IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, LEARNING MODEL GENERATION DEVICE, LEARNING MODEL PRODUCTION METHOD, LEARNED MODEL, AND PROGRAM

Abstract

To provide a technique capable of suppressing overlook of a small abnormality in an image and more accurately detecting the abnormality.SOLUTION: An image processing device detects an abnormality of an object by inputting a photographed image of the object to a learning model 410 that is machine-learned by using teacher data with each photographed image as the input and information about the abnormality in each photographed image as the output. The learning model 410 is the model which is machine-learned by using a value 270 obtained by generating a second feature map 250 integrated with a pixel value at the same position and / or neighboring position of each channel in a first feature map 230 on the basis of the first feature map 230 (being the intermediate feature map in a convolution neural network 200 and the feature map having a plurality of channels) and performing global max pooling processing on the second feature map 250.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an image processing device and a technique related thereto, and in particular to a technique for detecting an abnormality in an object using a captured image of the object. [Background technology]

[0002] There is a technology that uses a Convolutional Neural Network (CNN) to analyze images.

[0003] In a convolutional neural network, generally, a convolutional layer, a pooling layer, etc. are provided after an input layer, and a fully connected layer is provided at the end (see Patent Document 1, etc.). Also, as shown in Patent Document 1 (see paragraph 0040), etc., a global average pooling process (a process of calculating an average value of all channels for each channel in a feature map) may be performed before the fully connected layer. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-8896 A Summary of the Invention [Problem to be solved by the invention]

[0005] By the way, it is possible to use a convolutional neural network to detect the presence or absence of an abnormality (such as a scratch) in an object based on a captured image of the object.

[0006] For example, it is conceivable to apply a convolutional neural network such as that described in Patent Document 1 to the detection of abnormalities in an object.

[0007] However, when the above-mentioned global average pooling process is performed, the pixel values ​​of multiple pixels, including both abnormal and normal pixels, are averaged, which may result in small abnormalities (such as scratches) in the image being overlooked.

[0008] Therefore, an object of the present invention is to provide a technique that can prevent small abnormalities in an image from being overlooked and detect abnormalities more accurately. [Means for solving the problem]

[0009] In order to solve the above problem, the image processing device according to the present invention includes a control unit that detects an abnormality in a photographed image of an object based on a learning model that is machine-learned using teacher data in which each photographed image is input and information regarding an abnormality in each of the photographed images is output, and the learning model is ,tatami The present invention relates to a model that is machine-learned by generating a second feature map by integrating pixel values ​​at the same position and / or nearby positions of each channel in a first feature map, which is an intermediate feature map in a convolutional neural network and has a plurality of channels, and performing a global max pooling process on the second feature map, and using a value obtained by the second feature map, and the control unit detects an abnormality in the object by inputting the captured image of the object to the learning model.

[0010] The first feature map may be generated by an intermediate layer including a plurality of convolutional layers and a plurality of pooling layers arranged after an input layer in the convolutional neural network.

[0011] The convolutional neural network may not have a fully connected layer subsequent to a pooling layer that performs the global max pooling process on the second feature map.

[0012] The learning model may be a machine-learned model obtained by comparing the maximum value obtained by applying the global max pooling process to the second feature map, or a transformed value obtained by transforming the maximum value without multiplying the maximum value by a learning parameter, with correct data.

[0013] The learning model may be a machine-learned model obtained by performing the global max pooling process on the second feature map having two or more channels, converting each of the multiple maximum values ​​obtained for each channel without averaging the multiple maximum values, and comparing the converted values ​​with correct data.

[0014] The process of generating the second feature map may include a process of integrating pixel values ​​at the same pixel position in multiple channels of the first feature map using a 1×1 convolution filter that weights and adds the pixel values, and calculating a pixel value at each pixel position in the channel of the second feature map.

[0015] The process of generating the second feature map may include a process of integrating pixel values ​​at the same pixel position in multiple channels of a provisional feature map after convolution processing for each of the multiple channels of the first feature map by weighted addition, and calculating a pixel value at each pixel position in a channel of the second feature map.

[0016] The process of generating the second feature map may include a process of calculating a pixel value at each pixel position in one channel of the second feature map a predetermined number of times using a convolution filter that weights and adds pixel values ​​obtained by integrating pixel values ​​at the same position and / or adjacent positions in a plurality of channels of the first feature map, thereby generating a predetermined number of channels in the second feature map.

[0017] The second feature map may have two or more channels that are reduced in number compared to the number of channels of the first feature map, and the learning model may be a model machine-learned using values ​​obtained by applying the global max pooling process to at least a portion of the two or more channels.

[0018] The second feature map may be a feature map having a single channel, and the learning model may be a model machine-learned using a value obtained by performing the global max pooling process on the single channel.

[0019] The control unit may display information visualized based on the second feature map on a predetermined display unit.

[0020] In order to solve the above problem, the image processing method according to the present invention includes: a) detecting an abnormality in a photographed image of an object based on a learning model machine-learned using teacher data in which each photographed image is input and information on an abnormality in each of the photographed images is output, and the learning model ,tatami a model trained by machine learning using a value obtained by generating a second feature map by integrating pixel values ​​at the same position and / or nearby positions of each channel in a first feature map, which is an intermediate feature map in a convolutional neural network and has a plurality of channels, and performing a global max pooling process on the second feature map based on the second feature map; and wherein in the step a), the captured image of the object is input to the learning model to detect an abnormality in the object.

[0021] In order to solve the above problem, the present invention provides a program for causing a computer to execute the steps of: a) detecting an abnormality in a photographed image of an object based on a learning model machine-learned using teacher data in which each photographed image is input and information on an abnormality in each of the photographed images is output, and the learning model is ,tatamia model trained by machine learning using a value obtained by generating a second feature map by integrating pixel values ​​at the same position and / or nearby positions of each channel in a first feature map, which is an intermediate feature map in a convolutional neural network and has a plurality of channels, and performing a global max pooling process on the second feature map based on the second feature map; and wherein in the step a), the captured image of the object is input to the learning model to detect an abnormality in the object.

[0022] In order to solve the above problem, the learning model generation device of the present invention is a learning model generation device that inputs a captured image and generates a learning model that outputs information about abnormalities in the captured image, and is characterized in that it includes a control unit that generates a second feature map by integrating pixel values ​​at the same position and / or nearby positions of each channel in a first feature map (an intermediate feature map in a convolutional neural network of the learning model and a feature map having a plurality of channels) based on the first feature map, and performs machine learning in the learning model using values ​​obtained by performing a global max pooling process on the second feature map.

[0023] In order to solve the above problem, the learning model manufacturing method of the present invention is a learning model manufacturing method that inputs a captured image and produces a learning model that outputs information about anomalies in the captured image, and is characterized by comprising the steps of: a) generating a second feature map by integrating pixel values ​​at the same position and / or nearby positions of each channel in a first feature map (which is an intermediate feature map in a convolutional neural network of the learning model and has a plurality of channels) based on the first feature map, and performing machine learning in the learning model using values ​​obtained by performing a global max pooling process on the second feature map.

[0024] In order to solve the above problem, the trained model according to the present invention is: The trained model is machine-learned to output information regarding anomalies in an inference target image when an inference target image is input, and is characterized in that the trained model causes a computer to execute the following processes: a process of generating a first feature map for the inference target image, the first feature map being an intermediate feature map in a convolutional neural network and having a plurality of channels; a process of generating a second feature map by integrating pixel values ​​at the same position and / or nearby positions of each channel in the first feature map based on the first feature map; and a process of outputting information regarding the anomaly based on values ​​obtained by performing a global max pooling process on the second feature map.

[0025] In order to solve the above-mentioned problems, the present invention provides a program for causing a computer to execute the steps of: a) generating a second feature map by integrating pixel values ​​at the same position and / or nearby positions of each channel in a first feature map, the second feature map being an intermediate feature map having a plurality of channels in a convolutional neural network of a learning model that inputs a captured image and outputs information regarding anomalies in the captured image, and performing machine learning in the learning model by using values ​​obtained by performing a global max pooling process on the second feature map. Effect of the Invention

[0026] According to the present invention, it is possible to prevent small abnormalities in an image from being overlooked and to more accurately detect abnormalities. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 is a schematic diagram showing an image processing system. [Diagram 2] FIG. 2 is a conceptual diagram showing the processing in the learning stage of a learning model. [Diagram 3] FIG. 1 is a conceptual diagram showing the processing at the inference stage based on a learning model. [Figure 4] 10 is a flowchart showing the process of the controller in a learning stage. [Diagram 5] 13 is a flowchart showing the process of the controller in the inference stage. [Figure 6] FIG. 2 is a diagram showing a learning model according to the first embodiment. [Figure 7] FIG. 2 is a conceptual diagram illustrating an example of a configuration of a feature extraction layer. [Figure 8] FIG. 13 is a diagram showing how a second feature map is generated using a convolution filter of size 1×1. [Figure 9] FIG. 13 is a diagram showing how a second feature map is generated using a convolution filter of size N×N. [Figure 10] FIG. 13 is a diagram showing how a second feature map is generated using a convolution filter of size N×N. [Figure 11] FIG. 1 illustrates a convolution process within a channel. [Figure 12] FIG. 13 is a diagram showing how output values ​​are generated by GMP processing on a second feature map. [Figure 13] FIG. 13 is a diagram showing how output values ​​are generated by GMP processing and normalization processing on a second feature map. [Figure 14] FIG. 13 is a diagram showing GMP processing for one channel of the second feature map. [Figure 15] FIG. 11 is a diagram showing a learning model according to the second embodiment. [Figure 16] FIG. 2 shows how a second feature map 250 having C2 channels is generated using C2 convolution filters of size 1×1. [Figure 17] FIG. 2 shows how a second feature map 250 having C2 channels is generated using C2 convolution filters of size N×N. [Figure 18] FIG. 13 is a diagram showing how output values ​​are generated by GMP processing or the like for a second feature map having two channels. [Figure 19] FIG. 13 is a diagram showing how output values ​​are generated by GMP processing or the like for a second feature map having two channels. [Figure 20] FIG. 13 is a diagram showing a learning model according to the third embodiment. [Figure 21] FIG. 13 is a diagram showing how output values ​​are generated by performing GMP processing etc. on some of the five channels of the second feature map. [Figure 22] FIG. 13 is a diagram showing a screen for displaying a determination result regarding the presence or absence of an abnormality, etc. [Diagram 23] FIG. 13 is a diagram showing another screen for displaying the determination result regarding the presence or absence of an abnormality, etc. [Figure 24] FIG. 4 is a diagram illustrating an example of an input image. [Diagram 25]FIG. 13 is a diagram showing a learning model according to a comparative example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0029] <1. First embodiment> <1-1. System Overview> Fig. 1 is a schematic diagram showing an image processing system 1. As shown in Fig. 1, the image processing system 1 includes an imaging device 20 that captures an image of an object, and an image processing device 30 that processes the captured image. The image processing device 30 is also called an abnormality detection device (or an appearance inspection device) because it is a device that detects an abnormality in an object (e.g., a mechanical part) based on a captured image of the object (a device that inspects the appearance of the object). Similarly, the image processing system 1 is also called an abnormality detection system (or an appearance inspection system).

[0030] The image processing device 30 includes a learning model 410 (see FIG. 2). For example, a neural network model composed of multiple layers is used as the learning model 410. Then, weighting coefficients (learning parameters) between multiple layers (input layer, multiple intermediate layers, output layer) in the neural network model are adjusted by a predetermined machine learning method (deep learning, etc.).

[0031] The learning model 410 is a model that receives a captured image of an object (for example, a part such as a gear or a screw) as input and outputs information about an abnormality in the captured image (an abnormality of the object). In the machine learning (learning stage) of the learning model 410, for example, the presence or absence of an abnormality in each captured image (for example, a minute scratch on the surface of the object) is used as a label (correct answer data). In other words, a combination (dataset) of a captured image of the object and the presence or absence of an abnormality in the captured image is used as teacher data (labeled data) in the machine learning of the learning model 410. The presence or absence of an abnormality may be the presence or absence of one type of abnormality, or may be the presence or absence of each of multiple types of abnormalities (for example, "minute scratches," "minute tears," "minute stains," etc.).

[0032] The learning model 410 after being trained by machine learning is also referred to as a trained model 420. Specifically, the learning parameters of the learning model 410 (learning device) are adjusted using a predetermined machine learning method, and a trained learning model 410 (trained model 420) is generated (see FIG. 2).

[0033] In this embodiment, the image processing device 30 executes both the learning process of the learning model 410 (processing at the learning stage) and the inference process based on the learning model 410 (processing at the inference stage).

[0034] 2 is a conceptual diagram showing the processing of the learning stage of the learning model 410. As shown in FIG. 2, in the learning stage of the learning model 410, the image processing device 30 generates the learning model 410 by using teacher data (a large number of teacher data) in which each captured image is input and information on anomalies in each captured image (presence or absence of anomalies in the target object, etc.) is output. Note that the image processing device 30 is also a device that executes the processing of the learning stage (generation processing of the learning model 410), that is, a processing device that generates the learning model 410 (420), and is therefore also referred to as a learning model generation device.

[0035] Fig. 3 is a conceptual diagram showing the processing of the inference stage based on the learning model 410. As shown in Fig. 3, in the inference stage, the image processing device 30 detects an abnormality in a captured image of an object based on the learning model 410 that is machine-learned using the teacher data described above. Specifically, the image processing device 30 inputs a captured image of the object (an object to be determined whether or not there is an abnormality) to the learning model 410, thereby determining whether or not there is an abnormality in the object.

[0036] <1-2. Image processing device 30> As shown in FIG. 1, the image processing device 30 includes a controller 31 (also referred to as a control unit), a storage unit 32, a communication unit , and an operation unit .

[0037] The controller 31 is a control device that is built into the image processing device 30 and controls the operation of the image processing device 30 .

[0038] The controller 31 is configured as a computer system including one or more hardware processors (e.g., a central processing unit (CPU) and a graphics processing unit (GPU)). The controller 31 executes, in the CPU or the like, a predetermined software program (hereinafter also simply referred to as a program) stored in a storage unit (a non-volatile storage unit such as a ROM and / or a hard disk) 32, thereby realizing various processes. The program (more specifically, a group of program modules) may be recorded in a portable recording medium such as a USB memory, read out from the recording medium, and installed in the image processing device 30. Alternatively, the program may be downloaded via a communication network or the like and installed in the image processing device 30.

[0039] Specifically, the controller 31 executes a process (see FIG. 2) related to the learning stage of the learning model 410. Furthermore, the controller 31 executes a process (see FIG. 3) related to the inference stage using the learning model 410.

[0040] The storage unit 32 is configured with a storage device such as a hard disk drive (HDD) and / or a solid state drive (SSD), etc. The storage unit 32 stores a learning model 410 (including learning parameters and programs related to the learning model) (and thus a trained model 420) and the like.

[0041] The communication unit 34 is capable of performing network communication via a network. In this network communication, various protocols such as TCP / IP (Transmission Control Protocol / Internet Protocol) are used. By using the network communication, the image processing device 30 can exchange various data (teacher data (photographed image data, correct answer data, etc.)) with a desired counterpart (for example, the photographing device 20 or an information storage device (not shown)).

[0042] The operation unit 35 includes an operation input unit 35a that receives operation input to the image processing device 30, and a display unit 35b that displays and outputs various information. A mouse and a keyboard are used as the operation input unit 35a, and a display (such as a liquid crystal display) is used as the display unit 35b. A touch panel that functions as both a part of the operation input unit 35a and a part of the display unit 35b may also be provided.

[0043] <1-3. Learning model according to this embodiment> 6 is a diagram showing a learning model 410 according to the first embodiment. The learning model 410 is constructed using a convolutional neural network (CNN) 200.

[0044] 6, the learning model 410 has a hierarchical structure in which a plurality of layers (layers) are hierarchically connected. Specifically, the learning model 410 includes an input layer 310, a feature extraction layer 320, a convolutional layer 340, a pooling layer 360, and an output layer 370. In particular, the learning model 410 includes a pooling layer 360 that performs GMP processing (described later) immediately after the convolutional layer 340, and does not include a fully connected layer after the pooling layer 360 (on the output side).

[0045] <Input Layer 310> The input layer 310 is a layer that accepts the input image 210. Here, the input image 210 is a photographed image of an object (for example, a part such as a gear, or a finished product). For example, a color image (3 channels) having a pixel array (rectangular pixel array) with a width (horizontal) of W0 pixels and a height (vertical) of H0 pixels is input as the input image 210. In other words, the input image 210 is generated as voxel data of W0×H0×C0 (where C0=3).

[0046] 24 is a schematic diagram showing an example of an input image 210. In the input image 210 in FIG. 24, there is a minute "flaw" (scratch) in the lower left part (see the circle surrounded by a dashed line). Such an input image 210 is input to the input layer 310.

[0047] <Feature Extraction Layer 320> 6, the convolutional neural network 200 of the learning model 410 includes a feature extraction layer 320 following the input layer 310. The first feature map 230 is generated by processing the input image 210 through the feature extraction layer 320.

[0048] The feature extraction layer 320 is an intermediate layer including multiple convolution layers and multiple pooling layers (max pooling and / or average pooling, etc.). In this convolutional neural network 200, multiple intermediate layers are provided between the input layer 310 and the output layer 370. The feature extraction layer 320 is a part of the multiple intermediate layers. As the feature extraction layer 320, for example, a part (feature extraction part) of various convolutional neural network configurations (VGG, ResNet, etc.) can be used.

[0049] For example, a feature extraction layer (13 convolutional layers and 5 pooling layers) (see FIG. 7) provided just before the fully connected layer (3 layers) in VGG16 is provided as feature extraction layer 320 in convolutional neural network 200. In FIG. 7, a part (feature extraction part (just before the fully connected layer)) of the configuration of VGG16 (having 13 convolutional layers, 5 pooling layers, and 3 fully connected layers) is illustrated as feature extraction layer 320. Note that activation functions and the like are omitted as appropriate in FIG. 7.

[0050] Alternatively, all (or a part) of the feature extraction layer provided in other convolutional neural networks such as ResNet (Residual Network) (residual network) may be provided as the feature extraction layer 320 in the convolutional neural network 200. ResNet is a convolutional neural network that includes adding up residuals between layers. The feature extraction layer in ResNet is composed of a plurality of residual blocks, etc., which are composed of a combination of a convolutional layer, an activation function, and a skip connection (shortcut connection). Here, in a general convolutional neural network, a fully connected layer or the like is provided after the feature extraction layer as a layer (also called a classification layer) that performs classification processing based on the feature amount extracted in the feature extraction layer. All (or a part) of the feature extraction layer provided immediately before such a fully connected layer may be provided as the feature extraction layer 320 in the convolutional neural network 200.

[0051] The first feature map 230 is an intermediate feature map in the convolutional neural network 200 of the learning model 410. The first feature map 230 is generated as a feature map having a plurality of channels. The first feature map 230 is generated as three-dimensional array data (voxel data of W1×H1×C1) having C1 channels each composed of two-dimensional array data of a pixel array (rectangular pixel array) having a width of W1 pixels and a height of H1 pixels. The number of channels C1 of the first feature map 230 is, for example, 512. However, the number is not limited to this and may be 256, 1024, or the like.

[0052] Here, the feature extraction layer 320 is configured by repeatedly arranging one or more convolution layers and one pooling layer. In each convolution layer, features in an image are extracted by a filter that performs convolution processing. In addition, in each pooling layer, processing (maximum pooling processing, etc.) is performed to extract the maximum pixel value for each small pixel range (for example, a 2 x 2 pixel range), and the pixel size is reduced (for example, 1 / 2 in each vertical and horizontal direction) (the amount of information is condensed).

[0053] Then, the first feature map 230 is generated by performing the processing (convolution processing and pooling processing) by the feature extraction layer 320 on the input image 210. In this manner, the first feature map 230 is generated by an intermediate layer including a plurality of convolution layers and a plurality of pooling layers provided after the input layer 310 in the convolutional neural network 200. According to this, image features (such as minute scratches) in the input image 210 are extracted in a state where their rough positions are maintained in the two-dimensional image of each channel in the first feature map 230. Note that in the pooling layer included in the feature extraction layer 320, a maximum pooling process (or an average pooling process) or the like is executed. In order to more appropriately avoid missing detection of minute scratches, it is preferable to perform a maximum pooling process.

[0054] <Convolutional Layer 340> Furthermore, in the learning model 410 in the first embodiment, the second feature map 250 is generated by performing a convolution process on the first feature map 230. In other words, the convolution layer 340 is provided next to the feature extraction layer 320, and the second feature map 250 is generated based on the first feature map 230. According to such a process, image features (such as minute scratches) in the input image 210 are extracted in a state where their rough positions are still maintained in the two-dimensional image of each channel in the second feature map 250. The convolution layer 340 is also expressed as being a part of the intermediate layer in the convolutional neural network 200.

[0055] The second feature map 250 is generated by integrating (by weighting, etc.) pixel values ​​at the same position and / or nearby positions of each channel in the first feature map 230. The second feature map 250 is generated, for example, as three-dimensional array data (voxel data of W2×H2×C2) having C2 channels each composed of two-dimensional array data of pixel arrays (rectangular pixel arrays) each having a width of W2 pixels and a height of H2 pixels. The second feature map 250 is generated by using C2 filters (described later) of size N×N×C1.

[0056] However, in the first embodiment, a case where the second feature map 250 has a single channel (i.e., a case where C2=1) is exemplified. Also, a case where W2=W1 and H2=H1 is mainly exemplified. A case where the second feature map 250 has multiple channels will be described in the second embodiment.

[0057] Each of the values ​​W0, H0, W1, H1, W2, H2, C0, C1, and C2 can be set to an appropriate value depending on the configuration of the feature extraction layer 320. For example, W0=H0=512, W1=H1=16, W2=H2=16, C0=3, C1=512, and C2=1.

[0058] 8 and 9 are diagrams showing how a second feature map 250 (here, a second feature map 250 having a single channel) is generated by a convolution process on the first feature map 230. FIG.

[0059] In the convolution process for the first feature map 230, for example, a filter (341, 342, etc.) as shown in FIG. 8 or FIG. 9 is used to generate the second feature map 250.

[0060] Fig. 9 shows how the second feature map 250 is generated by a convolution filter 342 with a size of NxN (for example, N=3). Meanwhile, Fig. 8 shows how the second feature map 250 is generated by a convolution filter (kernel) 341 with a size of 1x1. Note that the convolution filter 341 in Fig. 8 can also be expressed as one aspect (when N=1) of the convolution filter 342 in Fig. 9. However, for convenience, the description will be given here in two parts (when N is 1 and when N is 2 or more).

[0061] The size in the depth direction of both the convolution filters 341 and 342 is a value C1 (the number of channels of the first feature map 230). In other words, the filter 341 has a size (three-dimensional size) of 1×1×C1, and the filter 342 has a size (three-dimensional size) of N×N×C1. Note that the value N is not limited to "3" and may be another value such as "5".

[0062] First, with reference to FIG. 8, a process for generating the second feature map 250 using a convolution filter 341 having a size of 1×1 (plane size) will be described.

[0063] Specifically, the convolution filter 341 in Fig. 8 has a size of 1 x 1 x C1. The convolution filter 341 is a filter that performs weighted addition of pixel values ​​at the same pixel position in multiple (C1) channels of the first feature map 230. Specifically, the convolution filter 341 is configured as a collection of C1 weighting coefficients (weighting coefficients for each channel) for the (i, j) pixel of each channel of the first feature map 230 in the channel direction. Note that the channel direction is the direction connecting the channels (the depth direction in Fig. 8) and is also referred to as the inter-channel direction.

[0064] Then, a value D2(i,j) is obtained by applying the filter 341 to a total of C1 pixel values ​​D1(i,j,h) (h=1,...,C1) existing at pixel positions (i,j) of each of the multiple channels of the first feature map 230. Here, h is the channel number. Specifically, first, the product is calculated by multiplying the pixel value D1(i,j,h) of the (i,j) pixel of the h-th channel of the first feature map 230 by the corresponding weighting coefficient W(h) in the convolution filter 341. Then, the value D2(i,j) is calculated as the sum (product sum value) of the products for the C1 channels (see equation (1)).

[0065]

number

[0066] The sum-of-products value D2(i,j) is calculated as the pixel value of the (i,j) pixel of the second feature map 250 (the first channel of the second feature map 250). In other words, the pixel values ​​of the same position (i,j) of each channel of the first feature map 230 are integrated to calculate the pixel value of the (i,j) pixel of the second feature map 250. In short, the weighted average (or a constant multiple value) of the pixel values ​​of the same position (i,j) of each channel of the first feature map 230 is calculated as the pixel value of the (i,j) pixel of the second feature map 250. Note that the value D2(i,j) of the k-th channel of the second feature map 250 is also expressed as the value D2(i,j,k). For example, the value D2(i,j) of the first channel of the second feature map 250 is also expressed as the value D2(i,j,1) (k=1 in the first embodiment).

[0067] In this manner, a process is executed to calculate a pixel value at a certain pixel position (i, j) in the second feature map 250 (specifically, the first channel (k=1)) based on the convolution filter 341 and pixel values ​​at a certain pixel position (i, j) in multiple channels of the first feature map 230. As a result, pixel values ​​(multiple pixel values) at the same position in multiple channels of the first feature map 230 are condensed into a pixel value at a corresponding pixel position in the second feature map 250.

[0068] Then, such processing is similarly performed for each of the pixels at all other pixel positions in the first feature map 230. In this way, pixel values ​​at all other pixel positions in the second feature map 250 (specifically, the single channel thereof) are calculated.

[0069] According to such a convolution process, pixel information at each position (the same position) across a plurality of channels is condensed into the value of the pixel at that position within a single new channel.

[0070] Next, with reference to FIG. 9, a process for generating the second feature map 250 using the convolution filter 342 having a size (plane size) of (N×N) will be described.

[0071] 9 has a size (three-dimensional size) of N×N×C1 in detail. The convolution filter 342 is a filter that performs weighted addition of pixel values ​​at the same pixel position and its neighboring (surrounding) pixel positions in multiple (C1) channels of the first feature map 230.

[0072] The convolution filter 342 is configured as a set of C1 weighting coefficients (weighting coefficients for each channel) for (N×N) pixels in a neighborhood centered on the (i,j) pixel of each channel of the first feature map 230 in the channel direction (depth direction in FIG. 9). Each weighting coefficient for each channel is composed of (N×N) (e.g., 3×3) weighting coefficients (see also FIG. 11). The weighting coefficient for each channel is composed of independent values ​​for each channel. In other words, each value of the weighting coefficient for a certain channel is basically different from each value of the weighting coefficient for another channel.

[0073] Then, a value D2(i,j) is obtained by applying a filter 342 of the same size to a voxel range of size N×N×C1 (a total of (N×N×C1) pixel values ​​existing in the three-dimensional area indicated by the dashed rectangular prism in Figure 9) centered on each pixel position (i,j) of the multiple channels of the first feature map 230.

[0074] This filter processing can be divided into processing within a channel and processing between channels (see FIG. 10). Like FIG. 9, FIG. 10 is a diagram showing the generation processing of the second feature map 250 using a convolution filter 342 of size (planar size) (N×N). However, the three-dimensional feature map and three-dimensional filter in FIG. 9 are broken down into channels (as a planar collection) and shown in FIG. 10. The left-right direction in FIG. 10 corresponds to the depth direction (channel direction) in FIG. 9.

[0075] Specifically, using the convolution filter 342, first, intra-channel weighting processing is performed on pixel values ​​of (N×N) pixels (including the central (i, j) pixel) in the vicinity of the (i, j) pixel in the h-th channel of the first feature map 230. In other words, (planar) convolution processing is performed on each of the multiple channels of the first feature map 230. As a result, a provisional feature map (also referred to as a provisional feature map) 245 is generated, as shown near the center in the horizontal direction of FIG. 10. The pixel values ​​of pixels in the vicinity range (dashed line area) of the position (i, j) of the h-th channel of the first feature map 230 in FIG. 10 are convoluted to obtain a pixel value D3(i, j, h) of the pixel in the position (i, j) of the provisional feature map 245. The pixel value D3(i, j, h) is the pixel value of the (i, j) pixel in the h-th channel of the provisional feature map 245 (see also FIG. 10).

[0076] Then, a value D2(i,j) is obtained by summing C1 pixel values ​​D3(i,j,h) (where h is a channel number, h=1, .., C1) for C1 channels of the temporary feature map 245 (see equation (2)). In other words, integration processing (summing processing) between channels is performed.

[0077]

number

[0078] Such a value D2(i,j) is calculated as the pixel value of the (i,j) pixel of the second feature map 250 (the first channel of the second feature map 250).

[0079] Here, the value D3(i,j,h) can also be expressed as the product of the in-plane component D4(i,j,h) and the inter-plane component W4(h). Taking this into consideration, the above formula (2) can be transformed into the following formula (3).

[0080]

number

[0081] In other words, planar convolution processing is performed in each channel using the intra-plane components D4(i,j,h) of the convolution filter 342 to obtain values ​​D4(i,j,h) of the provisional feature map 245. Furthermore, the pixel values ​​D4(i,j,h) (where h is a channel number, h=1, .., C1) are multiplied by the corresponding weighting coefficients W4(h) (inter-plane components) in the convolution filter 342 to obtain products. Then, the products for the C1 channels are summed up to obtain a value D2(i,j) (see equation (3)).

[0082] That is, pixel values ​​D4(i,j,h) obtained by convolving pixel values ​​in the same neighborhood range centered on the same pixel position (i,j) in multiple channels of the provisional feature map 245 are weighted and added together to be calculated as the pixel value of the (i,j) pixel in the second feature map 250. In short, the weighted average (or a constant multiple of that) of the pixel values ​​D4(i,j,h) is calculated as the pixel value of the (i,j) pixel in the second feature map 250.

[0083] In this case, it is assumed that the convolution process is performed on the first feature map 230 after padding (zero padding, etc.) is performed to fill the outer peripheral region of the peripheral pixels of each channel of the first feature map 230 with an appropriate value (for example, zero (or the same value as the peripheral pixels)). This makes it possible to prevent the pixel size (W×H) of the feature map (second feature map 250) after the convolution process from becoming small. According to the convolution process involving such padding, a situation of W2=W1 and H2=H1 can be realized. However, it is not limited to W2=W1 and H2=H1. For example, the plane size (W2×H2) of the second feature map 250 may be smaller than the plane size (W1×H1) of the first feature map 230. In FIG. 11, for the sake of simplification of the illustration, etc., the case of W2=W1=6 and H2=H1=6 is shown.

[0084] In this manner, a process is executed to calculate a pixel value of a corresponding pixel position (i, j) in the second feature map 250 (specifically, the kth channel) based on the convolution filter 342 and pixel values ​​of a certain pixel position (i, j) and its neighboring positions in the multiple channels of the first feature map 230. As a result, pixel values ​​(multiple pixel values) in the same range in the multiple channels of the first feature map 230 are condensed into the pixel value of the corresponding pixel position in the second feature map 250.

[0085] Then, such processing is similarly performed for pixels at all other pixel positions of each channel of the first feature map 230. In this way, pixel values ​​at all other pixel positions of the second feature map 250 (specifically, its k-th channel (k=1)) are calculated.

[0086] According to such a convolution process, information about pixels near each position (near the same position) across a plurality of channels is condensed into the value of the pixel at that position within a single new channel.

[0087] As described above (as shown in FIG. 8 or FIG. 9), the second feature map 250 is generated by integrating pixel values ​​at the same position and / or nearby positions in each channel in the first feature map 230. Specifically, a process of calculating pixel values ​​at all pixel positions in the first channel of the second feature map 250 (a process of calculating intra-channel pixel values) is executed once. This calculation process is performed using a convolution filter 341 or 342. The convolution filter 341 or 342 is a filter that performs weighted addition of pixel values ​​at the same position and / or nearby positions in multiple (C1) channels of the first feature map 230. In this way, a predetermined number (1) of channels in the second feature map 250 are generated.

[0088] It is preferable that each value in the second feature map 250 is further normalized and corrected (updated) by an activation function or the like.

[0089] <Pooling Layer 360 and Output Layer 370> Next, a global max pooling process is performed on the second feature map 250, and a score (output value) 270 of the learning model 410 is calculated (see FIG. 6). In other words, a pooling layer 360 is provided next to the convolution layer 340, and a score 270 (output value of the output layer 370) of the learning model 410 is generated. The pooling layer 360 is a layer that performs a global max pooling (GMP) process (hereinafter also referred to as a GMP process) on the second feature map 250. The pooling layer 360 is also referred to as a GMP layer. The pooling layer (GMP layer) 360, like the feature extraction layer 320 and the convolution layer 340, is also expressed as being a part of the intermediate layer in the convolution neural network 200.

[0090] Max Pooling is a process that extracts the maximum value (maximum pixel value) of multiple pixels corresponding to the kernel (filter) size as a feature value (output value). In general, the maximum pooling process often extracts the maximum value of multiple pixels (e.g., four pixels) corresponding to a filter size (e.g., 2×2 size) smaller than the channel size.

[0091] On the other hand, Global Max Pooling (GMP) processing is a type of Max Pooling processing, specifically, a Max Pooling processing targeting "the whole channel" (see FIG. 14). GMP processing (global max pooling processing) is a Max Pooling processing that extracts the maximum value among a plurality of pixels (all pixels in a channel) (for example, 256 pixels) corresponding to a filter size (for example, W2×H2=16×16) that is the same as the channel size.

[0092] Moreover, GMP processing (global max pooling processing) is processing that extracts (for each channel) the maximum pixel value for each channel of the feature map to be processed (here, the second feature map 250). When GMP processing is performed on a feature map having a single channel, a single value (maximum value) is output. On the other hand, when GMP processing is performed on a feature map having multiple (e.g., two) channels, multiple values ​​(maximum values ​​for each channel) (for example, two values, the maximum value in the first channel and the maximum value in the second channel) are output.

[0093] When the GMP processing is performed on the second feature map 250 having a single channel, the maximum value (maximum value) of a plurality of ((W2×H2)) pixel values ​​in the two-dimensional array data (W2×H2) of the single channel is output as the processing result of the GMP processing (see FIG. 14). FIG. 14 is a diagram showing the GMP processing for one channel of the second feature map. For the sake of simplicity, FIG. 14 illustrates a case where the second feature map 250 has a size of 6×6. In the case of FIG. 14, that is, when W2=H2=6, the maximum value of 6*6=36 pixel values ​​is output as the processing result of the GMP processing for (the first channel of) the second feature map 250. Alternatively, for example, when W2=H2=16, the maximum value of 16*16=256 pixel values ​​is output as the processing result of the GMP processing for (the first channel of) the second feature map 250.

[0094] If each value in the second feature map 250 has already been normalized, the value resulting from the GMP processing (a value between 0 and 1) is output as is as the output value 270 of the convolutional neural network 200 (output layer 370) (see FIG. 12).

[0095] On the other hand, if each value in the second feature map 250 has not yet been normalized (for example, each value is one of 8-bit values ​​(0 to 255)), the processing result of the GMP processing is normalized using a predetermined normalization function (activation function, etc.) (see FIG. 13). As a result, the processing result of the GMP processing is normalized to a value between 0 and 1. Note that the activation function may be a sigmoid function, a ReLU (Rectified Linear Unit) function, or the like.

[0096] The learning model 410 has the above-mentioned configuration, and when a captured image (input image) 210 of an object is input, an output value indicating information regarding an abnormality of the object (presence or absence of an abnormality, etc.) is output. As will be described later, in the learning stage, the normalized output value (output value of the learning model 410) is compared with the correct answer data, and machine learning of the learning model 410 is executed so as to reduce the difference between the two (difference value, etc.). In the inference stage, whether the object is normal or abnormal is determined based on the normalized output value (output value of the learning model 410).

[0097] Here, in the GMP layer 360, the GMP processing is performed on the second feature map 250. According to this, in the learning model 410 (420) after learning, if there is even one pixel (value close to "1") that is considered abnormal among the multiple (W2×H2) pixel values ​​of the second feature map 250, the value of that pixel (abnormal pixel) is output to the output value 270. Therefore, it is possible to prevent abnormal pixels from being overlooked.

[0098] <1-4. Learning stage processing> Next, the learning stage processing of the learning model 410 will be described with reference to FIG.

[0099] Fig. 4 is a flowchart showing the processing of the controller 31 in the learning stage. Fig. 4 is also a diagram showing a method for generating a trained model. In this application, generating the trained model 420 means manufacturing (producing) the trained model 420, and the "method for generating a trained model" means the "method for manufacturing a trained model".

[0100] 4, the controller 31 first acquires training data in step S11, which receives each captured image and outputs information about abnormalities in each captured image. For example, training data (plurality of training data) is acquired in which information indicating whether or not a "microscopic scratch" exists on the object of each captured image (for example, abnormal "1" / normal "0") is associated with each captured image as correct answer data. In particular, it is preferable to prepare a very large amount of training data.

[0101] Then, in step S12, the controller 31 performs machine learning of the learning model 410 as described above, using the teacher data acquired in step S11.

[0102] More specifically, as shown in FIG. 12 (and FIG. 13), the output value (normalized output value (a value between "0" and "1")) from the learning model 410 is compared with the correct answer data (for example, "0" for normal and "1" for abnormal). Then, machine learning of the learning model 410 is performed so as to reduce the difference between the two (such as a differential value). In other words, machine learning of the learning model 410 is performed using a value (output value 270) obtained by performing GMP processing on a single channel of the second feature map 250, and a trained model 420 is generated.

[0103] Specifically, the learning parameters (weighting coefficients and bias values ​​of each filter (kernel) and the like) of the learning model 410 are optimized using an error backpropagation method or the like. More specifically, the weighting coefficients and bias values ​​of each filter (kernel) of the feature extraction layer 320 and the weighting coefficients and bias values ​​of each filter (kernel) of the convolution layer 340 and the like are optimized.

[0104] Through such processing, a learning model 410 that has learned the relationship between each captured image and information about anomalies in the captured image (for example, the presence or absence of "microscopic scratches") is generated (step S13). That is, a trained learning model 410 (trained model 420) is generated.

[0105] According to this, as described below, by using the trained model 420, it is possible to estimate the presence or absence of abnormalities (such as "tiny scratches") in the subject (object) in a captured image based on the captured image of the object to be judged.

[0106] <1-5. Inference stage processing> Next, the processing of the inference stage using the trained model 420 will be described with reference to FIG.

[0107] Fig. 5 is a flowchart showing the processing of the controller 31 in the inference stage. Fig. 5 is also a diagram showing an anomaly detection method (such as an image processing method for anomaly detection) using a trained model.

[0108] First, in step S31, the controller 31 acquires a photographed image of an object. Specifically, the controller 31 acquires a photographed image (also referred to as a determination target image) captured by the photographing device 20 from the photographing device 20 via a network or the like.

[0109] In the next step S32, the controller 31 inputs the captured image acquired in step S31 to the learning model 410 (trained model 420). Then, the controller 31 acquires an output result from the trained model 420. The output result is an estimation result (inference result) regarding the presence or absence of an abnormality in the target object, etc.

[0110] When a value equal to or greater than the threshold value TH1 ("0.5") is output as the normalized output result (output value), it is determined that an abnormality exists in the object in the captured image. A normalized output result (output value) close to "1" (abnormal) indicates that the possibility of an abnormality is high (in short, that an abnormality exists). Conversely, an output result close to "0" (normal) indicates that the possibility of an abnormality is low (that an abnormality does not exist). In addition, the controller 31 also outputs the numerical value itself (or a converted value) related to the output result as the reliability of the determination result.

[0111] For example, when the output result value from the trained model 420 is "0.9", it is determined that a minute scratch is present and the reliability (reliability regarding the presence or absence of anomalies in the entire input image 210) is 90%. When the output result value from the trained model 420 is "0.2", it is determined that a minute scratch is not present and the reliability is 80% (=(1-0.2)*100(%)).

[0112] Then, in step S33, the controller 31 displays a screen 510 showing the determination result regarding the presence or absence of an abnormality on the display unit 35b. The screen 510 includes information based on the output result acquired in step S32.

[0113] Fig. 22 is a diagram showing an example of a screen 510. Fig. 22 shows a screen 510 (also referred to as 511) on which the input image 210 and the heat map of the second feature map 250 are arranged in the left-right direction. The screen 510 also displays information visualized based on the second feature map 250.

[0114] The input image 210 itself is displayed on the left side of the screen 511. Furthermore, in the screen 511, information regarding the presence or absence of an abnormality is displayed in a display area 520 above the input image 210. In FIG. 22, the character string "Abnormal (Scratch)" is displayed. This indicates that an abnormality ("scratch") exists in the object of the captured image. In detail, when the reliability regarding the presence or absence of an abnormality for the entire input image 210 is equal to or greater than a threshold value (for example, 50%), it is determined that "an abnormality exists." Note that, although the numerical value of the reliability is not displayed here, this is not limiting, and the numerical value of the reliability may also be displayed.

[0115] A heat map 530 of the second feature map 250 (visualized information of the second feature map 250) is displayed on the right side of the screen 511. Each pixel in the second feature map 250 has a pixel value indicating the possibility of the presence of an abnormality at the position of the pixel. It can also be said that each pixel value indicates the reliability of the pixel in the second feature map 250 (reliability regarding the presence or absence of an abnormality on a pixel-by-pixel basis).

[0116] In the heat map 530, the pixel value of each pixel in the second feature map 250 is divided into a plurality of stages, and the rectangle of each pixel is displayed by filling it with a color assigned to each stage. For example, pixels with a pixel value of 0.8 or more and 1.0 or less are displayed in red (dotted hatched area in FIG. 22), pixels with a pixel value of 0.5 or more and less than 0.8 are displayed in orange (diagonal hatched area in FIG. 22), and pixels with a pixel value less than 0.5 are displayed in white (colorless and transparent). The operator can identify the possibility of an abnormality (presence or absence of an abnormality, etc.) at each pixel position by the color of each pixel in the heat map 530.

[0117] The heat map 530 of the second feature map 250 is displayed to have the same size as the input image 210 (with its size adjusted).

[0118] The minimum rectangle of the heat map 530 corresponds to one pixel of the second feature map 250. Moreover, the minimum rectangle has a size equivalent to (W0 / W2) pixels of the input image 210 on each side. For example, when W0=H0=512 and W2=H2=16, one side of the minimum rectangle of the heat map 530 of the second feature map 250 has a size equivalent to 32 pixels (=512 / 16) of the input image 210.

[0119] According to such display information on screen 510 (information visualized based on second feature map 250), the operator can easily know the rough position of the abnormality in input image 210. In detail, by comparing input image 210 with heat map 530 (second feature map 250), it is possible to easily grasp at which position in input image 210 the abnormality exists. More specifically, it is found that the abnormality in input image 210 exists near a position corresponding to a red pixel position (or an orange pixel position) in heat map 530.

[0120] Note that, here, the input image 210 and the second feature map 250 (specifically, the heat map 530 thereof) are displayed in parallel arrangement, but the present invention is not limited to this.

[0121] For example, information (such as a heat map 530) that visualizes the second feature map 250 may be displayed superimposed on the input image (captured image) 210. FIG. 23 is a diagram showing such a screen 510 (512) or the like. In the screen 512 on the right side of FIG. 23, a heat map 530 (see the lower left in FIG. 23) having the same size as the input image 210 (see the upper left in FIG. 23) is displayed superimposed on the input image 210. In such a superimposed display or the like, it is preferable that the heat map 530 is displayed in a semi-transparent color.

[0122] Alternatively, a superimposed display state of the two images 210, 530 as shown in FIG. 23 (see screen 510) and a single display state of the input image 210 may be alternated. In particular, the input image 210 may be constantly displayed, and the presence or absence of the display of the heat map 530 may be switched at a predetermined time interval (such as every second). That is, a display state of only the input image 210 and a display state of the heat map 530 superimposed on the input image 210 may be alternately repeated. Alternatively, the input image 210 and the heat map 530 having the same size may be alternately displayed at the same position.

[0123] In addition, although pixels having a predetermined value or more in the second feature map 250 are filled with a predetermined color here, the present invention is not limited to this. For example, the information included in the second feature map 250 (such as the degree of abnormality at each pixel position) may be visualized and displayed by surrounding an area having pixels having a predetermined value or more in the second feature map 250 with a line.

[0124] As described above, in the inference stage, a captured image (input image 210) of an object is input to the learning model 410 (420) to detect an abnormality in the object.

[0125] <1-6. Comparative Example> Next, a learning model 910 according to a comparative example will be described.

[0126] 25 is a diagram showing a learning model 910 according to a comparative example. This learning model 910 has a similar configuration to the learning model 410 according to the first embodiment up to the generation of the first feature map 230. The learning model 910 is a convolutional neural network (CNN) having a general configuration.

[0127] In the learning model 910, after the generation of the first feature map 230, a process different from that of the learning model 410 is performed.

[0128] Specifically, in the learning model 910, a global average pooling (GAP) process (hereinafter also referred to as GAP process) is performed on the first feature map 230. In other words, the feature extraction layer 320 is followed by a pooling layer 940 that executes the GAP process.

[0129] GAP processing is an average pooling process that targets "all channels." GAP processing is a process that extracts (for each channel) the average pixel value for each channel of the target feature map (here, the first feature map 230). The average value of (W1×H1) pixels (1024 pixels when W1=H1=32) in each channel is extracted for each channel. If the first feature map 230 has C1 channels (for example, 512 channels), C1 average values ​​(average values ​​for each channel) are extracted.

[0130] Then, these C1 average values ​​are arranged one-dimensionally (converted into one-dimensional vector format) (see one-dimensional array 950) and passed to the fully connected layer 960. The fully connected layer 960 has a multiple hierarchical structure. In other words, multiple fully connected layers 960 are provided. Furthermore, an output layer 970 is provided next to the fully connected layer 960. The output layer 970 has nodes according to the number of classification destinations, and an output value is calculated for each node.

[0131] <1-7. Effects of the embodiment> In the above-described comparative example, the GAP process is performed on the first feature map 230. GAP (global average pooling) process is useful for capturing the overall features within a channel.

[0132] However, when GAP processing is used to detect abnormalities (especially minute scratches, etc.) in a captured image, there is a high possibility that information on minute scratches, etc. in each channel of the first feature map 230 will be lost. This is because the pixel values ​​of multiple pixels, including both abnormal pixels and normal pixels, are averaged. In particular, when a small number of abnormal pixels exist among a large number of normal pixels, there is a high possibility that information on the small number of abnormal pixels will be lost by the averaging process. Therefore, there is a possibility that minute abnormalities (such as scratches) in the image will be overlooked. For example, if GAP processing is performed on the second feature map 250 in FIG. 14, the output value will be a very small value (0.039 (=(1+0.4) / 36)). As a result, it will be determined that there is no abnormality.

[0133] On the other hand, in the first embodiment, as shown in FIG. 6 and FIG. 14, etc., GMP (global maximum pooling) processing is performed on the second feature map 250. According to the GMP processing, if there is even one abnormal pixel among a plurality of pixels ((W2×H2) pixels) in one channel in the second feature map 250, the value of the abnormal pixel is output as the output value 270. For example, if there is even one pixel having a pixel value indicating an abnormality (for example, a pixel value close to "1" in a normalized state) among all 256 pixels in one channel in the second feature map 250, the pixel value close to "1" is output as the output value 270. That is, the abnormal information remains as the output value 270 without disappearing (the maximum value close to "1" is output). Therefore, it is possible to prevent abnormal pixels from being overlooked.

[0134] In this way, it is possible to detect abnormalities such as minute scratches without overlooking them.

[0135] Here, if GMP processing is used, there is a possibility that the system may overreact to minute scratches (pick up noise, etc.). However, in detecting abnormalities such as minute scratches, it is preferable to avoid overlooking an abnormality rather than overdetecting an abnormality (detecting a normal state as abnormal). In other words, it is preferable to overdetect an abnormality with GMP processing rather than overlooking an abnormality (due to the GAP processing, etc.).

[0136] The above-mentioned learning model 410 is machine-trained using values ​​obtained by performing a global max pooling (GMP) process on the second feature map 250, and therefore it is possible to prevent abnormality information in the second feature map 250 from being lost due to an averaging process. Therefore, compared to a case where an averaging process is performed on the second feature map 250, it is possible to suppress overlooking minute abnormalities in an image and detect abnormalities more accurately.

[0137] In the above embodiment, a fully connected layer is not provided after the pooling layer (GMP layer) 360 that performs the GMP process (on the output side or in the subsequent stage). In other words, the learning model 410 does not have a fully connected layer after the GMP layer 360 (in the subsequent process). Here, the fully connected layer is a layer in which a process is performed to output, as an output value to each output node, an inner product (or a value obtained by further adding a bias to the inner product) between an input value vector in which all the input values ​​are arranged one-dimensionally and a weighting vector in which weighting coefficients for each input value are arranged one-dimensionally. The inner product is also expressed as a product-sum value obtained by adding (adding) values ​​obtained by multiplying each of the input values ​​by a weight (weighting coefficient). Note that the convolution layer differs from the fully connected layer in that the convolution layer is a layer in which a product-sum value (inner product) is obtained for some of the input values ​​(only pixel values ​​in the vicinity of a certain pixel position) among all the input values.

[0138] On the other hand, if a fully connected layer is placed after the GMP layer 360, optimization may be performed in the fully connected layer during machine learning. In that case, the immediately preceding position in the second feature map 250 and the abnormality position in the input image 210 may not be extracted in correspondence with each other. In other words, the pixel value distribution in the second feature map 250 may no longer have a direct correlation with the abnormality position in the input image 210.

[0139] In contrast, since no fully connected layer is provided after the pooling layer 360 that performs the GMP process, it is highly likely that the abnormality positions in the second feature map 250 maintain a corresponding relationship with the abnormality positions in the input image 210. Therefore, it is possible to accurately extract rough information about the abnormality positions in the input image 210.

[0140] In other words, in the learning model 410 of the above embodiment, the value (maximum value in the channel of the second feature map 250) obtained by performing the GMP process on the second feature map 250 is compared with the correct answer data, and the machine learning of the learning model 410 is executed. Alternatively, a converted value (a value normalized by an activation function (e.g., a sigmoid function) or the like) obtained by converting the maximum value without multiplying the maximum value by a learning parameter is compared with the correct answer data, and the machine learning of the learning model 410 is executed. That is, an output value output without multiplying (without intervening) the maximum value obtained by performing the GMP process on the second feature map 250 by a learning parameter is compared with the correct answer data, and the machine learning of the learning model 410 is executed. Therefore, it is highly likely that the abnormality position in the second feature map 250 maintains a correspondence relationship with the abnormality position in the input image 210. Therefore, it is possible to accurately extract rough abnormality position information in the input image 210.

[0141] In short, since the output layer 370 is provided immediately after the GMP layer 360, the output value from the GMP layer 360 is output almost as it is (either as it is or a value converted by an activation function or the like) as the output value of the learning model 410. This makes it possible to maintain the correspondence with the anomaly positions in the input image 210 in the second feature map 250 immediately before the GMP layer 360.

[0142] Furthermore, as described above, the image processing device 30 displays, on the display unit 35b, the information visualized based on the second feature map 250. Therefore, the user (operator) can easily grasp the rough position of the abnormality in the input image 210.

[0143] <2. Second embodiment> In the above-described first embodiment, the second feature map 250 has a single channel, but is not limited thereto. For example, the second feature map 250 may have a plurality of channels. In the second embodiment, such an aspect will be described.

[0144] Fig. 15 is a conceptual diagram showing a learning model 410 (also referred to as 410B) according to the second embodiment. As shown in Fig. 15, the second feature map 250 of the learning model 410B has multiple (C2) channels.

[0145] In the above first embodiment, as shown in FIG. 8 or FIG. 9, a second feature map 250 having only a single channel is generated using one convolution filter 341 of size 1×1×C1 or one convolution filter 342 of size N×N×C1.

[0146] On the other hand, in this second embodiment, as shown in Figure 16 or Figure 17, a second feature map 250 having C2 (multiple) channels is generated using C2 convolution filters 341 of size 1x1xC1 or C2 convolution filters 342 of size NxNxC1.

[0147] First, with reference to FIG. 16, a process in which the second feature map 250 having C2 channels is generated using a convolution filter 341 having a size of 1×1×C1 will be described.

[0148] The k-th convolution filter 341 (where k=1,...,C2) of the C2 convolution filters 341 of 1x1xC1 size is a filter for calculating pixel values ​​at each pixel position of the k-th channel of the second feature map 250. The k-th convolution filter 341 is a filter that performs weighted addition of pixel values ​​at the same position in multiple (C1) channels of the first feature map 230 (a filter that integrates pixel values ​​at the same position). Note that the C2 convolution filters 341 are basically configured with values ​​different from each other.

[0149] The k-th channel of the second feature map 250 is generated by the k-th convolution filter 341 in the same manner as in the first embodiment. In detail, a process of calculating a pixel value of a corresponding pixel position (i, j) of the second feature map 250 (specifically, the k-th channel) is executed based on the k-th convolution filter 341 and pixel values ​​of a certain pixel position (i, j) in a plurality of channels of the first feature map 230. Then, such a process is similarly executed for pixels of all other pixel positions of the first feature map 230. In this way, pixel values ​​of all other pixel positions of the second feature map 250 (specifically, the k-th channel) are calculated.

[0150] More specifically, first, the pixel values ​​of all pixel positions in the first channel of the second feature map 250 are calculated using the first convolution filter 341 in the same manner as in the first embodiment.

[0151] Next, the pixel values ​​of all pixel positions in the second channel of the second feature map 250 are calculated in a similar manner using the second convolution filter 341.

[0152] Thereafter, the same operation (the process of generating the k-th channel of the second feature map 250 using the remaining k-th convolution filter 341) is executed while incrementing the value k until the value k reaches C2, whereby the second feature map 250 having the C2 channel is generated.

[0153] In this manner, the process of calculating the pixel values ​​of each pixel position (pixel values ​​of all pixel positions) in the kth channel of the second feature map 250 using the kth convolution filter 341 is executed a predetermined number (C2 times) (k=1,...,C2). As a result, a predetermined number (C2) of channels in the second feature map 250 are generated.

[0154] Next, with reference to FIG. 17, a process in which the second feature map 250 having C2 channels is generated using a convolution filter 342 of size N×N×C1 will be described.

[0155] The k-th convolution filter 342 of the C2 convolution filters 342 of size N×N×C1 is a filter for calculating pixel values ​​at each pixel position of the k-th channel of the second feature map 250. The k-th convolution filter 342 is a filter that performs weighted addition of pixel values ​​at the same position and its neighboring positions in multiple (C1) channels of the first feature map 230 (a filter that integrates pixel values ​​at the same position and its neighboring positions). Note that the C2 convolution filters 342 are basically configured with values ​​different from each other.

[0156] 17, the k-th channel of the second feature map 250 is generated by the k-th convolution filter 342 in the same manner as in the first embodiment (see FIG. 9). Also, compared with FIG. 16, the k-th channel of the second feature map 250 is generated by using the k-th convolution filter 342 instead of the k-th convolution filter 341.

[0157] In detail, a process is executed to calculate a pixel value of a corresponding pixel position (i, j) of the second feature map 250 (specifically, the kth channel) based on the kth convolution filter 342 and pixel values ​​of a pixel position (i, j) and its neighboring pixel positions in multiple channels of the first feature map 230. Then, such a process is similarly executed for pixels of all other pixel positions of the first feature map 230. In this way, pixel values ​​of all other pixel positions of the second feature map 250 (specifically, the kth channel) are calculated.

[0158] Such processing is repeatedly performed using C2 convolution filters 342, thereby calculating pixel values ​​at all pixel positions in the channels for all of the C2 channels of the second feature map 250. More specifically, the processing of generating the kth channel of the second feature map 250 using the kth convolution filter 342 is performed C2 times while incrementing the value k until the value k reaches "C2" from "1". In this way, the second feature map 250 having C2 channels is generated.

[0159] In this manner, the second feature map 250 having C2 (multiple) channels is generated. Here, the second feature map 250 has two or more channels, which is reduced from the number of channels C1 of the first feature map 230. That is, the value C2 is smaller than the value C1 (the number of channels of the first feature map 230). For example, C2=2. However, this is not limited thereto, and the value C2 may be 3 or more. Moreover, the value C2 may be the same as the value C1, or may be larger than the value C1.

[0160] FIG. 18 shows how a second feature map 250 (C2=2) having two channels is output.

[0161] Here, the GMP process is performed on the second feature map 250 (second feature map 250 having two channels) having a size of W2×H2×2. The GMP process extracts the maximum pixel value for each of the two channels 251, 252 in the second feature map 250. Specifically, the maximum value in the first channel 251 of the second feature map 250 is obtained as the output of the first channel 251, and the maximum value in the second channel 252 of the second feature map 250 is obtained as the output of the second channel 252. Furthermore, the output for each channel is normalized by an activation function (for example, a sigmoid function) or the like, converted into a value between "0" and "1", and output from the output layer 370 of the learning model 410.

[0162] In the learning stage, as shown in Fig. 18, a comparison with the correct answer data is performed. Here, the correct answer data of the first channel 251 and the correct answer data of the second channel 252 are different types of correct answer data. For example, the correct answer data of the first channel 251 is correct answer data regarding the presence or absence of "microscopic scratches", and the correct answer data of the second channel 252 is correct answer data regarding the presence or absence of "microscopic tears". However, without being limited to this, the correct answer data of each channel may be other types of correct answer data (for example, "microscopic stains" etc.).

[0163] By performing machine learning based on such ground truth data, the output value of the first channel 251 is trained to output a value indicating the presence or absence of a "micro-scratches", and the output value of the second channel 252 is trained to output a value indicating the presence or absence of a "micro-tear".

[0164] The second embodiment can also achieve the same effect as the first embodiment. In the machine learning of the learning model 410 of the second embodiment, the multiple maximum values ​​obtained for each channel by performing GMP processing on the second feature map 250 having two or more channels are not averaged. Specifically, each of the multiple maximum values ​​is converted (by an activation function or the like) without averaging the multiple maximum values, and each converted value (or the maximum value itself) is compared with the correct answer data. According to this, multiple types of feature amounts can be extracted without averaging, so that multiple types of abnormalities (microscopic scratches, microscopic tears, microscopic stains, etc.) can be detected individually and appropriately.

[0165] Also, here, the output values ​​(multiple output values) for each channel by the GMP processing are normalized, but this is not limited thereto. For example, as shown in FIG. 19, the multiple output values ​​for each channel by the GMP processing may be normalized together by a softmax function. In detail, the multiple output values ​​for each channel by the GMP processing may be further processed by a softmax function, and the processing result may be compared with the correct answer data to perform machine learning. In this case, an output suitable for determining which of multiple types of abnormalities (for example, multiple types of abnormalities including "microscratch", "microtear", and "microstain") has occurred is obtained. In other words, the possibility of occurrence of each of multiple types of abnormalities (for example, abnormalities including "microscratch", "microtear", and "microstain") may be relatively determined.

[0166] <3. Third embodiment> In each of the above embodiments, the GMP processing is performed on all channels of the second feature map 250, but this is not limited to the above. For example, the GMP processing may be performed on only some of the C2 channels of the second feature map 250. In the third embodiment, such an aspect will be described.

[0167] 20 is a diagram showing a learning model 410 (also referred to as 410C) according to the third embodiment. In the learning model 410C, a second feature map 250 having a C1 channel is generated, similarly to the learning model 410 (410B) according to the second embodiment.

[0168] However, thereafter, GMP processing is performed on only some of all the C2 channels in the second feature map 250, and GAP processing is performed on the remaining channels of all the C2 channels.

[0169] 21, for example, when the second feature map 250 has a total of five channels (i.e., when C2=5), the GMP processing is performed on the first to third channels, and the GAP processing is performed on the fourth to fifth channels. In other words, a GMP layer 360 is provided for the first to third channels, and a GAP layer 369 is provided for the fourth to fifth channels.

[0170] In the learning stage, machine learning is performed using values ​​obtained by performing GMP processing on a plurality of channels from the first channel to the third channel. Specifically, machine learning is performed to reduce the difference between each output after GMP processing on the first channel to the third channel and ground truth data regarding the presence or absence of minute abnormalities (for example, minute scratches, minute tears, minute stains) in the input image 210. This generates a learning model 410 (trained model 420) that can accurately determine the presence or absence of each type of minute abnormality.

[0171] Furthermore, machine learning is performed using values ​​obtained by performing GAP processing on a plurality of channels from the fourth channel to the fifth channel. Specifically, machine learning is performed to reduce the difference between each output after GAP processing on the fourth channel to the fifth channel and the correct answer data regarding the presence or absence of a large abnormality (for example, a large stain, a general distortion, etc.) in the input image 210. This generates a learning model 410 (trained model 420) that can accurately determine the presence or absence of each type of large abnormality (anomaly over a relatively wide range in the image).

[0172] Meanwhile, in the inference stage, it is possible to accurately estimate the presence or absence of minute abnormalities (for example, minute scratches, minute tears, minute stains) in the input image 210 using the outputs after GMP processing for the first to third channels. Also, it is possible to accurately estimate the presence or absence of large abnormalities (abnormalities covering a relatively wide range in the image) in the input image 210 using the outputs after GAP processing for the fourth to fifth channels.

[0173] <4. Modifications, etc.> Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described contents.

[0174] For example, in the above embodiment, the image processing device 30 executes both the learning process and the inference process. However, the present invention is not limited to this, and the learning process and the inference process may be realized by separate devices (30A, 30B). In this case, the device 30B that executes the inference process may obtain, for example, information (learned learning parameters, etc.) related to the trained model 420 generated by the device 30A that executed the learning process from the device 30A, and use the trained model 420. [Explanation of symbols]

[0175] 1. Image Processing System 20 Imaging Equipment 30 Image processing device (learning model generation device) 35b Display section 200 Convolutional Neural Networks 210 Input images 230 First feature map 245 Provisional Feature Map 250 Second feature map 270 Score (output value) 310 Input Layer 320 Feature Extraction Layer 340 Convolutional Layer 341,342 Filters 360 Pooling layer (GMP layer) 370 Output Layer 410,420 Learning Model 530 Heatmap

Claims

1. a control unit that detects an abnormality in a photographed image of an object based on a learning model that is machine-learned using teacher data in which each photographed image is input and information regarding an abnormality in each of the photographed images is output; Equipped with the learning model is a model that is machine-learned by generating a second feature map by integrating pixel values ​​at the same position and / or adjacent positions of each channel in a first feature map, the second feature map being an intermediate feature map in a convolutional neural network and having a plurality of channels, and performing a global max pooling process on the second feature map; The image processing device is characterized in that the control unit detects abnormalities in the object by inputting the captured image of the object to the learning model.

2. The image processing device according to claim 1 , wherein the first feature map is generated by an intermediate layer including a plurality of convolution layers and a plurality of pooling layers provided after an input layer in the convolutional neural network.

3. 3 . The image processing device according to claim 1 , wherein the convolutional neural network does not have a fully connected layer subsequent to a pooling layer that performs the global max pooling process on the second feature map. 4 .

4. 4. The image processing device according to claim 3, wherein the learning model is a machine-learned model obtained by comparing a maximum value obtained by performing the global max pooling process on the second feature map, or a transformed value obtained by transforming the maximum value without multiplying the maximum value by a learning parameter, with correct answer data.

5. 5. The image processing device according to claim 3, wherein the learning model is a machine-learned model obtained by performing the global max pooling process on the second feature map having two or more channels, converting each of the multiple maximum values ​​obtained for each channel without averaging the multiple maximum values, and comparing the converted values ​​with correct data.

6. 6. The image processing device according to claim 1, wherein the process of generating the second feature map includes a process of integrating pixel values ​​at the same pixel position in multiple channels of the first feature map using a 1×1 convolution filter that weights and adds up pixel values ​​of the same pixel position in multiple channels of the second feature map, and calculating a pixel value of each pixel position in the channel of the second feature map.

7. 6. The image processing device according to claim 1, wherein the process of generating the second feature map includes a process of integrating pixel values ​​at the same pixel position in multiple channels of a provisional feature map after convolution processing for each of the multiple channels of the first feature map by weighted addition, to calculate a pixel value at each pixel position in the channel of the second feature map.

8. 6. The image processing device according to claim 1, wherein the process of generating the second feature map includes a process of calculating a pixel value at each pixel position in one channel of the second feature map a predetermined number of times using a convolution filter that weights and adds up pixel values ​​obtained by integrating pixel values ​​at the same position and / or adjacent positions in a plurality of channels of the first feature map, thereby generating a predetermined number of channels in the second feature map.

9. the second feature map has two or more channels that are reduced in number from the number of channels of the first feature map; 9. The image processing device according to claim 1, wherein the learning model is a model that is machine-learned using values ​​obtained by performing the global max pooling process on at least a portion of the two or more channels.

10. the second feature map is a feature map having a single channel; The image processing device according to claim 1 , wherein the learning model is a model machine-learned using a value obtained by performing the global max pooling process on the single channel.

11. The image processing device according to claim 1 , wherein the control unit displays, on a predetermined display unit, information visualized based on the second feature map.

12. a) detecting anomalies in a photographed image of an object based on a learning model machine-learned using teacher data in which each photographed image is input and information regarding anomalies in each of the photographed images is output; Equipped with the learning model is a model that is machine-learned by generating a second feature map by integrating pixel values ​​at the same position and / or adjacent positions of each channel in a first feature map, the second feature map being an intermediate feature map in a convolutional neural network and having a plurality of channels, and performing a global max pooling process on the second feature map; An image processing method characterized in that in the step a), an abnormality in the object is detected by inputting the captured image of the object to the learning model.

13. a) detecting anomalies in a photographed image of an object based on a learning model machine-learned using teacher data in which each photographed image is input and information regarding anomalies in each of the photographed images is output; A program for causing a computer to execute the above, the learning model is a model that is machine-learned by generating a second feature map by integrating pixel values ​​at the same position and / or adjacent positions of each channel in a first feature map, the second feature map being an intermediate feature map in a convolutional neural network and having a plurality of channels, and performing a global max pooling process on the second feature map; A program characterized in that in the step a), an abnormality in the object is detected by inputting the captured image of the object to the learning model.

14. A learning model generation device that receives a captured image and generates a learning model that outputs information about an abnormality in the captured image, a control unit that generates a second feature map by integrating pixel values ​​at the same position and / or adjacent positions of each channel in a first feature map, the second feature map being an intermediate feature map in a convolutional neural network of the learning model and having a plurality of channels, based on the first feature map, and executes machine learning in the learning model using a value obtained by performing a global max pooling process on the second feature map; A learning model generation device comprising:

15. A learning model production method for producing a learning model that inputs a captured image and outputs information about an anomaly in the captured image, comprising: a) generating a second feature map by integrating pixel values ​​at the same position and / or adjacent positions of each channel in a first feature map, the second feature map being an intermediate feature map in a convolutional neural network of the learning model and having a plurality of channels, based on the first feature map, and performing machine learning in the learning model using values ​​obtained by performing a global max pooling process on the second feature map; A learning model production method comprising:

16. A trained model that has been machine-trained to output information regarding an anomaly in an inference target image in response to an input of the inference target image, A process of generating a first feature map for the inference target image, the first feature map being an intermediate feature map in a convolutional neural network and having a plurality of channels; A process of generating a second feature map by integrating pixel values ​​at the same position and / or adjacent positions of each channel in the first feature map based on the first feature map; outputting information about the anomaly based on a value obtained by performing a global max pooling process on the second feature map; A trained model for running the above on a computer.

17. a) generating a second feature map by integrating pixel values ​​at the same position and / or adjacent positions of each channel in a first feature map, the second feature map being an intermediate feature map having a plurality of channels in a convolutional neural network of a learning model that inputs a captured image and outputs information about anomalies in the captured image, and performing machine learning in the learning model using values ​​obtained by performing a global max pooling process on the second feature map; A program for causing a computer to execute the following.

Images