Learning device, method, program, and inference device

Abstract

The detection performance can be improved.SOLUTION: A learning device according to the present embodiment includes a similarity calculation unit, a selection unit, a feature map calculation unit, a difference feature map calculation unit, a likelihood map calculation unit, a loss calculation unit, and an update unit. The similarity calculation unit calculates a similarity between the object image and at least one normal image. The selection unit selects a reference image from the normal images on the basis of the similarity. The feature map calculator calculates a first feature map of the target image and a second feature map of the reference image using a first machine learning model. The difference feature map calculation unit calculates a difference feature map that is a difference between the first feature map and the second feature map. The likelihood map calculation unit calculates a likelihood map based on the first feature map and the difference feature map using a second machine learning model. The loss calculation unit calculates a loss based on the likelihood based on the likelihood map. The updating unit updates the first machine learning model and the second machine learning model based on the loss.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a learning device, a method, a program, and an inference device. [Background technology]

[0002] The automation of infrastructure inspections such as roads and power equipment is progressing. Technologies are being developed to detect abnormalities (also known as anomalies, including damage and abnormal conditions) in inspection targets from images. For example, there is road surface anomaly detection technology that detects abnormalities on road surfaces from camera images. When training a machine learning model that estimates the location of road surface anomalies using supervised learning, it is possible to prepare a learning dataset that combines road images with circumscribing rectangles that indicate the location of anomalies in the target images as training labels, and then use that data for learning. However, preparing circumscribing rectangles that indicate the location of anomalies as training labels requires time-consuming training work. Furthermore, there are individual differences in the ability of training operators to identify abnormalities.

[0003] Therefore, there is a method to train a machine learning model using weakly supervised learning, which only teaches the presence or absence of anomalies in an image, and then use the trained model to detect the location of anomalies in an image.In addition, one method to improve image anomaly detection using supervised learning is to improve the accuracy of anomaly detection by using the differential feature amount between a normal image and a target image. However, the above-mentioned method has the problem of low accuracy in estimating the location of an anomaly on an image containing the detected anomaly.In addition, because it uses randomly sampled normal images, there is a problem that detection performance deteriorates due to the influence of variations in the background and the photographing angle of view when inspecting infrastructure facilities, etc. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Satoshi Ito, “Multiple Types of Road Deformation Detection Using Multiple Instance Learning”, ViEW2022, No.OS1-03, December 1, 2022. [Non-patent document 2] Ding, Choubo et al,. "Catching both gray and black swans: Open-set supervised anomaly detection." CVPR2022 Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a learning device, a method, a program, and an inference device that can improve detection performance. [Means for solving the problem]

[0006] The learning device according to this embodiment includes an acquisition unit, a similarity calculation unit, a selection unit, a feature map calculation unit, a differential feature map calculation unit, a likelihood map calculation unit, a loss calculation unit, and an update unit. The acquisition unit acquires a target image and at least one normal image. The similarity calculation unit calculates the similarity between the target image and the normal images. The selection unit selects a reference image from the normal images based on the similarity. The feature map calculation unit calculates a first feature map of the target image and a second feature map of the reference image using a first machine learning model. The differential feature map calculation unit calculates a differential feature map that is the difference between the first feature map and the second feature map. The likelihood map calculation unit calculates a likelihood map based on the first feature map and the differential feature map using a second machine learning model. The loss calculation unit calculates a loss based on the likelihood based on the likelihood map and a training label of the target image. The update unit updates the first machine learning model and the second machine learning model based on the loss. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an image inspection system according to the present embodiment. [Figure 2]FIG. 1 is a block diagram showing an example of the configuration of a learning device according to a first embodiment. [Figure 3] 4 is a flowchart showing an example of the operation of the learning device according to the first embodiment. [Figure 4] FIG. 10 is a diagram showing an example of a target image included in a learning dataset. [Figure 5] Conceptual diagram explaining Multiple Instance Learning for images. [Figure 6] FIG. 2 is a diagram showing an example of a learning method for a first machine learning model and a second machine learning model. [Figure 7] FIG. 10 is a block diagram showing an example of the arrangement of an image processing apparatus according to a second embodiment. [Figure 8] 10 is a flowchart showing an example of the operation of the image processing device according to the second embodiment. [Figure 9] FIG. 10 is a view showing an example of a display screen of the image processing apparatus according to the second embodiment. [Figure 10] 10A and 10B are diagrams showing examples of application of detection results of the image processing device to an application. [Figure 11] FIG. 1 is a block diagram showing an example of a hardware configuration of an information processing apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0008] The learning device, method, program, and inference device according to the present embodiment will be described in detail below with reference to the drawings. Note that in the following embodiments, parts with the same reference numerals perform similar operations, and redundant explanations will be omitted as appropriate.

[0009] (First embodiment) An example of the configuration of an image inspection system 1 according to the first embodiment will be described with reference to the block diagram of FIG.

[0010] The image inspection system 1 shown in FIG. 1 is used to capture images and use the captured images to inspect an object for abnormalities. An abnormality here refers to an abnormality that appears on the object being inspected. For example, an abnormality on a road surface could be a crack, rut, pothole, sinkhole, depression, or step that has appeared on the road surface. The severity of an abnormality may be defined according to its size, depth, etc. For example, the severity of a pothole may be defined as a grade such as "AA" or "A" depending on its size and depth.

[0011] The image inspection system 1 according to the first embodiment includes an information processing device 10 and an imaging device 20. The information processing device 10 includes a learning device 11 and an image processing device 13.

[0012] The learning device 11 trains a machine learning model using a learning dataset. The image processing device 13 performs image processing on captured images using the machine learning model trained by the learning device 11. The image processing device 13 performs image processing to check for abnormalities in images captured by the imaging device 20. The image processing device 13 may acquire images directly from the imaging device 20, or may acquire images from an external device or external medium, such as a server or media, in which images captured by the imaging device 20 are stored. Although FIG. 1 illustrates a single information processing device 10 including the learning device 11 and the image processing device 13, the learning device 11 and the image processing device 13 may each be independent devices.

[0013] Next, the functional configuration of the learning device 11 will be described with reference to the block diagram of FIG. The learning device 11 includes a target image acquisition unit 111, a teaching label acquisition unit 112, a normal image acquisition unit 113, a similarity calculation unit 114, a reference image selection unit 115, a feature map calculation unit 116, a differential feature map calculation unit 117, a likelihood map calculation unit 118, a likelihood calculation unit 119, a loss calculation unit 120, a model update unit 121, and a memory unit 122.

[0014] The target image acquisition unit 111 acquires a target image to be processed, which is included in the learning dataset. The target image is, for example, an image of an inspection target, and may or may not include an abnormality. The instruction label acquisition unit 112 acquires an instruction label that indicates whether or not an abnormality exists in a target image, which is included in the learning dataset.

[0015] The normal image acquisition unit 113 acquires at least one normal image. The normal image is an image of the inspection target, and is an image in which the inspection target does not contain any abnormality. The similarity calculation unit 114 uses the target image and the normal image to calculate the similarity of the normal image with respect to the target image.

[0016] The reference image selection unit 115 selects a reference image from the normal image based on the similarity calculated by the similarity calculation unit 114. The feature map calculation unit 116 uses the first machine learning model to obtain feature maps for each of the target image and the reference image. That is, the feature map calculation unit 116 uses the first machine learning model to calculate a first feature map that is a feature map of the target image and a second feature map that is a feature map of the reference image.

[0017] The differential feature map calculation unit 117 calculates a differential feature map that is the difference between the first feature map and the second feature map calculated by the feature map calculation unit 116. The likelihood map calculation unit 118 calculates a likelihood map based on the first feature map and the differential feature map using the second machine learning model. The likelihood map is a score map that indicates the likelihood of anomaly at each position in the image.

[0018] The likelihood calculation unit 119 calculates the likelihood of the target image from the likelihood map. The likelihood is a score that indicates the probability that an abnormality exists in the target image (that an abnormality is captured in the target image). The loss calculation unit 120 calculates a loss value based on the likelihood and the teaching label.

[0019] The model update unit 121 optimizes the parameters (weighting coefficients, biases, etc.) of the first machine learning model and the second machine learning model so that the loss value calculated by the loss calculation unit 120 is minimized. The storage unit 122 stores a target image, a normal image, a first machine learning model, a second machine learning model, and the like.

[0020] Next, an example of the operation of the learning device 11 according to the first embodiment will be described with reference to the flowchart in Fig. 3. The training flow executed by the learning device 11 according to the first embodiment is weakly supervised learning that combines only the image and the presence or absence of an abnormality in the image as teaching labels, and estimates the position of an abnormality in the image.

[0021] In step SA1, the target image acquisition unit 111 acquires a target image as learning data, and the instruction label acquisition unit 112 acquires an instruction label corresponding to the target image. The instruction label is a label indicating whether or not an abnormality is present in the target image. In step SA2, the normal image acquisition unit 113 acquires a plurality of normal images.

[0022] In step SA3, the similarity calculation unit 114 calculates the similarity between the target image and the normal image. Specifically, the similarity calculation unit 114 may calculate the similarity using a model constructed by machine learning. The similarity calculation unit 114 may extract features of the target image and the normal image using a neural network trained by supervised learning and calculate the similarity based on the extracted features. For example, the extracted features may be used to calculate the Euclidean distance and use it as the similarity. The similarity calculation unit 114 may receive multiple normal images and calculate the similarity between the multiple normal images. For example, the similarity calculation unit 114 may calculate the similarity between all images in a training dataset prepared for training the learning device 11 and all normal images in the training dataset. In another example, the similarity calculation unit 114 may calculate the similarity based on metadata such as the image capture date and time, the image capture location, and the weather. For example, if images are captured at the same location and on the same date and time, the similarity may be set high.

[0023] In step SA4, the reference image selection unit 115 selects at least one reference image. Specifically, the reference image selection unit 115 selects a normal image as a reference image based on the similarity calculated in step SA3. The reference image selection unit 115 may also acquire multiple normal images as reference images. In one example, the reference image selection unit 115 may probabilistically select an image to be acquired as a reference image from the normal images based on the similarity. For example, the reference image selection unit 115 may randomly select a reference image from normal images having a similarity equal to or greater than a threshold. Alternatively, the higher the similarity of a normal image, the higher the selection probability may be. The reference image selection unit 115 may also acquire the top N normal images (N is a natural number equal to or greater than 1) in terms of similarity as reference images. By probabilistically selecting the reference images, the first and second machine learning models can be trained to improve their generalization performance. In steps SA2 to SA4, an example has been shown in which at least one reference image is selected from a plurality of normal images. However, for example, if one normal image is acquired, the normal image may be selected as the reference image.

[0024] In step SA5, the feature map calculation unit 116 uses a first machine learning model to calculate a first feature map for the target image and a second feature map for the reference image. The first machine learning model is, for example, a convolutional neural network. The feature map calculation unit 116 functions as an encoder that converts the image into a lower-dimensional feature space using the first machine learning model. In one example, the feature map calculation unit 116 may calculate a feature map obtained from an intermediate layer of the first machine learning model. Specifically, when ResNet is used as the first machine learning model, the feature map calculation unit 116 calculates a feature map obtained from each intermediate layer. Since the output size of the feature map differs depending on the intermediate layer, feature maps having multiple different output sizes can be calculated.

[0025] In step SA6, the differential feature map calculation unit 117 calculates the difference between the first feature map and the second feature map as the differential feature map. For example, the differential feature map calculation unit 117 may use each intermediate layer calculated by the feature map calculation unit 116 to calculate differential feature maps with different output sizes from the intermediate layers. Alternatively, when the feature map calculation unit 116 receives multiple reference images and calculates respective feature maps, the differential feature map calculation unit 117 may calculate the difference between the average of the feature maps of the multiple reference images and the first feature map as the differential feature map. Alternatively, the differential feature map calculation unit 117 may calculate the differential feature map using a weighted average of the feature maps of the multiple reference images based on similarity. Specifically, the differential feature map calculation unit 117 may weight the multiple reference images so that the weight is larger for reference images with higher similarity, and calculate the difference between the average and the first feature map as the differential feature map.

[0026] In step SA7, the likelihood map calculation unit 118 uses a second machine learning model to calculate a likelihood map based on the first feature map and the differential feature map. The second machine learning model is, for example, a convolutional neural network that performs upsampling processing. The vertical and horizontal lengths of the likelihood map are assumed to be the same as those of the target image. The likelihood map, for example, takes a continuous value from 0 to 1 as a score for each region corresponding to each pixel of the target image, with a larger value indicating a higher likelihood of a specific anomaly. The likelihood map makes it possible to visualize which locations in the target image may be abnormal.

[0027] In step SA8, likelihood calculation unit 119 calculates the likelihood of the presence or absence of an abnormality in the target image using the likelihood map. The likelihood is, for example, a continuous value from 0 to 1, and a larger value indicates that the image contains an abnormality. For example, likelihood calculation unit 119 may calculate the maximum value of the likelihood map as the likelihood.

[0028] In step SA9, the loss calculation unit 120 calculates a loss value using a loss function based on the likelihood. As the loss function, for example, BinaryCrossEntropyLoss may be used. When the score for an image is represented as p and the label for the image is represented as y, the loss value (L) is calculated using equation (1).

[0029] L=-ylogp-(1-y)log(1-p)···(1) In step SA10, the model update unit 121 determines whether the training of the first machine learning model and the second machine learning model has finished. Specifically, for example, the model update unit 121 determines whether the loss value (L) calculated in step SA9 is equal to or less than a threshold. If the loss value (L) is equal to or less than the threshold, it is considered that the loss value (L) has converged, and therefore it may be determined that the training of the first machine learning model and the second machine learning model has finished. On the other hand, if the loss value (L) is greater than the threshold, it may be determined that the training of the first machine learning model and the second machine learning model has not finished. Note that the determination method is not limited to the above, and any general method for determining whether training has finished may be used.

[0030] If the training of the first machine learning model and the second machine learning model has been completed, proceed to step SA11; if the training of the first machine learning model and the second machine learning model has not been completed, proceed to step SA12.

[0031] In step SA11, the storage unit 122 stores a first trained model, which is a trained first machine learning model, and a second trained model, which is a trained second machine learning model. Note that, hereinafter, the first trained model and the second trained model are also collectively referred to as trained models.

[0032] In step SA12, the model update unit 121 updates the parameters of the first machine learning model and the second machine learning model, and the process returns to step SA1, where the process is repeated until the training is completed. That is, the parameters of the first machine learning model and the second machine learning model are updated so as to minimize the loss value (L).

[0033] Next, an example of a target image included in the learning dataset will be described with reference to FIG. 4, a captured image of a road surface is assumed as an example of the target image 41. As the instruction label corresponding to the target image 41, whether or not there is an abnormality on the road surface is indicated for each type of abnormality.

[0034] In the example of FIG. 4, the types of anomalies are shown as "vertical cracks, horizontal cracks, hexagonal cracks, and potholes," and the presence or absence of each type of anomaly is associated as a teaching label. Specifically, target image 41-1 does not contain "vertical cracks, horizontal cracks," but is assigned a teaching label indicating the presence of "hexagonal cracks and potholes." For example, the teaching label corresponding to target image 41-1 may be expressed as a vector [0,0,1,1]. Target image 41-N has "vertical cracks, horizontal cracks" but does not have "hexagonal cracks and potholes," so the teaching label may be expressed as a vector [1,1,0,0].

[0035] In this case, the teaching label does not indicate which part of the target image 41 contains an abnormality. The Multiple Instance learning framework can be applied to the problem setting of training using only the above-mentioned image and the presence or absence of an abnormality in the image as teaching labels, and estimating the position of an abnormality in the image.

[0036] Next, Multiple Instance learning for images will be described with reference to the conceptual diagram of FIG. Multiple-instance learning is a type of supervised learning. In general supervised learning, a label y is assigned to each sample x. In contrast, in Multiple-instance learning, a correct label is assigned to a set of multiple instances. In Multiple-instance learning, this set is called a bag. For example, let's explain the correct label for a two-class classification task. A bag containing all negative instances is assigned a negative label y=0, and a bag containing at least one positive instance is assigned a positive label y=1. The purpose of Multiple-instance learning is to train a machine learning model to estimate the label of a bag, and to train a machine learning model to estimate the label of each instance in the bag.

[0037] In the example of Figure 5, we consider a case where there is only one type of anomaly to be detected. If anomaly detection for road images is interpreted as Multiple Instance learning, each region (i, j) in image 51 corresponds to an instance. Here, i is an index indicating the horizontal position in the image, and j is an index indicating the vertical position in the image. For example, each region (i, j) may be in pixel units, or may be in patch units obtained by dividing the image. Furthermore, an image corresponds to a set bag that collects instances.

[0038] Image 51 is an image in which an abnormality is present, and it is shown that there is an abnormality in the regions (i, j) = (3, 4) and (4, 4). On the other hand, image 52 is an image in which there is no abnormality. Images 51 and 52 each have a label t for each region (i, j). ij and the label y for the image. Here, "1" indicates that there is an abnormality, and "0" indicates that there is no abnormality. An image 51 that contains even one abnormal region is assigned the label y=1 indicating that there is an abnormality, and an image 52 in which all regions are normal is assigned the label y=0 indicating that there is no abnormality.

[0039] Here, as shown in equation (2), the label t ijThe maximum value of is equal to the label y for the image. y=max i,j t ij ···(2) As described above, in the training method for the first machine learning model and the second machine learning model, by regarding the image as a bag and each region in the image as an instance, it is possible to apply a Multiple Instance learning framework that uses only the image and the presence or absence of anomalies in the image as teaching labels.

[0040] Next, an example of a learning method for the first machine learning model and the second machine learning model will be described with reference to FIG.

[0041] FIG. 6 shows the processing performed on the target image by the feature map calculation unit 116, the differential feature map calculation unit 117, the likelihood map calculation unit 118, and the likelihood calculation unit 119.

[0042] The first machine learning model and the second machine learning model are configured by a Fully Convolutional Network configured by convolution operations with spatial locality. In the first embodiment, it is assumed that a Feature Pyramid Network structure is used in which the first machine learning model is an encoder and the second machine learning model is a decoder. However, other models may be used as long as they have an encoder-decoder structure in which the first machine learning model is an encoder and the second machine learning model is a decoder. For example, a U-net may be used.

[0043] In the feature map calculation unit 116, the target image 60 and three reference images 65 are input to the encoder. When ResNet is used as the first machine learning model of the feature map calculation unit 116, feature maps obtained from each intermediate block can be calculated. Specifically, when the encoder shown in FIG. 6 is composed of five blocks with convolutional layers, first feature maps 61 with data sizes 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the data size of the target image 60 are obtained by convolution processing of the five blocks. Similarly, for the reference image 65, second feature maps 62 with data 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the data size of the reference image 65 are obtained. In this way, by using the output from the intermediate blocks, if the feature maps of the respective intermediate blocks have different sizes, multiple feature maps with different data sizes can be calculated. The differential feature map calculation unit 117 calculates the difference between each of the data sizes. In the example of Fig. 6, for the data sizes of 1 / 8, 1 / 16, and 1 / 32, the difference between the first feature map 61 and the corresponding second feature map 62 is calculated as the differential feature map 63. For example, the differential feature map calculation unit 117 calculates the Euclidean distance for each element between the first feature map 61 and the second feature map 62 to calculate the differential feature map 63 .

[0044] The likelihood map calculation unit 118 combines the differential feature map 63 and the first feature map 61 for each of the three types of data sizes and inputs the combined result to the decoder, which is the second machine learning model. The likelihood map calculation unit 118 outputs a likelihood map 66 for multiple types of anomalies using the output from the decoder, which is the second machine learning model. For example, the P3 feature map, P4 feature map, and P5 feature map output by the FeaturePyramidNetwork, which is the decoder, are combined and subjected to convolution processing, and then a sigmoid function, which is an activation function, is applied, and the likelihood map 66 for multiple types of anomalies is output. Here, there are K maps of likelihood maps 66, where K is the number of anomaly types. In the example of FIG. 6, the likelihood map 66 is calculated by dividing each region (i, j) in the target image into scores sij where s ij is a continuous value between 0 and 1, and the larger it is, the higher the probability of an anomaly. Using the second machine learning model, we assign a score s to every region in the image. ij After calculating the likelihood, the likelihood calculation unit 119 calculates the scores s for all regions in the image. ij The maximum value of is calculated as the score p for the entire image. GlobalMaxPooling is used to convert the likelihood map 66 into the score p for the entire image. The score p is expressed as in equation (3).

[0045] p=max i,j s ij ···(3) In Figure 6, the score p is expressed as a vector [p1, p2, , p K As mentioned above, K here indicates the number of types of anomalies. In other words, it corresponds to the number of likelihood maps to be calculated. If there is only one type of anomaly to be detected, K=1. If there are multiple types of anomalies, it is sufficient to generate the same number of likelihood maps as there are types of anomalies.

[0046] The score p for this target image 60 and the label y for the image are used to calculate a loss value using BinaryCrossEntropy, and the first machine learning model and the second machine learning model are trained so that this value decreases.

[0047] This training method allows us to obtain scores for each domain. ij The first and second machine learning models can be trained to be low for regions without anomalies and high for regions with anomalies.

[0048] In other words, the machine learning model is trained with data that instructs the presence or absence of a specific detection target in a specified unit for the image, outputs a likelihood in a unit smaller than the specified unit, and is trained so that the maximum value of the output likelihood matches the presence or absence of the instructed detection target.

[0049] According to the first embodiment described above, in weakly supervised learning, which combines only the target image and the presence or absence of an anomaly in the target image as teaching labels, a normal image similar to the target image is input as a reference image to a first machine learning model to train the model. In weakly supervised learning, the presence or absence of an anomaly is only taught for the entire image, not which region of the image contains an anomaly. Therefore, the accuracy of anomaly detection decreases due to the influence of the situation when the target image was captured, the angle of view, etc. However, according to the learning device according to the first embodiment described above, a reference image is selected from normal images based on the similarity with the target image, a difference feature map between the target image and the reference image is calculated, and a likelihood map is calculated based on the feature map and difference feature map of the target image. This makes it possible to train a model while reducing the influence of the shooting situation of the target image, and generate a trained model with improved anomaly detection accuracy.

[0050] (Second embodiment) In the second embodiment, an image processing device that executes inference processing using a learned model trained by the learning device 11 according to the first embodiment will be described.

[0051] FIG. 7 shows a block diagram of an image processing device 13 according to the second embodiment. The image processing device 13 includes a target image acquisition unit 111, a normal image acquisition unit 113, a similarity calculation unit 114, a reference image selection unit 115, a feature map calculation unit 116, a difference feature map calculation unit 117, a likelihood map calculation unit 118, a likelihood calculation unit 119, a memory unit 122, an image processing unit 131, and an information output unit 132.

[0052] The target image acquisition unit 111, the normal image acquisition unit 113, the similarity calculation unit 114, the feature map calculation unit 116, the difference feature map calculation unit 117, the likelihood map calculation unit 118, and the likelihood calculation unit 119 perform the same processing as in the first embodiment, and therefore detailed explanations thereof will be omitted.

[0053] Based on the similarity calculated by the similarity calculation unit 114, the reference image selection unit 115 selects at least one reference image in descending order of similarity to the normal image. The storage unit 122 stores a likelihood map of a target image, a superimposed image in which the likelihood map is superimposed on the target image, and the like.

[0054] Next, an example of the operation of the image processing device 13 according to the second embodiment will be described with reference to the flowchart of FIG.

[0055] In step SB1, the target image acquisition unit 111 acquires a target image that is to be subjected to inference processing. In step SB2, the normal image acquisition unit 113 acquires a plurality of normal images.

[0056] In step SB3, the similarity calculation unit 114 calculates the similarity between the target image and the normal image. The calculation method is the same as the process in step SA3 according to the first embodiment. In step SB4, the reference image selection unit 115 selects at least one reference image. Here, the top N normal images in terms of similarity may be acquired as the reference images.

[0057] In step SB5, the feature map calculation unit 116 calculates a first feature map of the target image and a second feature map of the reference image using the first trained model. In step SB6, the differential feature map calculation unit 117 calculates the difference between the first feature map and the second feature map as a differential feature map.

[0058] In step SB7, the likelihood map calculation unit 118 calculates a likelihood map using the second trained model. In step SB8, likelihood calculation unit 119 calculates the likelihood of the presence or absence of an abnormality in the target image. Note that the processes from step SB5 to step SB8 are performed when the first machine learning model and the second machine learning model in learning device 11 according to the first embodiment become the first trained model and the second trained model, and the same processes as those in the first embodiment may be performed.

[0059] In step SB9, the image processing unit 131 generates various output information indicating whether or not there is an abnormality in the target image, using the target image, the likelihood map, and the likelihood. For example, the image processing unit 131 generates a superimposed image in which the likelihood map is superimposed on the target image. In step SB10, the information output unit 132 outputs the various pieces of output information generated in step SB9.

[0060] Next, an example of a display screen of the image processing device 13 according to the second embodiment is shown in FIG. 9 is an example of a display screen of an abnormality detection result generated by the image processing device 13 when the target image is an image of a road surface. When an abnormality on the road surface is detected by the image processing unit 131, the information output unit 132 outputs information related to the abnormality. For example, the input image (target image) to the image processing device 13 and a superimposed image for each type of abnormality are displayed side by side. Also, a reference image used in processing the target image may be displayed. By looking at the display screen shown in FIG. 9, the user can easily grasp whether or not there is an abnormality, and further, if there is an abnormality, the type of abnormality and the location of the abnormality (area in the target image).

[0061] Next, an example of application of the detection results of the image processing device 13 will be described with reference to FIG. Fig. 10 is an example of a display screen that displays the point on the driving route at which an image in which an abnormality was detected was captured when inspecting road surface conditions while driving a car or the like and capturing images with the imaging device 20. In Fig. 10, a map information display area 91 and an inspection result display area 95 are displayed.

[0062] In the map information display area 91, a map, a driving route on the map 92, and thumbnail images of target images in which abnormalities have been detected, along with their attribute information, are displayed in a balloon 93. In addition, as shown in the upper left of the map information display area 91, a menu may be displayed so that the type of abnormal image to be displayed on the map can be selected.

[0063] The inspection result display area 95 displays inspection data, detected anomalies, which are a list of detected anomalies, and the selected anomaly. As inspection data, for example, the inspection date, start point, end point, and number of detected anomalies are displayed in association with each other. As detected anomalies, an ID that uniquely identifies the image containing the anomaly, the type of anomaly, the time and date of the image capture are displayed. As the selected anomaly, the target image and detailed attribute information about the anomaly included in the map information display area 91 are displayed.

[0064] 10, the speech bubble 93 with ID: 001 is selected, and the target image with ID: 001 and its attribute information are displayed on the image of the selected abnormality. Note that the image may be switched to a superimposed image by clicking the target image with the mouse cursor 96, or by touching the image if it is a touch screen.

[0065] By selecting "Go to detection details" in the item of the selected anomaly, a separate window may open, and you may transition to a display screen for the detection details, such as an enlarged view or likelihood map. For example, you may transition to a display screen for the detection results shown in Figure 9.

[0066] Also, a selection button 94 may be provided to enable transition to data on different anomaly detection results, such as data on a different route or a different day.

[0067] While FIG. 10 shows an example in which the inspection target is a road surface, the inspection target may be selectable from road surfaces, guardrails, sound barriers, etc. For example, a user may manually classify images of guardrails to set a set of target images for inspection. Alternatively, a trained model that performs semantic segmentation processing may be applied to the target images, and the segmentation results of road surfaces, guardrails, etc. for each pixel may be superimposed on a likelihood map to determine which segmentation result corresponds to an area estimated to have an abnormality, thereby identifying and classifying the inspection target.

[0068] According to the second embodiment described above, the reference image selection unit selects a normal image similar to the target image as a reference image, and performs inference on the target image using the trained model generated by the learning device according to the first embodiment, thereby enabling highly accurate anomaly detection.

[0069] Next, an example of the hardware configuration of an information processing device 10 including the learning device 11 and the image processing device 13 according to each of the above-described embodiments is shown in the block diagram of FIG.

[0070] The information processing device 10 includes a CPU (Central Processing Unit) 71, a RAM (Random Access Memory) 72, a ROM (Read Only Memory) 73, a storage 74, a display device 75, an input device 76, and a communication device 77, each of which is connected by a bus.

[0071] The CPU 71 is a processor that executes arithmetic processing, control processing, and the like according to a program. The CPU 71 uses a predetermined area of ​​the RAM 72 as a work area and executes the processing of each unit of the information processing device 10 described above in cooperation with programs stored in the ROM 73, storage 74, and the like. Note that each process of the information processing device 10 may be executed by one processor or may be executed in a distributed manner by multiple processors.

[0072] The RAM 72 is a memory such as an SDRAM (Synchronous Dynamic Random Access Memory), and functions as a work area for the CPU 71. The ROM 73 is a memory that stores programs and various information in a non-rewritable manner.

[0073] The storage 74 is a device that writes and reads data to a magnetic recording medium such as a hard disk drive (HDD), a semiconductor storage medium such as a flash memory, a magnetically recordable storage medium such as a HDD, an optically recordable storage medium, etc. The storage 74 writes and reads data to the storage medium in accordance with control from the CPU 71.

[0074] The display device 75 is a display device such as an LCD (Liquid Crystal Display), etc. The display device 75 displays various information based on a display signal from the CPU 71.

[0075] The input device 76 is an input device such as a mouse, a keyboard, etc. The input device 76 receives information input by a user as an instruction signal, and outputs the instruction signal to the CPU 71.

[0076] The communication device 77 communicates with external devices via a network under the control of the CPU 71 . In addition, if the learning device 11 and the image processing device 13 are configured as independent devices rather than being included as part of the information processing device 10, the learning device 11 and the image processing device 13 may each have the hardware configuration shown in Figure 11.

[0077] The instructions shown in the processing procedures described in the above-described embodiments can be executed based on a software program. A general-purpose computer system can store this program in advance and, by loading this program, achieve effects similar to those achieved by the control operations of the information processing device described above. The instructions described in the above-described embodiments can be recorded as a computer-executable program on a magnetic disk (such as a flexible disk or hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, or Blu-ray (registered trademark) Disc), a semiconductor memory, or a similar recording medium. The recording medium may take any storage format as long as it is readable by a computer or embedded system. A computer can achieve operations similar to those of the information processing device described in the above-described embodiments by loading the program from the recording medium and having the CPU execute the instructions described in the program based on the program. Of course, the computer may acquire or load the program via a network.

[0078] In addition, an OS (operating system), database management software, network middleware, etc. running on a computer may execute some of the processes required to realize this embodiment based on instructions from a program installed on the computer or embedded system from a recording medium. Furthermore, the recording medium in this embodiment is not limited to a medium independent of a computer or an embedded system, but also includes a recording medium that stores or temporarily stores a program downloaded via a LAN, the Internet, or the like. Furthermore, the number of recording media is not limited to one, and cases where the processing in this embodiment is executed from multiple media are also included in the recording media in this embodiment, and the media may have any configuration.

[0079] The computer or embedded system in this embodiment is for executing each process in this embodiment based on a program stored on a recording medium, and may be configured as either a device consisting of a single device such as a personal computer or a microcomputer, or a system in which multiple devices are connected to a network. Furthermore, the computer in this embodiment is not limited to a personal computer, but also includes an arithmetic processing unit, a microcomputer, etc. included in information processing equipment, and is a general term for equipment or devices that can realize the functions in this embodiment by a program.

[0080] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0081] 1...image inspection system, 10...information processing device, 11...learning device, 13...image processing device, 20...imaging device, 41, 41-1, 41-N...target image, 51, 52...image, 60...target image, 61...first feature map, 62...second feature map, 63...differential feature map, 65...reference image, 66...likelihood map, 71...CPU, 72...RAM, 73...ROM, 74...storage, 75...display device, 76...input device, 77...communication device, 91...map information display area, 9 2...driving route, 94...selection button, 95...inspection result display area, 96...mouse cursor, 111...target image acquisition unit, 112...teaching label acquisition unit, 113...normal image acquisition unit, 114...similarity calculation unit, 115...reference image selection unit, 116...feature map calculation unit, 117...differential feature map calculation unit, 118...likelihood map calculation unit, 119...likelihood calculation unit, 120...loss calculation unit, 121...model update unit, 122...storage unit, 131...image processing unit, 132...information output unit

Claims

1. an acquisition unit that acquires a target image and at least one normal image; a similarity calculation unit that calculates a similarity between the target image and the normal image; a selection unit that selects a reference image from the normal image based on the similarity; a feature map calculation unit that calculates a first feature map of the target image and a second feature map of the reference image using a first machine learning model; a differential feature map calculation unit that calculates a differential feature map that is a difference between the first feature map and the second feature map; a likelihood map calculation unit that calculates a likelihood map based on the first feature map and the differential feature map using a second machine learning model; a loss calculation unit that calculates a loss based on the likelihood based on the likelihood map and the teaching label of the target image; an update unit that updates the first machine learning model and the second machine learning model based on the loss; A learning device comprising:

2. the acquisition unit acquires a plurality of normal images, The learning device according to claim 1 , wherein the selection unit probabilistically selects the at least one reference image from a plurality of normal images based on the similarity.

3. the acquisition unit acquires a plurality of normal images, The learning device according to claim 1 , wherein the selection unit randomly selects the at least one reference image from a plurality of normal images whose similarity is equal to or greater than a threshold value.

4. The learning device according to claim 1 , wherein the selection unit selects at least one of the reference images according to a selection probability calculated from the similarity.

5. The learning device according to claim 4 , wherein the selection probability has a larger value as the degree of similarity increases.

6. The learning device according to claim 1 , wherein the feature map calculation unit calculates a plurality of first feature maps each having a different output size from an intermediate layer of the first machine learning model.

7. The learning device according to claim 6 , wherein the differential feature map calculation unit calculates a plurality of differential feature maps corresponding to the plurality of first feature maps, respectively.

8. the acquisition unit acquires a plurality of normal images, the selection unit selects a plurality of reference images from the normal image based on the similarity; The learning device according to claim 1 , wherein the feature map calculation unit calculates statistics of the feature maps of the plurality of reference images as the second feature map.

9. The learning device according to claim 1 , wherein the similarity calculation unit calculates the similarity based on a feature amount obtained by a neural network.

10. acquiring a target image and at least one normal image; Calculating a similarity between the target image and the normal image; selecting a reference image from the normal image based on the similarity; calculating a first feature map of the target image and a second feature map of the reference image using a first machine learning model; calculating a differential feature map that is a difference between the first feature map and the second feature map; calculating a likelihood map based on the first feature map and the differential feature map using a second machine learning model; calculating a loss based on the likelihood based on the likelihood map and the teaching label of the target image; updating parameters of the first machine learning model and the second machine learning model based on the loss; How to learn.

11. Computer, an acquisition means for acquiring a target image and at least one normal image; a similarity calculation means for calculating a similarity between the target image and the normal image; a selection means for selecting a reference image from the normal image based on the similarity; a feature map calculation means for calculating a first feature map of the target image and a second feature map of the reference image using a first machine learning model; a differential feature map calculation means for calculating a differential feature map that is a difference between the first feature map and the second feature map; a likelihood map calculation means for calculating a likelihood map based on the first feature map and the differential feature map using a second machine learning model; a loss calculation means for calculating a loss based on the likelihood based on the likelihood map and the teaching label of the target image; A learning program that functions as an update means that updates the first machine learning model and the second machine learning model based on the loss.

12. an acquisition unit that acquires a target image to be inspected and at least one normal image; a similarity calculation unit that calculates a similarity between the target image and the plurality of normal images; a selection unit that selects a reference image from the normal image based on the similarity; a feature map calculation unit that calculates a first feature map of the target image and a second feature map of the reference image using a first trained model trained by the learning device according to any one of claims 1 to 9; a differential feature map calculation unit that calculates a differential feature map that is a difference between the first feature map and the second feature map; a likelihood map calculation unit that calculates a likelihood map based on the first feature map and the differential feature map by using a second trained model trained by the learning device according to any one of claims 1 to 9; an image processing unit that generates output information relating to the target image and the likelihood map; An image processing device comprising:

13. The image processing device according to claim 12 , wherein the selection unit selects the normal image having the highest similarity as the reference image.

14. the acquisition unit acquires a plurality of normal images, The image processing device according to claim 12 , wherein the selection unit selects a plurality of normal images as the reference images in descending order of the similarity.

Images