Apparatus, method, and program for creating similar image sets

Abstract

To reduce the time and effort for creating a similar image set including an image that a user pays attention to, and an image similar to the image, from among many images.SOLUTION: A similar image set creating apparatus includes an acquisition unit, a first extraction unit, a second extraction unit, and a selection unit. The acquisition unit acquires a plurality of images. The first extraction unit extracts a plurality of first features from the images by using a first model that executes an image classification task. The second extraction unit extracts a plurality of second features from the images by using a second model that executes an image classification task. The second model is trained in such a manner that mutually similar images in a latent space are continuously distributed, compared to the first model. The selection unit selects, from the images, an image of interest serving as a reference of a similar image set, and an auxiliary image similar to the image of interest, on the basis of the first features and the second features.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to a similar image set creation device, method, and program.

Background Art

[0002] In the manufacturing field, efforts are expanding to monitor the occurrence of defects and improve productivity by classifying appearance images of products through machine learning. As a method for classifying appearance images by machine learning, supervised learning is used, in which teacher labels serving as classification criteria are manually assigned in advance, and a classification model is learned by a method such as deep learning. In order to learn a highly accurate classification model, it is necessary to accurately assign teacher labels to a large number of images.

[0003] In manufacturing, it may be difficult to accurately assign labels manually. For example, when assigning labels of good and defective products based on the color, size of scratches, gloss, etc. of product images, the criteria for distinguishing between good and defective products are not clear, and labels may be assigned according to different criteria by workers, or the criteria may change during the work.

[0004] In order to make the label assignment criteria constant, an image sample called a limit sample is created. The limit sample clarifies the label judgment criteria by showing examples of a plurality of images serving as samples of good and defective products. By performing label assignment based on the limit sample, even when a plurality of workers perform label assignment, they can work according to common criteria. Such limit samples are used not only for label assignment work for machine learning but also for visual inspections without using machine learning.

[0005] While limit samples standardize labeling and inspection criteria, they tend to be costly to create and use. Creating limit samples requires inspection specialists who are familiar with variations in the product's color, scratch size, gloss, etc., to select the samples. Furthermore, the characteristics of good and defective products may change depending on changes in the product's manufacturing process, requiring limit samples to be updated each time. In recent years, factories have been producing small quantities of a wide variety of products to meet specific needs, and each product may have multiple inspection processes, making the creation and use of limit samples for all inspections a costly undertaking. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] U.S. Patent No. 11,288,805 [Non-patent literature]

[0007] [Non-Patent Document 1] Yaling Tao, Kentaro Takagi, Kouta Nakata. “Clustering Friendly representation learning via instance discrimination and feature decorrelation”, arXiv:2106.00131 (ICLR2021) [Non-Patent Document 2] Wu, Zhirong, et al. "Unsupervised feature learning via non-parametric instance discrimination." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018. [Overview of the project] [Problems that the invention aims to solve]

[0008] The problem that the present invention aims to solve is to provide a similar image set creation device, method, and program that can reduce the effort required to create a similar image set that includes an image of interest to the user and images similar to that image from among a large number of images. [Means for solving the problem]

[0009] The similar image set creation apparatus according to the embodiment includes an acquisition unit, a first extraction unit, a second extraction unit, and a selection unit. The acquisition unit acquires a plurality of images. The first extraction unit extracts a plurality of first features from the plurality of images using a first model that performs an image classification task. The second extraction unit extracts a plurality of second features from the plurality of images using a second model that performs an image classification task. Here, the second model is trained so that similar images in the latent space are distributed more continuously than those in the first model. The selection unit selects a target image to serve as the basis for the similar image set and auxiliary images similar to the target image from among the plurality of images, based on the plurality of first features and the plurality of second features. [Brief explanation of the drawing]

[0010] [Figure 1] A diagram showing an example configuration of a similar image set creation device. [Figure 2] Diagram illustrating an examination image. [Figure 3] Diagram illustrating the first feature xp q. [Figure 4] Diagram showing an overview of the first model [Figure 5] A diagram illustrating the second feature Xp r. [Figure 6] Diagram showing an overview of the second model [Figure 7] This diagram schematically illustrates the selection process between the target image and the auxiliary image by the selection unit. [Figure 8] This diagram shows the processing procedure for creating limit samples using a similar image set creation device. [Figure 9] This diagram shows an example of a display screen for representative images of a cluster. [Figure 10]Figure showing an example of a selection screen for a target image [Figure 11] Figure showing an example of a selection screen for a target image [Figure 12] Figure illustrating the first selection method for auxiliary images [Figure 13] Figure illustrating the second selection method for auxiliary images [Figure 14] Figure illustrating a display screen of a set of similar images [Figure 15] Figure illustrating another display screen of a set of similar images [Figure 16] Figure illustrating a display screen of a limit sample

Embodiments for Carrying out the Invention

[0011] Hereinafter, a similar image set creation apparatus, method, and program according to this embodiment will be described with reference to the drawings.

[0012] FIG. 1 is a diagram showing a configuration example of a similar image set creation apparatus 100 according to this embodiment. As shown in FIG. 1, the similar image set creation apparatus 100 is a computer having a processing circuit 1, a storage device 2, an input device 3, a communication device 4, and a display device 5. Data communication between the processing circuit 1, the storage device 2, the input device 3, the communication device 4, and the display device 5 is performed via a bus or the like.

[0013] The processing circuit 1 includes a processor such as a CPU (Central Processing Unit) and memory such as RAM (Random Access Memory). By executing a program, the processing circuit 1 realizes an image acquisition unit 11, a first extraction unit 12, a second extraction unit 13, a selection unit 14, a limit sample creation unit 15, and a display control unit 16. The program is stored in a non-temporary recording medium readable by the processor. The program may be stored in a fixed recording medium such as a storage device 2, or in a portable recording medium. The hardware implementation of the processing circuit 1 is not limited to the above embodiment. For example, it may be composed of circuits such as an Application Specific Integrated Circuit (ASIC) that realize the image acquisition unit 11, the first extraction unit 12, the second extraction unit 13, the selection unit 14, the limit sample creation unit 15, and the display control unit 16. Each of these units 11 to 16 may be implemented in a single integrated circuit, or they may be implemented individually in multiple integrated circuits.

[0014] The image acquisition unit 11 acquires multiple images. The subject of the images in this embodiment is not particularly limited. The image capture equipment, shooting conditions, number of pixels, etc., are also not particularly limited. The multiple images acquired become candidates for images that constitute a set of similar images and, consequently, limit samples. For example, in the inspection of a manufacturing process, inspection images of manufactured goods are acquired. The inspection images may be acquired from shooting equipment installed in various manufacturing processes or from a computer that stores inspection images generated by such shooting equipment. Each of the multiple images acquired shows either a normal subject or a defective subject. Images showing normal subjects do not necessarily need to be acquired.

[0015] Figure 2 illustrates an example of an inspection image according to this embodiment. As shown in Figure 2, N (where N is a natural number greater than or equal to 2) inspection images are acquired. The inspection images are assumed to be semiconductor inspection images taken with an electron microscope of semiconductors manufactured in a semiconductor manufacturing plant. The inspection images show various semiconductors, including normal semiconductors and semiconductors with defects. As shown in Figure 2, the defects come in various sizes and shapes. The shapes of the defects vary, including circles and polygons. In some cases, the pattern of the product, such as vertical stripes, may be reflected in the background of the image.

[0016] The first extraction unit 12 extracts N first features from N images using a first model that performs an image classification task. The first model is a deep neural network that converts images into first features of lower dimension relative to the image itself. Through unsupervised representation learning, its parameters are trained so that the distance in the latent space between similar images becomes smaller, and the distance in the latent space between dissimilar images becomes larger. In this embodiment, the latent space is synonymous with the space defined by the features. The first model is stored in the memory device 2.

[0017] Figure 3 shows the first feature x p q This is an illustrative diagram. The subscript p represents the image number, and the subscript q represents the dimension (element) number of the first feature. In this embodiment, the dimension of the first feature is assumed to be 128. In this case, the first latent space, which is the space of the first feature, will have 128 dimensions.

[0018] The first model is trained, for example, using the technology described in Non-Patent Document 1. Non-Patent Document 1 proposes a representation learning method that uses a deep neural network to extract features from images without using teacher labels. Specifically, the weight parameters of the first model are trained to minimize the loss function L defined in equation (1) below. As shown in equation (1), the loss function L is a weighted sum of loss function L1 and loss function L2. The balancing parameter λ adjusts the influence of the first loss function L1 and loss function L2 on the loss function L. The balancing parameter λ can be set to any value.

[0019]

number

[0020] The loss function L1 is a loss function for image classification. The loss function L1 evaluates the error between the features of the images. Specifically, the loss function L1 is defined by equation (2) below. Here, n represents the number of images for which the loss function L1 is calculated. The subscripts i and j are the sequential numbers of the two types of images. i v is a vector representing the output (first feature) of the first model based on image i, and v j is a vector representing the output (first feature) of the first model based on image j. τ1 is the first temperature parameter, which adjusts the sensitivity of the inner product between vectors v. A smaller τ1 results in higher sensitivity, while a larger τ1 results in lower sensitivity.

[0021]

number

[0022] The loss function L2 is a loss function related to feature uncorrelation. The loss function L2 evaluates the correlation between the elements of the feature. The loss function L2 is defined by equation (3) below, where d represents the dimensionality of the feature, and l and m are the sequential numbers of the image elements. l f is a vector formed by arranging the l-th elements of the output (first feature) of the first model based on image l, and f mis a vector formed by arranging the m-th elements of the output (first feature) of the first model based on image m. τ2 is a second temperature parameter that adjusts the sensitivity of the inner product between vectors f. A smaller τ2 results in higher sensitivity, while a larger τ2 results in lower sensitivity. Non-patent document 1 describes a method for generating a model that extracts features that cause images to be distributed in a clustered manner by using a certain range of parameters, thereby making similar images closer together and dissimilar images further apart.

[0023]

number

[0024] Figure 4 shows an overview of the first model. As shown in Figure 4, the first model takes an image as input and outputs a first feature. Since the first model is trained based on equation (1), it can generate a first model that extracts first features in which similar images are closer together and dissimilar images are further apart compared to the second model. The first model can extract first features that are suitable for clustering, allowing relatively similar images to be classified into the same cluster and relatively dissimilar images to be classified into different clusters. The first features tend to have lower continuity in the image distribution compared to the second features, which are the output of the second model.

[0025] The second extraction unit 13 extracts N secondary features from N images using a second model that performs an image classification task. The second model is a deep neural network that converts images into secondary features with lower dimensions relative to the image itself. Similar to the first model, its parameters are trained through unsupervised representation learning so that the distance in the latent space between similar images becomes smaller and the distance in the latent space between dissimilar images becomes larger. The second model is stored in the memory device 2.

[0026] Figure 5 shows the second feature X. p rThis is a diagram illustrating this. The subscript p represents the image number, and the subscript r represents the dimension (component) number of the second feature. In this embodiment, the dimension of the second feature is assumed to be 128, the same as the first feature. In this case, the second latent space, which is the space of the second feature, will have 128 dimensions. Note that the dimensions of the second feature and the first feature may be different.

[0027] The second model is trained based on a different loss function than the first model. As an example, the second model is trained using the technique described in Non-Patent Literature 2. Non-Patent Literature 2 proposes a representation learning method that uses deep neural networks to extract features from images without using supervising labels. Specifically, the weight parameters of the second model are trained to minimize the loss function L defined in equation (4) below. As shown in equation (4), the loss function L for representation learning of the second model is the loss function L1 shown in equation (1).

[0028]

number

[0029] Figure 6 shows an overview of the second model. As shown in Figure 6, the second model takes an image as input and outputs a second feature. Since the second model is trained based on equation (4), it can generate a second model that extracts second features that bring similar images closer together. The second model brings similar images closer together in the second latent space and can extract second features that are continuously distributed in the second latent space. The second features tend to have a higher degree of continuity in the distribution of images in the second latent space compared to the first features, which are the output of the first model.

[0030] As shown in Figure 1, the selection unit 14 selects a subject image to serve as the basis for a similar image set and auxiliary images similar to the subject image from among N images, based on N first features and N second features. The subject image and auxiliary images constitute a similar image set. There may be one or more subject images and auxiliary images. As shown in Figure 1, the selection unit 14 has a first selection unit 17 and a second selection unit 18. The first selection unit 17 selects two or more subject images from N images based on N first features. The second selection unit 18 selects two or more auxiliary images similar to the subject images from among N images, based on N second features.

[0031] Figure 7 schematically shows the selection process of the selection unit 14 between the target images 71 and 72 and the auxiliary images 73, 74, and 75. As shown in the left diagram of Figure 7, the first selection unit 17 clusters N images in the first latent space LS1 using the first model. As described above, clustering by the first model tends to result in large distances between dissimilar images in the first latent space LS1 and low continuity of the image distribution. This means that similar images tend to form clusters. Images belonging to the same cluster have similar image features of defects. By checking the representative image of each cluster, it becomes possible to check the image features of defects without having to check all N images. Therefore, the first selection unit 17 selects the target images 71 and 72 from the representative images of each cluster. In Figure 7, it is assumed that two target images have been selected.

[0032] Next, as shown in the right-hand figure of Figure 7, the second selection unit 18 uses the second model to convert N images into features in the second latent space LS2. As described above, the features obtained by the second model tend to have small distances between dissimilar images in the second latent space LS2 and high continuity in the image distribution. In the second model, since the N images tend to be continuously distributed in the second latent space LS2, it is suitable for automatically selecting auxiliary images whose image features are between the target images. In the first model, images in adjacent clusters are not necessarily similar. Therefore, the second selection unit 18 selects images located on the path passing through the two target images 71 and 72 in the second latent space LS2 as auxiliary images 73, 74, and 75. This makes it possible to select auxiliary images 73, 74, and 75 that interpolate the image features of the two target images 71 and 72. It is also possible to select images located on the path but outside the two target images 71 and 72 as auxiliary images. This makes it possible to select auxiliary images that extrapolate the image features of the two target images 71 and 72.

[0033] The limit sample creation unit 15 creates a limit sample based on the focus image and auxiliary image selected by the selection unit 14. The limit sample includes an image representing the inspection target located at the boundary between good and defective products. The limit sample may be specified from among the focus image and auxiliary image selected by the selection unit 14, or it may be generated by image processing based on the focus image and auxiliary image.

[0034] The display control unit 16 displays various information via the display device 5. For example, the display control unit 16 displays images, featured images, auxiliary images, similar image sets, limit samples, etc., via the display device 5.

[0035] The storage device 2 is composed of ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), integrated circuit storage, etc. The storage device 2 stores various calculation results from the processing circuit 1 and various programs executed by the processing circuit 1.

[0036] Input device 3 receives various commands from the operator. Input device 3 can include a keyboard, mouse, various switches, touchpad, touch panel display, etc. Output signals from input device 3 are supplied to processing circuit 1. Input device 3 may also be a computer connected to processing circuit 1 via wired or wireless connection.

[0037] Communication device 4 is an interface for communicating information with external devices connected to the similar image set creation device 100 via a network.

[0038] The display device 5 displays various information according to the control of the display control unit 16. The display device 5 can be a CRT (Cathode-Ray Tube) display, a liquid crystal display, an organic EL (Electro-Luminescence) display, an LED (Light-Emitting Diode) display, a plasma display, or any other display known in the art.

[0039] The details of the similar image set creation device 100 according to this embodiment will be described below.

[0040] Figure 8 shows the processing procedure for creating limit samples using the similar image set creation device 100. The first and second models are assumed to have been generated in advance and stored in the storage device 2.

[0041] As shown in Figure 8, the image acquisition unit 11 first acquires an image dataset (step S1). The image dataset is assumed to contain N images. Each of the N images is assigned an ID to identify it.

[0042] When step S1 is performed, the first extraction unit 12 extracts first features using the first model (step S2). In step S2, the first extraction unit 12 reads the first model from the storage device 2 and applies the first model to N images to extract N first features. The extracted N first features are stored in the storage device 2. Each of the N first features is assigned the ID of the image from which the first feature was extracted.

[0043] When step S2 is performed, the second extraction unit 13 extracts second features using the second model (step S3). In step S3, the second extraction unit 14 reads the second model from the storage device 2 and applies the second model to N images to extract N second features. The extracted N second features are stored in the storage device 2. Each of the N second features is assigned the ID of the image from which the second feature was extracted.

[0044] When step S3 is performed, the first selection unit 17 selects an image of interest (step S4). In step S4, the first selection unit 17 classifies the N images into multiple clusters based on the N first features extracted in step S3. Specifically, the first selection unit 17 plots the first features of each of the N images in the first latent space, which is the space of the first features. The first selection unit 17 then clusters the N images plotted in the first latent space using a general clustering method such as k-means. The first latent space is set to have the same number of dimensions as the number of first features, for example. If the number of first features is 128-dimensional, the first latent space is also set to 128 dimensions. The first selection unit 17 may also form a first latent space reduced to 2 or 3 dimensions. The reduction algorithm is not particularly limited, but for example, t-SNE (t-distributed stochastic neighbor embedding) may be used. Next, the first selection unit 17 selects multiple representative images that represent each of the multiple clusters. The display control unit 16 displays multiple representative images on the display device 5 in a selectable format via the input device 3. The first selection unit 17 then selects the representative image selected via the input device 3 from among the multiple representative images as the image of interest.

[0045] Figure 9 shows an example of the display screen I1 for representative images of clusters. As shown in Figure 9, the display screen shows the cluster number, the representative image for each cluster, and the number of images belonging to each cluster. In this embodiment, the number of clusters is set to 50, and clustering is performed using the first feature. As a result of clustering, for example, cluster number "1" has 1000 images belonging to it, and the two representative images shown in the middle column are selected and displayed. As the representative image for each cluster, for example, an image close to the cluster central feature is selected from the images belonging to each cluster. The cluster central feature refers to the mean, median, or other statistical value of the first feature of the images belonging to each cluster. In the clustering results in Figure 9, it can be seen that circular defects of different sizes are classified into different clusters. Note that the number of representative images per cluster is not limited to two; it may be one or three or more. Also, the number of representative images may differ depending on the cluster.

[0046] Figure 10 shows an example of the selection screen I2 for the featured image. As shown in Figure 10, the selection screen I2 displays a display area I21 for the representative image display screen, a display area I22 for the featured image, and an auxiliary image selection button I23. The display area I21 displays the representative image display screen shown in Figure 9, with the display range adjustable by a slider bar I24. The display area I22 displays the featured image selected by the user via the input device 3. Figure 10 shows the selection screen I2 for selecting two featured images. There are various ways to select the featured image. For example, one method is to click on any representative image displayed in the display area I21. In this case, the clicked representative image is displayed as the featured image in the display area I22. Another selection method is to drag and drop any representative image displayed in the display area I21 to the display area I22. In this case, the dragged and dropped representative image is displayed as the featured image in the display area I22. The featured images in the display area I22 may be displayed in a sortable manner via the input device 3.

[0047] When creating a good product limit sample or a defective product limit sample, the two featured images should ideally be good and defective images related to the image features or defect types that the user is interested in. For example, if the image feature or defect type that the user is interested in is the size of the defect, then an image showing a defect of a size classified as good and an image showing a defect of a size classified as defective should be selected. The two featured images may be selected from different clusters or from the same cluster. Note that the featured images are not limited to the image features or defect types of interest; they may also be the shape, color, gloss, or background image pattern of the defect.

[0048] If two target images are selected, the auxiliary image selection button I23 is pressed via the input device 3. When the auxiliary image selection button I23 is pressed, the auxiliary image selection process (step S5) is performed.

[0049] Figure 11 shows another example of the selection screen I3 for the image of interest. As shown in Figure 11, the selection screen I3 displays the display area I31 for the representative image, the display area I32 for the image of interest, and the auxiliary image selection button I33. The display area I31 and the auxiliary image selection button I33 are the same as the display area I21 and the auxiliary image selection button I23 shown in Figure 10, respectively. Three images of interest can be selected on the selection screen I3, and the three images of interest are displayed in the display area I32. In Figure 11, as an example, three images of interest are selected and displayed in order according to criteria such as the size of the defect. The method for selecting images of interest is the same as the method shown in Figure 10. Note that the number of images of interest is not limited to two or three, but may be four or more. Also, the images of interest in the display area I32 may be displayed in a reorderable manner via the input device 3.

[0050] When step S4 is performed, the second selection unit 18 selects an auxiliary image (step S5). In step S5, the second selection unit 18 selects an auxiliary image whose image features are similar to the image of interest selected in step S4, utilizing the continuity of the distribution of similar images in the second latent space. Two methods for selecting auxiliary images are described below in detail.

[0051] In the first method of selecting auxiliary images, the second selection unit 18 calculates a line passing through the first and second focus images among the two or more focus images selected in step S4 in the second latent space relating to N second features, and selects as an auxiliary image an image from among the N images whose distance from this line is less than a threshold.

[0052] Figure 12 illustrates the first method of selecting auxiliary images. Specifically, as shown in Figure 12, the second selection unit 18 plots the second feature of each of the N images in the second latent space LS2, which is the space of the second feature. The second selection unit 18 may reduce the dimensionality of the second feature to 2 or 3 dimensions to form the second latent space LS2 in order to improve the continuity of the image distribution. The reduction algorithm is not particularly limited, but for example, t-SNE may be used. If the continuity of the image distribution can be ensured, a second latent space reduced to 4 dimensions or more may be used, or an unreduced second latent space (128-dimensional space) may be used. Figure 12 illustrates a 2-dimensional second latent space LS2.

[0053] The second latent space LS2 contains the first image of interest (more specifically, the second feature of the first image of interest) P1 and the second image of interest (more specifically, the second feature of the first image of interest) P2, which were selected in step S4. The second selection unit 18 calculates a straight line L1 that passes through the first image of interest P1 and the second image of interest P2. The second selection unit 18 calculates the distance in the second latent space LS2 to each of the N images' second features relative to the line L1. The second selection unit 18 then selects images whose distance in the second latent space LS2 is less than a threshold as images in the vicinity of the line L2. The selected images are set as auxiliary images. This threshold can be set to any value. Note that the calculation of the line L2 and the calculation of the distance in the second latent space LS2 can be done using general geometric methods.

[0054] By extending the line L1 beyond the boundary between the first image P1 and the second image P2, it is possible to select not only images with defect features that interpolate the image features of the first image P1 and the second image P2, but also images with defect features that extrapolate the image features of the first image P1 and the second image P2 as auxiliary images. Conversely, by keeping the line L1 inside the boundary between the first image P1 and the second image P2, it is possible to select only images with defect features that interpolate the image features of the first image P1 and the second image P2 as auxiliary images. The line L1 may also be a curve.

[0055] In the second method for selecting auxiliary images, the second selection unit 18 selects an auxiliary image from among the N images by performing a path search between the first and second focus images among the two or more focus images selected in step S4 in the second latent space relating to the N second features.

[0056] Figure 13 illustrates a second method for selecting auxiliary images. As shown in Figure 13, the second features of N images are plotted in the second latent space LS2. The second latent space LS2 includes the first focus image P1 and the second focus image P2, which were selected in step S4. The second selection unit 18 sets the first focus image P1 as the starting point and the second focus image P2 as the ending point, and performs a path search using the plots of other images as nodes along the path from the starting point P1 to the ending point P2 in terms of the second features, and selects multiple auxiliary images that have the second features between the first focus image P1 and the second focus image P2. In Figure 13, an example is shown in which images P3 and P4 are selected as auxiliary images. Dijkstra's algorithm, a general method, can be used for the path search.

[0057] In the selection of auxiliary images using pathfinding, the number of selected auxiliary images can be adjusted, for example, by restricting the movement between nodes to a distance greater than or equal to a threshold. The threshold for the distance moved in one step can be set to the value of any quantile, such as the quartile, decitile, or hypothetical quantile, using the distribution of distances between nodes. Alternatively, the threshold may be set based on the distance between representative images selected by clustering using the first feature, using the distance of the second feature.

[0058] According to the first and second selection methods, auxiliary images similar to the two target images can be efficiently selected by utilizing the continuous distribution of other images between the two target images in the second latent space. According to the first selection method, images that are close to the line passing through the two target images are selected as auxiliary images, making it simpler to select auxiliary images compared to the second selection method. According to the second selection method, auxiliary images are selected by sequentially selecting images whose second feature quantity is within a threshold distance from the first target image (start point) to the second target image (end point), making it possible to select auxiliary images in such a way that the image features change in stages.

[0059] Furthermore, if three or more focus images are selected, the second selection unit 18 selects an auxiliary image for each combination of two of those three or more focus images. This makes it possible to select an auxiliary image in the same way as when there are two focus images. For example, in Figure 11, for each combination of focus image 1 and focus image 2, and focus image 2 and focus image 3, which the user has arranged in order of defect size, an auxiliary image can be selected using either the first selection method or the second selection method described above. If necessary, an auxiliary image may also be selected for the combination of focus image 1 and focus image 3 using the same method.

[0060] When step S5 is performed, the display control unit 16 displays a set of similar images (step S6). In step S6, the display control unit 16 displays the focus image selected in step S4 and the auxiliary image selected in step S5 as a set of similar images on the display device 5.

[0061] Figure 14 illustrates a display screen I4 for a similar image set. Figure 14 illustrates an embodiment in which the image feature of interest is defect size, and two target images I41 and I45 are specified. As shown in Figure 14, the display screen I4 displays the two target images and three auxiliary images as a similar image set. The defect sizes of the three auxiliary images are located midway between the defect sizes of the two target images. The two target images and the three auxiliary images are arranged in order of defect size. By displaying a similar image set consisting of the target images and auxiliary images, the user can visually confirm the continuity of image features such as defect size. Furthermore, as a basis for selecting the auxiliary images, the projected image of the second latent space shown in Figure 12 or Figure 13 may be displayed on the display screen I4. In this case, it is preferable that the plot of the auxiliary images in the second latent space be visually emphasized by color, brightness, size, etc. Furthermore, the plot of the target image in the second latent space may be visually emphasized by color, brightness, size, etc.

[0062] If three or more images of interest are selected in step S4, the display control unit 16 displays an auxiliary image on the display device 5 for each combination of two images of interest. Specifically, for multiple combinations, the display control unit 16 displays the images of interest and the auxiliary images aligned along one axis with respect to image features such as defect size.

[0063] Figure 15 illustrates the display screen I5 of the similar image set. Figure 15 illustrates an example where three focus images are specified. As shown in Figure 15, the display screen I5 shows three focus images and six auxiliary images as a similar image set. The defect sizes of the three auxiliary images 1, 2, and 3 are located midway between the defect sizes of focus image 1 and focus image 2, and the defect sizes of the three auxiliary images 4, 5, and 6 are located midway between the defect sizes of focus image 2 and focus image 3. The three focus images and six auxiliary images are arranged in order of defect size. This allows the user to visually perceive the continuity of image features such as defect size, even when more than three focus images are selected.

[0064] However, the manner in which the set of similar images is displayed when three or more featured images are selected is not limited to this. For example, the display control unit 16 may display multiple combinations in a vertical arrangement, or each of the multiple combinations may be displayed in a separate window.

[0065] When step S6 is performed, the limit sample creation unit 15 creates a limit sample (step S7). In step S7, the limit sample creation unit 15 creates a limit sample based on the focus image selected in step S4 and the auxiliary image selected in step S5. Specifically, the limit sample creation unit 15 selects a limit sample from the set of similar images shown in Figures 14 and 15, following instructions from the user via the input device 3. The limit sample may be a good product limit sample, which is a good product close to a defective product; a defective product limit sample, which is a defective product close to a good product; or both. When a limit sample is selected, the display control unit 16 displays the display screen of the limit sample on the display device 5.

[0066] Figure 16 illustrates the display screen I6 for limit samples. The display screen I6 in Figure 16 illustrates the case where the focus image 1 shown in Figure 14 is selected as a good limit sample, and the auxiliary image 1 is selected as a defective limit sample. The display screen I6 displays a set of similar images, including the focus image and the auxiliary image, clearly indicating that they are good limit samples and / or defective limit samples. This allows the user to visually identify limit samples within a series of image features such as various defect sizes. In this case, to clearly distinguish between good and defective limit samples, it is preferable to label the focus image 1 as a good limit sample and the auxiliary image 1 as a defective limit sample. In addition, a mark I61 indicating the boundary between good and defective limit samples may be displayed to clearly indicate the boundary.

[0067] Limit samples are useful for tasks such as assigning training labels for machine learning classification models of defects and flaws, as well as for visual inspections that do not involve machine learning.

[0068] This completes the process of creating the limit sample.

[0069] Note that the processing procedure for creating the limit sample shown in Figure 8 is just one example, and various deletions, additions, and / or modifications are possible as long as they do not deviate from the essence of the invention.

[0070] (Variation 1) In the above embodiment, two or more images of interest are selected in step S4. However, the first selection unit 17 in Modification 1 may select only one image of interest. This is expected to reduce the effort required to select the image of interest and increase the variety of auxiliary images that can be selected. In this case, the second selection unit 18 may, for example, calculate an arbitrary line passing through the image of interest in the second latent space and select an image as an auxiliary image that has a second feature whose distance from the line is less than a threshold, similar to the first selection method. In this case, the second selection unit 18 may, for example, calculate a line corresponding to the shape of the cluster to which the image of interest belongs. Alternatively, the second selection unit 18 may select an auxiliary image by any path search starting from the image of interest.

[0071] (Modification 2) In the above embodiment, a limit sample is created based on a set of similar images, but it is not always necessary to create a limit sample. The similar image set creation device 100 according to Modification 2 does not have a limit sample creation unit 15. In this case, the similar image set creation device 100 may terminate the process shown in Figure 8 without creating a limit sample when the similar image set is created (step S5) or displayed (step S6).

[0072] (Variation 3) In the above embodiment, the first model and the second model were generated separately and independently. However, this embodiment is not limited to this. In Modification 3, the first model and the second model may be two models with the same learning method but at different learning stages. Specifically, the deep neural network at the initial stage of the learning process in Non-Patent Literature 1 may be used as the second model, and the deep neural network at the later stage may be used as the first model. As an example, the initial stage in the learning process refers to the stage where the number of iterations in the iterative process of parameter optimization in machine learning is below a threshold, and the later stage refers to the stage where the number of iterations is above the threshold. Of course, the initial stage and the later stage may be distinguished by other criteria.

[0073] (Summary) As described above, the similar image set creation apparatus 100 according to this embodiment includes an image acquisition unit 11, a first extraction unit 12, a second extraction unit 13, and a selection unit 14. The image acquisition unit 11 acquires N images. The first extraction unit 12 extracts N first features from the N images using a first model that performs an image classification task. The second extraction unit 13 extracts N second features from the N images using a second model that performs an image classification task. The second model is trained so that the distance between similar images in the latent space is narrower than that of the first model. The selection unit 14 selects a subject image to be used as the basis for the similar image set and auxiliary images similar to the subject image from among the N images, based on the N first features and the N second features.

[0074] With the above configuration, the selection unit 14 can easily select a target image of interest and a secondary image with similar image features to the target image by utilizing a first feature suitable for clustering, where the distance between similar images in the latent space is small and the distance between dissimilar images is large, and a second feature suitable for generating a continuous distribution of similar images in the latent space, where the distance between similar images in the latent space is small. Specifically, the selection unit 14 selects the target image based on the first feature suitable for clustering. Since similar images are grouped into the same cluster, it is possible to easily select a target image even if the user has no prior knowledge of the images. Furthermore, the selection unit 14 selects a secondary image based on the second feature suitable for generating a continuous distribution of similar images in the latent space. This makes it possible, for example, to automatically select a secondary image that has image features between two target images.

[0075] Thus, it becomes possible to provide a similar image set creation device, method, and program that can reduce the effort required to create a similar image set from a large number of images, including images that the user is interested in and images similar to those images.

[0076] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]

[0077] 1... Processing circuit, 2... Storage device, 3... Input device, 4... Communication device, 5... Display device, 11... Image acquisition unit, 12... First extraction unit, 13... Second extraction unit, 14... Selection unit, 15... Limit sample creation unit, 16... Display control unit, 17... First selection unit, 18... Second selection unit, 100... Similar image set creation device

Claims

1. An acquisition unit that acquires multiple images, A first extraction unit extracts multiple first features from the aforementioned multiple images using a first model that performs an image classification task, A second extraction unit extracts multiple second features from the aforementioned multiple images using a second model that performs an image classification task, wherein the second model is trained so that similar images in the latent space are continuously distributed compared to the first model. The system comprises a selection unit that selects a subject image to serve as the basis for a set of similar images and an auxiliary image similar to the subject image from among the plurality of images based on the plurality of first and plurality of second features, The aforementioned selection unit is A first selection unit selects two or more of the images of interest from the plurality of images based on the plurality of first features, The system includes a second selection unit that selects auxiliary images similar to two or more selected images of interest from the plurality of images based on the plurality of second features, Similar image set creation device.

2. The similar image set creation apparatus according to claim 1, wherein the first selection unit classifies the plurality of images into a plurality of clusters based on the plurality of first features, selects a plurality of representative images that represent each of the plurality of clusters, and selects the representative image selected via an input device from among the plurality of representative images as the image of interest.

3. The similar image set creation apparatus according to claim 2, further comprising a display control unit that selectively displays the plurality of representative images on a display device via the input device.

4. The display control unit displays the selected images of interest in a reorderable manner via the input device, as described in claim 3, for the similar image set creation apparatus.

5. The second selection unit is, In the latent space relating to the aforementioned multiple second features, a line is calculated that passes through the first and second images of the two or more images of interest. From the aforementioned plurality of images, the image whose distance from the line is less than a threshold is selected as the auxiliary image. The apparatus for creating a set of similar images according to claim 1.

6. The similar image set creation apparatus according to claim 1, wherein the second selection unit selects the auxiliary image from the plurality of images by pathfinding between the first image and the second image of the two or more images of interest in a latent space relating to the plurality of second features.

7. The similar image set creation apparatus according to claim 5 or 6, wherein the second selection unit plots the plurality of second feature quantities in the latent space reduced to two or three dimensions.

8. The similar image set creation apparatus according to claim 1, wherein the second selection unit, when three or more focus images are selected as two or more focus images, selects the auxiliary image for each combination of two of the selected three or more focus images.

9. The similar image set creation apparatus according to claim 8, further comprising a display control unit that displays the auxiliary image on a display device for each combination of the two target images.

10. The similar image set creation apparatus according to claim 1, wherein the first model is trained such that the distance between similar images in the latent space is greater than that between similar images in the second model.

11. The similar image set creation apparatus according to claim 1, wherein the first model and the second model are generated by unsupervised representation learning and trained on different loss functions.

12. The similar image set creation apparatus according to claim 1, further comprising a display control unit that displays the aforementioned image of interest and the auxiliary image as the similar image set on a display device.

13. The similar image set creation apparatus according to claim 1, further comprising a creation unit that creates a limit sample based on the aforementioned focus image and the aforementioned auxiliary image.

14. The process of acquiring multiple images, The process involves extracting multiple first features from the aforementioned multiple images using a first model that performs an image classification task, A step of extracting multiple second features from the aforementioned multiple images using a second model that performs an image classification task, wherein the second model is trained so that similar images in the latent space are distributed more continuously than those in the first model. The process includes selecting, based on the plurality of first features and the plurality of second features, a focus image that serves as the basis for a set of similar images and an auxiliary image similar to the focus image from among the plurality of images, The aforementioned selection process is, A first selection step of selecting two or more of the images of interest from the plurality of images based on the plurality of first features, The system includes a second selection step of selecting auxiliary images similar to two or more selected images of interest from the plurality of images based on the plurality of second features. A computer-based method for creating sets of similar images.

15. On the computer, A function to acquire multiple images, A function to extract multiple primary features from the aforementioned multiple images using a first model that performs an image classification task, A function that extracts multiple second features from the aforementioned multiple images using a second model that performs an image classification task, wherein the second model is trained so that similar images in the latent space are distributed more continuously than those in the first model. A similar image set creation program that, based on the plurality of first features and the plurality of second features, enables the selection of a target image to serve as the basis for a similar image set and auxiliary images similar to the target image from among the plurality of images, The aforementioned selection function is A function to select two or more of the aforementioned images based on the aforementioned multiple first features, The system has a function to select auxiliary images similar to two or more selected images of interest from the plurality of images based on the plurality of second features. A program for creating sets of similar images.

Images