Image processing method, program, and image processing device

Abstract

An image processing method according to one embodiment of the present disclosure is executed by a computer, and comprises: an acquisition step (S10) for acquiring an original image in which an object is reflected; a selection step (S20) for selecting, from among a plurality of small images generated by dividing the original image, two or more small images valid for machine learning, on the basis of learning contribution degrees indicating the degree of effect in machine learning for each of the plurality of small images; and an output step (S30) for outputting the two or more small images in a display mode corresponding to the learning contribution degree of each of the two or more small images.

Description

Technical Field

[0001] The present disclosure relates to an image processing method, a program, and an image processing apparatus. Background Art

[0002] Conventionally, there has been a system that uses a learning model to diagnose an analysis object shown in an image. When performing machine learning on the learning model, images are required as learning data for performing machine learning on the learning model.

[0003] Patent Document 1 discloses a program that cuts out a plurality of learning images for learning an identifier from an input image, classifies the learning images into one or more sets, and displays the learning images. The final learning images are determined by the user selecting the displayed learning images.

[0004] Prior Art Documents

[0005] Patent Documents

[0006] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2011-145791 Summary of the Invention

[0007] Problems to be Solved by the Invention

[0008] In machine learning of a learning model such as an identifier, a large number of images for machine learning are required. When there are a large number of images with a high similarity in structure to the images for machine learning, problems such as an increase in learning time and a deterioration in the discriminability of the identifier due to a difference in data distribution from the original distribution may occur. Therefore, it is desired to be able to simply select images that are effective for machine learning, such as having few images for machine learning and being able to improve the performance of the learning model.

[0009] The present disclosure provides an image processing method and the like that can easily select images effective for machine learning.

[0010] Solutions to the Problems

[0011] An image processing method according to one aspect of the present disclosure is executed by a computer, and the image processing method includes the following steps: an acquisition step of acquiring an original image showing an object; a selection step of selecting two or more small images effective for machine learning from the plurality of small images based on the learning contribution degree indicating the degree of effect in machine learning of each of the plurality of small images generated by dividing the original image; and an output step of outputting the two or more small images in a display manner corresponding to the learning contribution degree of each of the two or more small images.

[0012] A program according to one aspect of the present disclosure is a program for causing a computer to execute an image processing method according to one aspect of the present disclosure.

[0013] An image processing apparatus according to one aspect of the present disclosure includes: an acquisition unit that acquires an original image showing an object; a selection unit that selects two or more small images effective for machine learning from the plurality of small images based on the learning contribution degree indicating the degree of effect in machine learning of each of the plurality of small images generated by dividing the original image; and an output unit that outputs the two or more small images in a display manner corresponding to the learning contribution degree of each of the two or more small images.

[0014] Effect of the Invention

[0015] According to the present disclosure, it is possible to provide an image processing method and the like that can easily select an image effective for machine learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment.

[0017] Figure 2 is a diagram for explaining a process of determining a display manner of two or more small images by the image processing apparatus according to an embodiment.

[0018] Figure 3 is a diagram for explaining a normal region and an abnormal region in an original image according to an embodiment.

[0019] Figure 4 is a diagram for explaining a first example of an image output by the image processing apparatus according to an embodiment.

[0020] Figure 5 is a diagram for explaining a second example of an image output by the image processing apparatus according to an embodiment.

[0021] Figure 6 is a diagram for explaining a third example of an image output by the image processing apparatus according to an embodiment.

[0022] Figure 7 is a flowchart showing a processing procedure of the image processing apparatus according to an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In addition, the embodiments described below are used to show a specific example of the present disclosure. Therefore, the numerical values, shapes, materials, constituent elements, arrangement positions of the constituent elements, connection methods, etc. shown in the following embodiments are examples, and are not intended to limit the present disclosure. Therefore, with respect to the constituent elements in the following embodiments that are not recited in the independent claims, they will be described as optional constituent elements.

[0024] In addition, each drawing is a schematic diagram and is not necessarily drawn strictly. Also, in each drawing, the same reference numerals are given to substantially the same structures, and repeated descriptions are omitted or simplified.

[0025] (Embodiment)

[0026] [Structure]

[0027] First, the structure of the image processing apparatus 100 according to the embodiment will be described.

[0028] Figure 1 It is a block diagram showing the structure of the image processing apparatus 100 according to the embodiment.

[0029] The image processing apparatus 100 is a device that displays an image (small image) based on an image (original image) generated by photographing an object (workpiece) with a photographing device such as a camera. Specifically, the image processing apparatus 100 is a learning image automatic selection device that selects, from a plurality of small images generated by dividing the original image, a small image (hereinafter referred to as a learning image) for causing a learning model to perform machine learning (AI (Artificial Intelligence) learning), where the learning model is used to determine whether the object reflected in the original image contains a defect.

[0030] In machine learning, for example, various learning images obtained by photographing an object and information (annotation information) indicating the presence or absence of a defect or normality with respect to each learning image are used to cause the learning model to perform machine learning.

[0031] Here, there are images among the learning images that can cause the learning model to perform machine learning effectively, that is, can improve the performance of the learning model with a smaller number of images. On the other hand, depending on the learning images, there are also images that cannot cause the learning model to perform machine learning effectively. In particular, there is the following problem: there are a large number of candidates in the selection of small images in the normal area that does not contain defects, but it is not clear which candidate is effective for machine learning.

[0032] Therefore, the image processing apparatus 100 outputs learning images that enable the learning model to learn effectively in a manner that is easy for the user to understand.

[0033] In addition, the performance mentioned here is, for example, the accuracy rate that can correctly extract defects when the original image is input into the learning model after machine learning, or can correctly determine the situation of no defects.

[0034] The image processing apparatus 100 is a computer such as a personal computer or a tablet terminal. Specifically, for example, the image processing apparatus 100 is implemented by a communication interface for communicating with the display device 200 and the input device 210, a non-volatile memory storing a program, a volatile memory which is a transient storage area for executing the program, an input / output port for transmitting and receiving signals, a processor for executing the program, etc. The communication interface can be implemented by a connector connected with a communication line in a manner capable of wired communication, or can be implemented by an antenna and a wireless communication line in a manner capable of wireless communication.

[0035] The image processing apparatus 100 includes an information processing unit 110 and a storage unit 120.

[0036] The information processing unit 110 is a processing unit that performs various processes executed by the image processing apparatus 100. For example, the information processing unit 110 outputs a plurality of small images obtained by performing image processing on the acquired original image to the display device 200, thereby displaying a plurality of small images in the display image.

[0037] Figure 2 It is a diagram for explaining the process in which the image processing apparatus 100 according to the embodiment determines the display mode of two or more small images.

[0038] For example, the information processing unit 110 acquires Figure 2 the original image showing the object as shown in (a) of, and generates a plurality of small images as shown in (b) of Figure 2 by dividing the acquired original image. In the example shown in (b) of Figure 2 , the information processing unit 110 generates 14×9 = 126 small images from the original image. And, as shown in (c) of Figure 2 , the information processing unit 110 outputs two or more small images effective for machine learning selected based on the learning contribution degree indicating the effect of each of the plurality of small images in machine learning in a display mode corresponding to the learning contribution degree of the two or more small images. In the example shown in (c) of Figure 2 , for small images with a learning contribution degree equal to or higher than a specified learning contribution degree, the display mode is changed so that their outer edges become thick lines, dotted lines, or dashed lines, different from small images with a learning contribution degree less than the specified learning contribution degree.

[0039] In addition, being effective for machine learning, for example, means being equal to or higher than a specified learning contribution degree. The specified learning contribution degree can be arbitrarily determined.

[0040] The learning contribution degree of each of the plurality of small images is determined based on, for example, the similarity between the plurality of small images. For example, the similarity is calculated based on the average value of differences such as the luminance difference or color difference of each pixel at the same position of two small images. For example, the larger the average value, the lower the calculated similarity. The learning contribution degree is determined based on the calculated similarity. For example, the learning contribution degree is set such that the lower the similarity, the higher the learning contribution degree.

[0041] For example, the information processing unit 110 is implemented by one or more processors.

[0042] The information processing unit 110 includes an acquisition unit 111, a selection unit 112, an output unit 113, a reception unit 114, and a storage unit 120.

[0043] The acquisition unit 111 is a processing unit that acquires a raw image that reflects an object. Specifically, the acquisition unit 111 acquires a raw image that reflects a first object.

[0044] The object is an object to be inspected by a learning model. The acquisition unit 111 acquires, for example, Figure 2 a raw image that reflects the object as shown in (a) of the figure.

[0045] The object is, for example, an industrial product. In the present embodiment, the object is an electronic component such as an IC (Integrated Circuit).

[0046] In addition, the object may be an arbitrary object such as a substrate instead of an electronic component.

[0047] The imaging device is a camera that generates a raw image by imaging the object. The imaging device is implemented by, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0048] In addition, the acquisition unit 111 may acquire a raw image from a server device or the like via a communication interface included in the image processing device 100.

[0049] The selection unit 112 is a processing unit that selects two or more small images effective for machine learning from the plurality of small images based on the learning contribution degree of each of the plurality of small images generated by dividing the raw image, which indicates the degree of effect in machine learning.

[0050] First, the selection unit 112 generates a plurality of small images by dividing the original image. The division method of the original image can be arbitrarily determined. For example, the number of the plurality of small images can be arbitrarily determined. In addition, the plurality of small images can be rectangles, or can be any shapes such as triangles or circles. In addition, the sizes and shapes of the plurality of small images can be the same as each other or different from each other.

[0051] Next, the selection unit 112 selects any one image from the plurality of small images. The image to be selected here can be arbitrarily determined. In Figure 2 the example shown in (c) of Figure 2 the leftmost upper small image among the plurality of small images is first selected from the plurality of small images.

[0052] Next, the selection unit 112 calculates the similarity between the selected small image and the plurality of unselected small images. And, the selection unit 112 selects the image with the lowest similarity among the plurality of unselected small images.

[0053] The selection unit 112 selects two or more small images effective for machine learning by repeating the processes of selecting such small images and calculating the similarity (also referred to as the selection process) a predetermined number of times. That is, the selection unit 112 selects two or more small images by repeatedly executing the following process: based on the similarity between the plurality of small images other than all the selected small images and all the selected small images, one small image is selected from the plurality of small images other than all the selected small images. Through the above process, for example, the selection unit 112 selects two or more small images effective for machine learning based on the learning contribution degree (more specifically, the similarity) indicating the degree of effect in machine learning of each of the plurality of small images.

[0054] In addition, the predetermined number of times can be arbitrarily determined. For example, the predetermined number of times is determined based on a threshold value. For example, the selection unit 112 selects two or more small images based on the similarity between the plurality of small images and the threshold value of the similarity. For example, when the threshold value is 0.2, the selection unit 112 repeats the selection process until there are no longer any small images with a calculated similarity of 0.2 or less.

[0055] For example, the predetermined number of times can also be arbitrarily determined by the user. For example, the reception unit 114 can also receive information indicating the predetermined number of times or information indicating the threshold value from the user via the input device 210.

[0056] For example, the larger the threshold value, the larger the number of two or more small images. In other words, the larger the threshold value, the more small images the selection unit 112 selects.

[0057] In addition, the threshold value can be one or more. For example, the threshold values include a first threshold value and a second threshold value, and the value of the second threshold value is larger than that of the first threshold value. The selection unit 112 selects two or more small images from the multiple small images, including a first image with a similarity less than the first threshold value and a second image with a similarity equal to or greater than the first threshold value and less than the second threshold value.

[0058] In addition, for example, the two or more small images selected by the selection unit 112 are respectively images of normal regions in the original image that do not contain defects of the object.

[0059] Figure 3 It is a diagram for explaining the normal regions and abnormal regions in the original image related to the embodiment. Specifically, Figure 3 It is a diagram showing multiple small images obtained by segmenting the original image.

[0060] The normal region is a region in the original image that does not have defects such as scratches, defects, stains, or dust attachments. In Figure 3 the example shown, the small images included in the "normal region" are the small images among the multiple small images other than the four small images surrounded by thick lines. On the other hand, the abnormal region is a region in the original image that has such defects. In Figure 3 the example shown, the small images included in the "abnormal region" are the four small images surrounded by thick lines among the multiple small images.

[0061] When the selection unit 112 selects two or more small images, for example, it does not select the small images of the abnormal regions containing defects, but selects two or more small images from the small images of the normal regions that do not contain defects.

[0062] For example, the original image obtained by the acquisition unit 111 is output to the display device 200 through the output unit 113 and thus displayed on the display device 200. The user inputs the position of the defect in the original image by operating the input device 210. The reception unit 114 receives this input. The selection unit 112 selects two or more small images from the small images of the normal regions that do not contain defects based on this input received by the reception unit 114. At this time, for example, the selection unit 112 can attach information indicating normal (for example, no defects) or information indicating abnormal (for example, there are defects), that is, annotation information, to the multiple small images based on this input, and store them in the storage unit 120.

[0063] The output unit 113 is a processing unit that outputs two or more small images selected by the selection unit 112 in a display manner corresponding to the respective learning contribution degrees of the two or more small images. Specifically, the output unit 113 changes the two or more small images selected by the selection unit 112 to a display manner corresponding to the respective learning contribution degrees of the two or more small images, and outputs the image information including the two or more small images with the changed display manner to the display device 200, so as to display the two or more small images with the changed display manner on the display device 200.

[0064] In addition, the so-called output of two or more small images means that it is sufficient to output an image including two or more small images, which may be to output a plurality of small images generated by dividing the original image and including two or more small images, or to change the display manner of the parts corresponding to the two or more small images in the original image and output them in the changed display manner.

[0065] In addition, the display manner can be arbitrarily determined. For example, the output unit 113 outputs a plurality of small images with different decorations added around or inside each of the two or more small images based on the respective learning contribution degrees of the two or more small images.

[0066] Here, adding decorations includes, for example, adding a frame around each of the two or more small images. For example, the output unit 113 determines at least one display manner among the thickness, color, and shape of the frame based on the respective learning contribution degrees of the two or more small images. The shape of the frame refers to the type of line, which is the shape of a line such as a solid line, a dotted line, a dashed line, and a dot-dash line. For example, the output unit 113 adds a frame to the two or more small images in such a way that the higher the learning contribution degree, the thicker the frame, and the lower the learning contribution degree, the thinner the frame.

[0067] Figure 4 It is a diagram for explaining the first example of the image output by the image processing device 100 according to the embodiment. Specifically, it is a diagram showing an example of the image information output to the output unit 113 and displayed on the display device 200.

[0068] As Figure 4 shown, for example, on the display device 200, the part of the original image corresponding to the small images with a specified learning contribution degree or more is surrounded by any one of a solid line, a dashed line, and a dot-dash line. For example, it is assumed that the selection unit 112 selects a first image with a similarity less than the first threshold, a second image with a similarity of the first threshold or more and less than the second threshold, and a third image with a similarity of the second threshold or more and less than the third threshold from among a plurality of small images. In this case, for example, the output unit 113 changes the display manner of the two or more small images to: surround the part of the original image corresponding to the first image ( Figure 4The part corresponding to the "small image with the highest learning contribution" shown in the figure) is surrounded by a dotted line in the original image corresponding to the second image ( Figure 4 The part corresponding to the "small image with a learning contribution inferior to that of the solid quadrilateral" shown in the figure) is surrounded by a dashed-dotted line in the original image corresponding to the third image ( Figure 4 The part corresponding to the "small image with a learning contribution inferior to that of the dotted rectangle" shown in the figure). In this way, for example, the output unit 113 outputs the first image and the second image in different display manners. In this example, the output unit 113 outputs the original image showing the first image and the second image in different display manners. For example, the output unit 113 outputs information indicating that the first image is a small image with a higher learning contribution than the second image. This information is, for example, information such as Figure 4 Shown as the "small image with the highest learning contribution" and showing an explanation related to the learning contributions (i.e., similarity) of two or more small images.

[0069] In addition, for example, the output unit 113 outputs information related to two or more small images in descending order of learning contribution. In Figure 4 In the example shown, the output unit 113 outputs the image information in the following manner: starting from the upper part of the image displayed on the display device 200, explanations related to two or more small images (such as "the small image with the highest learning contribution") are arranged in descending order of learning contribution.

[0070] Furthermore, the so-called outputting information related to two or more small images in descending order of learning contribution may, for example, also include Figure 4 Displaying the solid lines, dotted lines, and dashed-dotted lines surrounding two or more images in this order and changing them sequentially over time. For example, it may be that only the solid line among the solid line, dotted line, and dashed-dotted line is displayed, only the dotted line is displayed after a specified time, only the dashed-dotted line is displayed after another specified time, and these displays are repeated and changed. In this way, the information related to two or more small images may also include information for explaining two or more small images and the display manners of two or more small images such as frame lines. In addition, regarding the descending order, it can be a spatial order such as starting from the upper part or a temporal order.

[0071] In addition, the so-called adding decorations to two or more small images includes, for example, performing at least one of hue correction, chroma correction, and lightness correction on each of the two or more small images. For example, the output unit 113 performs correction to attract the user's attention by performing correction such as making two or more small images approach expansion colors such as warm colors, or performing chroma-increasing correction, or performing lightness-increasing correction. Of course, the output unit 113 can also change the display mode by adding a frame around each of the two or more small images and performing image correction such as hue correction. In addition, for example, it is also possible to change the display mode such as hue correction for small images not selected by the selection unit 112 among the multiple small images (that is, small images other than the two or more small images). For example, it is also possible to perform correction such as reducing the lightness for small images not selected by the selection unit 112 among the multiple small images so that they are difficult to observe.

[0072] The reception unit 114 is a processing unit that receives the user's operations. The reception unit 114 receives the user's operations via the input device 210, for example. The reception unit 114 receives the input of position information indicating the position of an abnormal area (or defect) included in the original image, for example. The user observes the original image or small images displayed on the display device 200 and uses the input device 210 to input the position of the abnormal area included in the original image or the small images, or the small image including the defect. The reception unit 114 receives this input as position information, for example.

[0073] In addition, it may be that the reception unit 114 receives a first instruction indicating a first threshold or a second threshold, and the output unit 113 determines the display mode of two or more small images based on the first instruction received by the reception unit 114 and outputs the two or more small images in the determined display mode. That is, it is also possible to change the display mode of two or more images in the image information displayed on the display device 200 based on the first instruction.

[0074] Figure 5 FIG. is a second example for explaining the image output by the image processing apparatus 100 according to the embodiment. Figure 6 FIG. is a third example for explaining the image output by the image processing apparatus 100 according to the embodiment. In Figure 5 and Figure 6 In the example shown, similar to the first example shown in Figure 4 it is assumed that the selection unit 112 selects a first image with a similarity less than the first threshold, a second image with a similarity equal to or greater than the first threshold and less than the second threshold, and a third image with a similarity equal to or greater than the second threshold and less than the third threshold from among the multiple small images. In addition, in this example, the first threshold is threshold = 0.2, the second threshold is threshold = 0.4, and the third threshold is threshold = 0.6.

[0075] In the second example, first, the output unit 113 outputs image information in which a frame line is added to the small image corresponding to the first image. Thus, as Figure 5 shown, the original image in which a frame line is added to the portion corresponding to the first image, which is the small image selected by the selection unit 112 under the condition that the threshold is 0.2, is displayed on the display device 200.

[0076] Next, for example, it is assumed that the reception unit 114 receives a selection with a threshold of 0.4 as the first instruction. In this case, the output unit 113 outputs image information in which a frame line is added to the small images corresponding to the first image and the second image. Thus, as Figure 6 shown, the original image in which a solid frame line is added to the portion corresponding to the first image, which is the small image selected by the selection unit 112 under the condition that the threshold is 0.2, and a dashed frame line is added to the portion corresponding to the second image, which is the small image not selected by the selection unit 112 under the condition that the threshold is 0.2 but selected by the selection unit 112 under the condition that the threshold is 0.4, is displayed on the display device 200.

[0077] For example, the user selects the small images to be used in machine learning from two or more small images by selecting the threshold in this way. For example, when the user selects a threshold of 0.2, the first image is determined as the learning image to be used in machine learning. In addition, for example, when the user selects a threshold of 0.4, the first image and the second image are determined as the learning images to be used in machine learning. For example, when the reception unit 114 receives the first instruction, the output unit 113 determines the learning images from two or more small images based on the first instruction, and stores the information indicating that they are learning images in the storage unit 120. For example, when the reception unit 114 receives an instruction to perform machine learning, the output unit 113 selects the learning images based on this information, and inputs the selected learning images into the learning model to cause the learning model to perform machine learning.

[0078] In addition, the determination of the learning images from multiple small images can be performed arbitrarily.

[0079] For example, the reception unit 114 receives a second instruction, which indicates small images among two or more small images sorted based on the learning contribution degree, from the small image with the highest learning contribution degree to the nth small image. The learning image can be determined in this way. The selection unit 112, for example, can also calculate the similarity for all small images by repeatedly executing the above selection process for all small images, and calculate the learning contribution degree based on the calculated similarity. In addition, when calculating the learning contribution degree for all small images, the selection unit 112 can also select two or more small images for which the display method is to be changed after calculating the learning contribution degree of all small images.

[0080] In addition, for example, when the output unit 113 adds a frame line to two or more small images in such a way that the thicker the frame line, the higher the learning contribution degree, and the thinner the frame line, the lower the learning contribution degree, the reception unit 114 can receive a third instruction indicating the thickness of the frame line, and determine, as the images to be used in machine learning, the small images among the two or more small images that are decorated with a line thicker than the frame line indicated by the third instruction.

[0081] In addition, the acquisition unit 111, the selection unit 112, the output unit 113, and the reception unit 114 can be implemented by a shared processor, or can each be implemented by an independent processor.

[0082] The storage unit 120 is a storage device that stores programs executed by processing units such as the acquisition unit 111, the selection unit 112, the output unit 113, and the reception unit 114 for performing various processes, information required for the processes, and inspection images. The storage unit 120 is implemented by, for example, an HDD (Hard Disk Drive) and / or a semiconductor memory.

[0083] The display device 200 is a display that displays an image based on the control of the image processing device 100 (more specifically, the output unit 113). The display device 200 displays, for example, a plurality of small images (that is, the original image) including two or more small images. The display device 200 is implemented by a display device such as a liquid crystal panel or an organic EL (Electro Luminescence) panel.

[0084] The input device 210 is a user interface that receives the operations of the user. The input device 210 is implemented by a mouse, a keyboard, a touch panel, and / or hardware buttons.

[0085] In addition, the display device 200 and the input device 210 can also be integrally implemented by a touch panel display.

[0086] [Processing procedure]

[0087] Next, the processing procedure of the image processing apparatus 100 according to the embodiment will be described.

[0088] Figure 7 FIG. is a flowchart showing the processing procedure of the image processing apparatus 100 according to the embodiment.

[0089] First, the acquisition unit 111 acquires a raw image in which an object is reflected (S10). For example, the acquisition unit 111 acquires a raw image from a camera (not shown) via a communication interface or the like provided in the image processing apparatus 100.

[0090] In addition, the raw image can be stored in the storage unit 120, for example. In this case, the acquisition unit 111 acquires the raw image from the storage unit 120, for example.

[0091] Next, the selection unit 112 selects two or more small images effective for machine learning from the plurality of small images based on the learning contribution degree indicating the degree of the effect in machine learning of each of the plurality of small images generated by dividing the raw image (S20). Specifically, the selection unit 112 generates a plurality of small images by dividing the raw image acquired by the acquisition unit 111. Next, the selection unit 112 selects any one image from the plurality of small images. In the above example, for example, first, the small image located at the upper leftmost shown in (c) of Figure 2 is selected. Next, the selection unit 112 calculates the similarity between the selected small image and the plurality of unselected small images. The selection unit 112 calculates the similarity between all the selected small images and the unselected small images. The selection unit 112 selects two or more small images effective for machine learning by repeating such processing a predetermined number of times. The predetermined number of times can be arbitrarily determined. For example, in the above example, the predetermined number of times is determined based on a threshold value.

[0092] In addition, the similarity of each small image can be calculated based on the average value of the similarity with each small image or the like.

[0093] Next, the output unit 113 outputs the two or more small images selected by the selection unit 112 in a display manner corresponding to the learning contribution degree of each of the two or more small images (S30). Specifically, the output unit 113 causes the two or more small images selected by the selection unit 112 to be displayed on the display device 200 in a display manner corresponding to the learning contribution degree of each of the two or more small images.

[0094] In addition, the output unit 113 can cause the learning model to perform machine learning by outputting the two or more small images selected by the selection unit 112 to the learning model.

[0095] [Effects, etc.]

[0096] Next, the technology obtained based on the disclosure of this specification is illustrated, and the effects and the like obtained based on the illustrated technology are described.

[0097] Technology 1 is an image processing method executed by a computer. The image processing method includes the following steps: an acquisition step (S10) of acquiring an original image in which an object is reflected; a selection step (S20) of selecting two or more small images effective for machine learning from a plurality of small images based on the learning contribution degree indicating the degree of effect in machine learning of each of the plurality of small images generated by dividing the original image; and an output step (S30) of outputting the two or more small images in a display manner corresponding to the learning contribution degree of each of the two or more small images.

[0098] In machine learning that takes an image as input, a large number of images are required as learning data. Here, for example, a plurality of images with similar features of the object reflected in the image, such as the same shape and configuration, have a lower effect on machine learning than a plurality of images with dissimilar features. Therefore, by using a plurality of images with dissimilar features in machine learning, machine learning can be effectively executed. For example, appropriate output can be obtained even with a small number of images for machine learning. Therefore, in the image processing method according to one aspect of the present disclosure, two or more small images effective for machine learning are output from a plurality of small images in a display manner corresponding to the learning contribution degree indicating the degree of effect in machine learning of each of the plurality of small images generated by dividing the original image. Thereby, the small images can be set in a display manner corresponding to the learning contribution degree, so that the user can easily select the images effective for machine learning.

[0099] Technology 2 is the image processing method described in Technology 1, wherein the learning contribution degree of each of the plurality of small images is determined based on the similarity between the plurality of small images, and in the selection step, two or more small images are selected based on the similarity between the plurality of small images and a threshold value of the similarity.

[0100] In this way, the image processing method according to one aspect of the present disclosure is an automatic selection method of learning images using similarity, and selects small images effective for learning based on the similarity between small images.

[0101] Thereby, small images that are dissimilar to each other, that is, have a low similarity, can be automatically selected from a plurality of candidates (that is, a plurality of small images). Therefore, two or more small images effective for machine learning can be appropriately selected. By using two or more small images selected in this way in machine learning, the discrimination performance of the learning model can be improved with a smaller number of small images.

[0102] Technique 3 is an image processing method according to the image processing method described in Technique 2, wherein two or more small images are respectively images of normal regions in the original image that do not contain defects of the object.

[0103] There are many images of normal regions, and it is not clear which image of the normal region is effective for machine learning. It is necessary to try and error to determine which image is effective for machine learning. On the other hand, compared with the images of normal regions, the images of abnormal regions are local and the feature amounts of the images are clear. Therefore, it is not necessary to have so much trial and error to determine which image is effective for machine learning. Therefore, the image processing method according to one aspect of the present disclosure is particularly effective for images of normal regions.

[0104] Technique 4 is an image processing method according to the image processing method described in Technique 2 or 3, wherein the larger the threshold value, the more the number of two or more small images.

[0105] That is to say, when the threshold value becomes larger, the number of two or more small images displayed in the display device 200 increases.

[0106] As described above, it is considered that the lower the similarity between images, the higher the effect on machine learning, that is, the image with a higher learning contribution degree. Therefore, the larger the threshold value is set, the more the number of two or more small images selected. Therefore, for example, in the case where the user has a requirement to select an image to be used in machine learning from a large number of images, by setting the threshold value high, the image effective for machine learning can be simply changed to a display mode that is easy for the user to understand.

[0107] Technique 5 is an image processing method according to any one of Techniques 2 to 4, wherein in the selection step, two or more small images are selected by repeatedly executing the following process: based on the similarity between a plurality of small images other than all the selected small images and all the selected small images, one small image is selected from the plurality of small images other than all the selected small images.

[0108] That is to say, the similarity is calculated between the selected small images and the unselected small images, and the next small image effective for learning is selected.

[0109] Thereby, the images with low similarity to the selected images are repeatedly selected. Therefore, even if the similarities between all the small images are not all calculated, the images with low similarity to each other can be simply selected.

[0110] Technique 6 is an image processing method described in any one of Techniques 2 to 5. Among them, the threshold includes a first threshold and a second threshold, and the value of the second threshold is larger than the value of the first threshold. In the selection step, two or more small images are selected from a plurality of small images, including a first image with a similarity less than the first threshold and a second image with a similarity equal to or greater than the first threshold and less than the second threshold. In the output step, the first image and the second image are output in different display manners.

[0111] Thereby, images with similar similarities can be simply classified from each other according to each threshold.

[0112] Technique 7 is an image processing method described in Technique 6, and further includes an acceptance step. In this acceptance step, a first instruction indicating the first threshold or the second threshold is accepted. In the output step, based on the first instruction accepted in the acceptance step, the display manner of two or more small images is determined, and the two or more small images are output in the determined display manner.

[0113] For example, in the output step (first output step), first, as Figure 4 shown, the combination of the threshold and the frame line corresponding to the threshold is output (displayed). Next, in the acceptance step, the selection of the threshold desired by the user is accepted from the user. Next, in the output step (second output step), the display manner (for example, the frame line) of the small image is changed based on the selection (threshold) accepted in the acceptance step. For example, when the image shown in Figure 4 is output in the first output step and the selection of the threshold = 0.4 is accepted in the acceptance step, the image shown in Figure 5 is output in the second output step. Thereby, for example, small images of the learning contribution degree that the user wants to confirm can be simply output. For example, small images less than the threshold selected in this way are used in the machine learning of the learning model. In the selection of images to be used in machine learning by a computer according to a threshold, an objective evaluation image similarity value is used, and sometimes the similarity is different from that in the case of human observation. In this way, for example, finally, the images to be used in machine learning are selected based on the threshold selected by the user, thereby being able to make up for such judgment differences between the computer and humans.

[0114] Technique 8 is an image processing method described in any one of Techniques 2 to 7. Among them, the threshold includes a first threshold and a second threshold, and the value of the second threshold is larger than the value of the first threshold. In the selection step, two or more small images are selected from a plurality of small images, including a first image with a similarity less than the first threshold and a second image with a similarity equal to or greater than the first threshold and less than the second threshold. In the output step, information indicating that the learning contribution degree of the first image is higher than that of the second image is output.

[0115] That is, the small images selected when the similarity threshold is small are displayed on the display device 200 as small images with high learning contribution.

[0116] Thus, the images selected based on a relatively small threshold represent images that are not similar to each other. Therefore, the user can easily select small images with the same feature amount as the images to be used in machine learning, that is, the images with high learning contribution. In other words, the smaller the threshold, the less similar the selected small images are to each other. For the user, it is easy to select multiple small images with the same label (such as a specified feature like brightness) and various different feature amounts for that label as learning images.

[0117] Technique 9 is an image processing method according to any one of Techniques 1 to 8, further including an acceptance step in which a second instruction is accepted, where the second instruction indicates using, among two or more small images sorted based on the level of learning contribution, the small images from the small image with the highest learning contribution to the nth small image.

[0118] Thus, the user can easily select the images to be used in machine learning.

[0119] Technique 10 is an image processing method according to any one of Techniques 1 to 9, where in the output step, multiple small images with different decorations added around or inside each of the two or more small images based on the learning contribution of each of the two or more small images are output.

[0120] Thus, it is possible to simply notify the user of the learning contribution of each small image through decoration.

[0121] Technique 11 is an image processing method according to Technique 10, where adding decoration includes adding a frame around each of the two or more small images, and in the output step, at least one display method of the thickness, color, and shape of the frame is determined based on the learning contribution of each of the two or more small images.

[0122] Technique 12 is an image processing method according to Technique 10 or 11, where adding decoration includes performing at least one of hue correction, chroma correction, and lightness correction on each small image of the two or more small images.

[0123] Thus, the user can simply learn the learning contribution of each small image just by observing each small image that shows the learning contribution of each small image to the user through decoration.

[0124] Technique 13 is an image processing method described in accordance with Technique 11. In the output step, frame lines are added to two or more small images in such a way that the higher the learning contribution degree, the thicker the frame line, and the lower the learning contribution degree, the thinner the frame line. The image processing method further includes an acceptance step, in which, by accepting a third indication representing the thickness of the frame line, a small image decorated with a line thicker than the frame line represented by the third indication among the two or more small images is determined as an image to be used in machine learning.

[0125] Thereby, the user can simply know the learning contribution degree of each small image by merely observing each small image that shows the learning contribution degree of each small image to the user through decoration, and can simply select a small image to be used in machine learning.

[0126] Technique 14 is an image processing method described in accordance with any one of Techniques 1 to 13, wherein the object is an industrial product.

[0127] Machine learning using images is used for various purposes such as inspection of industrial products such as components of electronic devices, and human recognition. Regarding industrial products, for example, the same products are mechanically produced. Therefore, different from humans, if they are the same industrial product, there are many images with high similarity even if they show different objects. In addition, in order to make manufacturing easier, unnecessary processing is rarely performed, and sometimes there are many parts with high similarity even within one image. Therefore, the image processing method according to one aspect of the present disclosure is particularly effective when processing images that are likely to contain images with high similarity such as industrial products.

[0128] Technique 15 is an image processing method described in accordance with any one of Techniques 1 to 14, wherein, in the output step, information related to two or more small images is output in descending order of learning contribution degree.

[0129] Thereby, the user can easily select an image effective for machine learning.

[0130] Technique 16 is a program for causing a computer to execute the image processing method described in accordance with any one of Techniques 1 to 15.

[0131] Thereby, the same effect as the image processing method according to one aspect of the present disclosure is achieved.

[0132] Technique 17 is an image processing apparatus 100, comprising: an acquisition unit 111 that acquires a raw image showing an object; a selection unit 112 that selects two or more small images effective for machine learning from among a plurality of small images based on the learning contribution degree indicating the degree of effect in machine learning for each of the plurality of small images generated by dividing the raw image; and an output unit 113 that outputs the two or more small images in a display manner corresponding to the learning contribution degree of each of the two or more small images.

[0133] Thereby, the same effect as that of the image processing method according to one aspect of the present disclosure is achieved.

[0134] In addition, these inclusive or specific aspects can be implemented by a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a CD-ROM readable by a computer, or can be implemented by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0135] (Other embodiments, etc.)

[0136] The embodiments have been described above, but the present disclosure is not limited to the above embodiments.

[0137] In addition, in the above embodiment, the image processing apparatus 100 is implemented as a single apparatus, but it can also be implemented by a plurality of apparatuses. In the case where the image processing apparatus is implemented by a plurality of apparatuses, the components included in the image processing apparatus described in the above embodiment can be arbitrarily allocated to the plurality of apparatuses.

[0138] In addition, in the above embodiment, the processing performed by a specific processing unit can also be performed by other processing units. In addition, the order of a plurality of processes can be changed, and a plurality of processes can be executed in parallel.

[0139] In addition, in the above embodiment, each component (each processing unit) can also be implemented by executing a software program suitable for each component. Each component can be implemented by a program execution unit such as a CPU (Central Processing Unit) or a processor reading a software program recorded in a recording medium such as a hard disk or a semiconductor memory and executing the software program.

[0140] In addition, each component can also be implemented by hardware. Each component can be a circuit (or an integrated circuit). These circuits can form a single circuit as a whole, or can be independent circuits respectively. In addition, these circuits can be general-purpose circuits respectively, or can be dedicated circuits respectively.

[0141] In addition, the general or specific aspects of the present disclosure can be implemented by a system, apparatus, method, integrated circuit, computer program, or non-transitory recording medium such as a computer-readable CD-ROM. In addition, it can also be implemented by any combination of a system, apparatus, method, integrated circuit, computer program, and recording medium.

[0142] For example, the present disclosure can also be implemented as an image processing method executed by a computer such as an image processing apparatus. In addition, the present disclosure can also be implemented as a program for causing a computer to execute the image processing method, and can also be implemented as a non-transitory recording medium readable by a computer on which such a program is recorded.

[0143] Furthermore, modes obtained by applying various modifications conceived by those skilled in the art to each embodiment, or modes implemented by arbitrarily combining the constituent elements and functions in each embodiment without departing from the gist of the present disclosure are also included in the present disclosure.

[0144] Industrial Applicability

[0145] The present disclosure is useful as an image processing apparatus for presenting an image to a user.

[0146] Description of Reference Numerals

[0147] 100: Image processing apparatus; 110: Information processing unit; 111: Acquisition unit; 112: Selection unit; 113: Output unit; 114: Reception unit; 120: Storage unit; 200: Display device; 210: Input device.

Claims

1. An image processing method, executed by a computer, the image processing method comprises the following steps: An acquisition step of acquiring an original image showing an object; A selection step of selecting two or more small images effective for machine learning from the plurality of small images based on the learning contribution degree indicating the degree of effect in machine learning of each of the plurality of small images generated by segmenting the original image; And An output step of outputting the two or more small images in a display manner corresponding to the learning contribution degree of each of the two or more small images.

2. The image processing method according to claim 1, wherein, the learning contribution degree of each of the plurality of small images is determined based on the similarity between the plurality of small images, in the selection step, the two or more small images are selected based on the similarity between the plurality of small images and a threshold value of the similarity.

3. The image processing method according to claim 2, wherein, each of the two or more small images is an image of a normal area in the original image that does not contain a defect of the object.

4. The image processing method according to claim 2, wherein, the larger the threshold value is, the more the number of the two or more small images is.

5. The image processing method according to claim 2, wherein, in the selection step, the two or more small images are selected by repeatedly executing the following process: based on the similarity between the plurality of small images other than all the selected small images and all the selected small images, one small image is selected from the plurality of small images other than all the selected small images.

6. The image processing method according to claim 2, wherein, the threshold value includes a first threshold value and a second threshold value, and the value of the second threshold value is larger than the value of the first threshold value, in the selection step, the two or more small images including a first image with a similarity less than the first threshold value and a second image with a similarity of not less than the first threshold value and less than the second threshold value are selected from the plurality of small images, in the output step, the first image and the second image are output in different display manners.

7. The image processing method according to claim 6, wherein, it further comprises an acceptance step, in which a first instruction indicating the first threshold value or the second threshold value is accepted, in the output step, based on the first instruction accepted in the acceptance step, the display manner of the two or more small images is determined, and the two or more small images are output in the determined display manner.

8. The image processing method according to claim 2, wherein, the threshold value includes a first threshold value and a second threshold value, and the value of the second threshold value is larger than the value of the first threshold value, in the selection step, the two or more small images including a first image with a similarity less than the first threshold value and a second image with a similarity of not less than the first threshold value and less than the second threshold value are selected from the plurality of small images, In the output step, information indicating that the first image is a small image with a higher learning contribution degree than that of the second image is output.

9. The image processing method according to claim 1, wherein, it further includes an acceptance step, in which a second instruction is accepted, and the second instruction indicates small images from the small image with the highest learning contribution degree to the nth small image among the two or more small images sorted based on the level of learning contribution degree for use in machine learning.

10. The image processing method according to claim 1, wherein, in the output step, the plurality of small images with different decorations added around or inside each of the two or more small images based on the learning contribution degree of each of the two or more small images are output.

11. The image processing method according to claim 10, wherein, adding the decoration includes adding a frame around each of the two or more small images, in the output step, at least one display method of the thickness of the frame, the color of the frame, and the shape of the frame is determined based on the learning contribution degree of each of the two or more small images.

12. The image processing method according to claim 10, wherein, adding the decoration includes performing at least one of hue correction, chroma correction, and lightness correction on each small image of the two or more small images.

13. The image processing method according to claim 11, wherein, in the output step, the two or more small images are added with the frame in such a way that the higher the learning contribution degree, the thicker the frame, and the lower the learning contribution degree, the thinner the frame, the image processing method further includes an acceptance step, in which by accepting a third instruction indicating the thickness of the frame, the small images among the two or more small images decorated with a line thicker than the frame indicated by the third instruction are determined as images to be used in machine learning.

14. The image processing method according to claim 1, wherein, the object is an industrial product.

15. The image processing method according to claim 1, wherein, in the output step, information related to the two or more small images is output in the order of decreasing learning contribution degree.

16. A program for causing a computer to execute the image processing method according to any one of claims 1 to 15.

17. An image processing apparatus, comprising: an acquisition unit that acquires an original image showing an object; a selection unit that selects two or more small images effective for machine learning from the plurality of small images based on the learning contribution degree indicating the degree of effect in machine learning of each of the plurality of small images generated by segmenting the original image; and an output unit that outputs the two or more small images in a display manner corresponding to the learning contribution degree of each of the two or more small images.

Images