IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, IMAGE PROCESSING PROGRAM, IMAGE CLASSIFICATION DEVICE, AND TRAINED MODEL GENERATED THEREBY
The image processing device enhances machine learning accuracy by converting images into multiple color information, applying Fourier transform, and using diverse lighting to create robust training data, addressing the challenge of small data sets in varied object recognition.
Patent Information
- Application Number
- JP2022075252
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-04-28
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2042-04-28
AI Technical Summary
Existing image processing methods for machine learning face challenges in achieving high determination accuracy with a small amount of training data, particularly when dealing with diverse objects in images, such as mechanical parts or human bodies, where different parts can have varying features.
An image processing device that numerically converts images into multiple color information for each pixel, combines these, and applies fast Fourier transform to create data for machine learning, utilizing multiple light sources to illuminate objects from different directions for uniform imaging.
This approach enables stable determination accuracy by extracting features independent of positional or shape changes in the photographed subject, allowing effective machine learning with reduced data requirements.
Smart Images

Figure 0007583395000001 
Figure 0007583395000002 
Figure 0007583395000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an image processing device, an image processing method, and an image processing program for creating data used for machine learning based on information obtained from captured images, as well as to an image classification device that performs machine learning using data created by such an image processing device, and a trained model generated by the image classification device. [Background technology]
[0002] Conventionally, there are known methods for detecting anomalies from images captured by a camera, etc. For example, a method is known in which images of normal parts without scratches and images of abnormal parts with scratches are used as training data to train a deep learning model that can determine whether a part in an image is normal (without scratches) or abnormal (with scratches).
[0003] Such detection methods (judgment methods) based on captured images using machine learning are required to improve judgment accuracy, reduce processing weight, and perform in real time. In particular, achieving high judgment accuracy with a small amount of training data is recognized as a common challenge in current AI technology.
[0004] The technology described in Patent Document 1 was also devised to address such issues. Patent Document 1 describes an apparatus that uses a reformation stage classification model to determine whether or not the soil being transported needs to be reformed. This reformation stage classification model classifies the soil into one of the reformation stages, and is created by learning a number of small area images cut out from each sample image prepared for each reformation stage. In addition, Patent Document 1 describes that a fast Fourier transform is performed using the values of pixels (pixel values) constituting this small-area image to obtain an amplitude spectrum. Then, the values of each pixel obtained after the fast Fourier transform are normalized, and the small-area image after the transform is used for machine learning as learning and verification (paragraph 0040 of the specification of Patent Document 1).
[0005] Furthermore, Patent Document 2 describes a pedestrian crossing detection device that includes an extraction means that extracts a zebra pattern of a predetermined period corresponding to a pedestrian crossing from a predetermined area of the captured image based on a spectrum pattern obtained by Fourier transforming the predetermined area of the captured image each time the captured image of the vehicle-mounted camera is updated due to an update of the signal input from the camera. Patent Document 2 describes a method for determining whether or not a candidate zebra pattern corresponding to a pedestrian crossing is present in an edge-processed image signal based on a comparison between the average value of the amplitude spectrum and an intensity threshold (paragraph 0042 of the specification of Patent Document 2).
[0006] Patent Document 3 describes a detection device and a program for improving the detection rate when detecting a detection target in an image. Patent Document 3 describes that high accuracy can be achieved by using a second determination unit (second classification model) to determine only objects (undetected objects) that were not detected by a first determination unit (first classification model) (paragraphs 0039 and 0040 of the specification of Patent Document 3). [Prior art documents] [Patent documents]
[0007] [Patent Document 1] JP 2020-041801 A [Patent Document 2] JP 2013-114652 A [Patent Document 3] JP 2021-157550 A Summary of the Invention [Problem to be solved by the invention]
[0008] However, the technology described in Patent Document 2 does not create data to be used for machine learning in the first place, and therefore the spectrum patterns obtained by the technology described in Patent Document 2 are not sufficient data to bring about effective learning results. Moreover, the technology described in Patent Document 3 aims to improve the accuracy of judgment by creating and using multiple classification models, so to speak. Therefore, (a lot of) teacher data and learning are required to create the multiple classification models.
[0009] The technology described in Patent Document 1 converts a captured image, but efficiently obtains a large amount of learning data by extracting multiple small area images from one image as learning data. This is fine when a similar object to be determined (soil) appears no matter which part is extracted from one image, such as soil, but when the objects shown differ in each part extracted from a captured image, such as machine parts such as screws and nuts, or the human body, it is not possible to obtain effective training data.
[0010] Therefore, the present invention provides an image processing device and an image processing method for creating data used for machine learning to achieve stable determination accuracy even with a small amount of data. [Means for solving the problem]
[0011] The image processing device of the present invention has a first calculation means that numerically converts a captured image into multiple color information for each pixel and creates data by combining each of the numerically converted color information, and a second calculation means that creates data to be used for machine learning by performing a fast Fourier transform on the data created by the first calculation means. As a result, the subject captured in the captured image is numerically converted into multiple pieces of color information, which are then combined to create integrated data, and the integrated data is then subjected to a fast Fourier transform.
[0012] It is also preferable that the first calculation means scans the captured image in the row and column directions, numerically converts the pixels of the captured image into multiple pieces of color information, and creates data by combining the multiple pieces of color information obtained by scanning in the row direction with the multiple pieces of color information obtained by scanning in the column direction. In this way, the subject in the captured image is scanned from multiple directions, in the row and column directions, so that more color information is obtained and combined.
[0013] Moreover, it is preferable that the first calculation means converts the captured image, which is a three-dimensional captured image, into a plurality of pieces of color information for each volume element. This allows more and more color information to be obtained and combined.
[0014] The image processing system of the present invention includes the image processing device described above, a light source that illuminates an object to be photographed, and an image capturing means that captures the object illuminated by the light source. It is preferable that the object to be photographed is a mechanical part, and that there are a plurality of light sources that illuminate the object from a plurality of different directions. As a result, the subject is illuminated from a plurality of different directions by a plurality of light sources, and even if the subject is changed by the imaging means, a uniform image without shadows can be obtained.
[0015] The image classification device of the present invention has a learning means for performing machine learning using training data created by the above-mentioned image processing device. Furthermore, it is preferable that the learning means performs machine learning using only normal data created by the above-mentioned image processing device based on captured images of a subject in a normal state. This generates a trained model capable of classifying the subject captured in the captured image.
[0016] In addition, the image processing device of the present invention has a step of creating data by using a first calculation means to numerically convert a captured image into multiple color information for each pixel and combining each of the numerically converted color information, and a step of creating data to be used for machine learning by using a second calculation means to perform a fast Fourier transform on the data created by the first calculation means.
[0017] In addition, the image processing program of the present invention causes a computer to operate as an image processing device having a first calculation means that numerically converts a captured image into multiple color information for each pixel and creates data by combining each of the numerically converted color information, and a second calculation means that creates data to be used for machine learning by performing a fast Fourier transform on the data created by the first calculation means. Effect of the Invention
[0018] (1) The image processing device of the present invention is configured to have a first calculation means that numerically converts a captured image into multiple pieces of color information for each pixel and creates data by combining each of the numerically converted color information, and a second calculation means that creates data to be used for machine learning by performing a fast Fourier transform on the data created by the first calculation means.In this manner, the subject captured in the captured image is numerically converted into multiple pieces of color information and then combined to create integrated data, and the integrated data is then further subjected to a fast Fourier transform.As a result, features that are not affected by changes in the position or shape of abnormalities present in the subject can be clearly extracted, and data that can be effectively used for machine learning can be created.
[0019] (2) Furthermore, the first calculation means scans the captured image in row and column directions, numerically converts the pixels of the captured image into multiple pieces of color information, and creates data by combining the multiple pieces of color information obtained by scanning in the row direction with the multiple pieces of color information obtained by scanning in the column direction. Since the subject shown in the captured image is scanned in multiple directions, in the row and column directions, more multiple pieces of color information can be obtained and combined, making it possible to more clearly extract features that are not affected by changes in the position or shape of abnormalities present in the subject.
[0020] (3) Furthermore, the first calculation means is configured to numerically convert the captured image, which is a three-dimensional captured image, into multiple color information for each volume element, so that even more multiple color information can be acquired and combined, making it possible to more clearly extract features that are not limited by changes in the position or shape of abnormal areas present in the subject.
[0021] (4) The image processing system of the present invention is configured to include the image processing device described above, a light source that illuminates the subject to be photographed, and an imaging means that photographs the subject illuminated by the light source. In particular, the subject to be photographed is a mechanical part, and there are multiple light sources that illuminate the subject to be photographed from multiple different directions. As a result, the subject to be photographed is illuminated from multiple different directions by the multiple light sources, and even if the subject to be photographed changes depending on the imaging means, a uniform image without shadows can be obtained, making it possible to extract the features of the subject to be photographed stably and without unevenness.
[0022] (5) The image classification device of the present invention is configured to have a learning means that performs machine learning using training data created by the above-mentioned image processing device. In particular, the learning means is further configured to perform machine learning using only normal data created by the above-mentioned image processing device based on captured images of an object showing a normal state. As a result, a trained model capable of classifying the object captured in the captured images is generated, and the object captured in the captured images can be classified using the trained model.
[0023] According to the image processing device and the image processing program of the present invention, it is possible to achieve the same effects as those of the image processing device of the present invention. [Brief description of the drawings]
[0024] [Figure 1] 1 is a schematic configuration diagram of an image processing system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a flow diagram for explaining an image processing method according to an embodiment of the present invention. [Diagram 3] 1 is a diagram for explaining an image processing method according to an embodiment of the present invention; [Figure 4] FIG. 13 is a diagram for explaining captured images, where (A) is an image of a normal part, (B) is an image of an abnormal part (with rust), and (C) is an image of an abnormal part (with a dent). [Diagram 5] FIG. 5 is a diagram for explaining transformed images created by an image processing method according to an embodiment of the present invention based on the captured image shown in FIG. 4, where (A) is a transformed image created based on a captured image showing a normal part, (B) is a transformed image created based on a captured image showing an abnormal part (with rust), and (C) is a transformed image created based on a captured image showing an abnormal part (with a dent). [Figure 6] 13A to 13C are diagrams for explaining a converted image created based on another image by the image processing method according to the embodiment of the present invention. [Figure 7] 13A to 13C are diagrams for explaining a converted image created based on another image by the image processing method according to the embodiment of the present invention. [Figure 8] 13A to 13C are diagrams for explaining a converted image created based on another image by the image processing method according to the embodiment of the present invention. [Figure 9] 13A to 13C are diagrams for explaining a converted image created based on another image by the image processing method according to the embodiment of the present invention. [Figure 10] 13A to 13C are diagrams for explaining a converted image created based on another image by the image processing method according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0025] The following describes in detail an embodiment of the present invention. The following description of the components is an example (representative example) of an embodiment of the present invention, and the present invention is not limited to the following content as long as it does not deviate from the gist of the present invention.
[0026] [Image processing system] 1 is a schematic diagram of an image processing system according to an embodiment of the present invention. As shown in FIG. 1, the image processing system 1 according to the embodiment of the present invention includes an image processing device 10, an image capturing means 20, and a light source 30.
[0027] The light source 30 illuminates the subject O, and is, for example, an LED light or a strobe.
[0028] The photographing means 20 photographs the subject O, and may be, for example, a camera that photographs two-dimensional images or a camera with a depth sensor (3D camera) that photographs three-dimensional images that include depth in addition to two dimensions (plane).
[0029] The photographed image data of the photographing object O illuminated by the light source 30 and photographed by the photographing means 20 is sent to the image processing device 10. At this time, the photographed image data can be sent from the photographing means 20 to the image processing device 10 via wired communication or wireless communication, or via a storage medium such as a USB or an external hard disk.
[0030] [Image processing device] The image processing device 10 processes captured image data to generate a converted image. The converted image is used as data for machine learning, such as training data.
[0031] As shown in FIG. 1, the image processing device 10 includes an input unit 110, a first calculation unit 120, a second calculation unit 130, a display unit 140, a storage unit 150, and an output unit 160.
[0032] The input means 110 performs input processing of the photographed image photographed by the photographing means 20. As described above, the input means 110 receives the photographed image sent from the photographing means 20 via wired communication, wireless communication, or a storage medium, and stores it in the storage means 150. The input means 110 receives various input operations from the user of the image processing device 10 via a keyboard, a mouse, a touch panel, or the like.
[0033] The storage means 150 stores (preserves) information necessary for the image processing device 10, and is, for example, a memory or a database. In addition to the captured image, the storage means 150 stores a converted image, an image in the middle of conversion, or a program for operating the image processing device 10.
[0034] The first calculation means 120 converts the captured image into a plurality of pieces of color information for each pixel, and creates data by combining the color information thus converted into numerical values. When the captured image is a three-dimensional captured image, the first calculation means 120 converts the captured image into a plurality of pieces of color information for each volume element, and creates data by combining the respective pieces of color information that have been converted into numerical values.
[0035] The second calculation means 130 performs a fast Fourier transform on the data created by the first calculation means 120 to create a transformed image. This transformed image is used as data for machine learning.
[0036] The output means 160 outputs the converted image created by the second calculation means 130 to an external device or the like via wired or wireless communication, or via a storage medium.
[0037] The display means 140 displays various information to the user, and is, for example, a display.
[0038] The image processing device 10 can be used as an image processing device according to an embodiment of the present invention by constructing (implementing) a program capable of realizing the functions as described above and shown in FIG. 1 on, for example, a mobile PC, a desktop PC, or a tablet.
[0039] [Image classification device] Although not shown, an embodiment of the present invention is an image classification device having a learning means for performing machine learning using a converted image created by the image processing device 10 as training data, etc. As the training data, for example, a converted image created based on a captured image showing a machine part (screw) as shown in Figures 4(A), (B), and (C) is used as the training data.
[0040] In addition, the image classification device performs machine learning using learning methods such as deep learning such as CNN (Convolution Neural Network), RNN (Recurrent Neural Network), and AutoEncoder, or transfer learning, to generate a trained model.
[0041] The image classification device then uses the trained model to judge the captured image. For example, the image classification device performs inference using the trained model for a captured image of a screw manufactured in a factory, and judges whether the screw in the captured image is normal (no rust or dents) or abnormal (with rust or dents).
[0042] The image processing device 10 may have this learning means, so that the image processing device 10 has the above-mentioned determination function. In this case, the learning means performs learning using the converted image stored in the storage means 150 as training data, and the generated trained model is stored in the storage means 150.
[0043] [Image processing method] FIG. 2 is a flow diagram illustrating an image processing method according to an embodiment of the present invention. Hereinafter, an image processing method using an image processing device according to the present embodiment will be described with reference to the drawings. Note that in this description, it is assumed that the image processing method according to the present embodiment is performed by a user.
[0044] First, a photographed image photographed by the photographing means 20 is input to the image processing device 10 (step S110). Then, input processing is performed by the input means 110, and the photographed image is stored in the storage means 150. In this description, the photographed image will be described as a two-dimensional image.
[0045] Next, the captured image is converted into a plurality of pieces of color information for each pixel by the first calculation means 120 (step S120). Fig. 3 is a diagram for explaining an image processing method according to an embodiment of the present invention. To make it easier to understand, a part of a captured image is enlarged and explained. In the example shown in Fig. 3, the number of pixels in the enlarged part of the image is "8 x 8".
[0046] Then, the first calculation means 120 calculates a plurality of pieces of color information for each of the 8×8 pixels. For example, when the multiple color information is red, green, and blue information based on the RGB color model, the <red>, <green>, and <blue> color information for each 8×8 pixel is calculated as follows:
[0047] <Red> 255 175 72 32 210 216 53 8 202 158 99 66 199 195 49 24 119 110 120 96 156 139 105 95 100 141 114 112 114 86 217 224 69 115 104 112 146 110 244 247 62 85 103 129 122 116 73 66 112 94 63 108 154 175 35 0 140 107 32 78 181 208 43 0
[0048] <Green> 1 35 156 164 209 207 156 168 18 52 149 154 210 207 145 161 78 77 102 99 189 198 184 187 147 157 63 67 146 148 255 255 155 158 61 56 149 136 255 255 129 131 106 101 77 74 56 61 102 117 150 150 43 34 0 0 93 119 161 155 37 18 0 0
[0049] <Blue> 1 10 81 42 41 42 47 85 30 39 88 54 79 79 56 91 112 88 66 42 132 144 136 140 199 183 44 34 135 143 252 248 216 193 45 31 158 153 255 252 197 167 79 64 84 96 66 65 173 148 99 84 26 42 0 4 165 145 97 73 10 18 0 5
[0050] For example, the number in the top left corner is the color information of the top left pixel (the shaded area in Figure 3) of an 8 x 8 pixel array. Therefore, in this example, the <red> color information of the pixel in the shaded area in Figure 3 is 255, the <green> color information is 1, and the <blue> color information is 1.
[0051] Then, the first calculation means 120 combines the numerically converted color information to create data (step S130). In this explanation, the <red> color information for each pixel of the captured image is combined with the <green> color information for each pixel, and the <blue> color information for each pixel is combined with the <red> color information for each pixel.
[0052] For example, in the case of the color information in the above 8 x 8 pixels, when the captured image is scanned in the row direction (horizontal direction), this integrated data is "255,175,72,………,208,43,0,………,1,35,156,………,18,0,0,………,1,10,81,………,18,0,5" The data is a list of 64 x 3 = 192 color pieces of information like this:
[0053] Then, the second calculation means 130 performs a fast Fourier transform on the data created by the first calculation means, and a transformed image is created (step S140).
[0054] Finally, the converted image is output by the output means 160 (step S150). The converted images obtained by the above-mentioned procedure are used as training data in machine learning.
[0055] [Example 1] Fig. 4 is a diagram for explaining a photographed image. As shown in Fig. 4(A) to Fig. 4(C), the photographed subject is a machine part (screw). Here, the photographed image shown in Fig. 4(B) shows a screw with rust (see the dotted circle), and the photographed image shown in Fig. 4(C) shows a screw with a dent (see the dotted circle). On the other hand, the photographed image shown in Fig. 4(A) shows a normal screw without such rust or dents.
[0056] Moreover, Fig. 5 is a diagram for explaining a converted image created by the image processing method according to the present embodiment based on the captured image shown in Fig. 4. Here, Fig. 5(A) is a converted image created based on the captured image shown in Fig. 4(A), Fig. 5(B) is a converted image created based on the captured image shown in Fig. 4(B), and Fig. 5(C) is a converted image created based on the captured image shown in Fig. 4(C).
[0057] As shown in Fig. 5, the created converted image is an image in which frequency components are extracted as amplitude (vertical axis). Note that since color information does not have a time axis (time information), the second calculation means 130 performs fast Fourier transform using an arbitrary value (for example, a frequency of 100 Hz with a period of 10 milliseconds).
[0058] Here, when comparing Fig. 5(A) with Fig. 5(B), it can be seen that there are parts where the amplitudes are different (see the arrows in Fig. 5(B)). Also, when comparing Fig. 5(A) with Fig. 5(C), it can be seen that there are parts where the amplitudes are different (see the arrows in Fig. 5(C)). Also, when comparing Fig. 5(B) with Fig. 5(C), it can be seen that the amplitudes are different.
[0059] In other words, the converted image shown in Fig. 5 can be said to have extracted features of the subject captured in the captured image, such as rust, dents, and nothing. Therefore, this converted image can be said to be effective as training data indicating normality or abnormality in machine learning, and by performing machine learning using this training data, it can be said to be possible to generate a trained model that can achieve stable judgment accuracy even with a small amount of data.
[0060] [Example 2] 6 to 9 are diagrams for explaining a converted image created by the image processing method according to the present embodiment based on another image. In Fig. 6 to Fig. 9, the upper part shows an image assumed to be a photographed image. Meanwhile, the lower part shows a converted image created by the image processing method according to the present embodiment based on that image.
[0061] [Reposition] For example, the image shown in the top of Figure 6(A) shows a hexagram pattern that is assumed to be an anomaly (a hexagram-shaped scratch). From this image, a converted image like the one shown in the bottom of Figure 6(A) was created.
[0062] The image shown in the top of Figure 6(B) is the image in which only the position (location) of the hexagram pattern in the image shown in the top of Figure 6(A) has been changed. In other words, the position of the hexagram pattern has changed, but the size and color remain unchanged. Then, a converted image like that shown in the bottom of Figure 6(B) was created from this image.
[0063] Comparing the converted images in Figures 6(A) and (B), we can see that the amplitude and shape do not change. In other words, even if the position of scratches, rust, dents, etc. in the captured image changes, the information (features) can be accurately extracted. In short, when using converted images as training data in machine learning, it can be said that there is high robustness against changes in the position of abnormalities.
[0064] [Change Color] Figure 7(A) shows the same image and transformed image as shown in Figure 6(A), and is shown for comparison with Figure 7(B). On the other hand, FIG. 7(B) shows the hexagram pattern in the image shown at the top of FIG. 7(A) with the color changed.
[0065] Comparing the converted images in Figure 7(A) and (B), we can see that the amplitude changes, but the shape of the amplitude does not. In other words, by correcting (e.g., enlarging or reducing) the amplitude shown in the converted image, we can extract features that can be used to determine whether there are abnormal parts. Furthermore, while the shape of the amplitude does not change, the magnitude of the amplitude does, and so it is clear that it is possible to extract features that can distinguish between anomalies that differ only in color, such as red rust and blue rust.
[0066] [Resize] In addition, Fig. 8(A) is the same as the image and the converted image shown in Fig. 6(A), and is shown for comparison with Figs. 8(B) and (C). On the other hand, Figure 8(B) shows the size of the hexagram pattern in the image shown at the top of Figure 8(A) changed (enlarged), while Figure 8(C) shows the opposite, where the size has been made smaller.
[0067] Comparing the converted images in Figures 8(A) to 8(C), we can see that the amplitude and shape change. Therefore, we can see that it is possible to extract features that can distinguish the "degree of abnormality," such as the size of a scratch.
[0068] [shape change] In addition, Fig. 9(A) is the same as the image and the converted image shown in Fig. 6(A), and is shown for comparison with Figs. 9(B) and (C). On the other hand, in FIG. 9(B), the shape of the pattern has been changed from a hexagram to a hexagon, and in FIG. 9(C), the hexagram pattern has been rotated 90 degrees to the left.
[0069] Comparing the converted images in Figures 9(A) to 9(C), we can see that the amplitude and shape change. Therefore, we can see that it is possible to extract features that can distinguish between "types of abnormalities," such as scratches and dents.
[0070] [Example 3] Fig. 10 is a diagram for explaining a converted image created by the image processing method according to the present embodiment based on another image. In Fig. 10, the upper part also shows an image assumed to be a photographed image, and the lower part shows a converted image created by the image processing method according to the present embodiment based on that image.
[0071] For example, the image shown in the top of Fig. 10(A) shows a horizontal line pattern that can be assumed to be abnormal (horizontal scratches), the image shown in the top of Fig. 10(B) shows a vertical line pattern that can be assumed to be abnormal (vertical scratches), and the image shown in the top of Fig. 10(C) shows a diagonal line pattern that can be assumed to be abnormal (diagonal scratches). Then, from these images, transformed images such as those shown in the lower part of FIG. 10(A) to FIG. 10(C) were created.
[0072] Comparing the converted images in Figures 10(A) and (B), we can see that the amplitude and shape of each image are different. On the other hand, the converted image in Figure 10(C) is a combination of the amplitudes shown in the converted images in Figures 10(A) and (B). In other words, by scanning the captured image as described above not only in the row direction (horizontal direction) but also in the column direction (vertical direction) and creating data that combines the color information obtained from each scan, it can be said that the robustness to changes in the shape of the abnormality or the direction in which the abnormality is located (for example, scratches located horizontally, scratches located vertically, scratches located diagonally, etc.) is high.
[0073] Therefore, if the first calculation means 120 scans the captured image in the row and column directions to numerically convert the pixels of the captured image into multiple color information, combines the multiple color information obtained by scanning in the row direction with the multiple color information obtained by scanning in the column direction to create data, and then the second calculation means 130 performs a fast Fourier transform on the data to create a converted image, it is expected that the amplitude of the converted image based on the image shown at the top of Figure 9(A) and the amplitude of the converted image based on the image shown at the top of Figure 9(C) will be approximately the same size and shape.
[0074] Although the embodiment of the present invention has been described above, the described embodiment is merely an example, and the present invention is not limited to the contents thereof as long as it does not deviate from the gist of the present invention.
[0075] [Unsupervised learning] For example, an image classification device has been described that has a learning means for performing machine learning using training data of a converted image created by the image processing device 10. In other words, the learning means has been described as performing machine learning based on training data including both normal data and abnormal data as follows. Normal data: Training data created based on an image (Figure 4(A)) of a target (screw) in a normal state. Abnormal data: Training data created based on images (Fig. 4 (B) and (C)) of an object (screw) that shows abnormal conditions such as rust or dents.
[0076] The learning means can perform supervised learning using training data labeled with such terms as "normal," "abnormal (rust)," and "abnormal (dent)," as well as unsupervised learning. For example, the learning means can perform machine learning using only the normal data. Specifically, the learning means can use One Class SVM (Support Vector Machine), a method applied to one-class classification in unsupervised learning, to learn one class as normal data, determine a classification boundary, and detect outliers (abnormal data) based on the classification boundary.
[0077] That is, as is clear from the converted images shown in Fig. 5(A) to Fig. 5(C), the image processing device 10 can sufficiently extract features capable of determining the classification boundary between normal data (Fig. 5(A)) and abnormal data (Figs. 5(B) and (C)). Therefore, the learning means can perform learning such that, for example, using the data shown in Fig. 5(A) as a reference and based on the Euclidean distance or Mahalanobis distance, data that is determined to be within the classification boundary as indicated by the magnitude or shape of the amplitude is determined to be normal data, and data that is determined to be outside the classification boundary is determined to be abnormal data.
[0078] As a practical problem, while it is desirable to perform supervised learning that includes both normal and abnormal data, there are cases where it is difficult to obtain this abnormal data. For example, in a factory that manufactures machine parts, the probability of manufacturing an abnormal part is extremely low, such as 1 in 100,000 to 1 million parts. Therefore, it can be extremely difficult to obtain sufficient data on multiple types of abnormalities, such as red rust, blue rust, and dents. According to the image processing device 10 and image classification device of the present invention, even in such cases, unsupervised learning can be performed using the created converted images, making it possible to achieve stable judgment accuracy even with a small amount of data.
[0079] In this way, the transformed image created by the image processing device 10 can be used for both supervised learning and unsupervised learning.
[0080] [Preprocessing] It is also possible to perform pre-processing (image processing) on the captured image. For example, the first calculation means 120 converts the color tones of the captured image into red, green, and blue tones based on the RGB color model, respectively. In this way, a plurality of images are generated, such as a captured image converted into a red color tone, a captured image converted into a green color tone, and a captured image converted into a blue color tone, and the first calculation means 120 further converts the generated (converted) images into a plurality of numerical values of color information.
[0081] Color tone conversion based on the RGB color model means, for example, when the RGB values are specified as 0 to 255, image conversion is performed so that in the case of a captured image converted to a red color tone, the red value is set to 255 and the green and blue values are set to 0.
[0082] Then, the first calculation means 120 numerically converts the converted image into a plurality of pieces of color information. At this time, (1) the first calculation means 120 may create data by combining a plurality of pieces of color information based on the captured images converted into red, green, and blue tones, respectively. Also, (2) the first calculation means 120 may create data combining a plurality of pieces of color information based on the captured images converted into red tones, data combining a plurality of pieces of color information based on the captured images converted into green tones, and data combining a plurality of pieces of color information based on the captured images converted into blue tones, respectively.
[0083] By carrying out such pre-processing, it is possible to extract the feature quantities of abnormal areas captured in the captured image more clearly. In other words, it is possible to identify abnormal areas captured in the captured image more accurately. This is because some abnormal areas have distinctive colors, such as red rust or blue rust, and by carrying out such pre-processing, it is possible to make the characteristics of the abnormal areas more prominent. For example, in the case where combined data is created based on the captured image converted to a red color tone by the first calculation means 120 as in (2) above, and then learning data to be used for machine learning is created by the second calculation means 130, even if the learning data is not able to clearly extract the features of the abnormal area, the learning data created based on the captured image converted to a green or blue color tone may be able to clearly extract the features of the abnormal area.
[0084] Then, the image classification device performs machine learning using learning data created based on the captured images converted into red, green, and blue tones, and generates trained models. When inference is performed for a certain captured image using the multiple trained models created, if any one of the models is determined to be abnormal, it can be determined that the subject captured in the captured image has an abnormal part. In particular, in detection methods based on captured images using machine learning, it is acceptable to a certain extent to determine that something is normal as abnormal, but it is not permitted to determine that something is abnormal as normal.
[0085] [3D image] In addition, although the above description has mainly been given of the captured image as a two-dimensional image, the image processing device 10 can also create a converted image based on a three-dimensional image that includes depth (profoundness) in addition to two dimensions (plane). In this case, the first calculation means 120 numerically converts the captured image into a plurality of color information for each volume element, and creates data by combining the respective color information that has been numerically converted. Then, the second calculation means 130 creates a converted image by performing a fast Fourier transform on the data.
[0086] This allows color information to be obtained based on three-dimensional information (XYZ), which contains more information than two-dimensional information (XY), making it possible to extract the features of areas with abnormalities such as rust, dents, scratches, etc. For example, if the subject has a dent, the area behind the dent (deep areas) is darker than the area in front of the dent (shallow areas), meaning there is a difference in color information, and this feature can be extracted with greater precision.
[0087] Furthermore, the subject to be imaged may be not only mechanical parts such as screws and nuts, but also parts inside and outside the human body where abnormalities such as malignant tumors occur, such as inside the stomach or on the skin.
[0088] Furthermore, the multiple color information may be information on hue, saturation, and brightness based on the HSV model other than red, green, and blue information based on the RGB color model, or may be other color information. The color information can be appropriately designed and changed depending on the subject to be photographed, for example, whether the subject to be photographed is a light color, a dark color, or is glossy. [Industrial Applicability]
[0089] INDUSTRIAL APPLICABILITY The present invention is industrially useful because it can be effectively used in fields such as supervised learning and unsupervised learning as an image processing device or image processing method for creating data used in machine learning to achieve stable judgment accuracy even with a small amount of data. [Explanation of symbols]
[0090] 1. Image Processing System 10 Image processing device 110 Input means 120 First Calculation Means 130 Second Calculation Means 140 Display means 150 Memory means 160 Output Method 20. Shooting Method 30 light source O Shooting subject
Claims
1. a first calculation means for converting a captured image into a plurality of pieces of color information for each pixel, and creating data by combining the color information thus converted; A second calculation means for generating data to be used for machine learning by performing a fast Fourier transform on the data generated by the first calculation means; having The data created by the second calculation means can be used for machine learning to generate a trained model that can detect the presence or absence of an abnormality in a target and distinguish the type of the abnormality, The first calculation means scans the captured image in the row and column directions, converts the pixels of the captured image into a plurality of pieces of color information, and creates the data by combining the plurality of pieces of color information obtained by scanning in the row direction with the plurality of pieces of color information obtained by scanning in the column direction. Image processing device.
2. 2. The image processing apparatus according to claim 1, wherein the first calculation means converts the captured image, which is a three-dimensional captured image, into a plurality of pieces of color information for each volume element.
3. 3. The image processing apparatus according to claim 1, wherein the plurality of pieces of color information are information of red, green, and blue based on an RGB color model, or information of hue, saturation, and brightness based on an HSV model.
4. the plurality of color information are red, green, and blue information based on an RGB color model; 3. The image processing device according to claim 1, wherein the first calculation means converts the captured image into red, green, and blue color tones based on an RGB color model, and numerically converts the converted image into the plurality of color information.
5. A light source that illuminates the subject, An imaging means for imaging the object illuminated by the light source; The image processing device according to claim 3 ; An image processing system comprising:
6. The subject to be photographed is a mechanical part, The light source is provided in a plurality of different directions to illuminate the subject. The image processing system according to claim 5 .
7. 2. An image classification device comprising: a learning means for performing machine learning using training data created by the image processing device according to claim 1.
8. The image classification device according to claim 7 , wherein the learning means further performs machine learning using only normal data created by the image processing device based on captured images of a subject in a normal state.
9. a step of converting the captured image into a plurality of pieces of color information for each pixel by a first calculation means, and creating data by combining the respective pieces of color information thus converted; A step of generating data to be used for machine learning by performing a fast Fourier transform on the data generated by the first calculation means using a second calculation means; having The data created by the second calculation means can be used for machine learning to generate a trained model that can detect the presence or absence of an abnormality in a target and distinguish the type of the abnormality, The image processing method includes: a first calculation means for scanning the captured image in row and column directions, numerically converting the pixels of the captured image into a plurality of pieces of color information; and creating the data by combining the plurality of pieces of color information obtained by scanning in the row direction with the plurality of pieces of color information obtained by scanning in the column direction.
10. Computer, a first calculation means for converting a captured image into a plurality of pieces of color information for each pixel, and creating data by combining the color information thus converted; A second calculation means for generating data to be used for machine learning by performing a fast Fourier transform on the data generated by the first calculation means; having The data created by the second calculation means can be used for machine learning to generate a trained model that can detect the presence or absence of an abnormality in a target and distinguish the type of the abnormality, The first calculation means is an image processing program that operates as an image processing device that scans the captured image in the row and column directions, numerically converts the pixels of the captured image into multiple color information, and creates the data by combining the multiple color information obtained by scanning in the row direction with the multiple color information obtained by scanning in the column direction.
Citation Information
Patent Citations
Defect examining method
JP2004191112A
Crosswalk detection device
JP2013114652A
Image processing apparatus and image processing method
JP2018201156A
Soil reforming determination device and foreign matter removal method
JP2020041801A
Detection device and program
JP2021157550A
