Image generation device, learning device, image processing device, image generation method, learning method, and image processing method

Abstract

The image generation device has at least one first processor. The first processor acquires a first image, generates a plurality of frequency-processed images by emphasizing or extracting different frequency components with respect to the first image, and performs different arithmetic processes on each of the plurality of frequency-processed images. At least one second image is generated by synthesizing the frequency components of each of the plurality of frequency-processed images subjected to the arithmetic processing, and the first image and the second image are output as training data for machine learning of a mathematical model for performing a predetermined inference on the input image.

Description

Technical Field

[0001] The disclosed technology relates to an image generating device, a learning device, an image processing device, an image generating method, a learning method, and an image processing method. Background Art

[0002] As a technology related to machine learning of mathematical models for image processing, the following technology is known. For example, in International Publication No. 2020 / 017211, there is described a medical image learning device, which comprises: an image acquisition unit, which acquires a first image and a second image with different spectral distributions; an image processing unit, which performs image processing on the first image to generate a third image; and a learning unit, which uses the first to third images to train a recognizer for automatic recognition. The image processing unit generates the first to third images by performing at least any one of the following image processing: image processing that suppresses the signal of a frequency band included in the spectral distribution of the first image, whose characteristics are different from the frequency band corresponding to the second image; and emphasizing the signal of a frequency band included in the first image, whose characteristics are the same or similar to the frequency band corresponding to the second image.

[0003] International Publication No. 2020 / 175446 describes a learning method for machine learning of a generative model, which infers a second image including image information having a higher resolution than the first image based on a first image. In this method, a first learning image and a second learning image are used as learning data, the first learning image being an image including first resolution information having a lower resolution than the second image, and the second learning image being an image including second resolution information having a higher resolution than the first learning image and being a correct image corresponding to the first learning image. Summary of the invention

[0004] Technical issues to be solved by the invention

[0005] In the non-destructive inspection of industrial products using radiographic images, the following attempt is made: using mathematical models such as CNN (Convolutional Neural Network) constructed through machine learning to detect defects (damage) generated inside the product. In machine learning, there is usually a problem that the training data is insufficient relative to the complexity of the problem to be solved. In order to ensure as many variations as possible from a small amount of data, so-called "data expansion" is often performed. For example, in the case where the training data is image data, it can be expected that the accuracy of the mathematical model of the learning object can be improved by adding data that has been processed on the original image to the training data. The processing is to increase or decrease the brightness (pixel value), change the contrast, perform zooming, change the mixing ratio of the color channels of the color image, rotate the original image, and reverse the original image.

[0006] In machine learning of mathematical models for image recognition, even with the usual data expansion methods described above, it is sometimes impossible to make up for the lack of actual data. In particular, in radiographic images used for non-destructive inspection of industrial products, various information such as fluctuations accompanying thickness changes of the object (product) itself and shape characteristics of the object (product) itself are captured as background along with defects (damage) as the inspection object. As a result, although the variety of images to be prepared as training data becomes huge, it is not easy to prepare an image set that includes the diversity of background information.

[0007] The disclosed technology is created in view of the above viewpoints, and its purpose is to expand the variation of image sets used as training data in machine learning.

[0008] Means for solving technical problems

[0009] The image generation device involved in the disclosed technology has at least one first processor. The first processor acquires a first image, generates a plurality of frequency-processed images by emphasizing or extracting frequency components that are different from each other for the first image, performs different operation processing on each of the plurality of frequency-processed images, generates at least one second image by synthesizing the frequency components of the plurality of frequency-processed images that have been subjected to the operation processing, and outputs the first image and the second image as training data, which is used for machine learning of a mathematical model that performs a predetermined inference on an input image.

[0010] The first processor may perform a process of adding different weighting coefficients to a plurality of frequency-processed images as a calculation process. The first processor may generate a first frequency-processed image including a relatively low frequency component and a second frequency-processed image including a relatively high frequency component, and perform a process of adding a relatively small weighting coefficient to the first frequency-processed image and a relatively large weighting coefficient to the second frequency-processed image as a calculation process. The first processor may perform a process of amplifying the difference between the brightness value of each pixel and the average value at different magnifications for each of the plurality of frequency-processed images as a calculation process.

[0011] The first processor may generate a first frequency-processed image by filtering the first image, and generate a second frequency-processed image by subtracting the frequency component of the first frequency-processed image from the first image. The first processor may perform normalization processing on the first image and the second image to normalize the brightness. The first image is a radiographic image including an image of a specific structural portion of an object in a specific frequency region, and the mathematical model may be a model for detecting the image of the specific structural portion included in the radiographic image.

[0012] The learning device involved in the disclosed technology has at least one second processor. The second processor uses the first image and the second image provided by the above-mentioned image generation device as training data to train the mathematical model. The second processor can perform normalization processing on the first image and the second image to normalize the brightness.

[0013] The image processing device involved in the disclosed technology has at least one third processor. The third processor uses the mathematical model learned by the above-mentioned learning device to output the detection result of the image of the specific structural part of the object for the input image. The third processor can obtain the input image, generate a frequency-processed image by emphasizing or extracting a specific frequency component for the input image, obtain a first inference result by inputting the input image into the mathematical model, obtain a second inference result by inputting the frequency-processed image into the mathematical model, and output the detection result by comprehensively evaluating the first inference result and the second inference result.

[0014] The disclosed technology involves an image generation method in which at least one first processor of an image generation device performs the following processing: acquiring a first image; generating a plurality of frequency-processed images by emphasizing or extracting different frequency components of the first image; performing different operation processing on each of the plurality of frequency-processed images; generating at least one second image by synthesizing the frequency components of each of the plurality of frequency-processed images that have been subjected to the operation processing; and outputting the first image and the second image as training data, which is used for machine learning of a mathematical model that performs prescribed inferences on an input image.

[0015] The disclosed technology involves a learning method in which at least one second processor of a learning device performs the following processing: a first image and a second image provided by the above-mentioned image generation method are used as training data to train a mathematical model.

[0016] The image processing method involved in the disclosed technology is that at least one third processor of the image processing device executes the following processing: using the mathematical model learned by the above-mentioned learning method, outputs the detection result of the image of a specific structural part of the object for an input image.

[0017] Effects of the Invention

[0018] According to the disclosed technology, it is possible to expand variations of image sets used as training data in machine learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a diagram showing an example of the configuration of an image processing system according to an embodiment of the disclosed technology.

[0020] Figure 2This is a diagram showing an example of the hardware configuration of an image generating device according to an embodiment of the disclosed technology.

[0021] Figure 3 This is a functional block diagram showing an example of the functional configuration of an image generating device according to an embodiment of the disclosed technology.

[0022] Figure 4 This is a diagram showing an example of the flow of processing in the image generating device according to the embodiment of the disclosed technology.

[0023] Figure 5 This is a flowchart showing an example of the flow of image generation processing according to an embodiment of the disclosed technology.

[0024] Figure 6 This is a diagram showing an example of the hardware configuration of a learning device according to an embodiment of the disclosed technology.

[0025] Figure 7 This is a functional block diagram showing an example of the functional configuration of a learning device according to an embodiment of the disclosed technology.

[0026] Figure 8 This is a diagram showing an example of the hardware configuration of an image processing device according to an embodiment of the disclosed technology.

[0027] Fig. 9 This is a functional block diagram showing an example of the functional configuration of an image processing device according to an embodiment of the disclosed technology.

[0028] Fig.10 This is a diagram showing an example of the flow of processing in the image processing device according to the embodiment of the disclosed technology.

[0029] Fig.11 This is a flowchart showing an example of the flow of defect detection processing involved in the embodiment of the disclosed technology. DETAILED DESCRIPTION

[0030] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the accompanying drawings. In addition, in each of the drawings, the same reference numerals are given to the same or equivalent components and parts, and repeated descriptions are omitted.

[0031] Figure 11 is a diagram showing an example of the structure of an image processing system 1 involved in an embodiment of the disclosed technology. The image processing system 1 includes an image generating device 10, a learning device 20, and an image processing device 30. The image generating device 10 generates a second image by performing frequency processing on a first image, and outputs the first image and the second image as training data, which is machine learning of a mathematical model for performing a predetermined inference on an input image. The learning device 20 uses the first image and the second image provided by the image generating device 10 as training data to train the mathematical model. The image processing device 30 uses the mathematical model learned by the learning device 20 to output a detection result of an image of a specific structural part of an object for an input image.

[0032] The details of the image generation device 10, the learning device 20, and the image processing device 30 are described in sequence below. The following description takes the case where the image processed in the image processing system 1 is a radiographic image obtained in the non-destructive inspection of industrial products as an example. The radiographic image may include an image of a defect (damage) generated inside the object. The following description takes the following case as an example: the mathematical model processed in the image processing system 1 is a defect detection model for detecting defects (damage) included in the radiographic image, and the image processing device 30 uses the defect detection model to detect defects (damage) included in the radiographic image. In addition, in the following description, the case where the image generation device 10, the learning device 20, and the image processing device 30 are composed of different computers is described as an example.

[0033] [Image generation device]

[0034] Figure 2 1 is a diagram showing an example of the hardware structure of the image generating device 10. The image generating device 10 includes a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a non-volatile memory 103, an input device 104, a display 105, and a network interface 106. These hardware are connected to a bus 107. The display 105 is, for example, a liquid crystal display. The input device 104 includes, for example, a keyboard and a mouse, and may also include a proximity input device such as a touch display, and a sound input device such as a speaker. The network interface 106 is an interface for connecting the image generating device 10 to a network.

[0035] The nonvolatile memory 103 is a nonvolatile storage medium such as a hard disk and a flash memory. An image generation program 110 is stored in the nonvolatile memory 103. The RAM 102 is a working memory for the CPU 101 to execute processing. The CPU 101 loads the image generation program 110 stored in the nonvolatile memory 103 into the RAM 102 and executes processing according to the image generation program 110. The CPU 101 is an example of a "first processor" in the disclosed technology.

[0036] Figure 3 1 is a functional block diagram showing an example of the functional structure of the image generating device 10. The image generating device 10 functions as a first image acquiring unit 11, a frequency processing unit 12, an arithmetic processing unit 13, a synthesis processing unit 14, and an image output unit 15 by the CPU 101 executing an image generating program 110. Figure 4 FIG. 1 is a diagram showing an example of a process flow in the image generating device 10. Figure 4 The functions of the above-described functional units constituting the image generating device 10 will be described.

[0037] The first image acquisition unit 11 acquires a first image 41. The first image 41 is a radiographic image acquired in a non-destructive inspection of an industrial product. The first image 41 includes an image of a defect (damage) generated inside an object (product). Here, generally, the image can be interpreted as being able to be resolved into spatial frequency components. The image of the defect (damage) exists in a higher frequency region of the first image 41. For example, when the resolution of the first image 41 is 25 to 100 μm / px, the smallest image of the defect (damage) is 1 to 2 px. In addition, the first image 41 may be the original image or an image to which pre-processing other than the frequency processing described later, such as enlargement, reduction, rotation, and grayscale processing, has been applied to the original image. The first image acquisition unit 11 may perform this pre-processing.

[0038] The frequency processing unit 12 generates a first frequency-processed image 42 by performing a first frequency processing 51 on the first image 41. The first frequency processing 51 is a process of extracting low-frequency components from the first image 41 or a process of emphasizing the low-frequency components of the first image 41. The first frequency processing 51 may be, for example, a filtering process using a low-pass filter. In addition, the first frequency processing 52 is well known as a noise removal method. It may also be implemented using a median filter, a mean filter, a Gaussian filter, or a bilateral filter. In the first frequency-processed image 42, the image of the defect (damage) included in the first image 41 is substantially removed, while on the other hand, the image of the background such as the fluctuations accompanying the thickness change of the object (product) itself and the shape characteristics of the object (product) itself remain.

[0039] The frequency processing unit 12 generates a second frequency-processed image 43 by further performing a second frequency processing 52 on the first image 41. The second frequency processing 52 is a process of extracting high-frequency components from the first image 41 or a process of emphasizing high-frequency components of the first image 41. The second frequency processing 52 may be, for example, a filtering process using a high-pass filter. The second frequency-processed image 43 may also be generated by subtracting the frequency components of the first frequency-processed image 42 from the first image 41. In the second frequency-processed image 43, the image of the background such as the fluctuations accompanying the thickness change of the object (product) itself and the shape characteristics of the object (product) itself are substantially removed, while the image of the defect (damage) included in the first image 41 remains. That is, in the second frequency-processed image 43, the image of the defect (damage) is more clearly depicted.

[0040] The calculation processing unit 13 performs a first calculation processing 53 on the first frequency processed image 42, and performs a second calculation processing 54 different from the first calculation processing 53 on the second frequency processed image 43. For example, the calculation processing unit 13 may add a relatively small weighting coefficient w to the first frequency processed image 42. 1 (≥0) is performed as the first operation 53, and a relatively large weighting coefficient w is added to the second frequency processed image 43. 2 (>w 1 ) is the second operation process 54. That is, after setting the brightness value of each pixel of the first frequency processed image 42 to P1 x,y When the calculation processing unit 13 outputs w 1 ×P1 x,y The first frequency processed image 42A after the first operation is taken as the first frequency processed image. x,y When the calculation processing unit 13 outputs w 2 ×P2 x,y The second frequency processed image 43A after the second operation is obtained. In addition, the weighting coefficient w added to the first frequency processed image 42 mainly including low frequency components may be 1 The calculation processing unit 13 can also change the weighting coefficient w 1 With w 2 , and generates a plurality of image pairs consisting of the first frequency-processed image 42A and the second frequency-processed image 43A after the calculation process.

[0041] Instead of or in addition to the process of adding weighting coefficients of different magnitudes to the first frequency processed image 42 and the second frequency processed image 43, the calculation processing unit 13 may perform the following process as the calculation processing: contrast and emphasize the first frequency processed image 42 and the second frequency processed image 43 with different intensities. Specifically, the difference between the brightness value of each pixel and the average value may be multiplied by a factor m for the first frequency processed image 42. 1 The first operation 53 is to perform a magnification process of (>0) and to perform a magnification process of the second frequency processed image 43 so that the difference between the brightness value of each pixel and the average value is reduced to m. 1 Large magnification m 2 (>m 1 ) as the second operation processing 54. The operation processing unit 13 may also change the magnification m 1 With m 2 , and generates a plurality of image pairs consisting of the first frequency-processed image 42A and the second frequency-processed image 43A after the calculation process.

[0042] The synthesis processing unit 14 generates at least one second image 44 by performing a synthesis processing 55 for synthesizing the frequency components of the first frequency processed image 42A subjected to the first operation processing 53 and the frequency components of the second frequency processed image 43A subjected to the second operation processing 54. For example, the weighting coefficient w is added to each of the first frequency processed image 42 and the second frequency processed image 43 by the operation processing unit 13. 1 、w 2 In the case of the processing of w, the first frequency processed image 42 and the second frequency processed image 43 are subjected to the weighted addition processing, and an image is generated as the second image 44 through the synthesis processing 55. 1 <w 2 , a second image 44 in which the image of the defect (damage) existing in the high-frequency region is further emphasized can be obtained. 1 When the second frequency processed image 43 is set to zero, the second frequency processed image 43 becomes the second image 44, and the image of the defect (damage) is more clearly depicted in the second image 44. When a plurality of image pairs consisting of the first frequency processed image 42A and the second frequency processed image 43A after the operation are generated, the synthesis processing unit 14 performs synthesis processing 55 on each of the plurality of image pairs. Thus, a plurality of second images 44 are generated from one first image 41.

[0043] The image output unit 15 outputs the first image 41 acquired by the first image acquisition unit 11 and at least one second image 44 generated based on the first image 41 as training data, and the training data is used for machine learning of a mathematical model that performs a predetermined inference on the input image. The image output unit 15 may also output the first image 41 and the second image 44 after performing a normalization process to normalize the brightness of each pixel. The normalization process is represented by the following formula (1), for example. In formula (1), X norm is the brightness value after normalization. X is the brightness value before normalization. max is the maximum value of brightness (e.g. 255). min is the minimum value of brightness (for example, 0). By performing the normalization process, it is possible to suppress the brightness distribution from being significantly different between the first image 41 and the second image 44 .

[0044] X norm =(XX min ) / (X max -X min )・・・(1)

[0045] Figure 5 1 is a flowchart showing an example of the flow of the image generation process performed by the CPU 101 when the image generation program 110 is executed. The image generation program 110 is executed when the user instructs to start the process by operating the input device 104, for example.

[0046] In step S1, the first image acquisition unit 11 acquires the first image 41. The first image 41 may be an original image or an image pre-processed other than frequency processing such as enlargement, reduction, rotation, and grayscale processing on the original image. The first image acquisition unit 11 may perform the pre-processing.

[0047] In step S2, the frequency processing unit 12 generates a first frequency-processed image 42 and a second frequency-processed image 43 by emphasizing or extracting frequency components different from each other in the first image 41 acquired in step S1. The first frequency-processed image 42 mainly includes low-frequency components.

[0048] In step S3, the calculation processing unit 13 performs different calculation processing on each of the first frequency processed image 42 and the second frequency processed image 43 generated in step S2. For example, the calculation processing unit 13 adds a relatively small weighting coefficient w to the first frequency processed image 42. 1 (≥0) is performed as the first operation 53, and a relatively large weighting coefficient w is added to the second frequency processed image 43. 2 (>w 1 ) as the second operation processing 54. The operation processing unit 13 may also change the weighting coefficient w1 With w 2 , and generates a plurality of image pairs consisting of the first frequency-processed image 42A and the second frequency-processed image 43A after the calculation process.

[0049] In step S4, the synthesis processing unit 14 synthesizes the frequency components of the first frequency processed image 42A and the second frequency processed image 43A subjected to the computational processing in step S3, thereby generating at least one second image 44. When a plurality of image sets consisting of the first frequency processed image 42A and the second frequency processed image 43A subjected to the computational processing are generated, the synthesis processing unit 14 performs the synthesis processing 55 on each of the image sets. In this way, a plurality of second images 44 are generated.

[0050] In step S5, the image output unit 15 outputs the first image 41 acquired in step S1 and at least one second image 44 generated in step S4 as training data, which is used for machine learning of a mathematical model for performing a prescribed inference on the input image. The image output unit 15 may also output the first image 41 and the second image 44 after normalizing the brightness of each pixel.

[0051] As described above, the image generation device 10 involved in the embodiment of the disclosed technology acquires the first image 41, and generates a plurality of frequency-processed images by emphasizing or extracting different frequency components from the first image 41. The image generation device 10 performs different operation processing on each of the plurality of frequency-processed images, and generates at least one second image 44 by synthesizing the frequency components of each of the plurality of frequency-processed images that have been subjected to the operation processing. The image generation device 10 outputs the first image 41 and the second image 44 as training data, which is used for machine learning of a mathematical model that performs a predetermined inference on an input image.

[0052] In machine learning of mathematical models for image recognition, sometimes even using the usual data expansion method for the original image, such as increasing or decreasing brightness (pixel value), changing contrast, performing zooming, changing the mixing ratio of color channels of color images, rotating the original image, and reversing the original image, cannot make up for the lack of actual data. In particular, in radiographic images used for non-destructive inspection of industrial products, various information such as fluctuations accompanying thickness changes of the object (product) itself and shape features of the object (product) itself are captured as background along with defects (damage) of the inspection object. As a result, although the variation forms of images to be prepared as training data become huge, it is not easy to prepare a set of images that capture the diversity of background information.

[0053] According to the image generating device 10 according to the embodiment of the disclosed technology, the second image 44 is generated based on the first image 41 as the original image, and the first image 41 and the second image 44 are provided as training data, so that the training data can be expanded.

[0054] The second image 44 is an image obtained by performing different operation processing on each of the first frequency-processed image 42 and the second frequency-processed image 43, and synthesizing these frequency components. For example, the operation processing can be varied by changing the combination of weighting coefficients used for the operation processing, and a plurality of second images 44 can be generated. Thus, further data expansion can be performed, and the training data can be diversified. In this way, the image generation device 10 involved in the embodiment of the disclosed technology can expand the variation of the image set used as training data in machine learning.

[0055] In addition, since the second image 44 is an image obtained by synthesizing the frequency components of the first frequency processed image 42 and the second frequency processed image 43, the appearance of the image of the defect (damage) existing in the high frequency region and the image of the background existing in the low frequency region can be adjusted independently. For example, by adding a relatively small weighting coefficient w to the first frequency processed image 42 mainly including low frequency components, 1 A relatively large weighting coefficient w is added to the second frequency processed image 43 mainly including high frequency components. 2 By performing arithmetic processing on the frequency components of these images, a second image 44 in which the image of the defect (damage) existing in the high-frequency region is further emphasized can be obtained. 1 When the second frequency processed image 43 is set to zero, the second frequency processed image 43 becomes the second image 44, and the image of the defect (damage) is more clearly depicted in the second image 44. In the machine learning of the defect detection model that detects the defect (damage) included in the input image, by using the image in which the defect (damage) to be detected is emphasized as training data, the defect detection model can be taught the characteristics of the defect (damage) image more clearly. As a result, the accuracy of defect detection in the defect detection model can be improved.

[0056] In addition, in the present embodiment, the case where the frequency processing unit 12 generates two frequency-processed images (the first frequency-processed image 42 and the second frequency-processed image 43) is illustrated, but the disclosed technology is not limited thereto. The frequency processing unit 12 may also generate three or more frequency-processed images by emphasizing or extracting different frequency components from the first image 41. Thus, the variation of the operation processing in the operation processing unit 13 can be further increased, and a more diverse second image 44 can be generated.

[0057] [Learning device]

[0058] Figure 6 2 is a diagram showing an example of the hardware structure of the learning device 20. The hardware structure of the learning device 20 is the same as the hardware structure of the image generating device 10. The learning device 20 includes a CPU 201, a RAM 202, a nonvolatile memory 203, an input device 204, a display 205, and a network interface 206. These hardware are connected to a bus 207. The CPU 201 is an example of a "second processor" in the disclosed technology.

[0059] The non-volatile memory 203 stores a learning program 210, a defect detection model 220, and training data 60. The defect detection model 220 is a mathematical model based on CNN that is constructed to predict a class label in units of pixels for an input image. The defect detection model 220 may be, for example, a model that applies a well-known encoder-decoder model (ED-CNN: Encoder-Decoder Convolutional Neural Network). The encoder-decoder model is a model composed of an encoder that extracts features from an image through a convolutional layer and a decoder that outputs a probability map based on the extracted features. The probability map is the result obtained by deriving the probability that each pixel of the input image belongs to a certain category for each pixel. The defect detection model 220 is constructed by machine learning using multiple radiographic images with correct labels as training data. The training data 60 is a data set consisting of a first image 41, a second image 44, and correct labels 45 and 46 assigned to these images supplied from the image generation device 10 as a unit. A plurality of training data 60 are stored in the non-volatile memory 203. The training data 60 is used for machine learning of the defect detection model 220 .

[0060] Figure 7 2 is a functional block diagram showing an example of the functional structure of the learning device 20. The learning device 20 functions as a training data acquisition unit 21 and a learning unit 22 by executing a learning program 210 by the CPU 201. The training data acquisition unit 21 acquires the training data 60 stored in the non-volatile memory 203. The learning unit 22 uses the training data 60 acquired by the training data acquisition unit 21 to train the defect detection model 220, for example, using the error back propagation method.

[0061] In addition, the training data acquisition unit 21 or the learning unit 22 of the learning device 20 can replace the image output unit 15 of the image generation device 10 to perform normalization processing on the first image 41 and the second image 44 as the training data 60 to normalize the brightness of each pixel. By performing the normalization processing, it is possible to prevent the distribution of brightness from being significantly deviated between the first image 41 and the second image 44. In the case where the distribution of brightness of the first image 41 and the second image 44 is significantly deviated, the defect detection model 220 unintentionally distinguishes the first image 41 and the second image 42, and may not be able to extract the features of the defects (damages) that exist in common in these images. By performing the normalization processing on the first image 41 and the second image 44, it is possible to suppress the defect detection model 220 from unintentionally distinguishing the first image 41 and the second image 44.

[0062] As described above, the learning device 20 involved in the embodiment of the disclosed technology uses the first image 41 and the second image 44 generated by the image generating device 10 as training data to train the defect detection model 220. According to the learning device 20 involved in this embodiment, the defect detection model 220 is learned using the training data 60 expanded by the second image 44, so that the accuracy of defect detection in the defect detection model 220 can be improved.

[0063] In addition, each process for generating the second image 44 performed in the image generating device 10 may be performed in the learning device 20. In addition, each process for generating the second image 44 may be performed in the input layer of the defect detection model 220.

[0064] [Image processing device]

[0065] Figure 8 3 is a diagram showing an example of the hardware structure of the image processing device 30. The hardware structure of the image processing device 30 is the same as the hardware structure of the image generating device 10. The image processing device 30 includes a CPU 301, a RAM 302, a non-volatile memory 303, an input device 304, a display 305, and a network interface 306. These hardware are connected to a bus 307. The CPU 301 is an example of a "third processor" in the disclosed technology. A defect detection program 310 and a learned defect detection model 320 are stored in the non-volatile memory 303. The defect detection model 320 is a mathematical model learned by the learning device 20.

[0066] Fig. 9 3 is a functional block diagram showing an example of the functional configuration of the image processing device 30. The image processing device 30 functions as an input image acquisition unit 31, a frequency processing unit 32, an inference result acquisition unit 33, and a detection result output unit 34 when the CPU 301 executes a defect detection program 310. Fig.10FIG. 1 is a diagram showing an example of the flow of processing in the image processing device 30. Fig.10 The functions of the above-described functional units constituting the image processing device 30 will be described.

[0067] The input image acquisition unit 31 acquires an input image 71 input as a processing target of the image processing device 30. The input image 71 is a radiographic image similar to the first image 41. The input image 71 may be an image obtained by performing pre-processing other than frequency processing such as enlargement, reduction, rotation, and grayscale processing on the original image. The input image acquisition unit 31 may perform such pre-processing.

[0068] The frequency processing unit 32 generates a frequency-processed image 72 by performing a frequency processing 82 on the input image 71. The frequency processing 82 may be, for example, a process of extracting high-frequency components from the input image 71 or a process of emphasizing high-frequency components of the input image 71. The frequency processing 82 may be a process of extracting or emphasizing the same frequency components as the frequency components extracted or emphasized in the second frequency processing 52 performed in the image generating device 10, or a process of extracting or emphasizing different frequency components. The frequency processing unit 32 may also generate a plurality of frequency-processed images 72 by emphasizing or extracting frequency components that are different from each other with respect to the input image 71.

[0069] The inference result acquisition unit 33 acquires the first inference result 73 by inputting the input image 71 into the learned defect detection model 320. The inference result acquisition unit 33 acquires the second inference result 74 by further inputting the frequency-processed image 72 into the learned defect detection model 320. When a plurality of frequency-processed images 72 are generated, the inference result acquisition unit 33 acquires a plurality of second inference results 74 corresponding to each of the plurality of frequency-processed images 72. The first inference result 73 and the second inference result 74 may be, for example, a probability map, which is a result obtained by deriving the probability that the pixel is a pixel representing a defect (damage) for each pixel of the image.

[0070] The detection result output unit 34 outputs the detection result 75 of the image of the defect (damage) for the input image 71 by comprehensively evaluating the first inference result 73 and the second inference result 74. For example, the detection result output unit 34 can generate a comprehensive probability map by calculating the average value for the probability map as the first inference result 73 and the probability map as the second inference result 74. In addition, the detection result output unit 34 can also generate a comprehensive probability map by calculating the weighted average value for the probability map as the first inference result 73 and the probability map as the second inference result 74. In addition, the detection result output unit 34 can also generate a comprehensive probability map obtained by applying the maximum value to the probability map as the first inference result 73 and the probability map as the second inference result 74. The detection result output unit 34 can also derive the detection result 75 by, for example, performing threshold processing on the comprehensive probability map.

[0071] Fig.11 3 is a flowchart showing an example of the flow of defect detection processing performed by the CPU 201 by executing the defect detection program 310. The defect detection program 310 is executed when the user instructs to start the processing by operating the input device 304, for example.

[0072] In step S31, the input image acquisition unit 31 acquires an input image 71 input as a processing target of the image processing device 30. The input image 71 is the same radiographic image as the first image 41. In step S32, the frequency processing unit 32 generates a frequency-processed image 72 by emphasizing or extracting a specific frequency component of the input image 71 acquired in step S31.

[0073] In step S33, the inference result acquisition unit 33 acquires a first inference result 73 by inputting the input image 71 acquired in step S31 into the learned defect detection model 320. In step S34, the inference result acquisition unit 33 acquires a second inference result 74 by inputting the frequency processed image generated in step S32 into the learned defect detection model 320. In step S35, the detection result output unit 34 outputs a detection result 75 of an image of a defect (damage) for the input image 71 acquired in step S31 by comprehensively evaluating the first inference result 73 acquired in step S33 and the second inference result 74 acquired in step S34.

[0074] As described above, the image processing device 30 according to the embodiment of the disclosed technology uses the defect detection model 320 learned by the learning device 20 to output the detection result of the image of the defect (damage) for the input image 71. According to the image processing device 30 according to this embodiment, the defect (damage) is detected using the defect detection model 220 learned using the training data 60 expanded from the second image 44, so that the accuracy of defect detection can be improved.

[0075] In addition, since the detection result 75 is outputted by comprehensively evaluating the first inference result 73 and the second inference result 74, the balance between the detection accuracy and the detection sensitivity can be properly achieved. In addition, the detection result 75 of the image of the defect (damage) may be outputted based on only one of the first inference result 73 and the second inference result 74. When only the first inference result 73 is used, the frequency processing unit 32 and the frequency processed image 72 are not required.

[0076] In the above description, the image generating device 10, the learning device 20, and the image processing device 30 are illustrated as being composed of different computers, but the image generating device 10, the learning device 20, and the image processing device 30 may also be composed of one or two computers. For example, the image generating device 10 and the learning device 20 may be composed of the same computer. In addition, the learning device 20 and the image processing device 30 may be composed of the same computer. In addition, the image generating device 10, the learning device 20, and the image processing device 30 may also be composed of the same computer.

[0077] In addition, in the above description, the case where the image processed in the image processing system 1 is a radiographic image obtained in the non-destructive inspection of industrial products is illustrated, but the disclosed technology is not limited to this. The disclosed technology can be applied to all images that include a specific structural part of an object in a specific frequency region (for example, optical images, MRI (Magnetic Resonance Imaging) images, SEM (Scanning Electron Microscope) images, ultrasonic images, etc.). In addition, in the above description, the case where the mathematical model of the learning object is a defect detection model for detecting defects (damage) included in the radiographic image is illustrated, but the disclosed technology is not limited to this. The disclosed technology can be applied to a mathematical model for detecting or identifying a specific structural part of an object in a specific frequency region in an image.

[0078] In the above-mentioned embodiment, for example, as a hardware structure of a processing unit (processing unit) that performs various processes, such as the first image acquisition unit 11, the frequency processing unit 12, the calculation processing unit 13, the synthesis processing unit 14, the image output unit 15, the training data acquisition unit 21, the learning unit 22, the input image acquisition unit 31, the frequency processing unit 32, the inference result acquisition unit 33, and the detection result output unit 34, the following various processors can be used. Among the above-mentioned various processors, in addition to the general-purpose processors such as CPUs and GPUs that execute software (programs) to function as various processing units as described above, there are processors such as FPGAs whose circuit structures can be changed after manufacturing, such as programmable logic devices (PLDs), ASICs (Application Specific Integrated Circuits), and other processors having circuit structures specially designed to perform specific processes, such as dedicated circuits.

[0079] One processing unit may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, a combination of a CPU and an FPGA). In addition, multiple processing units may be composed of one processor.

[0080] As an example of a processor constituting multiple processing units, first, there is a method: represented by computers such as clients and servers, a processor is composed of a combination of one or more CPUs and software, and the processor functions as multiple processing units. Second, there is a method: represented by a system on chip (SoC), etc., a processor that uses an IC (Integrated Circuit) chip to implement the functions of the entire system including multiple processing units. In this way, various processing units are constituted as hardware structures using one or more of the above-mentioned various processors. In addition, as the hardware structure of these various processors, more specifically, a circuit (circuitry) composed of a combination of circuit elements such as semiconductor elements can be used.

[0081] In addition, in the above-mentioned embodiment, the method of pre-storing (installing) various programs in a non-volatile memory is described, but it is not limited to this. Various programs can be provided in the form of recording on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), and a USB (Universal Serial Bus) memory. In addition, various programs can also be downloaded from an external device via a network. That is, the program (i.e., program product) described in this embodiment can be provided by a recording medium or transmitted by an external computer.

[0082] Regarding the above embodiment, the following supplementary notes are further disclosed.

[0083] (Note 1)

[0084] An image generating device comprises at least one first processor, wherein:

[0085] The first processor

[0086] Get the first image,

[0087] By emphasizing or extracting frequency components different from each other in the first image to generate a plurality of frequency processed images,

[0088] performing different arithmetic processing on each of the plurality of frequency processed images,

[0089] At least one second image is generated by synthesizing the frequency components of each of the plurality of frequency processed images subjected to the calculation process,

[0090] The first image and the second image are output as training data, and the training data is used for machine learning of a mathematical model that performs a predetermined inference on an input image.

[0091] (Note 2)

[0092] The image generating device according to Supplement 1, wherein:

[0093] The first processor performs a process of adding mutually different weighting coefficients to the plurality of frequency processed images as the calculation process.

[0094] (Note 3)

[0095] An image generating device according to Supplement 1 or 2, wherein:

[0096] The first processor

[0097] generating a first frequency-processed image including relatively low frequency components and a second frequency-processed image including relatively high frequency components,

[0098] As the calculation processing, a process of adding a relatively small weighting coefficient to the first frequency processed image and a relatively large weighting coefficient to the second frequency processed image is performed.

[0099] (Note 4)

[0100] An image generating device according to any one of Supplementary Notes 1 to 3, wherein:

[0101] The first processor performs, as the arithmetic processing, a processing of amplifying a difference between a luminance value of each pixel and an average value at different magnifications for each of the plurality of frequency processed images.

[0102] (Note 5)

[0103] An image generating device according to any one of Supplementary Notes 1 to 4, wherein:

[0104] The first processor

[0105] generating a first frequency-processed image by performing a filtering process on the first image,

[0106] A second frequency-processed image is generated by subtracting the frequency components of the first frequency-processed image from the first image.

[0107] (Note 6)

[0108] An image generating device according to any one of Supplementary Notes 1 to 5, wherein:

[0109] The first processor

[0110] A normalization process is performed on the first image and the second image to normalize brightness.

[0111] (Note 7)

[0112] An image generating device according to any one of Supplementary Notes 1 to 6, wherein:

[0113] The first image is a radiographic image including an image of a specific structural portion of the object in a specific frequency region.

[0114] The mathematical model is a model for detecting the image of the specific structural portion included in the radiation image.

[0115] (Note 8)

[0116] A learning device having at least one second processor, wherein:

[0117] The second processor uses the first image and the second image provided by the image generating device described in any one of Supplementary Notes 1 to 7 as training data to train the mathematical model.

[0118] (Note 9)

[0119] The learning device according to Supplementary Note 8, wherein:

[0120] The second processor

[0121] A normalization process is performed on the first image and the second image to normalize brightness.

[0122] (Note 10)

[0123] An image processing device having at least one third processor, wherein:

[0124] The third processor outputs a detection result of an image of a specific structural portion of an object with respect to an input image, using the mathematical model learned by the learning device according to Supplementary Note 8 or Supplementary Note 9.

[0125] (Note 11)

[0126] An image processing device according to Supplementary Note 10, wherein:

[0127] The third processor

[0128] Get the input image,

[0129] A frequency-processed image is generated by emphasizing or extracting a specific frequency component of the input image,

[0130] Obtaining a first inference result by inputting the input image into the mathematical model,

[0131] Obtaining a second inference result by inputting the frequency processed image into the mathematical model,

[0132] The detection result is output by comprehensively evaluating the first inference result and the second inference result.

[0133] (Note 12)

[0134] An image generation method, wherein at least one first processor of an image generation device performs the following processing:

[0135] acquiring a first image;

[0136] generating a plurality of frequency-processed images by emphasizing or extracting frequency components different from each other in the first image;

[0137] performing a different arithmetic operation on each of the plurality of frequency-processed images;

[0138] generating at least one second image by synthesizing the frequency components of the plurality of frequency processed images subjected to the calculation processing; and

[0139] The first image and the second image are output as training data, and the training data is used for machine learning of a mathematical model that performs a predetermined inference on an input image.

[0140] (Note 13)

[0141] A learning method, wherein at least one second processor of a learning device performs the following processing:

[0142] The first image and the second image provided by the image generation method described in Supplement 12 are used as training data to train the mathematical model.

[0143] (Note 14)

[0144] An image processing method, wherein at least one third processor of an image processing device performs the following processing:

[0145] Using the mathematical model learned by the learning method described in Supplementary Note 13, a detection result of an image of a specific structural portion of an object is output for an input image.

[0146] In addition, the contents disclosed in Japanese Patent Application No. 2022-146397 filed on September 14, 2022 are incorporated herein by reference in their entirety. In addition, all documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each document, patent application, and technical standard were specifically and individually described as being incorporated by reference.

Claims

1. An image generating device having at least one first processor, wherein: The first processor Get the first image, By emphasizing or extracting frequency components different from each other in the first image to generate a plurality of frequency processed images, performing different arithmetic processing on each of the plurality of frequency processed images, At least one second image is generated by synthesizing the frequency components of each of the plurality of frequency processed images subjected to the calculation process, The first image and the second image are output as training data, and the training data is used for machine learning of a mathematical model that performs a predetermined inference on an input image.

2. The image generating device according to claim 1, wherein: The first processor performs a process of adding mutually different weighting coefficients to the plurality of frequency processed images as the calculation process.

3. The image generating device according to claim 2, wherein: The first processor generating a first frequency-processed image including relatively low frequency components and a second frequency-processed image including relatively high frequency components, As the calculation processing, a process of adding a relatively small weighting coefficient to the first frequency processed image and a relatively large weighting coefficient to the second frequency processed image is performed.

4. The image generating device according to claim 1, wherein: The first processor performs, as the arithmetic processing, a processing of amplifying a difference between a luminance value of each pixel and an average value at different magnifications for each of the plurality of frequency processed images.

5. The image generating device according to claim 1, wherein: The first processor generating a first frequency-processed image by performing a filtering process on the first image, A second frequency-processed image is generated by subtracting the frequency components of the first frequency-processed image from the first image.

6. The image generating device according to claim 1, wherein: The first processor A normalization process is performed on the first image and the second image to normalize brightness.

7. The image generating device according to claim 1, wherein: The first image is a radiographic image including an image of a specific structural portion of the object in a specific frequency region. The mathematical model is a model for detecting the image of the specific structural portion included in the radiation image.

8. A learning device having at least one second processor, wherein: The second processor uses the first image and the second image provided by the image generating device according to any one of claims 1 to 7 as training data to learn the mathematical model.

9. The learning device according to claim 8, wherein: The second processor A normalization process is performed on the first image and the second image to normalize brightness.

10. An image processing device having at least one third processor, wherein: The third processor uses the mathematical model learned by the learning device according to claim 8 to output a detection result of an image of a specific structural portion of an object with respect to an input image.

11. The image processing apparatus according to claim 10, wherein: The third processor Get the input image, A frequency-processed image is generated by emphasizing or extracting a specific frequency component of the input image, Obtaining a first inference result by inputting the input image into the mathematical model, Obtaining a second inference result by inputting the frequency processed image into the mathematical model, The detection result is output by comprehensively evaluating the first inference result and the second inference result.

12. A method for generating an image, wherein: The at least one first processor of the image generating device performs the following processing: acquiring a first image; generating a plurality of frequency-processed images by emphasizing or extracting frequency components different from each other in the first image; performing a different arithmetic operation on each of the plurality of frequency-processed images; generating at least one second image by synthesizing the frequency components of each of the plurality of frequency-processed images subjected to the arithmetic processing; as well as The first image and the second image are output as training data, and the training data is used for machine learning of a mathematical model that performs a predetermined inference on an input image.

13. A learning method, wherein: The at least one second processor of the learning device performs the following processing: The first image and the second image provided by the image generation method according to claim 12 are used as training data to cause the mathematical model to learn.

14. An image processing method, wherein: The at least one third processor of the image processing device performs the following processing: Using the mathematical model learned by the learning method according to claim 13, a detection result of an image of a specific structural portion of an object is output with respect to an input image.

Images