Image processing system and image processing program
The image processing system addresses the challenges of insufficient defect images and uneven defect distribution by optimizing a detector using a discrete optimization algorithm, improving classification accuracy through a point-of-interest image approach.
Patent Information
- Application Number
- JP2020201815
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-12-04
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2040-12-04
AI Technical Summary
Conventional image classification using neural networks faces challenges with insufficient numbers of defect images, uneven distribution of defect locations, varied types of non-defective products, and small defects, leading to reduced detection accuracy.
An image processing system utilizing a discrete optimization algorithm to optimize a detector for identifying points of interest, creating a point-of-interest image with the defect at the center, and training a classifier using a neural network with position, shape, and brightness information.
Improves the accuracy of image classification by efficiently reducing the number of non-defective images and stabilizing the arrangement of points of interest, enhancing the learning process.
Smart Images

Figure 0007581811000001 
Figure 0007581811000002 
Figure 0007581811000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an image processing technique for classifying images into desired classes, and more particularly to an image processing system that performs training of a classifier using a neural network. [Background technology]
[0002] Conventionally, learning using neural networks requires a large number of images to be used for learning, and in classification problems in particular, classification performance has been achieved by preparing the same number of images for each class. However, in reality, while it is relatively easy to obtain a large number of normal images, it is difficult to prepare abnormal images. In addition, there is also the problem that detection accuracy deteriorates when the object to be detected and classified from the image is very small compared to the image.
[0003] Specifically, in image classification using a neural network, a captured image is divided, and the resulting divided images are used to train a classifier. However, in this case, as shown in Fig. 8, defects to be classified are present only in a part of the captured image, so there are far fewer images of defects compared to images of good products. In addition, as shown in Fig. 9, the locations of defects in the divided images are not uniform, so there is a problem that images of the same defect are different. There is also a problem that there are various types of good products, so it is difficult to classify them into one class as good products. Furthermore, there are cases where the defect is small relative to the image, and in such cases there is also a problem that the detection accuracy is poor.
[0004] To solve this problem, in the past, the number of samples was increased to learn various cases. Transfer learning was also used to compensate for the number of samples. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] Special Publication No. 2003-503801 [Patent Document 2] International Publication No. 2017 / 068675 Brochure Summary of the Invention [Problem to be solved by the invention]
[0006] However, it is difficult to sufficiently solve the above problems by using a method that increases the number of samples. In other words, there is a need to propose a method that can appropriately solve the problems caused by the fact that there are fewer defective images compared to non-defective product images, that the locations of defects are not uniform, that there are various types of non-defective products, and that defects are small relative to the images.
[0007] Therefore, the inventors conducted extensive research and succeeded in solving the above problems by using a discrete optimization algorithm to efficiently reduce the number of non-defective images at the stage of creating a learning data set using a neural network, and by stably positioning points of interest such as defects in the image in the image, thereby completing the present invention.
[0008] Here, Patent Document 1 proposes an image classification system that does not require deductive definition of rules. Also, Patent Document 2 proposes a program generation device capable of generating an image classification program that realizes highly accurate image classification processing. However, these inventions are not capable of solving the above problems.
[0009] The present invention has been made in consideration of the above circumstances, and has an object to provide an image processing system and an image processing program that are capable of efficiently reducing images of non-defective products by using a discrete optimization algorithm at the stage of creating a learning data set for a neural network-based classifier, and creating a focus point image in which a focus point in an image is stably positioned, thereby improving the accuracy of learning by the classifier. [Means for solving the problem]
[0010] In order to achieve the above-mentioned object, the image processing system of the present invention is an image processing system that classifies images into desired classes, and is configured to include a detector optimization unit that optimizes a detector that detects points of interest in the image using a discrete optimization algorithm, a classification learning data creation unit that creates a point of interest image with the point of interest detected by the obtained optimized detector centered, and a classifier learning unit that uses the point of interest image to train a classifier using a neural network.
[0011] In addition, it is preferable that the image processing system of the present invention is configured such that the classifier learning section learns the classifier using the image of the point of interest and a detection result image created by the optimized detector.
[0012] It is also preferable that the image processing system of the present invention is configured such that the classifier learning unit further learns the classifier using at least one of position information, shape information, and luminance information of the point of interest as input data for the neural network.
[0013] It is also preferable that the image processing system of the present invention is configured such that the detector optimization unit optimizes the detector using divided images obtained by dividing the image, and integrates the detection result images created by the optimized detector.
[0014] It is also preferable that the image processing system of the present invention is configured so that the neural network has a convolutional layer.
[0015] It is also preferable that the image processing system of the present invention is configured such that the discrete optimization algorithm is evolutionary computation that performs optimization using genetic operations, which is an optimization method that mathematically simulates the evolution of living organisms.
[0016] Furthermore, it is also preferable that the image processing system of the present invention is configured so that the detector optimization unit includes a population memory unit that stores individual information, which is a plurality of the detectors that define the execution order of a plurality of functional units in a linear structure, a tree structure, or a network shape; an image memory unit that stores a plurality of learning images; an image processing unit that processes the learning images based on the individual information and generates output information that is a detection result image corresponding to the learning images; a teacher information memory unit that stores teacher information corresponding to the learning images for evaluating the output information; an evaluation value calculation unit that compares the output information with the teacher information, calculates an evaluation value for each of the individual information and for each of the learning images, and ranks the individual information based on the evaluation value for each of the individual information; and a genetic manipulation unit that selects or changes the individual information based on the ranking information of the individual information, and updates the population memory unit.
[0017] It is also preferable that the image processing system of the present invention is configured such that the detector optimization unit includes an image division unit that divides the learning images stored in the image memory unit to create divided images, the image processing unit generates output information corresponding to the divided images, and the detector optimization unit further includes a result integration unit that integrates the generated output information for each learning image to create a detection result image.
[0018] In addition, the image processing program of the present invention is an image processing program for classifying images into desired classes, and is configured to cause a computer to function as a detector optimization unit that optimizes a detector that detects points of interest in the image using a discrete optimization algorithm, a classification learning data creation unit that creates a point of interest image in which the point of interest detected by the obtained optimized detector is centered, and a classifier learning unit that uses the point of interest image to train a classifier using a neural network. Effect of the Invention
[0019] According to the present invention, it is possible to provide an image processing system and an image processing program that can efficiently reduce non-defective images using a discrete optimization algorithm at the stage of creating a learning data set for a neural network-based classifier, and create a focus point image in which the focus point in the image is stably positioned, thereby improving the accuracy of learning by the classifier. [Brief description of the drawings]
[0020] [Figure 1] 1 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention. [Diagram 2] 11A and 11B are explanatory diagrams illustrating detection of a singular point by an optimized detector in the image processing device according to the embodiment of the present invention. [Diagram 3] 1 is an explanatory diagram illustrating the creation of a point-of-interest image by an image processing device according to an embodiment of the present invention. [Figure 4] 1 is a flowchart showing a processing procedure (up to generation of an optimized detector) performed by an image processing device according to an embodiment of the present invention. [Diagram 5] 10 is a flowchart showing a genetic manipulation processing procedure in the processing procedure by the image processing device according to the embodiment of the present invention. [Figure 6] 1 is a flowchart showing a processing procedure (from the process following the optimized detector generation process to classifier learning) performed by an image processing device according to an embodiment of the present invention. [Figure 7] 1 is a flowchart showing a processing procedure for creating a classifier learning dataset in a processing procedure by an image processing device according to an embodiment of the present invention. [Figure 8] FIG. 1 is an explanatory diagram showing image division and defective images in image classification using a conventional neural network. [Figure 9] 1 is an explanatory diagram showing the non-uniformity of locations where defects appear in segmented images and various non-defective products in image classification using a conventional neural network; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0021] Hereinafter, an embodiment of the image processing system and the image processing program of the present invention will be described in detail. However, the present invention is not limited to the following embodiment and the specific contents of the examples described later.
[0022] First, an image processing system and an image processing program according to an embodiment of the present invention will be described with reference to Figs. 1 to 3. Fig. 1 is a block diagram showing the configuration of an image processing device corresponding to the image processing system of this embodiment. Fig. 2 is an explanatory diagram of detection of a singular point by an optimized detector in the image processing device according to this embodiment. Fig. 3 is an explanatory diagram of creation of a point of interest image by the image processing device according to this embodiment.
[0023] The image processing system of this embodiment is an image processing system that classifies images into desired classes, and includes a detector optimization unit that optimizes a detector that detects points of interest in an image using a discrete optimization algorithm, a classification learning data creation unit that creates a point of interest image with the point of interest detected by the obtained optimized detector centered, and a classifier learning unit that uses the point of interest image to train a classifier using a neural network. As this discrete optimization algorithm, it is preferable to use evolutionary computation, which is an optimization method that mathematically simulates the evolution of living organisms, and performs optimization using genetic operations, but other algorithms may also be used as long as they can achieve similar results.
[0024] Specifically, as shown in FIG. 1, the image processing device 1 of this embodiment includes a functional unit memory unit 10, an individual generation unit 11, a population memory unit 12, an image input unit 13, an image memory unit 14, an image segmentation unit 15, a segmented image memory unit 16, a detection teacher information input unit 17, a detection teacher information memory unit 18, a segmented image teacher information creation unit 19, a segmented image teacher information memory unit 20, an image processing unit 21, an output information memory unit 22, a result integration unit 23, an evaluation value calculation unit 24, an evaluation result memory unit 25, a gene manipulation unit 27, an optimized detector memory unit 26, a classification teacher information input unit 28, a classification teacher information memory unit 29, a classification learning data creation unit 30, a classification learning data memory unit 31, a classifier learning unit 32, and a classification result memory unit 33. These components in the image processing device 1 of this embodiment can all be provided in one information processing device as shown in Fig. 1. Furthermore, these components may be provided in a distributed manner in each device of an image processing system consisting of a plurality of information processing devices.
[0025] Of these components in the image processing device 1 of this embodiment, the detector optimization unit A is composed of a functional unit memory unit 10, an individual generation unit 11, a population memory unit 12, an image input unit 13, an image memory unit 14, an image segmentation unit 15, a divided image memory unit 16, a detection teacher information input unit 17, a detection teacher information memory unit 18, a divided image teacher information creation unit 19, a divided image teacher information memory unit 20, an image processing unit 21, an output information memory unit 22, a result integration unit 23, an evaluation value calculation unit 24, an evaluation result memory unit 25, a genetic manipulation unit 27, and an optimized detector memory unit 26.
[0026] The functional unit storage unit 10 stores nodes (corresponding to genes), which are functional units that execute a certain process, for each node number. In this embodiment, various types of nodes for performing image processing can be used as the nodes. Examples of this image processing include an image processing algorithm for detecting a detection target from an image, an image noise removal algorithm, an image resolution enhancement algorithm, and an image synthesis algorithm. These algorithms are created by combining multiple nodes. Furthermore, if there are parameters used in the execution of each process, these nodes are stored in the functional unit storage section 10 together with the parameters.
[0027] Furthermore, a node that does not execute any process may be used, or a functional unit that executes multiple processes may be used as one node (node set). By using nodes that do not execute any process, the actual number of nodes can be reduced, for example, when the setting value of the initial number of nodes for generating individual information is too large.
[0028] In this specification, "image processing" includes various filter processes for noise removal, correction processes such as brightness correction, binarization processing, edge detection processing, frequency filter processing, arithmetic processing such as arithmetic operations, morphology processing, detection target determination processing, branching processing, and the like.
[0029] The individual generation unit 11 generates individual information consisting of an array of a plurality of nodes, using the nodes stored in the functional unit storage unit 10. At this time, the individual generation unit 11 can randomly generate a plurality of pieces of individual information based on the plurality of nodes. This individual information constitutes an image processing algorithm. Also, the detector and the optimized detector in this embodiment correspond to this individual information, and constitute the image processing algorithm. The individual generation unit 11 can define the execution order of multiple nodes in the form of a linear structure, a tree structure, or a network in the individual information. The network is a directed network composed of directional links and can include feedback.
[0030] In addition, using network-like individual information allows learning to be performed including a network structure with more flexible expressiveness compared to using individual information consisting only of a linear structure or a tree structure, and is therefore advantageous in deriving optimal solutions (practical optimal solutions, practical solutions). In other words, since humans cannot know in advance which structure will be the optimal solution for a problem, it is difficult to determine the conditions for the structure before machine learning. For this reason, using individual information that defines the execution order of multiple nodes in a network that may include a closed circuit, and searching for optimal solutions under a wide range of conditions, is considered to be effective in increasing the possibility of deriving a true optimal solution.
[0031] The population storage unit 12 stores a population consisting of a plurality of pieces of individual information generated by the individual generation unit 11. As this population, one generation or a plurality of generations can be stored in the population storage unit 12, and the individual information can be stored in the population storage unit 12 for each generation number. The evaluation value calculation unit 24, which will be described later, can rank the individual information for each generation or for each set generation, and store the ranking information in the evaluation result storage unit 25.
[0032] The image input unit 13 inputs a plurality of learning images and a plurality of verification images to the image processing device 1, and stores them in the image storage unit 14 for each piece of image identification information. The training images can be used to process all individual information (image processing algorithms) to obtain output information, to compare the output information with teaching information to calculate an evaluation value, and to use the evaluation value to rank individuals. The verification image can be used to process the image using the best individual information among the specified individual information to obtain output information, to compare the output information with the teacher information to calculate an evaluation value, and to use the evaluation value to verify the validity of the image processing algorithm and whether overlearning has occurred.
[0033] Since the verification images are used by humans to analyze the results, they do not generally affect optimization using the discrete optimization algorithm. However, if the evaluation value of the verification image is used to determine termination, for example, if the process is terminated when the evaluation value of the verification image reaches a threshold value, or if the process is terminated early when a large difference occurs between the evaluation values of the learning image and the verification image, then the optimization using the discrete optimization algorithm will automatically be affected.
[0034] The image division unit 15 divides each of the multiple learning images and multiple verification images stored in the image storage unit 14 into multiple divided images (hereinafter, may be simply referred to as images) of a set size, and stores them in the divided image storage unit 16 for each image identification information. FIG. 2 shows an input image and an image obtained by dividing it. If a point of interest such as a defect is shown in the input image, the point of interest is displayed in the divided image corresponding to that location. At this time, the area of one point of interest may be displayed separately in multiple divided images. Note that the term "point of interest" corresponds to a detection target in an image (a singular point detected by image processing), and does not mean a single point on a coordinate system.
[0035] Moreover, information indicating the area in which the point of interest exists is added to the detection result image corresponding to the divided image in which the point of interest exists (indicated by filling in white in FIG. 2). In the image processing system of this embodiment, such a detection result image is created by the image processing unit 21, which will be described later, making it possible to detect the point of interest.
[0036] It is also possible to omit the image division unit 15, the divided image memory unit 16, and the divided image teacher information creation unit 19, the divided image teacher information memory unit 20, and the result integration unit 23 described later from the image processing system of this embodiment, and perform image processing using multiple learning images and multiple verification images without dividing them.
[0037] The detection teacher information input unit 17 inputs teacher information for evaluating the output information, and stores the teacher information in the detection teacher information storage unit 18 for each piece of identification information of the corresponding image. As this teacher information, a teacher image corresponding to a plurality of learning images, a teacher image corresponding to a plurality of verification images, etc. This teacher image indicates the shape and position of the point of interest, and can be, for example, an image in which the point of interest is depicted in white on a black background, as in the detection result image in FIG. In the image processing system of this embodiment, the teaching information is not limited to the image alone. In addition to the image, the number and positions of detection targets in the image, or the range in which the detection targets exist, can also be used as teaching information.
[0038] The divided image teacher information creating unit 19 receives the detection teacher information from the detection teacher information storage unit 18 , creates teacher information corresponding to each divided image, and stores the information in the divided image teacher information storage unit 20 . In this case, the divided image teacher information creation unit 19 can create teacher information by dividing the teacher image into multiple images of a set size. In addition, if the teacher information for detection includes the number and positions of detection targets, or the range in which the detection targets exist, the teacher information can be created by calculating the number, positions, and range of detection targets for each divided image.
[0039] The image processing unit 21 inputs an image from the divided image storage unit 16 and also inputs individual information from the population storage unit 12, and sequentially executes a plurality of nodes included in the individual information based on the individual information. Note that the image processing unit 21 also executes nodes for executing the image input processing and the output image storage processing, respectively.
[0040] At this time, each node executes a process corresponding to each function using information obtained from the image. Also, each node may selectively execute the next node corresponding to the processing result based on the processing result. Therefore, even if the node is included in the individual information, not all nodes are necessarily executed in the image processing. Of course, there are cases where all nodes in the individual information are executed. Also, by performing feedback, a node that is defined only once in one individual information may be executed multiple times.
[0041] In addition, as described above, the functions of a node include various filter processes for noise removal, correction processes such as brightness correction, binarization processes, edge detection processes, frequency filter processes, arithmetic operations such as morphology processes, branching processes, and other narrow-sense image processing. Another function of the node is a detection target determination process, which is performed by the image processing unit 21 to label detection target candidates in a certain range in the image, calculate their feature amounts, and determine whether or not the detection target candidates having the feature amounts are correct detection targets based on thresholds or the like. In addition, in the detection target determination process, it is also possible to perform a process in which non-detection target candidates are labeled and their features are calculated, and if the candidate does not have that feature based on a threshold value or the like, it is determined that the candidate is not a detection target.
[0042] Furthermore, the image processing unit 21 can create an image (detection result image) as output information for each learning image and each verification image, and store the image for each individual information (identification information) and for each image (identification information) in the output information storage unit 22. When the teacher information includes the number of detection targets in the image, positions, or range in which the detection targets exist, the number of detection targets, positions, or range in which the detection targets exist are also created as output information and stored in the output information storage unit 22.
[0043] For each of the learning image and the verification image, if a detection target is detected, the image processing unit 21 creates a detection result image to which information indicating the area in which the detection target is present is added, and if the detection target is not detected, it creates a detection result image without the information added. This output information can then be stored in the output information storage unit 22 for each individual piece of information and for each image.
[0044] The function of creating a detection result image can also be performed by the evaluation value calculation unit 24. That is, after image processing is performed by the image processing unit 21, the evaluation value calculation unit 24 can generate a detection result image and store it in the output information storage unit 22, and the evaluation value can be calculated using this.
[0045] The result integration unit 23 integrates the detection result images as output information stored in the output information storage unit 22 for each learning image and verification image, creates a detection result image of the same size as the image before division, and stores it in the output information storage unit 22. This makes it possible to create a focus point image with the defect located at the center, as described below, even if one defect area is divided into multiple divided images.
[0046] The evaluation value calculation unit 24 inputs the output information (detection result image for each divided image) from the output information storage unit 22, and also inputs the teacher information corresponding to the output information from the divided image teacher information storage unit 20. Then, the output information and the teacher information are compared to calculate an evaluation value, and the obtained evaluation value can be stored in the evaluation result storage unit 25 for each piece of individual information and for each image.
[0047] The evaluation value calculation unit 24 can calculate the evaluation value using, for example, the mean square error. In addition, when the evaluation value calculation unit 24 has finished calculating the evaluation values for all the individual information (or the individual information for one generation) stored in the population memory unit 12 and stored them in the evaluation result memory unit 25, the evaluation value calculation unit 24 can rank the individual information based on those evaluation values and store the obtained ranking information in the evaluation result memory unit 25. Furthermore, the evaluation value calculation unit 24 can also rank the learning images for each piece of individual information based on the evaluation value, and store the obtained ranking information in the evaluation result storage unit 25.
[0048] Furthermore, the evaluation value calculation unit 24 can perform an end judgment to judge whether or not to end image processing by the image processing unit 21. The end judgment can be performed based on whether or not an end condition is satisfied. When the end condition is satisfied, the individual information with the highest evaluation value stored in the evaluation result storage unit 25 is obtained as individual information (optimized detector) optimized for detecting a point of interest (singular point) in an image, and is stored in the optimized detector storage unit 26.
[0049] The image processing unit 21 inputs the divided image from the divided image memory unit 18 and also inputs the optimized detector from the optimized detector memory unit 26, and sequentially executes multiple nodes contained in the individual information based on this detector, and creates images (detection result images) as output information and stores them in the output information memory unit 22.
[0050] The termination conditions for the termination judgment by the evaluation value calculation unit 24 may include a case where image processing is completed for populations of all preset generations, or a case where no evolution has occurred for a certain number of generations or more. A case where no evolution has occurred is a case where the evaluation value has not improved since the time of comparison. The function of this end determination may be separated from the evaluation value calculation unit 24, and an end determination unit for making the end determination may be provided in the image processing device 1 of this embodiment.
[0051] The gene manipulation unit 27 selects or modifies the individual information based on the ranking information of the individual information in the evaluation result storage unit 25 , and updates the population storage unit 12 . This makes it possible to sequentially add individual information of new generations of populations to the population storage unit 12.
[0052] Specifically, the gene manipulation unit 27 can have an elite individual preservation unit that passes on individual information with a high evaluation value to the next generation, an individual selection unit that passes on individual information to the next generation with a certain probability based on the evaluation value, a crossover unit that exchanges parts of two pieces of individual information with each other, and a mutation unit that randomly rewrites part or all of the selected individual information or node parameters.
[0053] The elite individual preservation unit can carry out a process of unconditionally passing on to the next generation information on the individual with the best evaluation value in the population of each generation. The individual selection unit can select, with a high probability, individual information with a high ranking in evaluation value from the population of individuals in each generation, and perform processing such as leaving the information to the next generation.
[0054] The crossover unit can perform one-way crossover, which swaps identical (node numbers) sequences in the sequences of each node of a plurality of pieces of individual information with a certain probability. The crossover unit can also perform one-point crossover, which swaps sequences with each other at one point of the identical sequence in the sequences of each node, or multiple-point crossover, which swaps sequences with each other at multiple points of the identical sequence. The crossover unit can perform such crossover processing on the individual information selected by the individual selection unit.
[0055] As a mutation process, the mutation unit can use the nodes stored in the functional unit storage unit 10 to rewrite part of the individual information with a certain probability for each piece of individual information, or rewrite all of the individual information. The mutation unit can perform such a mutation process on the individual information selected by the individual selection unit. Furthermore, as a mutation process, the mutation unit can also generate new individual information using the nodes stored in the functional unit storage unit 10. Furthermore, as a mutation process, the mutation unit can also perform parameter mutation by randomly rewriting some or all of the parameters of the nodes of the selected parent individual information.
[0056] Furthermore, if the number of individuals of the same series generated based on the same individual information increases, diversity will be lost and evolution will enter a dead end (a mathematically localized solution state). On the other hand, if the number of individuals outside the same series is increased too much, it will approach random search and efficient evolution will not occur. Therefore, in the early stages of learning, it is preferable to increase individuals other than the same series by using the mutation unit to provide diversity, and in the middle stages of learning, to increase the number of individuals of the same series by using the individual selection unit in order to perform efficient evolution. Also, when evolution stops, it is preferable to use the mutation unit to increase individuals other than the same series in order to provide diversity again.
[0057] Then, the genetic manipulation unit 27 generates a population of a new generation and stores it in the population storage unit 12. At this time, the genetic manipulation unit 27 can update the population storage unit 12 by overwriting the population of the previous generation. Also, the population storage unit 12 can be updated by storing populations of multiple generations for each generation.
[0058] The classification teacher information input unit 28 inputs teacher information for training the classifier and stores it in the classification teacher information storage unit 29. This teacher information is class information for classifying points of interest (singular points), and may be, for example, a non-defective product, dirt, scratches, dents, etc. This teacher information may be, for example, an image in which dirt, scratches, dents, etc. are displayed in white on a black background, and the respective classes (for example, 1, 2, 3, respectively), and an image of only a black background as a non-defective class (for example, 0).
[0059] The classification learning data creation unit 30 creates a learning data set for the classifier, and stores it in the classification learning data storage unit 31 . At this time, the classification learning data creation unit 30 inputs the integrated detection result image from the output information storage unit 22. In addition, the classification learning data creation unit 30 inputs the learning image and the verification image (input image) from the image storage unit 14.
[0060] Here, the classification learning data creation unit 30 can select and input only images in which the focus point exists as the detection result images. This makes it possible to create a learning data set for the classifier by excluding many good-quality images, such as good-quality images that do not have the focus point, thereby making it possible to efficiently reduce the number of good-quality images.
[0061] The classification learning data creation unit 30 uses these images to label points of interest (singular points detected by image processing) in the detection result image, and creates a point of interest image in which each point of interest is located at the center. Note that the "point of interest" corresponds to the detection target in the image (the classification target in classifier learning), and means the area of the singular points detected by image processing.
[0062] At this time, the classification learning data creation unit 30 can adjust the size of the point of interest by enlarging, reducing, or cutting the image of the point of interest, and can also adjust the shading of the point of interest image by performing shading correction. It is also preferable to create an image of the same size as the region corresponding to the focus image based on the detection result image, and include this image as the detection result image in the learning data set for the classifier.
[0063] As shown in FIG. 3, the classification learning data creation unit 30 creates cut-out images by cutting out points of interest from an input image, and uses these to create point-of-interest images in which each point of interest is arranged in the center. At this time, when the focus point is displayed separately in multiple divided images, the classification learning data creation unit 30 can create a focus point image in which the focus point is located at the center by referring to the learning image or verification image corresponding to the integrated detection result image.
[0064] Furthermore, the classification learning data creation unit 30 inputs teacher information for training the classifier from the classification teacher information storage unit 29, and sets the class of the point of interest for each image of the point of interest. Then, the classification learning data creation unit 30 creates a learning data set for the classifier including the created focus point image, detection result image, and class.
[0065] The classifier may be trained by including only the image of the point of interest and the class in the training data set for the classifier. The classifier may also be trained by including at least one of the position information, shape information, brightness information, etc. of the point of interest in the training data set for the classifier.
[0066] The classifier learning unit 32 performs learning by a neural network using the classifier learning data set stored in the classification learning data storage unit 31. As this neural network, it is preferable to use a convolutional neural network (CNN) having a convolution layer.
[0067] Then, the classifier learning unit 32 stores the obtained classification results and the trained classifier in the classification result storage unit 33. The classification results can include the probability of each class and the classification type (generally the class with the highest probability) for each image of interest. In addition, these can be combined for each learning image and verification image to include the number of detections of the classification type and the judgment result of whether the product is good or bad.
[0068] Next, a processing procedure performed by the image processing device according to this embodiment will be described with reference to FIGS. First, each process up to the generation process of an optimized detector in the processing procedure by the image processing device according to this embodiment and the genetic manipulation process will be described with reference to Figs. The image input unit 13 in the image processing device 1 inputs an image and stores it in the image storage unit 14 (step 10). These images can be a plurality of learning images and a plurality of verification images.
[0069] Furthermore, the detection teacher information input unit 17 inputs a teacher image as detection teacher information and stores it in the detection teacher information storage unit 18 (step 11). At this time, the detection teacher information input unit 17 can also use, together with the image, the number and positions of detection targets in the image, or the range in which the detection targets exist, as the detection teacher information.
[0070] Next, the following process is repeatedly executed for the number of learning images and verification images (step 12). That is, the image division unit 15 divides each of the multiple learning images and multiple verification images stored in the image memory unit 14 to create divided images, and stores them in the divided image memory unit 16 for each image identification information (step 13). In addition, the divided image teacher information creation unit 19 inputs the detection teacher information from the detection teacher information storage unit 18, creates teacher information corresponding to each divided image, and stores it in the divided image teacher information storage unit 20 (step 14).
[0071] Next, the individual generation unit 11 generates an initial population of individuals (step 15). Specifically, the individual generation section 11 can generate individual information by randomly selecting and arranging nodes from the functional unit storage section 10, and store the information in the individuals group storage section 12 for each piece of identification information. The order of steps 10 to 14 and step 15 may be interchanged.
[0072] At this time, the individual generation unit 11 can store an individual group consisting of a plurality of pieces of individual information as one generation in the individual group storage unit 12 for each generation number. The new generation population can be created by the genetic manipulation unit 27 performing genetic manipulation on the individual information stored in the population storage unit 12. That is, the new generation population can be created by the genetic manipulation unit 27 selecting or changing the individual information and updating the population storage unit 12.
[0073] In this way, generations in the population storage unit 12 can be added sequentially according to the execution of the generation loop shown in step 16. The generation loop is executed for a preset number of generations. Note that if a termination condition is met in the termination determination described below, the generation loop may terminate before the execution of the set number of generations.
[0074] Next, a population loop is executed for all the individual information of the population of the relevant generation in the population storage unit 12 (step 17). That is, the image processing unit 21 executes image processing for each of the divided images in the divided image storage unit 16, using the individual information of the individuals of the relevant generation in the individuals storage unit 12 (step 18). At this time, the image processing unit 21 can sequentially execute each of the multiple functional units that constitute the individual information on the divided image, create a detection result image or the like as output information corresponding to the divided image, and store it in the output information memory unit 22.
[0075] Next, the evaluation value calculation unit 24 compares the output information in the output information storage unit 22 with the teacher information in the divided image teacher information storage unit 20 for each individual information and each divided image of the population of the generation in the output information storage unit 22, and calculates an evaluation value (step 19). At this time, the evaluation value calculation unit 24 can use the output information and the teacher information to calculate the mean square error as the evaluation value by, for example, the least squares method.
[0076] Then, the evaluation value calculation unit 24 can store the obtained evaluation value in the evaluation result storage unit 25 for each piece of individual information and for each divided image. It is also possible to calculate and store a statistical value of the evaluation value for each piece of individual information by the evaluation value calculation unit 24. As the statistical value of the evaluation value, for example, the average value or the standard deviation can be used.
[0077] Next, the evaluation value calculation unit 24 ranks the individual information based on the evaluation values stored in the evaluation result storage unit 25, and stores the results in the evaluation result storage unit 25 (step 20). This ranking of the individual information can be performed based on a statistical value of the evaluation values, etc.
[0078] Moreover, the evaluation value calculation unit 24 performs a process of determining whether or not a termination condition is satisfied (step 21). If the termination condition is satisfied, the evaluation value calculation unit 24 stores the highest ranked individual information stored in the evaluation result storage unit 25 in the optimized detector storage unit 27 as an optimized detector (step 23).
[0079] If the termination condition is not satisfied, the gene manipulation unit 27 executes the gene manipulation process (step 22). This enables the gene manipulation unit 27 to update the population storage unit 12 by adding a new generation of population.
[0080] Specifically, as shown in FIG. 5, the elite individual preservation unit in the gene manipulation unit 27 selects the individual information with the highest evaluation value as the individual information of the next generation population (step 30). Moreover, the individual selection section in the gene manipulation section 27 selects individual information with a higher ranking of evaluation value as individual information for the next generation population with a higher probability (step 31).
[0081] Furthermore, the crossover section in the gene manipulation section 27 executes a crossover process on the individual information selected by the individual selection section to generate new individual information (step 32). Furthermore, the mutation section in the gene manipulation section 27 performs a mutation process or the like based on the individual information selected by the individual selection section, or without using this individual information, to generate new individual information (step 33). The order of each process by the elite individual preservation unit, the individual selection unit, the crossover unit, and the mutation unit may be changed within the range of an order in which each process can be executed.
[0082] This genetic manipulation process makes it possible to add individual information of the second generation population to the population storage unit 12. In this way, by repeating the generation loop, individual information of the population of new generations can be added sequentially to the population storage unit 12.
[0083] Next, with reference to FIG. 6 and FIG. 7, each process from the process following the optimized detector generation process to the classifier learning process in the processing procedure by the image processing device according to this embodiment and the classifier learning dataset creation process will be described. First, the following process is repeatedly executed for the number of learning images and verification images (step 40).
[0084] That is, the image division unit 15 divides each of the multiple learning images and multiple verification images stored in the image memory unit 14 to create divided images, and stores them in the divided image memory unit 16 for each image identification information (step 41). Furthermore, the divided image teacher information creation unit 19 inputs the detection teacher information from the detection teacher information storage unit 18, creates teacher information corresponding to each divided image, and stores it in the divided image teacher information storage unit 20 (step 42).
[0085] Here, the reason why steps 41 and 42 perform the same processing as steps 13 and 14 is that, taking into consideration the time required for the evolutionary computation learning performed in steps 10 to 23, an optimized detector is created using a smaller number of images compared to the number of images required for training a classifier using a neural network performed in step 48, and then a detection result image is created using a large number of images, making it possible to create a data set for training the classifier. Therefore, when performing evolutionary computation using a sufficient number of images for training a classifier, it is possible to omit steps 41 and 42 and perform subsequent processing using the segmented images and teaching information obtained in steps 13 and 14.
[0086] Next, the following process is repeatedly executed for the number of divided images (step 43). The image processing unit 21 inputs the divided image from the divided image storage unit 18 and the optimized detector from the optimized detector storage unit 26, and sequentially executes the multiple nodes included in the individual information based on this detector, creates images (detection result images) as output information, and stores them in the output information storage unit 22 (step 44). Then, the result integration unit 23 integrates the detection result images stored in the output information storage unit 22 for each learning image and verification image, creates an integrated detection result image, and stores it in the output information storage unit 22 (step 45).
[0087] Next, a process of creating a data set for classifier training is performed (step 46). 7, the classification learning data creation unit 30 inputs the integrated detection result images (only those having a point of interest) from the output information storage unit 22. The classification learning data creation unit 30 also inputs the learning image and the verification image from the image storage unit 14, and inputs the corresponding divided images from the divided image storage unit 16 (step 50).
[0088] Moreover, the classification learning data creation unit 30 inputs the teacher information for training the classifier from the classification teacher information storage unit 29 (step 51). Then, the following loop is executed for the number of detection result images (step 52). First, the classification learning data creation unit 30 labels points of interest (singular points detected by image processing) in the detection result image (step 53).
[0089] Next, the following process is repeated for the number of labels (step 54). That is, the classification learning data creation unit 30 calculates the center of gravity of the labeled point of interest (step 55). Then, a point of interest image is created in which the center of gravity of the point of interest is located at the center (step 56). At this time, the classification learning data creation unit 30 can perform processing to match the size of the point of interest by enlarging, reducing, or cutting the point of interest image, and can also perform processing to match the shade of the point of interest image by performing shade correction. It is also preferable to create an image of the same size as the area corresponding to the point of interest image based on the detection result image, and include this as the detection result image in the learning dataset for the classifier.
[0090] Furthermore, the classification learning data creation unit 30 sets a class of the point of interest for each image of the point of interest using the teacher information for training the classifier input from the classification teacher information storage unit 29 (step 57). Then, the classification learning data creation unit 30 creates a learning data set for the classifier including the created focus point image, detection result image, position information, shape information, brightness information, etc. of the focus point, and classes, and stores the data in the classification learning data storage unit 31 (step 58).
[0091] Next, the classifier learning unit 32 inputs the classifier learning data set from the classification learning data storage unit 31 (step 47) and performs learning using a neural network (step 48). As this neural network, it is preferable to use a convolutional neural network (CNN) having a convolutional layer. Then, the classifier learning unit 32 stores the obtained classification result and the trained classifier in the classification result storage unit 33.
[0092] The image processing device of the above embodiment can be realized by using a computer controlled by the image processing program of the present invention. The CPU of the computer sends commands to each component of the computer based on the image processing program, and causes them to perform predetermined processes required for the operation of the image processing device, such as individual generation process, image processing, evaluation value calculation process, gene manipulation process, classification learning data creation process, classifier learning process, etc. In this way, each process and operation in the image processing device of the present invention can be realized by specific means in which the program and the computer work together.
[0093] The program is stored in advance on a recording medium such as ROM or RAM, and is executed by having the computer read the program from the recording medium installed in the computer; however, the program can also be read into the computer via a communication line, for example. Furthermore, the recording medium for storing the program can be configured by any recording means readable by any computer, such as a semiconductor memory, a magnetic disk, an optical disk, or the like.
[0094] As described above, according to the image processing system and image processing program of the embodiment of the present invention, a discrete optimization algorithm is used at the stage of creating a learning data set for a neural network-based classifier to efficiently reduce the number of non-defective images, and a focus point image in which the focus point in the image is stably positioned is created, thereby making it possible to improve the accuracy of learning by the classifier. This also makes it possible to appropriately determine whether the inspection image is an image of a non-defective product, and if the image is an image of a defective product, what type of defective product it is.
[0095] The present invention is not limited to the above-described embodiment, and it goes without saying that various modifications can be made within the scope of the present invention. For example, it is possible to distribute each component of the image processing device among a plurality of information processing devices, and to use information other than the above-mentioned as the classification teacher information. [Industrial Applicability]
[0096] INDUSTRIAL APPLICABILITY The present invention can be suitably used to obtain an information processing device for image inspection that is excellent at classifying images when performing inspection based on image data of equipment, products, etc. [Explanation of symbols]
[0097] 1 Image processing device 10 Functional unit memory section 11 Individual Generation Section 12 Population Memory Unit 13 Image input section 14 Image storage section 15 Image division section 16 Divided image storage section 17. Detection teacher information input section 18 Detection teacher information storage unit 19. Segmented image teacher information creation unit 20 Segmented image teacher information storage unit 21 Image processing section 22 Output information storage unit 23 Results Integration Division 24 Evaluation value calculation unit 25 Evaluation result storage unit 26 Optimized detector memory 27 Genetic Engineering Department 28 Classification teacher information input section 29 Classification teacher information storage unit 30 Classification learning data creation unit 31 Classification learning data storage unit 32 Classifier training section 33 Classification result storage unit A Detector optimization section
Claims
1. 1. An image processing system for classifying an image into a desired class, comprising: The system includes a detector optimization unit that optimizes a detector for detecting points of interest in the image by a discrete optimization algorithm, a classification learning data creation unit that creates a point of interest image in which the points of interest detected by the optimized detector are positioned at the center, and a classifier learning unit that uses the point of interest image to train a classifier by a neural network.
1. An image processing system comprising:
2. 2. The image processing system according to claim 1, wherein the classifier learning unit performs learning of the classifier using the image of the point of interest and a detection result image created by the optimized detector.
3. 3. The image processing system according to claim 1, wherein the classifier learning unit further uses at least one of position information, shape information, and brightness information of the point of interest as input data for the neural network to learn the classifier.
4. The image processing system according to any one of claims 1 to 3, characterized in that the detector optimization unit optimizes the detector using divided images obtained by dividing the image, and integrates detection result images created by the optimized detector.
5. 5. The image processing system according to claim 1, wherein the neural network has a convolutional layer.
6. The image processing system according to any one of claims 1 to 5, characterized in that the discrete optimization algorithm is evolutionary computation that performs optimization using genetic operations, which is an optimization method that mathematically simulates the evolution of living organisms.
7. The detector optimization unit: A population storage unit that stores information on a plurality of detectors that define an execution sequence of a plurality of functional units in a linear structure, a tree structure, or a network structure; an image storage unit that stores a plurality of learning images; an image processing unit that processes the learning image based on the individual information and generates output information that is a detection result image corresponding to the learning image; a teacher information storage unit configured to store teacher information corresponding to the learning image for evaluating the output information; an evaluation value calculation unit that compares the output information with the teacher information, calculates an evaluation value for each of the individual information and for each of the learning images, and ranks the individual information based on the evaluation value for each of the individual information; A genetic manipulation unit that selects or changes the individual information based on the ranking information of the individual information, and updates the population storage unit.
7. The image processing system according to claim 1,
8. the detector optimization unit includes an image division unit that divides the learning image stored in the image storage unit to create a divided image, the image processing unit generates output information corresponding to the divided images; The detector optimization unit further includes a result integration unit that integrates the generated output information for each learning image to create a detection result image.
8. The image processing system according to claim 7.
9. An image processing program for classifying an image into a desired class, comprising: Computer, The system functions as a detector optimization unit that optimizes a detector for detecting points of interest in the image by a discrete optimization algorithm, a classification learning data creation unit that creates a point of interest image in which the points of interest detected by the optimized detector are positioned at the center, and a classifier learning unit that uses the point of interest image to learn a classifier by a neural network.
2. An image processing program comprising:
Citation Information
Patent Citations
Improved neural network-based power equipment image fault identification method
CN111310722A
Image classification using evolutionary parameters
JP2003503801A
Estimation unit generator, inspection device, method for generating estimation unit, and estimation generation program
JP2020060398A
Method and apparatus for video interpretation of carotid intima-media thickness
US20170238909A1
Gabor filtering for improved microcalcification detection in digital mammograms
US6137898A
