Method for creating a pre-trained model, image processing method, program, and image processing device.

Abstract

To provide a method for creating a learned model that can prevent a reduction in image recognition accuracy despite the small number of images used as learning data.SOLUTION: A method for creating a learned model includes: an image acquisition step of acquiring a plurality of candidate images to be candidates for learning data; a selection step of executing rule-based image recognition processing on each of the plurality of candidate images to select images used as the learning data; and a model creation step of causing an algorithm of machine learning to learn the images selected in the selection step to create a learned model.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a method for creating a learned model, an image processing method, a program, and an image processing apparatus.

Background Art

[0002] Conventionally, in various fields such as sorting of agricultural products, appearance inspection of products, inspection of structures, autonomous driving, work support, reading of characters and numbers, etc., image recognition technology using artificial intelligence (AI) has been utilized. Image processing technology includes a rule-based model in which judgment criteria are set by a user and a machine learning model that performs image recognition by AI.

[0003] Also, AI can be classified into cloud AI that performs learning and inference on the cloud and edge AI that incorporates a model learned on a terminal (edge) into the terminal (edge) and performs inference on the terminal side (see, for example, Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In recent years, an AI that performs learning on the terminal side in addition to inference has been proposed. However, when performing machine learning on AI on a terminal, it is difficult to use a large amount of images as learning data like cloud AI. On the other hand, when the number of images for machine learning on AI is small (for example, several tens), noise in the learned model increases, and there is a risk that the image recognition accuracy will decrease.

[0006] The present invention has been made in view of the above, and aims to provide a method and apparatus for creating a trained model, an image processing method, an image processing apparatus, and a program that can suppress a decrease in image recognition accuracy even when the number of images used as training data is small. [Means for solving the problem]

[0007] To solve the above problems, a method for creating a trained model according to one aspect of the present invention includes: an image acquisition step of acquiring a plurality of candidate images to be used as training data; a selection step of selecting an image to be used as training data by applying rule-based image recognition processing to each of the plurality of candidate images; and a model creation step of creating a trained model by having a machine learning algorithm learn the images selected in the selection step.

[0008] In the method for creating the trained model described above, the image acquisition step may involve acquiring 10 to 30 candidate images.

[0009] In the method for creating the trained model described above, the selection step may include: an alignment step of aligning a pre-selected or created reference image with the candidate image; an outlier determination step of determining outlier pixels as pixels whose brightness values ​​differ significantly from those of corresponding pixels in the candidate image; and a step of selecting the candidate image from which the outlier pixels have been removed as an image to be used as training data.

[0010] In the method for creating the trained model described above, the outlier pixel determination step may involve calculating the brightness difference between corresponding pixels in the reference image and the candidate image, and determining pixels in the candidate image whose brightness difference exceeds a reference range as outlier pixels.

[0011] The method for creating the trained model described above further includes a step of setting the reference range for the brightness difference to a range of 0% or more and 50% or less of the brightness value of the pixel in the reference image, and the outlier pixel determination step may determine as an outlier pixel any pixel whose absolute value of the brightness difference exceeds the set reference range.

[0012] The method for creating the trained model described above further includes a step of determining the distribution of brightness values ​​of corresponding pixels among the plurality of candidate images, and setting the reference range within the range of 0σ to 3σ in the distribution, wherein the outlier pixel determination step may determine pixels whose absolute value of brightness difference exceeds the set reference range as outlier pixels.

[0013] In the method for creating the trained model described above, the reference image may be selected from among the multiple candidate images.

[0014] In the method for creating the trained model described above, the reference image may be created by aligning the multiple candidate images, calculating the average or median of the brightness values ​​of corresponding pixels among the multiple candidate images, and using the average or median as the brightness value of each pixel.

[0015] In the method for creating the trained model described above, the selection step may include: an alignment step of aligning a pre-selected or created reference image with the candidate image; an outlier determination step of determining outlier pixels as pixels whose brightness values ​​differ significantly from those of corresponding pixels in the candidate image; and an exclusion step of excluding candidate images in which the proportion of outlier pixels exceeds a standard from the images to be used as training data. The candidate images that were not excluded may be selected as images to be used as training data.

[0016] The method for creating the trained model described above further includes a step of setting the reference ratio in the range of 0% to 50%, and the exclusion step may exclude candidate images in which the number of pixels determined to be outliers exceeds the set reference ratio to the total number of pixels in the candidate image.

[0017] In the method for creating the trained model described above, the reference image may be selected from among the multiple candidate images.

[0018] In the method for creating the trained model described above, the reference image may be created by aligning the multiple candidate images, calculating the average or median of the brightness values ​​of corresponding pixels among the multiple candidate images, and using the average or median as the brightness value of each pixel.

[0019] Another aspect of the present invention is an image processing method which includes: a second selection step of determining whether or not to include a target image in the judgment of a trained model by applying rule-based image recognition processing to the target image; and a determination step of performing image recognition on the target images selected as targets for judgment in the second selection step using a trained model created by the method for creating a trained model, wherein target images that were not selected as targets for judgment in the second selection step are treated as unsuitable images and image recognition is not performed using the trained model.

[0020] In the image processing method described above, the second selection step may involve setting an inspection area in the target image and performing selection based on the pixels within the inspection area.

[0021] In the image processing method described above, the second selection step may be performed by comparing the target image with a pre-selected or created reference image for image recognition.

[0022] In the above image processing method, the second sorting step may include: a step of setting inspection regions in the image recognition reference image and the target image; a matching rate calculation step of performing image matching of the inspection region in the target image with respect to the inspection region in the image recognition reference image, and calculating a matching rate for all or part of the inspection region; and a step of determining that the case where the matching rate is lower than a reference value is a non-conforming image.

[0023] The above image processing method may further include a step of setting the reference value of the matching rate within a range of 0.3 or more and 1.0 or less.

[0024] In the above image processing method, before the matching rate calculation step, a step of performing normalized correlation transformation on the luminance value of the inspection region in the target image so that the luminance value of the inspection region in the target image is equal to the luminance value of the inspection region in the image recognition reference image may be further included, and the matching rate calculation step may calculate the matching rate for the inspection region in the target image subjected to the normalized correlation transformation.

[0025] In the above image processing method, the image recognition reference image may be selected from among a plurality of images used as learning data when creating the learned model.

[0026] In the above image processing method, the image recognition reference image performs alignment among a plurality of images used as learning data when creating the learned model, calculates an average value or a median value of luminance values of corresponding pixels among the plurality of images, determines pixels with luminance values of corresponding pixels in each image deviating significantly from the average value or the median value as outlier pixels, and may be created by using the average value or the median value of luminance values of corresponding pixels among the plurality of images excluding the outlier pixels as the luminance value of each pixel.

[0027] Another aspect of the present invention is a program that causes a computer to execute an image acquisition step of acquiring a plurality of candidate images as candidate learning data, a selection step of selecting an image to be used as learning data by performing rule-based image recognition processing on each of the plurality of candidate images, and a model creation step of creating a learned model by causing the computer to learn the image selected in the selection step using a machine learning algorithm.

[0028] The above program may further cause the computer to execute a display step of displaying a location within the image that is the basis for the omission from the selection for candidate images that were not selected as images to be used as learning data in the selection step.

[0029] The above program further causes the computer to execute a second selection step of selecting whether or not to subject a target image to determination by a learned model by performing rule-based image recognition processing on the target image, and a determination step of performing image recognition on the target image selected as a determination target in the second selection step using the learned model. The target image not selected as a determination target in the second selection step may be regarded as an inappropriate image, and image recognition by the learned model may not be performed.

[0030] The above program may further cause the computer to execute a display step of displaying a location within the image that is the basis for the recognition as inappropriate for the inappropriate image.

[0031] The above program may further cause the computer to execute a guidance step of guiding imaging conditions according to the state of the location within the image.

[0032] Another aspect of the present invention is an image processing device comprising: an image acquisition unit that acquires a plurality of candidate images to be used as training data; a selection unit that selects an image to be used as training data by performing rule-based image recognition processing on each of the plurality of candidate images; and a model creation unit that creates a trained model by having a machine learning algorithm train the images selected by the selection unit.

[0033] The above-described image processing device further comprises: a second selection unit that performs rule-based image recognition processing on a target image to determine whether or not to include the target image in the training model's judgment; and a determination unit that performs image recognition using the training model on the target image selected as a target for judgment by the second selection unit. In this case, the second selection unit may also treat the target image that is not selected as a target for judgment by the second selection unit as an unsuitable image and not perform image recognition using the training model.

[0034] The above-mentioned image processing device may be implemented using edge computing for at least the image acquisition unit, the sorting unit, the model creation unit, the second sorting unit, and the determination unit.

[0035] In the above-described image processing apparatus, the image processing apparatus may be composed of a terminal device comprising at least a processor and memory, and the image acquisition unit may acquire the plurality of candidate images and the target image based on image data input from an imaging device connected to the image processing apparatus by wire or wireless. [Effects of the Invention]

[0036] According to the present invention, since candidate images selected by applying rule-based image recognition processing to multiple candidate images are used as training data for a machine learning algorithm, it is possible to suppress a decrease in the image recognition accuracy of the trained model even when the number of images used as training data is small. [Brief explanation of the drawing]

[0037] [Figure 1] This is a block diagram showing an example configuration of an image processing system including an image processing device according to an embodiment of the present invention. [Figure 2] This is a network diagram showing another example configuration of an image processing system including an image processing apparatus according to an embodiment of the present invention. [Figure 3] This is a flowchart showing the process for creating a learning model in an embodiment of the present invention. [Figure 4] This flowchart shows the process of selecting candidate images using rule-based image recognition. [Figure 5] This is a schematic diagram illustrating the candidate image selection process. [Figure 6] This flowchart shows a modified version of the process for selecting candidate images using rule-based image recognition. [Figure 7] This flowchart shows the image recognition process performed in the image processing method according to an embodiment of the present invention. [Figure 8] This flowchart shows the process of selecting target images using rule-based image recognition. [Figure 9] This is a schematic diagram illustrating the image selection process. [Modes for carrying out the invention]

[0038] Hereinafter, a method for creating a trained model, an image processing method, a program, and an image processing apparatus according to embodiments of the present invention will be described with reference to the drawings. Note that the present invention is not limited to these embodiments. Furthermore, in each drawing, the same parts are denoted by the same reference numerals.

[0039] The method for creating a pre-trained model and the image recognition using the created pre-trained model according to this embodiment can be applied to various fields, such as product appearance inspection, circuit board inspection, structural inspection, crop sorting, and optical character recognition (OCR).

[0040] (Configuration of the image processing device) Figure 1 is a block diagram showing the schematic configuration of an image processing system including an image processing device according to an embodiment of the present invention. As shown in Figure 1, the image processing system 1 includes an image processing device 10, an imaging unit 20, a display unit 30, and an operation input unit 40 connected to the image processing device 10 by wire or wireless.

[0041] The image processing device 10 is a terminal device consisting of a personal computer equipped with, for example, a computing unit such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), a storage device such as memory and storage, and a communication interface such as USB (Universal Serial Bus) and Ethernet. The image processing device 10 performs so-called edge computing, which involves creating a trained model, performing image recognition using this model, and performing various processing using the image recognition results, all on a single device.

[0042] The imaging unit 20 is an optical camera equipped with an image sensor, such as a CMOS image sensor. The imaging unit 20 generates image data by imaging an object and inputs it to the image processing device 10. The imaging unit 20 may be a so-called industrial or commercial camera, or a general digital camera. Alternatively, a mobile communication terminal such as a tablet or smartphone may be used as the imaging unit 20. In short, the configuration of the imaging unit 20 should be appropriately selected according to the application of the image recognition technology according to this embodiment (for example, the object to be inspected, the inspection location, etc.).

[0043] The display unit 30 is, for example, a liquid crystal display or an organic EL display, and displays information such as images and characters based on the image signal output from the image processing device 10.

[0044] The operation input unit 40 is an input device such as a keyboard, mouse, touch panel on the surface of the display unit 30, or various adjustment knobs, and accepts user operations and inputs signals corresponding to the operations to the processor 13, which will be described later.

[0045] In addition, the image processing system 1 may be equipped with an illumination device for illuminating the object to be imaged, and a position adjustment means (such as a movable stage) for adjusting the position of the object to be imaged.

[0046] Figure 2 is a network diagram showing another example configuration of an image processing system including an image processing device 10. The image processing system 2 shown in Figure 2 comprises an image processing device 10 and an imaging terminal 50.

[0047] As shown in Figure 1, the image processing device 10 may directly acquire image data from the imaging unit 20 connected to the image processing device 10, or as shown in Figure 2, it may acquire image data from the imaging terminal 50 via the communication network N. For example, when inspecting structures outdoors or when reading water meters using OCR, it is conceivable to send and receive image data from the imaging terminal 50, which has communication and imaging functions, via the communication network N. The communication network N may consist of, for example, an internet line, telephone line, LAN, dedicated line, mobile communication network, WiFi (Wireless Fidelity), Bluetooth (registered trademark), or a combination thereof, and may be wired, wireless, or a combination thereof.

[0048] As shown in Figure 1, the image processing device 10 includes a communication interface 11, a storage unit 12, and a processor 13. The communication interface 11 is an interface that connects the image processing device 10 to other devices directly or via a communication network N, and enables communication with other devices. The communication interface 11 can be configured using, for example, a soft modem, cable modem, wireless modem, ADSL modem, etc.

[0049] The storage unit 12 is configured using a computer-readable storage medium such as semiconductor memory like ROM or RAM or a hard disk, and stores operating system programs and driver programs, application programs that execute various functions, parameters used during the operation of these programs, and other information. In this embodiment, the storage unit 12 stores a program 121 that causes the processor 13 to create a trained model and perform image recognition using the trained model, image data 122 input from the imaging unit 20, training data 123 for training the trained model, various parameters 124 used in image recognition processing, and the created trained model 125.

[0050] The processor 13 is configured using, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and by reading the program 121 stored in the memory unit 12, it comprehensively controls each part of the image processing device 10, and also performs various calculations for creating a trained model and performing image recognition using the trained model. The functional units realized by the processor 13 include an imaging control unit 131, a display control unit 132, a setting unit 133, an image generation unit 134, a first rule-based image recognition unit 135, a trained model creation unit 136, a second rule-based image recognition unit 137, an AI judgment unit 138, and a guide unit 139.

[0051] The imaging control unit 131 controls imaging conditions such as imaging timing and frame rate in the imaging unit 20 according to signals input from the operation input unit 40. If the system 1 is equipped with an illumination device or position adjustment means for the object to be imaged, the illumination intensity and the position of the object to be imaged may also be controlled as imaging conditions.

[0052] The display control unit 132 controls the display of images and various information on the display unit 30. The setting unit 133 sets various parameters used in image recognition in the first rule-based image recognition unit 135 and the second rule-based image recognition unit 137. These parameters may also be set arbitrarily by the user using the operation input unit 40.

[0053] The image generation unit 134 generates images by performing image processing such as demosaicing, white balance processing, and gamma correction based on image data input from the imaging unit 20. The image generation unit 134 also functions as an image acquisition unit that acquires multiple candidate images to be used as training data, and an image acquisition unit that acquires target images for image recognition.

[0054] The first rule-based image recognition unit 135 functions as a selection unit that selects images to be used as training data by applying rule-based image recognition processing to each of a plurality of candidate images. Specifically, the first rule-based image recognition unit 135 includes an alignment unit 135a that aligns candidate images with a pre-selected or created reference image, and a selection unit 135b that selects candidate images to be used as training data by determining pixels in the reference image whose brightness values ​​differ significantly from those of corresponding pixels in the candidate image as outliers.

[0055] The learning model creation unit 136 creates a trained model 125 by training a machine learning algorithm with the images selected by the selection unit 135b. This trained model 125 is stored in the storage unit 12.

[0056] The second rule-based image recognition unit 137 functions as a second selection unit that selects images to be judged by a trained model by applying rule-based image recognition processing to the target images for image recognition. Specifically, the second rule-based image recognition unit 137 includes an inspection area setting unit 137a that sets an inspection area within the target image, and a selection unit 137b that performs selection by comparing the target image with a pre-selected or created reference image for image recognition.

[0057] The AI ​​judgment unit 138 uses the trained model 125 created by the learning model creation unit 136 to perform image recognition on the target images selected as targets for judgment by the selection unit 137b. The AI ​​judgment unit 138 considers target images that were not selected as targets for judgment to be unsuitable images and does not perform image recognition using the trained model 125 for those images. Hereafter, image recognition by the trained model 125 will also be referred to as AI judgment.

[0058] The guide unit 139 displays on the display unit 30 the location in the image that is deemed unsuitable for the unsuitable image, and guides the user on the imaging conditions according to the condition of that location. Examples of imaging conditions include the number of pixels of the imaging unit 20, resolution, exposure time, lens aperture, brightness, and focus. The guide content may be displayed on the display unit 30, or a speaker may be connected to the image processing device 10 to read the guide aloud.

[0059] (How to create a pre-trained model) Figure 3 is a flowchart showing the process of creating a learning model in an embodiment of the present invention. The process of creating a learning model is basically carried out in the following steps.

[0060] First, candidate images to be used as training data are acquired based on the image data input from the imaging unit 20 (step S10). The candidate images acquired at this stage do not need to be as numerous as those used in the creation of typical machine learning models; a dozen to several dozen images are sufficient. For example, the number of acquired images may be between 10 and 30.

[0061] Next, candidate images are selected by applying rule-based image recognition processing to each of the multiple candidate images (step S20). The candidate image selection process will be described in detail later. In this application, "selection" includes the concept of obtaining an image from among multiple images to be used in subsequent processing. For example, obtaining an image that can be used in subsequent processing by applying a predetermined process is also included in selection.

[0062] Then, a trained model is created by training a machine learning algorithm with the selected images (step S30). Various machine learning algorithms can be applied, such as neural networks, support vector machines, or deep learning methods such as convolutional neural networks and reinforcement learning.

[0063] (Candidate image selection process 1-1) Figure 4 is a flowchart illustrating the process of selecting candidate images using rule-based image recognition. Figure 5 is a schematic diagram illustrating the candidate image selection process. Candidate image selection is performed by comparing the candidate images with pre-selected or created reference images. The method for selecting or creating reference images will be described later.

[0064] First, the reference image and the candidate image are aligned (step S201). Alignment can be performed using known methods, such as extracting feature points like edges and corners and associating these feature points with each other.

[0065] Next, pixels in the reference image whose brightness values ​​deviate significantly from those of corresponding pixels in the candidate images are identified as outliers. Specifically, as shown in Figure 5, the brightness difference between each corresponding pixel (pixel g0 and pixel g1, pixel g0 and pixel g2, ...) between the reference image r0 and the candidate images c1, c2, ... is calculated (step S202). Then, pixels in the candidate images whose brightness difference exceeds the reference range are identified as outliers.

[0066] The reference range may be set as appropriate by the user using the operation input unit 40. For example, the reference range for brightness difference may be set to be within a predetermined ratio of the brightness values ​​of each pixel g0 in the reference image r0. The predetermined ratio can be arbitrarily set by the user, for example, within the range of 0% to 50%. As an example, if the reference range is set to within 30% of the brightness value, and the brightness value of pixel g0 in the reference image r0 is x0, and the brightness value of pixel g1 in the candidate image c1 corresponding to pixel g0 is x1, then if the brightness difference |x1-x0| exceeds 0.3×x0, pixel g1 is determined to be an outlier pixel.

[0067] Furthermore, the reference range for brightness difference may be set by taking into account the brightness values ​​of other candidate images. In detail, the distribution of luminance values ​​of corresponding pixels g1, g2, ... among multiple candidate images c1, c2, ... is determined, and the reference range is set within the range of 0σ to 3σ based on the standard deviation σ of this distribution. For example, if the reference range is set to 2σ, and the luminance value of pixel g0 in reference image r0 is x0, and the luminance value of pixel g1 in candidate image c1 corresponding to pixel g0 is x1, then if the luminance difference |x1-x0| exceeds 2σ, pixel g1 is determined to be an outlier pixel. Note that when determining the distribution, the luminance value of pixel g0 in reference image r0 may or may not be included.

[0068] Next, candidate images with outlier pixels removed are selected as images for training data (step S204). For example, if pixel g1 in candidate image c1 shown in Figure 5 is determined to be an outlier pixel, candidate image c1 with pixel g1 removed (in other words, the brightness value of pixel g1 is converted to null) will be used as the image for training data.

[0069] (Candidate image selection process 1-2) Figure 6 is a flowchart showing a modified version of the process of selecting candidate images using rule-based image recognition (step S20 in Figure 3).

[0070] First, the reference image and the candidate image are aligned (step S211). The alignment method is the same as in step S201.

[0071] Next, pixels in the reference image whose brightness value differs significantly from that of the corresponding pixel in the candidate image are identified as outliers. Specifically, the brightness difference between each corresponding pixel in the reference image and the candidate image is calculated (step S212). Then, pixels in the candidate image whose brightness difference exceeds the reference range are identified as outliers (step S213). The method for setting the reference range is the same as in step S203.

[0072] Next, it is determined whether the candidate images contain more outlier pixels than the standard proportion (step S214). Candidate images in which the number of outlier pixels does not exceed the standard proportion are selected as images for training data (step S214: No, step S215). On the other hand, candidate images in which the number of outlier pixels exceeds the standard proportion are excluded from being used as training data (step S214: Yes, step S216).

[0073] The reference ratio may be set as appropriate by the user using the operation input unit 40. For example, the reference ratio may be set within the range of 0% to 50%. As an example, if the reference ratio is set to 30%, and more than 30% of the total number of pixels in a candidate image are outliers, that candidate image will not be used as an image for training data. Also, if the reference ratio is set to 0%, if even one outlier pixel is present in a candidate image (i.e., more than 0%), that candidate image will be excluded. Also, if the reference ratio is set to 50%, candidate images in which more than half of the total pixels are outliers will be excluded.

[0074] Furthermore, if candidate images are excluded, a guide may be displayed to enable the user to improve the imaging conditions (step S217). For example, in the excluded candidate images, the display unit 30 may display the area within the image that was deemed unsuitable (e.g., an area where outlier pixels are concentrated). Alternatively, a guide for imaging conditions may be displayed depending on the state of that area. As a specific example, if the brightness value of the area where outlier pixels are concentrated is extremely small compared to the brightness value of the corresponding area in the reference image, a guide such as increasing the illumination intensity or increasing the exposure time may be displayed. By displaying such a guide, even if candidate images are excluded, the user can improve the imaging conditions and take new images, thereby adding new candidate images. This makes it possible to efficiently collect candidate images that can be used as training data.

[0075] (Method for selecting or creating reference images) As the reference image used in the above selection processes 1-1 and 1-2, for example, an image arbitrarily selected by the user from among the multiple candidate images acquired in step S10 can be used.

[0076] Alternatively, a reference image may be created based on multiple candidate images. Specifically, the first step is to align the multiple candidate images. For alignment, known methods can be used, such as extracting feature points like edges and corners and associating these feature points with each other.

[0077] Next, the average or median of the luminance values ​​of corresponding pixels among multiple candidate images is calculated. For example, in Figure 5, the average or median of the pixel values ​​of corresponding pixels g1, g2, ... among candidate images c1, c2, ... is calculated. The image in which the luminance values ​​of each pixel are set to the calculated average or median is used as the reference image.

[0078] (Image recognition processing) Figure 7 is a flowchart showing the image recognition process performed in the image processing method according to an embodiment of the present invention. The image recognition process is basically performed in the following steps.

[0079] First, a target image is acquired based on the image data input from the imaging unit 20 (step S40). Here, the target image is, for example, an image of the product to be inspected in the case of a visual inspection. In the case of water meter reading using OCR, it is an image of the water meter to be read.

[0080] Next, rule-based image recognition processing is applied to the target image to determine whether or not to include it in the judgment of the trained model (AI judgment) (step S50).

[0081] Next, the target images selected as targets for judgment in step S50 are input into the trained model 125 (see Figure 1) created in steps S10 to S30 in Figure 3 for image recognition (step S60). The image recognition results are then displayed on the display unit 30 (step S70). For target images that were not selected as targets for judgment in step S50, image recognition by the trained model 125 is not performed.

[0082] (Selection process for target images 2-1) Figure 8 is a flowchart illustrating the process of selecting target images using rule-based image recognition. Figure 9 is a schematic diagram illustrating the target image selection process. Target image selection is performed by comparing the target image with a pre-selected or created reference image for image recognition. The method for selecting or creating the reference image for image recognition will be described later.

[0083] First, an inspection area is set in the target image and the reference image for image recognition (step S501). The selection process for the target image is performed based on the pixels within the set inspection area. The method for setting the inspection area is not particularly limited. For example, as shown in Figure 9, if a common marker m11 is present in both the target image and the reference image for image recognition, the inspection area m12 may be set based on the marker m11.

[0084] Next, image matching is performed between the inspection area in the reference image for image recognition and the inspection area in the target image, and the matching rate for all or part of the inspection area is calculated (step S502). That is, template matching is performed using the inspection area in the reference image for image recognition as a template, and a matching score is obtained.

[0085] Here, the area to be targeted for image matching may be the entire inspection area or only a part of it. For example, as shown in Figure 9, when reading numerical values ​​using OCR, only the frame portion m13 surrounding the numerical values ​​may be targeted for image matching. This is because if the frame portion m13 in the target image m10 matches the same portion in the reference image for image recognition with a high score, the inside of the frame portion m13 (the numerical portion) is also considered to have good image quality, making it easy to subject to AI judgment (high recognition accuracy can be expected).

[0086] Next, it is determined whether the matching rate falls below a certain threshold (step S503). If the matching rate is above the threshold (step S503: No), the target image is selected for AI judgment (step S504). On the other hand, if the matching rate falls below the threshold (step S503: Yes), the target image is deemed a non-conforming image (step S505). For example, low-resolution images that are considered difficult to judge with AI, or images whose judgment result (bad) is obvious without even needing AI judgment, may be judged as non-conforming images. Such non-conforming images cannot be subjected to AI judgment.

[0087] The standard value for the matching rate may be set as appropriate by the user using the operation input unit 40. For example, the standard value may be set in the range of 0.3 to 1.0. As an example, if the standard value is set to 0.7, target images with a matching rate of 0.8 will be selected as subjects for AI judgment. Conversely, target images with a matching rate of 0.3 will be judged as unsuitable images.

[0088] Furthermore, if an image is determined to be unsuitable (step S505), a guide may be displayed to enable the user to improve the imaging conditions (step S506). For example, in the case of an unsuitable image, the display unit 30 may show the part of the image that is recognized as unsuitable (for example, an area with a particularly low matching rate). Alternatively, a guide for imaging conditions may be displayed according to the state of the part in question. As a specific example, in the case of an unsuitable image where part of the frame portion m13 around a number is missing, the user may be guided to ensure that the entire frame portion m13 is captured in the image.

[0089] (Selection process for target images 2-2) In the above selection process 2-1, after setting the inspection area (step S501), before calculating the agreement rate for all or part of the inspection area (step S502), the brightness values ​​of the inspection area in the target image may be subjected to a normalized correlation transformation so that they become equivalent to the brightness values ​​of the inspection area in the reference image for image recognition. In this case, the agreement rate with the inspection area in the reference image for image recognition is calculated for the inspection area in the target image that has undergone the normalized correlation transformation. In this case, the risk of the target image being judged as an unsuitable image due to the imaging environment of the target image (for example, being dark due to cloudy weather) can be reduced. The processing from step S503 onwards is the same as in selection process 2-1.

[0090] (Method for selecting or creating reference images for image recognition) The reference images used for image recognition in the above selection processes 2-1 and 2-2 may be arbitrarily selected by the user from among candidate images or images used as training data when creating the trained model.

[0091] Alternatively, a reference image for image recognition may be created based on the images used as training data. In detail, first, the multiple images used as training data when creating the trained model are aligned, and the average or median brightness values ​​of corresponding pixels in these multiple images are calculated. Then, pixels whose brightness values ​​deviate significantly from the average or median are determined as outliers. Outliers can be determined in the same way as described in candidate image selection process 1-1.

[0092] Furthermore, by using the average or median of the brightness values ​​of corresponding pixels among multiple images from which outlier pixels have been removed as the brightness value of each pixel, a reference image for image recognition can be created.

[0093] As explained above, according to this embodiment, since the candidate images selected by applying rule-based image recognition processing to multiple candidate images are used as training data for a machine learning algorithm, it is possible to suppress a decrease in the image recognition accuracy of the trained model even when the number of images used as training data is small.

[0094] Furthermore, according to this embodiment, rule-based image recognition processing is applied to the target images, and AI judgment using the trained model is performed only on the selected target images, while AI judgment is not performed on non-suitable images, thereby improving image recognition accuracy and processing efficiency.

[0095] The present invention is not limited to the above embodiments and modifications, and various inventions can be formed by appropriately combining the multiple components disclosed in the above embodiments and modifications. For example, the invention may be formed by excluding some components from all the components shown in the above embodiments and modifications, or by appropriately combining the components shown in the above embodiments and modifications. [Explanation of Symbols]

[0096] 1,2…Image processing system, 10…Image processing device, 11…Communication interface, 12…Storage unit, 13…Processor, 20…Imaging unit, 30…Display unit, 40…Operation input unit, 50…Imaging terminal, 121…Program, 122…Image data, 123…Training data, 124…Parameters, 125…Trained model, 131…Imaging control unit, 132…Display control unit, 133…Setting unit, 134…Image generation unit, 135…First rule-based image recognition unit, 135a…Alignment unit, 135b…Sorting unit, 136…Learning model creation unit, 137…Second rule-based image recognition unit, 137a…Inspection area setting unit, 137b…Sorting unit, 138…AI judgment unit, 139…Guide unit

Claims

1. An image acquisition step of acquiring multiple candidate images to be used as training data, A selection step in which rule-based image recognition processing is applied to each of the above-mentioned multiple candidate images to select an image to be used as training data, The model creation step involves creating a trained model by having a machine learning algorithm learn the images selected in the above selection step, Includes, The aforementioned sorting step is, A positioning step of aligning a pre-selected or created reference image with the candidate image, A step of determining an outlier pixel in which a pixel in the reference image whose brightness value deviates significantly from the corresponding pixel in the candidate image is determined to be an outlier pixel, The steps include selecting candidate images from which the aforementioned outlier pixels have been removed as images to be used as training data, How to create a trained model that includes [specific data / features].

2. The outlier pixel determination step involves calculating the brightness difference between corresponding pixels in the reference image and the candidate image, and determining pixels in the candidate image whose brightness difference exceeds a reference range as outlier pixels. A method for creating a trained model as described in claim 1.

3. The step further includes setting the reference range for the brightness difference to a range of 0% or more and 50% or less of the brightness value of the pixel in the reference image, The aforementioned outlier pixel determination step determines pixels whose absolute value of the brightness difference exceeds the set reference range as outlier pixels. A method for creating a trained model as described in claim 2.

4. The process further includes the step of determining the distribution of brightness values ​​of corresponding pixels among the plurality of candidate images, and setting the reference range within the range of 0σ to 3σ in the distribution, The aforementioned outlier pixel determination step determines pixels whose absolute value of the brightness difference exceeds the set reference range as outlier pixels. A method for creating a trained model as described in claim 2.

5. The aforementioned reference image was selected from among the multiple candidate images. A method for creating a trained model as described in claim 1.

6. The aforementioned reference image is Alignment is performed between the aforementioned multiple candidate images, The average or median of the brightness values ​​of corresponding pixels among the multiple candidate images is calculated. This is created by using the aforementioned average or median value as the brightness value of each pixel. A method for creating a trained model as described in claim 1.

7. An image acquisition step of acquiring multiple candidate images to be candidates for training data, A selection step in which rule-based image recognition processing is applied to each of the above-mentioned multiple candidate images to select an image to be used as training data, The model creation step involves creating a trained model by having a machine learning algorithm learn the images selected in the above selection step, Includes, The aforementioned sorting step is, A positioning step of aligning a pre-selected or created reference image with the candidate image, A step of determining an outlier pixel in which a pixel in the reference image whose brightness value deviates significantly from the corresponding pixel in the candidate image is determined to be an outlier pixel, An exclusion step is performed to exclude candidate images with a higher percentage of outlier pixels from the images used as training data, Includes, Select the candidate images that were not excluded as images to be used as training data. How to create a pre-trained model.

8. The step further includes setting the aforementioned reference percentage within a range of 0% to 50%, The exclusion step excludes candidate images in which the number of pixels determined to be outliers exceeds the set standard ratio to the total number of pixels in the candidate image. A method for creating a trained model as described in claim 7.

9. The aforementioned reference image was selected from among the multiple candidate images. A method for creating a trained model as described in claim 7.

10. The aforementioned reference image is Alignment is performed between the aforementioned multiple candidate images, The average or median of the brightness values ​​of corresponding pixels among the multiple candidate images is calculated. This is created by using the aforementioned average or median value as the brightness value of each pixel. A method for creating a trained model as described in claim 7.

11. The image acquisition step involves acquiring 10 to 30 candidate images. A method for creating a trained model according to any one of claims 1 to 10.

12. A second selection step involves applying rule-based image recognition processing to the target image to determine whether or not to include the target image in the evaluation by the trained model. A determination step in which, for target images selected as targets for determination in the second selection step, image recognition is performed using a trained model created by the method for creating a trained model described in any one of claims 1 to 10, Includes, In the second selection step described above, target images that were not selected as targets for judgment are treated as unsuitable images, and image recognition is not performed using the trained model. Image processing methods.

13. An image acquisition step of acquiring multiple candidate images to be candidates for training data, A selection step in which rule-based image recognition processing is applied to each of the above-mentioned multiple candidate images to select an image to be used as training data, The model creation step involves creating a trained model by having a machine learning algorithm learn the images selected in the above selection step, How to create a pre-trained model that includes, A second selection step involves applying rule-based image recognition processing to the target image to determine whether or not to include the target image in the evaluation by the trained model. A determination step in which, for target images selected as targets for determination in the second selection step, image recognition is performed using the trained model created by the method for creating the trained model, Includes, In the second selection step described above, the target images that were not selected as targets for judgment are treated as unsuitable images, and image recognition is not performed using the trained model. The second sorting step involves setting an inspection area in the target image and performing sorting based on the pixels within the inspection area. Image processing methods.

14. An image acquisition step of acquiring multiple candidate images to be used as training data, A selection step in which rule-based image recognition processing is applied to each of the above-mentioned multiple candidate images to select an image to be used as training data, The model creation step involves creating a trained model by having a machine learning algorithm learn the images selected in the above selection step, How to create a pre-trained model that includes, A second selection step involves applying rule-based image recognition processing to the target image to determine whether or not to include the target image in the evaluation by the trained model. A determination step in which, for target images selected as targets for determination in the second selection step, image recognition is performed using the trained model created by the method for creating the trained model, Includes, In the second selection step described above, the target images that were not selected as targets for judgment are treated as unsuitable images, and image recognition is not performed using the trained model. The second selection step involves selecting the target image by comparing it with a pre-selected or created reference image for image recognition. Image processing methods.

15. The second sorting step described above is: The steps include setting an inspection area in the aforementioned reference image for image recognition and the aforementioned target image, A matching rate calculation step involves performing image matching of the inspection area in the target image with respect to the inspection area in the reference image for image recognition, and calculating the matching rate for all or part of the inspection area. The steps include determining that an image is unsuitable if the aforementioned agreement rate falls below a standard value, including, The image processing method according to claim 14.

16. The step further includes setting the reference value for the agreement rate in the range of 0.3 or more and 1.

0. The image processing method according to claim 15.

17. Prior to the matching rate calculation step, the method further includes a step of applying a normalized correlation transformation to the brightness values ​​of the inspection areas in the target image so that the brightness values ​​of the inspection areas in the target image become equivalent to the brightness values ​​of the inspection areas in the reference image for image recognition. The aforementioned agreement rate calculation step involves calculating the agreement rate for the examination area within the target image that has undergone the normalized correlation transformation. The image processing method according to claim 15.

18. The aforementioned reference image for image recognition is selected from among multiple images used as training data when creating the aforementioned trained model. The image processing method according to claim 14.

19. The aforementioned reference image for image recognition is When creating the aforementioned trained model, alignment is performed between multiple images used as training data, and the average or median of the brightness values ​​of corresponding pixels between the multiple images is calculated. Pixels whose brightness values ​​deviate significantly from the aforementioned average or median are determined to be outliers in each image. This method is created by taking the average or median of the brightness values ​​of corresponding pixels in multiple images, excluding outlier pixels, and using that as the brightness value for each pixel. The image processing method according to claim 14.

20. An image acquisition step to obtain multiple candidate images that will serve as training data, A selection step in which rule-based image recognition processing is applied to each of the above-mentioned multiple candidate images to select an image to be used as training data, The model creation step involves creating a trained model by having a machine learning algorithm learn the images selected in the above selection step, Have the computer run it, The aforementioned sorting step is, A positioning step of aligning a pre-selected or created reference image with the candidate image, A step of determining an outlier pixel in which a pixel in the reference image whose brightness value deviates significantly from the corresponding pixel in the candidate image is determined to be an outlier pixel, The steps include selecting candidate images from which the aforementioned outlier pixels have been removed as images to be used as training data, A program that includes this.

21. An image acquisition step of acquiring multiple candidate images to be candidates for training data, A selection step in which rule-based image recognition processing is applied to each of the above-mentioned multiple candidate images to select an image to be used as training data, The model creation step involves creating a trained model by having a machine learning algorithm learn the images selected in the above selection step, Have the computer run it, The aforementioned sorting step is, A positioning step of aligning a pre-selected or created reference image with the candidate image, A step of determining an outlier pixel in which a pixel in the reference image whose brightness value deviates significantly from the corresponding pixel in the candidate image is determined to be an outlier pixel, An exclusion step is performed to exclude candidate images with a higher percentage of outlier pixels from the images used as training data, Includes, Select the candidate images that were not excluded as images to be used as training data. program.

22. For candidate images that were not selected as training data in the selection step, the computer is further instructed to perform a display step in which the area containing the missing pixels is displayed as the location within the image that was excluded from the selection. The program according to claim 20 or 21.

23. A second selection step involves applying rule-based image recognition processing to the target image to determine whether or not to include the target image in the evaluation by the trained model. A determination step in which image recognition is performed using the trained model on the target images selected as targets for determination in the second selection step, The computer is then made to perform the following: In the second selection step described above, target images that were not selected as targets for judgment are treated as unsuitable images, and image recognition is not performed by the trained model. The program according to claim 20 or 21.

24. The second selection step is to perform selection by comparing the target image with a pre-selected or created reference image for image recognition, The steps include setting an inspection area in the aforementioned reference image for image recognition and the aforementioned target image, A matching rate calculation step involves performing image matching of the inspection area in the target image with respect to the inspection area in the reference image for image recognition, and calculating the matching rate for all or part of the inspection area. The steps include determining that an image is unsuitable if the aforementioned agreement rate falls below a standard value, Includes, The program according to claim 23, further comprising causing the computer to perform a display step in which, with respect to the non-conforming image, the area with a particularly low matching rate is displayed as the portion of the image that is the basis for being recognized as non-conforming.

25. The computer is further instructed to perform a guide step that guides the imaging conditions according to the state of the area within the aforementioned image. The program according to claim 22.

26. The computer further performs a guide step that guides the imaging conditions according to the state of the area in the image. The program according to claim 24.

27. ​​An image acquisition step of acquiring a plurality of candidate images to be used as training data, A selection step in which rule-based image recognition processing is applied to each of the above-mentioned multiple candidate images to select an image to be used as training data, The model creation step involves creating a trained model by having a machine learning algorithm learn the images selected in the above selection step, A second selection step involves applying rule-based image recognition processing to the target image to determine whether or not to include the target image in the evaluation by the trained model. A determination step in which image recognition is performed using the trained model on the target images selected as targets for determination in the second selection step, Have the computer run it, In the second selection step described above, the target images that were not selected as targets for judgment are treated as unsuitable images, and image recognition is not performed using the trained model. The second sorting step involves setting an inspection area in the target image and performing sorting based on the pixels within the inspection area. program.

28. An image acquisition step of acquiring multiple candidate images to be candidates for training data, A selection step in which rule-based image recognition processing is applied to each of the above-mentioned multiple candidate images to select an image to be used as training data, The model creation step involves creating a trained model by having a machine learning algorithm learn the images selected in the above selection step, A second selection step involves applying rule-based image recognition processing to the target image to determine whether or not to include the target image in the evaluation by the trained model. A determination step in which image recognition is performed using the trained model on the target images selected as targets for determination in the second selection step, Have the computer run it, In the second selection step described above, the target images that were not selected as targets for judgment are treated as unsuitable images, and image recognition is not performed using the trained model. The second selection step involves selecting the target image by comparing it with a pre-selected or created reference image for image recognition. program.

29. An image acquisition unit that acquires multiple candidate images to be used as training data, A selection unit that selects images to be used as training data by applying rule-based image recognition processing to each of the aforementioned multiple candidate images, A model creation unit creates a trained model by having a machine learning algorithm learn the images selected by the aforementioned selection unit. Equipped with, The sorting unit is, Align the candidate image with a pre-selected or created reference image. For a pixel in the aforementioned reference image, a pixel whose brightness value deviates significantly from that of the corresponding pixel in the candidate image is determined to be an outlier pixel. The candidate images from which the aforementioned outlier pixels have been removed are selected as images to be used as training data. Image processing device.

30. An image acquisition unit that acquires a plurality of candidate images to be used as training data, A selection unit that selects images to be used as training data by applying rule-based image recognition processing to each of the aforementioned multiple candidate images, A model creation unit creates a trained model by having a machine learning algorithm learn the images selected by the aforementioned selection unit. Equipped with, The sorting unit is, Align the pre-selected or created reference image with the candidate image, For a pixel in the aforementioned reference image, a pixel whose brightness value deviates significantly from that of the corresponding pixel in the candidate image is determined to be an outlier pixel. Candidate images with outliers exceeding a certain percentage are excluded from the images used as training data. Select the candidate images that were not excluded as images to be used as training data. Image processing device.

31. A second selection unit performs rule-based image recognition processing on the target image to determine whether or not to include the target image in the evaluation by the trained model, A determination unit performs image recognition on target images selected as targets for determination in the second selection unit using the trained model, Furthermore, Images that were not selected as targets for judgment by the second selection unit are treated as unsuitable images, and image recognition is not performed using the trained model. The image processing apparatus according to claim 29 or 30.

32. An image acquisition unit that acquires a plurality of candidate images to be used as training data, A selection unit that selects images to be used as training data by applying rule-based image recognition processing to each of the aforementioned multiple candidate images, A model creation unit creates a trained model by having a machine learning algorithm learn the images selected by the aforementioned selection unit. A second selection unit performs rule-based image recognition processing on the target image to determine whether or not to include the target image in the evaluation by the trained model, A determination unit performs image recognition on target images selected as targets for determination in the second selection unit using the trained model, Equipped with, Images that were not selected as targets for judgment by the second selection unit are treated as unsuitable images, and image recognition is not performed using the trained model. The second sorting unit sets an inspection area in the target image and performs sorting based on the pixels within the inspection area. Image processing device.

33. An image acquisition unit that acquires a plurality of candidate images to be used as training data, A selection unit that selects images to be used as training data by applying rule-based image recognition processing to each of the aforementioned multiple candidate images, A model creation unit creates a trained model by having a machine learning algorithm learn the images selected by the aforementioned selection unit. A second selection unit performs rule-based image recognition processing on the target image to determine whether or not to include the target image in the evaluation by the trained model, A determination unit performs image recognition on target images selected as targets for determination in the second selection unit using the trained model, Equipped with, Images that were not selected as targets for judgment by the second selection unit are treated as unsuitable images, and image recognition is not performed using the trained model. The second selection unit performs selection by comparing the target image with a pre-selected or created reference image for image recognition. Image processing device.

34. An image acquisition unit that acquires a plurality of candidate images to be used as training data, A selection unit that selects images to be used as training data by applying rule-based image recognition processing to each of the aforementioned multiple candidate images, A model creation unit creates a trained model by having a machine learning algorithm learn the images selected by the aforementioned selection unit. A second selection unit performs rule-based image recognition processing on the target image to determine whether or not to include the target image in the evaluation by the trained model, A determination unit performs image recognition on target images selected as targets for determination in the second selection unit using the trained model, Equipped with, Images that were not selected as targets for judgment by the second selection unit are treated as unsuitable images, and image recognition is not performed using the trained model. At least the image acquisition unit, the sorting unit, the model creation unit, the second sorting unit, and the determination unit are implemented by edge computing. Image processing device.

35. An image processing apparatus, An image acquisition unit that acquires multiple candidate images to be used as training data, A selection unit that selects images to be used as training data by applying rule-based image recognition processing to each of the aforementioned multiple candidate images, A model creation unit creates a trained model by having a machine learning algorithm learn the images selected by the aforementioned selection unit. A second selection unit performs rule-based image recognition processing on the target image to determine whether or not to include the target image in the evaluation by the trained model, A determination unit performs image recognition on target images selected as targets for determination in the second selection unit using the trained model, Equipped with, Images that were not selected as targets for judgment by the second selection unit are treated as unsuitable images, and image recognition is not performed using the trained model. The image processing device is composed of a terminal device comprising at least a processor and memory, The image acquisition unit acquires the plurality of candidate images and the target image based on image data input from an imaging device connected to the image processing device by wire or wireless connection. Image processing device.

Images