Image inspection apparatus

Abstract

To provide an image inspection apparatus which is adaptable to the increase of recognizable character size while reducing a processing load.SOLUTION: An image inspection apparatus includes, as functions of DSP equipped in a control section of a control unit: a learning data generation unit 310b: and a determiner update unit 315 which functions as a learning execution unit that execute update of a pre-trained model equipped with a feature extraction unit configured to extract feature quantities indicating features of image data and a character type output unit configured to output a character type based on the feature quantities. The learning data generation unit 310b generates learning data having learning image data, designated position information, designated character type information, and designated size information. The determiner update unit 315 updates the pre-trained model so that the feature quantities, which are extracted by inputting the learning image data to the feature extraction unit and correspond to a position indicated by the designated position information, may be classified into a designated character type class corresponding to the combination of the character type indicated by the designated character type information and the size indicated by the designated size information.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an image inspection device that implements a trained network that performs classification tasks for character recognition. [Background technology]

[0002] For example, in the factory automation industry, image inspection devices are widely known to replace the process of visually inspecting workpieces by humans. Image inspection devices capture images of workpieces and judge the captured images based on certain criteria. If the image inspection device is an image sensor, the judgment results are output via I / O to an external device such as a programmable logic controller (PLC).

[0003] When the criteria used for judgment by an image inspection device are based on certain image features, even changes in the image features that would not affect the results of a visual inspection by a human may affect the judgment results of the image inspection device. Therefore, an image inspection device that uses an image recognition model obtained by machine learning to obtain judgment results closer to those of a visual inspection is known, such as the image inspection device disclosed in Patent Document 1. The image inspection device of Patent Document 1 is configured to generate a learning device that distinguishes between good and defective product images by learning good product images that have been assigned attributes as good products and defective product images that have been assigned attributes as defective products, and to input newly acquired images into the learning device during operation to make a pass / fail judgment. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-187072 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, there is a device equipped with character recognition (OCR) that recognizes the character image, which is the character portion of a captured image of a workpiece to which characters have been added, identifies the characters on the workpiece, and determines whether the characters meet the evaluation criteria using an image recognition model.

[0006] One OCR recognition method is the sliding window method. In the sliding window method, the size for extracting characters from an image is first determined. For example, a frame is drawn around the characters on an image of an actual workpiece with text on it. This frame size becomes the character extraction size. After the character extraction size is determined, when the image to be OCR is input during operation, the image is extracted into multiple image regions using the predetermined character extraction size. Each of the multiple extracted image regions is verified to determine whether it has features similar to a given character. For example, if a region is determined to have the feature "A" (or has features similar to the "A" feature compared to other classes to the extent that it can be considered to be in the "A" class), that region is recognized as an image of the character "A." Since the feature verification target is shifted little by little during verification, conceptually the verification target frame (window) slides, hence the name sliding window method.

[0007] However, the sliding window method determines which character the features of the area being cut out correspond to, so character images whose sizes vary depending on the character type (for example, the letter "I" or a period in a proportional font) may be cut out as a character image that includes adjacent characters, which could reduce recognition accuracy.

[0008] In addition, in OCR, there are cases where it is desired to increase the variety of character sizes that can be recognized by the image recognition model, but in the case of the sliding window method, it is necessary to scan and classify features while changing the window size, which results in an enormous amount of calculation processing.

[0009] The present disclosure has been made in consideration of the above points, and its purpose is to make it possible to accommodate an increase in recognizable character sizes while reducing the processing load. [Means for solving the problem]

[0010] In order to achieve the above objective, an aspect of the present disclosure can be based on an image inspection device that executes a pre-trained character recognition model that classifies an image area, which is a partial area of ​​an image of input image data, into a first character type class.

[0011] The image inspection device includes a control unit that functions as a training data generation unit that executes the pre-trained model and generates training data, and a training execution unit that updates the pre-trained model based on the training data. The pre-trained model includes a feature extraction unit that extracts features that indicate characteristics of the input image data from the image data, and a character type output unit that outputs a character type based on the features extracted by the feature extraction unit. The training data generation unit generates training data that includes training image data, designated position information that indicates the relative position of a character in an image of the training image data, designated character type information that indicates the character type of the character located at the relative position, and designated size information that indicates the size of the character in the image.

[0012] The learning execution unit updates the pre-trained model so as to classify the features extracted when the training image data is input to the feature extraction unit and corresponding to the relative position indicated in the specified position information into a specified character type class corresponding to the combination of the character type indicated in the specified character type information and the size indicated in the specified size information.

[0013] According to this configuration, the training data generated by the training data generation unit includes not only designated character type information indicating the character type, but also designated position information indicating the relative position of the character in the training image data image and designated size information indicating the size of the character in the image. As a result, the training execution unit learns combinations of character sizes and character types specified in the training data as new character type classes, thereby reducing the learning load related to character sizes and increasing the number of recognizable character sizes.

[0014] The control unit may also be configured to include a convolutional operation network inference accelerator that executes operations of a convolutional neural network as the feature extraction unit.

[0015] The feature extraction unit may include a character type class feature extraction unit that extracts features related to a character type class of the image region, and a foreground / background feature extraction unit that extracts foreground features that are features indicating whether the image region is foreground or background. The feature extraction unit may also include a foreground output unit that determines whether the image region is the foreground based on the foreground features, and the character type output unit may output, for an image region that corresponds to the image region determined to be the foreground, a value indicating the character type of the image region based on the character type feature corresponding to the image region.

[0016] That is, for example, if a classifier includes the background, there will be any "character" class close to the "background" class in the feature space handled by the classifier, and if such a "character" is labeled, all nearby "background" vectors will be classified into that "character" class. However, by narrowing down the image area to whether it is foreground or background and then having the classifier learn, as in this embodiment, such problems are less likely to occur, and the accuracy of judgment is improved.

[0017] The feature extraction unit may include a character type class feature extraction unit that extracts features related to a character type class of the image region, and a size feature extraction unit that extracts a size feature that is a feature related to the size of the image region. In this case, when the character type class feature belongs to the designated character type class corresponding to the combination of the character type and the designated size, the character type output unit can output the size of the character image in the image region based on the designated size included in the designated character type class.

[0018] The image processing apparatus may further include a size output unit that outputs a size of the character image in the image region based on the size feature. In this case, the size output unit can output the size of the character image when the character type class feature does not belong to the designated character type class corresponding to the combination of the character type and the designated size. [Effects of the Invention]

[0019] As described above, the learning data includes designated character type information, designated position information indicating the relative position of the character in the image, and designated size information indicating the size of the character. Therefore, a combination of character size and character type can be learned as a new character type class. This reduces the learning load related to character size while increasing the number of recognizable character sizes. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a diagram illustrating an image inspection device according to an embodiment of the present invention during operation. FIG. [Figure 2] FIG. 2 is a hardware configuration diagram of the image inspection device. [Figure 3] FIG. 2 is a functional block diagram of a DSP. [Figure 4] 10 is a flowchart showing an example of a procedure for setting up an image inspection device. [Figure 5] 10 is a flowchart illustrating an example of a procedure for setting an inspection tool. [Figure 6] 10 is a flowchart showing an example of a procedure for performing an inspection. [Figure 7] FIG. 1 is a diagram illustrating the processing concept of an AI-OCR tool. [Figure 8] FIG. 10 is a diagram illustrating character types in pre-learning. [Figure 9] FIG. 10 is a diagram illustrating the character type output unit when pre-learning is completed. [Figure 10] FIG. 10 is a diagram illustrating the character type output unit when on-site learning is completed. [Figure 11] FIG. 10 is a diagram showing a specific example of character recognition processing. [Figure 12] FIG. 10 is a diagram showing a specific example of dictionary registration processing. [Figure 13] FIG. 10 is a diagram illustrating a case where the same character that appears different is additionally learned. [Figure 14] 10 is a flowchart showing an example of a procedure for relearning after operation of the image inspection device. [Figure 15] FIG. 10 is a diagram showing an example of a driving result display screen. [Figure 16] FIG. 10 is a diagram illustrating an example of a display of an additional learning window. [Figure 17] FIG. 10 is a diagram showing an example of a screen displayed during additional learning. [Figure 18] FIG. 10 is a diagram showing an example of a display of a character type selection window. [Figure 19] FIG. 18 is a diagram equivalent to FIG. 17 showing a state in which a character type has been selected. [Figure 20] FIG. 10 is a diagram showing an example of a screen displayed when additionally learning an unregistered character type. [Figure 21] FIG. 10 is a diagram showing an example of a screen displayed when registering a character type in a user dictionary. [Figure 22] FIG. 10 is a diagram showing an example of a window display for newly registering a character type. [Figure 23] FIG. 19 is a diagram showing newly registered character types. [Figure 24] FIG. 10 is a diagram showing an example of a screen displayed before additional learning starts. [Figure 25] FIG. 10 is a diagram showing an example of a screen displayed when checking the results of additional learning. [Figure 26]FIG. 10 is a diagram showing an example of a screen displayed during operation when checking the date. [Figure 27] FIG. 10 is a diagram showing an example of a selection window that accepts the selection of an OCR mode. [Figure 28] FIG. 10 is a diagram illustrating an example of a screen displayed when specifying a template. [Figure 29] FIG. 10 is a diagram illustrating an example of a window for setting a determination condition. [Figure 30] FIG. 10 is a diagram illustrating an example of a window for setting a date and time offset. [Figure 31] 10A and 10B are diagrams illustrating a case where a master character string is changed for each image capture. [Figure 32] FIG. 10 is a diagram showing an example of a screen displayed when custom settings are made. [Figure 33] FIG. 10 is a diagram showing an example of a format setting window. [Figure 34] This is a diagram equivalent to FIG. 42 showing the case where the data is divided into elements. [Figure 35] FIG. 10 is a diagram showing a screen displayed when setting the content of an element. [Figure 36] FIG. 10 is a diagram showing a screen on which a first selection window showing options for the content of an element is displayed. [Figure 37] FIG. 10 is a diagram showing a screen on which a second selection window showing options for the content of an element is displayed. [Figure 38] FIG. 10 is a diagram showing an example of a count up / down setting window. [Figure 39] FIG. 10 is a diagram showing an example of a replacement setting window. [Figure 40] FIG. 10 is a diagram showing an example of a link window for linking the notation of the first, middle, and last days of the month with a calendar. [Figure 41] FIG. 10 is a diagram illustrating an example of a batch input window. [Figure 42] FIG. 10 is a diagram showing a screen displayed when an image including character strings arranged in an arc is input. [Figure 43] FIG. 42 shows a case where the phase of character strings arranged in an arc shape has changed. [Figure 44]FIG. 10 is a diagram illustrating an example of an arc setting window. [Figure 45] FIG. 10 is a diagram showing a screen displayed when additional learning is performed using test images with different character string phases. [Figure 46] FIG. 10 is a diagram showing an example of a screen displayed during operation when inspecting character strings arranged in an arc shape. [Figure 47] FIG. 47 is a diagram equivalent to FIG. 46 in which a part of the character strings arranged in an arc shape is missing. [Figure 48] FIG. 10 is a diagram illustrating an example of an additional learning confirmation window. [Figure 49] 10A and 10B are diagrams illustrating a case where a box enclosing a character to be learned is moved during additional learning. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the following description of the preferred embodiments is merely exemplary in nature and is not intended to limit the present invention, its applications, or its uses.

[0022] FIG. 1 is a diagram illustrating an image inspection device S according to an embodiment of the present invention during operation. The image inspection device S captures an image of a workpiece W transported by a transport means A according to an imaging setting to acquire a workpiece image, judges the acquired workpiece image according to a judgment setting, and outputs an inspection result using the judgment result to an external device as a sensor output. An example of the external device is a programmable logic controller (PLC) 5, but the external device may be a device other than the PLC 5. The PLC 5 controls the transport means A based on the received inspection result, for example, to separate the storage destination of the workpiece W. The following explanation will be given for the case where the external device is the PLC 5. Note that the workpiece W may be a workpiece that is not transported by the transport means A.

[0023] The image inspection device S includes an imaging unit 1 for capturing an image of a workpiece W, a control unit 2 to which image data captured by the imaging unit 1 is input, a PC (personal computer) 3 for configuring the image inspection device S, and a display device 4 for displaying a setting screen, a selection screen, workpiece images, inspection results, etc. The control unit 2 is capable of executing a trained model that performs a classification task into multiple character type classes in order to recognize characters in an image area that is a part of the image of the input image data. The control unit 2 transmits a sensor output to the PLC 5 according to the character recognition results obtained by the trained model.

[0024] Here, the image inspection device S may be used, for example, to inspect the workpiece W from various angles at various points in a manufacturing device or production line. For this reason, multiple image inspection devices S may be installed in a single manufacturing device or production line, potentially resulting in insufficient installation space and power supply. Therefore, the image inspection device S must be compact to accommodate the installation space and power-efficient to accommodate the power supply. To meet these requirements, the image inspection device S according to this embodiment does not include a GPU. In other words, the control unit 2 is equipped with a trained model that has been trained to the extent that basic character recognition is possible. However, the desired classification accuracy can be achieved without the user having to perform advanced additional training, for which the use of a GPU is recommended. Furthermore, the fact that the user does not need to perform advanced training means that the user does not need to prepare a GPU for training, thereby reducing the time required for training. Note that a single image inspection device S may be installed and operated in a manufacturing device or production line. The image inspection device S may also be referred to as an image sensor.

[0025] (Configuration of imaging unit) The imaging unit 1 is separate from the control unit 2 and is installed so as to be able to capture an image of the workpiece W from a desired direction. The workpieces W are sequentially transported by the transport means A into the imaging field of view of the imaging unit 1.

[0026] As shown in FIG. 2, the imaging unit 1 includes an illumination module 10 for illuminating the workpiece W, and a camera module 11 for capturing an image of the workpiece W illuminated by the illumination module 10.

[0027] The lighting module 10 has an LED (light emitting diode) 10a that irradiates light toward the workpiece W, and an LED driver 10b that controls the light intensity and light emission timing of the LED 10a. The LED driver 10b is connected to a head communication section 20 (described later) of the control unit 2, and is controlled by a control section 21 (described later) of the control unit 2.

[0028] The camera module 11 has an AF motor 11a and an imaging board 11b. The AF motor 11a is a member for automatically focusing on the workpiece W by driving a focusing lens of an optical system (not shown). The autofocus method is not particularly limited, and examples include a contrast method.

[0029] The imaging board 11b is equipped with a CMOS sensor 11c, an FPGA 11d, and a DSP 11e. The CMOS sensor 11c is an image sensor that receives light that is irradiated from the LED 10a onto the workpiece W and reflected by the workpiece W. This CMOS sensor 11c is connected to the head communication section 20 of the control unit 2, and is controlled by the control section 21 of the control unit 2 to perform exposure processing at a predetermined timing for a predetermined time.

[0030] The FPGA 11d is a processing device whose internal processing contents can be changed. The DSP 11e is a signal processing device. A light-receiving amount signal of the light-receiving element of the CMOS sensor 11c is output to the FPGA 11d for processing, and is also output to the DSP 11e for processing. The processing by the FPGA 11d and the DSP 11e is not particularly limited, but examples thereof include various types of filter processing. The image data processed by the FPGA 11d and the DSP 11e is transmitted from the imaging unit 1 to the control unit 2.

[0031] The imaging unit 1 and the control unit 2 are connected via a communication cable 6. Therefore, the control unit 2 can be installed in a location away from the location where the imaging unit 1 is installed.

[0032] (PC configuration) PC3 is configured as a general-purpose personal computer or the like. In this example, the personal computer can be used as PC3 by installing a predetermined program on it. PC3 includes operating devices such as a keyboard 3a and a mouse (not shown). A user of the image inspection device S can perform setting operations and selection operations on the image inspection device S by operating the operating devices on PC3. Specific setting operations and selection operations will be described later.

[0033] The PC 3 and the communication board 22 of the control unit 2 are connected to be able to communicate with each other, and information based on setting operations by the user is sent from the PC 3 to the control unit 2. In addition, the PC 3 is able to receive image data of the workpiece W, inspection results, etc. output from the control unit 2. The PC 3 and the control unit 2 are connected via a communication cable 7. Therefore, the PC 3 can be installed in a location away from where the control unit 2 is installed.

[0034] (Configuration of display device 4) The display device 4 is configured, for example, with a liquid crystal display or an organic EL display. In this example, the display device 4 includes a touch panel 4a. The touch panel 4a is a component capable of detecting operations by the user's finger. The type of the touch panel 4a is not particularly limited, and examples include a capacitive type and an infrared type. The display device 4 and the communication board 22 of the control unit 2 are connected to be able to communicate with each other. Operation information of the touch panel 4a by the user is transmitted from the display device 4 to the control unit 2. In addition, the display device 4 is capable of receiving image data of the workpiece W output from the control unit 2. The display device 4 and the control unit 2 are connected via a communication cable 7. Therefore, the display device 4 can be installed in a location away from the installation location of the control unit 2.

[0035] The PC 3 and the display device 4 may be configured as an integrated unit. For example, the display device 4 may be configured as a display device that the PC 3 has. In this case, the main body of the PC 3 and the display device 4 may be integrated or may be separate. In this example, the communication board 22 and the PLC 5 are connected via a communication cable 7.

[0036] (Configuration of control unit 2) 2, the control unit 2 includes a head communication section 20, a control section 21, a communication board 22, a power supply 23, a connector board 24, an I / O board 25, and a storage device (storage section) 26. The head communication section 20 is connected to the control section 21 and is a section that executes mutual communication between the control section 21 and the imaging unit 1. A control signal for the imaging unit 1 output from the control section 21 is transmitted to the imaging unit 1 via the head communication section 20. The control signal for the imaging unit 1 includes a signal that controls the light emission timing and light emission amount of the LED 10a, and a signal that controls the AF motor 11a and the imaging board 11b. Furthermore, image data acquired by the imaging unit 1 is output from the imaging unit 1 and then transmitted to the control section 21 via the head communication section 20.

[0037] The control unit 21 has a DSP 21a and FPGA 21b that perform various signal processing, an accelerator 21c that speeds up the processing, and a memory 21d made up of RAM, ROM, etc. The specific configuration of the control unit 21 will be described later.

[0038] The communication board 22 is connected to the control unit 21, and is a member that executes mutual communication between the control unit 21 and the PC 3, the display device 4, and the PLC 5.

[0039] The connector board 24 includes a power supply interface 24a. A power cable (not shown) for supplying power from an external source is connected to the power supply interface 24a. The connector board 24 is connected to a power supply 23, and power supplied from the outside to the power supply interface 24a is adjusted to a predetermined voltage by the power supply 23 and then supplied to the control unit 21. The power supplied to the control unit 21 is supplied to the imaging unit 1 via the head communication unit 20.

[0040] The I / O board 25 is connected to the control unit 21. The inspection results output from the control unit 21 are input to the PLC 5 via the I / O board 25.

[0041] (Details of DSP21a) 3 is a functional block diagram of the control unit 21 included in the control unit 2. The control unit 21 includes an imaging setting unit 100, an examination setting unit 200, and an examination execution unit 400.

[0042] The imaging setting unit 100 sets imaging setting parameters and reflects the set imaging setting parameters when capturing an image of the workpiece W. The imaging setting parameters include, for example, multiple parameters such as the timing of illumination by the illumination module 10, brightness (light emission amount), exposure time by the camera module 11, and focus (focal position) of the camera module 11. When setting the image inspection device S, the imaging setting unit 100 displays a GUI (graphical user interface) related to setting the imaging setting parameters on the display device 4. Although not shown, the GUI related to setting the imaging setting parameters has areas where the illumination brightness, exposure time, focus, etc. can be individually set. When a user inputs each parameter on the GUI using the touch panel 4a, keyboard 3a, etc., the imaging setting unit 100 accepts the user's setting operation and outputs each parameter to, for example, the storage device 26 for storage therein. Each parameter can be read from the storage device 26 as needed.

[0043] When acquiring image data of the workpiece W, the imaging setting unit 100 outputs the setting data including imaging setting parameters to the camera module 11. When the camera module 11 receives the setting data, it sets the imaging setting parameters included in the received setting data within the camera module 11 so that they are reflected when capturing an image. When the camera module 11 receives an imaging trigger signal, it illuminates the workpiece W in accordance with the imaging setting parameters and captures an image of the workpiece W to acquire image data of the workpiece W.

[0044] The inspection setting unit 200 includes a tool setting unit 201, a master image registration unit 220, and an inspection condition setting unit 230. The tool setting unit 201 includes a tool selection unit 202, a parameter setting unit 205, and a learning tool setting unit 305. The tool selection unit 202 allows the user to select a tool to be set. When setting up the image inspection device S, the tool selection unit 202 displays a GUI for tool selection on the display device 4. Although not shown, the GUI for tool selection has an area where the user can select a desired tool from multiple tools. When the user selects an arbitrary tool on the GUI using the touch panel 4a, keyboard 3a, or the like, the tool selection unit 202 accepts the user's selection operation and outputs the selected tool to, for example, the storage device 26 for storage therein. The selected tool can be read from the storage device 26 as needed.

[0045] When the tool selected by the tool selection unit 202 is a rule-based tool, information about the tool selected by the tool selection unit 202 is output to the parameter setting unit 205 as selected tool information. When setting the image inspection device S, the parameter setting unit 205 displays a GUI (Graphical User Interface) related to setting tool parameters on the display device 4. Tool parameters are parameters that can be set for each tool. When the user sets tool parameters on the GUI using the touch panel 4a, keyboard 3a, etc., the parameter setting unit 205 accepts the setting operation by the user and outputs the tool parameters to, for example, the storage device 26 for storage in the storage device 26. The tool parameters can be read from the storage device 26 as necessary. The learning tool setting unit 305 will be described later.

[0046] The master image registration unit 220 is a part that registers the captured image I1 as a master image I2. Specifically, the master image registration unit 220 incorporates the captured image I1 into a GUI and displays it on the display device 4, thereby presenting it to the user. When the user wants to register the captured image I1 displayed on the display device 4 as the master image I2, the user performs a registration operation using the touch panel 4a, the keyboard 3a, or the like. The master image registration unit 220 registers the captured image I1 for which the user has performed the registration operation as the master image I2, and also outputs it to, for example, the storage device 26, where it is stored.

[0047] The master image registration unit 220 has an inspection window setting unit 221. The inspection window setting unit 221 receives instructions, for example, regarding the position, range, shape, etc. of the inspection window for the captured image I1 registered as the master image I2. Specifically, the inspection window setting unit 221 displays a GUI for setting the inspection window on the display device 4. When the user inputs the position, range, shape, etc. of the inspection window on the GUI using the touch panel 4a, the keyboard 3a, etc., the inspection window setting unit 221 receives the setting operation by the user and outputs the information related to the inspection window to, for example, the storage device 26, and stores it in the storage device 26. The inspection window setting unit 221 reflects the setting of the inspection range in the imaging range based on the positional relationship of the inspection window with respect to the captured image I1.

[0048] The inspection condition setting unit 230 is a unit that sets the tools to be used for inspection and the conditions under which the judgment results of each tool must satisfy to output a predetermined inspection result, i.e., the output conditions of the inspection result. The inspection condition setting unit 230 displays a GUI on the display device 4 that enables selection of the tools to be used for inspection and setting of the output conditions. Although not shown, this GUI has an area where any tool, such as a rule-based tool or a learning tool, can be selected as a tool. When a user selects a tool on the GUI using the touch panel 4a, keyboard 3a, or the like, the inspection condition setting unit 230 accepts the user's selection operation. The GUI also has an area where output conditions can be set. When a user sets output conditions on the GUI using the touch panel 4a, keyboard 3a, or the like, the inspection condition setting unit 230 accepts the user's operation to set the output conditions. Examples of output conditions include the condition "how many detected objects are required to output an ON signal as a 'good' inspection result" when the tool is the AI ​​object detection tool T2 described below, or the condition "what conditions must a character string meet to output an ON signal as a 'good' inspection result" when the judgment tool is the AI-OCR tool T3.

[0049] The learning tool setting unit 305 includes a learning data setting unit 310 and a classifier update unit 315. The learning data setting unit 310 is a unit that receives settings for treating the captured image I1 as learning data D1 and reflects the settings. The settings for treating the captured image I1 as learning data D1 include settings for label information. Specifically, the learning data setting unit 310 includes a learning image selection unit 310a, a learning data generation unit 310b, and a label information setting unit 310c.

[0050] The training image selection unit 310a is a unit that selects a captured image I1 to be treated as a training image. The training image selection unit 310a accepts an image selection operation by a user using the touch panel 4a, keyboard 3a, or the like on the GUI, and outputs a training image I3, which is image data of the selected training image, to the training data generation unit 310b. The image selected by the training image selection unit 310a may be an image other than the captured image I1. Alternatively, the configuration may be such that the training image I3 is automatically selected without accepting a selection operation by the user.

[0051] The label information setting unit 310c displays a GUI incorporating the training image I3 selected by the training image selection unit 310a on the display device 4, accepts a specification operation related to setting the label information, and sets the label information based on the information specified by the operation.

[0052] The training data generation unit 310b generates training data D1 based on the training image data and label information. The training data generation unit 310b acquires training image data from the training image selection unit 310a, acquires label information from the label information setting unit 310c, and outputs the generated training data D1 to the classifier update unit 315. In this way, the control unit 21 functions as the training data generation unit 310b that generates the training data D1.

[0053] The information that the training data generation unit 310b acquires to generate the training data D1 varies depending on the type of training tool used in the training data D1. Details will be described later, but for example, when generating training data D1 for training an AI-OCR tool T3, the training data generation unit 310b acquires training image data and, as label information, designated position information indicating the relative position of a character in a training image, designated character type information indicating the character type of the character, and designated size information indicating the size of the character in the image.

[0054] The learning data setting unit 310 may change the method of selecting learning images and the method of setting label information depending on the type of learning tool that uses the learning data D1 that reflects the settings of the learning data setting unit 310.

[0055] When the learning tool T1 is a learning tool that performs judgment based on the feature tendency of the entire image region included in the inspection window set by the inspection window setting unit 221, the entire image region of the selected training image I3 that corresponds to the inspection window is considered to be the image region to which a label is assigned. In other words, the label information setting unit 310c does not need to set the position to which the label is assigned. In this case, the label information setting unit 310c can set the label information by simply accepting the label specification, thereby reducing the operational steps related to setting the training data D1. The label specification here corresponds to the specification of a class when the label is a class label indicating a class.

[0056] Furthermore, in the above case, if the tool's judgment result is treated as the inspection result (e.g., "good" or "bad"), the user can easily grasp the characteristics of the image that should be classified into that class. In this case, the training image selection unit 310a may accept an image selection operation while the class of the assigned class label is fixed. For example, when the training image selection unit 310a accepts a selection operation of the training image I3 while displaying a GUI on the display device 4 guiding the user to "select an image you want to classify into the first class 'good'," the inspection window determines the image region to which the label is assigned. Furthermore, the label information setting unit 310c can set label information assuming that the image region is designated as the first class "good" without accepting an operation to specify a class. Because label information can be set based on a fixed class, the operational steps involved in setting the training data D1 are reduced.

[0057] When the learning tool is an AI object detection tool T2 that detects an object instructed by the user from an image region included in the inspection window set by the inspection window setting unit 221, i.e., the label information setting unit 310c accepts an operation to specify a position on the image and an operation to specify a label to be assigned to the position. The learning image I3 is an image containing the object to be detected, and includes an image region where the object is located and a background region where the object is not located. Because different class labels should be assigned to the image region where the object is located and the background region where the object is not located, it is necessary to accept the specification of the position to be labeled. Based on the accepted specification operation, the label information setting unit 310c outputs, as label information, a combination of specified position information indicating the position and specified class information based on the class label to be assigned to the position to the learning data generation unit 310b. In this way, the user assigns a class label of the class "object (foreground)" to a predetermined position on the image, but the operation to specify the class may be omitted. This is because AI object detection tool T2 is a learning tool for detecting objects, and so it is presumed that the position specified by the user on the image is the position to which a class label of the class "object" should be assigned. For example, label information setting unit 310c can omit the acceptance of the operation to specify a class by accepting the specification of a position on the image while displaying a guide such as "Please specify the position of the object in the image."

[0058] When the learning tool is an AI-OCR tool T3 that recognizes characters in an image area included in the inspection window set by the inspection window setting unit 211, i.e., the label information setting unit 310c accepts the designation of a position on the image, similar to the AI ​​object detection tool T2. Because the AI-OCR tool T3 not only detects characters but also identifies the character type of the characters, the label information setting unit 310c accepts an operation to designate a character type as a label designation. By accepting an operation to designate a character type, the label information setting unit 310c designates two classes: a class "character" and a character type class, and the designated character type information includes a class label for the class "character" and a class label for the character type class. The label information setting unit 310c outputs a combination of the designated position information and the designated character type information to the learning data generation unit 310b as label information.

[0059] The label information output by the label information setting unit 310c is not limited to specified position information and specified class information, and may also include size information and angle information. The size information and angle information may be specified by a size label or angle label separately assigned by the user. However, when the label information setting unit 310c allows the user to specify a position using a rectangular bounding box, the bounding box provides specified size information and specified angle information. The specified size information and specified angle information may be used to update the classifier.

[0060] In this embodiment, the learning data D1 for the AI-OCR tool T2 is set using specified position information, specified size information, and specified character type information. As will be described in detail later, in this embodiment, the specified size information is incorporated into the specified character type information. In other words, a character type whose size information is specified is treated as a single character type class. At this time, a position on the image is specified using a bounding box on the GUI, and the label information setting unit 310c identifies the character type class corresponding to the specified position by combining the size corresponding to the bounding box and the character type specified as the class of the label to be assigned to that position. Therefore, the label information when setting the learning data D1 for the AI-OCR tool T2 is a combination of a position in the captured image and a character type class specified for that position.

[0061] The classifier update unit 315 is a part that receives the learning data D1 generated by the learning data generation unit 310b and updates the classifier based on the learning data D1 using a method including a machine learning technique, and has a classification unit update unit 315a and a class addition unit 315b. In this embodiment, the classifier update unit 315 is a learning execution unit, and therefore the control unit 21 also functions as a learning execution unit that executes learning based on the learning data and controls on-site learning of the pre-trained model.

[0062] Note that pre-training here refers to the creation of a model using machine learning techniques by a business providing image inspection device S before providing the product to a user. Therefore, a pre-trained model refers to the model at the time the business provides image inspection device S. Furthermore, on-site training refers to the creation and modification of a model using techniques including machine learning, with the user who has received image inspection device S submitting training data.

[0063] The classifier update unit 315 updates the classifier used in the inspection by performing a predetermined update method based on the received learning data D1. In other words, the classifier update unit 315 performs learning using the learning data D1. The classifier to be updated varies depending on the selected learning tool. The classifier update unit 315 updates the classifier using a method including a machine learning technique that uses feature amounts (feature vectors or feature maps) extracted by inputting image data included in the learning data D1 to the feature extraction unit 510 of the classifier to be updated and label information corresponding to the image data.

[0064] The classification unit update unit 315a updates the classification unit, which is the determination unit 520 of the classifier to be updated, which classifies image data into classes. Specifically, the classification unit update unit 315a executes the feature extraction unit 510 of the classifier to be updated, inputs the training image data of the training data D1, and obtains a feature F (feature vector). As a result, the training data group is treated as a group of "combinations of feature vectors x class labels corresponding to each feature vector." Based on this group of combinations, a boundary surface is generated so that each class identified by the class label is distinguished in a feature space to which the feature vector of the feature F is mapped. When additional training data is available, the boundary surface is readjusted so that the additional feature F in the feature space is classified into the class associated with the additional feature F.

[0065] When the classifier to be updated by the classifier update unit 315 is a classifier executed by the learning tool T1, the classifier update unit 315a executes the feature extraction unit 510 to obtain a feature F1 from all of the training image data included in the training data D1. Then, the classifier update unit 315a updates the image classifier, which is the determination unit 520 that classifies the image data into classes set by the user. The image classifier is an SVM (support vector machine) as a linear classifier.

[0066] When the classifier to be updated by classifier update unit 315 is a classifier executed by AI object detection tool T2, classifier update unit 315a updates the image region classifier, just as in the case of a classifier executed by learning tool T1. The image region classifier is also a linear classifier, just like the image classifier. The difference compared to when classifier update unit 315 updates a classifier executed by learning tool T1 is that a partial region image of the training image data is input to feature extraction unit 510 to obtain feature F2 (feature vector), and the classifier is otherwise updated in the same way as in the case of learning tool T1.

[0067] The class addition unit 315b updates the classifier to be updated so that it can classify images into additional classes specified by the training data D1 in addition to the existing classes that could be classified before the update. Since the parameters of a classifier are typically optimized by comparing them with other classes, training is required using at least one image data set for each class to be classified. The parameter update by the classifier update unit 315a uses at least one image data set for each class to be classified. In on-site training, there is a limit to the amount of image data collected. If an attempt is made to update the parameters of the classifier using only image data that does not cover all classes, the image may be classified into the presented class, but the accuracy of classification into other classes or the accuracy of classification of images that should belong to other classes may decrease. Even if all classes are covered, it is difficult to prepare the same amount of image data in on-site training as in pre-training, so at least the accuracy of classification into existing classes decreases. Furthermore, when there are many classes to be classified, the amount of calculation required for training by the classifier update unit 315a tends to increase, resulting in a heavy learning load. In order to reduce the learning load and increase the degree of correspondence to classes added in additional learning, the class addition unit 315b updates the parameters by labeling the arrangement of extracted features obtained by the classification unit before the update. In this embodiment, when the classifier update unit 315 updates the classifier of the AI-OCR tool T3, the class addition unit 315b updates the classifier.

[0068] Similar to when the classifier of the AI ​​object detection tool T2 is updated, the class addition unit 315b inputs a partial region image of the training image data into the feature extraction unit 510 to obtain feature F3. What differs from when the AI ​​object detection tool T2 is updated is that the classifier's determination unit 520 is a character type output unit 523 obtained using a distance learning method in pre-learning. More specifically, the character type output unit 523 pre-learns a group of feature vectors of classifiable character type classes so that, in the embedding space, the distance between feature vectors of the same character type class is small and the distance between feature vectors of different character type classes is large. At this time, a representative vector is defined as an arrangement of representative feature vectors for each character type class, and the character type output unit 523 classifies the image data into the character type class of the representative vector whose distance is close to the feature vector of the image data and outputs the classification result. In the character type output unit 523 that has undergone such pre-training, a feature vector to be classified into a new character type class based on the training data D1 is arranged in the embedding space so as to be spaced apart from the representative vector of the pre-trained existing character type class if the feature vector is a character type different from the existing character type class. The class adding unit 315b takes advantage of this arrangement of feature vectors to define the feature F3 extracted from the training data D1 as a new representative vector. In other words, classification into a new character type class is possible while leaving the calculation parameters used up until the feature F (feature vector) is arranged in the embedding space unchanged. Furthermore, the class adding unit 315b adds a class that combines a specified size and a specified character type from the label information of the training data, i.e., the character type class "specified character type with specified size," as a new character type class.

[0069] The test execution unit 400 includes a rule determination unit 402 , a learning tool execution unit 403 , and a test result output unit 410 .

[0070] When the inspection tools used for the inspection include a rule-based tool, the rule determination unit 402 performs a determination on the captured image I1 of the inspection target based on the inspection tool. The parameters set by the parameter setting unit 205 are used.

[0071] The learning tool execution unit 403 is a part that executes the classifier when a learning tool is included in the inspection tools used for inspection, and outputs a judgment result for the captured image I1 of the inspection target. The classifier includes a feature extraction unit 510 and a judgment unit 520.

[0072] The feature extraction unit 510 is configured, for example, by a convolutional neural network. The feature extraction unit 510 is a part that extracts features that indicate the characteristics of input image data from the image data, and can also extract features for any position in the input image data, for example. The feature extraction unit 510 is a part that is pre-trained to extract features according to each learning tool, and has parameters that are used in the process of extracting the feature F from the image data.

[0073] The feature extraction unit 510 can extract the feature F in a format appropriate for the learning tool. For example, when learning tool T1 is executed, image data (captured image I1) is input to the feature extraction unit 510, and a multidimensional feature vector is obtained as the feature F. Furthermore, when a judgment is made utilizing spatial information of the input image, the image data is input to the feature extraction unit 510, and a feature map is obtained in which a feature vector is calculated for each convolution pixel corresponding to a certain range of pixel region in the input image data.

[0074] Determination unit 520 is a part that outputs a determination result based on the input feature amount F (feature vector or feature map). The determination unit 520 executed by the learning tool execution unit varies depending on the learning tool, and at least one of an image classification unit, an image region classification unit, a character type output unit 523, a size output unit 524, and an image region angle output unit 525 is executed.

[0075] The image classification unit classifies the feature F4 obtained from the feature extraction unit 510. The feature F4 is a feature vector. The image classification unit outputs information indicating which class the image data corresponding to the feature F4 belongs to. When the image classification unit is executed during execution of the learning tool T1, and the learning tool T1 is set to classify into a first class "good" and a second class "bad" in the settings by the learning tool setting unit 305, the image classification unit outputs information indicating whether the captured image I1 belongs to the first class or the second class, based on the feature F4 obtained from the captured image I1.

[0076] The image region classification unit classifies each convoluted pixel of the feature map based on the feature map as feature F5 obtained from the feature extraction unit 510. The image region classification unit classifies each convoluted pixel of the feature map into a class, and therefore can output, based on the spatial information contained in the feature map, where in the captured image I1 there is an image region classified into a specified class. When the image region classification unit is executed during execution of the AI ​​object detection tool T2, the image region classification unit classifies the convoluted pixels of the feature map F5 into the first class "object (foreground)" or the second class "background." This classification result can then be used to determine where in the captured image I1 corresponding to the feature map F5 there is an area image belonging to the first class "foreground," i.e., where the workpiece W as the object is present.

[0077] The image area classification unit executed during execution of the AI-OCR tool T3 classifies each convolved pixel of the feature map as feature F6 into the first class "foreground" and the second class "background." Based on this classification result, it is possible to determine where in the captured image I1 the character image area corresponding to feature F6 is located.

[0078] The character type output unit 523 is executed during execution of the AI-OCR tool T3. The character type output unit 523 outputs a character type class classification result based on the feature F7 obtained from the feature extraction unit 510. Because the feature F7 is a feature map, the character type output unit 523 can determine where in the captured image I1 corresponding to the feature F7 there is a character image area belonging to a given character type class. When the character type output unit 523 classifies an additional character type class, the classification result is used to obtain a determination result as to where in the image data corresponding to the input feature map a character image of a given character type exists, and how large it is.

[0079] The size output unit 524 is executed when the AI-OCR tool T3 is executed, and outputs the size of the corresponding image area based on the feature F8 output from the feature extraction unit 510. The feature F8 is a feature map. The size output unit 524, which is executed when the AI-OCR tool T3 is executed, outputs the size of the character image area corresponding to each pixel of the feature map.

[0080] The image area angle output unit 525 outputs information about the angle at which the corresponding image area exists, based on the feature F9 output from the feature extraction unit 510. The image area angle output unit 525, which is executed during execution of the AI ​​object detection tool T2, outputs angle information when the image area of ​​the input data corresponding to each pixel of the feature map as the feature F9 is the image area of ​​the workpiece W.

[0081] The inspection result output unit 410 generates and outputs inspection results based on the judgment results obtained by execution of the judgment unit 520, in accordance with the output conditions set by the inspection condition setting unit 230. For example, if the only tool used for inspection is the learning tool T1 and the output conditions are set to use the judgment results of the learning tool T1 as the inspection results, the inspection result output unit 410 obtains the class classification results of the image data of the captured image I1 as a first class "good" or a second class "bad" as the judgment results. If the obtained judgment result is the first class "good," the inspection result output unit 410 generates an inspection result indicating that the captured image I1 is a non-defective image, and outputs the inspection result from the I / O board 25 to a PLC or the like.

[0082] When the inspection condition setting unit 230 sets an inspection using the AI ​​object detection tool T2 and the output condition for outputting "ON" as the inspection result is "detection of one or more workpieces W," the inspection result output unit 410 obtains the classification result from the image area classification unit 521 as the judgment result and displays the judgment result indicating where and at what angle the workpieces W are present in the image data corresponding to the input feature F2. The inspection result output unit 410 then generates an inspection result based on the number of workpieces W in the image data indicated in the judgment result, displays the area corresponding to the detected workpieces W on the target image data, and outputs the inspection result to a PLC or the like. In this example, the number of workpieces W to be detected is specified as the output condition, but parameters such as the number of workpieces W may also be set as the judgment condition for the AI ​​object detection tool T2. Furthermore, when the angle of the image area is estimated in the judgment based on the AI ​​object detection tool T2, an angle-based condition such as "the detected workpieces W are within ±15° of the workpieces W in the master image" may also be set as the output condition.

[0083] It is possible to register multiple types of objects for object detection in the AI ​​object detection tool T2. For example, by registering a first object, a workpiece W1, and a second object, a workpiece W2, that has a different shape, color, and size from the first object, the inspection result generating unit 332a obtains as a determination result where and at what angle the workpiece W1 exists in the image data, and also obtains as a determination result where and at what angle the workpiece W2 exists in the image data.

[0084] When the inspection condition setting unit 230 sets an inspection using the AI-OCR tool T3 and sets an output condition stating that "the characters following 'best before' must be a string indicating a date within three days of the inspection date," the inspection result output unit 410 obtains a determination result indicating where in the image a character image is located. The inspection result output unit 410 generates an inspection result based on the character information indicated in the determination result, displays an area corresponding to the detected character image in the target image data and the character type corresponding to that area, and outputs the inspection result to a PLC or the like. At this time, for the existing character type class, the inspection result output unit 410 obtains a determination result indicating where in the image, what character image is located, and what size it is located in the image data of the captured image I1 corresponding to the input feature F, based on the classification result by the character type output unit 523 and the size output unit 524. For the additional character type class, the inspection result output unit 410 obtains a determination result indicating where in the image, what character image is located, and what size it is located in the image, based on the classification result by the character type output unit 523.

[0085] The inspection result output unit 410 may generate and output inspection results by combining multiple inspection tools, including rule-based tools and learning tools. For example, if the learning tool T1 determines the first class as "good," the AI ​​object detection tool T2 determines that there are two objects in the image, and the AI-OCR tool T3 determines that the expiration date is correctly printed in a predetermined position on the object detected by the AI ​​object detection tool T2, the inspection result output unit 410 may output an "ON" inspection result.

[0086] The inspection result output unit 410 also includes an additional training image designation unit 415. Based on the determination result obtained from the determination unit 520 and the inspection result generated based on the determination result, a GUI that accepts designation of a training image I3 to be added as a target for on-site training can be displayed on the display device 4. The user can designate a training image using the additional training image designation unit 415. When an image is designated by the additional training image designation unit 415, information related to the designated image is output to the training data setting unit 310. The training data setting unit 310 treats the image designated by the additional training image designation unit 415 in the same way as the training image I3 selected by the training image selection unit 310a. In other words, the training data generation unit 310b generates training data D1 based on the training image data of the training image I3 designated by the additional training image designation unit 415 and the label information set by the label information setting unit 310c.

[0087] (When setting up an image inspection device) 4 is a flowchart showing the control of the control unit 21 when setting up the image inspection device S. In step SA1 of imaging setting after the start, the control unit 21 accepts imaging setting parameters from the user via the imaging setting unit 100 and sets the imaging setting parameters. In step SA2, the imaging setting parameters set in step SA1 are applied to cause the imaging unit 1 to perform imaging. Image data acquired by the imaging unit 1 is stored in the storage device 26 and treated as a master image I2 and a learning image I3.

[0088] In step SA3, an input relating to the settings for the image inspection of the captured image I1 is accepted. In step SA4, input of signals from outside, output of signals to outside, communication settings, etc. are performed.

[0089] Fig. 5 is a flowchart showing the details of step SA3 when a learning tool is used as an inspection tool. In step SB1, the control unit 21 registers the image acquired in step SA2 shown in Fig. 4 as a master image I2 using the master image registration unit 220. In step SB2, the inspection window setting unit 221 sets an inspection range on the master image I2 registered in step SB1.

[0090] In step SB3, the control unit 21 selects the learning image I2 using the learning image selection unit 310a.

[0091] In step SB4, control unit 21 performs settings related to the judgment of the learning tool. More specifically, label information is set by label information setting unit 310c, learning data generation unit 310b generates learning data D1 based on the label information, and classifier update unit 315 updates the classifier based on learning data D1. For example, in learning tool T1, when a class is determined and the user is prompted to select image data belonging to that class, the user's selection of image data corresponds to the acceptance of the selection of learning image I3 in step SB3 and also to the acceptance of input for judgment setting in step SB4. In this way, depending on how the user's input is accepted, steps SB3 and SB4 may be performed simultaneously.

[0092] In cases where a label assignment target is not identified upon selection of the training image I3, such as in the AI-OCR tool T3, after the training image I3 is selected by the training image selection unit 310a in step SB3, the label information setting unit 310c accepts a user specification for step SB4 using the selected training image I3. However, by using a classifier to set step SB4, the acceptance of the user specification can be omitted. For example, class labels may be assigned to the training images selected in step SB3 using an existing classifier, and training data may be generated based on the assigned class labels. In such cases, steps SB3 and SB4 may be performed simultaneously, as in the above-described training tool T1.

[0093] In step SB5, the output conditions for generating inspection results based on the judgment results are set by the inspection condition setting unit 230. For example, if the inspection result is the result of counting the workpiece W using the AI ​​object detection tool T2 as the inspection tool, the number of areas that must be judged as targets by the AI ​​object detection tool T2 before the inspection result is output as "ON" is set.

[0094] (When operating image inspection equipment) Fig. 6 is a flowchart showing the control procedure of the control unit 21 when the image inspection device S is operating in an inspection mode including a learning tool. The image inspection device S can be switched between a setting mode in which settings are made as shown in Fig. 4 and Fig. 5, and an operation mode shown in Fig. 6, but this mode switching does not have to be clear, and continuous operation is also possible, for example, by temporarily switching to the setting mode to change settings while operating in the operation mode, and then returning to the operation mode.

[0095] In step SC1 after the start, the imaging unit 1 captures an image of the workpiece W, and acquires an image I1 of the workpiece W. In step SC2, the inspection execution unit 400 starts an inspection based on the set learning tool for the image of the workpiece W acquired in step SC1. First, in step SC3, based on the inspection range set in the inspection window setting unit 221, the range to be inspected is cut out from the image of the workpiece W acquired in step SC1.

[0096] In step SC4, feature extraction unit 510 extracts feature amounts from the image data of the inspection range extracted in step SC3. In step SC5, the feature amounts extracted in step SC4 are judged by judgment unit 520. As described above, in steps SC4 and SC5, the feature amounts change depending on the learning tool or the judger to be executed.

[0097] In step SC6, a judgment result is generated by the learning tool. If inspection conditions related to the judgment result are set in the learning tool, the judgment result is determined by comparing the result with the inspection conditions. For example, if a condition such as "the angle must be within ±15° of the workpiece W in the master image" is set for the AI ​​object detection tool T2, the judgment result is generated based on this condition. In step SC7, it is determined whether there are any unprocessed configured inspection tools. If there are any unprocessed configured inspection tools, the process proceeds to step SC3; if there are no unprocessed configured inspection tools, the process proceeds to step SC8. In step SC8, the inspection based on the configured inspection tools ends.

[0098] In step SC9, the inspection results are output by the inspection result output unit 410. If the inspection includes one inspection tool, the judgment result related to that inspection tool is output as the inspection result as is, but if the judgment results of multiple inspection tools are combined, the inspection results are generated and output based on the judgment results of each inspection tool.

[0099] (Details of AI-OCR) 7 is a conceptual diagram illustrating the feature extraction unit 510 and determination unit 520 involved in character type class classification when the learning tool execution unit 403 executes the AI-OCR tool T3. When executing the AI-OCR tool T3, the learning tool execution unit 403 executes the character type output unit 523 for character type class classification. The learning tool execution unit 403 also executes the character type feature extraction unit 510a as the feature extraction unit 510 that extracts the feature F7 for output by the character type output unit 523. During operation of the image inspection device S, when the imaging unit 1 acquires image data of a captured image I1 that includes a character in part, the image data is input to the pre-trained feature extraction unit 510. The character type feature extraction unit 510a is a computational model having a network structure, and is a convolutional neural network trained by machine learning techniques to extract the feature F7 suitable for the output of the character type output unit 523. The feature F7 is input to the character type output unit 523 to recognize characters in an image area that is part of the image of input image data, and classifies the image area into an existing character type (first character type) class. At this time, the learning tool execution unit 403 controls the calculation processing by the pre-trained character type output unit 523. Specifically, it controls the classification task into an existing character type class by the pre-trained character type output unit 523. Because the position where characters are printed is sometimes important in image inspection, the image area 500a on which the AI-OCR tool T3 is executed may be set in advance, and the feature F7 for the image area 500a may be input to the character type output unit 523.

[0100] There are multiple existing character type classes, and existing character type classes include, for example, alphabets, numbers, katakana, and hiragana. Characters may also include symbols, in which case the symbols are also included in existing character type classes. The storage device 26 stores character type class representative features that represent the character type class features of character images belonging to the existing character type classes described above. The character type class representative features stored in the storage device 26 can be read out when performing character recognition processing.

[0101] The character type output unit 523 is a linear classifier that reads from the storage device 26 character type class representative features that represent the character type class features of character images belonging to existing character type classes. Based on the character type class features extracted from a character image region at an arbitrary position in the input image data and the character type class representative features, the unit can output a value indicating the character type of the image region corresponding to the character type class features. For example, when the character type output unit 523 determines that the character type class features belong to a specified character type class corresponding to a combination of a character type and a specified size, the unit can output the size of the character image of the image region based on the specified size included in the specified character type class. Note that even when a character image combining multiple characters is registered as a single character type class, the feature F7 input to the character type output unit 523 is treated as a feature corresponding to a single character image region.

[0102] In this embodiment, the AI-OCR tool T3 estimates the size of the character image area and outputs a character recognition result corresponding to the size. Therefore, the learning tool execution unit 403 executes a size output unit 524, which outputs the size of the image area as a determination unit 520, and a size feature extraction unit 510c, which extracts a feature F8 suitable for the output of the size output unit 524. The size feature extraction unit 510c is a convolutional neural network pre-trained to extract a feature F8 suitable for the output of the size output unit 524, and the feature F8 is a feature map. The size output unit 524 estimates and outputs the location and size of the image area in the captured image I1 based on the feature F8. The AI-OCR tool T3 combines the output from the character type output unit 523 and the output from the size output unit 524 to output a character recognition result. This improves character recognition accuracy. For example, in the example of FIG. 7, the width of the character image area occupied by the letter "I" is smaller than the widths of the character image areas occupied by other characters. If no part for outputting the size of the character image area is provided, character recognition is performed with the size of the character image area fixed. If the size of the character image areas other than "I" is fixed, other characters will be included in the character image area of ​​"I", which will reduce the accuracy of character recognition. By outputting the size of the character image area by size output unit 524, character type output unit 523 can output character types assuming image areas of various sizes, thereby improving the accuracy of character recognition.

[0103] In the example shown in FIG. 7, the training of the feature extraction unit 510 is performed by the business operator that provides the image inspection device S, and on-site training by the user is not possible. The character type output unit 523 can be trained on-site by the user. That is, the parameters of the feature extraction unit 510, including the convolution operation that imposes a large training load, are fixed, and only additional training of the character type output unit 523 is possible. This allows additional training by the CPU installed in the control unit 21 without the need for a high-performance processing device such as a GPU. The size output unit 524 is also a part obtained by the business operator that provides the image inspection device S using machine learning techniques, i.e., a part provided by pre-training. However, re-training requires a large amount of computational processing, and therefore on-site training is not possible.

[0104] Furthermore, during inference, the inference by the feature extraction unit 510 is processed by, for example, an accelerator 21c specialized for convolution operations, and the inference by the determination unit 520 such as the character type output unit 523 can be processed by, for example, the DSP 21a or accelerator 21c.

[0105] The concept of pre-learning of the character type output unit 523 will be explained with reference to Fig. 8. This is also called distance learning. The circle E1 in Fig. 8 schematically shows the feature space, which is actually a multidimensional space. For example, when learning character types "A" and "B," the feature vectors of the same character type are learned to approach each other, and the feature vectors of different character types are learned to move away from each other.

[0106] 9 shows the state of the character type output unit 523 when the image inspection device S is provided to a user by a business operator, i.e., the state after pre-learning of the character types "A," "B," and "C" has been completed. As shown in this figure, basic alphanumeric characters have been pre-learned so that they can be classified, and representative vectors, which are representative feature vectors, have been registered.

[0107] After image inspection device S is provided to a user by the manufacturer, when the character "β" is read by image inspection device S, it is read as "B," as shown in Figure 10. Explaining this in more detail, first, because "β" and "B" have different characteristics, the feature vector extracted from the character image of "β" is located at a position away from the representative vector of character type class "B." However, because the representative vector in the vicinity of where the feature vector of "β" is located is a representative vector of character type class "B," the feature vector of character image "β" is classified into character type class "B."

[0108] If the character image of "β" is read as "B" in this way, the user may want the character image "β" to be read as "β". In other words, since the character image of "β" is similar in appearance to "B" that is registered as an existing character type class, it is read as "B" as shown in FIG. 10, and such a reading result is output by the inspection result output unit 410. After checking the inspection result, the user designates the character image "β" as training image I3 in the additional training image designation unit 415 and instructs the character type output unit 523 to perform on-site training.

[0109] Specifically, the learning data generator 310b generates learning data D1 that includes image region data including “β” and designated character type information that specifies the character type shown in the image region data including “β.” In this case, since the character type is “β,” the designated character type information is information that specifies “β.”

[0110] The class addition unit 315b of the classifier update unit 315 adjusts the parameters of the character type output unit 523 so as to classify image regions of the training image data included in the training data D1 generated by the training data generation unit 310b into the “β” class corresponding to the specified character type, i.e., into an additional character type class different from the existing character type classes. More specifically, the character type output unit 523 is adjusted so as to add the representative vector of “β” to the representative vectors read during execution of the character type output unit 523, and to classify image regions corresponding to feature vectors included in feature F7 and determined to be close to the representative vector of “β” into the character type “β.” At this time, the class addition unit 315b registers the representative vector of “β” based on the arrangement of the feature vector of “β” so that feature F3 extracted from the character image “β,” i.e., the feature vector of “β” arranged in the feature space by the character type output unit 523 before the update, is not classified into the existing character type class “B.” In this case, the feature vector of "B" remains registered, so when a character image closer to "B" than "β" is input, the feature F7 extracted from that character image remains positioned closer to the representative vector "B," and as a result, it is read as "B."

[0111] In this way, there is no need for a process of adjusting the arrangement of predetermined feature amounts in a multidimensional space, and feature vectors based on feature vectors of additionally specified image data are simply stored, so re-learning equivalent to pre-learning is not necessary, and on-site learning can be performed so that the user can classify new character types without deteriorating the reading performance of the image inspection device S at the time it is provided to the user by the business operator. Also, a single training image may be used for on-site learning.

[0112] (Example of character recognition) FIG. 11 specifically illustrates the flow of character recognition processing. First, during operation of the image inspection device S, image data of the captured image I1 acquired by the imaging unit 1 is input to the feature extraction unit 510. The feature extraction unit 510 is a pre-trained part that has undergone pre-training to extract features appropriate for the intended use. The feature extraction unit 510 includes a character type feature extraction unit 510a, an object feature extraction unit 510b, and a size feature extraction unit 510c. The object feature extraction unit 510b extracts a feature F6 indicating the object-like nature of an image region, i.e., whether the image region is in the foreground or background. The size feature extraction unit 510c extracts a feature F8 related to the size of the image region. For convenience, in this embodiment, each feature extraction unit is described as a separate unit. However, each feature extraction unit may be configured to extract individual features from features extracted using a common model structure. Furthermore, the same feature may be extracted for different purposes.

[0113] The three feature amounts extracted by the feature amount extraction unit 510 are sent to a determination unit 520 serving as a feature output unit.

[0114] The feature F6 extracted by the object feature extraction unit 510b is a feature map and is sent to the candidate area output unit 522a, which functions as an image area classification unit. Based on the feature F6, the candidate area output unit 522a calculates a probability value that the image area corresponding to each convolved pixel is an object. The probability value is calculated using a known method, such as calculating similarity with features that indicate the image area of ​​the object. The candidate area output unit 522a then designates image areas for which the probability value is equal to or greater than a threshold as candidate areas and outputs candidate area information Dxy indicating the position of the candidate area in the captured image I1. Note that the candidate area information does not include information regarding the size of the candidate area. In this embodiment, the candidate area information Dxy is output without considering spatial information in the captured image I1 in order to detect all closely spaced character image areas and character image areas that may be detected as multiple separate character image areas in the captured image I1. However, a heat map in which character likelihoods are arranged according to the spatial information of the captured image I1 may also be generated, and the candidate area information may be output based on the peak positions in the heat map. With this configuration, it is possible to eliminate candidate areas that overlap with the same character image area, thereby reducing the amount of calculation required for processing that uses candidate area information. When the candidate area information Dxy is output without taking into account the spatial information in the captured image I1 as in this embodiment, for example, for a captured image I1 in which the correct character image area is located near the peak position, it is possible to prevent omission of the candidate area information Dxy from being output, thereby further improving OCR accuracy.

[0115] The feature F7 extracted by the character type feature extraction unit 510a is sent to the character type output unit 523. The feature F7 is a feature map having an H dimension (height) and a C dimension (feature). The character type output unit 523 outputs a value indicating the character type of the image region based on the feature corresponding to the image region indicated in the candidate region information Dxy. More specifically, the feature f71 corresponding to the first candidate region P1 is identified from the feature F7 using the candidate region information Dxy, and the character type is output based on the feature f71. Here, the character type feature may be normalized to a length of 1. Note that the feature F6 and the feature F7 may be configured as a common feature, which reduces the processing load of the feature extraction unit 510. In this embodiment, the feature f71 is identified from the feature F7, but the feature extraction unit 510 may also be configured to extract features only for the candidate region based on the candidate region information Dxy. Furthermore, in this embodiment, the character type output unit 523 identifies the feature f71 based on the candidate area information Dxy to speed up the processing of the character type output unit 523, but the character type may also be output for all or part of the convolved pixels of the feature F7 as a feature map.

[0116] The character type output unit 523 calculates the product of the dictionary matrix DM shown in FIG. 11 and the feature value f71 as a feature vector to infer which character type class the candidate area P1 corresponding to the feature value f71 belongs to. The dictionary matrix DM contains as many representative vectors for each character type class as the number of registered characters (the number of character type classes). The character type output unit 523 calculates the product of the feature value f71 and the dictionary matrix DM to output the probability value that the candidate area P1 belongs to each character type class as a score. In this way, the character type output unit 523 calculates the score of each character type class for each candidate area specified by the candidate area information Dxy. Therefore, the information output by the character type output unit 523 is a combination of the candidate area information Dxy and character type information Dcl related to the character type of each candidate area. The character type information Dcl is composed of scores related to all character type classes indicated in the dictionary matrix DM, but may also be information related to some character type classes, such as character type classes with high scores, or may not include any scores.

[0117] The feature F8 extracted by the size feature extraction unit 510c is sent to the size output unit 524. The size output unit 524, which serves as the determination unit 520, outputs size information Dhw relating to the size (width, height) of the image region based on the feature F8. Because the feature F8 is a feature map, the size output unit 320f can output size information Dhw of the character image of each image region based on the feature of each convoluted pixel constituting the feature F8, and outputs a combination of the position of the image region included in the captured image I1 and the size information Dhw of the image region. For example, when the character type output unit 523 determines, based on the feature F7 extracted by the character type feature extraction unit 510a, that an image region does not belong to a designated character type class corresponding to the combination of character type and designated size (described later), it outputs size information Dhw of the image region, i.e., the character image region. Note that the feature map serving as the feature F8 is extracted without consideration of whether the image region is a character image region, and therefore values ​​are included in convoluted pixels that are not character image regions. In this embodiment, the size information is expressed as width and height, but may also be expressed as a sub-pixel position relative to the pixel that specifies the position of the image region.

[0118] The overlap determination unit 526 obtains the combination of candidate area information Dxy and character type information Dcl output by the character type output unit 320b, and size information Dhw for each image area, and determines whether the candidate areas overlap based on the character type class and size when the candidate area is a character image area.

[0119] When outputting candidate area information Dxy from the captured image I1, the character type class of the candidate area is not taken into consideration. Therefore, there is a risk that multiple candidate areas will be output from the same character image area. Therefore, if multiple adjacent candidate areas are determined to belong to the same character type class, it is preferable to determine one of the candidate areas as an overlapping candidate area and delete it. However, even if a candidate area is correctly determined from adjacent character images of the same character type in the captured image I1, multiple adjacent candidate areas will be determined to belong to the same character type class. Therefore, it is preferable to consider spatial overlap of each candidate area in the captured image I1. Therefore, the overlap determination unit 526 deletes overlapping candidate areas based on the candidate area information Dxy, character type information Dcl, and size information Dhw. As will be described in detail later, if an image area is determined to belong to the specified character type class corresponding to the combination of character type and specified size based on feature F7, the size of the candidate area in the captured image I1 is determined by the candidate area information Dxy and character type information Dcl. Therefore, spatial overlap in the captured image I1 does not necessarily need to be determined based on the size information Dhw. As a specific processing method, known techniques such as non-maximum suppression can be used. Depending on the determination result, the overlap determination unit 526 deletes the candidate areas and outputs the character reading result R1 of the captured image I1 based on the remaining candidate areas.

[0120] By going through the above processing flow, the determination unit 520 as a feature output unit outputs the reading result, and the user can obtain the reading result including a reading result display in which a rectangular GUI based on the output result by the size output unit 320f is superimposed and character type information based on the output result by the character type output unit 320b is displayed adjacent to the rectangular GUI.

[0121] In this embodiment, overlapping candidate areas are eliminated based on the candidate area information Dxy, character type information Dcl, and size information Dhw, but candidate areas may also be eliminated based on the candidate area information Dxy and the character type information Dcl, or based only on the candidate area information Dxy. In this case, as described above, there is a risk of a decrease in the accuracy of overlap determination, but the amount of calculation processing (processing by the character type output unit 523 in this embodiment) whose calculation amount increases depending on the number of candidate areas can be reduced. The same applies when overlap determination is performed based on the candidate area information Dxy and the size information Dhw.

[0122] (Dictionary registration details) 12 is a diagram showing a specific example of dictionary registration processing. The dictionary registration processing is on-the-job training of the AI-OCR tool T3, including updating of the character type output unit 523 by the class addition unit 315b. For example, a case where a user registers the character "β" in the dictionary will be described. The imaging unit 1 acquires image data of a captured image I1 that includes the character "β" in part, and the user selects the captured image I1 as a learning image I3 using the learning image selection unit 310a.

[0123] FIG. 13 is a diagram illustrating a GUI 540 generated by the training data setting unit 310. The user sets label information for a training image I3 using the label information setting unit 310c. When registering a training image I3 in a dictionary, the label information setting unit 310c generates a designation receiving GUI 540 for receiving the user's designation of a character position, a character size, and a label, and displays the GUI 540 on the display device 4. The designation receiving GUI 540 includes a target image display area 541 that displays the training image I3, and a label information display area 542 that displays label information to be set for the training image I3. A rectangular box 544, whose shape changes in response to a user operation, is superimposed on the target image display area 541. The label information display area 542 displays the designated character type, designated position, and designated size for each box 544. When setting label information for the training image I3 using the label information setting unit 310c, the user first manipulates the position and size of the box 544 so that it corresponds to the character image area of ​​the training image I3 displayed in the target image display area 541. When the user manipulates the box 544, the values ​​of the designated position and the designated size in the label information display area 542 change depending on the manipulation. The label information display area 542 also functions as an area for accepting user input, and the user inputs a character type corresponding to the character area into the designated character type in the label information display area 542. When a character type is input into the label information display area 542, the input character type is displayed in the character type input area 543 near the corresponding box 544 in the target image display area 541. The user can add or delete boxes 544 depending on the number of character areas on the image 500, and the label information display area 542 has display areas corresponding to the number of boxes 544. In addition, the areas in the label information display area 542 where the designated position and designated size are displayed also function as input areas. The specified position can be input as x-coordinates and y-coordinates to specify the position of the character, and the specified size can be input as width and height to specify the character size. The input operation of the character type may be performed in a character type input area 543 that is superimposed on the target image display area 541.

[0124] The example in FIG. 13 shows GUI 540 when the user performs on-site learning of two additional "β" characters with different fonts. The two "β" characters in learning image I3 look different because they are in different fonts. The user places two boxes 544 for the two character image regions "β" and specifies the position and size of the character image regions "β." For convenience, the box 544 assigned the number 1 in FIG. 13 is referred to as box 544a, and the box 544 assigned the number 2 is referred to as box 544b. The label information display area 542 displays the position and size of the corresponding character image regions according to boxes 544a and 544b, but as shown in label information display area 542, the two "β" characters are also different sizes. These two character image regions have the same character type, "β." However, when these two character image regions are additionally subjected to in-situ learning, the feature vector representing "β" specified in box 544a in GUI 540 and the feature vector representing "β" specified in box 544b are added as separate classes to dictionary matrix DM. In this manner, training data generation unit 310b generates training data D1 having designated size information indicating the size of the character image region in the image data of training image I3. The generated training data D1 is input to classifier update unit 315. In classifier update unit 315, feature extraction unit 510 executes character type feature extraction unit 510a to extract feature F7 from training image I3, and executes object feature extraction unit 510b to extract feature F6 from training image I3.

[0125] Returning to FIG. 12 , the details of the processing by the classifier update unit 315 will be described. The class addition unit 315b, which functions as the classifier update unit 315, executes the character type feature extraction unit 510a to extract feature F7 and identifies feature F3 corresponding to the character image region designated by the user based on the specified position information (x and y coordinates designated by the user) included in the training data D1. In this embodiment, two character image regions are specified by boxes 544a and 544b in the training image I3 included in the training data D1, so two feature vectors are identified, and the class addition unit 315b adds the two identified feature vectors to the dictionary matrix DM as is. Boxes 544a and 544b are character image regions of the same character type, but are registered as feature vectors corresponding to different classes. In this embodiment, the identified feature is added to the dictionary matrix DM as is; however, a feature that can be sufficiently distinguished from existing representative features based on the identified feature may be selected and added to the dictionary matrix DM.

[0126] The classification unit update unit 315a, which functions as the classifier update unit 315, executes the object feature extraction unit 510b to extract feature F6 as a feature map, and extracts feature F10 corresponding to an image area designated by the user based on designated position information included in the training data D1. Here, the image area identified by the designated position information is an image area designated by the user as a character image area, so feature F10 corresponds to an image area treated as a foreground area, not a background area. Therefore, the candidate area output unit 522a is updated so that the score, which is the probability value that the image area corresponding to feature F10 is an object area, is higher than the score used by the candidate area output unit 522a when determining a candidate area. The method for updating the candidate area output unit 522a is not particularly limited. For example, the updated foreground output unit 320c can be trained by a known method, as long as it is configured to incorporate SVM learning or cosine similarity evaluation.

[0127] While the learning tool execution unit 403 executes the size feature extraction unit 510c, the classifier update unit 315 does not execute the size feature extraction unit 510c during the dictionary registration process. Generally, updating a model that infers the size of a specific image region included in input image data using machine learning techniques increases the amount of calculation. In other words, updating the size output unit 524 using machine learning techniques so that size information Dhw based on the specified size information for the image region in the learning data D1 is difficult, resulting in a heavy learning load that makes on-site learning difficult. For this reason, the size output unit 524 is not updated during on-site learning in this embodiment, and therefore the size feature extraction unit 510c is not executed.

[0128] For this reason, in the on-site learning of this embodiment, the specified size information of the training data D1, i.e., the width and height specified by the user via the specification receiving GUI 540, is reflected in the dictionary matrix DM and applied during inference after the dictionary registration process. More specifically, the added character type class features are registered in the dictionary matrix DM as features corresponding to a combination of a character type and an image region size. Because the character image "β" specified in box 544a and the character image "β" specified in box 544b are of the same character type "β" but different sizes, the feature vector specified in box 544a is registered in the dictionary matrix DM as a feature vector corresponding to the combination of the character type "β" and the size specified in box 544a, and the feature vector specified in box 544b is registered as a feature vector corresponding to the combination of the character type "β" and the size specified in box 544b. When the learning tool execution unit 403 executes the character type output unit 523, if it determines that the feature value f71 belongs to the added character type class, it determines that the candidate area P1 is a character image of the character type specified by the learning data D1 and is an image area having the size specified by the learning data D1. In the example of Fig. 12, two character type classes with the character type "β" are added, and if it is determined that the candidate area P1 belongs to one of the two added character type classes, the character type of the candidate area P1 is determined to be "β", and the size of the candidate area P1 reflects the size of the box 544 used to identify the feature vector of the classified class.

[0129] The class adding unit 315b updates the character type output unit 523 not only when adding a character type other than those that can be classified by pre-learning, but also when adding a character image area that could not be read with an existing character type. If the inspection results output by the inspection execution unit 400 include a captured image I1 that failed to be read, the captured image I1 is used as a learning image I3 and subjected to on-site learning, thereby improving the accuracy of the character type output unit 523.

[0130] In this case, a typical example of a captured image I1 that fails to be read is a captured image I1 that includes a faded character image region. In this case, if the learning tool execution unit 403 is configured to execute the candidate area output unit 522a separately from the character type output unit 523 when executing the AI-OCR tool T3, the degradation of reading accuracy due to on-site learning can be reduced. More specifically, the faded character image region may be similar in feature quantity to the background image region. In this case, if a feature vector corresponding to the faded character image region is added to the dictionary matrix DM, the character type output unit may classify the background image region into the class corresponding to the faded character image region, resulting in a false detection. In this embodiment, the candidate area output unit 522a is executed separately from the character type output unit 523, and the character type output unit 523 performs processing based on the output from the candidate area output unit 522a, thereby reducing the risk of false detection.

[0131] (Relearn after driving) As described above, the image inspection device S is a device capable of executing a pre-trained character recognition model. After the image inspection device S is put into operation (also referred to as after operation), misreadings may occur in the character recognition model. However, when classifying a character, the image inspection device S of this embodiment cuts out an image area, which is a partial area of ​​the image of the input image data, and the image area can be used to allow the user to provide instructions on the misreading results. The image inspection device S can then accept instructions from the user on the misreading results, and by accepting the user's instructions on the misreading results, the image inspection device S can execute a model that requires a small amount of calculation for learning based on the user's instructions. Furthermore, because the image area is cut out, the user can provide instructions without setting the image area.

[0132] The process of enabling relearning of a pre-trained model after operation of the image inspection device S will be described with reference to the flowchart shown in FIG. 14. This flowchart starts after the additional training image designation unit 415 has designated a training image. In step SD1, the control unit 21 acquires the operation results of the image inspection device S and displays them on the display device 4. This step can be called a driving result display step or a presentation step for presenting the driving results to the user. FIG. 15 shows a driving result display screen 600 as an example of a GUI for displaying the driving results. The driving result display screen 600 can be generated by the control unit 21 and displayed on the display device 4.

[0133] The driving result display screen 600 has a driving result display area 601 and an operation reception area 602. The operation reception area 602 has a reading setting button 602a for setting reading, a character learning button 602b for starting character learning, a judgment condition button 602c for setting judgment conditions, a button 602d for performing test driving / character learning, and a learning content list button 602e for displaying a list of learning contents. Each of the buttons 602a, 602b, 602c, 602d, and 602e is configured as an image displayed on the display device 4, and therefore can be touch operated using the touch panel 4a. However, this is not limited to this, and the buttons may be configured as physically operable buttons. The same applies to buttons hereinafter.

[0134] The driving result display area 601 displays the image input to the model. In this example, "100 kΩ" is the character string to be read by the model. The driving result display area 601 also displays a first box 601a surrounding the image area including the character string to be read and a second box 601b surrounding the area of ​​the characters read by the model, which are superimposed on each other, and further displays "1", "0", and "0" as the reading results by the model. The driving result display screen 600 allows the user to understand the reading results.

[0135] In the example shown in Figure 15, "k" and "Ω" are misread. "k" is a character that is pre-registered in image inspection device S, but it is a character that has been misread. "Ω" is a character that has not been pre-registered in image inspection device S, so it is a character that has been misread. In other words, "k" and "Ω" have in common the fact that they are both misread characters, but they differ in whether they are characters that are pre-registered in image inspection device S or not.

[0136] After the driving result display step of step SD1 in Fig. 14, the process proceeds to step SD2, where the control unit 21 accepts the user's designation of a character image area. Specifically, when the user operates the character learning button 602b shown in Fig. 15, the control unit 21 displays an additional learning window 610 shown in Fig. 16 in the operation acceptance area 602 of the driving result display screen 600. The additional learning window 610 is provided with an add button 610a for adding a character to be learned and a start learning button 610b for starting learning.

[0137] When the user operates the add button 610a, the control unit 21 displays the additional learning window 620 shown in FIG. 17 in the operation reception area 602 of the driving result display screen 600. The additional learning window 620 allows for reading settings. This example shows the case of additionally learning the letter "k." When making the reading settings, the user first forms a third box 601c that surrounds the letter "k," which is the letter to be learned, on the image displayed in the driving result display area 601. The third box 601c can be formed by a touch operation or the like by the user. For example, the third box 601c can be formed by dragging from the top left to the bottom right of the letter to be learned. Once the third box 601c is formed, the control unit 21 identifies the character image area surrounded by the third box 601c and accepts the designation of the character image area.

[0138] 14, the process proceeds to step SD3, where control unit 21 accepts the designation of the character type of the characters included in the character image area accepted in step SD2. Additional learning window 620 shown in Fig. 17 allows the user to set whether or not the character surrounded by third box 601c is to be learned. Specifically, additional learning window 620 is provided with character setting area 620a for setting the character surrounded by third box 601c as a correct character, and reading exclusion setting area 620b for setting the character surrounded by third box 601c as a character not to be read.

[0139] When the setting button 620c in the character setting area 620a is operated, the control unit 21 generates a character type selection window 630 shown in FIG. 18 and displays it on the display device 4. The character type selection window 630 has a character type display area 631 in which a list of character types (registered character types) pre-registered in the image inspection device S is displayed. The character type display area 631 displays multiple tabs, allowing the user to select any tab. The multiple tabs include, for example, a tab for categorized numbers / symbols, a tab for categorized uppercase letters, a tab for categorized lowercase letters, and a user dictionary tab for categorized characters registered by the user. When the user performs an operation to select a desired tab, the character types classified in the selected tab are displayed in a list in the character type display area 631. The character types displayed in the character type display area 631 can be selected by the user, and the example shown in FIG. 18 shows a case where the user has selected "k."

[0140] When the control unit 21 identifies the character type selected by the user, it displays the character type selected by the user in the character setting area 620a, as shown in Fig. 19. This allows the user to confirm the character type selected by the user.

[0141] 20 shows an example of accepting the specification of a character type that is not registered in the image inspection device S, i.e., an unregistered character type. When "Ω" is a character type that is not registered in the image inspection device S, the user first forms a fourth box 601d that surrounds the character "Ω" that the user wants to learn on the image displayed in the driving result display area 601. The fourth box 601d can be formed in the same manner as the third box 601c.

[0142] After the fourth box 601d is formed, when the user operates the setting button 620c in the character setting area 620a, the control unit 21 generates a character type selection window 630 shown in Fig. 21 and displays it on the display device 4. This example shows a case where no character type is registered in the user dictionary, and a new registration button 631a for registering a new character type is displayed.

[0143] When the new registration button 631a is operated, the control unit 21 displays a new registration window 640 shown in Fig. 22. The new registration window 640 accepts the input of only one character. However, if it is desired to input only one character out of the multiple characters that make up a phrase, it is also possible to accept an operation of deleting unnecessary characters and leaving only one necessary character after the input of the phrase is once accepted. This makes it easier to input characters. Fig. 23 shows the state in which the input of "Ω" has been completed.

[0144] After the character type specification is accepted as described above, the process proceeds to step SD4 shown in Fig. 14. In step SD4, the control unit 21 accepts a learning execution instruction from the user. However, before accepting the learning execution instruction, the character type accepted in steps SD2 and SD3 is displayed in the character type display area 612 of the additional learning window 610 shown in Fig. 24. When the user operates the learning start button 610b, the process proceeds to step SD5 in Fig. 14, where the control unit 21 executes additional learning of the model.

[0145] After step SD5, a test run can be performed to test the effectiveness of the additional learning. When the control unit 21 receives a test run instruction from the user, the control unit 21 executes a process in which the character recognition model after the additional learning classifies the input image area into character type classes. FIG. 25 shows an example of a result display screen 650 that displays the results of the test run. This example shows that "k" and "Ω" that underwent additional learning have been successfully classified. If a misreading occurs during the test run, the user can check this on the result display screen 650, and can make corrections at this stage in steps SD2 and SD3 of FIG. 14.

[0146] (template specification) FIG. 26 is a diagram showing an example of an in-operation screen 700 that is displayed during operation when inspecting a date that is printed or the like on the workpiece W. The in-operation screen 700 is a screen that is generated by the control unit 21 and displayed on the display device 4, and is provided with an image display area 701 in which an image input to the model is displayed, and an additional learning button 702 that is operated when additional learning is performed. In the example shown in FIG. 26, a date is registered as the master. The image display area 701 displays the master date and time and the read date and time read by the model.

[0147] Here, let's assume that workpieces W with a manufacturing date printed on them are to be inspected. If "January 1, 2024" is registered as the master in the image inspection device S, the workpieces W manufactured on "January 1, 2024" will be inspected. However, the workpieces W manufactured the following day will have "January 2, 2024" printed on them. Therefore, if the master remains "January 1, 2024," all of the workpieces W that are actually fine will be judged as "defective." This requires the user to change the master date setting every day before operation, which is a significant burden. The same applies to expiration dates, which change in relation to the manufacturing date.

[0148] In response to this, this embodiment is equipped with a template designation function. With the template designation function, when the user designates the date format, for example, the date setting is automatically changed using the current date and time information held internally by the image inspection device S. A specific setting method when using the template designation function will be described below.

[0149] First, the control unit 21 generates a selection window 710 (shown in FIG. 27) for accepting the selection of an OCR mode and displays it on the display device 4. The selection window 710 is provided with a character string button 710a for selecting a character string mode suitable for reading fixed character strings (character strings that do not change), a date and time button 710b for selecting a date and time mode suitable for reading variable character strings such as the date and time, and a custom button 710c that allows customization of the judgment conditions according to the print content of the work W.

[0150] When the date and time button 710b is operated by the user, the control unit 21 generates a tool setting screen 720 as shown in Fig. 28 and displays it on the display device 4, and also generates an editing window 722 and displays it on the display device 4. The tool setting screen 720 is provided with an image display area 721 in which the image input to the model is displayed. The editing window 722 displays options such as year / month / date, day / month / year, month / day / year, month / year, hour / minute, hour, minute, etc., and the user selects from these options an option that suits the object of inspection.

[0151] FIG. 29 shows a case where the user selects a date. In this case, the control unit 21 generates a judgment condition setting window 730 for setting judgment conditions and displays it on the display device 4. The judgment condition setting window 730 includes a master date / time input area 731 for inputting a master date / time and a synchronization setting area 732 for setting calendar synchronization. The date / time input in the master date / time input area 731 is registered as the master. In the synchronization setting area 732, the user can select either a synchronous mode, which automatically changes the date setting using the date / time information within the image inspection device S, or an asynchronous mode, which does not synchronize with the date / time information within the image inspection device S. Changing the master date / time using the date / time information within the image inspection device S is called calendar synchronization. When the synchronous mode is selected, the control unit 21 acquires date / time information within the image inspection device S, and if it determines that the acquired date / time is the next day, it automatically changes the master date / time to the next day. The same applies to the time; the control unit 21 acquires date / time information within the image inspection device S and automatically changes the acquired time to the master time.

[0152] When the synchronization setting button 732a shown in FIG. 29 is operated, the control unit 21 generates an offset setting window 740 shown in FIG. 30 and displays it on the display device 4. The offset setting window 740 is a window for accepting the setting of the offset amount when the date and time registered as the master is to be offset by a predetermined date and time. The offset setting window 740 includes a setting area 741 in which the offset amount can be set in months, days, and minutes, a tolerance setting area 742 for setting the allowable error range, a calculation order setting area 743 for setting the offset calculation order, a one-month offset setting area 744, a time display format setting area 745, and a setting confirmation area 746. The offset setting window 740 allows the user to easily set any offset amount. For example, as shown in the setting confirmation area 746, if the manufacturing date is "January 1, 2023," and a 12-month offset is applied, the master date can be confirmed to be "January 1, 2024." The set offset amount is reflected in the master by the control unit 21.

[0153] (Custom Settings) The image inspection device S may be configured to allow the user to customize the master string when a template cannot be specified. FIG. 31 illustrates a case in which a master string is changed for each image capture. The "0001" in the lower right corner of the image displayed in the operation result display area 601 is a count-up master string that increments by one for each image capture, similar to a serial number, for example. In the case of such a master string, the user operates the custom button 710c in the selection window 710 shown in FIG. 27. The control unit 21 then generates a judgment condition setting window 750 shown in FIG. 32 for setting judgment conditions for custom settings and displays it on the display device 4. The judgment condition setting window 750 includes a setting button 751 for executing format setting. When the setting button 751 is operated, the control unit 21 generates a format setting window 770 shown in FIG. 33 and displays it on the display device 4. The format setting window 770 includes a division setting area 771 for dividing the master string into elements. In the example shown in FIG. 33, the character string is made up of eight elements: "best before date," "2024," ".", "01," ".", "01," "+K," and "0001." In this case, as shown in FIG. 34, separators are set between "best before date" and "2024," between "2024" and ".", between "." and "01," between "01" and ".", between "." and "01," between "01" and "+K," and between "+K" and "0001." To set separators, an operation for selecting the mark indicating the separator can be performed.

[0154] After dividing the master character string into elements, content can be specified for each element. After division is complete, a content setting area 772 is displayed in the format setting window 770, as shown in FIG. 35. A list of multiple elements is displayed in the content setting area 772. When the user selects one element, a first selection window 772a shown in FIG. 36 or a second selection window 772b shown in FIG. 37 is displayed. The user can select the desired content from the multiple options displayed in the first selection window 772a or the second selection window 772b. The selected content is reflected in the master. At this time, if calendar synchronization is performed, an offset setting window 740 shown in FIG. 30 is displayed on the display device 4, allowing the offset amount to be set.

[0155] Furthermore, if the element is a count-up type character string or a count-down type character string, the control unit 21 generates a count-up / down setting window 780 as shown in Fig. 38 and displays it on the display device 4. The count-up / down setting window 780 has a basic setting area 781, a count-up / reset setting area 782, and a number-of-digits setting area 783. In the basic setting area 781, it is possible to set a start value, an end value, an addition / subtraction value, and a serial number value. In the count-up / reset setting area 782, it is possible to set the trigger number, etc. In the number-of-digits setting area 783, it is possible to set whether to enable zero padding and the number of digits.

[0156] 39 shows a replacement setting window 790. In this replacement setting window 790, it is possible to link the display of months corresponding to each country with the calendar. This makes it possible to accommodate the different languages ​​and date display formats depending on the country.

[0157] Fig. 40 shows a link window 800 for linking the notation of the first, middle, and last days of the month with a calendar. When making settings in the link window 800 for linking the notation of the first, middle, and last days of the month with a calendar, the same input items such as the first, middle, and last days of the month can be input in a lump using a bulk input window 810 as shown in Fig. 41.

[0158] (OCR circular search function) As shown in FIG. 42, an image may contain multiple characters arranged in an arc shape. In this case, in this embodiment, rather than setting one tool per character, it is possible to specify the character image and input it into the model, and therefore, by specifying "arc" in the editing window 722, it is possible to specify a character string range 721a in which the characters are arranged in an arc shape. However, as shown in FIG. 43, an image in which the phase of the character string arranged in an arc shape has changed may be input. With a general position correction tool, it is possible to determine the center position of the circle, but it is not possible to determine the phase, so the character string to be detected may fall outside the character string range 721a.

[0159] To address this issue, the image inspection device S is able to handle changes in the phase of character strings arranged in an arc using the OCR circular search function. Specifically, as shown in Fig. 44, the control unit 21 generates an arc setting window 725 and displays it on the display device 4. The arc setting window 725 has a reading direction designation area 725a for designating whether the reading direction is clockwise or counterclockwise, and a circular search setting area 725b for setting whether the circular search function is enabled. When the circular search is enabled in the circular search setting area 725b, a circular search range 721b is displayed in the image display area 721 in addition to the character string range 721a.

[0160] Figure 45 shows test images with different character string phases, and additional learning can be performed using such test images. This makes it possible to perform inspection even if images with different character string phases are input while the image inspection device S is in operation, as shown in Figure 46.

[0161] Figure 47 shows a case where a part of the arc-shaped character string registered as the master is missing, resulting in a bad inspection result. When such a missing part is found, the model is configured to read out the character string with the closest edit distance.

[0162] 48 is an image showing an additional learning confirmation window 820 for confirming whether or not additional learning will be performed with an input image in which the phases of character strings arranged in an arc are different. If the user wishes to perform additional learning with the image displayed in the additional learning confirmation window 820, the user can select OK, or if the user does not wish to perform additional learning, select Cancel.

[0163] Figure 49 is a diagram illustrating the case where the third box 601c surrounding the characters to be learned is moved during additional learning. The direction of movement of the third box 601c is set so that when the user moves the third box 601c, the third box 601c automatically moves along the circular search range 721b. This makes it easy to move the third box 601c in the desired direction. When reading a character string arranged in an arc using a model, it is first expanded into a rectangular band before reading is performed.

[0164] The above-described embodiments are merely examples in all respects and should not be construed as limiting. Furthermore, all modifications and variations within the scope of the claims are within the scope of the present invention. [Industrial Applicability]

[0165] As described above, the image inspection device according to the present disclosure can be used to inspect various types of workpieces. [Explanation of symbols]

[0166] 21 Control section 26 Storage device (storage unit) 310b Learning data generation unit 315 Classifier update unit (learning execution unit) 320 Feature Extraction Unit S Image inspection device

Claims

1. An image inspection device that executes a pre-trained character recognition model that classifies an image area that is a partial area of ​​an image of input image data into a first character type class, a control unit that functions as a training data generation unit that executes the pre-trained character recognition model and generates training data, and a training execution unit that executes updating of the pre-trained model based on the training data; The pre-trained model is a feature extraction unit that extracts, from the input image data, a feature that indicates a feature of the image data; a character type output unit that outputs a character type based on the feature amount extracted by the feature amount extraction unit, the learning data generation unit generates learning data including learning image data, designated position information indicating a relative position of a character in an image of the learning image data, designated character type information indicating a character type of the character located at the relative position, and designated size information indicating a size of the character in the image; The learning execution unit updates the pre-trained model so as to classify the feature that is extracted when the training image data is input to the feature extraction unit and corresponds to the relative position indicated in the specified position information into a specified character type class that corresponds to the combination of the character type indicated in the specified character type information and the size indicated in the specified size information.

2. 2. The image inspection device according to claim 1, The control unit includes a convolutional operation network inference accelerator that executes operations of a convolutional neural network as the feature extraction unit.

3. 2. The image inspection device according to claim 1, The feature extraction unit a character type class feature extraction unit that extracts features related to a character type class of the image region; a foreground / background feature extraction unit that extracts a foreground feature that is a feature indicating whether the image region is foreground or background, a foreground output unit that determines whether the image region is a foreground based on the foreground feature amount; The character type output unit outputs a value indicating the character type of the image area corresponding to the image area determined to be the foreground by the foreground output unit, based on the character type feature corresponding to the image area.

4. 2. The image inspection device according to claim 1, The feature extraction unit a character type class feature extraction unit that extracts features related to a character type class of the image region; a size feature extraction unit that extracts a size feature that is a feature related to the size of the image region; The character type output unit an image inspection device that outputs a size of the character image in the image region based on a designated size included in the designated character type class when the character type class feature belongs to the designated character type class corresponding to the combination of the character type and the size.

5. 5. The image inspection device according to claim 4, a size output unit that outputs a size of the character image in the image region based on the size feature amount; The size output unit outputs the size of the character image when the character type class feature does not belong to the designated character type class corresponding to the combination of the character type and the designated size.

Images