Image sensor

Abstract

To facilitate learning of a user while taking advantage of the performance of a trained model for character recognition.SOLUTION: An image sensor includes: a control unit which functions as a learning data generation unit and a learning execution unit; and a storage unit which stores character type class representative features that represent features of a character image for each class of a first character type class. A pre-trained model includes a feature extraction unit and a character type output unit. The learning data generation unit generates learning data having image region data and designated character type information. The learning execution unit causes the storage unit to store character type class representative features of a second character type class, which is different from the character type class representative features of the first character type class, based on the features extracted from the image region data of the learning data, so that the image region data of the learning data may be classified into a second character type which is a designated character type.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to image sensors capable of running trained models that perform classification tasks for character recognition. [Background technology]

[0002] For example, in the factory automation industry, image inspection devices are widely known to replace the process of visually inspecting workpieces by humans. Image inspection devices capture images of workpieces and judge the captured images based on certain criteria. If the image inspection device is an image sensor, the judgment results are output via I / O to an external device such as a programmable logic controller (PLC).

[0003] When the criteria used for judgment by an image inspection device are based on certain image features, even changes in the image features that would not affect the results of a visual inspection by a human may affect the judgment results of the image inspection device. Therefore, an image inspection device that uses an image recognition model obtained by machine learning to obtain judgment results closer to those of a visual inspection is known, such as the image inspection device disclosed in Patent Document 1. The image inspection device of Patent Document 1 is configured to generate a classifier that distinguishes between good and defective product images by learning good product images that have been assigned attributes as good products and defective product images that have been assigned attributes as defective products, and to input newly acquired images into the classifier during operation to make a pass / fail judgment. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-187072 Summary of the Invention [Problem to be solved by the invention]

[0005] Generally, creating an image recognition model obtained through machine learning requires a large amount of image data, and it is difficult for each user to collect the necessary image data, or the burden on the user increases. Various methods have been considered to reduce the burden of collecting image data required for machine learning. However, when creating a model that judges the quality of input images, such as the image inspection device described in Patent Document 1, the objects to be inspected vary from user to user, and the standards for determining whether an image is good or bad also vary. This requires the user to assign attributes for good products and attributes for bad products to each piece of image data, which is likely to further increase the burden on the user before operation. Another issue is the high computational load required when creating a model using machine learning techniques.

[0006] Some image inspection devices are equipped with optical character recognition (OCR) technology, which recognizes character images—the text portion of a captured image of a workpiece bearing text—and uses an image recognition model to identify the characters on the workpiece and determine whether the characters conform to a criterion. Character recognition results are generally consistent across users. In other words, while the criteria for determining pass / fail differ from user to user, the criteria for determining what character an OCR image represents do not differ across users. Therefore, it is possible to pre-set character types to be classified and provide a classifier that can recognize the set character types. This classifier can be considered a substitute for the criteria for determining what character a character image represents. Providing users with such pre-trained models reduces the burden on users prior to operating the classifier.

[0007] In this specification, the creation of a model using machine learning techniques by a business providing a product related to image inspection before providing the product to a user is referred to as "pre-training," and a model created through pre-training is referred to as a "pre-trained model." Furthermore, the creation and modification of a model using techniques including machine learning by a user who has received a product related to image inspection and submits training data is referred to as "on-site training."

[0008] OCR is also used in the field of document scanners, where it is used to recognize character strings on the document being scanned. For this purpose, it is common to improve character recognition accuracy by using a machine learning model that has learned not only image features corresponding to characters, but also character string (context) features corresponding to characters.

[0009] On the other hand, characters recognized by OCR in image inspection in the factory automation industry are characters contained in strings of characters that do not change depending on the context, such as expiration dates. Furthermore, compared to document scanners, image inspection in the factory automation industry is subject to greater changes in the external environment that affect OCR recognition accuracy. In the field of document scanners, images are read in largely unchanging environments such as the scanner table and autofeeder path, so changes in the external environment that affect OCR recognition accuracy are significantly smaller. In contrast, the surrounding environment changes over time and with the diversity of workpiece types in OCR for image inspection in the factory automation industry. Therefore, the type of character image that should be recognized as a certain character type depends not only on the shape of the characters in the character image but also on the environment that affects the character image. In other words, compared to OCR in document scanners, the number of character image types that should be recognized as a certain character type in OCR for image inspection in the factory automation industry is proportional to the external environment in which the OCR is performed. However, training using string features other than image features corresponding to characters is not suitable. For this reason, to improve reading accuracy, large amounts of training data and relatively computationally intensive pre-training are required. Furthermore, it is difficult to achieve the required character recognition accuracy through pre-training alone; additional training by users, i.e., on-site training, is required. In other words, OCR in image inspection in the FA industry is required to be able to recognize characters to a certain extent without the need for on-site training, because the identification standards are independent of the user, but on the other hand, it is also required to be able to perform on-site training in order to improve character recognition accuracy.

[0010] The demand for on-site learning is driven not only by the external environment, but also by the unique handling of character images in factory automation (FA) environments. When OCR is performed on documents, the documents themselves are a means of transmitting information from person to person, so common characters are often used. However, in the FA industry, some users sometimes use special characters that are not commonly used as mere symbols. In this way, when recognizing a character type that is not set as a classifiable character type in the pre-trained model, it is necessary to add that character type as a new character type.

[0011] Here, a model created using machine learning techniques and classifying objects into classes determines classification criteria, i.e., parameters for outputting classification results, according to the classes to be classified. Therefore, adding a new character type class to a pre-trained OCR model requires relearning of the pre-trained model, i.e., machine learning similar to pre-training. Therefore, in order for the OCR model after the class addition to have the same reading accuracy as before the class addition, training data and computational processing equivalent to those used in pre-training are required. If only the added character type is provided as training data, the pre-trained model is updated according to the training data. In other words, a model capable of classifying existing character types, which are pre-set character types, is changed to a model capable of only classifying the added character type, which is the new character type. Furthermore, even if character images of all existing character types are prepared as training data, if the number of various types of training data is insufficient, the classification accuracy of the existing character types will be significantly reduced. To avoid this, users need to prepare training data equivalent to the training data used in pre-training for the existing character type classes as well, and perform machine learning that requires computational processing equivalent to that used in pre-training. It has been extremely difficult for users to prepare a classifier that can classify into additional character type classes in addition to the existing character type classes.

[0012] In short, when it comes to OCR in the FA industry, there are users who want to conduct a certain amount of on-site training, but it is necessary to avoid placing a heavy burden on users as described above, and to avoid worsening the classification accuracy of existing character types.

[0013] The present disclosure has been made in consideration of these points, and its purpose is to make it easy for users to learn while taking advantage of the performance of pre-trained models for character recognition. [Means for solving the problem]

[0014] In order to achieve the above-mentioned objective, an aspect of the present disclosure can be based on an image sensor that executes a pre-trained character recognition model to classify an image area that is part of an image of input image data into a first character type class, and that outputs a sensor output according to the character recognition result.

[0015] The image sensor includes a control unit that functions as a training data generation unit that executes the pre-trained model and generates training data and a training execution unit that updates the pre-trained model based on the training data, and a memory unit that stores, for each of the first character type classes, character type class representative features that represent the features of character images classified into that class. The pre-trained model includes a feature extraction unit that includes a character type class feature extraction unit that extracts features related to character type classes from an image region of the input image data as features that indicate the features of the image region, and a character type output unit that outputs the character type of the image region based on the features extracted by the character type class feature extraction unit and the character type class representative features of the memory unit.

[0016] The learning data generation unit generates learning data having image region data and designated character type information that specifies the character type shown in the image region data, and the learning execution unit stores in the storage unit the character type class representative feature of a second character type class that is different from the character type class representative feature of the first character type class based on the features extracted from the image region data of the learning data, so as to classify the image region data of the learning data into a second character type that is the designated character type.

[0017] With this configuration, when an image region including characters is input to the pre-trained model, the pre-trained model executes a classification task into a first character type class, which is an existing character type class. If the target image data contains special characters used by the user or characters that are difficult to read due to background factors, the training data generation unit generates training data including image region data containing such characters and specified character type information that specifies the additional character type indicated in the image region data. The training execution unit then adjusts the parameters of the character type output unit so as to classify the image region data contained in the training data generated by the training data generation unit into an additional character type class, i.e., a second character type class, different from the first character type class, which is an existing character type class. This makes it possible to classify the image region data into the additional character type class in addition to the existing character type classes without deteriorating the classification accuracy of the existing character types.

[0018] The character type output unit can perform advance learning using a distance learning method based on image region data classified into the first character type class.

[0019] The feature extraction unit may include a size feature extraction unit that extracts a size feature, which is a feature related to the size of the image region, and the feature extraction unit may extract a foreground feature, which is a feature indicating whether the image region is foreground or background. In this case, the feature output unit may determine and output the position and size of the image region within the image based on the size feature and foreground feature extracted by the size feature extraction unit, and the character type output unit may output a value indicating the character type based on the character type class feature corresponding to the image region of the size determined by the feature output unit. With this configuration, candidates for image regions containing a given character, i.e., the foreground, are narrowed down based on the size feature, thereby reducing the amount of calculation required for the character type output unit to output a value indicating the character type for the image region. In other words, character size adaptability is improved while reducing processing load.

[0020] The feature output unit may include a foreground output unit that determines whether the image area is the foreground based on the foreground feature extracted by the feature extraction unit. In this case, the character type output unit can output, for an image area corresponding to the image area determined to be the foreground by the foreground output unit, a value indicating the character type of the image area based on the character type feature corresponding to the image area.

[0021] In other words, improving classification accuracy is expected by narrowing down whether an image region is foreground or background before inputting it into a trained model. In a configuration in which the character type output unit outputs a value indicating that the image region is background, any character class close to the background class will exist in the feature space handled by the character type output unit. If a character close to the background class is labeled as an additional character type class by user specification, there is a risk that a feature vector that should be classified into a nearby background class will be classified into that character class. In contrast, as in this configuration, a foreground output unit is provided separately from the character type output unit, and for an image region corresponding to an image region determined to be foreground, the character type output unit outputs a value indicating the character type of the image region. In other words, by configuring in such a way that the character type output unit does not need to output a value indicating the background class, the erroneous recognition of background as character is reduced.

[0022] When the character type specified by the specified character type information is a first character type corresponding to the first character type class, the feature output unit can also output the image region data classified into the second character type class as the first character type. For example, even among characters in the same character type, some characters may be easy to read and some may be difficult to read, and the same character type as a previously learned character type may be specified. In such cases, by treating the specified character type as a new class, the processing load on the image sensor can be reduced compared to when all additional learning is performed using learning equivalent to pre-learning, while output as the same character type can be obtained in terms of character recognition, thereby improving user responsiveness. [Effects of the Invention]

[0023] As described above, since the parameters can be adjusted so that the image region data in the training data is classified into, for example, an additional character type class different from the existing character type classes, classification into the additional character type class in addition to the existing character type classes becomes possible without deteriorating the classification accuracy of the existing character types. Therefore, the performance of the trained model for character recognition can be utilized while the user can easily perform training. [Brief explanation of the drawings]

[0024] [Figure 1] 1A and 1B are diagrams illustrating an image sensor according to an embodiment of the present invention during operation. [Figure 2] FIG. 2 is a hardware configuration diagram of an image sensor. [Figure 3] FIG. 2 is a functional block diagram of a DSP. [Figure 4] 10 is a flowchart illustrating an example of a procedure for setting an image sensor. [Figure 5] 10 is a flowchart illustrating an example of a procedure for setting an inspection tool. [Figure 6] 10 is a flowchart showing an example of a procedure for performing an inspection. [Figure 7] FIG. 1 is a diagram illustrating the processing concept of an AI-OCR tool. [Figure 8] FIG. 10 is a diagram illustrating character types in pre-learning. [Figure 9] FIG. 10 is a diagram illustrating the character type output unit when pre-learning is completed. [Figure 10] FIG. 10 is a diagram illustrating the character type output unit when on-site learning is completed. [Figure 11] FIG. 10 is a diagram showing a specific example of character recognition processing. [Figure 12] FIG. 10 is a diagram showing a specific example of dictionary registration processing. [Figure 13] FIG. 10 is a diagram illustrating a case where the same character that appears different is additionally learned. DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the following description of the preferred embodiments is merely exemplary in nature and is not intended to limit the present invention, its applications, or its uses.

[0026] FIG. 1 is a diagram illustrating an operation of an image sensor S according to an embodiment of the present invention. The image sensor S captures an image of a workpiece W transported by a transport means A according to an imaging setting to acquire a workpiece image, judges the acquired workpiece image according to a judgment setting, and outputs an inspection result using the judgment result to an external device as a sensor output. An example of the external device is a programmable logic controller (PLC) 5, but the external device may be a device other than the PLC 5. The PLC 5 controls the transport means A based on the received inspection result, for example, to separate the storage destinations of the workpieces W. The following description will be given of the case where the external device is the PLC 5. Note that the workpiece W may also be a workpiece that is not transported by the transport means A.

[0027] The image sensor S includes an imaging unit 1 for capturing an image of a workpiece W, a control unit 2 to which image data captured by the imaging unit 1 is input, a PC (personal computer) 3 for configuring the image sensor S, and a display device 4 for displaying a setting screen, a selection screen, workpiece images, inspection results, etc. The control unit 2 is capable of executing a trained model that performs a classification task into multiple character type classes in order to recognize characters in an image area that is a part of the image of the input image data. The control unit 2 outputs a sensor output to the PLC 5 according to the character recognition results obtained by the trained model.

[0028] Here, the image sensor S may be used, for example, to inspect the workpiece W from various angles at various points in a manufacturing device or production line. For this reason, multiple image sensors S may be installed in a single manufacturing device or production line, potentially resulting in insufficient installation space and power supply. Therefore, the image sensor S must be compact to accommodate the installation space and power-efficient to accommodate the power supply. To meet these requirements, the image sensor S according to this embodiment does not include a GPU. In other words, the control unit 2 is equipped with a trained model that has been trained to the extent that basic character recognition is possible. However, the desired classification accuracy can be achieved without the user having to perform advanced additional training, for which the use of a GPU is recommended. Furthermore, eliminating the need for advanced training by the user means that the user does not need to prepare a GPU for training, thereby reducing the time required for training. Note that a single image sensor S may be installed and operated in a manufacturing device or production line.

[0029] (Configuration of imaging unit) The imaging unit 1 is separate from the control unit 2 and is installed so as to be able to capture an image of the workpiece W from a desired direction. The workpieces W are sequentially transported by the transport means A into the imaging field of view of the imaging unit 1.

[0030] As shown in FIG. 2, the imaging unit 1 includes an illumination module 10 for illuminating the workpiece W, and a camera module 11 for capturing an image of the workpiece W illuminated by the illumination module 10.

[0031] The lighting module 10 has an LED (light emitting diode) 10a that irradiates light toward the workpiece W, and an LED driver 10b that controls the light intensity and light emission timing of the LED 10a. The LED driver 10b is connected to a head communication section 20 (described later) of the control unit 2, and is controlled by a control section 21 (described later) of the control unit 2.

[0032] The camera module 11 has an AF motor 11a and an imaging board 11b. The AF motor 11a is a member for automatically focusing on the workpiece W by driving a focusing lens of an optical system (not shown). The autofocus method is not particularly limited, and examples include a contrast method.

[0033] The imaging board 11b is equipped with a CMOS sensor 11c, an FPGA 11d, and a DSP 11e. The CMOS sensor 11c is an image sensor that receives light that is irradiated from the LED 10a onto the workpiece W and reflected by the workpiece W. This CMOS sensor 11c is connected to the head communication section 20 of the control unit 2, and is controlled by the control section 21 of the control unit 2 to perform exposure processing at a predetermined timing for a predetermined time.

[0034] The FPGA 11d is a processing device whose internal processing contents can be changed. The DSP 11e is a signal processing device. A light-receiving amount signal of the light-receiving element of the CMOS sensor 11c is output to the FPGA 11d for processing, and is also output to the DSP 11e for processing. The processing by the FPGA 11d and the DSP 11e is not particularly limited, but examples thereof include various types of filter processing. The image data processed by the FPGA 11d and the DSP 11e is transmitted from the imaging unit 1 to the control unit 2.

[0035] The imaging unit 1 and the control unit 2 are connected via a communication cable 6. Therefore, the control unit 2 can be installed in a location away from the location where the imaging unit 1 is installed.

[0036] (PC configuration) The PC 3 is configured with a general-purpose personal computer or the like. In this example, the personal computer can be used as the PC 3 by installing a predetermined program on it. The PC 3 includes operation devices such as a keyboard 3a and a mouse (not shown). A user of the image sensor S can perform setting operations and selection operations on the image sensor S by operating the operation devices on the PC 3. Specific setting operations and selection operations will be described later.

[0037] The PC 3 and the communication board 22 of the control unit 2 are connected to be able to communicate with each other, and information based on setting operations by the user is sent from the PC 3 to the control unit 2. In addition, the PC 3 is able to receive image data of the workpiece W, inspection results, etc. output from the control unit 2. The PC 3 and the control unit 2 are connected via a communication cable 7. Therefore, the PC 3 can be installed in a location away from where the control unit 2 is installed.

[0038] (Configuration of display device 4) The display device 4 is configured, for example, with a liquid crystal display or an organic EL display. In this example, the display device 4 includes a touch panel 4a. The touch panel 4a is a component capable of detecting operations by the user's finger. The type of the touch panel 4a is not particularly limited, and examples include a capacitive type and an infrared type. The display device 4 and the communication board 22 of the control unit 2 are connected to be able to communicate with each other. Operation information of the touch panel 4a by the user is transmitted from the display device 4 to the control unit 2. In addition, the display device 4 is capable of receiving image data of the workpiece W output from the control unit 2. The display device 4 and the control unit 2 are connected via a communication cable 7. Therefore, the display device 4 can be installed in a location away from the installation location of the control unit 2.

[0039] The PC 3 and the display device 4 may be configured as an integrated unit. For example, the display device 4 may be configured as a display device that the PC 3 has. In this case, the main body of the PC 3 and the display device 4 may be integrated or may be separate. In this example, the communication board 22 and the PLC 5 are connected via a communication cable 7.

[0040] (Configuration of control unit 2) 2, the control unit 2 includes a head communication section 20, a control section 21, a communication board 22, a power supply 23, a connector board 24, an I / O board 25, and a storage device (storage section) 26. The head communication section 20 is connected to the control section 21 and is a section that executes mutual communication between the control section 21 and the imaging unit 1. A control signal for the imaging unit 1 output from the control section 21 is transmitted to the imaging unit 1 via the head communication section 20. The control signal for the imaging unit 1 includes a signal that controls the light emission timing and light emission amount of the LED 10a, and a signal that controls the AF motor 11a and the imaging board 11b. Furthermore, image data acquired by the imaging unit 1 is output from the imaging unit 1 and then transmitted to the control section 21 via the head communication section 20.

[0041] The control unit 21 has a DSP 21a and FPGA 21b that perform various signal processing, an accelerator 21c that speeds up the processing, and a memory 21d made up of RAM, ROM, etc. The specific configuration of the control unit 21 will be described later.

[0042] The communication board 22 is connected to the control unit 21, and is a member that executes mutual communication between the control unit 21 and the PC 3, the display device 4, and the PLC 5.

[0043] The connector board 24 includes a power supply interface 24a. A power cable (not shown) for supplying power from an external source is connected to the power supply interface 24a. The connector board 24 is connected to a power supply 23, and power supplied from the outside to the power supply interface 24a is adjusted to a predetermined voltage by the power supply 23 and then supplied to the control unit 21. The power supplied to the control unit 21 is supplied to the imaging unit 1 via the head communication unit 20.

[0044] The I / O board 25 is connected to the control unit 21. The inspection results output from the control unit 21 are input to the PLC 5 via the I / O board 25.

[0045] (Details of DSP21a) 3 is a functional block diagram of the control unit 21 included in the control unit 2. The control unit 21 includes an imaging setting unit 100, an examination setting unit 200, and an examination execution unit 400.

[0046] The imaging setting unit 100 sets imaging setting parameters and reflects the set imaging setting parameters when capturing an image of the workpiece W. The imaging setting parameters include, for example, multiple parameters such as the timing of illumination by the illumination module 10, brightness (light emission amount), exposure time by the camera module 11, and focus (focal position) of the camera module 11. When setting the image sensor S, the imaging setting unit 100 displays a GUI (graphical user interface) related to setting the imaging setting parameters on the display device 4. Although not shown, the GUI related to setting the imaging setting parameters has areas where the illumination brightness, exposure time, focus, etc. can be individually set. When a user inputs each parameter on the GUI using the touch panel 4a, keyboard 3a, etc., the imaging setting unit 100 accepts the user's setting operation and outputs each parameter to, for example, the storage device 26 for storage therein. Each parameter can be read from the storage device 26 as needed.

[0047] When acquiring image data of the workpiece W, the imaging setting unit 100 outputs the setting data 400 including imaging setting parameters to the camera module 11. When the camera module 11 receives the setting data 400, it sets the imaging setting parameters included in the received setting data 400 so that they are reflected when imaging. When the camera module 11 receives an imaging trigger signal, it illuminates the workpiece W in accordance with the imaging setting parameters and captures an image to acquire image data of the workpiece W. The acquired image data of the workpiece W is shown as captured image 401 in FIG.

[0048] The inspection setting unit 200 includes a tool setting unit 201, a master image registration unit 220, and an inspection condition setting unit 230. The tool setting unit 210 includes a tool selection unit 202, a parameter setting unit 205, and a learning tool setting unit 305. The tool selection unit 202 allows the user to select a tool to be set. When setting the image sensor S, the tool selection unit 202 displays a GUI for tool selection on the display device 4. Although not shown, the GUI for tool selection includes an area where the user can select a desired tool from multiple tools. When the user selects a tool on the GUI using the touch panel 4a, keyboard 3a, or the like, the tool selection unit 202 accepts the user's selection operation and outputs the selected tool to, for example, the storage device 26 for storage therein. The selected tool can be read from the storage device 26 as needed.

[0049] When the tool selected by the tool selection unit 211 is a rule-based tool, information about the tool selected by the tool selection unit 202 is output to the parameter setting unit 205 as selected tool information 203. When setting the image sensor S, the parameter setting unit 205 displays a GUI (Graphical User Interface) related to setting tool parameters on the display device 4. Tool parameters are parameters that can be set for each tool. When the user sets tool parameters on the GUI using the touch panel 4a, keyboard 3a, etc., the parameter setting unit 205 accepts the setting operation by the user and outputs the tool parameters to, for example, the storage device 26 for storage in the storage device 26. The tool parameters can be read from the storage device 26 as necessary. The learning tool setting unit 305 will be described later.

[0050] The master image registration unit 220 is a part that registers the captured image I1 as a master image I2. Specifically, the master image registration unit 220 incorporates the captured image I1 into a GUI and displays it on the display device 4, thereby presenting it to the user. When the user wants to register the captured image I1 displayed on the display device 4 as the master image I2, the user performs a registration operation using the touch panel 4a, the keyboard 3a, or the like. The master image registration unit 220 registers the captured image I1 for which the user has performed the registration operation as the master image I2, and also outputs it to, for example, the storage device 26, where it is stored.

[0051] The master image registration unit 220 has an inspection window setting unit 221. The inspection window setting unit 221 receives instructions, for example, regarding the position, range, shape, etc. of the inspection window for the captured image I1 registered as the master image I2. Specifically, the inspection window setting unit 221 displays a GUI for setting the inspection window on the display device 4. When the user inputs the position, range, shape, etc. of the inspection window on the GUI using the touch panel 4a, the keyboard 3a, etc., the inspection window setting unit 221 receives the setting operation by the user and outputs the information related to the inspection window to, for example, the storage device 26, and stores it in the storage device 26. The inspection window setting unit 221 reflects the setting of the inspection range in the imaging range based on the positional relationship of the inspection window with respect to the captured image I1.

[0052] The inspection condition setting unit 230 is a unit that sets the tools to be used for inspection and the conditions under which the judgment results of each tool must satisfy to output a predetermined inspection result, i.e., the output conditions for the inspection result. The inspection condition setting unit 230 displays a GUI on the display device 4 that enables selection of the tools to be used for inspection and setting of the output conditions. Although not shown, this GUI has an area where any tool, such as a rule-based tool or a learning tool, can be selected as a tool. When a user selects a tool on the GUI using the touch panel 4a, keyboard 3a, or the like, the inspection condition setting unit 230 accepts the user's selection operation. The GUI also has an area where output conditions can be set. When a user sets output conditions on the GUI using the touch panel 4a, keyboard 3a, or the like, the inspection condition setting unit 314 accepts the user's operation to set the output conditions. Examples of output conditions include the condition "how many detected objects are required to output an ON signal as a 'good' inspection result" when the tool is the AI ​​object detection tool T2 described below, or the condition "what conditions must a character string meet to output an ON signal as a 'good' inspection result" when the judgment tool is the AI-OCR tool T3.

[0053] The learning tool setting unit 305 includes a learning data setting unit 310 and a classifier update unit 315. The learning data setting unit 310 is a unit that receives settings for treating the captured image I1 as learning data D1 and reflects the settings. The settings for treating the captured image I1 as learning data D1 include settings for label information. Specifically, the learning data setting unit 310 includes a learning image selection unit 310a, a learning data generation unit 310b, and a label information setting unit 310c.

[0054] The training image selection unit 310a is a unit that selects a captured image I1 to be treated as a training image. The training image selection unit 310a accepts an image selection operation by a user using the touch panel 4a, keyboard 3a, or the like on the GUI, and outputs a training image I3, which is image data of the selected training image, to the training data generation unit 310b. The image selected by the training image selection unit 310a may be an image other than the captured image I1. Alternatively, the configuration may be such that the training image I3 is automatically selected without accepting a selection operation by the user.

[0055] The label information setting unit 310c displays a GUI incorporating the training image I3 selected by the training image selection unit 310a on the display device 4, accepts a specification operation related to setting the label information, and sets the label information based on the information specified by the operation.

[0056] The training data generation unit 310b generates training data D1 based on the training image data and label information. The training data generation unit 310b acquires training image data from the training image selection unit 310a, acquires label information from the label information setting unit 310c, and outputs the generated training data D1 to the classifier update unit 315. In this way, the control unit 21 functions as the training data generation unit 310b that generates the training data D1.

[0057] The information that the training data generation unit 310b acquires to generate the training data D1 varies depending on the type of training tool used in the training data D1. Details will be described later, but for example, when generating training data D1 for training an AI-OCR tool T3, the training data generation unit 310b acquires training image data and, as label information, designated position information indicating the relative position of a character in a training image, designated character type information indicating the character type of the character, and designated size information indicating the size of the character in the image.

[0058] The learning data setting unit 310 may change the method of selecting learning images and the method of setting label information depending on the type of learning tool that uses the learning data D1 that reflects the settings of the learning data setting unit 310.

[0059] When the learning tool T1 is a learning tool that performs judgment based on the feature tendency of the entire image region included in the inspection window set by the inspection window setting unit 221, the entire image region of the selected training image I3 that corresponds to the inspection window is considered to be the image region to which a label is assigned. In other words, the label information setting unit 310c does not need to set the position to which the label is assigned. In this case, the label information setting unit 310c can set the label information by simply accepting the label specification, thereby reducing the operational steps related to setting the training data D1. The label specification here corresponds to the specification of a class when the label is a class label indicating a class.

[0060] Furthermore, in the above case, if the tool's judgment result is treated as the inspection result (e.g., "good" or "bad"), the user can easily grasp the characteristics of the image that should be classified into that class. In this case, the training image selection unit 310a may accept an image selection operation while the class of the assigned class label is fixed. For example, when the training image selection unit 310a accepts a selection operation of the training image I3 while displaying a GUI on the display device 4 guiding the user to "select an image you want to classify into the first class 'good'," the inspection window determines the image region to which the label is assigned. Furthermore, the label information setting unit 310c can set label information assuming that the image region is designated as the first class "good" without accepting an operation to specify a class. Because label information can be set based on a fixed class, the operational steps involved in setting the training data D1 are reduced.

[0061] When the learning tool is an AI object detection tool T2 that detects an object instructed by the user from an image region included in the inspection window set by the inspection window setting unit 221, i.e., the label information setting unit 310c accepts an operation to specify a position on the image and an operation to specify a label to be assigned to the position. The learning image I3 is an image containing the object to be detected, and includes an image region where the object is located and a background region where the object is not located. Because different class labels should be assigned to the image region where the object is located and the background region where the object is not located, it is necessary to accept the specification of the position to be labeled. Based on the accepted specification operation, the label information setting unit 310c outputs, as label information, a combination of specified position information indicating the position and specified class information based on the class label to be assigned to the position to the learning data generation unit 310b. In this way, the user assigns a class label of the class "object (foreground)" to a predetermined position on the image, but the operation to specify the class may be omitted. This is because AI object detection tool T2 is a learning tool for detecting objects, and so it is presumed that the position specified by the user on the image is the position to which a class label of the class "object" should be assigned. For example, label information setting unit 310c can omit the acceptance of the operation to specify a class by accepting the specification of a position on the image while displaying a guide such as "Please specify the position of the object in the image."

[0062] When the learning tool is an AI-OCR tool T3 that recognizes characters in an image area included in the inspection window set by the inspection window setting unit 211, i.e., the label information setting unit 310c accepts the designation of a position on the image, similar to the AI ​​object detection tool T2. Because the AI-OCR tool T3 not only detects characters but also identifies the character type of the characters, the label information setting unit 310c accepts an operation to designate a character type as a label designation. By accepting an operation to designate a character type, the label information setting unit 310c designates two classes: a class "character" and a character type class, and the designated character type information includes a class label for the class "character" and a class label for the character type class. The label information setting unit 310c outputs a combination of the designated position information and the designated character type information to the learning data generation unit 310b as label information.

[0063] The label information output by the label information setting unit 310c is not limited to specified position information and specified class information, and may also include size information and angle information. The size information and angle information may be specified by a size label or angle label separately assigned by the user. However, when the label information setting unit 310c allows the user to specify a position using a rectangular bounding box, the bounding box provides specified size information and specified angle information. The specified size information and specified angle information may be used to update the classifier.

[0064] In this embodiment, the learning data D1 for the AI-OCR tool T2 is set using specified position information, specified size information, and specified character type information. As will be described in detail later, in this embodiment, the specified size information is incorporated into the specified character type information. In other words, a character type whose size information is specified is treated as a single character type class. At this time, a position on the image is specified using a bounding box on the GUI, and the label information setting unit 310c identifies the character type class corresponding to the specified position by combining the size corresponding to the bounding box and the character type specified as the class of the label to be assigned to that position. Therefore, the label information when setting the learning data D1 for the AI-OCR tool T2 is a combination of a position in the captured image 401 and the character type class specified for that position.

[0065] The classifier update unit 315 is a part that receives the training data D1 generated by the training data generation unit 310b and updates the classifier based on the training data D1 using a method including a machine learning technique, and includes a classifier update unit 315a and a class addition unit 315b. In this embodiment, the classifier update unit 315 is a learning execution unit, and therefore the control unit 21 also functions as a learning execution unit that executes learning based on the training data 317, and controls on-site training of the pre-trained model.

[0066] The classifier update unit 315 updates the classifier used in the inspection by performing a predetermined update method based on the received learning data D1. In other words, the classifier update unit 315 performs learning using the learning data D1. The classifier to be updated varies depending on the selected learning tool. The classifier update unit 315 updates the classifier using a method including a machine learning technique that uses feature amounts (feature vectors 323 or feature maps 324) extracted by inputting image data included in the learning data D1 to a feature amount extraction unit 510 of the classifier to be updated and label information corresponding to the image data.

[0067] The classification unit update unit 315a updates the classification unit, which is the determination unit 520 of the classifier to be updated, which classifies image data into classes. Specifically, the classification unit update unit 315a executes the feature extraction unit 510 of the classifier to be updated, inputs the training image data of the training data D1, and obtains a feature F (feature vector). As a result, the training data group is treated as a group of "combinations of feature vectors x class labels corresponding to each feature vector." Based on this group of combinations, a boundary surface is generated so that each class identified by the class label is distinguished in a feature space to which the feature vector of the feature F is mapped. When additional training data is available, the boundary surface is readjusted so that the additional feature F in the feature space is classified into the class associated with the additional feature F.

[0068] When the classifier to be updated by the classifier update unit 315 is a classifier executed by the learning tool T1, the classifier update unit 315a executes the feature extraction unit 510 to obtain a feature F1 from all of the training image data included in the training data D1. Then, the classifier update unit 315a updates the image classifier 521, which is a determination unit 520 that classifies image data into classes set by the user. The image classifier 521 is an SVM (support vector machine) as a linear classifier.

[0069] When the classifier to be updated by classifier update unit 315 is a classifier executed by AI object detection tool T2, classifier update unit 315a updates image region classifier 522, just as in the case of a classifier executed by learning tool T1. Image region classifier 522 is also a linear classifier, just like image classifier 521. The difference compared to when classifier update unit 315 updates a classifier executed by learning tool T1 is that a partial region image of the training image data is input to feature extraction unit 510 to obtain feature F2 (feature vector), and the classifier is otherwise updated in the same way as in the case of learning tool T1.

[0070] The class addition unit 315b updates the classifier to be updated so that it can classify images into additional classes specified by the training data D1 in addition to the existing classes that could be classified before the update. Since the parameters of a classifier are typically optimized by comparing them with other classes, training is required using at least one image data set for each class to be classified. The parameter update by the classifier update unit 315a uses at least one image data set for each class to be classified. In on-site training, there is a limit to the amount of image data collected. If an attempt is made to update the parameters of the classifier using only image data that does not cover all classes, the image may be classified into the presented class, but the accuracy of classification into other classes or the accuracy of classification of images that should belong to other classes may decrease. Even if all classes are covered, it is difficult to prepare the same amount of image data in on-site training as in pre-training, so at least the accuracy of classification into existing classes decreases. Furthermore, when there are many classes to be classified, the amount of calculation required for training by the classifier update unit 315a tends to increase, resulting in a heavy learning load. In order to reduce the learning load and increase the degree of correspondence to classes added in additional learning, the class addition unit 315b updates the parameters by labeling the arrangement of extracted features obtained by the classification unit before the update. In this embodiment, when the classifier update unit 315 updates the classifier of the AI-OCR tool T3, the class addition unit 315b updates the classifier.

[0071] Similar to when the classifier of the AI ​​object detection tool T2 is updated, the class addition unit 315b inputs a partial region image of the training image data into the feature extraction unit 510 to obtain feature F3. What differs from when the AI ​​object detection tool T2 is updated is that the classifier's determination unit 520 is a character type output unit 523 obtained using a distance learning method in pre-learning. More specifically, the character type output unit 523 pre-learns a group of feature vectors of classifiable character type classes so that, in the embedding space, the distance between feature vectors of the same character type class is small and the distance between feature vectors of different character type classes is large. At this time, a representative vector is defined as an arrangement of representative feature vectors for each character type class, and the character type output unit 523 classifies the image data into the character type class of the representative vector whose distance is close to the feature vector of the image data and outputs the classification result. In the character type output unit 523 that has undergone such pre-training, a feature vector to be classified into a new character type class based on the training data D1 is arranged in the embedding space so as to be spaced apart from the representative vector of the pre-trained existing character type class if the feature vector is a character type different from the existing character type class. The class adding unit 315b takes advantage of this arrangement of feature vectors to define the feature F3 extracted from the training data D1 as a new representative vector. In other words, classification into a new character type class is possible while leaving the calculation parameters used up until the feature F (feature vector) is arranged in the embedding space unchanged. Furthermore, the class adding unit 315b adds a class that combines a specified size and a specified character type from the label information of the training data, i.e., the character type class "specified character type with specified size," as a new character type class.

[0072] The test execution unit 400 includes a rule determination unit 402 , a learning tool execution unit 403 , and a test result output unit 410 .

[0073] When the inspection tools used for the inspection include a rule-based tool, the rule determination unit 402 performs a determination on the captured image I1 of the inspection target based on the inspection tool. The parameters set by the parameter setting unit 205 are used.

[0074] The learning tool execution unit 403 is a part that executes the classifier when a learning tool is included in the inspection tools used for inspection, and outputs a judgment result for the captured image I1 of the inspection target. The classifier includes a feature extraction unit 510 and a judgment unit 520.

[0075] The feature extraction unit 510 is configured, for example, by a convolutional neural network. The feature extraction unit 510 is a part that extracts features that indicate the characteristics of input image data from the image data, and can also extract features for any position in the input image data, for example. The feature extraction unit 510 is a part that is pre-trained to extract features according to each learning tool, and has parameters that are used in the process of extracting the feature F from the image data.

[0076] The feature extraction unit 510 can extract the feature F in a format appropriate for the learning tool. For example, when learning tool T1 is executed, image data (captured image I1) is input to the feature extraction unit 510, and a multidimensional feature vector is obtained as the feature F. Furthermore, when a judgment is made utilizing spatial information of the input image, the image data is input to the feature extraction unit 510, and a feature map 324 is obtained in which a feature vector is calculated for each convolution pixel corresponding to a certain range of pixel region in the input image data.

[0077] Determination unit 520 is a part that outputs a determination result based on the input feature amount F (feature vector or feature map). The determination unit 520 executed by the learning tool execution unit varies depending on the learning tool, and at least one of image classification unit 521, image region classification unit 522, character type output unit 523, size output unit 524, and image region angle output unit 525 is executed.

[0078] The image classification unit 521 classifies the feature F4 obtained from the feature extraction unit 510. The feature F4 is a feature vector. The image classification unit 331a outputs information indicating to which class the image data corresponding to the feature F4 belongs. When the image classification unit 521 is executed during execution of the learning tool T1, and the learning tool T1 is set to be classified into a first class "good" and a second class "bad" in the setting by the learning tool setting unit 305, the image classification unit 521 outputs information indicating to which class, the first class or the second class, the captured image I1 belongs, based on the feature F4 obtained from the captured image I1.

[0079] The image region classification unit 522 classifies each convolved pixel of the feature map based on the feature map as feature F5 obtained from the feature extraction unit 510. The image region classification unit 522 performs class classification for each convolved pixel of the feature map, and can output where in the captured image I1 there is an image region classified into a specified class based on the spatial information contained in the feature map. When the image region classification unit 522 is executed during execution of the AI ​​object detection tool T2, the image region classification unit 522 classifies the convolved pixels of the feature map F5 into the first class "object (foreground)" or the second class "background." This classification result makes it possible to determine where in the captured image I1 corresponding to the feature map F5 there is an area image belonging to the first class "foreground," i.e., whether a workpiece W as an object exists.

[0080] The image region classification unit 522 executed during execution of the AI-OCR tool T3 classifies each convolved pixel of the feature map as the feature F6 into a first class, "foreground," and a second class, "background." Based on this classification result, it is possible to determine where in the captured image I1 the character image region corresponding to the feature F6 is located.

[0081] The character type output unit 523 is executed during execution of the AI-OCR tool T3. The character type output unit 523 outputs a character type class classification result based on the feature F7 obtained from the feature extraction unit 510. Because the feature F7 is a feature map, the character type output unit 523 can determine where in the captured image I1 corresponding to the feature F7 there is a character image area belonging to a given character type class. When the character type output unit 523 classifies an additional character type class, the classification result is used to obtain a determination result as to where in the image data corresponding to the input feature map a character image of a given character type exists, and how large it is.

[0082] The size output unit 524 is executed when the AI-OCR tool T3 is executed, and outputs the size of the corresponding image area based on the feature F8 output from the feature extraction unit 510. The feature F8 is a feature map. The size output unit 524, which is executed when the AI-OCR tool T3 is executed, outputs the size of the character image area corresponding to each pixel of the feature map.

[0083] The image area angle output unit 525 outputs information about the angle at which the corresponding image area exists, based on the feature F9 output from the feature extraction unit 510. The image area angle output unit 525, which is executed during execution of the AI ​​object detection tool T2, outputs angle information when the image area of ​​the input data corresponding to each pixel of the feature map as the feature F9 is the image area of ​​the workpiece W.

[0084] The inspection result output unit 410 generates and outputs inspection results based on the judgment results obtained by execution of the judgment unit 520, in accordance with the output conditions set by the inspection condition setting unit 230. For example, if the only tool used for inspection is the learning tool T1 and the output conditions are set to use the judgment results of the learning tool T1 as the inspection results, the inspection result output unit 410 obtains the class classification results of the image data of the captured image I1 as a first class "good" or a second class "bad" as the judgment results. If the obtained judgment result is the first class "good," the inspection result output unit 410 generates an inspection result indicating that the captured image I1 is a non-defective image, and outputs the inspection result from the I / O board 25 to a PLC or the like.

[0085] When the inspection condition setting unit 230 sets an inspection using the AI ​​object detection tool T2 and the output condition for outputting "ON" as the inspection result is "detection of one or more workpieces W," the inspection result output unit 410 obtains the classification result from the image area classification unit 521 as the judgment result and displays the judgment result indicating where and at what angle the workpieces W are present in the image data corresponding to the input feature F2. The inspection result output unit 410 then generates an inspection result based on the number of workpieces W in the image data indicated in the judgment result, displays the area corresponding to the detected workpieces W on the target image data, and outputs the inspection result to a PLC or the like. In this example, the number of workpieces W to be detected is specified as the output condition, but parameters such as the number of workpieces W may also be set as the judgment condition for the AI ​​object detection tool T2. Furthermore, when the angle of the image area is estimated in the judgment based on the AI ​​object detection tool T2, an angle-based condition such as "the detected workpieces W are within ±15° of the workpieces W in the master image" may also be set as the output condition.

[0086] It is possible to register multiple types of objects for object detection in the AI ​​object detection tool T2. For example, by registering a first object, a workpiece W1, and a second object, a workpiece W2, that has a different shape, color, and size from the first object, the inspection result generating unit 332a obtains as a determination result where and at what angle the workpiece W1 exists in the image data, and also obtains as a determination result where and at what angle the workpiece W2 exists in the image data.

[0087] When the inspection condition setting unit 230 sets an inspection using the AI-OCR tool T3 and sets an output condition stating that "the characters following 'best before' must be a string indicating a date within three days of the inspection date," the inspection result output unit 410 obtains a determination result indicating where in the image a character image is located. The inspection result output unit 410 generates an inspection result based on the character information indicated in the determination result, displays an area corresponding to the detected character image in the target image data and the character type corresponding to that area, and outputs the inspection result to a PLC or the like. At this time, for the existing character type class, the inspection result output unit 410 obtains a determination result indicating where in the image, what character image is located, and what size it is located in the image data of the captured image I1 corresponding to the input feature amount F, based on the classification result by the character type classification unit 523 and the image area size output unit 523. For the additional character type class, the inspection result output unit 410 obtains a determination result indicating where in the image, what character image is located, and what size it is located, based on the classification result by the character type classification unit 523.

[0088] The inspection result output unit 410 may generate and output inspection results by combining multiple inspection tools, including rule-based tools and learning tools. For example, if the learning tool T1 determines the first class as "good," the AI ​​object detection tool T2 determines that there are two objects in the image, and the AI-OCR tool T3 determines that the expiration date is correctly printed in a predetermined position on the object detected by the AI ​​object detection tool T2, the inspection result output unit 410 may output an "ON" inspection result.

[0089] The inspection result output unit 410 also includes an additional training image designation unit 415. Based on the judgment result obtained from the judgment unit 510 and the inspection result generated based on the judgment result, a GUI that accepts designation of a training image I3 to be added as a target for on-site training can be displayed on the display device 4. The user can designate a training image using the additional training image designation unit 415. When an image is designated by the additional training image designation unit 415, information related to the designated image is output to the training data setting unit 310. The training data setting unit 310 treats the image designated by the additional training image designation unit 415 in the same way as the training image I3 selected by the training image selection unit 310a. In other words, the training data generation unit 310b generates training data D1 based on the training image data of the training image I3 designated by the additional training image designation unit 415 and the label information set by the label information setting unit 310c.

[0090] (When setting up the image sensor) 4 is a flowchart showing the control of the control unit 21 when setting the image sensor S. In image capture setting in step SA1 after the start, the control unit 21 accepts image capture setting parameters from the user via the image capture setting unit 100 and sets the image capture setting parameters. In step SA2, the image capture setting parameters set in step SA1 are applied to cause the image capture unit 1 to capture an image. Image data acquired by the image capture unit 1 is stored in the storage device 26 and treated as a master image I2 and a learning image I3.

[0091] In step SA3, an input relating to the settings for the image inspection of the captured image I1 is accepted. In step SA4, input of signals from outside, output of signals to outside, communication settings, etc. are performed.

[0092] Fig. 5 is a flowchart showing the details of step SA3 when a learning tool is used as an inspection tool. In step SB1, the control unit 21 registers the image acquired in step SA2 shown in Fig. 4 as a master image I2 using the master image registration unit 220. In step SB2, the inspection window setting unit 221 sets an inspection range on the master image I2 registered in step SB1.

[0093] In step SB3, the control unit 21 selects the learning image I2 using the learning image selection unit 310a.

[0094] In step SB4, control unit 21 performs settings related to the judgment of the learning tool. More specifically, label information is set by label information setting unit 310c, learning data generation unit 310b generates learning data D1 based on the label information, and classifier update unit 315 updates the classifier based on learning data D1. For example, in learning tool T1, when a class is determined and the user is prompted to select image data belonging to that class, the user's selection of image data corresponds to the acceptance of the selection of learning image I3 in step SB3 and also to the acceptance of input for judgment setting in step SB4. In this way, depending on how the user's input is accepted, steps SB3 and SB4 may be performed simultaneously.

[0095] In cases where a label assignment target is not identified upon selection of the training image I3, such as in the AI-OCR tool T3, after the training image I3 is selected by the training image selection unit 310a in step SB3, the label information setting unit 310c accepts a user specification for step SB4 using the selected training image I3. However, by using a classifier to set step SB4, the acceptance of the user specification can be omitted. For example, class labels may be assigned to the training images selected in step SB3 using an existing classifier, and training data may be generated based on the assigned class labels. In such cases, steps SB3 and SB4 may be performed simultaneously, as in the above-described training tool T1.

[0096] In step SB5, the output conditions for generating inspection results based on the judgment results are set by the inspection condition setting unit 230. For example, if the inspection result is the result of counting the workpiece W using the AI ​​object detection tool T2 as the inspection tool, the number of areas that must be judged as targets by the AI ​​object detection tool T2 before the inspection result is output as "ON" is set.

[0097] (When using the image sensor) Fig. 6 is a flowchart showing the control procedure of the control unit 21 when the image sensor S is in inspection operation including a learning tool. The image sensor S can be switched between a setting mode in which settings are made as shown in Fig. 4 and Fig. 5, and an operation mode shown in Fig. 6, but this mode switching does not have to be clear, and continuous operation is also possible, for example, by temporarily switching to the setting mode to change settings while operating in the operation mode, and then returning to the operation mode.

[0098] In step SC1 after the start, the imaging unit 1 captures an image of the workpiece W, and acquires an image I1 of the workpiece W. In step SC2, the inspection execution unit 400 starts an inspection based on the set learning tool for the image of the workpiece W acquired in step SC1. First, in step SC3, based on the inspection range set in the inspection window setting unit 221, the range to be inspected is cut out from the image of the workpiece W acquired in step SC1.

[0099] In step SC4, feature extraction unit 510 extracts feature amounts from the image data of the inspection range extracted in step SC3. In step SC5, the feature amounts extracted in step SC4 are judged by judgment unit 520. As described above, in steps SC4 and SC5, the feature amounts change depending on the learning tool or the judger to be executed.

[0100] In step SC6, a judgment result is generated by the learning tool. If inspection conditions related to the judgment result are set in the learning tool, the judgment result is determined by comparing the result with the inspection conditions. For example, if a condition such as "the angle must be within ±15° of the workpiece W in the master image" is set for the AI ​​object detection tool T2, the judgment result is generated based on this condition. In step SC7, it is determined whether there are any unprocessed configured inspection tools. If there are any unprocessed configured inspection tools, the process proceeds to step SC3; if there are no unprocessed configured inspection tools, the process proceeds to step SC8. In step SC8, the inspection based on the configured inspection tools ends.

[0101] In step SC9, the inspection results are output by the inspection result output unit 410. If the inspection includes one inspection tool, the judgment result related to that inspection tool is output as the inspection result as is, but if the judgment results of multiple inspection tools are combined, the inspection results are generated and output based on the judgment results of each inspection tool.

[0102] (Details of AI-OCR) 7 is a conceptual diagram illustrating the feature extraction unit 510 and determination unit 520 involved in character type class classification when the learning tool execution unit 403 executes the AI-OCR tool T3. When executing the AI-OCR tool T3, the learning tool execution unit 403 executes the character type output unit 523 for character type class classification. The learning tool execution unit 403 also executes the character type feature extraction unit 510a as the feature extraction unit 510 that extracts the feature F7 for output by the character type output unit 523. During operation of the image sensor S, when the imaging unit 1 acquires image data of a captured image I1 that partially contains characters, the image data is input to the pre-trained feature extraction unit 510. The character type feature extraction unit 510a is a computational model having a network structure, and is a convolutional neural network trained by machine learning techniques to extract the feature F7 suitable for the output of the character type output unit 523. The feature F7 is input to the character type output unit 523 to recognize characters in an image area that is part of the image of input image data, and classifies the image area into an existing character type (first character type) class. At this time, the learning tool execution unit 403 controls the calculation processing by the pre-trained character type output unit 523. Specifically, it controls the classification task into an existing character type class by the pre-trained character type output unit 523. Because the position where characters are printed is sometimes important in image inspection, the image area 500a on which the AI-OCR tool T3 is executed may be set in advance, and the feature F7 for the image area 500a may be input to the character type output unit 523.

[0103] There are multiple existing character type classes, and existing character type classes include, for example, alphabets, numbers, katakana, and hiragana. Characters may also include symbols, in which case the symbols are also included in existing character type classes. The storage device 26 stores character type class representative features that represent the character type class features of character images belonging to the existing character type classes described above. The character type class representative features stored in the storage device 26 can be read out when performing character recognition processing.

[0104] The character type output unit 523 is a linear classifier that reads from the storage device 26 character type class representative features that represent the character type class features of character images belonging to existing character type classes. Based on the character type class features extracted from a character image region at an arbitrary position in the input image data and the character type class representative features, the unit can output a value indicating the character type of the image region corresponding to the character type class features. For example, when the character type output unit 523 determines that the character type class features belong to a specified character type class corresponding to a combination of a character type and a specified size, the unit can output the size of the character image of the image region based on the specified size included in the specified character type class. Note that even when a character image combining multiple characters is registered as a single character type class, the feature F7 input to the character type output unit 523 is treated as a feature corresponding to a single character image region.

[0105] In this embodiment, the AI-OCR tool T3 estimates the size of the character image area and outputs a character recognition result corresponding to the size. Therefore, the learning tool execution unit 403 executes a size output unit 524, which outputs the size of the image area as a determination unit 520, and a size feature extraction unit 510c, which extracts a feature F8 suitable for the output of the size output unit 524. The size feature extraction unit 510c is a convolutional neural network pre-trained to extract a feature F8 suitable for the output of the size output unit 524, and the feature F8 is a feature map. The size output unit 524 estimates and outputs the location and size of the image area in the captured image I1 based on the feature F8. The AI-OCR tool T3 combines the output from the character type output unit 523 and the output from the size output unit 524 to output a character recognition result. This improves character recognition accuracy. For example, in the example of FIG. 7, the width of the character image area occupied by the letter "I" is smaller than the widths of the character image areas occupied by other characters. If no part for outputting the size of the character image area is provided, character recognition is performed with the size of the character image area fixed. If the size of the character image areas other than "I" is fixed, other characters will be included in the character image area of ​​"I", which will reduce the accuracy of character recognition. By outputting the size of the character image area by size output unit 524, character type output unit 523 can output character types assuming image areas of various sizes, thereby improving the accuracy of character recognition.

[0106] In the example shown in FIG. 7, the learning of the feature extraction unit 510 is performed by the business providing the image sensor S, and on-site learning by the user is not possible. The character type output unit 523 can be learned on-site by the user. That is, the parameters of the feature extraction unit 510, including the convolution operation that imposes a large learning load, are fixed, and only additional learning of the character type output unit 523 is possible. This allows additional learning by the CPU installed in the control unit 21 without the need for a high-performance processing device such as a GPU. The size output unit 524 is also a part obtained by the business providing the image sensor S using machine learning techniques, i.e., a part provided by prior learning. However, since re-learning requires a large amount of computational processing, on-site learning is not possible.

[0107] Furthermore, during inference, the inference by the feature extraction unit 510 is processed by, for example, an accelerator 21c specialized for convolution operations, and the inference by the determination unit 520 such as the character type output unit 523 can be processed by, for example, the DSP 21a or accelerator 21c.

[0108] The concept of pre-learning of the character type output unit 523 will be explained with reference to Fig. 8. This is also called distance learning. The circle E1 in Fig. 8 schematically shows the feature space, which is actually a multidimensional space. For example, when learning character types "A" and "B," the feature vectors of the same character type are learned to approach each other, and the feature vectors of different character types are learned to move away from each other.

[0109] 9 shows the state of the character type output unit 523 when the image sensor S is provided to the user by the business operator, that is, the state after pre-learning of the character types "A," "B," and "C" is completed. As shown in this figure, basic alphanumeric characters have been pre-learned so that they can be classified, and representative vectors, which are representative feature vectors, have been registered.

[0110] After image sensor S is provided to a user by the manufacturer, when the character "β" is read by image sensor S, it is read as "B," as shown in Figure 10. Explaining this in more detail, first, because "β" and "B" have different characteristics, the feature vector extracted from the character image of "β" is located at a position away from the representative vector of character type class "B." However, because the representative vector in the vicinity where the feature vector of "β" is located is a representative vector of character type class "B," the feature vector of character image "β" is classified into character type class "B."

[0111] If the character image of "β" is read as "B" in this way, the user may want the character image "β" to be read as "β". In other words, since the character image of "β" is similar in appearance to "B" that is registered as an existing character type class, it is read as "B" as shown in FIG. 10, and such a reading result is output by the inspection result output unit 410. After checking the inspection result, the user designates the character image "β" as training image I3 in the additional training image designation unit 415 and instructs the character type output unit 523 to perform on-site training.

[0112] Specifically, the learning data generator 310b generates learning data D1 that includes image region data including “β” and designated character type information that specifies the character type shown in the image region data including “β.” In this case, since the character type is “β,” the designated character type information is information that specifies “β.”

[0113] The class addition unit 315b of the classifier update unit 315 adjusts the parameters of the character type output unit 523 so as to classify image regions of the training image data included in the training data D1 generated by the training data generation unit 310b into the “β” class corresponding to the specified character type, i.e., into an additional character type class different from the existing character type classes. More specifically, the character type output unit 523 is adjusted so as to add the representative vector of “β” to the representative vectors read during execution of the character type output unit 523, and to classify image regions corresponding to feature vectors included in feature F7 and determined to be close to the representative vector of “β” into the character type “β.” At this time, the class addition unit 315b registers the representative vector of “β” based on the arrangement of the feature vector of “β” so that feature F3 extracted from the character image “β,” i.e., the feature vector of “β” arranged in the feature space by the character type output unit 523 before the update, is not classified into the existing character type class “B.” In this case, the feature vector of "B" remains registered, so when a character image closer to "B" than "β" is input, the feature F7 extracted from that character image remains positioned closer to the representative vector "B," and as a result, it is read as "B."

[0114] In this way, since there is no need for a process of adjusting the arrangement of predetermined feature amounts in a multidimensional space and only feature vectors based on feature vectors of additionally specified image data are stored, re-learning equivalent to pre-learning is not required, and on-site learning can be performed so that the user can classify new character types without deteriorating the reading performance of the image sensor S at the time it is provided to the user by the business operator. Also, a single learning image may be used for on-site learning.

[0115] (Example of character recognition) FIG. 11 specifically illustrates the flow of character recognition processing. First, during operation of the image sensor S, image data of the captured image I1 acquired by the imaging unit 1 is input to the feature extraction unit 510. The feature extraction unit 510 is a pre-trained unit that has undergone pre-training to extract features appropriate for the intended use. The feature extraction unit 510 includes a character type feature extraction unit 510a, an object feature extraction unit 510b, and a size feature extraction unit 510c. The object feature extraction unit 510b extracts a feature F6 indicating the likelihood of an image region being an object, i.e., whether the image region is in the foreground or background. The size feature extraction unit 510c extracts a feature F8 related to the size of the image region. For convenience, in this embodiment, each feature extraction unit is described as a separate unit. However, each feature extraction unit may be configured to extract individual features from features extracted using a common model structure. Furthermore, the same feature may be extracted for different purposes.

[0116] The three feature amounts extracted by the feature amount extraction unit 510 are sent to a determination unit 520 serving as a feature output unit.

[0117] The feature F6 extracted by the object feature extraction unit 510b is a feature map, which is sent to the candidate area output unit 522a, which functions as the image area classification unit 522. Based on the feature F6, the candidate area output unit 522a calculates a probability value that the image area corresponding to each convolved pixel is an object. The probability value is calculated using a known method, such as calculating similarity with features that indicate the image area of ​​the object. The candidate area output unit 522a then designates image areas for which the probability value is equal to or greater than a threshold as candidate areas and outputs candidate area information Dxy indicating the position of the candidate area in the captured image I1. Note that the candidate area information does not include information regarding the size of the candidate area. In this embodiment, the candidate area information Dxy is output without considering spatial information in the captured image I1 in order to detect all closely spaced character image areas and character image areas that may be detected as multiple separate character image areas in the captured image I1. However, a heat map in which character likelihoods are arranged according to the spatial information of the captured image I1 may also be generated, and the candidate area information may be output based on the peak positions in the heat map. With this configuration, it is possible to eliminate candidate areas that overlap with the same character image area, thereby reducing the amount of calculation required for processing that uses candidate area information. When the candidate area information Dxy is output without taking into account the spatial information in the captured image I1 as in this embodiment, for example, for a captured image I1 in which the correct character image area is located near the peak position, it is possible to prevent omission of the candidate area information Dxy from being output, thereby further improving OCR accuracy.

[0118] The feature F7 extracted by the character type feature extraction unit 510a is sent to the character type output unit 523. The feature F7 is a feature map having an H dimension (height) and a C dimension (feature). The character type output unit 523 outputs a value indicating the character type of the image region based on the feature corresponding to the image region indicated in the candidate region information Dxy. More specifically, the feature f71 corresponding to the first candidate region P1 is identified from the feature F7 using the candidate region information Dxy, and the character type is output based on the feature f71. Here, the character type feature may be normalized to a length of 1. Note that the feature F6 and the feature F7 may be configured as a common feature, which reduces the processing load of the feature extraction unit 510. In this embodiment, the feature f71 is identified from the feature F7, but the feature extraction unit 510 may also be configured to extract features only for the candidate region based on the candidate region information Dxy. Furthermore, in this embodiment, the character type output unit 523 identifies the feature f71 based on the candidate area information Dxy to speed up the processing of the character type output unit 523, but the character type may also be output for all or part of the convolved pixels of the feature F7 as a feature map.

[0119] The character type output unit 523 calculates the product of the dictionary matrix DM shown in FIG. 11 and the feature value f71 as a feature vector to infer which character type class the candidate area P1 corresponding to the feature value f71 belongs to. The dictionary matrix DM contains as many representative vectors for each character type class as the number of registered characters (the number of character type classes). The character type output unit 523 calculates the product of the feature value f71 and the dictionary matrix DM to output the probability value that the candidate area P1 belongs to each character type class as a score. In this way, the character type output unit 523 calculates the score of each character type class for each candidate area specified by the candidate area information Dxy. Therefore, the information output by the character type output unit 523 is a combination of the candidate area information Dxy and character type information Dcl related to the character type of each candidate area. The character type information Dcl is composed of scores related to all character type classes indicated in the dictionary matrix DM, but may also be information related to some character type classes, such as character type classes with high scores, or may not include any scores.

[0120] The feature F8 extracted by the size feature extraction unit 510c is sent to the size output unit 524. The size output unit 524, which serves as the determination unit 520, outputs size information Dhw relating to the size (width, height) of the image region based on the feature F8. Because the feature F8 is a feature map, the size output unit 320f can output size information Dhw of the character image of each image region based on the feature of each convoluted pixel constituting the feature F8, and outputs a combination of the position of the image region included in the captured image I1 and the size information Dhw of the image region. For example, when the character type output unit 523 determines, based on the feature F7 extracted by the character type feature extraction unit 510a, that an image region does not belong to a designated character type class corresponding to the combination of character type and designated size (described later), it outputs size information Dhw of the image region, i.e., the character image region. Note that the feature map serving as the feature F8 is extracted without consideration of whether the image region is a character image region, and therefore values ​​are included in convoluted pixels that are not character image regions. In this embodiment, the size information is expressed as width and height, but may also be expressed as a sub-pixel position relative to the pixel that specifies the position of the image region.

[0121] The overlap determination unit 526 obtains the combination of candidate area information Dxy and character type information Dcl output by the character type output unit 320b, and size information Dhw for each image area, and determines whether the candidate areas overlap based on the character type class and size when the candidate area is a character image area.

[0122] When outputting candidate area information Dxy from the captured image I1, the character type class of the candidate area is not taken into consideration. Therefore, there is a risk that multiple candidate areas will be output from the same character image area. Therefore, if multiple adjacent candidate areas are determined to belong to the same character type class, it is preferable to determine one of the candidate areas as an overlapping candidate area and delete it. However, even if a candidate area is correctly determined from adjacent character images of the same character type in the captured image I1, multiple adjacent candidate areas will be determined to belong to the same character type class. Therefore, it is preferable to consider spatial overlap of each candidate area in the captured image I1. Therefore, the overlap determination unit 526 deletes overlapping candidate areas based on the candidate area information Dxy, character type information Dcl, and size information Dhw. As will be described in detail later, if an image area is determined to belong to the specified character type class corresponding to the combination of character type and specified size based on feature F7, the size of the candidate area in the captured image I1 is determined by the candidate area information Dxy and character type information Dcl. Therefore, spatial overlap in the captured image I1 does not necessarily need to be determined based on the size information Dhw. As a specific processing method, known techniques such as non-maximum suppression can be used. Depending on the determination result, the overlap determination unit 526 deletes the candidate areas and outputs the character reading result R1 of the captured image I1 based on the remaining candidate areas.

[0123] By going through the above processing flow, the determination unit 520 as a feature output unit outputs the reading result, and the user can obtain the reading result including a reading result display in which a rectangular GUI based on the output result by the size output unit 320f is superimposed and character type information based on the output result by the character type output unit 320b is displayed adjacent to the rectangular GUI.

[0124] In this embodiment, overlapping candidate areas are eliminated based on the candidate area information Dxy, character type information Dcl, and size information Dhw, but candidate areas may also be eliminated based on the candidate area information Dxy and the character type information Dcl, or based only on the candidate area information Dxy. In this case, as described above, there is a risk of a decrease in the accuracy of overlap determination, but the amount of calculation processing (processing by the character type output unit 523 in this embodiment) whose calculation amount increases depending on the number of candidate areas can be reduced. The same applies when overlap determination is performed based on the candidate area information Dxy and the size information Dhw.

[0125] (Dictionary registration details) 12 is a diagram showing a specific example of dictionary registration processing. The dictionary registration processing is on-the-job training of the AI-OCR tool T3, including updating of the character type output unit 523 by the class addition unit 315b. For example, a case where a user registers the character "β" in the dictionary will be described. The imaging unit 1 acquires image data of a captured image I1 that includes the character "β" in part, and the user selects the captured image I1 as a learning image I3 using the learning image selection unit 310a.

[0126] FIG. 13 is a diagram illustrating a GUI 540 generated by the training data setting unit 310. The user sets label information for a training image I3 using the label information setting unit 310c. When registering a training image I3 in a dictionary, the label information setting unit 310c generates a designation receiving GUI 540 for receiving the user's designation of a character position, a character size, and a label, and displays the GUI 540 on the display device 4. The designation receiving GUI 540 includes a target image display area 541 that displays the training image I3, and a label information display area 542 that displays label information to be set for the training image I3. A rectangular box 544, whose shape changes in response to a user operation, is superimposed on the target image display area 541. The label information display area 542 displays the designated character type, designated position, and designated size for each box 544. When setting label information for the training image I3 using the label information setting unit 310c, the user first manipulates the position and size of the box 544 so that it corresponds to the character image area of ​​the training image I3 displayed in the target image display area 541. When the user manipulates the box 544, the values ​​of the designated position and the designated size in the label information display area 542 change depending on the manipulation. The label information display area 542 also functions as an area for accepting user input, and the user inputs a character type corresponding to the character area into the designated character type in the label information display area 542. When a character type is input into the label information display area 542, the input character type is displayed in the character type input area 543 near the corresponding box 544 in the target image display area 541. The user can add or delete boxes 544 depending on the number of character areas on the image 500, and the label information display area 542 has display areas corresponding to the number of boxes 544. In addition, the areas in the label information display area 542 where the designated position and designated size are displayed also function as input areas. The specified position can be input as x-coordinates and y-coordinates to specify the position of the character, and the specified size can be input as width and height to specify the character size. The input operation of the character type may be performed in a character type input area 543 that is superimposed on the target image display area 541.

[0127] The example in FIG. 13 shows GUI 540 when the user performs on-site learning of two additional "β" characters with different fonts. The two "β" characters in learning image I3 look different because they are in different fonts. The user places two boxes 544 for the two character image regions "β" and specifies the position and size of the character image regions "β." For convenience, the box 544 assigned the number 1 in FIG. 13 is referred to as box 544a, and the box 544 assigned the number 2 is referred to as box 544b. The label information display area 542 displays the position and size of the corresponding character image regions according to boxes 544a and 544b, but as shown in label information display area 542, the two "β" characters are also different sizes. These two character image regions have the same character type, "β." However, when these two character image regions are additionally subjected to in-situ learning, the feature vector representing "β" specified in box 544a in GUI 540 and the feature vector representing "β" specified in box 544b are added as separate classes to dictionary matrix DM. In this manner, training data generation unit 310b generates training data D1 having designated size information indicating the size of the character image region in the image data of training image I3. The generated training data D1 is input to classifier update unit 315. In classifier update unit 315, feature extraction unit 510 executes character type feature extraction unit 510a to extract feature F7 from training image I3, and executes object feature extraction unit 510b to extract feature F6 from training image I3.

[0128] Returning to FIG. 12 , the details of the processing by the classifier update unit 315 will be described. The class addition unit 315b, which functions as the classifier update unit 315, executes the character type feature extraction unit 510a to extract feature F7 and identifies feature F3 corresponding to the character image region designated by the user based on the specified position information (x and y coordinates designated by the user) included in the training data D1. In this embodiment, two character image regions are specified by boxes 544a and 544b in the training image I3 included in the training data D1, so two feature vectors are identified, and the class addition unit 315b adds the two identified feature vectors to the dictionary matrix DM as is. Boxes 544a and 544b are character image regions of the same character type, but are registered as feature vectors corresponding to different classes. In this embodiment, the identified feature is added to the dictionary matrix DM as is; however, a feature that can be sufficiently distinguished from existing representative features based on the identified feature may be selected and added to the dictionary matrix DM.

[0129] The classification unit update unit 315a, which functions as the classifier update unit 315, executes the object feature extraction unit 510b to extract feature F6 as a feature map, and extracts feature F10 corresponding to an image area designated by the user based on designated position information included in the training data D1. Here, the image area identified by the designated position information is an image area designated by the user as a character image area, so feature F10 corresponds to an image area treated as a foreground area, not a background area. Therefore, the candidate area output unit 522a is updated so that the score, which is the probability value that the image area corresponding to feature F10 is an object area, is higher than the score used by the candidate area output unit 522a when determining a candidate area. The method for updating the candidate area output unit 522a is not particularly limited. For example, the updated foreground output unit 320c can be trained by a known method, as long as it is configured to incorporate SVM learning or cosine similarity evaluation.

[0130] While the learning tool execution unit 403 executes the size feature extraction unit 510c, the classifier update unit 315 does not execute the size feature extraction unit 510c during the dictionary registration process. Generally, updating a model that infers the size of a specific image region included in input image data using machine learning techniques increases the amount of calculation. In other words, updating the size output unit 524 using machine learning techniques so that size information Dhw based on the specified size information for the image region in the learning data D1 is difficult, resulting in a heavy learning load that makes on-site learning difficult. For this reason, the size output unit 524 is not updated during on-site learning in this embodiment, and therefore the size feature extraction unit 510c is not executed.

[0131] For this reason, in the on-site learning of this embodiment, the specified size information of the training data D1, i.e., the width and height specified by the user via the specification receiving GUI 540, is reflected in the dictionary matrix DM and applied during inference after the dictionary registration process. More specifically, the added character type class features are registered in the dictionary matrix DM as features corresponding to a combination of a character type and an image region size. Because the character image "β" specified in box 544a and the character image "β" specified in box 544b are of the same character type "β" but different sizes, the feature vector specified in box 544a is registered in the dictionary matrix DM as a feature vector corresponding to the combination of the character type "β" and the size specified in box 544a, and the feature vector specified in box 544b is registered as a feature vector corresponding to the combination of the character type "β" and the size specified in box 544b. When the learning tool execution unit 402 executes the character type output unit 523, if it determines that the feature value f71 belongs to the added character type class, it determines that the candidate area P1 is a character image of the character type specified by the learning data D1 and is an image area having the size specified by the learning data D1. In the example of Fig. 12, two character type classes with the character type "β" are added, and if it is determined that the candidate area P1 belongs to one of the two added character type classes, the character type of the candidate area P1 is determined to be "β", and the size of the candidate area P1 reflects the size of the box 544 used to identify the feature vector of the classified class.

[0132] The class adding unit 315b updates the character type output unit 523 not only when adding a character type other than those that can be classified by pre-learning, but also when adding a character image area that could not be read with an existing character type. If the inspection results output by the inspection execution unit 400 include a captured image I1 that failed to be read, the captured image I1 is used as a learning image I3 and subjected to on-site learning, thereby improving the accuracy of the character type output unit 523.

[0133] In this case, a typical example of a captured image I1 that fails to be read is a captured image I1 that includes a faded character image region. In this case, if the learning tool execution unit 403 is configured to execute the candidate area output unit 522a separately from the character type output unit 523 when executing the AI-OCR tool T3, the degradation of reading accuracy due to on-site learning can be reduced. More specifically, the faded character image region may be similar in feature quantity to the background image region. In this case, if a feature vector corresponding to the faded character image region is added to the dictionary matrix DM, the character type output unit may classify the background image region into the class corresponding to the faded character image region, resulting in a false detection. In this embodiment, the candidate area output unit 522a is executed separately from the character type output unit 523, and the character type output unit 523 performs processing based on the output from the candidate area output unit 522a, thereby reducing the risk of false detection.

[0134] The above-described embodiments are merely examples in all respects and should not be construed as limiting. Furthermore, all modifications and variations within the scope of the claims are within the scope of the present invention. [Industrial Applicability]

[0135] As described above, the image sensor according to the present disclosure can be used to inspect various types of workpieces. [Explanation of symbols]

[0136] 21 Control section 26 Storage device (storage unit) 310b Learning data generation unit 315 Classifier update unit (learning execution unit) 320 Feature Extraction Unit 320a Feature output unit S Image Sensor

Claims

1. An image sensor that executes a pre-trained character recognition model to classify an image region that is a part of an image of input image data into a first character type class, and that outputs a sensor output according to the character recognition result, a control unit that functions as a learning data generation unit that executes the pre-trained model and generates learning data, and a learning execution unit that executes updating of the pre-trained model based on the learning data; a storage unit that stores, for each of the first character type classes, a character type class representative feature that represents a feature of a character image classified into the class; The pre-trained model is a feature extraction unit including a character type class feature extraction unit that extracts features related to a character type class from the image region of the input image data as features indicating characteristics of the image region; a character type output unit that outputs a character type of the image region based on the feature extracted by the character type class feature extraction unit and the character type class representative feature of the storage unit, the learning data generation unit generates learning data having image region data and designated character type information that designates a character type shown in the image region data; The learning execution unit stores in the storage unit the character type class representative feature of a second character type class, which is different from the character type class representative feature of the first character type class, based on the feature extracted from the image region data of the learning data, so as to classify the image region data of the learning data into the second character type, which is the specified character type.

2. 10. The image sensor of claim 1, The character type output unit is an image sensor that is pre-trained using a distance learning method based on image region data classified into the first character type class.

3. 2. The image sensor of claim 1, The image sensor, wherein the control unit has a convolutional operation network inference accelerator that executes operations of a convolutional neural network as the feature extraction unit.

4. 2. The image sensor of claim 1, The pre-trained model executed by the control unit is a size feature extraction unit that extracts a size feature that is a feature related to the size of the image region; an object feature extraction unit that extracts a foreground feature that is a feature indicating whether the image region is a foreground or a background; a candidate area output unit that identifies a position of a candidate area that is a candidate for a character image area of ​​the input image data based on the foreground feature amount; an overlap determination unit that determines whether the candidate regions overlap based on the size feature amount, The control unit outputs a sensor signal in accordance with the character recognition result obtained by deleting at least one of the candidate areas determined by the overlap determination unit to be overlapping.

5. 5. The image sensor of claim 4, The overlap determination unit determines whether candidate areas corresponding to the same character image area overlap based on the character type output by the character type output unit.

6. 2. The image sensor of claim 1, the pre-trained model includes a candidate area output unit that identifies a position of a candidate area that is a candidate for a character area in the input image data based on the feature extracted by the feature extraction unit; The character type output unit outputs the character type of the image area specified by the candidate area output unit.

7. 2. The image sensor of claim 1, An image sensor in which, when the character type specified by the specified character type information is a first character type corresponding to the first character type class, the character type output unit outputs the image area data classified into the second character type class as the specified first character type.

8. 2. The image sensor of claim 1, the learning data generation unit generates learning data having designated size information indicating a size of an image area of ​​a character type designated by the designated character type information; The character type output unit outputs, as an image area of ​​a specified size, the size of the image area data classified into the second character type class in which the character type class representative feature is determined based on the learning data.

9. 2. The image sensor of claim 1, The character type class representative feature of the second character type class is stored in the non-volatile storage unit in an area separate from an area for storing the character type class representative feature of the first character type class.

10. 2. The image sensor of claim 1, The control unit outputs a character recognition result for the first image data, and generates learning data by accepting a designation for image data for which the character recognition result differs from a visual recognition result by a user.

11. 2. The image sensor of claim 1, The feature extraction unit is an image sensor that extracts the feature for an arbitrary position in the input image data.

12. 2. The image sensor of claim 1, The character type output unit is an image sensor that outputs a value indicating a character type based on a character type class feature for an arbitrary position in the input image data.

Images