Image processing device, image processing method and program

Abstract

To provide an image processing device capable of performing image processing using an AI model uniquely created by a user.SOLUTION: An image processing device includes: a flow creation unit that creates an image processing flow by selecting one or more processing items to be executed from among a plurality of processing items and setting the order of execution of the one or more processing items; and an image processing unit that executes the image processing flow. The plurality of processing items include at least one AI processing item. Each item of the at least one AI processing item defines: acquiring an AI model created externally; and outputting at least one of an inference result obtained by inputting a target image to the AI model and output information generated from the inference result.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In the field of product manufacturing, technologies have been developed that photograph objects such as parts, intermediate products, or finished products, and automatically determine attributes related to the appearance of the objects based on the images obtained. Specifically, the attributes of the objects are determined using a model obtained by performing machine learning using multiple images of objects with known attributes.

[0003] Japanese Patent Application Laid-Open No. 2022-136563 (Patent Document 1) discloses an image processing device including a learning unit and a classification unit. The learning unit generates a model by performing machine learning using the feature amount of the learning image and label data indicating the classification of the learning image. The classification unit inputs the feature amount of the image of the measurement target to the model and outputs the classification of the image of the measurement target from the model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-136563 A [Non-patent literature]

[0005] [Non-Patent Document 1] Mike Van Ness, Madeleine Udell, "CDF Normalization for Controlling the Distribution of Hidden Nodes", I (Still) Can't Believe It's Not Better Workshop at NeurIPS 2021 (October 19, 2021) Summary of the Invention [Problem to be solved by the invention]

[0006] In the technology described in Patent Document 1, a model is generated by a learning unit included in an image processing device. The learning unit generates the model according to a learning algorithm adopted by a company that provides the image processing device.

[0007] In recent years, the so-called "democratization of AI (Artificial Intelligence)" has progressed. Therefore, users of image processing devices can create their own AI models (trained models). However, since users of image processing devices do not know the details of the internal processing of the image processing devices, they cannot use the AI ​​models they have created instead of the models generated by the learning unit of the image processing devices.

[0008] The present disclosure has been made in consideration of the above-mentioned situation, and its purpose is to provide an image processing device, an image processing method, and a program capable of performing image processing using an AI model created by a user. [Means for solving the problem]

[0009] An image processing device according to one aspect of the present disclosure includes a flow creation unit and an image processing unit. The flow creation unit creates an image processing flow by selecting one or more processing items to be executed from among a plurality of processing items and setting an execution order of the one or more processing items. The image processing unit executes the image processing flow. The plurality of processing items include at least one AI processing item. Each of the at least one AI processing item defines acquiring an AI model created externally and outputting at least one of an inference result obtained by inputting a target image into the AI ​​model and output information generated from the inference result.

[0010] According to this disclosure, the image processing device can perform image processing using an AI model created by the user.

[0011] In the above disclosure, each of the at least one AI processing item defines obtaining a first parameter indicating an input image format of the AI ​​model, converting a target image to meet the input image format using the first parameter, and inputting the converted target image into the AI ​​model. According to this disclosure, the target image in an appropriate format is input into the AI ​​model.

[0012] In the above disclosure, the at least one AI processing item includes a first AI processing item using a first AI model that outputs an output image in which each pixel has a floating-point type value as an inference result. The first AI processing item defines performing one or more image processes on the output image and outputting the output image on which the one or more image processes have been performed as output information. The one or more image processes include at least one of a first process that converts the value of each pixel to an integer and a second process that normalizes the value of each pixel.

[0013] According to this disclosure, the output information output by executing the first AI process item can be easily used in subsequent processing.

[0014] In the above disclosure, the value of each pixel of the output image represents the probability of belonging to a specific class, and the first AI processing item defines obtaining a second parameter representing the difference between a threshold optimized for determining classification into a specific class and a predetermined value, and performing a second processing using the second parameter.

[0015] According to this disclosure, the predetermined value may be used as an optimal threshold for determining classification into a particular class in the normalized output image.

[0016] In the above disclosure, at least one AI processing item includes a second AI processing item using a second AI model that performs an autoencoder on a target image to output a restored image as an inference result. The second AI processing item defines generating a difference image between the target image and the restored image, and outputting the difference image or a processed image obtained by performing one or more image processes on the difference image as output information.

[0017] According to this disclosure, the difference image or the processed image may be used to detect anomalies in the target image.

[0018] In the above disclosure, the one or more image processes include at least one of a first process that converts the value of each pixel to an integer, and a second process that normalizes the value of each pixel.

[0019] According to this disclosure, the output information output by executing the second AI process item can be easily used in subsequent processing.

[0020] In the above disclosure, the second AI processing item defines obtaining a second parameter representing the difference between a threshold optimized for detecting abnormal portions from the difference image and a predetermined value, and performing a second process using the second parameter.

[0021] According to this disclosure, the predetermined value can be used as an optimal threshold for detecting abnormalities in the normalized difference image.

[0022] In the above disclosure, the flow creation unit accepts designation of an AI model to be used in the target AI processing item in response to a target AI processing item being selected as one or more processing items among at least one AI processing item, and determines whether the designated AI model is compatible with the target AI processing item.

[0023] According to this disclosure, a user can recognize that he or she has mistakenly specified an AI model that is not suitable for an AI processing item by checking the judgment results made by the flow creation unit.

[0024] In the above disclosure, at least one AI processing item includes a third AI processing item using a third AI model that outputs an output image as an inference result. The third AI processing item defines generating a first processed image obtained by performing a first image processing on the output image and a second processed image obtained by performing a second image processing on the output image, displaying the second processed image on a display, and outputting the first processed image to a subsequent processing item of the one or more processing items.

[0025] According to this disclosure, the image processing device can display a second processed image on a display that is easy for a user to view, and can output a first processed image suitable for a subsequent processing item to the subsequent processing item.

[0026] In the above disclosure, the image processing device further includes an update unit that updates the AI ​​model to a re-trained AI model. The re-trained AI model is obtained by re-training the AI ​​model using a target image. According to this disclosure, the image processing device can execute image processing using the latest AI model.

[0027] An image processing method according to one aspect of the present disclosure includes a processor creating an image processing flow by selecting one or more processing items to be executed from among a plurality of processing items and setting an execution order of the one or more processing items, and the processor executing the image processing flow. The plurality of processing items include at least one AI processing item. Each of the at least one AI processing item defines acquiring an AI model created externally and outputting at least one of an inference result obtained by inputting a target image into the AI ​​model and output information generated from the inference result.

[0028] A program according to one aspect of the present disclosure causes a computer to execute the image processing method described above. These disclosures also enable the image processing method and program to perform image processing using an AI model created by a user. Effect of the Invention

[0029] According to the present disclosure, an image processing device, an image processing method, or a program can perform image processing using an AI model created by a user. [Brief description of the drawings]

[0030] [Figure 1] FIG. 1 illustrates an example of an image processing apparatus according to an embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an example of a procedure for creating an AI model. [Diagram 3] FIG. 1 is a diagram showing a schematic structure of an AI model. [Figure 4] FIG. 1 is a diagram illustrating an example of an AI model. [Diagram 5] FIG. 13 is a diagram illustrating another example of an AI model. [Figure 6] FIG. 13 is a diagram illustrating yet another example of an AI model. [Figure 7] 2 is a schematic diagram illustrating an example of a hardware configuration of the image processing device illustrated in FIG. 1. [Figure 8] 2 is a block diagram showing an example of a functional configuration of the image processing device shown in FIG. 1. [Figure 9] FIG. 13 is a diagram illustrating an example of a screen provided by a flow creation unit. [Figure 10] FIG. 10 is a diagram showing an example of an image processing flow created using the screen shown in FIG. [Figure 11] FIG. 13 is a diagram showing an example of a screen for setting processing conditions for AI processing items. [Figure 12] FIG. 13 is a diagram showing another example of a screen for setting processing conditions for AI processing items. [Figure 13] 13 is a diagram illustrating a first example of an internal configuration of an item execution unit. [Figure 14] 13 is a diagram illustrating a second example of the internal configuration of the item execution unit. [Figure 15] FIG. 13 is a diagram illustrating a third example of the internal configuration of the item execution unit. [Figure 16] FIG. 13 is a diagram illustrating a fourth example of the internal configuration of the item execution unit. [Figure 17] FIG. 13 is a diagram illustrating a fifth example of the internal configuration of the item execution unit. [Figure 18] FIG. 4 is a diagram illustrating an example of an operation procedure of the image processing device. [Figure 19] FIG. 13 is a diagram showing a modified example of the item execution unit 14C. [Figure 20] FIG. 13 is a block diagram showing a functional configuration of an image processing device according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference characters and the description thereof will not be repeated.

[0032] §1 Examples of application First, an example of a situation in which the present invention is applied will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of an image processing device according to an embodiment. As shown in Fig. 1, a camera 200 is externally attached to an image processing device 100 according to an embodiment.

[0033] The image processing device 100 performs an image processing flow that combines one or more processing items selected from a plurality of predetermined processing items.

[0034] A user M of the image processing device 100 creates his / her own AI model 15 using a learning device 300. The learning device 300 is, for example, a server that provides a known cloud service. Alternatively, the learning device 300 may be a device owned by the user M of the image processing device 100. The learning device 300 includes a learning tool 302 and an evaluation tool 304.

[0035] The learning tool 302 generates an AI model by performing machine learning using a training data set that includes learning images and correct answer data.

[0036] The correct answer data indicates correct answer information corresponding to the training image. The type of information indicated by the correct answer data depends on the type of AI model to be generated. For example, when generating an AI model for area detection (segmentation), the correct answer data indicates an area in the training image (e.g., an area where a defect exists) and an attribute of the area (e.g., "defect").

[0037] The evaluation tool 304 evaluates the performance of the AI ​​model generated by the learning tool 302. For example, the evaluation tool 304 evaluates the performance of the AI ​​model by comparing an inference result obtained by inputting a plurality of evaluation images into the AI ​​model with correct answer data corresponding to the plurality of evaluation images.

[0038] A user M of the image processing device 100 acquires an AI model that he or she has created by using the learning device 300.

[0039] The image processing device 100 includes a flow creation unit 11 and an image processing unit 12. The flow creation unit 11 creates an image processing flow by selecting one or more processing items to be executed from a plurality of processing items 44 and setting the execution order of the selected one or more processing items. The flow creation unit 11 creates an image processing flow according to an input from a user M. The image processing unit 12 executes the created image processing flow.

[0040] The multiple processing items 44 include at least one processing item that does not use an external AI model (hereinafter referred to as a "general-purpose processing item 42") and at least one AI processing item 40 that uses an external AI model.

[0041] Each of at least one AI processing item 40 is defined to obtain an externally created AI model, and output at least one of an inference result obtained by inputting an image to be measured (hereinafter referred to as "target image") into the AI model and output information generated from the inference result. The externally created AI model includes the AI model created by the learning device 300. The target image is typically obtained from a camera externally attached to the image processing device 100. Alternatively, the target image is output from a previous processing item.

[0042] Thereby, the user M of the image processing device 100 can cause the image processing device 100 to execute an image processing flow in which one or more selected processing items 44 including AI processing items using the AI model created by the user M are executed in a desired order. In this way, the image processing device 100 can perform image processing using the AI model created by the user M.

[0043] §2 Specific Examples <Example of AI Model> In a scene of manufacturing a product, an AI model can be adopted for the appearance inspection of objects such as parts, semi-finished products, or products. Therefore, hereinafter, the AI model used for appearance inspection in the inspection line will be described. However, the use of the AI model is not limited to appearance inspection.

[0044] FIG. 2 is a diagram showing an example of the creation procedure of an AI model. Skills related to machine learning may be required for the creation of an AI model. Therefore, in the example shown in FIG. 2, the inspector and the AI model creator cooperate with each other to generate an AI model.

[0045] First, the inspector accumulates images of the object in the inspection line (step S1). The accumulated images are used as learning images for generating an AI model or evaluation images for evaluating an AI model.

[0046] Next, the inspector performs annotation for each of the stored images (step S2). Annotation is a process of adding correct answer information to an image. That is, the inspector creates correct answer data indicating the correct answer information for each image. This generates a data set including the image and the correct answer data.

[0047] Next, the inspector provides the AI ​​model creator with a plurality of data sets (step S3). The AI ​​model creator uses a portion of the plurality of data sets as a training data set and the remaining portion as an evaluation data set.

[0048] Next, the AI ​​model creator creates an AI model by applying multiple training data sets to the learning device 300 (step S4). The learning tool 302 included in the learning device 300 uses a known machine learning library such as TensorFlow (registered trademark) or Pytorch (registered trademark).

[0049] In general, an AI model receives an input of an image in which the luminance value of each pixel is expressed in a floating-point type (e.g., float type or double type). On the other hand, the luminance value of each pixel of an image accumulated in an inspection line is generally expressed in an integer type (e.g., uchar type or byte type). Therefore, the learning tool 302 converts the learning images included in the training data set from the integer type to the floating-point type.

[0050] Additionally, the learning tool 302 typically performs a standardization process on the training images in the multiple training data sets by dividing, for each pixel, the deviation of the luminance value from the mean value μ by the standard deviation σ.

[0051] FIG. 3 is a diagram showing a schematic structure of an AI model. FIG. 3 shows an AI model configured by a neural network. The neural network includes an input layer, an intermediate layer, and an output layer. However, the structure of the neural network may not be limited to such an example, and may be appropriately determined according to the embodiment. For example, the number of intermediate layers may not be limited to one, and may be two or more.

[0052] The number of neurons (nodes) in the input layer corresponds to the number of pixels in the input image. The number of neurons in the intermediate layer corresponds to the number of dimensions of the feature. The number of neurons in the output layer corresponds to the inference result.

[0053] In a neural network, neurons in adjacent layers are appropriately connected to each other, and a weight (connection weight) is set for each connection. A threshold is set for each neuron, and the output of each neuron is basically determined by whether the sum of the products of each input and each weight exceeds the threshold. The weights of the connections between neurons and the threshold of each neuron are examples of calculation parameters that define an AI model.

[0054] The learning tool 302 prepares a neural network that constitutes an AI model. The learning tool 302 executes a learning process for the neural network using multiple training data sets.

[0055] Since the training data set includes the correct answer data, the learning tool 302 can execute the learning process of the neural network using supervised learning. For example, in the first step, the learning tool 302 inputs each learning image to the input layer and executes the calculation process of the neural network. In the second step, the learning tool 302 calculates the error between the output value obtained from the output layer and the correct answer data corresponding to each learning image based on the loss function. In the third step, the learning tool 302 calculates the error of the calculation parameters of the neural network, that is, the weight of the connection between neurons and each threshold of each neuron, using the error of the calculated output value, for example, by the back propagation method. In the fourth step, the learning tool 302 updates the value of the weight of the connection between neurons and each threshold of each neuron based on each calculated error. The learning tool 302 adjusts the value of the calculation parameters so that the sum of the error between the output value output from the output layer and the correct answer data is reduced by repeating the above first to fourth steps.

[0056] The learning tool 302 may use unsupervised learning to perform the learning process of the neural network. For example, the learning tool 302 may create an AI model that performs autoencoding using a plurality of learning images that show flawless objects. The AI ​​model encodes an input image into a low-dimensional latent representation, and outputs a restored image obtained by decoding the latent representation as an inference result.

[0057] Returning to FIG. 2, next, the evaluation tool 304 of the learning device 300 evaluates the AI ​​model using multiple evaluation datasets (step S5).

[0058] The AI ​​model applied to the classification outputs the probability of belonging to a specific class (also referred to as a "score"). The specific class is, for example, a class in which a "defect" exists. The AI ​​model may output a score for an image, or may output a score for each pixel in the image. The score may generally take a value between 0 and 1. Depending on the result of comparing the score with a threshold, the classification of the image or pixel into a specific class is determined. In step S5, the evaluation tool 304 may calculate a threshold (hereinafter referred to as an "optimum threshold Th1") optimized for classification into a specific class. For example, the evaluation tool 304 calculates the threshold that maximizes the F value when the threshold is changed between 0 and 1 as the optimal threshold Th1. The F value is the harmonic mean of the precision and the recall. Furthermore, the AI ​​model may output the processing result in the middle of the network in the AI ​​model. For example, the AI ​​model may use a Grad-CAM (Gradient-weighted Class Activation Mapping) method to output information indicating where in the image attention is focused on in terms of classification into a specific class.

[0059] An AI model that performs autoencoder is generally used to detect anomalies in an object. Specifically, a difference image between a restored image, which is an inference result of the AI ​​model, and an input image represents an abnormal part of the object. Therefore, the presence or absence of a defect in the object is determined based on the difference image between the restored image and the input image. The evaluation tool 304 inputs each evaluation image included in a plurality of evaluation data sets to the AI ​​model, and generates a difference image between the restored image and the evaluation image. In the difference image, a pixel having a value exceeding a threshold is determined as a pixel showing a defect. The evaluation tool 304 calculates a threshold optimized for detecting an abnormal part (hereinafter referred to as "optimum threshold Th2") by comparing the difference image with the correct answer data.

[0060] Next, the AI ​​model creator provides the AI ​​model created by the learning device 300 to the inspection personnel (step S6).

[0061] Furthermore, the AI ​​model creator acquires a parameter set for the created AI model from the learning device 300, and provides the parameter set to the inspection staff (step S7).

[0062] The parameter set includes a first parameter indicating an input image format of the AI ​​model. For example, the first parameter includes a parameter indicating a size (image height and image width) of the input image of the AI ​​model. The first parameter includes a parameter indicating a type of luminance value of each pixel of the input image of the AI ​​model (e.g., float type or double type). In addition, the first parameter may include a mean value μ and a standard deviation σ used in the standardization process by the learning tool 302.

[0063] The parameter set may include a second parameter for normalizing an image that is an inference result of the AI ​​model. The second parameter is also referred to as a "normalization parameter." One example of the second parameter represents the difference between an optimal threshold Th1 optimized for classification into a specific class and a predetermined value. The predetermined value is typically the median (0.5) of the range of values ​​that the probability (score) of belonging to the specific class can take. Another example of the second parameter represents the difference between an optimal threshold Th2 optimized for detecting an abnormal part from a difference image and a predetermined value. The predetermined value is the median (0.5) of the range of values ​​that each pixel of the difference image can take.

[0064] Fig. 4 is a diagram showing an example of an AI model. Fig. 4 shows an AI model 15A that performs region detection (segmentation). Each training data set 90A includes a learning image 92A containing an object 91, and correct answer data 94A. The correct answer data 94A indicates a defect region 93 in the learning image 92A.

[0065] When an AI model 15A created by machine learning using multiple training data sets 90A receives an input image 96A, it outputs an output image 98A as an inference result. The value of each pixel in the output image 98A indicates the probability (score) of belonging to the "defect" class (in other words, the probability that a defect exists).

[0066] Fig. 5 is a diagram showing another example of an AI model. Fig. 5 shows an AI model 15B that reads a character string in a font specific to a user. Each training data set 90B includes a learning image 92B in which a character string in a font specific to a user appears, and correct answer data 94B. The correct answer data 94B indicates the character string appearing in the learning image 92B.

[0067] An AI model 15B created by machine learning using multiple training data sets 90B receives an input image 96B and outputs, as an inference result, a character string 98B that appears in the input image 96B.

[0068] FIG. 6 is a diagram showing yet another example of an AI model. FIG. 6 shows an AI model 15F that performs object detection. Each training data set 90F includes a learning image 92F in which an object to be detected is shown, and correct answer data 94F. In FIG. 6, a code (a barcode and a two-dimensional code) is shown as an example of an object to be detected. Note that the object to be detected is not limited to a code. For example, the object to be detected may be a specific part of a product for determining an inspection area of ​​the product. The correct answer data 94F indicates the center coordinates, width, and height of a partial area (bounding box) surrounding an object in the learning image 92F.

[0069] An AI model 15F created by machine learning using multiple training datasets 90F receives an input image 96F and outputs as an inference result the center coordinates, width, and height of a bounding box surrounding an object appearing in the input image 96F, the type label of the object, and the probability (score) of belonging to that type label.

[0070] <Example of hardware configuration of image processing device> The image processing device 100 is typically a computer having a general-purpose architecture, and executes a pre-installed program (instruction code) to perform the image processing according to the present embodiment. Such a program is typically distributed in a state stored in various recording media, or is installed in the image processing device 100 via a network, etc.

[0071] When such a general-purpose computer is used, in addition to the application for executing the image processing according to the present embodiment, an OS (Operating System) for executing basic processing of the computer may be installed. In this case, the program according to the present embodiment may call necessary modules among program modules provided as part of the OS in a predetermined sequence at a predetermined timing to execute processing. That is, the program according to the present embodiment itself may not include the above-mentioned modules, and may execute processing in cooperation with the OS. The program according to the present embodiment may be in a form that does not include some of these modules.

[0072] Furthermore, the program according to the present embodiment may be provided by being incorporated into a part of another program. In this case, the program itself does not include the modules included in the other program to be combined as described above, and the processing is executed in cooperation with the other program. That is, the program according to the present embodiment may be in a form incorporated into such other program. Note that some or all of the functions provided by the execution of the program may be implemented as a dedicated hardware circuit.

[0073] Fig. 7 is a schematic diagram showing an example of a hardware configuration of the image processing device shown in Fig. 1. As shown in Fig. 7, the image processing device 100 includes a CPU (Central Processing Unit) 110, which is an example of a processor, a main memory 112, a hard disk 114, a camera interface 116, an input interface 118, a display controller 120, a communication interface 124, and a data reader / writer 126. These components are connected to each other via a bus 128 so as to be able to communicate data with each other.

[0074] CPU 110 loads programs 115 installed on hard disk 114 into main memory 112 and executes them in a predetermined order to perform various calculations. Main memory 112 typically includes a volatile storage device such as a dynamic random access memory (DRAM), and in addition to programs 115 read from hard disk 114, it holds images acquired from camera 200, etc. Furthermore, hard disk 114 stores various data, etc., as described below. In addition to hard disk 114, or instead of hard disk 114, a semiconductor storage device such as a flash memory may be used.

[0075] Camera interface 116 mediates data transmission between CPU 110 and camera 200. That is, camera interface 116 is connected to camera 200. Camera interface 116 gives an imaging command to camera 200 in accordance with an internal command generated by CPU 110. Note that the imaging command may be output to camera 200 in response to a detection signal of a photoelectric sensor. Alternatively, the imaging command may be output to camera 200 in response to an external command from a programmable logic controller (PLC).

[0076] The camera interface 116 includes an image buffer 116a for temporarily storing an image received from the camera 200. The camera 200 includes a plurality of image sensors. Each image sensor outputs a gradation value of a corresponding pixel. The gradation value is expressed by integer digital data, such as 8-bit, 10-bit, or 12-bit, depending on the image sensor. The camera 200 includes either a monochrome image sensor or a color image sensor as an image sensor. The color image sensor may include a single-chip image sensor using a Bayer filter. The image sensor may include an A / D converter. In this case, the digital data representing the gradation value is directly output from the image sensor. The A / D converter may be external to the image sensor. The image stored in the image buffer 116a may be in a format in which the gradation values ​​of pixels are arranged according to the order of raster scan, for example. When a color image is received from the camera 200, the image buffer 116a may store an image after demosaicing. Alternatively, the image buffer 116a may store an image after performing some kind of compression processing on the image received from the camera 200. For example, if the imaging element outputs a 10-bit grayscale value, the image buffer 116a may store the most significant 8 bits of data for each pixel, excluding the least significant 2 bits.

[0077] 7, the camera 200 is externally attached to the image processing device 100. However, the camera 200 may be built into the image processing device 100.

[0078] The input interface 118 mediates data transmission between the CPU 110 and the input device 160. That is, the input interface 118 accepts input information input to the input device 160 by the user.

[0079] The display controller 120 is connected to a display 150, and controls the screen of the display 150 so as to notify the user of the processing results of the CPU 110, etc.

[0080] The communication interface 124 mediates data transmission between the CPU 110 and an external device (for example, a PC). The communication interface 124 is typically implemented by Ethernet (registered trademark) or USB (Universal Serial Bus).

[0081] Data reader / writer 126 mediates data transmission between CPU 110 and memory card 106, which is a recording medium. That is, memory card 106 is distributed in a state in which a program executed by image processing device 100 and the like are stored, and data reader / writer 126 reads out the program from memory card 106. In addition, data reader / writer 126 writes images received from camera 200 and / or processing results in image processing device 100, etc., to memory card 106 in response to an internal command from CPU 110. Note that memory card 106 is made up of a general-purpose semiconductor storage device such as SD (Secure Digital), a magnetic storage medium such as a flexible disk, an optical storage medium such as a CD-ROM (Compact Disk Read Only Memory), etc.

[0082] <Example of functional configuration of image processing device> Fig. 8 is a block diagram showing an example of a functional configuration of the image processing device shown in Fig. 1. As shown in Fig. 8, the image processing device 100 includes a flow creation unit 11, an image processing unit 12, and a storage unit 13. The flow creation unit 11 and the image processing unit 12 are realized by the CPU 110 shown in Fig. 7 executing a program 115. The storage unit 13 is realized by a main memory 112 or a hard disk 114.

[0083] The storage unit 13 stores the AI ​​model 15 created using an external learning device 300 and a parameter set 16 related to the AI ​​model 15. For example, an inspector stores the AI ​​model 15 and the parameter set 16 received from the AI ​​model creator in the storage unit 13 according to the procedure shown in FIG.

[0084] The flow creation unit 11 creates an image processing flow by setting one or more process items to be executed among a plurality of predetermined process items 44 and the execution order of the one or more process items.

[0085] As described above, one or more AI processing items 40 that perform image processing using an AI model are predefined in the image processing device 100. This allows a user of the image processing device 100 to appropriately select an AI processing item suitable for the desired appearance inspection.

[0086] Furthermore, the flow creation unit 11 sets various conditions for the process items to be executed in response to input to the input device 160.

[0087] The image processing unit 12 executes the image processing flow created by the flow creation unit 11. The image processing unit 12 includes an item execution unit 14 corresponding to each processing item to be executed. The image processing unit 12 includes, for example, the OpenVINO (registered trademark) tool to perform inference using an AI model. The OpenVINO tool is provided by Intel and is an inference engine that uses an AI model.

[0088] <Example of flow creation process> 9 is a diagram showing an example of a screen provided by the flow creation unit 11. A screen 30 shown in FIG.

[0089] The screen 30 includes a set item display area 32, a process item selection area 34, a camera image display area 36, ​​an insert / add process item button 38, and an execution order swap button 39. The set item display area 32 graphically displays the contents of the currently set process flow.

[0090] Icons representing a plurality of predefined process items 44 are displayed together with their names in a list in the process item selection area 34. As shown in Fig. 9, the list displayed in the process item selection area 34 includes a plurality of AI process items 40 and a plurality of general-purpose process items 42 that do not use an external AI model.

[0091] The AI ​​processing items 40 include, for example, AI processing items 40a, 40c to 40e. The AI ​​processing item 40c uses the AI ​​model 15C that performs classification. That is, the AI ​​model 15C outputs the belonging probability (score) of each class based on the characteristics of the input image.

[0092] The AI ​​processing item 40d uses the AI ​​model 15D that performs autoencoding. That is, the AI ​​model 15D encodes an input image into a low-dimensional latent representation, and outputs a restored image obtained by decoding the latent representation as an inference result.

[0093] The AI ​​processing item 40e uses the AI ​​model 15E that performs object recognition. That is, the AI ​​model 15E outputs area information indicating a rectangular area in which an object appears in an input image and the probability (score) of each class of an object moving to the rectangular area as an inference result.

[0094] The general-purpose processing items 42 include known image processing items, for example, a general-purpose processing item 42a that performs labeling.

[0095] A user of the image processing device 100 selects a processing item required for the desired image processing in the processing item selection area 34 of the screen 30 ((1) Select processing item). Furthermore, the user selects the position (order) at which the selected processing item should be added in the set item display area 32 ((2) Select adding position). The user adds a processing item to the set item display area 32 by selecting the insert / add processing item button 38 ((3) Press the insert / add button) ((4) The processing item is added). The user can create a desired image processing flow by repeating this process as appropriate. Furthermore, the user can change the execution order as appropriate during or after the creation of the processing settings by selecting a processing item in the set item display area 32 and then selecting the change execution order button 39.

[0096] Fig. 10 is a diagram showing an example of an image processing flow created using the screen shown in Fig. 9. The image processing flow 50 shown in Fig. 10 includes a general-purpose processing item 42b for acquiring a target image from the camera 200, an AI processing item 40a, and a general-purpose processing item 42a for performing labeling.

[0097] The flow creator 11 sets a processing condition for each of the one or more processing items selected as the processing items to be executed.

[0098] Fig. 11 is a diagram showing an example of a screen for setting processing conditions for AI processing items. A screen 60 shown in Fig. 11 is used to set an AI model 15. The screen 60 includes setting areas 61 and 62, display areas 63 and 64, and a button 65.

[0099] The setting area 61 is used to set a hardware device that executes the inference processing using the AI ​​model 15. The setting area 61 includes an input field 61a for inputting a hardware device. In response to an operation of the input field 61a, the flow creation unit 11 displays a pull-down menu of hardware devices capable of executing the inference processing, and prompts the user to select a device from the pull-down menu. The flow creation unit 11 sets the selected device as the hardware device that executes the inference processing using the AI ​​model 15.

[0100] The setting area 62 is used to specify the AI ​​model 15. The setting area 62 includes a radio button 62a and input fields 62b and 62c.

[0101] As described above, the image processing unit 12 includes, for example, the OpenVINO (registered trademark) tool. The formats of the AI ​​model executable in the OpenVINO tool include the IR (Intermediate Representation) format and the ONNX (Open Neural Network Exchange) format. Therefore, the user converts the format of the AI ​​model 15 created by the user using the learning device 300 into the IR format or the ONNX format in advance. The radio button 62a is used to select the format of the AI ​​model 15. Note that, if the format of the AI ​​model created using the learning device 300 matches or is compatible with the format of the AI ​​model that the image processing unit 12 can read, the user does not need to convert the format of the AI ​​model 15.

[0102] The input fields 62b and 62c receive the designation of the AI ​​model 15. Specifically, the file path of the AI ​​model 15 is input into the input fields 62b and 62c. The AI ​​model 15 in the IR format is composed of an xml format file and a bin format file. The input field 62b is used to designate the xml format file. The input field 62c is used to designate the bin format file.

[0103] The display area 63 is used to display the input image conditions of the AI ​​model that can be used in the corresponding AI processing item and the output (inference result) conditions of the AI ​​model. The input image format includes the batch number "n", the number of channels "c", the image height "h", and the image width "w".

[0104] The conditions of the inference result of the AI ​​model that outputs an image include the number of batches “n”, the number of channels “c”, the image height “h”, and the image width “w”. The AI ​​model that outputs an image includes, for example, the AI ​​model 15A shown in FIG.

[0105] The conditions for the inference result of the AI ​​model that does not output images include the number of batches “n” and the number of channels “c.” The AI ​​model that does not output images includes, for example, the AI ​​model 15B shown in FIG.

[0106] The display area 64 is used to display the time taken to load the AI ​​model 15 .

[0107] The button 65 is used to start reading the AI ​​model 15. In response to clicking the button 65, the flow creation unit 11 starts reading the AI ​​model 15 according to the file paths input in the input fields 62b and 62c.

[0108] The format of the AI ​​model 15 may differ depending on the AI ​​processing item. For example, the AI ​​processing item 40a corresponding to the image processing "segmentation" uses the AI ​​model 15 (for example, the AI ​​model 15A shown in FIG. 4) that outputs an output image in which the value of each pixel indicates the probability of belonging to a specific class. Therefore, the AI ​​processing item 40a cannot use the AI ​​model 15 that does not output an image (for example, the AI ​​model 15B shown in FIG. 5). Therefore, the flow creation unit 11 may determine whether or not the AI ​​model specified in the input fields 62b and 62c is suitable for the AI ​​processing item. Specifically, the flow creation unit 11 compares the format of the output data of the AI ​​model specified in the input fields 62b and 62c with the condition (hereinafter referred to as "output condition") of the output (inference result) of the AI ​​model usable in the AI ​​processing item. The flow creation unit 11 determines that the specified AI model is suitable for the AI ​​processing item when the format of the output data of the specified AI model satisfies the output condition. When the format of the output data of the specified AI model does not satisfy the output condition, the flow creation unit 11 determines that the specified AI model does not match the AI ​​processing item.

[0109] If the flow creation unit 11 determines that the specified AI model does not match the AI ​​processing item, it may output an error message.

[0110] Fig. 12 is a diagram showing another example of a screen for setting processing conditions for an AI processing item. A screen 60A shown in Fig. 12 is used for setting conditions (hereinafter referred to as "judgment conditions") for judging the quality of the appearance of an object based on the inference result of AI model 15. The judgment conditions are set, for example, for AI processing item 40c that uses AI model 15C that performs classification.

[0111] As shown in FIG. 12, the screen 60A includes an input field 66 for inputting a lower limit value and an input field 67 for inputting an upper limit value for each class. As described above, the AI ​​model 15C outputs the belonging probability of each class based on the features of the input image. The belonging probability has a value from 0 to 1. Therefore, values ​​from 0 to 1 are input into the input fields 66 and 67. The user inputs the lower limit value and the upper limit value of the range that the belonging probability (score) can take when the appearance of the object is good into the input fields 66 and 67, respectively.

[0112] Alternatively, the flow creation unit 11 may set, as a judgment condition, a condition that a character string can take, for an AI processing item that uses an AI model 15B (see FIG. 5) that outputs a character string as an inference result in response to a user input.

[0113] Furthermore, the flow creation unit 11 sets a parameter set 16 corresponding to the AI ​​model 15 set in the AI ​​processing item in response to the user's input. The user only needs to input the parameter set 16 provided in step S7 of FIG. 2.

[0114] <Example of item execution section> Next, first to fifth examples of the internal configuration of the item execution section 14 will be described with reference to Figs.

[0115] (First example) Fig. 13 is a diagram showing a first example of the internal configuration of an item execution unit 14A that executes AI processing items.

[0116] 13, the item execution unit 14A includes an AI model reading unit 71, an image conversion unit 72, and an inference unit 73. The AI ​​model reading unit 71 and the inference unit 73 are realized by, for example, the OpenVINO (registered trademark) tool.

[0117] The AI ​​model reading unit 71 reads a file of a specified AI model 15. For example, the AI ​​model reading unit 71 reads an AI model 15A (see FIG. 4) that outputs an output image in which the value of each pixel indicates the probability of belonging to a specific class as an inference result. Alternatively, the AI ​​model reading unit 71 may read an AI model 15B (see FIG. 5) that outputs a character string appearing in an input image as an inference result. Alternatively, the AI ​​model reading unit 71 may read an AI model 15C that performs classification or an AI model 15E that performs object recognition. Alternatively, the AI ​​model reading unit 71 may read an AI model 15F (see FIG. 6) that outputs the center coordinates, width, and height of a bounding box surrounding an object appearing in an input image, the type label of the object, and a score as an inference result.

[0118] The AI ​​model reading unit 71 reads a file in a format of an AI model executable in the OpenVINO tool. However, when a file having a format different from the format of an AI model executable in the OpenVINO tool is specified, the AI ​​model reading unit 71 may convert the format of the specified file. The AI ​​model reading unit 71 may convert the format of the AI ​​model using a "model optimizer" provided by the OpenVINO tool.

[0119] The image conversion unit 72 converts the format of the target image into a format suitable for the AI ​​model 15 based on the first parameter included in the parameter set 16.

[0120] Specifically, the image conversion unit 72 converts the size (height and width) of the target image into the size indicated by the first parameter.

[0121] In addition, the image conversion unit 72 converts the format of the value of each pixel of the target image. For example, the image conversion unit 72 converts the value of each pixel of the target image from an integer type (e.g., uchar type or byte type) to a floating-point type (e.g., float type or double type) indicated by the first parameter.

[0122] Furthermore, the image conversion unit 72 performs the same standardization process on the target image as the standardization process performed on the learning image when the AI ​​model 15 was learned. That is, the image conversion unit 72 performs the standardization process on the target image using the average value μ and standard deviation σ included in the first parameter. As described above, the standardization process is a process of dividing the deviation of the luminance value of each pixel from the average value μ by the standard deviation σ.

[0123] The inference unit 73 performs inference using the AI ​​model 15 read by the AI ​​model reading unit 71. The inference unit 73 inputs the target image converted by the image conversion unit 72 to the AI ​​model 15 and obtains an inference result. The item execution unit 14A outputs the inference result to the outside.

[0124] (Second example) Fig. 14 is a diagram showing a second example of the internal configuration of the item execution unit. Fig. 14 shows the internal configuration of item execution unit 14B that executes AI processing items. As shown in Fig. 14, item execution unit 14B differs from item execution unit 14A shown in Fig. 13 in that it includes a determination unit 74.

[0125] The judgment unit 74 judges whether or not the inference result obtained by the inference unit 73 satisfies a judgment condition. The judgment condition is set by the flow creation unit 11 in advance.

[0126] The item execution unit 14B outputs the determination result by the determination unit 74 to the outside. The determination result is an example of "output information" in the present disclosure. The item execution unit 14B may output the inference result obtained by the inference unit 73 to the outside in addition to the determination result.

[0127] When the AI ​​model reading unit 71 reads the AI ​​model 15A (see FIG. 4), for example, a condition is set as the judgment condition that the probability of belonging to the "defect" class does not exceed a preset upper or lower limit value. Alternatively, a condition is set as the judgment condition that the area of ​​the region in which the probability of belonging to the "defect" class exceeds a first threshold value is equal to or smaller than a second threshold value. In this case, a user of the image processing device 100 can recognize whether the appearance of the object is good or bad by checking the judgment result.

[0128] Alternatively, when the AI ​​model reading unit 71 reads the AI ​​model 15B (see FIG. 5), for example, a condition that the character string may take is set as the judgment condition. In this case, the user of the image processing device 100 can recognize the presence or absence of an abnormality in the character string by checking the judgment result.

[0129] Alternatively, when the AI ​​model reading unit 71 reads the AI ​​model 15C that performs classification, for example, a condition that the probability of belonging to the "defect" class is less than a threshold is set as a judgment condition. In this case, a user of the image processing device 100 can recognize the presence or absence of a defect in the object by checking the judgment result.

[0130] (Third example) Fig. 15 is a diagram showing a third example of the internal configuration of the item execution unit. Fig. 15 shows the internal configuration of item execution unit 14C that executes AI processing items. As shown in Fig. 15, item execution unit 14C differs from item execution unit 14A shown in Fig. 13 in that it includes a normalization unit 75 and an image conversion unit 76.

[0131] The AI ​​model reading unit 71 of the item execution unit 14C reads the AI ​​model 15 that outputs an image as an inference result. For example, the AI ​​model reading unit 71 reads the AI ​​model 15A (see FIG. 4) that outputs an output image in which the value of each pixel indicates the probability of belonging to a specific class.

[0132] The normalization unit 75 performs normalization processing on the image that is the inference result obtained by the inference unit 73, based on the second parameter included in the parameter set 16. As described above, the second parameter represents the difference between the optimal threshold Th optimized for classification into a specific class and a predetermined value (typically 0.5).

[0133] The normalization unit 75 normalizes the value of each pixel of the image according to, for example, a method based on the Min-Max method. Specifically, the normalization unit 75 normalizes the value val of each pixel according to the following equation (1). val'={(val-threshold) / (max-min)}+0.5 (1) The threshold is a value represented by the second parameter. The max is the maximum value of the pixel. The min is the minimum value of the pixel. The pixel value indicates the probability of belonging, and can take a value between 0 and 1. Therefore, a fixed value of "1" may be used as (max-min).

[0134] By performing normalization according to equation (1) above, a predetermined value (typically 0.5) can be used as an optimal threshold for determining classification into a particular class in the normalized image.

[0135] In addition, the normalization unit 75 may perform normalization using a cumulative distribution function (CDF) (see Mike Van Ness, Madeleine Udell, "CDF Normalization for Controlling the Distribution of Hidden Nodes", I (Still) Can't Believe It's Not Better Workshop at NeurIPS 2021 (October 19, 2021) (Non-Patent Document 1)).

[0136] The image conversion unit 76 converts the value of each pixel of the image normalized by the normalization unit 75 from a floating-point type (e.g., float type or double type) to an integer type (e.g., uchar type or byte type). That is, the image conversion unit 76 converts the value of each pixel to an integer. The image conversion unit 76 not only converts to an integer type, but also performs a magnification conversion from the value range (0 to 1) changed by the above standardization process to a range that can be expressed as an image (e.g., 0 to 255).

[0137] The item execution section 14C outputs to the outside the image (hereinafter referred to as a "processed output image") converted by the image conversion section 76. The processed output image is an example of "output information" in the present disclosure.

[0138] (Example 4) Fig. 16 is a diagram showing a fourth example of the internal configuration of the item execution unit. Fig. 16 shows the internal configuration of an item execution unit 14D that executes AI processing items. As shown in Fig. 16, the item execution unit 14D differs from the item execution unit 14C shown in Fig. 15 in that it includes a difference image generation unit 77.

[0139] The AI ​​model reading unit 71 of the item execution unit 14D reads the AI ​​model 15D that performs autoencoding. As described above, the AI ​​model 15D encodes an input image into a low-dimensional latent representation, and outputs a restored image obtained by decoding the latent representation as an inference result.

[0140] The difference image generating unit 77 generates a difference image between the restored image, which is the inference result obtained by the inference unit 73 , and the target image converted by the image converting unit 72 .

[0141] The normalization unit 75 performs normalization processing on the difference image based on a second parameter included in the parameter set 16. The second parameter represents the difference between an optimal threshold Th2 optimized for detecting an abnormal portion from the difference image and a predetermined value (typically 0.5). As a result, the predetermined value (typically 0.5) can be used as the optimal threshold for detecting an abnormal portion in the normalized difference image.

[0142] The image conversion unit 76 converts the value of each pixel of the difference image normalized by the normalization unit 75 from a floating-point type (e.g., float type or double type) to an integer type (e.g., uchar type or byte type). The image conversion unit 76 not only converts to an integer type, but also performs a magnification conversion from the value range (0 to 1) changed by the above standardization process to a range that can be expressed as an image (e.g., 0 to 255).

[0143] The item execution unit 14D outputs to the outside the image (hereinafter referred to as "processed difference image") converted by the image conversion unit 76. The processed difference image is an example of "output information" in the present disclosure.

[0144] In the fourth example of the internal configuration of the item execution unit 14D, the normalization unit 75 may be omitted. In this case, the item execution unit 14D may perform integer type conversion and magnification conversion on the restored image generated by the inference unit 73, and then generate a difference image between the integer-converted restored image and the target image.

[0145] (Fifth Example) Fig. 17 is a diagram showing a fifth example of the internal configuration of the item execution unit. Fig. 17 shows the internal configuration of item execution unit 14E corresponding to the general-purpose processing item "labeling". As shown in Fig. 17, item execution unit 14E includes binarization processing unit 81, blobbing unit 82, and determination unit 83.

[0146] The binarization processing unit 81 binarizes the input image. Specifically, the binarization processing unit 81 converts the value of each pixel of the input image into 0 or 1 based on a preset threshold value.

[0147] The blobbing unit 82 assigns a label to a pixel cluster that has consecutive pixels with a value of "1" (or "0").

[0148] The determination unit 83 determines whether or not the pixel block to which the label is assigned by the blobbing unit 82 satisfies a determination condition. The determination condition is set in advance by the flow creation unit 11. For example, the determination condition is that the area of ​​the pixel block is equal to or smaller than a threshold value. The item execution unit 14E outputs the determination result by the determination unit 83 to the outside.

[0149] (Examples of application of the item execution part of the first to fifth examples) A user of the image processing device 100 creates an image processing flow shown in Fig. 10 using the screen 30 shown in Fig. 9. In this case, the image processing unit 12 includes an item execution unit 14C corresponding to the AI ​​processing item 40a included in the image processing flow shown in Fig. 10, and an item execution unit 14E corresponding to the general-purpose processing item 42b. The item execution unit 14E performs labeling on the processed output image output from the item execution unit 14C.

[0150] The normalization unit 75 of the item execution unit 14C performs normalization processing on the image that is the inference result obtained by the inference unit 73 according to the above formula (1). The difference between the optimal threshold Th1 determined by the evaluation tool 304 and a predetermined value (typically 0.5) is substituted for threshold in formula (1). Therefore, the predetermined value (typically 0.5) is an optimal threshold for determining classification into a specific class in the normalized image. Therefore, the user may set a value (typically 127 or 128) obtained by converting the predetermined value (typically 0.5) into an integer as the threshold of the binarization processing unit 81 of the item execution unit 14E. This allows the user to easily set an optimal threshold for the binarization processing unit 81. In other words, the user does not need to adjust the threshold of the binarization processing unit 81.

[0151] A user of the image processing device 100 may create an image processing flow including an AI processing item 40d and a general-purpose processing item 42b of "labeling" by using the screen 30 shown in Fig. 9. In this case, the image processing unit 12 includes an item execution unit 14D corresponding to the AI ​​processing item 40d and an item execution unit 14E corresponding to the general-purpose processing item 42b. The item execution unit 14E performs labeling on the processed difference image output from the item execution unit 14D.

[0152] The normalization unit 75 of the item execution unit 14D performs normalization processing on the difference image generated by the difference image generation unit 77 according to the above formula (1). The difference between the optimal threshold Th2 determined by the evaluation tool 304 and a predetermined value (typically 0.5) is substituted for threshold in formula (1). Therefore, in the normalized difference image, the predetermined value (typically 0.5) is the optimal threshold for detecting an abnormal part from the difference image. Therefore, the user may set a value (typically 127 or 128) obtained by converting the predetermined value (typically 0.5) into an integer as the threshold of the binarization processing unit 81 of the item execution unit 14E. This allows the user to easily set an optimal threshold for the binarization processing unit 81. In other words, the user does not need to adjust the threshold of the binarization processing unit 81.

[0153] A user of the image processing device 100 may create an image processing flow including an AI processing item 40c and a general-purpose processing item 42 of "threshold judgment" using the screen 30 shown in FIG. 9. In this case, the image processing unit 12 includes an item execution unit 14A corresponding to the AI ​​processing item 40c and an item execution unit 14 corresponding to the general-purpose processing item 42 of "threshold judgment". The item execution unit 14 corresponding to the general-purpose processing item 42 of "threshold judgment" judges whether or not the inference result output from the item execution unit 14A satisfies a preset judgment condition, and outputs the judgment result. In this case, for example, a condition that the belonging probability of a specific class is within a set range is set as the judgment condition.

[0154] A user of the image processing device 100 may create an image processing flow including an AI processing item 40e and a general-purpose processing item 42 of "threshold judgment" using the screen 30 shown in FIG. 9. In this case, the image processing unit 12 includes an item execution unit 14A corresponding to the AI ​​processing item 40e and an item execution unit 14 corresponding to the general-purpose processing item 42 of "threshold judgment". The item execution unit 14 corresponding to the general-purpose processing item 42 of "threshold judgment" judges whether or not the inference result output from the item execution unit 14A satisfies a preset judgment condition, and outputs the judgment result. In this case, as the judgment condition, for example, a condition regarding the size or position of a rectangular area in which the belonging probability of a specific class exceeds a threshold is set.

[0155] <Operation procedure of image processing device> 18 is a diagram showing an example of an operation procedure of the image processing device. First, an inspection personnel prepares an AI model 15 (step S11). Specifically, the inspection personnel stores a file of the AI ​​model 15 in the image processing device 100. Furthermore, the inspection personnel prepares a parameter set 16 (step S12).

[0156] Next, the inspector sets the file path of the AI ​​model 15 in the image processing device 100 (step S13).

[0157] Next, the inspector instructs the image processing device 100 to read the AI ​​model 15 (step S14). As a result, the image processing device 100 reads the file of the AI ​​model 15 based on the set file path.

[0158] The image processing device 100 notifies that the reading of the file of the AI ​​model 15 is completed (step S15).

[0159] Next, the inspector sets the parameter set 16 in the image processing device 100 (step S16).

[0160] Next, the inspection personnel instructs the image processing device 100 to execute image processing (step S17).

[0161] The CPU 110 of the image processing device 100 executes the specified image processing (step S18). Specifically, the CPU 110 executes the image processing according to a previously created image processing flow.

[0162] Next, CPU 110 outputs the result of the image processing (step S19), whereby an inspector can check the output result and determine whether the appearance of the object is good or bad.

[0163] 18, in step S18, the CPU 110 executes image processing including at least one AI processing item. Therefore, step S18 includes steps S18a to S18c corresponding to each of the at least one AI processing items.

[0164] In step S18a, the CPU 110 acquires a target image. For example, the CPU 110 acquires the target image from the camera 200. Alternatively, the CPU 110 acquires an image obtained by executing a processing item preceding the AI ​​processing item as the target image.

[0165] In step S18b, the CPU 110 converts the target image based on the first parameter included in the parameter set 16 so as to satisfy the input image format of the AI ​​model 15.

[0166] In step S18c, the CPU 110 executes an inference process using the AI ​​model 15. Specifically, the CPU 110 inputs the converted target image to the AI ​​model 15, thereby acquiring an inference result of the AI ​​model 15.

[0167] <Advantages> As described above, the image processing device 100 according to the present embodiment includes a flow creation unit 11 that creates an image processing flow by selecting one or more processing items to be executed from among a number of processing items and setting the execution order of the one or more processing items, and an image processing unit 12 that executes the image processing flow. The multiple processing items include at least one AI processing item 40. Each of the at least one AI processing item 40 defines acquiring an AI model 15 created externally and outputting at least one of an inference result obtained by inputting a target image to the AI ​​model 15 and output information generated from the inference result.

[0168] This allows a user of the image processing device 100 to cause the image processing device 100 to execute an image processing flow that executes multiple processing items, including AI processing items that use an AI model that the user has created himself / herself, in a desired order.

[0169] Each of the at least one AI processing item 40 defines obtaining a first parameter indicating an input image format of the AI ​​model 15, converting a target image using the first parameter to satisfy the input image format, and inputting the converted target image to the AI ​​model 15.

[0170] At least one AI processing item 40 includes, for example, an AI processing item 40a that uses an AI model 15 that outputs an output image in which each pixel has a floating-point value as an inference result. The AI ​​processing item 40a is an example of a “first AI processing item” in the present disclosure.

[0171] The AI ​​processing item 40a defines the execution of one or more image processes on an output image and the output image on which the one or more image processes have been performed as output information. The one or more image processes include an integerization process that converts the value of each pixel into an integer, and a normalization process that normalizes the value of each pixel. This makes it easier to use the output information from the item execution unit 14C that executes the AI ​​processing item 40a in subsequent processing.

[0172] For example, the value of each pixel in the output image represents the probability of belonging to a specific class. The AI ​​processing item 40a defines obtaining a second parameter representing the difference between an optimal threshold Th1 optimized for determining classification into a specific class and a predetermined value, and performing a normalization process using the second parameter. As a result, the predetermined value can be used as an optimal threshold for determining classification into a specific class in the normalized output image.

[0173] At least one of the AI ​​processing items 40 includes, for example, an AI processing item 40d that uses an AI model 15D that outputs a restored image as an inference result by performing an autoencoder on a target image. The AI ​​processing item 40d is an example of a "second AI processing item" in the present disclosure.

[0174] The AI ​​processing item 40d defines the steps of generating a difference image between the converted target image and the restored image, and outputting the processed difference image obtained by executing one or more image processes on the difference image as output information. The one or more image processes include an integerization process that converts the value of each pixel into an integer, and a normalization process that normalizes the value of each pixel. This makes it easier to use the output information from the item execution unit 14D that executes the AI ​​processing item 40d in subsequent processing.

[0175] The AI ​​processing item 40d defines obtaining a second parameter representing the difference between an optimal threshold Th2 optimized for detecting an abnormal part from the difference image and a predetermined value, and performing a normalization process using the second parameter, so that the predetermined value can be used as an optimal threshold for detecting an abnormal part in the normalized difference image.

[0176] In response to an AI processing item 40 being set as one or more processing items to be executed, the flow creation unit 11 accepts designation of an AI model 15 and determines whether or not the designated AI model 15 is compatible with the AI ​​processing item 40. This allows a user of the image processing device 100 to recognize that he or she has erroneously designated an AI model that is not suitable for the AI ​​processing item by checking the determination result by the flow creation unit 11.

[0177] <Modification> In item execution units 14C and 14D, one of normalization unit 75 and image conversion unit 76 may be omitted.

[0178] FIG. 19 is a diagram showing a modified example of the item execution unit shown in FIG. 15. As shown in FIG. 19, item execution unit 14F differs from item execution unit 14C shown in FIG. 15 in that it includes an image conversion unit 78. The item execution unit 14F uses an AI model 15 (e.g., AI model 15A) that outputs an output image as an inference result. AI model 15A is an example of a "third AI model" of the present disclosure.

[0179] The image conversion unit 78 performs image processing for display on the image of the inference result obtained from the inference unit 73. For example, the image conversion unit 78 performs image processing to add a specific color to an area where the belonging probability (score) of a specific class exceeds a preset threshold. The item execution unit 14F causes the display 150 to display the image that has been image-processed by the image conversion unit 78 (corresponding to the "second processed image" of the present disclosure).

[0180] Fig. 20 is a block diagram showing a functional configuration of an image processing device according to a modified example. As shown in Fig. 19, the image processing device 100 according to the modified example is different from the image processing device 100 shown in Fig. 8 in that it includes an update unit 17.

[0181] The update unit 17 updates the AI ​​model 15 to a re-learned AI model. The re-learned AI model is obtained by re-learning the AI ​​model using a target image.

[0182] Specifically, the update unit 17 accumulates the target image input to the AI ​​model 15. The update unit 17 may upload the target image to a server on a cloud system, or may upload the target image to an on-premise hard disk drive. The update unit 17 may upload the target image to a storage destination every time the target image is input to the AI ​​model 15, or may upload a plurality of target images input to the AI ​​model 15 during a certain period of time to a storage destination all at once.

[0183] The AI ​​model creator performs annotation on the accumulated target images. As a result, each target image is assigned with correct answer data. The AI ​​model creator sets the target images as re-learning images, and creates a re-learning dataset including the re-learning images and correct answer data. The AI ​​model creator inputs the re-learning dataset into the learning device 300. As a result, the learning device 300 re-learns the AI ​​model 15 using the re-learning dataset.

[0184] The update unit 17 acquires the re-learned AI model periodically or in response to an instruction from a user. At this time, the update unit 17 also acquires a parameter set corresponding to the re-learned AI model. The update unit 17 updates the AI ​​model 15 and the parameter set 16 to the re-learned AI model and the latest parameter set. As a result, the item execution unit 14, which executes the AI ​​processing items, executes the inference process using the re-learned AI model. Furthermore, the item execution unit 14 executes the conversion process and normalization process of the target image based on the latest parameter set.

[0185] §3 Supplementary Note As described above, the present embodiment includes the following disclosure.

[0186] (Configuration 1) An image processing device (100), a flow creation unit (11, 110) that creates an image processing flow by selecting one or more processing items to be executed from among a plurality of processing items and setting an execution order of the one or more processing items; An image processing unit (12, 110) that executes the image processing flow, The plurality of processing items includes at least one AI processing item (40); Each of the at least one AI processing item (40) Acquiring an externally created AI model (15); and The image processing device (100) defines outputting at least one of an inference result obtained by inputting a target image into the AI ​​model (15) and output information generated from the inference result.

[0187] (Configuration 2) Each of the at least one AI processing item (40) Obtaining a first parameter indicating an input image format of the AI ​​model (15); converting the target image to conform to the input image format using the first parameters; and inputting the transformed target image into the AI ​​model (15).

[0188] (Configuration 3) The at least one AI processing item (40) includes a first AI processing item (40a) using a first AI model (15A) that outputs an output image in which each pixel has a floating-point value as the inference result; The first AI processing item (40a) is performing one or more image processes on the output image; outputting the output image on which the one or more image processing operations have been performed as the output information; The image processing device (100) according to configuration 1, wherein the one or more image processes include at least one of a first process for converting a value of each pixel to an integer, and a second process for normalizing a value of each pixel.

[0189] (Configuration 4) the value of each pixel in the output image represents a probability of belonging to a particular class; The first AI processing item (40a) is Obtaining a second parameter representing a difference between a threshold value optimized for determining classification into the specific class and a predetermined value; and executing the second process using the second parameter.

[0190] (Configuration 5) the at least one AI processing item (40) includes a second AI processing item (40d) using a second AI model (15D) that performs an autoencoder on the target image to output a restored image as the inference result; The second AI processing item is: generating a difference image between the target image and the restored image; and outputting, as the output information, the difference image or a processed image obtained by performing one or more image processes on the difference image.

[0191] (Configuration 6) The image processing device (100) according to configuration 5, wherein the one or more image processes include at least one of a first process for converting a value of each pixel to an integer, and a second process for normalizing a value of each pixel.

[0192] (Configuration 7) The second AI processing item (40d) is acquiring a second parameter representing a difference between a threshold value optimized for detecting an abnormal portion from the difference image and a predetermined value; and executing the second process using the second parameters.

[0193] (Configuration 8) The flow creation unit (11, 110) In response to a target AI processing item being selected as the one or more processing items among the at least one AI processing item, a designation of an AI model (15) to be used in the target AI processing item is accepted; The image processing device (100) according to any one of configurations 1 to 7, which determines whether or not a specified AI model is compatible with the target AI processing item.

[0194] (Configuration 9) the at least one AI processing item (40) includes a third AI processing item using a third AI model that outputs an output image as the inference result; The third AI processing item is: generating a first processed image obtained by performing a first image processing on the output image and a second processed image obtained by performing a second image processing on the output image; displaying the second processed image on a display (150); and outputting the first processed image to a subsequent processing item of the one or more processing items.

[0195] (Configuration 10) The image processing device (100) according to configuration 1, further comprising an update unit (17, 110) that updates the AI ​​model (15) to a re-learned AI model, and the re-learned AI model is obtained by re-learning the AI ​​model (15) using the target image.

[0196] (Configuration 11) 1. An image processing method, comprising: A processor (110) selects one or more processing items to be executed from among a plurality of processing items, and sets an execution order of the one or more processing items to create an image processing flow; The processor (110) executes the image processing flow, The plurality of process items includes at least one AI process item; Each of the at least one AI process item comprises: Acquiring an externally created AI model (15); and The image processing method defines outputting at least one of an inference result obtained by inputting a target image into the AI ​​model (15) and output information generated from the inference result.

[0197] (Configuration 12) A program (115) for causing a computer (110) to execute an image processing method, The image processing method includes: creating an image processing flow by selecting one or more processing items to be executed from among a plurality of processing items and setting an execution order of the one or more processing items; Executing the image processing flow, The plurality of process items includes at least one AI process item; Each of the at least one AI process item comprises: Acquiring an externally created AI model (15); and A program (115) that defines outputting at least one of an inference result obtained by inputting a target image into the AI ​​model (15) and output information generated from the inference result.

[0198] Although the embodiment of the present invention has been described, the embodiment disclosed herein should be considered as illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, and it is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0199] 11 flow creation unit, 12 image processing unit, 13 memory unit, 14, 14A to 14F item execution unit, 15, 15A to 15F AI model, 16 parameter set, 17 update unit, 20, 30, 60, 60A screen, 32 set item display area, 34 processing item selection area, 36 camera image display area, 38 insert / add button, 39 order change button, 40, 40a to 40e AI processing item, 42, 42a, 42b processing item, 50 image processing flow, 61, 62 setting area, 61a, 62b, 62c, 66, 67 input field, 62a radio button, 63, 64 display area, 65 button, 71 AI model loading unit, 72, 76, 78 image conversion unit, 73 inference unit, 74, 83 judgment unit, 75 normalization unit, 77 Difference image generation unit, 81 binarization processing unit, 82 blobbing unit, 90A, 90B, 90F training data set, 91 object, 92A, 92B, 92F learning image, 93 defect area, 94A, 94B, 94F correct answer data, 96A, 96B, 96F input image, 98A output image, 98B character string, 98F inference result, 100 image processing device, 106 memory card, 110 CPU, 112 main memory, 114 hard disk, 115 program, 116 camera interface, 116a image buffer, 118 input interface, 120 display controller, 124 communication interface, 126 data reader / writer, 128 bus, 150 display, 160 input device, 200 camera, 300 learning device, 302 learning tool, 304 evaluation tool, M user.

Claims

1. An image processing device, a flow creation unit that creates an image processing flow by selecting one or more processing items to be executed from among a plurality of processing items and setting an execution order of the one or more processing items; an image processing unit that executes the image processing flow, The plurality of process items includes at least one AI process item, Each of the at least one AI processing item comprises: Acquiring an externally created AI model; and The image processing device is defined as outputting at least one of an inference result obtained by inputting a target image into the AI ​​model and output information generated from the inference result.

2. Each of the at least one AI processing item comprises: Obtaining a first parameter indicating an input image format of the AI ​​model; converting the target image to conform to the input image format using the first parameters; The image processing device according to claim 1 , further comprising: inputting the transformed target image into the AI ​​model.

3. The at least one AI processing item includes a first AI processing item using a first AI model that outputs an output image as the inference result, in which each pixel has a floating-point value; The first AI processing item is: performing one or more image processes on the output image; outputting the output image on which the one or more image processes have been performed as the output information; The image processing apparatus according to claim 1 , wherein the one or more image processes include at least one of a first process for converting a value of each pixel to an integer and a second process for normalizing a value of each pixel.

4. the value of each pixel in the output image represents a probability of belonging to a particular class; The first AI processing item is: obtaining a second parameter representing a difference between a threshold value optimized for determining classification into the specific class and a predetermined value; The image processing device according to claim 3 , further comprising a step of defining execution of the second process using the second parameter.

5. The at least one AI processing item includes a second AI processing item using a second AI model that performs an autoencoder on the target image to output a restored image as the inference result, The second AI processing item is: generating a difference image between the target image and the restored image; The image processing apparatus according to claim 1 , further comprising: outputting, as the output information, the difference image or a processed image obtained by performing one or more image processes on the difference image.

6. The image processing apparatus according to claim 5 , wherein the one or more image processes include at least one of a first process for converting a value of each pixel to an integer, and a second process for normalizing a value of each pixel.

7. The second AI processing item is: acquiring a second parameter representing a difference between a threshold value optimized for detecting an abnormal portion from the difference image and a predetermined value; The image processing device according to claim 6 , further comprising: a step of defining execution of the second process using the second parameter.

8. The flow creation unit In response to a target AI processing item being selected as the one or more processing items among the at least one AI processing item, a designation of an AI model to be used in the target AI processing item is accepted; The image processing device according to claim 1 , further comprising: a processor configured to determine whether a specified AI model is compatible with the target AI processing item.

9. The at least one AI processing item includes a third AI processing item using a third AI model that outputs an output image as the inference result; The third AI processing item is: generating a first processed image obtained by performing a first image processing on the output image and a second processed image obtained by performing a second image processing on the output image; displaying the second processed image on a display; The image processing apparatus according to claim 1 , further comprising: a step of outputting the first processed image to a subsequent processing item of the one or more processing items.

10. The image processing device according to claim 1 , further comprising an update unit that updates the AI ​​model to a re-learned AI model, and the re-learned AI model is obtained by re-learning the AI ​​model using the target image.

11. 1. An image processing method, comprising: A processor selects one or more process items to be executed from a plurality of process items, and sets an execution order of the one or more process items to create an image processing flow; The processor executes the image processing flow, The plurality of process items includes at least one AI process item, Each of the at least one AI processing item comprises: Acquiring an externally created AI model; and An image processing method that defines outputting at least one of an inference result obtained by inputting a target image into the AI ​​model and output information generated from the inference result.

12. A program for causing a computer to execute an image processing method, The image processing method includes: creating an image processing flow by selecting one or more processing items to be executed from among a plurality of processing items and setting an execution order of the one or more processing items; Executing the image processing flow, The plurality of process items includes at least one AI process item, Each of the at least one AI processing item comprises: Acquiring an externally created AI model; and A program that defines outputting at least one of an inference result obtained by inputting a target image into the AI ​​model and output information generated from the inference result.

Images