Image correction device, image correction method, and program

Abstract

To enable an algorithm for correcting an image to be easily changed according to a purpose of correcting the image.SOLUTION: An image correction device 120 inputs an image to be corrected into a trained learning model, for calculating a degree of confidence regarding pass / fail of the image, and corrects the image to be corrected that is inputted into the trained learning model on the basis of the calculated degree of confidence. Then, the image correction device 120 determines the image as a final corrected image if a prescribed calculation termination condition based on the corrected image is satisfied, and if the condition is unsatisfied, inputs the corrected image into the trained learning model, for recalculating the degree of confidence regarding the pass / fail of the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image correction device, an image correction method, and a program. [Background technology]

[0002] As images are digitized, digital images are often corrected. Patent Document 1 discloses that when a flash is emitted during image capture by a camera, a point spread function is calculated, and then the point spread function is corrected based on flash information related to the emission of light from a flash emitting unit, and image restoration is performed based on the corrected point spread function. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-205802 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology described in Patent Document 1 is limited to correcting an image to a specific state. Specifically, Patent Document 1 uses an algorithm for correcting an image that is specialized for correcting camera shake regardless of whether a flash is emitted. Therefore, it is not easy to change the algorithm to perform corrections other than correcting camera shake in an image.

[0005] The present invention has been made in view of the above problems, and has as its object to make it possible to easily change an algorithm for correcting an image depending on the purpose of the image correction. [Means for solving the problem]

[0006] The image correction device of the present invention is an image correction device that corrects an image using a trained learning model that takes an image as input and outputs a confidence level regarding the quality of the image, and a plan representation image in which each of a plurality of items included in the product data corresponds to one of rows and columns, the processing order of a plurality of products included in the product data corresponds to the other of rows and columns, and the value of each item corresponds to a pixel value. an image acquisition means for acquiring the image acquired by the image acquisition means; Plan Expression Input the image into the learning model Plan Expression image The plan represented by a certainty calculation means for calculating a certainty of the pass / fail of the Plan Expression and an image correction means for correcting the image, and the confidence calculation means calculates the confidence level of the image corrected by the image correction means. Plan Expression If the image does not satisfy a predetermined calculation termination condition for the calculation to correct the plan-representative image, the image corrected by the image correcting means is Plan Expression The image is input to the learning model and the confidence is calculated again, and the image correction means Plan Expression If the image satisfies the calculation termination condition, Plan Expression The image above after final corrections Plan Expression Determine as an image.

[0007] The image correction method of the present invention is an image correction method that corrects an image using a trained learning model that takes an image as input and outputs a confidence level regarding the quality of the image, and a plan representation image in which each of a plurality of items included in the product data corresponds to one of rows and columns, the processing order of a plurality of products included in the product data corresponds to the other of rows and columns, and the value of each item corresponds to a pixel value. an image acquisition step of acquiring the image obtained by the image acquisition step; Plan Expression Input the image into the learning model Plan Expression image The plan represented by a confidence level calculation step of calculating a confidence level for the pass / fail of the test data; and Plan Expression an image correction step of correcting the image, and the confidence calculation step is Plan Expression The image is Plan Expression If a predetermined calculation termination condition for the calculation for correcting the image is not met, the image corrected by the image correction step is Plan ExpressionThe image is input to the learning model and the confidence is calculated again, and the image correction step , Osamu Corrected above Plan Expression If the image satisfies the calculation termination condition, Plan Expression The image above after final corrections Plan Expression Determine as an image.

[0008] The program of the present invention causes a computer to function as each of the means of the image correction device. [Effects of the Invention]

[0009] According to the present invention, the algorithm for correcting an image can be easily changed depending on the purpose of the image correction. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the functional configuration of a learning model creation device and an image correction device. [Figure 2] FIG. 10 is a diagram showing an example of a plan representation original image for creating a learning model and pass / fail judgment data. [Figure 3] FIG. 1 is a diagram conceptually illustrating an example of a CNN. [Figure 4] FIG. 1 is a diagram illustrating an example of an outline of a convolution operation. [Figure 5] FIG. 10 is a diagram illustrating an example of a convolution operation in detail. [Figure 6] FIG. 10 is a diagram illustrating an example of processing in a convolution layer and a pooling layer. [Figure 7] FIG. 10 is a diagram illustrating an example of an outline of a method for correcting an image. [Figure 8] 10 is a flowchart illustrating an example of a learning model creation method. [Figure 9] 10 is a flowchart illustrating an example of an image correction method. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. 1 is a diagram showing an example of the functional configuration of the learning model creation device 110 and the image correction device 120. The hardware of the learning model creation device 110 and the image correction device 120 is realized, for example, by using an information processing device including a processor, a main memory device, an auxiliary memory device, and various interfaces. Note that this embodiment illustrates a case where the learning model creation device 110 and the image correction device 120 are separate devices. However, the learning model creation device 110 and the image correction device 120 may also be realized by a single device.

[0012] Although details will be described later, this embodiment illustrates an example in which product data is acquired, a plan-representation image is created, and a production plan or a logistics plan is created by correcting the plan-representation image using a learning model that uses the plan-representation image as an input and outputs a confidence level regarding the success or failure of the production plan or logistics plan represented by the plan-representation image. Here, the product data includes values ​​of multiple items, including specifications, for each product included in a certain production plan or logistics plan, and the processing order of the product in the production plan or logistics plan. The plan-representation image is an image in which each of the multiple items included in the product data corresponds to one of rows and columns, the processing order of the multiple products included in the product data corresponds to the other of rows and columns, and the value of each item corresponds to a pixel value.

[0013] In such a plan-representing image, pixel values ​​are assigned to correspond to a production plan or a logistics plan. Therefore, the way features appear in the plan-representing image differs from images captured by an imaging device. Therefore, it is preferable to create a learning model taking into account the way features appear in the plan-representing image. Therefore, below, we will first explain an example of a learning model creation device 110 that creates such a learning model, and then explain an example of an image correction device 120.

[0014] However, the image to be corrected in the image correction device 120 is not limited to a plan-representation image, and may be other digital images, such as an image captured by an imaging device. Details will be described later, but in this embodiment, a case where the learning model is a convolutional neural network (CNN) is exemplified (however, the CNN algorithm itself is modified from a known CNN algorithm so that the calculations described below can be performed, and is different from the known CNN algorithm). Note that if the image to be corrected in the image correction device 120 is not a plan-representation image, but is, for example, an image captured by an imaging device, a known CNN algorithm may be used as the learning model. In the following description, an image captured by an imaging device will be referred to as a captured image as necessary.

[0015] [Learning model creation device 110] The learning model creation device 110 creates a learning model used to modify an image. In this embodiment, the learning model creation device 110 creates a learning model used to create a production plan or a logistics plan represented by the plan representation image by modifying the plan representation image. More specifically, the learning model creation device 110 creates a learning model used to create a production plan for producing a product. An example of a production plan is a hot rolling schedule (a schedule including the rolling order of slabs (steel) to be hot rolled). However, the production plan is not limited to a hot rolling schedule as long as it includes the production order of the product. The learning model creation device 110 may also create a learning model used to create a logistics plan instead of a production plan. When creating a learning model used to create a logistics plan, for example, the production order may be replaced with the transport order in the following description. The learning model may be a learning model (e.g., a machine learning model) created by various types of learning. However, as described above, the present embodiment illustrates a case in which the learning model is a convolutional neural network (CNN). The learning method used to create a learning model is preferably supervised learning, which allows for easy construction of a learning model with highly reliable calculated values ​​of the objective variable by collecting training data, but is not limited to supervised learning and may also be unsupervised learning, reinforcement learning, or a learning method other than machine learning.

[0016] The learning model creation device 110 includes a learning image acquisition unit 111, a learning model creation unit 112, and a storage unit 113.

[0017] The learning image acquisition unit 111 acquires a learning target image used to create a learning model and pass / fail judgment data for the image. In this embodiment, the learning image acquisition unit 111 includes a teacher data acquisition unit 111a and a learning model creation image creation unit 111b.

[0018] <<Teacher data acquisition unit 111a>> The teacher data acquisition unit 111a acquires teacher data including product data for creating a learning model and pass / fail judgment data corresponding to the product data for creating a learning model. The product data for creating a learning model is product data for each of the multiple products included in the plan, and has values ​​for multiple items including product specifications and the processing order of the products in the plan.

[0019] In this embodiment, the multiple items in the product data for creating a learning model are exemplified as product size, material, quality, and delivery date. The product size, material, quality, and delivery date are product specifications. Note that the multiple items may include items other than product specifications. For example, the multiple items may include product production conditions (e.g., the processes performed when producing the product, the equipment used when producing the product, and the materials used when producing the product). Furthermore, since this embodiment exemplifies a production plan, the processing order is the production order.

[0020] Thus, in this embodiment, the product data includes values ​​of multiple items, including specifications, for each product included in a certain production plan, and the production order of the product in the production plan. Note that the image correction device 120 (plan creation product data acquisition unit 121a), which will be described later, also acquires product data. In the following description, the product data acquired by the training data acquisition unit 111a will be referred to as product data for learning model creation as needed to distinguish it from the product data acquired by the image correction device 120. In the following description, the multiple items, including specifications, will also be referred to as manufacturing specifications as needed.

[0021] The pass / fail judgment data is data that indicates whether the image is good or bad (in this embodiment, the pass / fail of the plan represented by the plan representation image). In this embodiment, a case is exemplified in which the pass / fail judgment data is set to "1" for a good plan and "0" for a bad plan. A good plan is a plan that does not need to be corrected, for example, a plan that was used in actual production. A good plan may also be, for example, a plan that was used when a product was produced as planned. A bad plan is a plan that needs to be corrected (a plan that is the opposite of a good plan), for example, a plan that was drawn up but not used in actual production, or a plan that was used when a product was not produced as planned.

[0022] In this embodiment, the pass / fail judgment data includes a confidence level indicating the likelihood that the image is good (i.e., the plan represented by the plan representation image is a good plan) and a confidence level indicating the likelihood that the image is bad (i.e., the plan represented by the plan representation image is a bad plan). This embodiment also illustrates a case in which the confidence level takes a value between 0 and 1. A confidence level of "1" indicates the highest confidence level, and a confidence level of "0" indicates the lowest confidence level. The teacher data specifies whether the product data for creating a learning model is based on a good plan or a bad plan. Therefore, when the product data for creating a learning model is based on a good plan, the pass / fail judgment data corresponding to the product data for creating a learning model includes data indicating that the confidence level of the good plan is "1" and the confidence level of the bad plan is "0." On the other hand, if the product data for creating a learning model is based on a poor plan, the pass / fail judgment data corresponding to the product data for creating a learning model will be data containing a confidence level of "0" that the plan is good and a confidence level of "1" that the plan is poor. In other words, the pass / fail judgment data corresponding to the product data for creating a learning model contains data with a value of "0" and data with a value of "1."

[0023] The teacher data acquisition unit 111a acquires multiple teacher data as the teacher data. The teacher data acquired by the teacher data acquisition unit 111a preferably includes multiple teacher data including product data for creating a learning model based on a good plan and multiple teacher data including product data for creating a learning model based on a poor plan. Furthermore, from the viewpoint of performing highly accurate learning of the learning model (CNN in this embodiment), it is preferable that the number of teacher data including product data for creating a learning model based on a good plan and the number of teacher data including product data for creating a learning model based on a poor plan acquired by the teacher data acquisition unit 111a are large. However, if the number of teacher data acquired by the teacher data acquisition unit 111a is too large, the calculation load will increase. The number of teacher data acquired by the teacher data acquisition unit 111a is determined appropriately from this viewpoint.

[0024] The method of acquiring the teacher data may be, for example, at least one of the following: receiving teacher data transmitted from an external device; reading out teacher data stored on a portable storage medium; or inputting the teacher data by an operator operating the user interface of the learning model creation device 110.

[0025] <<Learning model creation image creation unit 111b>> The learning model creation image creation unit 111b creates a plan expression image for creating a learning model based on the learning model creation product data included in the training data acquired by the training data acquisition unit 111a.

[0026] The plan-representing image is an image in which each of the multiple items included in the product data corresponds to one of rows and columns, the processing order of the multiple products included in the product data corresponds to the other of rows and columns, and the value of each item corresponds to a pixel value. Note that the image correction device 120 (plan-creation image creation unit 121b), which will be described later, also creates plan-representing images. In the following description, the plan-representing image created by the learning-model-creation image creation unit 111b will be referred to as a plan-representing image for learning-model creation as needed to distinguish it from the plan-representing image created by the image correction device 120.

[0027] In this embodiment, an example is shown in which each of the multiple manufacturing specifications (items) included in the product data for creating a learning model corresponds to a column, and the production order of the multiple products included in the product data for creating a learning model corresponds to a row.

[0028] FIG. 2 is a diagram showing an example of a plan-representation original image 210 for creating a learning model and pass / fail judgment data 220. The plan-representation original image is obtained by treating product data as an image and converting the values ​​of each manufacturing specification (item) included in the product data into pixel values. The plan-representation original image refers to an image that is the source of the plan-representation image. The image correction device 120 (plan-representation image creation unit 121b), which will be described later, also creates plan-representation original images. To distinguish the plan-representation original image created by the image correction device 120, the plan-representation original image created by the learning-model creation image creation unit 111b will be referred to as the plan-representation original image for creating a learning model as needed.

[0029] In this embodiment, multiple categories are set for each manufacturing specification (size, material, quality, delivery date) included in the product data for creating a learning model. Then, the value of each manufacturing specification included in the product data for creating a learning model is classified into one of the multiple categories set for the manufacturing specification. In the example shown in FIG. 2, six categories 1 to 6 are set for size. Here, a larger number indicating the size category represents a larger size. Also, six categories A to F are set for material. Here, an example is shown in which the closer the category letters are in alphabetical order, the more similar the materials are. Also, an example is shown in which five categories 1 to 5 are set for quality. Here, an example is shown in which the larger the number indicating the quality category, the higher the quality of the product. Also, an example is shown in which five categories 1 to 5 are set for delivery date. Here, an example is shown in which the larger the number indicating the delivery date category, the further ahead the delivery date (the more time is available for delivery).

[0030] FIG. 2 illustrates an example in which the production plan includes 12 products. The learning model creation image creation unit 111b identifies which of the above-mentioned categories the values ​​of the manufacturing specifications (size, material, quality, delivery date) of each of the 12 products belong to. As described above, in this embodiment, each of the multiple manufacturing specifications included in the learning model creation product data corresponds to a column, and the production order of the multiple products included in the learning model creation product data corresponds to a row. Therefore, in FIG. 2, the manufacturing specifications (size, material, quality, delivery date) of the 12 products included in the production plan correspond to the columns of the plan representation source image 210 for learning model creation. Furthermore, the production order of the 12 products included in the production plan corresponds to the rows.

[0031] In the example shown in FIG. 2, the production order of the products is assigned to each pixel in ascending order from the top of the row of the original image 210 representing a plan for creating a learning model. Therefore, for example, the product that will be produced earliest among 12 products included in the production plan is assigned to the pixel of the first row of the original image 210 representing a plan for creating a learning model. In this embodiment, a variable that identifies the row of the original image 210 representing a plan for creating a learning model is v, and the original image 210 representing a plan for creating a learning model is assigned to the pixel of the first row of the original image 210 representing a plan for creating a learning model from the v1 row (=first row) to the v max It is assumed that up to line (=line 12) exists.

[0032] In the example shown in FIG. 2, six size categories, six material categories, five quality categories, and five delivery categories are assigned to each pixel in the column of the original image 210 representing the plan used to create a learning model, starting from the left. Therefore, for example, the pixel value of a pixel corresponding to a category to which the value of each product's manufacturing specification (size, material, quality, delivery date) belongs is set to a different value from the pixel value of a pixel corresponding to other categories with the same manufacturing specification as the relevant manufacturing specification. For example, a bit pattern is created in which the pixel value of a pixel corresponding to a category to which the value of each product's manufacturing specification (size, material, quality, delivery date) belongs is set to "1," and the pixel value of other categories with the same manufacturing specification as the relevant manufacturing specification is set to "0." In FIG. 2, pixels with a pixel value of "1" are shown in black, and pixels with a pixel value of "0" are shown in white (this is the same in FIG. 3 and subsequent figures. This corresponds to one-hot encoding). For example, in Figure 2, the categories of product, material, quality, and delivery date that can be produced earliest among the 12 products included in the production plan are 5, C, 4, and 5, respectively, as shown in the first row.

[0033] In this embodiment, the variable that identifies the column of the original image 210 expressing a plan for creating a learning model is set to h, and the variable that identifies the manufacturing specification is set to m. However, the variable m is not assigned to the delivery date. The values ​​of m for size, material, and quality are set to 1, 2, and 3, respectively. Then, as shown in FIG. 2, in the original image 210 expressing a plan for creating a learning model, the area corresponding to the manufacturing specification m is set to h1 m column to h max m Therefore, the area corresponding to the size (m=1) is h1 1 Column (=1st column) to h max 1 The area corresponding to the material (m=2) is h1 2 Column (=7th column) to h max 2 The area corresponding to quality (m=3) is h1 3 Column (=13th column) to h max 3For example, h1 1 corresponds to the symbol h1 with a 1 above it in Figure 2. This notation method is 1 The same applies to symbols other than those, and they will be written in the same way in each figure and each formula.

[0034] In this embodiment, the part of the plan-representing original image 210 for creating a learning model to which the variable m has been assigned in this manner is designated as the plan-representing image 230 for creating a learning model (the number m is designated as M). Note that this embodiment illustrates a case in which the variable m is not assigned to the delivery date, but the variable m may also be assigned to the delivery date. In this case, the plan-representing original image 210 for creating a learning model becomes the same as the plan-representing image 230 for creating a learning model.

[0035] As described above, in this embodiment, the pass / fail judgment data 220 includes a confidence level of a good plan and a confidence level of a bad plan, with "1" indicating the highest confidence level and "0" indicating the lowest confidence level. The teacher data specifies whether the product data for creating a learning model is based on a good plan or a bad plan. In FIG. 2, a confidence level of a good plan of "1" and a confidence level of a bad plan of "0" are represented as "Good: 1 Bad: 0." This indicates that the product data for creating a learning model, from which the plan expression image 230 shown in FIG. 2 was created, is based on a good plan.

[0036] In the following explanation, the confidence Y1 that the plan represented by the plan-representative image input to the input layer 310 is a good plan will be referred to as the confidence Y1 for a good plan, as necessary. Also, the confidence Y2 that the plan represented by the plan-representative image input to the input layer 310 is a bad plan will be referred to as the confidence Y2 for a bad plan, as necessary.

[0037] In this embodiment, the learning model creation image creation unit 111b creates the plan-representing original image 210 for learning model creation and the plan-representing image 230 for learning model creation in the above manner. Note that the plan-representing original image 210 for learning model creation and the plan-representing image 230 for learning model creation are not limited to those shown in Fig. 2. For example, when the pixel values ​​of the plan-representing original image 210 for learning model creation and the plan-representing image 230 for learning model creation are represented by 8 bits, the values ​​of the multiple manufacturing specifications m may be classified into one of 256 gradation values.

[0038] In addition, in this embodiment, a case where each of the multiple manufacturing specifications included in the product data for creating a learning model corresponds to a column (horizontal direction), and the production order of the multiple products included in the product data for creating a learning model corresponds to a row (vertical direction) is exemplified. However, each of the multiple manufacturing specifications included in the product data for creating a learning model may correspond to a row, and the production order of the multiple products included in the product data for creating a learning model may correspond to a column.

[0039] When the image to be corrected in the image correction device 120 is a captured image, the learning image acquisition unit 111 only needs to acquire multiple pieces of training data, each of which includes a captured image and pass / fail judgment data for the captured image. Therefore, the learning image acquisition unit 111 does not need to acquire product data for creating a learning model or generate an image different from the captured image. When the image to be corrected in the image correction device 120 is a captured image, the pass / fail judgment data is data indicating the quality of the captured image. When correcting a captured image for camera shake as described in Patent Document 1, the pass / fail judgment data for a captured image that is not affected by camera shake is data containing a confidence level of "1" that the image is good and a confidence level of "0" that the image is bad. On the other hand, the pass / fail judgment data for a captured image that is affected by camera shake is data containing a confidence level of "0" that the image is good and a confidence level of "1" that the image is bad. In this way, a good image is an image that does not need to be corrected, and a bad image is an image that needs to be corrected. In the following description, the confidence that an image is good will also be referred to as the confidence that an image is good, and the confidence that an image is bad will also be referred to as the confidence that an image is bad, as necessary. Note that the confidence that an image is good and the confidence that an image is bad are confidence levels specific to the case where the image is a plan-representing image, and are concepts that are included in the confidence that an image is good and the confidence that an image is bad, respectively.

[0040] <Learning model creation unit 112, storage unit 113> The learning model creation unit 112 performs machine learning using the images acquired by the learning image acquisition unit 111, thereby creating a learning model that takes the image as input and outputs a confidence level as to whether the image is good or bad, and stores the model in the storage unit 113. In this embodiment, the learning model creation unit 112 performs supervised learning using the plan-representing image for learning model creation 230 created by the learning model creation image creation unit 111b, thereby creating a learning model that takes the plan-representing image as input and outputs a confidence level as to whether the plan is good or bad, and stores the model in the storage unit 113. As described above, this embodiment also illustrates the case where the learning model is a CNN.

[0041] FIG. 3 is a conceptual diagram illustrating an example of a CNN. In the example illustrated in FIG. 2, the CNN includes an input layer 310, a convolutional layer 320, a pooling layer 330, and an output layer 340. This embodiment illustrates a case where the CNN has the configuration illustrated in FIG. 3. However, the configuration of the CNN is not limited to the configuration illustrated in FIG. 3, and other known CNN configurations may be adopted. For example, for simplicity of explanation and notation, FIG. 3 illustrates a case where there is one convolutional layer 320 and one pooling layer 330. However, the CNN may include multiple pairs of convolutional layers 320 and pooling layers 330. Furthermore, this embodiment illustrates a case where the neurons in the output layer 340 and each component of the pooling layer 330 are fully connected, and illustrates a case where the CNN does not include a fully connected layer. However, the CNN may include a fully connected layer.

[0042] In CNN, an image is input to an input layer 310. In this embodiment, a case where a plan-representing image 230 for creating a learning model is input to the input layer 310 will be exemplified. In the convolution layer 320, a convolution operation using a kernel (filter) is performed on the image. FIG. 4 is a diagram illustrating an example of an outline of the convolution operation. The example shown in FIG. 4 illustrates a case where the kernel used in the convolution operation is individually set for one item (manufacturing specification m) among multiple items (manufacturing specification m) assigned to a column of the learning model creation plan expression image 230. Specifically, FIG. 4 illustrates a case where kernel 411 is set for size, kernel 412 is set for material, and kernel 413 is set for quality.

[0043] Here, the number of multiple items (manufacturing specifications m) included in the plan representation image 230 for creating a learning model is assumed to be M. In the example shown in FIG. 2, M=3. The same kernel may be set for 2 or more but less than M items (manufacturing specifications m) in the plan representation image 230 for creating a learning model. In the example shown in FIG. 2, for example, the same kernel may be set for size and material, the same kernel may be set for size and quality, the same kernel may be set for material and quality, or the same kernel may be set for size, material, and quality.

[0044] When the same kernel is set for items (manufacturing specifications m) that are equal to or greater than 2 and less than M in the planning expression image 230 for creating a learning model, one kernel may be applied to each item (manufacturing specifications m), or multiple kernels may be applied to one item (manufacturing specification m). In the example shown in Fig. 2, for example, one kernel may be set for size and material and another kernel for quality, or one kernel may be set for size and material and another kernel for quality and material. As described above, when the same kernel is set for items (manufacturing specifications m) that are 2 or more and less than M in the planning expression image 230 for creating a learning model, in the following explanation, the items that are 2 or more and less than M (for example, size and material in the example shown in Figure 2) can be considered as one item (manufacturing specifications m).

[0045] FIG. 4 also illustrates a case in which the size in the longitudinal direction (i.e., row direction (horizontal direction)) of the rows corresponding to the processing orders (production orders) of multiple products in the plan representation image for creating a learning model 230 of the kernels 411 to 413 set for one item (manufacturing specification m) is the same as the size in the longitudinal direction of the area corresponding to that one item (manufacturing specification m) in the plan representation image for creating a learning model 230. That is, as illustrated with reference to FIG. 2, in the plan representation image for creating a learning model 230, the production orders of multiple products correspond to rows, and multiple items (manufacturing specifications m) correspond to columns. Therefore, the row direction (horizontal direction) size of the kernel 411 set for size is the same as the row direction size of the area corresponding to size in the plan representation image for creating a learning model 230. Similarly, the row direction (horizontal direction) sizes of the kernels 412 and 413 set for material and quality are the same as the row direction sizes of the areas corresponding to material and quality in the plan representation image for creating a learning model 230, respectively.

[0046] Figure 4 also illustrates an example in which the longitudinal size (i.e., column direction (vertical direction)) of the columns of kernels 411 to 413 corresponding to multiple items (manufacturing specifications m) in the planning expression image 230 for creating a learning model is set to a size according to the item (manufacturing specifications m) to which the kernels 411 to 413 are set. For example, it may be preferable that the product to be produced after a product made of a certain material is made of a specific material (for example, in FIG. 4, the product to be produced after a product made of material F is made of material C). In such cases, a good production plan is one that includes many relationships in which the product to be produced after a product made of a certain material is made of a specific material. Therefore, to make it easier to extract such features, the size in the column direction (vertical direction) of the kernel 412 set for the material is made small.

[0047] Furthermore, for example, if high-quality products and low-quality products are distributed in production order, the low-quality products may need to be produced using the same processing as the high-quality products, which may increase the production cost of the low-quality products. In such cases, a production plan that produces products of the same quality in succession is a good production plan. Therefore, to make it easier to extract such features, the size of the column direction (vertical direction) of the kernel 413 set for the material is increased.

[0048] Also, for example, if the size of adjacent products in the production sequence suddenly changes, it may become difficult to produce the products. In other words, in such cases, a production plan in which the size changes stepwise (gradually) is a good production plan. Therefore, in order to extract such features, the size in the column direction (vertical direction) of the kernel 411 set for the size is set to a medium size.

[0049] 4 illustrates an example in which kernels 411 to 413 set for each item (each manufacturing specification m) are slid only within an area corresponding to the item (manufacturing specification m) in the learning model creation plan representation image 230. Specifically, in FIG. 4, the kernel 411 is slid only within an area corresponding to the size in the learning model creation plan representation image 230. Similarly, the kernels 412 and 413 are slid only within areas corresponding to the material and quality, respectively, in the learning model creation plan representation image 230. More specifically, FIG. 4 illustrates an example in which the kernels 411 to 413 are slid only in the direction of the columns (column direction (vertical direction)) corresponding to multiple items (manufacturing specifications m) (see the outline arrows below the kernels 411 to 413 in FIG. 4).

[0050] However, it is not necessary to set the kernels 411 to 413 as shown in Fig. 4. For example, the row size of a kernel set for one item (manufacturing specification m) may be smaller than the row size of the area corresponding to that one item (manufacturing specification m) in the learning model creation plan representation image 230. In this case, the kernel set for that one item (manufacturing specification m) is slid, for example, in the row direction (horizontal direction) and column direction (vertical direction) within the area corresponding to that item (manufacturing specification m) in the learning model creation plan representation image 230.

[0051] FIG. 5 is a diagram illustrating an example of a convolution operation in detail. In this embodiment, in the planning representation image 230 for creating a learning model, the pixel value of the vth row and hth column of the product specification m is expressed as x v,h m Here, the pixel value x v,h m When expressing this, it is assumed that a numerical value is set for h for each area to which the kernels 411 to 413 are applied in the plan representation image for creating a learning model 230. In this embodiment, the kernel 411 is set in an area corresponding to the size in the plan representation image for creating a learning model 230. Therefore, for example, when the pixel value x v,h m When expressing this, the area h1 corresponding to the size (m=1) is shown in Figure 2. 1 The value of h, which indicates the column, is "1", and max 1 The value of h, which indicates the column, is "6". Also, the pixel value x v,h m When expressing the material (m=2), h1 2 The value of h, which indicates the column, is "1", and max 2 The value of h, which indicates the column, is "6". Also, the pixel value x v,h m When expressing the quality (m=3), h1 3 The value of h, which indicates the column, is "1", and max 3 The value of h, which indicates the column, is "5". In this way, the x v,hm is defined as the pixel value of the planning representation image 230 for creating a learning model. In addition, the weights of the kernels 411 to 413 are defined as w l,s m_fc Here, m_f c is a symbol that identifies the convolutional layer 320 for the manufacturing specification m. In the example shown in FIGS. 3 to 5, one convolutional layer 320 is created for each manufacturing specification m, so f c = 1. When multiple convolution layers 320 are created for one manufacturing specification m, f c can take a value of 2 or more (f c The number of F c l is the position of an element in the feature map that constitutes the convolution layer 320, and is a number (row number) that identifies the position of the element in the direction corresponding to the column direction (vertical direction) of the image input to the input layer 310. s is the position of an element in the feature map that constitutes the convolution layer 320, and is a number (column number) that identifies the position of the element in the direction corresponding to the row direction (horizontal direction) of the image input to the input layer 310. Furthermore, the number of elements in the column direction (vertical direction) of the kernels 411 to 413 is k r m and the number of elements in the row direction (horizontal direction) of kernels 411 to 413 is k c m It is written as follows.

[0052] Then, in the convolution operation in the convolution layer 320, the following equation (1) is calculated, and u l,s m_fc is calculated.

[0053]

number

[0054] FIG. 6 is a diagram illustrating an example of processing in the convolution layer 320 and the pooling layer 330. For convenience of notation, FIG. 6 illustrates processing of only an area of ​​size (m=1) in the plan representation image for creating a learning model 230. FIG. 6 also illustrates processing in the case where kernels 411 to 413 are set as shown in FIGS. 4 and 5 for the plan representation image for creating a learning model 230 shown in FIG. 2. When the kernels 411 to 413 are set as shown in FIGS. 4 and 5, k c m =h max m Therefore, h max m -k c m +1=1. In this case, as shown in FIG. 6, the image of the region corresponding to one manufacturing specification m in the planning representation image for creating a learning model 230 becomes a single column of images through the convolution operation in the convolution layer 320. In FIG. 3, as a result of applying the kernel 411 to the region of size (m=1) in the planning representation image for creating a learning model 230, the calculated value u of the convolution operation 1,1 1_1 is stored in element 321 of the feature map, as shown in gray. Similarly, as a result of applying kernel 412 to the material (m=2) region in the learning model creation planning representation image 230, the calculated value u 1,1 2_1 is stored in element 322 of the feature map in gray, and the kernel 413 is applied to the quality (m=3) region in the planning representation image 230 for creating a learning model, resulting in the calculated value u of the convolution operation. 1,1 3_1 is stored in element 323 of the feature map.

[0055] Also, in equation (1), d l+q-1 is shown in Figure 2. v In the example shown in Figure 2, v takes values ​​from 1 to 12, so in Figure 5, d v d1~d 12 It is written as d v (d1~d 12) is the value of the delivery date category shown in FIG. 2. As mentioned above, the larger the value of the delivery date category, the further ahead the delivery date (in other words, the smaller the value of the delivery date category, the closer the delivery date). In FIG. 2, for example, the value of the delivery date category for a product with a production order of "6" is "1", while the value of the delivery date category for products with production orders of "1" to "5" is "4" or "5". That is, in the example shown in FIG. 2, the plan representation image 230 for creating a learning model is created based on a production plan that includes a product with an earlier delivery date to be produced before a product with an approaching delivery date. Therefore, in this embodiment, a product to be produced before a product with an approaching delivery date is considered to be a product with high importance from the perspective of the manufacturing specification m, and a weight is applied to the pixel value corresponding to that product in the plan representation image 230 for creating a learning model. In this way, in this embodiment, as shown in equation (1), the weights w set for the kernels 411 to 413 are p,q m For d l+q-1 By multiplying the pixel value x of the planning representation image 230 for creating a learning model, v,h m (x in equation (1) l+q-1,s+r-1 m ) is weighted according to the delivery date of the product corresponding to the pixel value.

[0056] In the convolution operation, for example, a bias (threshold) is added to the right side of equation (1), but the bias (threshold) itself is realized by a known technique for CNN. Therefore, notation and detailed explanation of the bias (threshold) will be omitted here.

[0057] Then, in the convolution layer 320, the calculated value u of the convolution operation obtained as described above is l,s m_fc As shown in the following equation (2), the activation function σ c (u) to the u in the convolution layer 320. l,s m_fc is calculated.

[0058]

number

[0059] In the example shown in Figure 6, the calculated value u 1,1 1_1 ,···,u 8,1 1_1 , respectively, the calculated value c of the convolutional layer 320 1,1 1_1 ,···,c 8,1 1_1 In FIG. 3, elements 321, 322, and 323 of the feature map are respectively represented by u 1,1 1_1 , u 1,1 2_1 , u 1,1 3_1 is the activation function σ c (u) is calculated by 1,1 1_1 , c 1,1 2_1 , c 1,1 3_1 Similarly, elements 324, 325, and 326 of the feature map shown in FIG. 3 store u 2,1 1_1 , u 2,1 2_1 , u 2,1 3_1 is the activation function σ c (u) is calculated by 2,1 1_1 , c 2,1 2_1 , c 2,1 3_1 is stored. Note that the activation function σ c As (u), for example, a known function used in CNN, such as a sigmoid function, may be used.

[0060] Processing in the convolutional layer 320 is performed in the above manner. When the image to be corrected in the image correction device 120 is a captured image, the processing in the convolution layer 320 is realized by the same processing as that in a known CNN. In this case, it is not necessary to set kernels individually for multiple regions of the captured image. In other words, only one type of kernel is required to be applied to the captured image. Also, d l+q-1 However, it is also possible to set kernels individually for multiple regions of the captured image. For example, if the imaging range of the imaging device is fixed and it is known in advance that a portion of the captured image has characteristics different from those of other regions, kernels of different sizes may be used for the kernel for that portion and the kernel for the other regions.

[0061] The processing in the pooling layer 330 and the output layer 340 is realized by the same processing as that in a known CNN. In FIG. 3, the calculated value c l,s 1_fc The two elements 321 and 324 of the feature map storing the σ are compressed (downsampled) and stored in the output table (compressed feature map elements) 331 of the pooling layer 330 (see also the feature map elements 321, 324, and 331 in FIG. 6). Similarly, the calculated value c l,s 2_fc The two feature map elements 322, 325 storing the data are compressed and stored in the output table 332 of the pooling layer 330, as shown in gray. The calculated value c l,s 3_fc The two elements 323 and 326 of the feature map storing the above data are compressed and stored in the output table 333 of the pooling layer 330, as shown in gray. Compression of multiple elements of the feature map is achieved, for example, by calculating the maximum or average value of the values ​​stored in the multiple elements.

[0062] In this embodiment, the calculated value of the pooling layer 330 is pi,j m_fp Here, i is the position of the output table of the pooling layer 330, and is a number (row number) that identifies the position in the direction corresponding to the column direction (vertical direction) of the image input to the input layer 310, and j is the position of the output table of the pooling layer 330, and is a number (column number) that identifies the position in the direction corresponding to the row direction (horizontal direction) of the image input to the input layer 310. p is a symbol that identifies the pooling layer 330 for the manufacturing specification m. In this embodiment, one pooling layer 330 is created for each manufacturing specification m, so f p = 1. When multiple pooling layers 330 are created for one manufacturing specification m, f p can take a value of 2 or more (f p The number of F p In FIG. 6, the calculated value c of the convolution layer 320 for the region of the size (m=1) of the planning representation image 230 for creating a learning model is l,s 1_fc The calculated value c for each of the two elements of the feature map that stores 1,1 1_1 ~c 2,1 1_1 , c 3,1 1_1 ~c 4,1 1_1 , c 5,1 1_1 ~c 6,1 1_1 , c 7,1 1_1 ~c 8,1 1_1 Compress it to size (m=1), f p The calculated values ​​p 1,1 1_1 , p 2,1 1_1 , p 3,1 1_1 , p 4,1 1_1 are calculated as the components of the pooling layer 330.

[0063] The learning model creation unit 112 creates a CNN in which the above-described processing in the convolution layer 320 is performed by performing supervised learning using the plan representation image for learning model creation 230 created by the learning model creation image creation unit 111b and the pass / fail judgment data 220, and stores the CNN in the storage unit 113. Specifically, in the configuration example of the CNN shown in FIG. 3, the learning model creation unit 112 calculates, as CNN learning parameters, parameters including the weights w and biases (thresholds) in the convolution layer 320 and the weights w and biases (thresholds) used when fully connecting the pooling layer 330 and the output layer 340. The CNN learning method is realized by a well-known method, such as a method using backpropagation, and therefore a detailed description of the CNN learning method will be omitted.

[0064] [Image correction device 120] The image correction device 120 corrects images using a trained learning model created by the learning model creation device 110. As described above, this embodiment illustrates a case in which the image correction device 120 creates a production plan or a logistics plan by correcting a plan representation image using a learning model that takes a plan representation image as input and outputs a confidence level regarding the quality of the plan representation image (a confidence level regarding the quality of the production plan or logistics plan represented by the plan representation image). More specifically, this embodiment illustrates a case in which the learning model creation device 110 creates a learning model used to create a production plan for producing products such as steel plates. Therefore, this embodiment illustrates a case in which the image correction device 120 also functions as a plan creation device that creates a production plan.

[0065] The image correction device 120 includes a correction image acquisition unit 121, a planning image creation unit 121b, a confidence factor calculation unit 122, an image correction unit 123, and a planning unit .

[0066] FIG. 7 is a diagram showing an example of an outline of an image correction method in image correction device 120. As shown in FIG. In FIG. 7, a plan-representing image different from the plan-representing image 230 for creating a learning model included in the training data is input to the CNN created by the learning model creation device 110. Then, a confidence Y1 for a good plan and a confidence Y2 for a bad plan are output from the output layer 340. The image correction device 120 calculates confidence errors, which are errors of the confidence Y1 for a good plan and the confidence Y2 for a bad plan from their respective target values. As described above, in this embodiment, a confidence of "1" indicates the highest confidence, and a confidence of "0" indicates the lowest confidence. Therefore, in this embodiment, a case is illustrated in which the target value of the confidence Y1 for a good plan is "1" and the target value of the confidence Y2 for a bad plan is "0."

[0067] The image correction device 120 corrects the plan-representing image input to the CNN based on a good-plan confidence error, which is the error of the confidence Y1 for a good plan from its target value, and a bad-plan confidence error, which is the error of the confidence Y2 for a bad plan from its target value. The image correction device 120 repeatedly performs this correction of the plan-representing image until a calculation termination condition is met, and determines the plan-representing image when the calculation termination condition is met as the final corrected plan-representing image. The calculation termination condition may be, for example, any of the following: the confidence error is equal to or less than a threshold; the difference between the calculated representation image and the previous value is equal to or less than a predetermined value; or the number of repeated calculations is equal to a predetermined value.

[0068] When the image to be corrected in the image correction device 120 is a captured image, the explanation of Figure 7 above is changed to one in which "plan-representing image" is replaced with "captured image," "certainty Y1 of a good plan" is replaced with "certainty Y1 of a good image," and "certainty Y2 of a bad plan" is replaced with "certainty Y2 of a bad image."

[0069] <Correction image acquisition unit 121> The correction image acquisition unit 121 acquires an image to be corrected. For the image acquired by the learning image acquisition unit 111, pass / fail judgment data for the image is required, but pass / fail judgment data for the image acquired by the correction image acquisition unit 121 is not required. In this embodiment, an example is shown in which the correction image acquisition unit 121 has a planning use product data acquisition unit 121a and a planning use image creation unit 121b.

[0070] <<Planning Product Data Acquisition Unit 121a>> The planning product data acquisition unit 121a acquires planning product data, which is product data different from the learning model creation product data included in the training data acquired by the training data acquisition unit 111a. Like the learning model creation product data, the planning product data includes values ​​of multiple items, including specifications, for each product included in a certain production plan, and the production order of the product in that production plan. To distinguish the product data acquired by the planning product data acquisition unit 121a from the product data (learning model creation product data) included in the training data acquired by the learning model creation device 110 (trainer data acquisition unit 111a), the planning product data is referred to as planning product data as necessary. The planning product data includes values ​​of multiple items, including specifications (manufacturing specifications m), for each product included in the production plan to be corrected, and the production order of the product in that production plan. The image correction device 120 does not need to acquire pass / fail judgment data 220. Furthermore, the product data for creating a learning model acquired by the planning product data acquisition unit 121 may be one (i.e., the product data for creating a learning model acquired by the planning product data acquisition unit 121 may be product data in one production plan).

[0071] <<Planning Image Creation Unit 121b>> The planning image creation unit 121b creates a plan-representation original image for planning and a plan-representation image for planning based on the planning product data acquired by the planning product data acquisition unit 121a. The method for creating the planning-representation original image for planning and the plan-representation image for planning is the same as the method for creating the plan-representation original image for learning model creation 210 and the plan-representation image for learning model creation 230. Therefore, in the following description, the plan-representation image created by the planning image creation unit 121b will be referred to as a "plan-representation image for planning" as needed to distinguish it from the plan-representation original image (plan-representation image for learning model creation 230) created by the learning model creation device 110 (learning model creation image creation unit 111b). Furthermore, the image correction device 120 uses the CNN created by the learning model creation device 110. Therefore, the following description will include mathematical expressions as needed, and use the same symbols as those described with reference to FIGS. 2 to 6. For convenience of explanation, the plan representation image for planning will be the image shown in FIG. 2 and the like, and the plan representation image for planning will also be described by adding the reference numeral 230 as necessary.

[0072] When the image to be corrected in the image correction device 120 is a captured image, the correction image acquisition unit 121 only needs to acquire the captured image to be corrected, and there is no need to acquire product data for planning or create an image different from the captured image.

[0073] <Confidence calculation unit 122> The certainty calculation unit 122 calculates a certainty factor for the pass / fail of an image to be corrected by inputting the image to be corrected into a learning model (input layer 310 of CNN) stored in the storage unit 113. In this embodiment, an example is shown in which the certainty calculation unit 122 inputs a plan representation image for planning 230 created by the plan representation image creation unit 121b into the learning model (input layer 310 of CNN) stored in the storage unit 113, and calculates a certainty factor for the pass / fail of a production plan represented by the plan representation image for planning 230. More specifically, an example is shown in which the certainty calculation unit 122 calculates a certainty factor Y1 for a good plan and a certainty factor Y2 for a bad plan.

[0074] <Image Correction Unit 123> The image correction unit 123 corrects the image input to the input layer 310 of the learning model (CNN) by the confidence calculation unit 122 based on the confidence calculated by the confidence calculation unit 122 (confidence Y1 for a good image and confidence Y2 for a bad image). In this embodiment, a case is illustrated in which the image correction unit 123 corrects the plan-representing image for planning 230 input to the input layer 310 of the learning model (CNN) by the confidence calculation unit 122 based on the confidence calculated by the confidence calculation unit 122 (confidence Y1 for a good plan and confidence Y2 for a bad plan). In addition, in this embodiment, a case is illustrated in which the image correction unit 123 includes an evaluation value calculation unit 123a, an error partial differential value calculation unit 123b, a correction unit 123c, and a determination unit 123d.

[0075] <<Evaluation Value Calculation Unit 123a>> The evaluation value calculation unit 123a calculates the value of the evaluation function E based on the confidence calculated by the confidence calculation unit 122 (in this embodiment, confidence Y1 for a good plan and confidence Y2 for a bad plan). The evaluation function E includes a confidence error, which is an error of the confidence with respect to the target value (as a parameter that defines the function). As described above, this embodiment illustrates a case in which the confidence errors are a good plan confidence error, which is an error of the confidence Y1 for a good plan with respect to the target value of 1, and a bad plan confidence error, which is an error of the confidence Y2 for a bad plan with respect to the target value of 0, and illustrates a case in which the evaluation function E is the following formula (3).

[0076]

number

[0077] In equation (3), (1-Y1) represents the good plan confidence error, and (0-Y2) represents the bad plan error confidence. Therefore, the smaller the absolute values ​​of the good plan confidence error (1-Y1) and the bad plan confidence error (0-Y2), the smaller the value of the evaluation function E. In this embodiment, we illustrate a case where the plan representation image 230 for planning input to the input layer 310 of the learning model (CNN) is modified to reduce the value of the evaluation function E. Alternatively, we can modify the plan representation image 230 for planning input to the input layer 310 of the learning model (CNN) to increase the value of the evaluation function E. In this case, for example, we can use a function obtained by multiplying the entire right-hand side of equation (3) by (-1) as the evaluation function E. In this embodiment, we illustrate a case where the value of the evaluation function E is calculated by the evaluation value calculation unit 123a, because the value of the evaluation function E may be used in equation (17), which will be described later. However, in this embodiment, the value of the evaluation function E itself is not used in the calculation for correcting the image to be corrected (the plan-representing image for plan creation 230). Therefore, for example, when equation (17) described later is not used, the plan-representing image correcting unit 123 does not need to have the evaluation value calculating unit 123a.

[0078] <<Error partial differential value calculation unit 123b>> The error partial differential value calculation unit 123b calculates the error partial differential value ∂E / ∂x using the confidence factors calculated by the confidence factor calculation unit 122 (in this embodiment, the confidence factor Y1 for a good plan and the confidence factor Y2 for a bad plan). v,h m Calculate the error partial differential value ∂E / ∂x v,h m is a function of calculating the value of the evaluation function E by the pixel value x of the image to be corrected (in this embodiment, the plan-representing image 230 for plan creation). v,h mAs described above, in this embodiment, the confidence calculated by the confidence calculation unit 122 is the confidence Y1 for a good plan and the confidence Y2 for a bad plan, and the confidence errors are the good plan confidence error (1-Y1) and the bad plan confidence error (0-Y2). In this embodiment, the error partial differential value calculation unit 123b includes a first partial partial differential value calculation unit 123b1, a second partial partial differential value calculation unit 123b2, and a partial partial differential value combination unit 123b3.

[0079] <<<First Partial Differential Value Calculation Unit 123b1>>> The first partial differential value calculation unit 123b1 calculates the confidence factor calculated by the confidence factor calculation unit 122 (in this embodiment, the confidence factor Y1 for a good plan and the confidence factor Y2 for a bad plan) and the calculated value u input to the output layer 340. k 0 and the value of the evaluation function E is calculated as the value u k 0 The first partial differential value ∂E / u k 0 Calculate.

[0080] In this embodiment, the processing in the convolution layer 320 uses the above-described equations (1) and (2). An example of calculations performed in the processing in the pooling layer 330 and the output layer 340 will be described below.

[0081] First, in this embodiment, a case where the CNN does not include a fully connected layer is illustrated. Therefore, in this embodiment, in the pooling layer 330, the calculated value (component) p i,j m_fpis calculated, and the neurons of the output layer 340 and each component of the pooling layer 330 are fully connected. For convenience of notation, the example shown here is a case where 2 × 2 elements of the feature map constituting the convolutional layer 320 are compressed into one element. When the calculation of formula (4a) is performed, the pooling layer 330 is a so-called max pooling layer. When the calculation of formula (4b) is performed, the pooling layer 330 is a so-called average pooling layer.

[0082]

number

[0083] Next, in this embodiment, the calculated value p stored in each output table of the pooling layer 330 is calculated by the following equation (5): i,j m_fp The weighted linear sum u k 0 is the calculated value u k 0 Here, k is a parameter for identifying neurons in the output layer 340, and indicates a value that is equal to or greater than 1 and equal to or less than the number of neurons in the output layer 340 (the number of calculated values ​​output from the output layer 340) (in the following description, k will be referred to as the neuron identification parameter k of the output layer 340 as necessary). In this embodiment, the confidence Y1 for a good plan and the confidence Y2 for a bad plan are output from the output layer 340, so "1" and "2" are set as the neuron identification parameter k of the output layer 340 (i.e., u k 0 is expressed as u1 0 and u2 0 becomes).

[0084]

number

[0085] Here, J mis the number of output tables of the pooling layer 330, and indicates the number of rows (columns) in the direction corresponding to the row direction (horizontal direction) of the image input to the input layer 310. m is the number of output tables of the pooling layer 330, and indicates the number of rows in the direction corresponding to the column direction (vertical direction) of the image input to the input layer 310. i,j_k m_fp is the calculated value (each component of the pooling layer 330) p stored in each output table of the neurons of the output layer 340 and the pooling layer 330. i,j m_fp As described above, in this embodiment, the parameter k for identifying the neurons in the output layer 340 is set to "1" or "2" (i.e., w i,j_k m_fp is expressed as w as in equation (5). i,j_1 m_fp and w i,j_2 m_fp In practice, a bias (threshold) is added to the right side of equation (5), but the bias (threshold) itself is realized using known CNN technology. Therefore, the notation and detailed explanation of the bias (threshold) will be omitted here.

[0086] Next, in this embodiment, an example is shown in which the confidence level Y1 for a good image (confidence level for a good plan in this embodiment) and the confidence level Y2 for a bad image (confidence level for a bad plan in this embodiment) are calculated in the output layer 340 using the following equation (6).

[0087]

number

[0088] where σ 0 (u) represents the activation function. Note that the activation function σ used in the convolution layer 320 is c Similar to (u), the activation function σ used in the output layer 340 0 For (u), a known function used in CNN, such as a sigmoid function, may be used.

[0089] In this embodiment, an example of the processing in the first partial differential value calculation unit 123b1 and the second partial differential value calculation unit 123b2 will be described, taking as an example a case where the above calculations are performed in the convolution layer 320, the pooling layer 330, and the output layer 340. Therefore, the certainty factor calculation unit 122 calculates the pixel value x v,h m is input to the input layer 310, the following calculation value is calculated. That is, the confidence calculation unit 122 calculates the pixel value x v,h m The calculated value u of the convolution operation in the convolution layer 320 is calculated using l,s m_fc and the calculated value u of the convolution operation l,s m_fc The calculated value c of the convolution layer 320 is calculated using l,s m_fc and the calculated value c l,s 3_fc The calculated value (component) p of the pooling layer 330 is calculated using i,j m_fp and each calculated value (each component) p i,j m_fp The weighted linear sum of the calculated value u k 0 The calculated value u k 0 and then calculating the calculated values ​​Y1 and Y2 (output values ​​of the output layer 340) output from the output layer 340 using the above.

[0090] The pixel value x of the image to be corrected (in this embodiment, the plan representation image 230 for plan creation) v,h m The value of the evaluation function E is partially differentiated by ∂E / ∂x v,h m is the pixel value x of the image to be corrected (in this embodiment, the plan-representing image 230 for plan creation). v,h m represents the change in the evaluation function E relative to the change in . v,hm x becomes "0" v,h m is the x that minimizes the value of the evaluation function E. v,h m Therefore, ∂E / ∂x v,h m Depending on x v,h m is corrected, the pixel value x of the image to be corrected (in this embodiment, the plan-representing image 230 for plan creation) is corrected so that the value of the evaluation function E becomes smaller. v,h m That is, the pixel value x of the image to be corrected (in this embodiment, the plan-representing image 230 for plan creation) can be corrected. v,h m The correction amount is ∂E / ∂x v,h m It can be determined based on ∂E / ∂x v,h m is expressed by the following equation (7) according to the chain rule. That is, based on the evaluation function E, the pixel value x v,h m ∂E / ∂x v,h m is ∂E / u k 0 and ∂u k 0 / ∂x v,h m (k=1, 2) can be expressed as the product of two differential coefficients. k 0 The first partial differential value calculation unit 123b1 calculates the latter ∂u k 0 / ∂x v,h m is calculated by the second partial differential value calculation unit 123b2.

[0091]

number

[0092] (7) ∂E / ∂u1 on the right side of equation (7) 0 , ∂E / ∂u2 0is expressed as the following equation (8).

[0093]

number

[0094] where {σ 0 (u1 0 )}' is σ 0 (u1 0 ) to u1 0 It shows that it is the value differentiated by {σ 0 (u2 0 )}' is σ 0 (u2 0 ) to u2 0 This notation (expressing ' as a differential operator) is also used in other expressions. 0 , u2 0 is the calculated value p stored in each output table of the pooling layer 330, which is input from the pooling layer 330 to the output layer 340. i,j m_fp (each component of the pooling layer 330).

[0095] As mentioned above, Y1, Y2, and u1 on the right side of equation (8) 0 , u2 0 is the pixel value x of the image to be corrected (in this embodiment, the plan-representing image 230 for plan creation). v,h m is input to the input layer 310 and calculated by the confidence calculation unit 122. Also, the activation function σ 0 The function form of is predetermined. Therefore, the first partial differential value calculation unit 123b1 calculates the first partial differential value ∂E / u1 0 , ∂E / u2 0 can be calculated.

[0096] <<<Second Partial Differential Value Calculation Unit 123b2>>> The second partial differential value calculation unit 123b2 calculates the calculated value u to be input to the output layer 340 using the results of the calculation performed by the trained CNN by inputting the image to be corrected (in this embodiment, the plan-representing image for plan creation 230) to the input layer 310 by the confidence calculation unit 122. k 0 (The weighted linear sum u of each component of the pooling layer 330 input to the output layer 340 from the pooling layer 330 k 0 ) is calculated by multiplying the pixel value x of the image to be corrected (in this embodiment, the plan-representing image 230 for plan creation). v,h m The second partial differential value ∂u k 0 / ∂x v,h m Calculate.

[0097] In this embodiment, the second partial differential value calculation unit 123b2 calculates a value calculated by the second calculation, which is a calculation performed on the upstream side of the CNN (the input layer 310 side) before the first calculation, which is a calculation performed on the downstream side of the CNN (the output layer 340 side) by inputting the image to be corrected (in this embodiment, the plan representation image for planning 230) to the input layer 310 by the confidence calculation unit 122, as the pixel value x v,h m The partial derivative value ∂u l,s m_fc / ∂x v,h m , ∂c l,s m_f / ∂x v,h m , ∂p i,j m_fp / ∂x v,h m Using this, the calculated value c l,s m_fc , p i,j m_fp , u k 0 is the pixel value x of the image to be corrected (in this embodiment, the plan-representing image 230 for plan creation). v,h m The partial derivative value ∂c l,s m_fc / ∂xv,h m , ∂p i,j m_fp / ∂x v,h m , ∂u k 0 / ∂x v,h m By performing calculations including sequentially calculating the second partial differential value ∂u k 0 / ∂x v,h m This example shows the case where ∂u is calculated. l,s m_fc / ∂x v,h m Using ∂c l,s m_fc / ∂x v,h m Calculate the calculated ∂c l,s m_f / ∂x v,h m Using ∂p i,j m_fp / ∂x v,h m Calculate the calculated ∂p i,j m_fp / ∂x v,h m Using ∂u k 0 / ∂x v,h m This calculation is performed by using ∂u k 0 / ∂x v,h m is calculated (i.e., the pixel value x of the image to be corrected (in this embodiment, the plan-representing image 230 for plan creation) v,h m The first calculation of partial differentiation with respect to l,s m_fc , p i,j m_fp , u k 0 is the calculated value u k 0 until the pixel value x v,h m The first calculation of partial differentiation with l,sm_fc , p i,j m_fp , u k 0 This is done by changing the position of the CNN where the is calculated from the input layer 310 side to the output layer 340 side.

[0098] As described above, the pixel value x of the image to be corrected (in this embodiment, the plan-representing image 230 for plan creation) v,h m ∂E / ∂x shown in equation (7) v,h m If the pixel value x of the image to be corrected (in this embodiment, the plan-representing image 230 for plan creation) is corrected in accordance with v,h m We can modify the right-hand side of equation (7) as follows: ∂E / u1 0 , ∂E / u2 0 The (first partial differential value) is calculated by the first partial differential value calculation unit 123b1. Therefore, the second partial differential value calculation unit 123b2 calculates ∂u k 0 / ∂x v,h m the second partial differential value ∂u k 0 / ∂x v,h m It is calculated as follows.

[0099] As described above, the confidence calculation unit 122 calculates the pixel value x of the image to be corrected (in this embodiment, the plan-representing image 230 for plan creation). v,h m is input to the input layer 310, and u l,s m_fc , c l,s m_fc , p i,j m_fp , u k 0 are calculated in this order. In this embodiment, the second partial differential value calculation unit 123b2 uses these values ​​to calculate ∂u l,s m_fc / ∂x v,h m , ∂c l,sm_fc / ∂x v,h m , ∂p i,j m_fp / ∂x v,h m , ∂u k 0 / ∂x v,h m The case where the values ​​are calculated in this order will be exemplified.

[0100] First, ∂u l,s m_fc / ∂x v,h m is expressed as the following equation (10) from the following equation (9) which is a transformation of equation (1).

[0101]

number

[0102] The second partial differential value calculation unit 123b2 calculates ∂u l,s m_fc / ∂x v,h m Here, the weights w of the kernels 411 to 413 used in the convolution calculation in the convolution layer 320 are calculated. l,s m_fc (w v-q+1,h-r+1 m_fc ) is calculated by the learning model creation device 110 (learning model creation unit 112). Therefore, according to equation (10), ∂u l,s m_fc / ∂x v,h m can be calculated.

[0103] Next, ∂c l,s m_fc / ∂x v,h m is c l,s m_fc =σ c (u l,s m_fc )(σ c (* is the activation function of the convolutional layer), it is expressed as the following equation (11).

[0104]

number

[0105] The second partial differential value calculation unit 123b2 calculates ∂c l,s m_fc / ∂x v,h m Here, the calculated value u of the convolution operation in the convolution layer 320 is calculated. l,s m_fc is the pixel value x of the image to be corrected (in this embodiment, the plan-representing image 230 for plan creation). v,h m is input to the input layer 310 and calculated by the confidence calculation unit 122. Also, the activation function σ c The function form of is predetermined, and its differential coefficient can be calculated. c (u l,s m_fc )}' can be calculated, and by combining with equation (10) and equation (11), ∂c l,s m_fc / ∂x v,h m can be calculated.

[0106] Next, ∂p i,j m_fp / ∂x v,h m is expressed by the following equation (12) from equation (4b): Here, however, an example is shown in which 2 × 2 elements (elements (2i-1, 2j-1), (2i, 2j-1), (2i-1, 2j), (2i, 2j)) of the feature map constituting the convolution layer 320 are compressed into the output table (i, j) of the pooling layer 330.

[0107]

number

[0108] The second partial differential value calculation unit 123b2 calculates ∂p i,j m_fp / ∂x v,h mHere, the calculated value c of the convolution layer 320 is calculated. l,s m_fc is the pixel value x of the image to be corrected (in this embodiment, the plan-representing image 230 for plan creation). v,h m is input to the input layer 310 and calculated by the confidence calculation unit 122. i,j m_fp The function form of (*) is predetermined, and its differential coefficient can be calculated. i,j m_fp (c l,s m_fc )}' can be calculated, and by combining equation (11) with equation (12), ∂p i,j m_fp / ∂x v,h m can be calculated.

[0109] The calculated value p of the pooling layer 330 i,j m_fp When calculated by equation (4a), ∂p i,j m_fp / ∂x v,h m is calculated using the following equation (13).

[0110]

number

[0111] Finally, ∂u k 0 / ∂x v,h m is expressed by the following equation (15) from the following equation (14) in which equation (5) is expressed in terms of the neuron discrimination parameter k of the output layer 340:

[0112]

number

[0113] The second partial differential value calculation unit 123b2 calculates ∂u k 0 / ∂x v,hm (k=1, 2) is calculated. Here, the calculated values ​​(each component of the pooling layer 330) p i,j m_fp The weight w used when fully connecting i,j m_fp is calculated by the learning model creation device 110 (learning model creation unit 112). Therefore, by combining equation (12) with equation (14), ∂u k 0 / ∂x v,h m can be calculated.

[0114] As described above, in this embodiment, the second partial differential value calculation unit 123b2 calculates w v-q+1,h-r+1 m_fc Using equation (10), ∂u l,s m_fc / ∂x v,h m Calculate the ∂u l,s m_fc / ∂x v,h m and u l,s m_fc Using equation (11), ∂c l,s m_fc / ∂x v,h m Calculate the value of ∂c l,s m_f / ∂x v,h m and c l,s m_fc Using equation (12) or (13), ∂p i,j m_fp / ∂x v,h m Calculate the value of ∂p i,j m_fp / ∂x v,h m and w i,j m_fp Using equation (15), the second partial differential value ∂u k 0 / ∂x v,h m An example of calculating (k=1, 2) is shown below.

[0115] That is, ∂c l,s m_fc / ∂x v,h m When calculating c l,s m_fc u, which is calculated upstream of the CNN (on the input layer 310 side) from the calculation of (first calculation) l,s m_fc Calculation value u by calculation (second calculation) l,s m_fc x v,h m The partial derivative value ∂u l,s m_fc / ∂x v,h m is used (see equation (11)). i,j m_fp / ∂x v,h m When calculating p i,j m_fp The calculation of c is performed upstream of the CNN (first calculation). l,s m_fc Calculated value c by calculation (second calculation) l,s m_fc x v,h m The partial derivative value ∂c l,s m_fc / ∂x v,h m is used (see equation (12) or equation (13)). k 0 / ∂x v,h m When calculating u k 0 p, which is calculated upstream of the CNN calculation (first calculation) i,j m_fp Calculation value p by calculation (second calculation) i,j m_fp x v,h m The partial derivative value ∂p i,j m_fp / ∂x v,h m is used (see equation (15)).

[0116] As described above, in this embodiment, the pixel value x of the image to be corrected (in this embodiment, the plan-representing image for plan creation 230) input to the input layer 310 of the CNN by the confidence factor calculation unit 122 is v,h m The calculated value u in each layer when corrected l,s m_fc , c l,s m_fc , p i,j m_fp By propagating the change in x from the input layer 310 side to the output layer 340 side of the CNN, v,h m If you correct the u k 0 The change in ∂u k 0 / ∂x v,h m is calculated.

[0117] <<<Partial partial differential value synthesis unit 123b3>>> As described above, the first partial differential value calculation unit 123b1 calculates the ∂E / u1 0 , ∂E / u2 0 (first partial differential value) is calculated, and the second partial differential value calculation unit 123b2 calculates ∂u1 0 / ∂x v,h m , ∂u2 0 / ∂x v,h m The partial differential value synthesis unit 123b3 calculates the first partial differential value ∂E / u1 0 , ∂E / u2 0 and the second partial differential value ∂u1 0 / ∂x v,h m , ∂u2 0 / ∂x v,h m Using equation (7), the error partial differential value ∂E / ∂x v,h m (v=1,2,...,v max , h=1,2,···,h max m ) is calculated.

[0118] <<Correction part 123c>> The correction unit 123c corrects the error partial differential value ∂E / ∂x calculated by the error partial differential value calculation unit 123b (partial partial differential value synthesis unit 123b3). v,h m In this embodiment, the correction unit 123c corrects the image to be corrected (the plan-representing image for plan creation 230 in this embodiment) input to the learning model using the following equation (16): v,h m The pixel value x of the image to be corrected (in this embodiment, the plan representation image 230 for plan creation) from which the calculation of v,h m Here is an example of correcting the above.

[0119]

number

[0120] Here, (R) and (R-1) indicate the number of iterations of the correction calculation for the image to be corrected (in this embodiment, the plan-representing image 230 for plan creation) (R is an integer equal to or greater than 1). Also, η is a learning coefficient (the pixel value x in one iteration of the calculation). v,h m The larger the absolute value of the learning coefficient η, the lower the calculation load, but the lower the calculation accuracy (pixel value x v,h m On the other hand, the smaller the absolute value of the learning coefficient η, the lower the calculation accuracy (resolution of pixel value x v,h m The learning coefficient η is set in advance from this perspective. However, the learning coefficient η is the sum of the pixel values ​​x after correction. v,h m(R) is the pixel value x v,h m The range of values ​​that can be taken as the above should be determined.

[0121] <<Judgment unit 123d>> The determination unit 123d determines whether a predetermined calculation end condition is satisfied for calculation to modify the image to be modified (in this embodiment, the plan representation image for planning 230). The calculation end condition is not particularly limited, but in this embodiment, a case will be exemplified in which the determination unit 123d determines whether the calculation end condition is satisfied based on the image modified by the modification unit 123c (in this embodiment, the plan representation image for planning 230). For example, as in the following equation (17), the determination unit 123d determines that the calculation end condition is satisfied when the value of the evaluation function E is equal to or less than a threshold value ε1, and otherwise determines that the calculation end condition is not satisfied. Furthermore, the determination unit 123d determines, for example, as in the following equation (18), that the pixel value x v,h m(R) and the pixel value x of the image to be corrected when the number of repetitions is R-1 (the image corrected by the correcting unit 123c one time before (in this embodiment, the plan-representing image 230 for plan creation)). v,h m(R-1) If the maximum absolute value of the difference between and is equal to or less than the threshold value ε2, it may be determined that the calculation termination condition is met, and if not, it may be determined that the calculation termination condition is not met. Also, as mentioned above, the calculation termination condition may be that the number of iterative calculations is a predetermined value.

[0122]

number

[0123] If the determination unit 123d determines that the calculation termination condition is not satisfied, the certainty factor calculation unit 122 inputs the latest image corrected by the correction unit 123c (in this embodiment, the plan-representing image for plan creation 230) to the input layer 310 as a new image to be corrected. Then, the certainty factor calculation unit 122 and the image correction unit 123 perform the above-mentioned processing again. The certainty factor calculation unit 122 and the image correction unit 123 repeatedly perform such processing until the calculation termination condition is satisfied. Then, if the judgment unit 123d determines that the calculation termination condition is met, the image correction unit 123 determines that the latest image corrected by the correction unit 123c (in this embodiment, the plan representation image 230 for plan creation) is the final corrected image.

[0124] <Planning Department 124> When the plan representation image for planning 230 corrected by the image correction unit 123 satisfies a predetermined calculation termination condition for the calculation to correct the plan representation image for planning 230, the plan creation unit 124 creates a production plan based on the corrected plan representation image for planning 230. In the following description, the plan representation image for planning 230 will be referred to as the plan representation image for planning 230 after final correction, as necessary. Note that when the image to be corrected in the image correction device 120 is a captured image or the like, the image correction device 120 does not need to include the plan creation unit 124.

[0125] In this embodiment, the pixel value x of the plan representation image 230 for planning after the final correction is v,h m can take a value other than "0" and "1" by the calculation of equation (16). Therefore, the planning unit 124 calculates the pixel value x v,h m The maximum value of the pixel value x is converted to "1" and the rest are converted to "0" for each manufacturing specification m. v,h m The final modified plan-representing image for planning 230, into which the above has been converted, is referred to as the optimal plan-representing image for planning 230, as necessary. For example, assume that the optimal plan-representing image for planning 230 is as shown in FIG. 4. In this case, the black areas in FIG. 4, which indicate pixel values ​​of "1," identify the production order in which products with specific manufacturing specifications m (size, material, quality) should be manufactured. In this embodiment, the image modification device 120 creates the optimal plan-representing image for planning 230 as a production plan.

[0126] When actually producing a product, the production plan is used, for example, in the following manner. First, the plan creation unit 124 inputs, for example, data indicating the manufacturing specification m for each of multiple products to be produced. Then, for each of all products included in the input data, the plan creation unit 124 identifies products for which the values ​​of all manufacturing specifications m correspond to pixels with a pixel value of "1" (black pixels) in the optimal plan creation plan representation image 230. This determines the production order of all products included in the input data. The plan creation unit 124 outputs information including the determined product production order as information indicating a production plan. The output may be, for example, at least one of display on a computer display, storage in a storage device internal or external to the image correction device 120, and transmission to an external device.

[0127] Alternatively, the plan creation unit 124 may output an optimal plan representation image 230 for plan creation and present it to the planner. In this case, the planner determines the production sequence of each product by identifying, for each of a plurality of products to be produced, products whose manufacturing specifications m correspond to pixels with pixel values ​​of "1" (black pixels) in the optimal plan representation image 230 for plan creation. In this case, the image correction device 120 may input and store information including the manufacturing specifications m, delivery dates, and production sequences of each product determined in this manner.

[0128] [flowchart] Next, an example of a learning model creation method performed using the learning model creation device 110 of this embodiment will be described with reference to the flowchart of FIG.

[0129] First, in step S800, the learning image acquisition unit 111 acquires an image to be learned and pass / fail judgment data for that image. In this embodiment, the processing of step S800 is realized by steps S801 and S802. In step S801, the training data acquisition unit 111a acquires training data. The training data includes product data for creating a learning model and pass / fail judgment data 220. The product data for creating a learning model includes the value of the manufacturing specification m for each product included in a certain production plan and the production order of the product in that production plan. The pass / fail judgment data 220 includes data indicating "1" or "0" as the confidence level Y1 for a good plan and the confidence level Y2 for a bad plan.

[0130] Next, in step S802, the learning model creation image creation unit 111b creates the learning model creation plan expression image 230 based on the learning model creation product data included in the teacher data in step S801.

[0131] Next, in step S803, the learning model creation unit 112 performs machine learning using the learning target image acquired in step S800 to create a learning model (CNN) that takes the image as input and outputs confidence levels for the image's quality (confidence level Y1 for a good image and confidence level Y2 for a bad image). In this embodiment, the learning model creation unit 112 performs supervised learning using the learning model creation plan representation image 230 created in step S802 to create a learning model (CNN) that takes the plan representation image as input and outputs confidence levels for the quality of the plan (confidence level Y1 for a good plan and confidence level Y2 for a bad plan).

[0132] Next, in step S804, the learning model creation unit 112 stores in the storage unit 113 information about the learning model (CNN) created in step S803.

[0133] Next, an example of an image correction method performed using the image correction device 120 of this embodiment will be described with reference to the flowchart of Fig. 9. The flowchart of Fig. 9 is executed after information on the learning model (CNN) is stored in the storage unit 113 according to the flowchart of Fig. 8.

[0134] First, in step S900, the correction image acquisition unit 121 acquires an image to be corrected. In this embodiment, the processing of step S900 is realized by steps S901 and S902. In step S901, the planning use product data acquisition unit 121a acquires planning use product data. Like the learning model creation product data, the planning use product data includes the value of the manufacturing specification m of each product included in a certain production plan and the production order of the product in the production plan.

[0135] Next, in step S902, the planning image creating unit 121b creates the planning representation image 230 for planning based on the planning product data acquired in step S901. Next, in step S903, the certainty calculation unit 122 calculates certainty factors (certainty factor Y1 for a good image and certainty factor Y2 for a bad image) as to whether the image to be corrected, acquired in step S900, is good or bad, by inputting the image to be corrected acquired in step S900 into the learning model (input layer 310 of CNN) stored in the storage unit 113 by executing the processing according to the flowchart of Fig. 8. In this embodiment, the certainty calculation unit 122 inputs the plan representation image for planning 230 created in step S902 into the learning model (input layer 310 of CNN) stored in the storage unit 113 by executing the processing according to the flowchart of Fig. 8, and calculates certainty factors (certainty factor Y1 for a good plan and certainty factor Y2 for a bad plan) as to whether the production plan represented by the plan representation image for planning 230 is good or bad.

[0136] Next, in step S904, the evaluation value calculation unit 123a uses the confidence levels calculated in step S903 (in this embodiment, the confidence level Y1 for a good plan and the confidence level Y2 for a bad plan) to calculate the value of the evaluation function E according to equation (3). Next, in step S905, the first partial differential value calculation unit 123b1 calculates the confidence factors calculated in step S903 (in this embodiment, the confidence factor Y1 for a good plan and the confidence factor Y2 for a bad plan) and the calculated value u1 0 , u2 0 Using equation (8), the first partial differential value ∂E / u1 0 , ∂E / u2 0 Calculate.

[0137] Next, in step S906, the second partial differential value calculation unit 123b2 calculates ∂u l,s m_fc / ∂x v,h m Using ∂c l,s m_fc / ∂x v,h m Calculate ∂c using equation (13) or (14). l,s m_f / ∂x v,h m Using ∂p i,j m_fp / ∂x v,h m Calculate ∂p by equation (15) i,j m_fp / ∂x v,h m Using ∂u k 0 / ∂x v,h m The second partial differential value ∂u is calculated by performing a calculation including the steps of: k 0 / ∂x v,h m Calculate.

[0138] Next, in step S907, the partial differential value synthesis unit 123b3 synthesizes the first partial differential value ∂E / u1 calculated in step S905. 0 , ∂E / u2 0 and the second partial differential value ∂u1 calculated in step S906. 0 / ∂x v,h m , ∂u2 0 / ∂x v,h m Using equation (7), the error partial differential value ∂E / ∂x v,h m (v=1,2,...,v max , h=1,2,···,h max m ) is calculated.

[0139] Next, in step S908, the correction unit 123c calculates the error partial differential value ∂E / ∂x calculated in step S907. v,h m By performing the calculation of equation (16) using the above, the image to be corrected (in this embodiment, the plan-representing image 230 for plan creation) is corrected.

[0140] Next, in step S909, the determination unit 123d determines whether or not the calculation termination condition is satisfied by performing calculation of equation (18) or equation (19). If the result of this determination is that the calculation termination condition is not satisfied (NO in step S909), the processing of step S903 is executed again. In this case, in the processing of step S903, the image corrected in step S908 (the plan representation image 230 for planning) is used as the image to be corrected (rather than the image acquired in step S900 (the plan representation image 230 for planning created in step S902)).

[0141] On the other hand, if the calculation termination condition is satisfied (YES in step S909), the process of step S910 is performed. In step S910, the plan creation unit 124 creates a production plan based on the plan representation image for plan creation 230 corrected in step S908, which was determined to satisfy the calculation termination condition in step S909. Note that if the image to be corrected is not a plan representation image, the process of step S910 is not executed.

[0142] [summary] As described above, the image correction device 120 inputs the image to be corrected (in this embodiment, the plan-representing image for planning 230) into the trained learning model (CNN) to calculate the confidence level for the image (confidence level Y1 for a good plan and confidence level Y2 for a bad plan), and corrects the image to be corrected (in this embodiment, the plan-representing image for planning 230) input into the trained learning model (CNN) based on the calculated confidence levels. If a predetermined calculation termination condition based on the corrected image (in this embodiment, the plan-representing image for planning 230) is satisfied, the image correction device 120 determines the image as the final corrected image. If the predetermined calculation termination condition is not satisfied, the image correction device 120 inputs the corrected image (in this embodiment, the plan-representing image for planning 230) into the trained learning model and recalculates the confidence level for the image. The image to which the learning model (CNN) can be applied is not limited to the plan-representing image for planning 230, but can also be various other images, such as captured images. Furthermore, data other than product data, including multiple data items with desired content for each data item, can be visualized in the same manner as creating a plan-representing image from product data. That is, the content of the data items can be digitized into pixel values ​​using one-hot encoding or similar techniques, and an image having those pixel values ​​can be used in place of the plan-representing image. Therefore, the algorithm for correcting an image does not need to be significantly modified depending on the purpose of the image correction, and the algorithm for correcting an image can be easily modified depending on the purpose of the image correction. Furthermore, image correction can be repeated while successively checking whether the corrected image meets the purpose of the correction. Therefore, an image that meets the purpose of the correction can be generated.

[0143] The above-described embodiments of the present invention can be realized by a computer executing a program. A computer-readable recording medium on which the program is recorded and a computer program product such as the program can also be applied as embodiments of the present invention. Examples of recording media that can be used include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, magnetic tapes, non-volatile memory cards, and ROMs. The embodiments of the present invention may also be realized by dedicated hardware such as an ASIC (Application Specific Integrated Circuit). Furthermore, the above-described embodiments of the present invention are merely examples of specific embodiments for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited by these. In other words, the present invention can be embodied in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0144] 110 Learning model creation device 111 Learning image acquisition unit 111a Teacher data acquisition section 111b Image creation unit for creating learning models 112 Learning Model Creation Department 113 Storage section 120 Image correction device 121 Correction image acquisition unit 121a Product data acquisition section for planning 121b Planning Image Creation Unit 122 Confidence calculation part 123 Image Correction Department 123a Evaluation value calculation unit 123b Error partial differential value calculation unit 123b1 First partial differential value calculation unit 123b2 Second partial differential value calculation unit 123b3 Partial partial differential value synthesis unit 123c Correction section 123d Judgment section 124 Planning Department 210 Original image of plan representation for creating learning model 220 Pass / fail judgment data 230 Plan representation original image (Plan representation image for creating learning model, Plan representation image for creating plan) 310 Input Layer 320 convolutional layers 321~326 Elements of feature maps that make up convolutional layers 330 Pooling Layer 331~333 Pooling layer output table (compressed feature map elements) 340 Output Layer 411~413 Kernel (filter)

Claims

1. An image correction device that corrects an image using a trained learning model that takes an image as input and outputs a confidence level regarding the quality of the image, an image acquisition means for acquiring, as an image to be corrected, a plan representation image in which each of a plurality of items included in the product data corresponds to one of rows and columns, the processing order of a plurality of products included in the product data corresponds to the other of rows and columns, and the value of each item corresponds to a pixel value; a certainty calculation means for inputting the plan-representing image acquired by the image acquisition means into the learning model and calculating a certainty of the success or failure of the plan represented by the plan-representing image; an image correction means for correcting the plan-representing image input to the learning model based on the certainty calculated by the certainty calculation means; Equipped with when the plan-representing image corrected by the image correction means does not satisfy a predetermined calculation end condition for calculation to correct the plan-representing image, the certainty calculation means inputs the plan-representing image corrected by the image correction means into the learning model and calculates the certainty again; The image correction means determines the corrected plan-representing image as the final corrected plan-representing image if the corrected plan-representing image satisfies the calculation termination condition.

2. The image correction means an error partial differential value calculation means for calculating an error partial differential value, which is a value obtained by partially differentiating a value of an evaluation function including a certainty error, which is an error of the certainty from a target value of the certainty, by a pixel value of the plan-representing image, using the certainty calculated by the certainty calculation means; a correcting means for correcting the plan-representing image input to the learning model by using the error partial differential value calculated by the error partial differential value calculating means; 2. The image correction device of claim 1, comprising:

3. the learning model is a convolutional neural network (CNN), The error partial differential value calculation means a first partial differential value calculation means for calculating a first partial differential value, which is a value obtained by partially differentiating a value of the evaluation function with a calculated value input to the output layer of the CNN, using the confidence calculated by the confidence calculation means and a calculated value input to the output layer of the CNN; a second partial differential value calculation means for calculating a second partial partial differential value, which is a value obtained by partially differentiating a calculated value input to an output layer of the CNN by a pixel value of the plan representation image, using a result of calculation performed by the CNN by inputting the plan representation image to an input layer of the CNN; a partial partial differential value synthesizing means for calculating the error partial differential value using the second partial partial differential value and the first partial partial differential value; 3. The image correction device according to claim 2, further comprising:

4. the second partial differential value calculation means calculates the second partial differential value by performing a calculation including calculating a second partial differential value using the first partial differential value; the first partial differential value is a value obtained by partially differentiating a calculated value by a second calculation, which is a calculation performed on the input layer side of the CNN relative to the first calculation, by a pixel value of the plan representation image; the second partial differential value is a value obtained by partially differentiating the value calculated by the first calculation with respect to the pixel value of the plan representation image, 4. The image correction device according to claim 3, wherein the calculation of the second partial differential value is performed by changing a position of the CNN where the calculated value by the first calculation is calculated from the input layer side of the CNN toward the output layer side thereof until the calculated value by the first calculation becomes a calculated value to be input to the output layer of the CNN.

5. An image correction device as described in claim 3 or 4, in which, in the convolutional layer of the CNN, a convolution operation is performed on the planned representation image using a kernel of a size corresponding to the size of an area corresponding to one or more of the items.

6. An image correction device as described in any one of claims 3 to 5, wherein in the convolution operation performed in the convolution layer of the CNN, the weight assigned to the pixel value of a pixel corresponding to a product with a relatively early delivery date is made larger than the weight assigned to the pixel value of a pixel corresponding to a product with a relatively late delivery date.

7. An image correction device as described in any one of claims 1 to 6, wherein the calculation termination condition is a condition that the error of the confidence level with respect to the target value is below a threshold, a condition that the difference from the previous value of the planned representation image is below a predetermined value, or a condition that the number of repeated calculations of the confidence level is a predetermined value.

8. An image correction method for correcting an image using a trained learning model that takes an image as input and outputs a confidence level regarding the quality of the image, an image acquisition step of acquiring, as an image to be corrected, a plan-representing image in which each of a plurality of items included in the product data corresponds to one of rows and columns, the processing order of a plurality of products included in the product data corresponds to the other of rows and columns, and the value of each item corresponds to a pixel value; a certainty calculation step of inputting the plan-representing image acquired in the image acquisition step into the learning model and calculating a certainty of the success or failure of the plan represented by the plan-representing image; an image correction step of correcting the plan-representing image input to the learning model based on the confidence calculated by the confidence calculation step; Equipped with the certainty calculation step, when the plan-representing image corrected by the image correction step does not satisfy a predetermined calculation termination condition for calculation to correct the plan-representing image, inputs the plan-representing image corrected by the image correction step into the learning model and calculates the certainty again; The image correction method includes determining the corrected plan-representing image as the final corrected plan-representing image when the corrected plan-representing image satisfies the calculation termination condition.

9. A program for causing a computer to function as each of the means of the image correction device according to any one of claims 1 to 7.

Images