Image processing device, image processing method, and image processing program

Abstract

To provide an image processing device, an image processing method, and an image processing program that are capable of expressing with sufficient accuracy the relationship between a first image and a second image associated with the first image.SOLUTION: An image processing device, an image processing method, and an image processing program use a controller to which teacher data including a first image and a second image associated with the first image is input. When the first image is input to a learning model, the controller calculates a model image to be output from the learning model, and trains the learning model based on a loss function calculated based on the model image and the second image. The learning model includes an encoder that outputs a first feature map based on the first image, a pixel shuffle layer that generates a second feature map by rearranging pixels that make up the first feature map, and a decoder that outputs the model image based on the second feature map.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] Patent Document 1 discloses a technique for training an image processing model so as to reduce adverse effects such as output of false patterns caused by noise patterns when performing noise reduction processing. According to this technique, the image processing model is trained using low-noise images, which are training data for training the image processing model, and multiple high-noise images of the same scene but with different noise patterns. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-163319 Summary of the Invention [Problem to be solved by the invention]

[0004] According to the technology described in Patent Document 1, in order to train an image processing model, it is necessary to prepare multiple high-noise images of the same scene but with different noise patterns corresponding to a low-noise image. Therefore, if there is a shortage of training data for training the image processing model, there is a problem in that the relationship between a high-noise image and a low-noise image associated with the high-noise image cannot be expressed with sufficient accuracy.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and has as its object to provide an image processing device, an image processing method, and an image processing program that can represent, with sufficient accuracy, the relationship between a first image and a second image associated with the first image. [Means for solving the problem]

[0006] The image processing device, image processing method, and image processing program according to the present disclosure use a controller to which training data including a first image and a second image associated with the first image is input. When the first image is input to a learning model, the controller calculates a model image to be output from the learning model and trains the learning model based on a loss function calculated based on the model image and the second image. The learning model includes an encoder that outputs a first feature map based on the first image, a pixel shuffle layer that generates a second feature map by rearranging the pixels constituting the first feature map, and a decoder that outputs the model image based on the second feature map.

[0007] A plurality of pixel shuffle layers may be connected in multiple stages between the encoder and the decoder.

[0008] The pixel shuffle layer may include a convolutional layer, a shuffle layer, a normalization layer, and a ReLU layer. Here, the convolutional layer receives the first feature map. The shuffle layer rearranges the pixels of the convolved map output from the convolutional layer. The normalization layer normalizes the output from the shuffle layer. The ReLU layer performs thresholding on the output from the normalization layer.

[0009] The controller may acquire an input image, input the input image into a trained learning model, calculate an output image to be output from the learning model, and output the output image.

[0010] The input image may be a curvature image acquired by photometric stereo. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to provide an image processing device, an image processing method, and an image processing program that can express the relationship between a first image and a second image that corresponds to the first image with sufficient accuracy. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram illustrating a configuration of an image processing device according to an embodiment of the present disclosure. [Figure 2] 10 is a flowchart showing a processing procedure of the image processing device. [Figure 3] FIG. 2 is a block diagram showing the configuration of a learning model. [Figure 4] FIG. 1 is a block diagram showing the configuration of a pixel shuffle layer. [Figure 5] FIG. 10 is a schematic diagram illustrating an example of pixel rearrangement by a shuffle layer. [Figure 6A] FIG. 10 is a diagram showing an example of a processing result by a learning model (not including a pixel shuffle layer). [Figure 6B] FIG. 10 is a diagram showing an example of a processing result by a learning model (including two pixel shuffling layers). [Figure 6C] FIG. 10 is a diagram showing an example of a processing result by a learning model (including four pixel shuffle layers). [Figure 6D] FIG. 10 is a diagram showing an example of a processing result by a learning model (including six pixel shuffling layers). DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, several exemplary embodiments will be described with reference to the drawings. Note that common parts in the drawings are given the same reference numerals, and duplicated explanations will be omitted.

[0014] [Configuration of image processing device] 1 is a block diagram showing the configuration of an image processing device according to an embodiment of the present disclosure. The image processing device 20 includes an input unit 21, an output unit 23, and a controller 25. The controller 25 is connected to the input unit 21 and the output unit 23 so as to be able to communicate with them. The image processing device 20 may also be connected to an imaging device 10 and a database 30.

[0015] The input unit 21 acquires training data used for training a learning model (described later) and input images to be subjected to image processing. The training data includes a first image and a second image associated with the first image. The acquired training data and input images are transmitted to the controller 25.

[0016] For example, the input unit 21 may acquire the first image and the input image from the imaging device 10. Here, the imaging device 10 may have an imaging element such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The first image and the input image include an image to be detected, such as a character or an object. In addition to the image to be detected, the first image and the input image also include noise that may hinder the detection of the image.

[0017] For example, a case will be described in which the first image and the input image include engraved characters engraved on the object to be imaged by the imaging device 10. For example, the engraved characters are embossed or engraved characters such as serial numbers. If the object to be imaged is metal, the engraved characters may be engraved using a wide variety of methods, such as dot marking, which stamps dots on the metal surface, or vibromarking, which thermally carves the metal surface into the shape of characters. The engraved characters may be handwritten by a person or engraved by a machine. The material of the object to be imaged may be various, such as wood, concrete, or metal. The materials of the engraved characters and the object to be imaged are not limited to the examples given here.

[0018] In addition to the engraved characters, the first image and the input image may contain noise such as scratches and polishing marks. This noise can reduce the accuracy of recognizing engraved characters. To improve the accuracy of recognizing engraved characters, it is necessary to remove the noise through image processing.

[0019] The first image and the input image may be curvature images acquired by photometric stereo. The curvature image may be generated based on multiple images acquired by irradiating the surface of the object to be imaged from different directions and capturing the surface.

[0020] Photometric stereo uses known combinations of illumination and reflectance in multiple images to analyze surface curvature and highlight changes in surface normals. Changes in surface normals can represent the shape of engraved characters. Other changes in surface normals can also represent defects such as cracks, scratches, and dents. However, changes in surface normals caused by defects such as cracks, scratches, and dents are often smaller than changes in surface normals caused by the shape of engraved characters. The curvature image can be a normal vector image that shows the inclination of the surface of the imaged object, or an image that shows the curvature of the surface of the imaged object.

[0021] The first image and the input image are not limited to those including printed characters. For example, the first image and the input image may be image data obtained by scanning a printed material on which characters, line drawings, etc. are printed. In this case, the image to be detected is the characters, line drawings, etc. contained in the image data. Furthermore, noise may be dust adhering to the surface of the printed material, fading ink, or other stains on the printed material.

[0022] The second image is an image associated with the first image, and is an image obtained by removing noise contained in the first image. For example, a user may modify the first image to remove noise, and the image after the noise removal may be used as the second image. Alternatively, the first image may be binarized and then manually removed, and the image after the noise removal may be used as the second image. The method for creating the second image is not limited to the examples given here.

[0023] The noise to be removed may vary depending on the type of detection target contained in the first image. If the detection target is text, pixels other than those that make up the text must be considered to be the background of the text. If the detection target is a line drawing, pixels other than those that make up the line drawing must be considered to be the background of the line drawing. Therefore, noise may be removed from pixels other than those that make up the text or line drawing so that they are the same color, and the second image may be generated.

[0024] Alternatively, the input unit 21 may acquire the training data (first image, second image) and the input image from the database 30. The database 30 stores the first image and the second image associated with the first image in association with each other. For example, the database 30 may be any of various storage devices such as a hard disk, flash memory, ROM, RAM, optical disk, and magnetic tape.

[0025] The database 30 may store a learning model, which will be described later. More specifically, the database 30 may store "parameters" related to the learning model.

[0026] The output unit 23 outputs information generated by the controller 25. For example, the output unit 23 outputs a model image output from a learning model. The information output from the output unit 23 may be presented to a user on a display (not shown) or may be stored in the database 30.

[0027] The controller 25 is a general-purpose computer equipped with a CPU (Central Processing Unit), a memory, and an input / output unit. A computer program (recognition program) for functioning as the image processing device 20 is installed in the controller 25. By executing the computer program, the controller 25 functions as a plurality of information processing circuits (251, 253, 255) equipped in the image processing device 20.

[0028] In this disclosure, an example is shown in which multiple information processing circuits (251, 253, 255) are realized by software. However, it is also possible to configure the information processing circuits (251, 253, 255) by preparing dedicated hardware for executing each of the information processes described below. Also, the multiple information processing circuits (251, 253, 255) may be configured by individual hardware.

[0029] As shown in FIG. 1, the controller 25 includes a learning model holding unit 251, a calculation unit 253, and a parameter adjustment unit 255 as a plurality of information processing circuits (251, 253, 255).

[0030] The learning model holding unit 251 holds a learning model to be trained using teacher data. Here, a first image or an input image is input to the "learning model," and a model image calculated based on the input image is output from the learning model. After training of the learning model, which will be described later, is completed, the learning model holding unit 251 may output the trained learning model via the output unit 23.

[0031] The learning model is composed of a neural network, which contains multiple layers, through which signals are propagated in order. Each layer is composed of one or more units.

[0032] Units between layers are connected to each other, and each unit may have an activation function (e.g., a sigmoid function, a normalized linear function, a softmax function, etc.). A weighted sum is calculated based on multiple inputs to the unit, and the output of the unit is the value of the activation function, with the sum value as a variable. For example, in machine learning, the weights used when calculating the sum in each unit of a neural network are adjusted as parameters related to the learning model.

[0033] The controller 25 may store, in a memory (not shown), the connection relationships between units in the neural network and the weights used to calculate the sum for each unit of the neural network. Hereinafter, the "connection relationships" and "weights" between units will be referred to as "parameters."

[0034] Next, the structure of the learning model will be explained in more detail. Figure 3 is a block diagram showing the configuration of the learning model. The learning model includes an encoder, an intermediate network, and a decoder.

[0035] The encoder is composed of N convolutional layers EL1 to ELN (N is an integer equal to or greater than 1) and outputs a first feature map based on a first image. The intermediate network is composed of N pixel shuffle layers PSB1 to PSBN (N is an integer equal to or greater than 1) and generates a second feature map by rearranging the pixels that make up the first feature map. The structure of the pixel shuffle layers and the manner in which the pixels are rearranged will be described later. The decoder is composed of N convolutional layers DL1 to DLN (N is an integer equal to or greater than 1) and outputs a model image based on the second feature map. The numbers of convolutional layers that make up the encoder, intermediate network, and decoder may differ from one another.

[0036] Units are connected between each layer that makes up the encoder, intermediate network, and decoder. More specifically, multiple layers are arranged in the order of convolutional layers EL1-ELN, pixel shuffle layers PSB1-PSBN, and convolutional layers DL1-DLN, and units in adjacent layers are connected.

[0037] Note that by connecting units, the set of signals that a layer of interest receives from adjacent layers can be considered as an "image." The set of signals is processed in the order of convolutional layers EL1-ELN, pixel shuffle layers PSB1-PSBN, and convolutional layers DL1-DLN, and the "images" processed in the above layer order can be considered to be transmitted sequentially. Therefore, the units that make up each layer can be considered as pixels of the "image" input and output to each layer (hereinafter, "unit" and "pixel" may be used interchangeably).

[0038] Each convolutional layer applies a "convolution operation" to the received signal pair and transmits the obtained result. For example, convolutional layer EL1 applies a "convolution operation" to the first image and transmits the obtained result to convolutional layer EL2. In this way, the signal pair flows in the order of convolutional layers EL1 to ELN, pixel shuffle layers PSB1 to PSBN, and convolutional layers DL1 to DLN.

[0039] Let us explain "convolution operation". Of the multiple convolution layers that make up the learning model, the convolution layer of interest (first convolution layer) receives an "image" from the adjacent convolution layer (second convolution layer) consisting of a set of signals "W1 × W2". The value of the signal contained in the "image" is multiplied by x ij (Note that i=0,1,···,W1-1, j=0,1,···,W2-1).

[0040] On the other hand, to define the operation of convolution, we consider a "filter" consisting of a set of signals "H1 × H2". The size of the "filter" ("filter size") is "H1 × H2". The value of the signal included in the "filter" is expressed as h pq (Note that p = 0, 1, , H1-1, q = 0, 1, , H2-1). "H1 × H2" is an integer greater than or equal to 2.

[0041] In this case, the convolution operation "u ij " is defined as the following equation 1 between the above "image" and "filter".

[0042]

number

[0043] In other words, each unit in the first convolutional layer of interest is connected to the adjacent units in the second convolutional layer, numbering "H1 × H2". The value "h" defines the "filter" for each unit. pq " is the parameter that defines the learning model to which the first convolutional layer belongs.

[0044] As described above, the encoder generates a first feature map, which is a latent representation of the input first image. The decoder generates a model image from the second feature map, which is a latent representation. Here, each pixel of the first feature map consists of multiple channels (a set of data indicating feature amounts). Each pixel of the second feature map consists of multiple channels (a set of data indicating feature amounts).

[0045] In the intermediate network, pixel shuffle layers may be connected in multiple stages. Here, "connected in multiple stages" means that the pixel shuffle layers are connected in series. In other words, the output of the Mth stage (M=1, 2, . . . , N-1) pixel shuffle layer is the input to the (M+1)th stage pixel shuffle layer. In the example shown in Figure 3, multiple pixel shuffle layers are connected in multiple stages between the encoder and decoder.

[0046] 4 is a block diagram showing the configuration of the pixel shuffle layer. As shown in FIG. 4, the pixel shuffle layer includes a convolution layer LY1, a shuffle layer LY2, a normalization layer LY3, and a ReLU layer LY4.

[0047] The convolutional layer LY1 receives the first feature map as input. That is, the convolutional layer LY1 is located on the input side of the pixel shuffle layer and receives the input of the pixel shuffle layer. The convolutional layer LY1 applies a "convolution operation" to the input and outputs the result to the shuffle layer LY2.

[0048] The shuffle layer LY2 rearranges the pixels of the convolved map output from the convolution layer LY1. Figure 5 is a schematic diagram showing an example of pixel rearrangement by the shuffle layer. The shuffle layer LY2 outputs a rearranged map that has a larger number of pixels than the number of pixels of the convolved map based on the multiple channels for each pixel that make up the convolved map.

[0049] For example, the upper part of Fig. 5 shows an example in which four channels CH1 to CH4 exist in a convoluted map consisting of 4 x 4 = 16 pixels. The number of pixels and the number of channels are not limited to the example shown in Fig. 5.

[0050] In contrast, as shown in the bottom of Figure 5, shuffle layer LY2 generates a sorted map consisting of 8 x 8 = 64 pixels. Looking at area UC of the sorted map, we can see that it is made up of four pixels, with one piece of data for each of channels CH1 to CH4 arranged in it. Looking at the adjacent 2 x 2 areas in this way, we can see that there is one piece of data for each of channels CH1 to CH4.

[0051] In Fig. 5, data is arranged as shown by data A1 to A16 for channel CH1. As distinguished by different patterns in Fig. 5, data for channels CH2 to CH4 is arranged in the same way.

[0052] If the number of pixels in the convolved map is "W1 × W2" and the number of channels is "r1 × r2" (r1 and r2 are natural numbers), the number of pixels in the sorted map will be "(W1 × r1) × (W2 × r2)". Note that the number of channels is not limited to being expressed as a product of natural numbers.

[0053] The normalization layer LY3 normalizes the output from the shuffle layer. More specifically, the normalization layer LY3 performs a normalization process on the sorted map to generate a normalized sorted map. The normalization process performed by the normalization layer LY3 may be instance normalization or batch normalization.

[0054] The ReLU layer LY4 performs threshold processing on the output from the normalization layer. More specifically, the ReLU layer LY4 replaces the value of a pixel that constitutes the normalized sorted map with 0 if the value is less than a predetermined threshold, and retains the value of the pixel if the value is equal to or greater than the predetermined threshold. Here, the predetermined threshold may be a value that is set in advance. In particular, the predetermined threshold may be set to 0.

[0055] The calculation unit 253 calculates a model image to be output from the learning model when the first image is input to the learning model. The calculation unit 253 also calculates the value of a loss function based on the model image and the second image. Here, the loss function is a function that evaluates the deviation (error) between the model image and the second image. The loss function is a function configured so that the greater the deviation between the model image and the second image, the larger its value becomes, and the smaller the deviation between the model image and the second image, the smaller its value becomes.

[0056] Furthermore, the calculation unit 253 may calculate an output image output from a trained learning model when an input image is input to the trained learning model. The "trained learning model" refers to a learning model that has been trained by the parameter adjustment unit 255, which will be described later.

[0057] The parameter adjustment unit 255 trains the learning model based on the loss function calculated by the calculation unit 253. Specifically, the parameter adjustment unit 255 adjusts the parameters related to the learning model so as to reduce the deviation between the model image output from the learning model to which the first image is input and the second image. For example, the parameter adjustment unit 255 changes the parameters related to the learning model in a direction that reduces the loss function. By adjusting the parameters related to the learning model based on a series of training data, a learning model trained with the training data can be obtained.

[0058] The parameter adjustment unit 255 may use gradient descent, stochastic gradient descent, etc. to adjust the parameters in a direction that reduces the loss function. Here, backpropagation may be used for gradient calculation in gradient descent or stochastic gradient descent.

[0059] Additionally, when training a learning model in the parameter adjustment unit 255, techniques such as regularization that restricts the degrees of freedom of weights during learning may be used to mitigate overfitting to training data. Other techniques may also be used, such as dropout, that stochastically selects units in a neural network and disables other units. Furthermore, techniques such as data regularization, data standardization, and data augmentation that eliminate bias in data may also be used to improve generalization performance.

[0060] The parameter adjustment unit 255 determines whether a training end condition is satisfied for the training of the learning model. For example, the parameter adjustment unit 255 may determine that the training end condition is satisfied when the loss function has a predetermined value for all the training data.

[0061] Alternatively, the training termination condition may be determined based on the number of times the learning model is adjusted based on the teacher data. For example, the parameter adjustment unit 255 may measure the number of epochs, which indicates the number of times the learning model has been adjusted, for each training data, and determine that the training termination condition is met when the number of epochs for all training data exceeds a predetermined number.

[0062] [Processing procedure of image processing device] 2 is a flowchart showing the processing procedure of the image processing device 20. In particular, FIG. 2 shows the processing of the image processing device 20 when training a learning model based on teacher data.

[0063] In step S101, the input unit 21 acquires training data to be used for training the learning model. For example, the input unit 21 selects one of a plurality of training data.

[0064] In step S103, the calculation unit 253 inputs the first image into the learning model.

[0065] In step S105, the calculation unit 253 calculates a model image that is output from the learning model when the first image is input to the learning model.

[0066] In step S107, the calculation unit 253 calculates the value of the loss function based on the model image and the second image.

[0067] In step S109, the parameter adjustment unit 255 adjusts the parameters related to the learning model based on the loss function calculated by the calculation unit 253.

[0068] In step S111, the parameter adjustment unit 255 determines whether or not a training end condition is satisfied for the training of the learning model.

[0069] If the training end condition is not satisfied (NO in step S111), the process returns to step S101. On the other hand, if the training end condition is satisfied (YES in step S111), in step S113, the learning model holding unit 251 outputs the trained learning model via the output unit 23.

[0070] Although not shown in Figure 2, after a trained learning model is obtained, the calculation unit 253 may calculate an output image that is output from the learning model when an input image is input into the trained learning model.

[0071] [Effects of the embodiment] As described above in detail, the image processing device, image processing method, and image processing program according to the present disclosure use a controller to which training data including a first image and a second image associated with the first image is input. When the first image is input to a learning model, the controller calculates a model image to be output from the learning model and trains the learning model based on a loss function calculated based on the model image and the second image. The learning model includes an encoder that outputs a first feature map based on the first image, a pixel shuffle layer that generates a second feature map by rearranging the pixels constituting the first feature map, and a decoder that outputs the model image based on the second feature map.

[0072] This allows the relationship between a first image and a second image associated with the first image to be expressed with sufficient accuracy. In particular, the pixel shuffle layer emphasizes the features of the image to be detected contained in the first image, and the relationship between the first image and the second image can be expressed in a learning model based on a small amount of training data. The training data allows automatic setting of image processing to emphasize the features contained in the images, improving user work efficiency.

[0073] The effect of using pixel shuffle layers will be explained using Figures 6A to 6D. Figure 6A is a diagram showing an example of a processing result by a learning model (not including a pixel shuffle layer). Figure 6B is a diagram showing an example of a processing result by a learning model (including two-stage pixel shuffle layers). Figure 6C is a diagram showing an example of a processing result by a learning model (including four-stage pixel shuffle layers). Figure 6D is a diagram showing an example of a processing result by a learning model (including six-stage pixel shuffle layers).

[0074] Six images were prepared by photographing characters engraved on six metal plates by different writers using the photometric stereo method, and four of these images were used as first images included in the training data to train the learning model. Of the two images not used as training data, one was used as an evaluation image, and the remaining image was input to the trained learning model as an input image to generate an output image. Figure 6A shows the output image generated by the learning model without a pixel shuffle layer after training. Meanwhile, Figures 6B to 6D show the output images generated by the learning model with a pixel shuffle layer after training.

[0075] In Fig. 6A, the inscribed characters are almost unreadable in the region DS, whereas in Fig. 6B to 6D, the inscribed characters are highlighted to the extent that they can be read in the region DS. In this way, by using the pixel shuffle layer, it is possible to emphasize the features of the image to be detected contained in the first image.

[0076] A plurality of pixel shuffle layers may be connected in multiple stages between the encoder and decoder. This allows the features of the image to be detected and contained in the first image to be emphasized. For example, as shown in Figures 6B to 6D, the more the number of pixel shuffle layers is increased, the more the features of the image are emphasized. Note that, compared to Figure 6C, Figure 6D shows that in addition to the image to be detected, noise contained in the first image is also emphasized. For example, it can be seen that a blurred image appears in the region ER where no inscription characters are present. Therefore, while increasing the number of pixel shuffle layers, the number of pixel shuffle layers may be appropriately limited according to the characteristics of the input image to be processed.

[0077] The pixel shuffle layer may include a convolutional layer, a shuffle layer, a normalization layer, and a ReLU layer. Here, the first feature map is input to the convolutional layer. The shuffle layer rearranges the pixels of the convolutional map output from the convolutional layer. The normalization layer normalizes the output from the shuffle layer. The ReLU layer performs threshold processing on the output from the normalization layer. This eliminates the need to prepare multiple training data sets with different noise patterns for training a learning model. Therefore, the relationship between the first image and the second image can be expressed in the learning model based on a small amount of training data. As a result, the load for building a learning model can be reduced.

[0078] The controller may acquire an input image, input the input image to a trained learning model, calculate an output image to be output from the learning model, and output the output image. This makes it possible to extract the image to be detected contained in the input image and remove noise contained in the input image. As a result, by detecting the image to be detected using the output image instead of the input image, the possibility of false detection can be reduced.

[0079] The input image may be a curvature image acquired using photometric stereo. This allows the learning model to represent the characteristics of the image expressed by the curvature image, such as engraved characters. It also makes it possible to remove noise contained in the curvature image.

[0080] Each function described in the above embodiments may be implemented by one or more processing circuits, including programmed processors, electrical circuits, and even devices such as application specific integrated circuits (ASICs), or circuit components arranged to perform the described functions.

[0081] According to the present disclosure, image processing for enhancing features contained in an image can be automatically set, thereby improving the user's work efficiency. As a result, the user's labor productivity can be improved. Therefore, for example, it is possible to contribute to Goal 8 of the Sustainable Development Goals (SDGs) led by the United Nations, "Promote inclusive and sustainable economic growth, full and productive employment and decent work for all."

[0082] Although several embodiments have been described, the embodiments can be modified or varied based on the above disclosure. All components of the above embodiments and all features described in the claims may be individually extracted and combined, unless they are mutually inconsistent. [Explanation of symbols]

[0083] 10. Imaging device 20 Image processing device 21 Input section 23 Output section 25 Controller 251 Learning Model Storage Unit 253 Calculation Unit 255 Parameter Adjustment Section 30 databases LY1 convolutional layer LY2 shuffle layer LY3 Normalization layer LY4 ReLU layer

Claims

1. a first image and a second image associated with the first image; An image processing device including a controller to which training data including The controller When the first image is input to a learning model, a model image output from the learning model is calculated; training the learning model based on a loss function calculated based on the model image and the second image; The learning model is an encoder that outputs a first feature map based on the first image; a pixel shuffle layer that rearranges pixels constituting the first feature map to generate a second feature map; a decoder that outputs the model image based on the second feature map; An image processing device comprising:

2. The image processing device according to claim 1 , wherein a plurality of the pixel shuffle layers are connected in multiple stages between the encoder and the decoder.

3. The pixel shuffle layer comprises: a convolution layer to which the first feature map is input; a shuffle layer that rearranges the pixels of the convolved map output from the convolution layer; a normalization layer that normalizes the output from the shuffle layer; a ReLU layer that performs thresholding on the output from the normalization layer; The image processing device according to claim 1 , comprising:

4. The controller Take an input image, When the input image is input to the trained learning model, an output image output from the learning model is calculated; outputting the output image; The image processing device according to any one of claims 1 to 3.

5. The input image is a curvature image acquired by photometric stereo. The image processing device according to claim 4 .

6. a first image and a second image associated with the first image; An image processing method for controlling a controller to which training data including the following is input, The controller When the first image is input to a learning model, a model image output from the learning model is calculated; training the learning model based on a loss function calculated based on the model image and the second image; The learning model is an encoder that outputs a first feature map based on the first image; a pixel shuffle layer that rearranges pixels constituting the first feature map to generate a second feature map; a decoder that outputs the model image based on the second feature map; An image processing method comprising:

7. a first image and a second image associated with the first image; An image processing program to be executed in a controller to which training data including: calculating a model image output from a learning model when the first image is input to the learning model; training the learning model based on a loss function calculated based on the model image and the second image; Including, The learning model is an encoder that outputs a first feature map based on the first image; a pixel shuffle layer that rearranges pixels constituting the first feature map to generate a second feature map; a decoder that outputs the model image based on the second feature map; An image processing program comprising:

Images