Machine learning model products, generation devices, and recording media

Abstract

Output image data representing an output object having a detailed shape is generated. The machine learning model includes an input layer and an output layer. The input layer includes a first extraction unit, a second extraction unit, one or more displacement information acquisition units, and an addition unit. The first extraction unit extracts shape features of a portion of the input object that is included in a first input range. The second extraction unit is provided in parallel with the first extraction unit and extracts shape features of a portion of the input object that is included in a second input range that is larger than the first input range. The one or more displacement information acquisition units acquire displacement information related to the displacement of the position of the shape feature extracted by the first extraction unit within the input image. The addition unit adds a plurality of data including data output from the first extraction unit, data output from the second extraction unit, and data output from the one or more displacement information acquisition units.

Description

Technical Field

[0001] This specification relates to machine learning models that generate image data. Background Art

[0002] In recent years, the use and development of machine learning models such as neural networks and support vector machines has continued to advance. For example, the technology disclosed in Non-Patent Document 1 uses a data generation model called a variational autoencoder (VAE) for anomaly detection in industrial products.

[0003] Prior art literature

[0004] Non-patent literature

[0005] Non-Patent Literature 1: Ryosuke Tachibana, Takashi Matsubara, and Kuniaki Uehara, “Industrial Product Anomaly Detection Using Unregularized Anomaly Scores Based on Deep Generative Models,” 2018 National Conference of the Chinese Society for Artificial Intelligence (32nd)

[0006] Models that extract features by compressing information, such as VAEs, can extract abstract features. However, this information compression loses detailed features. When such models are used to generate image data containing an object, the generated image data struggles to represent the object's detailed shape. This issue isn't limited to models that extract features by compressing information; it also applies to models that use other methods to extract features. Summary of the Invention

[0007] This specification discloses a technique capable of generating image data representing an object having a detailed shape.

[0008] The technology disclosed in this specification can be realized as the following application examples.

[0009] [Application Example 1] A machine learning model for generating output image data corresponding to input image data of an input image containing an input object is a machine learning model that has been learned, wherein the machine learning model is a model that extracts shape features of an input object contained in the input image of the input image data and generates output image data of an output image containing an output object based on the extracted shape features by performing operations using multiple operation parameters for the input image data. The machine learning model comprises: an input layer that performs operations using the input image data; and an output layer that is provided on the downstream side of the input layer and generates the output image data. The input layer comprises: a first extraction unit that extracts the input image data; a shape feature of a portion of the input object contained in a first input range that is smaller than the input image; a second extraction unit, which is provided in parallel with the first extraction unit and extracts the shape feature of a portion of the input object contained in a second input range that is smaller than the input image and larger than the first input range; one or more displacement information acquisition units, which use the data output from the first extraction unit to perform an operation, thereby obtaining displacement information related to the displacement of the position of the shape feature extracted by the first extraction unit within the input image; and an addition operation unit, which adds a plurality of data including the data output from the first extraction unit, the data output from the second extraction unit, and the data output from the one or more displacement information acquisition units.

[0010] According to this configuration, the first extraction unit outputs data corresponding to the shape features of the portion contained within the relatively small first input range, i.e., the relatively fine shape features, while the second extraction unit outputs data corresponding to the shape features of the portion contained within the relatively large second input range, i.e., the relatively coarse shape features. Furthermore, since these data are summed up in the input layer, it is possible to generate output image data representing an output object having a relatively fine shape in addition to a relatively coarse shape. Furthermore, by adding data output from the displacement information acquisition unit, i.e., data corresponding to displacement information related to the positional displacement of the shape features extracted by the first extraction unit within the input image, robustness against displacement of the relatively fine shape features within the input image can be improved.

[0011] [Application Example 2] A machine learning model according to Application Example 1, wherein the output layer includes: a first restoration unit, which uses a portion of the data of the processing object corresponding to a first output range smaller than the output image to restore the shape feature; and a second restoration unit, which is arranged in parallel with the first restoration unit and uses a portion of the data of the processing object corresponding to a second output range smaller than the output image and larger than the first output range to restore the shape feature.

[0012] With this configuration, the first restoration unit uses the portion of the object data corresponding to the smaller first output range to restore the shape features, i.e., the relatively fine shape features. The second restoration unit uses the portion of the object data corresponding to the larger second output range to restore the shape features, i.e., the relatively coarse shape features. This allows the generation of output image data that represents an output object having both a relatively coarse shape and a relatively fine shape.

[0013] [Application Example 3] A machine learning model according to Application Example 1 or 2, wherein the input layer includes the first extraction unit as a convolutional layer, the second extraction unit as a convolutional layer, and the displacement information acquisition unit as a pooling layer, and the output layer does not include a pooling layer but includes the first restoration unit as a transposed convolutional layer and the second restoration unit as a transposed convolutional layer.

[0014] Because the input layer includes a displacement information acquisition unit that functions as a pooling layer, the effect of displacement of the shape features within the input image on the output layer's operations is mitigated. Consequently, an output layer that does not include a pooling layer but instead includes a first restoration unit that functions as a transposed convolutional layer and a second restoration unit that functions as a transposed convolutional layer can generate output image data that represents an output object with both relatively coarse and detailed shapes.

[0015] [Application Example 4] A machine learning model according to any one of Application Examples 1 to 3, wherein the one or more displacement information acquisition units include: a first displacement information acquisition unit, which acquires first displacement information related to the displacement of a position within a first displacement range that is smaller than the input image; and a second displacement information acquisition unit, which is arranged in parallel with the first displacement information acquisition unit and acquires second displacement information related to the displacement of a position within a second displacement range that is smaller than the input image and larger than the first displacement range, and the addition operation unit adds the multiple data including the data output from the first extraction unit, the data output from the second extraction unit, the data output from the first displacement information acquisition unit, and the data output from the second displacement information acquisition unit.

[0016] According to this configuration, the first displacement information acquisition unit acquires first displacement information related to displacements of positions within a relatively small first displacement range, i.e., relatively small positional displacements, and the second displacement information acquisition unit acquires second displacement information related to displacements of positions within a relatively large second displacement range, i.e., relatively large positional displacements. Furthermore, since these data are summed in the input layer, robustness can be improved against relatively small positional displacements and relatively large positional displacements.

[0017] [Application Example 5] A machine learning model according to any one of Application Examples 1 to 4, wherein the input layer includes a third extraction unit, which is arranged in parallel with the first extraction unit and the second extraction unit, and extracts shape features of a portion of the input object contained in a third input range that is smaller than the input image and larger than the second input range, and the addition unit adds the multiple data including the data output from the first extraction unit, the data output from the second extraction unit, the data output from the third extraction unit, and the data output from the one or more displacement information acquisition units.

[0018] According to this configuration, three shape features corresponding to three levels of detail are extracted, and three types of data corresponding to the three features are added together, thereby generating output image data representing an output object having the three shape features corresponding to the three levels of detail.

[0019] [Application Example 6] A machine learning model according to any one of Application Examples 1 to 5, wherein the multiple operation parameters are adjusted through training so that the difference between the baseline image data of the baseline image containing the baseline object and the output image data generated by inputting the baseline image data into the machine learning model becomes smaller.

[0020] According to this structure, by inputting input image data of an input image including an input object into a machine learning model, output image data including an output image of an object based on shape features extracted by the input layer and having a small difference from a reference object can be generated.

[0021] [Application Example 7] A generating device for generating image data, the generating device comprising: an acquiring unit for acquiring input image data of an input image including an input object; an output image data generating unit for generating output image data corresponding to the input image data by inputting the input image data into a machine learning model that has been learned; and a differential image data generating unit for generating differential image data of a differential image representing the difference between the input image and an output image of the output image data, the machine learning model being a model that extracts shape features of an input object included in the input image of the input image data and generates output image data of an output image including an output object based on the extracted shape features by performing operations using a plurality of operation parameters for the input image data, the machine learning model comprising: an input layer for performing operations using the input image data; and an output layer provided on the downstream side of the input layer and generating the output image data, the input The input layer includes: a first extraction unit, which extracts the shape features of a portion of the input object contained in a first input range smaller than the input image; a second extraction unit, which is set in parallel with the first extraction unit and extracts the shape features of a portion of the input object contained in a second input range smaller than the input image and larger than the first input range; one or more displacement information acquisition units, which use data output from the first extraction unit to perform operations, thereby obtaining displacement information related to the displacement of the position of the shape feature extracted by the first extraction unit in the input image; and an addition operation unit, which adds a plurality of data including data output from the first extraction unit, data output from the second extraction unit and data output from the one or more displacement information acquisition units, and the plurality of operation parameters are adjusted through training so that the difference between the benchmark image data of the benchmark image containing the benchmark object and the output image data generated by inputting the benchmark image data into the machine learning model becomes smaller.

[0022] According to this structure, by inputting input image data including an input image of an input object into a machine learning model, it is possible to generate output image data including an output image of an output object that is less different from a reference object based on the shape features extracted by the input layer. Consequently, it is possible to generate differential image data representing the difference between the input object and the reference object.

[0023] [Application Example 8] A generating device according to Application Example 7, wherein the differential image data generating unit obtains smoothed input image data by smoothing the input image data, obtains smoothed output image data by smoothing the output image data, and generates the differential image data using the smoothed input image data and the smoothed output image data.

[0024] According to this configuration, noise included in the difference image data can be suppressed.

[0025] [Application Example 9] A generating device according to Application Example 8, wherein the differential image data generating unit performs edge extraction processing and corrects the differential image data by extracting a non-edge portion different from the edge portion from the differential image, wherein the edge extraction processing extracts the edge portion within the image of the smoothed output image data.

[0026] The difference image may contain edges due to the misalignment of objects between the input image and the output image. According to the above configuration, the corrected difference image data can appropriately represent the difference from the edges caused by the misalignment between the input image and the output image.

[0027] [Application Example 10] A computer program for a computer that generates image data, the computer program causing the computer to implement the following functions: an acquisition function that acquires input image data of an input image containing an input object; an output image data generation function that generates output image data corresponding to the input image data by inputting the input image data into a machine learning model that has been learned; and a differential image data generation function that generates differential image data of a differential image representing the difference between the input image and the output image of the output image data, the machine learning model being a model that extracts shape features of an input object contained in the input image of the input image data and generates output image data of an output image containing an output object based on the extracted shape features by performing operations using a plurality of operation parameters for the input image data, the machine learning model comprising: an input layer that performs operations using the input image data; and an output layer that is provided on the downstream side of the input layer and generates the output image data. The output image data, the input layer includes: a first extraction unit that extracts shape features of a portion of the input object included in a first input range smaller than the input image; a second extraction unit that is provided in parallel with the first extraction unit and extracts shape features of a portion of the input object included in a second input range smaller than the input image and larger than the first input range; one or more displacement information acquisition units that perform operations using data output from the first extraction unit to thereby obtain displacement information related to the displacement of the position of the shape features extracted by the first extraction unit within the input image; and an addition unit that adds a plurality of data including the data output from the first extraction unit, the data output from the second extraction unit, and the data output from the one or more displacement information acquisition units, the plurality of operation parameters being adjusted through training so that a difference between reference image data of a reference image including a reference object and output image data generated by inputting the reference image data into the machine learning model becomes smaller.

[0028] [Application Example 11] A computer program according to Application Example 10, wherein the differential image data generation function obtains smoothed input image data by smoothing the input image data, the differential image data generation function obtains smoothed output image data by smoothing the output image data, and the differential image data generation function uses the smoothed input image data and the smoothed output image data to generate the differential image data.

[0029] [Application Example 12] A computer program according to Application Example 11, wherein the differential image data generation function performs edge extraction processing, and corrects the differential image data by extracting a non-edge portion different from the edge portion from the differential image, and the edge extraction processing extracts the edge portion within the image of the smoothed output image data.

[0030] [Application Example 13] A generating device for generating image data, comprising: an acquiring unit, which acquires input image data of an input image including an input object; and an output image data generating unit, which generates output image data corresponding to the input image data by inputting the input image data into a machine learning model that has been learned, wherein the machine learning model is a model that extracts shape features of the input object included in the input image of the input image data and generates output image data of an output image including an output object based on the extracted shape features by performing operations using a plurality of operation parameters for the input image data, wherein the machine learning model comprises: an input layer, which performs operations using the input image data; and an output layer, which is provided on the downstream side of the input layer and generates the output image data, wherein the input layer comprises: a first extraction unit, which extracts shape features of the input object included in the input image compared to the first extraction unit. shape features of a portion within a first input range that is smaller than an input image; a second extraction unit that is provided in parallel with the first extraction unit and extracts shape features of a portion of the input object that is contained in a second input range that is smaller than the input image and larger than the first input range; one or more displacement information acquisition units that perform operations using data output from the first extraction unit to thereby obtain displacement information related to the displacement of the position of the shape features extracted by the first extraction unit within the input image; and an addition operation unit that adds a plurality of data including data output from the first extraction unit, data output from the second extraction unit, and data output from the one or more displacement information acquisition units, wherein the plurality of operation parameters are adjusted through training so that a difference between a benchmark image data of a benchmark image containing a benchmark object and output image data generated by inputting the benchmark image data into the machine learning model becomes smaller.

[0031] In addition, the technology disclosed in this specification can be implemented in various forms, for example, in the form of a training method for a machine learning model and a training device for performing training, a method for generating image data using a learned machine learning model and a generation device, a judgment method and a judgment device using image data, a computer program for implementing the functions of these methods or devices, a recording medium (e.g., a non-temporary recording medium) recording the computer program, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is an explanatory diagram showing the image processing system according to the embodiment.

[0033] Figure 2 It is an explanatory diagram of an example of the learning model 234.

[0034] Figure 3 It is a block diagram showing the structure of the input layer L10.

[0035] Figure 4 It is a block diagram showing the structure of the intermediate layer L20.

[0036] Figure 5 It is a block diagram showing the structure of the output layer L30.

[0037] Figure 6 This is a flowchart showing an example of a training process.

[0038] Figure 7 (A) is an explanatory diagram of the reference label sheet 800 . Figure 7 (B) is an explanatory diagram of an example of reference image data. Figure 7 (C) is an explanatory diagram of an example of output image data.

[0039] Figure 8 This is a flowchart showing an example of differential data generation processing.

[0040] Figure 9 This is an explanatory diagram of image processing. DETAILED DESCRIPTION

[0041] A. First embodiment:

[0042] A1.Device structure:

[0043] Figure 1 This is an explanatory diagram of an image processing system according to an embodiment. Image processing system 1000 is a system for processing image data of the appearance of a product 700. Product 700 can be any product and is predetermined. In this embodiment, product 700 is a label sheet printed with various objects such as text and graphics. Hereinafter, product 700 is also referred to as label sheet 700.

[0044] Image processing system 1000 includes a data processing device 200 and a digital camera 100 connected to data processing device 200. Data processing device 200 is, for example, a personal computer. Data processing device 200 includes a processor 210, a storage device 215, a display unit 240 for displaying images, an operation unit 250 for receiving user operations, and a communication interface 270. These elements are interconnected via a bus. Storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0045] The processor 210 is a device for performing data processing, such as a CPU. The volatile storage device 220 is, for example, a DRAM, and the non-volatile storage device 230 is, for example, a flash memory.

[0046] The non-volatile storage device 230 stores a first program 232, a second program 233, and a prediction model 234. In this embodiment, the prediction model 234 is an artificial neural network model, a machine learning model trained through a training process described later (hereinafter also referred to as "machine learning model 234" or simply "learning model 234"). In this embodiment, the learning model 234 is a program module. Details of the learning model 234 will be described later.

[0047] The processor 210 temporarily stores various intermediate data used in executing the first program 232 , the second program 233 , and the learning model 234 in the storage device 215 (for example, either the volatile storage device 220 or the nonvolatile storage device 230 ).

[0048] The display unit 240 is a device that displays images, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device that receives user operations, such as a touch panel, buttons, or a joystick, and is superimposed on the display unit 240. The user can input various instructions to the data processing device 200 by operating the operation unit 250. The communication interface 270 is an interface for communicating with other devices (e.g., a USB interface, a wired LAN interface, or an IEEE802.11 wireless interface). The digital camera 100 is connected to the communication interface 270.

[0049] A2. Learning model structure:

[0050] Figure 2 2 is an explanatory diagram of an example of the learning model 234. In this embodiment, the function of the learning model 234 is realized by the processor 210 executing the learning model 234 as a program module. The input image data 910 is input to the learning model 234. The input image data 910 is a label sheet 700 ( Figure 1) is optically read, and the input image 910i represented by the input image data 910 represents the label sheet 700. In this embodiment, a graphic 701 and a character string 702 are printed on the label sheet 700.

[0051] Learning model 234 extracts features of input image 910i of input image data 910 and reconstructs input image 910i based on the extracted features, thereby generating output image data 940 representing output image 940i. Output image 940i represents label slice 950 reconstructed from label slice 700 of input image 910i. Label slice 950 of output image 940i includes a graphic 951 corresponding to graphic 701 and a character string 952 corresponding to character string 702.

[0052] Input image 910i and output image 940i are rectangular images having two sides parallel to a first direction D1 and two sides parallel to a second direction D2 perpendicular to first direction D1. These images 910i and 940i are represented by the color values ​​of a plurality of pixels arranged in a matrix along first direction D1 and second direction D2. The width W represents the number of pixels in the first direction D1, and the height H represents the number of pixels in the second direction D2. In this embodiment, input image data 910 and output image data 940 are bitmap data representing the color values ​​of a plurality of pixels. Furthermore, in this embodiment, the color value of each pixel is represented by a grayscale value of R (red), G (green), and B (blue) ranging from 0 to 255.

[0053] Furthermore, the actual label sheet 700 may have defects such as a missing portion of an object (in this embodiment, the graphic 701 or the character string 702). In this embodiment, the learning model 234 is trained to generate output image data 940 representing a label sheet without defects, even when the label sheet 700 in the input image 910i has defects (details will be described later).

[0054] The learning model 234 has: an input layer L10 to which the input image data 910 is input; an intermediate layer L20 that processes the data from the input layer L10; and an output layer L30 that processes the data from the intermediate layer L20.

[0055] Figure 3 This is a block diagram showing the structure of input layer L10. Input layer L10 uses input image data 910 to generate feature data 920 representing the features of input image data 910 (i.e., features of input image 910i). This input layer L10 is also called an encoder. In this embodiment, input layer L10 includes three convolutional layers 311-313, two pooling layers 321 and 322, and an addition unit 331.

[0056] The three convolutional layers 311 to 313 are each input with input image data 910. The convolutional layers 311 to 313 each perform convolution processing and bias addition processing.

[0057] The convolution processing performed by the first convolutional layer 311 involves applying s first-type input filters FL11 of size "p1×q1×r" to the input data and calculating correlation values ​​representing the correlation between the input data and the first-type input filters FL11. The value p1 represents the size (number of pixels) of the first-type input filter FL11 in the first direction D1, the value q1 represents the size (number of pixels) of the first-type input filter FL11 in the second direction D2, and the value r represents the number of color components (3 (RGB) in this embodiment). Each first-type input filter FL11 includes "p1×q1×r" weights. When applying the first-type input filter FL11, a list of "p1×q1×r" grayscale values, representing the RGB grayscale values ​​of the "p1×q1" pixels corresponding to the position of the first-type input filter FL11, is obtained from the input image data 910. The inner product of this obtained list and the list of "p1×q1×r" weights for the first-type input filter FL11 is calculated. The offset addition process adds a bias, prepared for each first-type input filter FL11, to the inner product. "Inner product + offset" represents the correlation value. While sliding the first-type input filter FL11, the correlation value is calculated for each of the multiple positions of the first-type input filter FL11. Each first-type input filter FL11 generates bitmap data 911, which represents the correlation value at each of the multiple positions of the first-type input filter FL11. The image 911i represented by this bitmap data 911 is also called a feature map or convolution map. The position of the pixels in convolution map 911i represents the position of the first-type input filter FL11. In this embodiment, the convolution process is performed so that the size of convolution map 911i (i.e., the number of pixels in the first direction D1 and the number of pixels in the second direction D2) is the same as the size of the input image 910i (i.e., W×H). Therefore, the step size (the amount by which the filter is slid) is 1. Furthermore, pixels are padded around the perimeter of the input image 910i using zero padding.

[0058] The total number of filters s is also called the number of channels. When s first-type input filters FL11 are used, s bitmap data 911 are generated. In this embodiment, the number of channels s is 8.

[0059] The convolutional layers 311 - 313 respectively use the input filters FL11 - FL13 to generate the bitmap data 911 - 913. The sizes of the three input filters FL11 - FL13 are different. The second input filter FL12 is larger than the first input filter FL11, and the third input filter FL13 is larger than the second input filter FL12. The size comparison between two filters is carried out as follows. When the first filter is overlapped on the second filter in such a way that the second filter contains the whole of the first filter and the second filter contains a part that does not overlap with the first filter, it is judged that the second filter is larger than the first filter. Not limited to filters, the size comparison for two-dimensional regions such as images and windows is the same.

[0060] In this embodiment, the size of the first input filter FL11 is p1×q1, the size of the second input filter FL12 is p2×q2, and the size of the third input filter FL13 is p3×q3. And, p1 < p2 < p3, q1 < q2 < q3. For example, p1 = 1, p2 = २, p3 = 3, q1 = 1, q2 = 2, q3 = 3. The difference between the convolutional layers 311 - 313 is only that the sizes of the filters are different. For example, the number of channels s is the same for the three convolutional layers 311 - 313. The second convolutional layer 312 generates s bitmap data 912 (i.e., s convolutional maps 912i), and the third convolutional layer 313 generates s bitmap data 913 (i.e., s convolutional maps 913i). In addition, any one of the input filters FL11 - FL13 is smaller than the input image 910i of the input image data 910. The sizes of the input filters FL11 - FL13 can be set to larger values as the size of the input image 910i is larger.

[0061] It should be noted that in the original text, the symbol "२" in "p2 = २" seems incorrect. It might be a typo and should probably be "2". This translation assumes it's a typo and translates it as "2" in the English version. If this is not a typo, please provide more context or clarify the meaning.The value of each pixel in the convolution maps 911i-913i generated by the convolution layers 311-313 is significantly influenced by the shape of the portion of the object in the input image 910i contained in the input filters FL11-FL13. Thus, the convolution maps 911i-913i can represent the shape characteristics of the portion contained in the input filters FL11-FL13. The first convolution map 911i can represent the shape characteristics of the small portion contained in the small-sized first input filter FL11. The second convolution map 912i can represent the shape characteristics of the medium-sized portion contained in the medium-sized second input filter FL12. The third convolution map 913i can represent the shape characteristics of the large portion contained in the large-sized third input filter FL13. For example, the first convolution layer 311 can extract fine-grained shape features such as points, the second convolution layer 312 can extract the features of straight lines, and the third convolution layer 313 can extract the features of curves. In this way, convolutional layers 311-313 can extract shape features of the portion of the object in input image 910i that is contained within input filters FL11-FL13. The first convolution map 911i can extract high-resolution information. The second convolution map 912i can extract intermediate-resolution information. The third convolution map 913i can extract low-resolution information. Furthermore, the ability to extract fine shape features can also be translated as the ability to extract high-resolution information. Similarly, the ability to extract coarse shape features can also be translated as the ability to extract low-resolution information.

[0062] The two pooling layers 321 and 322 are each fed with the s bitmap data 911 from the first convolutional layer 311. The pooling layers 321 and 322 perform processing to determine a representative value using multiple values ​​of multiple pixels contained in a window. In this embodiment, the pooling layers 321 and 322 each perform so-called max pooling.

[0063] The first pooling layer 321 applies a first window WN21 of “pa×qa” to the input bitmap data 911. The value pa represents the size (number of pixels) in the first direction D1 of the first window WN21, and the value qa represents the size (number of pixels) in the second direction D2 of the first window WN21. The selection of the maximum value using the first window WN21 is performed at each of the multiple positions of the first window WN21 while sliding the first window WN21. The first pooling layer 321 generates bitmap data 914, and this bitmap data 914 represents the maximum value at each of the multiple positions of the first window WN21. The image 914i represented by this bitmap data 914 is also referred to as a feature map or a pooling map. The position of the pixel of the pooling map 914i represents the position of the first window WN21. In the present embodiment, max pooling is performed such that the size of the pooling map 914i to be output (i.e., the number of pixels in the first direction D1 and the number of pixels in the second direction D2) is the same as the size of the input convolution map 911i (i.e., W×H which is the size of the input image 910i). Therefore, the stride (the amount by which the window slides) is 1. In addition, pixels are supplemented by zero-padding around the input convolution map 911i.

[0064] The first pooling layer 321 is input with s bitmap data 911. The first pooling layer 321 generates s bitmap data 914 from the s bitmap data 911.

[0065] The second pooling layer 322 generates bitmap data 915 using a second window WN22. The second window WN22 is larger than the first window WN21. Specifically, the size of the first window WN21 is pa×qa, and the size of the second window WN22 is pb×qb. And, pa < pb, qa < qb. For example, pa = 3, pb = 5, qa = 3, qb = 5. The only difference between the pooling layers 321 and 322 is the difference in the size of the windows. The second pooling layer 322 generates s bitmap data 915 (i.e., s pooling maps 915i) from the s bitmap data 911 (i.e., s convolution maps 911i).

[0066] As described above, the convolution map 911i input to the pooling layers 321 and 322 represents the shape features of the object in the input image 910i. The pooling layers 321 and 322 determine the representative value (the maximum value in this embodiment) within the windows WN21 and WN22. Therefore, when the positional displacement of the feature portion within the convolution map 911i is within the range of windows WN21 and WN22, that is, when the positional displacement of the feature portion of the object in the input image 910i is within the range of windows WN21 and WN22, approximately the same representative value can be determined from windows WN21 and WN22. In this way, the pooling layers 321 and 322 can extract the shape features extracted by the first convolution layer 311 while allowing the positional displacement of the feature portion within the range of WN21 and WN22. The influence of the positional displacement of the feature portion on the input image 910i on the pooling maps 914i and 915i is mitigated when the positional displacement is within the range of windows WN21 and WN22. When the positional displacement exceeds the range of windows WN21 and WN22, pooling maps 914i and 915i change according to the positional displacement. In this way, pooling maps 914i and 915i represent information related to the positional displacement of the shape features extracted by the first convolutional layer 311 within the input image 910i.

[0067] The addition unit 331 generates feature data 920 using the five bitmap data 911-915 from the convolutional layers 311-313 and the pooling layers 321 and 322. Feature data 920 represents a map of the same size as the input maps 9911i-915i (also referred to as feature map 920i). In this embodiment, the addition unit 331 inputs the sum of the five values ​​of the five pixels at the same position in the five maps 911i-915i into an activation function and uses the calculated value of the activation function as the value of the pixel at the same position in the feature map 920i. In this embodiment, a so-called ReLU (Rectified Linear Unit) is used as the activation function. In addition, the addition unit 331 performs processing to generate one feature data 920 from the five bitmap data 911-915 for each channel. In this embodiment, since the number of channels s is 8, the addition unit 331 generates 8 feature data 920. The s feature data 920 are supplied to the intermediate layer L20 ( Figure 2 ).

[0068] Furthermore, the first convolutional layer 311 uses a computational parameter set 601 comprising "p1×q1×r×s" weights for the s first-type input filters FL11 and s biases corresponding to the s first-type input filters FL11. Similarly, the second convolutional layer 312 uses a computational parameter set 602 comprising "p1×q1×r×s" weights for the s second-type input filters FL12 and s biases. The third convolutional layer 313 uses a computational parameter set 603 comprising "p1×q1×r×s" weights for the s third-type output filters FL33 and s biases.

[0069] Figure 4 This is a block diagram showing the structure of intermediate layer L20. Intermediate layer L20 uses s feature data 920 to generate s intermediate data 930 for output. In this embodiment, intermediate layer L20 includes a first conversion unit 410, a first fully connected layer 415, a second fully connected layer 420, and a second conversion unit 430.

[0070] A feature map 920i represents the value of each of "W×H" pixels (hereinafter, the value corresponding to the pixel of the feature map is also referred to as the feature value). The s feature data 920 represent "W×H×s" feature values. The first fully connected layer 415 described later processes the "W×H×s" feature values ​​as a one-dimensional vector. The first conversion unit 410 converts the reference order of the "W×H×s" feature values ​​into the reference order of the elements of the one-dimensional vector based on the predetermined correspondence between the "W×H×s" feature values ​​and the elements of the one-dimensional vector. In addition, the first conversion unit 410 can be omitted, and the first fully connected layer 415 can refer to the "W×H×s" feature values ​​based on the above-mentioned correspondence.

[0071] The first fully connected layer 415 is similar to the fully connected layers used in typical neural networks. In this embodiment, the first fully connected layer 415 uses "W×H×s" eigenvalues ​​to calculate N eigenvalues. N can be an integer greater than 1 or greater than 2. N can be the same as "W×H×s" or an integer smaller than "W×H×s". Alternatively, N can be an integer greater than "W×H×s". The N eigenvalues ​​are also referred to as latent variables (hereinafter also referred to as latent variables 920z). The first fully connected layer 415 calculates the inner product of the vector consisting of the input "W×H×s" eigenvalues ​​and the weight vector consisting of the "W×H×s" weights. The calculated inner product is output as a single eigenvalue. Bias addition and activation function operations are omitted. A weight vector is prepared for each of the N intermediate values. The first fully connected layer 415 uses a calculation parameter set 608 containing N weight vectors (i.e., "W×H×s×N" weights).

[0072] The second fully connected layer 420 is the same as the fully connected layer used in a general neural network. In this embodiment, the second fully connected layer 420 uses the N eigenvalues ​​from the first fully connected layer 415 to calculate "W×H×s" intermediate values. The second fully connected layer 420 calculates the value obtained by adding a bias to the inner product of the vector composed of the input N eigenvalues ​​and the weight vector composed of the N weights (inner product + bias), and inputs the calculated value into the activation function. The calculated value of the activation function is used as one intermediate value. In this embodiment, the so-called ReLU is used as the activation function. A weight vector and a bias are prepared for each of the "W×H×s" intermediate values. The second fully connected layer 420 uses a calculation parameter set 604 containing "W×H×s" weight vectors (i.e., "N×W×H×s" weights) and "W×H×s" biases.

[0073] The output layer L30 ( Figure 2 ) "W×H×s" intermediate values ​​are processed as s maps. Similar to an image of size "W×H", one map is represented by the values ​​of "W×H" pixels. The second conversion unit 430 converts the reference order of the "W×H×s" intermediate values ​​into the reference order of the multiple pixels of the s maps based on the predetermined correspondence between the "W×H×s" intermediate values ​​and the multiple pixels of the s maps. Hereinafter, the s maps will also be referred to as intermediate maps 930i. In addition, the data representing the intermediate maps 930i will also be referred to as intermediate data 930. The intermediate layer L20 generates s intermediate data 930. The s intermediate data 930 are supplied to the output layer L30 ( Figure 2 The intermediate values ​​of intermediate mapping 930i are calculated using the N feature values ​​of latent variable 920z. Therefore, the intermediate values ​​represent the characteristics of the object in input image 910i. Alternatively, the second conversion unit 430 may be omitted, and the output layer L30 may refer to the "W×H×s" intermediate values ​​based on the above correspondence.

[0074] Figure 5 This is a block diagram showing the structure of the output layer L30. The output layer L30 uses s intermediate data 930 to generate output image data 940 representing the target image. This output layer L30 is also called a decoder. In this embodiment, the output layer L30 includes three transposed convolutional layers 511, 512, and 513 and an adder 520.

[0075] Each of the three transposed convolution layers 511-513 receives s intermediate data 930 as input. Each of the transposed convolution layers 511-513 performs transposed convolution and bias addition. Transposed convolution is a process that restores features extracted by convolution, and includes, for example, upscaling the input map and performing convolution using the upscaled map. Such transposed convolution is utilized as part of the decoder. In this embodiment, upscaling the map is omitted. The transposed convolution layers 511-513 then generate a single image from the s intermediate maps 930i through convolution.

[0076] The convolution processing of the first transposed convolution layer 511 is as follows: the first output filter FL31 of size "p1×q1×s" is applied to the input s intermediate data 930, thereby calculating the correlation value indicating the correlation between the input data and the first output filter FL31. The values ​​p1 and q1 are the size of the first output filter FL31 and the first convolution layer 311 ( Figure 3) have the same size as the first-type input filter FL11. As described later, the first transposed convolutional layer 511 uses r first-type output filters FL31. Each first-type output filter FL31 contains "p1×q1×s" weights. In the process of applying the first-type output filter FL31, the intermediate values ​​of the "p1×q1" pixels corresponding to the position of the first-type output filter FL31, that is, a list of "p1×q1×s" intermediate values, are obtained from each of the s intermediate data 930. The inner product of this obtained list and the list of "p1×q1×s" weights for the first-type output filter FL31 is calculated. The bias addition process adds a bias, which is prepared for each first-type output filter FL31, to the inner product. "Inner product + bias" represents the correlation value. While sliding the first-type output filter FL31, the correlation value is calculated for each of the multiple positions of the first-type output filter FL31. A first-type output filter FL31 generates bitmap data 931, which represents the correlation value at each of multiple positions of the first-type output filter FL31. As described later, the correlation value is calculated as a value that restores the feature represented by the median value of multiple pixels corresponding to the first-type output filter FL31. Hereinafter, image 931i represented by bitmap data 931 will be referred to as a partially restored image. The positions of the pixels in partially restored image 931i represent the positions of the first-type output filter FL31. In this embodiment, convolution processing is performed so that the size of partially restored image 931i (i.e., the number of pixels in the first direction D1 and the number of pixels in the second direction D2) is the same as the size of intermediate map 930i (i.e., W×H). Therefore, the step size (the amount by which the filter is slid) is 1. Furthermore, pixels around intermediate map 930i are padded with zeros.

[0077] In this embodiment, similar to the pixels of input image 910i, each pixel in partially restored image 931i is associated with the grayscale values ​​of r color components. In this embodiment, the value r is 3, and each pixel is associated with three grayscale values ​​of RGB. The first transposed convolutional layer 511 calculates r values ​​for each pixel using r first-type output filters FL31. The bitmap data 931 generated by the first transposed convolutional layer 511 is r-colored bitmap data.

[0078] Transposed convolution layers 511-513 use output filters FL31-FL33 to generate bitmap data 931-933, respectively. The three output filters FL31-FL33 have different sizes. The size of the second output filter FL32 is the same as the size of the second input filter FL12 (p2×q2), and the size of the third output filter FL33 is the same as the size of the third input filter FL13 (p3×q3). The only difference between the transposed convolution layers 511-513 is the filter size. The second transposed convolution layer 512 generates r-colored bitmap data 932 (i.e., an r-colored partially restored image 932i), and the third transposed convolution layer 513 generates r-colored bitmap data 933 (i.e., an r-colored partially restored image 933i). Furthermore, each output filter FL31-FL33 is smaller than the output image 940i of the output image data 940.

[0079] The values ​​of each pixel of the partially restored images 931i-933i generated by the transposed convolutional layers 511-513 are greatly affected by the intermediate values ​​of the parts of the intermediate map 930i included in the output filters FL31-FL33. Figure 4 As described, the intermediate values ​​of the intermediate map 930i represent the features (including shape features) of the object in the input image 910i. Therefore, the partially restored images 931i-931i can represent the shape features represented by the portions corresponding to the output filters FL31-FL33. The first partially restored image 931i can represent the small-scale shape features represented by the portion corresponding to the small-size first-type output filter FL31. The second partially restored image 932i can represent the medium-scale shape features represented by the portion corresponding to the medium-size second-type output filter FL32. The third partially restored image 933i can represent the large-scale shape features represented by the portion corresponding to the large-size third-type output filter FL33. For example, the first transposed convolutional layer 511 can restore fine shape features such as points, the second transposed convolutional layer 512 can restore the features of straight lines, and the third transposed convolutional layer 513 can restore the features of curves. In this way, the transposed convolutional layers 511-513 can restore shape features using the portions of the intermediate data 930 corresponding to the output filters FL31-FL33. The first partially restored image 931i can restore high-resolution information. The second partially restored image 932i can restore intermediate-resolution information. The third partially restored image 933i can restore low-resolution information. Furthermore, the ability to restore fine shape features can also be translated as the ability to restore high-resolution information. Furthermore, the ability to restore coarse shape features can also be translated as the ability to restore low-resolution information.

[0080] The adder 520 uses the three bitmap data 931-933 from the transposed convolutional layers 511-513 to generate output image data 940. Output image data 940 represents an output image 940i of the same size as the input partially restored images 931i-932i. In this embodiment, the adder 520 inputs the sum of the three values ​​of the three pixels at the same position in the three partially restored images 931i-933i into an activation function and uses the calculated value of the activation function as the value of the pixel at the same position in the output image 940i. In this embodiment, a so-called Sigmoid function is used as the activation function. Furthermore, the adder 520 calculates pixel values ​​for each r color component. In this embodiment, the three red (R) grayscale values ​​of the three pixels at the same position in the three partially restored images 931i-933i are used to calculate the red (R) grayscale value of the pixel at the same position in the output image 940i. The grayscale values ​​of green G and blue B are calculated in the same manner. The generated output image 940i can represent the restored shape features of various sizes, that is, the label pieces.

[0081] Furthermore, the first transposed convolutional layer 511 uses a computational parameter set 605 comprising "p1×q1×s×r" weights for the r first-type output filters FL31 and r biases corresponding to the r first-type output filters FL31. Similarly, the second transposed convolutional layer 512 uses a computational parameter set 606 comprising "p1×q1×s×r" weights for the r second-type output filters FL32 and r biases. The third transposed convolutional layer 513 uses a computational parameter set 607 comprising "p1×q1×s×r" weights for the r third-type output filters FL33 and r biases.

[0082] A3. Training Process:

[0083] Figure 6 2 is a flowchart showing an example of a training process for generating a learning model 234. In this embodiment, the learning model 234 is trained to generate image data of a label sheet without defects when the image data of the label sheet 700 is input. Through training, the above-mentioned operation parameter set 601-608 ( Figure 3-Figure 5 ) is used to adjust multiple operation parameters. The training is performed by the data processing device 200 ( Figure 1 In this embodiment, the processor 210 performs the training process according to the first program 232.

[0084] In S110 , the processor 210 initializes a plurality of operation parameters (including the operation parameter sets 601 - 608 ) of the learning model 234 . For example, each operation parameter is set to a random value.

[0085] In S115, the processor 210 obtains reference image data. The reference image data is image data of a non-defective label sheet (also referred to as a reference label sheet). Figure 7 (A) is an explanatory diagram of a reference label sheet 800. A graphic 801 and a character string 802 are printed on the reference label sheet 800. Figure 7 B is an explanatory diagram of an example of reference image data. Reference image data 810 is image data obtained by photographing the reference label sheet 800. In this embodiment, the operator uses the digital camera 100 ( Figure 1 ). The processor 210 then provides a shooting instruction to the digital camera 100. The digital camera 100 captures the reference label sheet 800 in accordance with the instruction, generating reference image data. The processor 210 obtains the reference image data from the digital camera 100 and stores the reference image data in the non-volatile storage device 230. Alternatively, the processor 210 may perform a cropping process to cut out the portion representing the reference label sheet 800 from the image data obtained from the digital camera 100, thereby obtaining reference image data 810.

[0086] In S120, the processor 210 uses the reference image data 810 to generate a plurality of learning input image data, and stores the generated plurality of learning input image data in the non-volatile storage device 230. In this embodiment, a plurality of learning input image data are generated by changing the position or orientation of the reference label sheet 800 within the reference image 810i. Among the plurality of learning input image data, one or more of the four parameters of movement direction, movement amount, rotation direction, and rotation angle are different from each other. These parameters are changed within the range of the offset that may be generated when the image data of the label sheet is obtained by optically reading the label sheet. In this way, the process of generating a plurality of image data for learning by processing the existing image data is also called "data augmentation". In addition, it is not limited to the change of position or orientation, and the color can be changed, noise can be added, or smoothing can be performed. In any case, the plurality of learning input image data represents a defect-free reference label sheet 800.

[0087] At S130, processor 210 selects V (V is an integer greater than or equal to 1) pieces of learning input image data from the plurality of learning input image data, inputs the V pieces of learning input image data into learning model 234, and generates V pieces of output image data. Alternatively, unused pieces of learning input image data may be selected as the V pieces of learning input image data. Alternatively, the V pieces of learning input image data may be randomly selected from the plurality of learning input image data.

[0088] At S135, processor 210 calculates an error value representing the difference between the V learning input image data input to learning model 234 and the output image data corresponding to the learning input image data. The error value is calculated based on a predetermined loss function. For example, mean squared error (MSE) is used to calculate the error value. The smaller the difference between the learning input image data and the output image data, the smaller the error value.

[0089] At S140, processor 210 uses the V error values ​​to adjust multiple computational parameters (including computational parameter sets 601-608) of learning model 234. Specifically, processor 210 adjusts the multiple computational parameters according to a predetermined algorithm so that the error values ​​are reduced, that is, the difference between the learned input image data and the output image data is reduced. For example, an algorithm utilizing error backpropagation and gradient descent is used as the algorithm.

[0090] In S150, the processor 210 determines whether the training is completed. The condition for the completion of the training may be, for example, the condition that all the error values ​​calculated in S135 are smaller than a predetermined error threshold. Alternatively, the condition for the completion of the training may be the condition that a completion instruction from the operator is input. The operator confirms whether the image represented by the output image data can fully reproduce the image represented by the corresponding learning input image data. Based on the confirmation result, the operator inputs a completion instruction or a continuation instruction of the training via the operation unit 250. In addition, the condition for the completion of the training may be the condition that the number of times the processing of S130-S140 is performed is greater than a predetermined number threshold.

[0091] If it is determined that the training is not completed (S150: No), the processor 210 transfers to S130. If it is determined that the training is completed (S150: Yes), in S160, the learning model 234 with the adjusted operation parameters (i.e., the learned learning model 234) is stored in the non-volatile storage device 230. Then, Figure 6 The processing is completed. The learned model 234 is used in the difference data generation process described later.

[0092] The learned model 234 is configured to enable the computer to perform processing using the adjusted parameters. Figure 2-Figure 5 As described above, the learning model 234 extracts features (including shape features) of an object (here, a graphic or character string of a label piece) in an input image, and restores the image of the object based on the extracted features. Figure 7 (C) is an explanatory diagram of an example of output image data generated by the learned model 234. The output image data 840 represents the output image data generated by Figure 7 (B) is an example of image data generated by inputting the image data 810 into the learned model 234. The output image 840i of the output image data 840 represents the same as Figure 7 (B) Label piece 800 (including graphic 801 and character string 802) is nearly identical to reference label piece 800 in image 810i. As described above, learning model 234 is trained using learning input image data of a non-defective object. In this case, regardless of whether the object in the input image has defects, the trained learning model 234 generates output image data representing a non-defective object.

[0093] A4. Differential data generation process:

[0094] Figure 8 1 is a flowchart showing an example of differential data generation processing. The differential data can be used to determine whether the sample of the label sheet has a defect. The differential data is generated by the data processing device 200 ( Figure 1 In this embodiment, the processor 210 executes the differential data generation process according to the second program 233.

[0095] In S210, the processor 210 obtains target input image data as image data of a sample of the label sheet, and stores the obtained target input image data in the storage device 215 (eg, the non-volatile storage device 230). Figure 6 Similarly to the process of S115 , the target input image data 710 is acquired using the digital camera 100 .

[0096] Figure 9 This is an explanatory diagram of image processing. The diagram shows an example of multiple images 710i, 740i, 710bi, 740bi, 760di, 740bi, 770i, 770bi, and 780i being processed in the differential data generation process. These images are arranged in the order of processing from top to bottom in the diagram.

[0097] exist Figure 9 The upper left corner of FIG. 1 shows a target input image 710i of the target input image data 710. The target input image 710i represents a label sheet 700x having a defect (also referred to as input label sheet 700x). Specifically, the label sheet 700x has a missing portion 701x of the graphic 701 and a missing portion 702x of the character string 702. Furthermore, the target input image 710i includes noise 710n. The noise 710n may be generated due to various reasons, such as data processing in the digital camera 100.

[0098] In S220( Figure 8), the processor 210 generates object output image data by inputting the object input image data into the learned model 234. Figure 9 The upper right corner of the target output image data 740 shows the target output image 740i. The target output image 740i represents a non-defective label sheet 750 (also referred to as the output label sheet 750). The position and orientation of the output label sheet 750 in the target output image 740i are substantially the same as the position and orientation of the input label sheet 700x in the target input image 710i. The graphic 751 and the character string 752 of the output label sheet 750 are substantially the same as those of the reference label sheet 800 ( Figure 7 (A)) The graphic 801 and the character string 802 are substantially the same. The target output image 740i includes noise 740n. The noise 740n may be generated due to various reasons such as the calculation of the learning model 234.

[0099] In S230( Figure 8 ), the processor 210 performs smoothing processing on the object input image data and the object output image data to generate smoothed input image data and smoothed output image data (smoothing processing is also called blurring processing). In this embodiment, the smoothing processing is a processing using an average filter. Figure 9 The second layer from the top shows smoothed input image 710bi of smoothed input image data 710b and smoothed output image 740bi of smoothed output image data 740b. As shown in the figure, noise 710n and 740n have been removed. Furthermore, the smoothing filter can be a median filter, Gaussian filter, or other various filters. Furthermore, smoothing processing is not limited to processing using a smoothing filter; various processes can also be used to make the color value (also called pixel value) of a pixel in an image change more gradually with respect to changes in its position on the image.

[0100] In S240( Figure 8 ), the processor 210 performs edge extraction processing on the smoothed output image data to generate edge image data. In this embodiment, the processor 210 calculates the brightness value based on the pixel value (RGB value), applies the well-known Sobel filter to the brightness value, and calculates the edge amount of each pixel. Then, the processor 210 extracts pixels with an edge amount greater than a predetermined threshold as edge pixels. Figure 9The right side of the third layer from the top shows an edge image 740 of the edge image data 740e. The edge image 740 represents the edge portion 750e (e.g., outline) of the output label sheet 750, the edge portion 751e (e.g., outline or pattern of the graphic, etc.) of the graphic 751, and the edge portion 752e (e.g., outline) of the character string 752. In this embodiment, the edge image data 740e is binary bitmap data. A zero pixel value represents an edge pixel, and a 1 pixel value represents a non-edge pixel. In addition, the edge extraction filter can be various other filters such as a Prewitt filter and a Roberts filter. In addition, the edge extraction process is not limited to the process using the edge extraction filter, and can also be various processes for extracting pixels of a portion where the change in pixel value on the image is sharp relative to the change in position on the image.

[0101] In S250( Figure 8 ), the processor 210 generates differential image data representing the difference between the smoothed input image data and the smoothed output image data. In the present embodiment, the differential image data represents a differential image showing the difference between the smoothed input image 710bi and the smoothed output image 740bi. The processor 210 uses the difference between the two color values ​​of the two pixels at the same position of the smoothed input image 710b and the smoothed output image 740bi as the color value of the pixel at the same position of the differential image. Specifically, the pixel value of the differential image is set to the absolute value of the difference between the brightness value calculated based on the pixel value of RGB of the smoothed input image data and the brightness value calculated based on the pixel value of RGB of the smoothed output image data. The zero pixel value of the differential image indicates that the difference between the two image data is zero. Furthermore, the larger the pixel value of the differential image, the larger the difference between the two image data. In Figure 9 The third layer from the top shows a differential image 760di of the differential image data 760d on the left side. As described above, the smoothed input image 710bi includes a missing portion 701x of the graphic 701 and a missing portion 702x of the character string 702. On the other hand, the smoothed output image 740bi does not have these missing portions. Therefore, the differential image 760di shows a missing portion 701z corresponding to the missing portion 701x of the graphic 701 and a missing portion 702z corresponding to the missing portion 702x of the character string 702. Furthermore, the positions of the label pieces 700 and 750 within the image may differ between the smoothed input image 710bi and the smoothed output image 740bi. Therefore, the differential image 760di can represent the edge portion 700d of the label piece, the edge portion 701d of the graphic, and the edge portion 702d of the character string. Furthermore, the pixel values ​​of the differential image can also be represented by other color components. For example, the pixel values ​​of the difference image may represent grayscale values ​​of three colors: the absolute value of the difference of red R, the absolute value of the difference of green G, and the absolute value of the difference of blue B.

[0102] In S260( Figure 8 ), the processor 210 uses the edge image data 740e to correct the differential image data 760d. Specifically, a process is performed to extract non-edge portions different from edge portions from the differential image 760di. In this embodiment, the processor 210 refers to the edge image data 740e and sets the pixel values ​​of the edge portions in the differential image data 760d to zero, thereby generating corrected differential image data. Figure 9 The fourth layer from the top shows a corrected differential image 770i of the corrected differential image data 770 on the left side. Corrected differential image 770i shows missing portions 701z and 702z. Edge portions 700d, 701d, and 702d included in differential image 760di have been deleted. Furthermore, in this embodiment, a zero pixel value in edge image data 740e represents an edge pixel, and a one pixel value represents a non-edge pixel. Therefore, processor 210 can generate corrected differential image data 770 by multiplying the pixel value of a pixel in differential image data 760d by the pixel value of a pixel at the same position in edge image data 740e.

[0103] In S270( Figure 8 ), the processor 210 binarizes the corrected differential image data 770 to generate binary differential image data. In this embodiment, the processor 210 performs binarization by comparing the pixel values ​​of the corrected differential image data 770 with a predetermined binarization threshold. Figure 9 The left side of the bottommost layer shows a binary difference image 770bi of the binary difference image data 770b. Binary difference image 770bi shows difference portions 701zb and 702zb corresponding to the missing portions 701z and 702z of the corrected difference image data 770. Various methods for distinguishing large and small values ​​can be used for binarization. For example, so-called Otsu binarization can be used.

[0104] In S280( Figure 8 ), the processor 210 synthesizes the target output image data 740 and the binary difference image data 770b to generate synthesized image data. In this embodiment, the processor 210 generates the synthesized image data by changing the color of the difference portion in the target output image to a predetermined specific color (such as white, red, etc.). Figure 9 The right side of the bottommost layer shows a composite image 780i of the composite image data 780. The difference parts 701zb and 702zb in the output label sheet 750 are shown in a specific color.

[0105] In S290( Figure 8), the processor 210 stores image data representing the difference between the target input image and the target output image in the storage device 215 (e.g., the non-volatile storage device 230). In this embodiment, the difference image data (S250), the corrected difference image data (S260), the binary difference image data (S270), and the synthesized image data (S280) represent the difference. These four image data are all obtained using the difference image data (S250). The image data stored in the storage device in S290 can be one or more image data arbitrarily selected from the four aforementioned image data. In this embodiment, the processor 210 stores the binary difference image data and the synthesized image data in the non-volatile storage device 230.

[0106] Through the above process, Figure 8 The processing ends. The image data stored in the storage device in S290 can be used in various processes. For example, the processor 210 can display the synthesized image 780i ( Figure 9 A viewer of the composite image 780i can easily identify the different portions 701zb and 702zb (particularly their positions and shapes). Furthermore, the processor 210 can also analyze the binary difference image data 770b to determine whether the input label sheet 700x has a defect. For example, if the size (e.g., the number of pixels) of a continuous difference portion exceeds a predetermined threshold, the input label sheet can be determined to have a defect.

[0107] As mentioned above, Figure 2 The learning model 234 is a machine learning model that generates output image data 940 corresponding to the input image data 910 including the input image 910i of the label piece 700. Figure 8 In the processing, the learned model 234 is used. Figure 2-Figure 5 As described, the learning model 234 is a model that performs operations using a plurality of operation parameters (including the operation parameter sets 601-608) for the input image data 910. Figure 3 As described above, the shape features of the input object (the graphic 701 and the character string 702 of the label sheet 700 in this embodiment) included in the input image 910i of the input image data 910 are extracted by calculation based on the learning model 234. Figure 4 、 Figure 5 As described above, the output image data 940 including the output image 940 i of the output object (the label sheet 950 in this embodiment) based on the extracted shape features is generated by the calculation based on the learning model 234 .

[0108] like Figure 2As shown, the learning model 234 includes an input layer L10 that performs operations using input image data 910 , and an output layer L30 that is provided downstream of the input layer L10 and generates output image data 940 .

[0109] like Figure 3 As shown, input layer L10 includes convolutional layers 311-313, pooling layers 321 and 322, and an addition unit 331. First convolutional layer 311 is an example of a first extraction unit that extracts shape features of portions of an input object (e.g., graphic 701, character string 702) serving as the input image 910i and contained within the range of a first input filter FL11, which is smaller than the input image 910i. Second convolutional layer 312 is provided in parallel with first convolutional layer 311. Second convolutional layer 312 is an example of a second extraction unit that extracts shape features of portions of an input object contained within the range of a second input filter FL12, which is smaller than the input image 910i and larger than the range of the first input filter FL11.

[0110] The pooling layers 321 and 322 generate bitmap data 914 and 915 of pooling maps 914i and 915i by performing operations using the data output from the first convolutional layer 311. Figure 3 As described above, pooling maps 914i and 915i can be affected by the positional displacement of the shape features extracted by the first convolutional layer 311 within the input image 910i. When the positional displacement is within the range of windows WN21 and WN22, the influence is mitigated. When the positional displacement exceeds the range of windows WN21 and WN22, pooling maps 914i and 915i can change in accordance with the positional displacement. Thus, pooling maps 914i and 915i are examples of displacement information related to positional displacement. Furthermore, pooling layers 321 and 322 are examples of displacement information acquisition units that acquire displacement information.

[0111] The addition unit 331 adds a plurality of data including the data output from the first convolutional layer 311 , the data output from the second convolutional layer 312 , and the data output from the pooling layers 321 and 322 .

[0112] With the above structure, the first convolutional layer 311 outputs data corresponding to relatively fine shape features, while the second convolutional layer 312 outputs data corresponding to relatively coarse shape features. For example, the first convolutional layer 311 can extract fine shape features, while the second convolutional layer 312 can extract line features. The adder 331 of the input layer L10 then adds these data together.

[0113] Therefore, the learning model 234 can generate output image data 940 representing an output object having not only a relatively coarse shape but also a relatively fine shape. Assume that, rather than having multiple convolutional layers using filters of different sizes arranged in parallel in the input layer, multiple processing layers (e.g., multiple convolutional layers) are connected in series. In this case, while relatively coarse shape features can be extracted, it is difficult to extract relatively fine shape features. The above configuration can suppress this problem.

[0114] Furthermore, the data output from pooling layers 321 and 322 is added to the addition operation of addition unit 331. Pooling layers 321 and 322 use the data corresponding to relatively fine shape features from first convolutional layer 311 to generate bitmap data 914 and 915. The effects of positional displacement within windows WN21 and WN22 are mitigated in data 914 and 915 output from pooling layers 321 and 322. This improves robustness against positional displacement of relatively fine shape features within the input image.

[0115] In addition, if you use Figure 5 As described, the output layer L30 includes transposed convolutional layers 511-513 and an adder 520. The first transposed convolutional layer 511 is an example of a first restoration unit that restores shape features using a portion of the intermediate data 930 being processed that corresponds to the range of the first output filter FL31, which is smaller than the output image 940i. The second transposed convolutional layer 512 is provided in parallel with the first transposed convolutional layer 511. The second transposed convolutional layer 512 is an example of a second restoration unit that restores shape features using a portion of the intermediate data 930 being processed that corresponds to the range of the second output filter FL32, which is smaller than the output image 940i and larger than the range of the first output filter FL31.

[0116] The first transposed convolutional layer 511 restores relatively fine shape features, while the second transposed convolutional layer 512 restores relatively coarse shape features. Therefore, the learning model 234 can generate output image data 940 representing an output object having both relatively coarse and fine shapes.

[0117] As described above, the input layer L10 includes the first convolutional layer 311, which serves as an example of a first extraction unit; the second convolutional layer 312, which serves as an example of a second extraction unit; and the pooling layers 321 and 322, which serve as examples of displacement information acquisition units. Furthermore, the output layer L30 does not include a pooling layer, but instead includes the first transposed convolutional layer 511, which serves as an example of a first restoration unit, and the second transposed convolutional layer 512, which serves as an example of a second restoration unit. Because the input layer L10 includes the pooling layers 321 and 322, the effect of positional displacement of shape features within the input image 910i on the operations performed by the output layer L30 is mitigated. Therefore, the output layer L30, which does not include a pooling layer but includes the first transposed convolutional layer 511 and the second transposed convolutional layer 512, can generate output image data 940 that represents an output object having both a relatively coarse shape and a relatively fine shape.

[0118] In addition, if you use Figure 3 As described, the input layer L10 includes a first pooling layer 321 and a second pooling layer 322. The first pooling layer 321 is an example of a first displacement information acquisition unit that acquires bitmap data 914 related to positional displacement within a first window WN21 that is smaller than the input image 910i. The second pooling layer 322 is provided in parallel with the first pooling layer 321. The second pooling layer 322 is an example of a second displacement information acquisition unit that acquires bitmap data 915 related to positional displacement within a second window WN22 that is smaller than the input image 910i and larger than the first window WN21.

[0119] Furthermore, the addition unit 331 adds together multiple data, including the data output from the first convolutional layer 311, the second convolutional layer 312, the first pooling layer 321, and the second pooling layer 322. In this structure, the first pooling layer 321 obtains the first bitmap data 914 associated with relatively small positional displacements, and the second pooling layer 322 obtains the second bitmap data 915 associated with relatively large positional displacements. Furthermore, the addition unit 331 of the input layer L10 adds these data together, thereby improving robustness against relatively small and relatively large positional displacements.

[0120] In addition, the input layer L10 ( Figure 3) includes a third convolutional layer 313 provided in parallel with the first convolutional layer 311 and the second convolutional layer 312. The third convolutional layer 313 is an example of a third extraction unit that extracts shape features of a portion of the input object that is contained within the range of the third input filter FL13, which is smaller than the input image 910i and larger than the range of the second input filter FL12. Furthermore, the addition unit 331 adds together a plurality of data items, including the data output from the first convolutional layer 311, the second convolutional layer 312, the third convolutional layer 313, the first pooling layer 321, and the second pooling layer 322. In this way, three types of shape features corresponding to three levels of detail are extracted and the three types of data corresponding to the three features are added together, thereby generating output image data 940 representing the output object having the three types of shape features corresponding to the three levels of detail.

[0121] In addition, as in Figure 6 As described in S140 of , the plurality of operation parameters (including the operation parameter sets 601-608) of the learning model 234 are adjusted through training so that the difference between the learning input image data and the output image data generated by inputting the learning input image data into the learning model 234 becomes smaller. As described in S120, the learning input image data is generated using the reference image data 810 of the reference image 810i including the reference label piece 800. The plurality of learning input image data are image data of the reference image including the reference label piece 800. Therefore, as described using Figure 8 、 Figure 9 As described above, by inputting the target input image data 710 including the target input image 710i of the input label sheet 700x into the learning model 234, it is possible to generate the target output image data 740 including the target output image 740i of the output label sheet 750 that is less different from the reference label sheet 800. Such target output image data 740 can be used to detect defects in the target input image 710i (here, the input label sheet 700x).

[0122] In addition, when using Figure 8 、 Figure 9In the described process, the processor 210 performs the following processing using the already learned learning model 234. At S210, the processor 210 obtains object input image data 710 including an input object's object input image 710i. At S220, the processor 210 inputs the object input image data 710 into the already learned learning model 234, generating object output image data 740 corresponding to the object input image data 710. At S230 and S250, the processor 210 generates difference image data 760d representing a difference image 760di between the object input image 710i and the object output image 740i. The learning model 234 can generate object output image data 740 including the object output image 740i of the output label sheet 750, which is based on the shape features extracted from the input layer L10 and has a small difference from the reference label sheet 800. Consequently, the processor 210 can generate difference image data 760d representing the difference between the input label sheet 700x and the reference label sheet 800.

[0123] In addition, as in S230, S250 ( Figure 8 ), processor 210 performs smoothing processing on target input image data 710 and target output image data 740 to obtain smoothed input image data 710b and smoothed output image data 740b, and generates difference image data 760d using smoothed input image data 710b and smoothed output image data 740b. Therefore, noise included in difference image data 760d can be reduced.

[0124] In addition, as in S240, S260 ( Figure 8 ), processor 210 performs edge extraction processing on difference image data 760d, extracting non-edge portions different from edge portions from difference image 760di, thereby correcting the difference image data (S260). This edge extraction processing extracts edge portions (S240) within smoothed output image 740bi of smoothed output image data 740b. Due to the misalignment of the object between target input image 710i and target output image 740i, difference image 760di may contain edges. With this configuration, corrected difference image data 770 can appropriately represent differences from edges caused by the misalignment between target input image 710i and target output image 740i.

[0125] B. Modification:

[0126] (1) The input layer of the learning model is not limited to the convolutional layers 311-313 ( Figure 3), it may also include various operation units (also referred to as extraction units) for extracting shape features of an input object contained in an input image. The extraction unit may be configured to extract shape features of a portion of the input object contained in the input range using the respective pixel values ​​of one or more pixels in the input image contained in a portion that is smaller than the input image. Preferably, the input layer includes a plurality of extraction units that use input ranges of different sizes. Thus, the input layer can extract shape features of different sizes. In other words, the input layer can extract information of different resolutions. The total number of extraction units may be any number greater than 2.

[0127] Furthermore, the larger the input image size (number of pixels), the greater the computational effort required for the learning model. Similarly, the greater the number of extractors, the greater the computational effort required for the learning model. Therefore, when the input image size is large, it is preferable to have a smaller number of extractors.

[0128] (2) The pooling layers 321 and 322 included in the input layer L10 of the learning model 234 are not limited to performing maximum pooling processing, and can also perform various processing that uses one or more pixels within the windows WN21 and WN22 to determine representative values. For example, the pooling layers 321 and 322 can perform average pooling processing. In addition, the bitmap data 912 from the second convolutional layer 312 can be input to the pooling layers 321 and 322. This bitmap data 912 represents detailed shape features compared to the bitmap data 913 from the third convolutional layer 313. Therefore, it is possible to improve robustness against displacement of the position of relatively detailed shape features within the input image.

[0129] The data generated by the pooling layers 321 and 322 is an example of displacement information related to the displacement of the position of the shape feature extracted by the extraction unit (e.g., the convolutional layers 311-313) within the input image. The displacement information may be various information that changes according to the displacement of the position. The displacement information may be determined such that when the displacement of the position of the object on the input image is within a predetermined displacement range, the influence of the position displacement on the displacement information becomes smaller, and when the position displacement exceeds the displacement range, the influence of the position displacement on the displacement information becomes larger. That is, the displacement information may be calculated as follows: when the position difference of the object between the two input images is within the displacement range, the difference between the two displacement information obtained from the two input images becomes smaller, and when the position difference exceeds the displacement range, the difference between the two displacement information obtained from the two input images becomes larger. The input layer is not limited to having a pooling layer, and may also have various operation units (referred to as displacement information acquisition units) for obtaining displacement information.

[0130] The input layer preferably has a plurality of displacement information acquisition units that use displacement ranges of different sizes. This makes it possible to improve robustness against displacements of different sizes. In addition, the total number of displacement information acquisition units may be any number greater than 1. When the size of the input image is large, it is preferred that the total number of displacement information acquisition units is small. In either case, it is preferred that the addition unit of the input layer adds data from each of the plurality of extraction units and one or more displacement information acquisition units. This allows the input layer to extract shape features of parts of various sizes and improve robustness against misalignment of objects within the input image. In addition, more than one displacement information acquisition unit may use the same data from the same extraction unit to acquire displacement information.

[0131] Furthermore, when the input layer's multiple extraction units include a first extraction unit that uses a first input range and a second extraction unit that uses a second input range larger than the first input range, it is preferred that at least one displacement information acquisition unit use data from the first extraction unit. This improves robustness against positional displacements of relatively fine-grained shape features. In this case, the input layer need not include a displacement information acquisition unit that uses data from the second extraction unit. This simplifies the input layer's structure.

[0132] (3) The output layer of the learning model is not limited to having transposed convolutional layers 511-513 ( Figure 5 ), and may also have various operation units (also called restoration units) for restoring shape features. The restoration unit may process the processing object data representing the features (including shape features) of the object as image data. Furthermore, the restoration unit may be configured to restore the shape features using the pixel values ​​of one or more pixels in the portion of the image represented by the processing object data that is included in an output range that is smaller than the output image. The output layer preferably has a plurality of restoration units that use output ranges of different sizes. Thus, the output layer can restore shape features of different sizes. In other words, the output layer can restore information of different resolutions. Here, preferably, a plurality of restoration units are provided in parallel to process the same processing object data, and the output layer has an addition operation unit that adds the data output from each of the plurality of restoration units. Furthermore, the data output from the addition operation unit may be used as output image data. Thus, shape features of different sizes can be appropriately restored. Here, the output layer may not have a pooling layer (and therefore may not have a displacement information acquisition unit).

[0133] In addition, the total number of restoration units can be any number greater than 1, preferably greater than 2. When the size of the input image is large, the total number of restoration units is preferably small. The size of the output range of the restoration unit of the output layer can be different from the size of the input range of the extraction unit of the input layer. The total number of restoration units of the output layer can be different from the total number of extraction units of the input layer. However, it is preferred that the input layer has U (U is an integer greater than 2) extraction units using input ranges of different sizes, the output layer has U restoration units using output ranges of different sizes, and the sizes of the U input ranges are respectively the same as the sizes of the U output ranges. In this way, it is possible to appropriately extract and restore a variety of shape features of multiple sizes that are different from each other.

[0134] (4) Figure 2-Figure 5 In the learning model 234, the fully connected layers 415 and 420 ( Figure 4 ) from the input layer L10( Figure 3 ) of the adding unit 331 to the output layer L30 ( Figure 5 ) (specifically, the transposed convolutional layers 511-513). Fully connected layers 415 and 420 can transmit information while minimizing information loss. Therefore, information representing shape features of various sizes extracted by input layer L10 is transmitted to output layer L30. As a result, output layer L30 can restore shape features of various sizes.

[0135] The structure of the intermediate layer connecting the input layer's addition unit and the output layer's restoration unit is not limited to fully connected layers 415 and 420 and can be a variety of structures. It is preferable that the intermediate layer transmit information while minimizing information loss. For example, the intermediate layer can include multiple fully connected layers connected in series. Alternatively, the intermediate layer can be omitted. In this case, data from the input layer's addition unit is input to the output layer's restoration unit.

[0136] (5) The structure of the learning model can be any other structure instead of the above structure. The number of channels s can be any number greater than 1. The more channels s is, the more capable it is to extract and restore various shape features. The fewer channels s is, the easier it is to train the learning model. The number r of color components can be any number greater than 1. The color components can be any color components such as grayscale, RGB, YCbCr, CMYK, etc. When r = 1, the learning model can process grayscale bitmap data.

[0137] In addition, the learning model can be replaced by various models that extract and restore the shape features of the object in the input image. Figure 2-Figure 5For example, the learning model may be a model called Variational Autoencoder (VAE). VAE is a model that compresses input image data into parameters (mean and variance) of a multidimensional normal distribution. When VAE is used, more detailed shape features can be extracted and restored. When VAE is used, the structure of the intermediate layer can be various structures that process parameters of a normal distribution, similar to the intermediate layer of the well-known VAE. For example, Figure 4 The N (N is an even number) eigenvalues ​​output by the first fully connected layer 415 can represent the parameters of the normal distribution.

[0138] In either case, the learning model is trained using a method suitable for the learning model. The multiple operating parameters of the learning model are preferably adjusted so that the difference between the reference image data of the reference image containing the reference object (e.g., the reference label sheet 800) and the output image data generated by inputting the reference image data into the learning model becomes smaller. In addition, it is preferred to use multiple reference image data of multiple reference images containing the reference object to adjust the multiple operating parameters. Here, it is preferred that at least one of the position and orientation of the reference object is different between the multiple reference images. Thus, the learned model can appropriately process various input image data.

[0139] (6) The differential data generation process can be replaced by various other processes Figure 8 processing. For example, the smoothing process (S230) can be omitted. In this case, in S250, differential image data representing the difference between the object input image data 710 and the object output image data 740 is generated. In addition, the edge extraction process (S240) and the correction process of the differential image data (S260) can be omitted. In this case, in S270, the uncorrected differential image data (for example, the differential image data 760d) is binarized. The differential image can be an image representing the difference in pixel values ​​of each of a plurality of pixels. The pixel value difference can be represented by various color components (for example, RGB, brightness value, YCbCr, etc.).

[0140] (7) Figure 8 In the embodiment, data processing device 200 operates as a generating device that generates differential image data. Alternatively, data processing device 200 may operate as a generating device that generates output image data by inputting input image data into a learning model, rather than generating differential image data. Processor 210 may store the generated output image data in storage device 215 (e.g., nonvolatile storage device 230). The output image data can be used in various processes (e.g., displaying an output image, generating differential image data, etc.).

[0141] (8) The structure of the image processing system 1000 may be replaced by various other structures. Figure 1 For example, with the execution Figure 6 The training data processing device can be performed by different data processing devices Figure 8 Alternatively, a scanner may be used instead of the digital camera 100. The product 700 is not limited to a label sheet, but may be any product such as a paper product such as a box or envelope for storing goods, a cloth product such as a shirt or towel, or a machine component.

[0142] (9) Figure 1 The data processing device 200 may be a device different from a personal computer (e.g., a digital camera, a scanner, or a smartphone). Alternatively, a plurality of devices (e.g., computers) capable of communicating with each other via a network may each share a portion of the data processing function of the data processing device, thereby providing data processing functions as a whole (a system including these devices corresponds to the data processing device).

[0143] In each of the above embodiments, a portion of the structure implemented by hardware may be replaced with software, and conversely, a portion or all of the structure implemented by software may be replaced with hardware. For example, the learning model 234 may be implemented as a hardware circuit such as an ASIC (Application Specific Integrated Circuit) instead of a program module.

[0144] In addition, when part or all of the functions of the present invention are implemented by a computer program, the program can be provided in the form of a recording medium (e.g., a non-transitory recording medium) that can be read by a computer. The program can be used in a state where it is stored in the same or different recording medium (computer-readable recording medium) as when it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but may also include internal storage devices in computers such as various ROMs, external storage devices connected to computers such as hard disk drives, etc.

[0145] The present invention has been described above based on the embodiments and modifications, but the embodiments are for easy understanding of the present invention and do not limit the present invention. The present invention can be changed and improved without departing from the spirit and scope of the present invention, and its equivalents are included in the present invention.

[0146] Explanation of symbols

[0147] 100…digital camera, 200…data processing device, 210…processor, 215…storage device, 220…volatile storage device, 230…non-volatile storage device, 232…first program, 233…second program, 234…prediction model (machine learning model), L10…input layer, L20…intermediate layer, L30…output layer, 240…display unit, 250…operation unit, 270…communication interface, 311-313…convolution layer, 321…first pooling layer, 322…second pooling layer, 331…addition unit, 4 10…first conversion unit, 420…fully connected layer, 430…second conversion unit, 511-513…transposed convolution layer, 520…addition operation unit, 601-608…operation parameter set, 700, 700x, 750, 800, 950…label slice, 701…graph, 702…character string, 701z, 702z…missing part, 701zb, 702zb…difference part, FL11-FL13…input filter, WN21…first window, WN22…second window, FL31-FL33…output filter.

Claims

1. A machine learning model product, comprising a machine learning model for generating output image data corresponding to input image data including an input image of an input object, the machine learning model being a learned machine learning model, characterized in that: The machine learning model is a model that extracts shape features of an input object contained in an input image of the input image data by performing operations using a plurality of operation parameters on the input image data, and generates output image data containing an output object based on the extracted shape features. The machine learning model includes: an input layer that performs a calculation using the input image data; and an output layer that is provided on the downstream side of the input layer and generates the output image data by restoring shape features. The input layer includes: a first extraction unit configured to extract a shape feature of a portion of the input object included in a first input range smaller than the input image; a second extraction unit provided in parallel with the first extraction unit and configured to extract shape features of a portion of the input object included in a second input range that is smaller than the input image and larger than the first input range; one or more displacement information acquisition units that perform calculations using the data output from the first extraction unit to thereby acquire displacement information related to the displacement of the position of the shape feature extracted by the first extraction unit within the input image; and An adding unit that adds a plurality of data including the data output from the first extraction unit, the data output from the second extraction unit, and the data output from the one or more displacement information acquisition units.

2. The machine learning model product according to claim 1, wherein: The output layer includes: a first restoration unit configured to restore a shape feature using a portion of the data to be processed that corresponds to a first output range that is smaller than the output image; and A second restoration unit is provided in parallel with the first restoration unit and restores shape features using a portion of the processing target data corresponding to a second output range that is smaller than the output image and larger than the first output range.

3. The machine learning model product according to claim 2, wherein: The input layer includes: The first extraction part is a convolutional layer; The second extraction unit is a convolutional layer; and The displacement information acquisition unit is a pooling layer. The output layer does not include a pooling layer but includes: The first restoration part is a transposed convolutional layer; and The second restoration unit is a transposed convolutional layer.

4. The machine learning model product according to any one of claims 1 to 3, characterized in that The one or more displacement information obtaining units include: a first displacement information acquisition unit that acquires first displacement information related to a displacement at a position within a first displacement range smaller than the input image; and a second displacement information acquisition unit provided in parallel with the first displacement information acquisition unit and configured to acquire second displacement information related to a displacement of a position within a second displacement range that is smaller than the input image and larger than the first displacement range; The adding unit adds the plurality of data including the data output from the first extracting unit, the data output from the second extracting unit, the data output from the first displacement information acquiring unit, and the data output from the second displacement information acquiring unit.

5. The machine learning model product according to any one of claims 1 to 3, characterized in that The input layer includes a third extraction unit provided in parallel with the first extraction unit and the second extraction unit, and extracting shape features of a portion of the input object contained in a third input range that is smaller than the input image and larger than the second input range. The adding unit adds the plurality of data including the data output from the first extracting unit, the data output from the second extracting unit, the data output from the third extracting unit, and the data output from the one or more displacement information acquiring units.

6. The machine learning model product according to any one of claims 1 to 3, characterized in that The plurality of operation parameters are adjusted through training so that a difference between reference image data of a reference image including a reference object and output image data generated by inputting the reference image data into the machine learning model becomes smaller.

7. A generating device for generating image data, characterized in that: have: an acquisition unit that acquires input image data including an input image of an input object; an output image data generating unit configured to generate output image data corresponding to the input image data by inputting the input image data into a learned machine learning model; as well as a differential image data generating unit that generates differential image data representing a difference between the input image and the output image of the output image data, The machine learning model is a model that extracts shape features of an input object contained in an input image of the input image data by performing operations using a plurality of operation parameters on the input image data, and generates output image data containing an output object based on the extracted shape features. The machine learning model includes: an input layer that performs a calculation using the input image data; and an output layer that is provided on the downstream side of the input layer and generates the output image data by restoring shape features. The input layer includes: a first extraction unit configured to extract a shape feature of a portion of the input object included in a first input range smaller than the input image; a second extraction unit provided in parallel with the first extraction unit and configured to extract shape features of a portion of the input object included in a second input range that is smaller than the input image and larger than the first input range; one or more displacement information acquisition units that perform calculations using the data output from the first extraction unit to thereby acquire displacement information related to the displacement of the position of the shape feature extracted by the first extraction unit within the input image; and an adding unit that adds a plurality of data including the data output from the first extraction unit, the data output from the second extraction unit, and the data output from the one or more displacement information acquisition units; The plurality of operation parameters are adjusted through training so that a difference between reference image data of a reference image including a reference object and output image data generated by inputting the reference image data into the machine learning model becomes smaller.

8. The generating device according to claim 7, characterized in that The differential image data generating unit obtains smoothed input image data by performing a smoothing process on the input image data. The differential image data generating unit obtains smoothed output image data by performing a smoothing process on the output image data. The difference image data generating unit generates the difference image data using the smoothed input image data and the smoothed output image data.

9. The generating device according to claim 8, characterized in that The differential image data generating unit performs edge extraction processing for extracting edge portions within the image of the smoothed output image data. The difference image data generating unit performs processing on the difference image data to extract a non-edge portion different from the edge portion from the difference image, thereby correcting the difference image data.

10. A recording medium having a computer program recorded thereon, the computer program being used for a computer generating image data, the computer program being characterized in that: The computer program enables the computer to implement the following functions: an acquisition function for acquiring input image data including an input image of an input object; an output image data generation function for generating output image data corresponding to the input image data by inputting the input image data into a learned machine learning model; as well as a differential image data generating function for generating differential image data representing a difference between the input image and an output image of the output image data, The machine learning model is a model that extracts shape features of an input object contained in an input image of the input image data by performing operations using a plurality of operation parameters on the input image data, and generates output image data containing an output object based on the extracted shape features. The machine learning model includes: an input layer that performs a calculation using the input image data; and an output layer that is provided on the downstream side of the input layer and generates the output image data by restoring shape features. The input layer includes: a first extraction unit configured to extract a shape feature of a portion of the input object included in a first input range smaller than the input image; a second extraction unit provided in parallel with the first extraction unit and configured to extract shape features of a portion of the input object included in a second input range that is smaller than the input image and larger than the first input range; one or more displacement information acquisition units that perform calculations using the data output from the first extraction unit to thereby acquire displacement information related to the displacement of the position of the shape feature extracted by the first extraction unit within the input image; and an adding unit that adds a plurality of data including the data output from the first extraction unit, the data output from the second extraction unit, and the data output from the one or more displacement information acquisition units; The plurality of operation parameters are adjusted through training so that a difference between reference image data of a reference image including a reference object and output image data generated by inputting the reference image data into the machine learning model becomes smaller.

11. The recording medium according to claim 10, wherein The differential image data generating function obtains smoothed input image data by performing a smoothing process on the input image data. The differential image data generating function obtains smoothed output image data by performing a smoothing process on the output image data. The difference image data generation function generates the difference image data using the smoothed input image data and the smoothed output image data.

12. The recording medium according to claim 11, wherein The differential image data generating function performs edge extraction processing for extracting edge portions within the image of the smoothed output image data. The difference image data generating function performs processing on the difference image data to extract a non-edge portion different from the edge portion from the difference image, thereby correcting the difference image data.

13. A generating device for generating image data, the generating device comprising: an acquisition unit that acquires input image data including an input image of an input object; and an output image data generating unit that generates output image data corresponding to the input image data by inputting the input image data into a learned machine learning model, The machine learning model is a model that extracts shape features of an input object contained in an input image of the input image data by performing operations using a plurality of operation parameters on the input image data, and generates output image data containing an output object based on the extracted shape features. The machine learning model includes: an input layer that performs a calculation using the input image data; and an output layer that is provided on the downstream side of the input layer and generates the output image data by restoring shape features. The input layer includes: a first extraction unit configured to extract a shape feature of a portion of the input object included in a first input range smaller than the input image; a second extraction unit provided in parallel with the first extraction unit and configured to extract shape features of a portion of the input object included in a second input range that is smaller than the input image and larger than the first input range; one or more displacement information acquisition units configured to perform calculations using the data output from the first extraction unit to thereby acquire displacement information related to the displacement of the position of the shape feature extracted by the first extraction unit within the input image; as well as an adding unit that adds a plurality of data including the data output from the first extraction unit, the data output from the second extraction unit, and the data output from the one or more displacement information acquisition units; The plurality of operation parameters are adjusted through training so that a difference between reference image data of a reference image including a reference object and output image data generated by inputting the reference image data into the machine learning model becomes smaller.

Images