Image processing method, image processing apparatus, and storage medium

Abstract

Provided is an image processing method capable of performing image processing using a machine learning model on images of various sizes with high accuracy.SOLUTION: A first process S204 of dividing a first image to generate a plurality of second images smaller than the first image, and a second process S205 of inputting the plurality of second images to a machine-learning model to generate a plurality of third images, in which each of the plurality of second images includes an overlapping region overlapping with an adjacent second image, and the plurality of second images are generated to include an image including an overlapping region having a first pixel number and an image including an overlapping region having a second pixel number different from the first pixel number; SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a method for processing images using machine learning models. [Background technology]

[0002] In image processing using machine learning models, an input image is sometimes divided into multiple patches and processed sequentially rather than processed all at once. Also, a process is known in which multiple patches are made the same size (number of pixels) by using padding, which adds pixels to the periphery of the input image, and overlapping, which overlaps some pixels of adjacent patches.

[0003] Patent Document 1 discloses an image processing method that generates an estimated image by dividing an input image of a predetermined number of pixels and inputting multiple patches that overlap with each other by a certain number of pixels into a machine learning model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US2022 / 0309027 Summary of the Invention [Problem to be solved by the invention]

[0005] However, since the number of pixels is set to a fixed value, when an input image with a number of pixels different from the predetermined number is processed in the image processing method of Cited Document 1, there is a risk that the number of pixels of multiple patches will differ. As a result, it is not possible to accurately process input images with various numbers of pixels using a machine learning model. [Means for solving the problem]

[0006] An image processing method according to one aspect of the present invention includes a first step of dividing a first image to generate a plurality of second images smaller than the first image, and a second step of generating a plurality of third images by inputting the plurality of second images into a machine learning model, wherein each of the plurality of second images includes an overlapping area that overlaps with an adjacent second image, and the plurality of second images generated include an image including an overlapping area with a first number of pixels and an image including an overlapping area with a second number of pixels different from the first number of pixels. [Effects of the Invention]

[0007] According to an image processing method according to one embodiment of the present invention, it is possible to provide an image processing method or the like that is capable of performing image processing using a machine learning model with high accuracy on images of various sizes. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating an outline of image processing in each embodiment. [Figure 2] 10A and 10B are diagrams illustrating tile division in each embodiment. [Figure 3] 10A and 10B are diagrams illustrating tile synthesis in each embodiment. [Figure 4] 1 is a block diagram of an image processing system according to a first embodiment. [Figure 5] 1 is an external view of an image processing system according to a first embodiment. [Figure 6] FIG. 1 is a diagram illustrating a flow of learning of a machine learning model in a first embodiment. [Figure 7] 1 is a flowchart relating to learning of a machine learning model in the first embodiment. [Figure 8] 1 is a flowchart relating to image processing using a machine learning model in the first embodiment. [Figure 9] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 10] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 11]FIG. 10 is a block diagram of an image processing system according to a third embodiment. [Figure 12] 11 is a flowchart relating to image processing using a machine learning model in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted.

[0010] First, image processing using a machine learning model in each embodiment will be described with reference to FIGS.

[0011] In this embodiment, as shown in FIG. 1(a), a padded image (first image) is generated by adding (adding) a predetermined number of extra pixels to the periphery of an input image of 256 × 256 pixels (number of vertical × horizontal pixels). In FIG. 1(a), the additional area added by padding is represented by a diagonally shaded area. Note that the predetermined number of pixels in FIG. 1 is six pixels, and the same number of pixels are added to all four sides of the input image in FIG. 1. Therefore, the size of the image (first image) obtained by padding is 268 × 268 pixels. For example, if a long, narrow patch one pixel wide is to be divided from the edge of the input image, there is no guarantee that the machine learning model can process this patch with the same accuracy as other patches. In such cases, it is preferable to add pixels by padding.

[0012] In addition, the padding method used in this embodiment is a method called mirroring, which adds extra pixels to the periphery of the input image by folding six pixels inward from the edge of the input image and then outward using the edge of the input image as the rotation axis.

[0013] Next, as shown in FIG. 1(b), the padding image is divided into a plurality of tiles (second image) each having a predetermined number of pixels. In FIG. 2, for the sake of explanation, the padding image (258×258 pixels) and the upper left tile (64×64 pixels) are indicated by thick lines. In FIG. 1(b), each tile is generated so that adjacent tiles have a predetermined number of overlapping portions (overlapping areas) when the padding image is divided into tiles. Therefore, when the padding image is divided into tiles, the padding image is divided into a total of 25 tiles (5×5), and each tile extends beyond the edge of the padding image by 4 pixels. Note that in FIG. 1(b), the predetermined tile size is 64×64 pixels and the predetermined number of overlapping pixels is 12 pixels (first number of pixels), but this embodiment is not limited to this.

[0014] Here, the order of tile division is shown in Figure 1(d). In this embodiment, the padding image is divided into tiles horizontally (horizontally) starting from the upper left corner, then goes back one step to the left corner, and the tiles are divided horizontally (horizontally) again, repeating this process in a left-to-right order. Therefore, in Figure 1(b), some tiles extend beyond the right and bottom edges of the padding image.

[0015] In this embodiment, the number of pixels of the tile that extend beyond the right edge (horizontal direction) of the padding image can be calculated by subtracting the number of pixels of the padding image in the horizontal direction from the sum of (number of overlapping pixels) × 4, (number of pixels obtained by subtracting twice the number of overlapping pixels from the number of pixels of the tile in the horizontal direction) × 3, and (number of pixels obtained by subtracting the number of overlapping pixels from the number of pixels of the tile in the horizontal direction) × 2. Note that the number of pixels of the tile that extend beyond the bottom edge (vertical direction) of the padding image can be calculated in a similar manner. Note that the tile division method, number of pixels, and calculation method used here are merely examples, and the present invention is not limited to these.

[0016] In this embodiment, to deal with tiles that extend beyond the padding image, the overlap pixels of some tiles are made different, as shown in Fig. 1(c). Specifically, in Fig. 1(c), the number of overlap pixels of the edge tiles that extend beyond the padding image is increased by four pixels to 16 pixels.

[0017] Tile T5, which extends beyond the right edge of the padding image in Figure 1, overlaps with the adjacent tile T4 on the left, so the number of overlapping pixels for tile T5 is set to 16. Similar processing is also performed on tiles T10, T15, and T20, which extend beyond the right edge of the padding image. Furthermore, similar processing is performed on tiles T21, T22, T23, T24, and T25, which extend beyond the bottom edge of the padding image and differ only in that their adjacent tiles are located above them. This configuration allows the entire padding image to be processed by the machine learning model, even when tiles extend beyond the padding image.

[0018] Next, a process for generating an output image from a plurality of tiles (second image) will be described with reference to Figures 2 and 3. Figure 2 is a diagram showing a process for generating a plurality of tiles (third image) after processing from a plurality of tiles using a machine learning model, and Figure 3 is a diagram showing a process for generating an output image (fourth image) by combining a plurality of tiles after processing.

[0019] As shown in Figure 2, in this embodiment, the machine learning model performs processing by sequentially inputting multiple tiles. This configuration reduces the processing load. As a result, desired image processing can be performed with high accuracy without relying on the CPU, GPU, or ASIC memory used for processing.

[0020] FIG. 3(a) shows an example of processing for combining images by connecting them in the spatial direction, and FIG. 3(a) shows an example of processing for combining images by weighted averaging.

[0021] In Figure 3(a), half of the 12 overlapping pixels contained in the processed tiles (6 pixels) are removed from each of the processed tiles T'1 and T'2, and then the tiles are connected to adjacent tiles. This is not a limitation, and for example, 10 of the 12 overlapping pixels contained in the processed tile T'1 may be removed, and 2 of the 12 overlapping pixels contained in the processed tile T'2 may be removed, before connection. The process of combining by connecting in the spatial direction is preferable because it reduces the processing load and enables faster processing.

[0022] In Figure 3(b), the overlapping pixels contained in processed tile T'1 and processed tile T'2 are combined using a weighted average process. This configuration allows for smooth transitions between processed tiles, which is preferable in terms of producing a high-quality output image. Note that tiles may be combined after removing pixels corresponding to padding pixels added to the periphery of the input image.

[0023] In this embodiment, the number of overlapping pixels is increased beyond a predetermined number only for edge tiles that extend beyond the padding image. Therefore, exceptional processing is performed only on tiles in predetermined locations (e.g., edges). This configuration reduces the load of tile division processing in an image processing method that can accurately process input images of various sizes. Furthermore, in the processing of this embodiment, it is preferable that the number of pixels in the overlapping area, the additional area, and the tile are each specified as a predetermined number of pixels, regardless of the size of the input image. This configuration allows the division processing into tiles to be performed regardless of the size of the input image. Furthermore, the adjustment step described below adjusts the tile size, allowing the entire input image or padding image to be processed accurately.

[0024] The above image processing method is an example, and the present invention is not limited to this. Details of other image processing methods will be described in the following examples. [Example]

[0025] Next, an image processing system using a machine learning model according to a first embodiment will be described with reference to FIGS.

[0026] The machine learning model in this embodiment is a model structured by a neural network, but is not limited to this. The machine learning model in this embodiment may be a mathematical model obtained by deep learning. As the machine learning model in this embodiment, for example, a model structured by a Convolutional Neural Network (CNN) may be used. Furthermore, as the machine learning model, a model structured by a Generative Adversarial Network (GAN), a Recurrent Neural Network (RNN), a Fully Connected Network (FCN), or a Transformer may also be used. In the following, to avoid redundant expression, a model structured by a neural network may be simply referred to as a neural network.

[0027] 4 is a block diagram of the image processing system 100 according to this embodiment, and FIG.

[0028] The image processing system 100 includes a learning device 101, an imaging device 102, an image estimation device 103, a display device 104, a recording medium 105, an input device 106, an output device 107, and a network 108.

[0029] The learning device 101 includes a storage unit (storage means) 101a, an acquisition unit (acquisition means) 101b, and a learning unit (learning means) 101c. Details of the learning device 101 will be described later.

[0030] The imaging device 102 has an optical system 102a and an imaging element 102b. The optical system 102a collects light incident on the imaging device 102 from the subject space. The imaging element 102b receives an optical image of the subject formed via the optical system 102a and acquires a captured image (blurred image). The imaging element 102b is a CCD (Charge Coupled Device) sensor, a CMOS (Complementary Metal-Oxide Semiconductor) sensor, or the like.

[0031] The imaging device 102 can acquire information about the shooting conditions of the captured image (focal length and aperture of the optical system 102a, the number of pixels, pixel pitch, and type of optical low-pass filter of the image sensor 102b, the shooting mode, ISO sensitivity at the time of shooting, etc.) along with the image. It can also acquire the development conditions of the captured image (image format, noise reduction level, sharpness level, image compression rate, etc.) along with the image. This information acquired together with the image can also be transmitted together with the image to an acquisition unit 103b of the image estimation device 103 (described later). A storage unit for saving the acquired image, a display unit for displaying it, a transmission unit for transmitting it to the outside, an output unit for saving it in an external storage medium, etc. are not shown. A control unit for controlling each unit of the imaging device 102 is also not shown.

[0032] The image estimation device 103 includes a storage unit 103a, an acquisition unit 103b, an addition unit 103c, a division unit 103d, an adjustment unit 103e, a processing unit (estimation means) 103f, and a synthesis unit 103g. Details of the image processing executed by the image estimation device 103 will be described later.

[0033] In the image processing in this embodiment, a machine learning model is used, and the weight information of the machine learning model is read from the storage unit 103a.

[0034] The weight information in this embodiment is learned by the learning device 101. The image estimation device 103 reads the weight information from the storage unit 101a via the network 108 in advance and stores it in the storage unit 103a. The stored weight information may be the weight numerical value itself or may be in an encoded format. Details regarding the learning of the machine learning model and image processing using weights in this embodiment will be described later.

[0035] The blur-reduced image (output image) is output to at least one of a display device 104, a recording medium 105, and an output device 107. The display device 104 is, for example, a liquid crystal display or a projector. A user can check the image being processed via the display device 104 and perform image editing work via the input device 106. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, etc. The input device 106 is, for example, a keyboard or a mouse, etc. The output device 107 is, for example, a printer, etc.

[0036] Next, a learning (training) method for a machine learning model executed by the learning device 101 in this embodiment will be described with reference to Figures 6 and 7. The learning method for a machine learning model corresponds to a method for generating a trained model.

[0037] FIG. 6 is a diagram showing the flow of updating the weights of a neural network (machine learning model). FIG. 7 is a flowchart related to neural network learning. The steps in FIG. 7 are mainly executed by the acquisition unit 101b and the learning unit 101c. Note that, in this embodiment, an example is shown in which the processing by the machine learning model is processing to reduce image blur (image restoration, sharpening), but this is not limited to this. The processing by the machine learning model may also include upscaling (super-resolution), increasing contrast, improving brightness, denoising, defocus blur conversion, lighting conversion, etc. By using training images according to the desired processing, it is possible to learn a machine learning model capable of the desired processing, similar to the method described below.

[0038] CN in Figure 6 represents a convolutional layer. In the convolutional layer CN, the sum of the input, the convolution of the filter, and the bias is calculated, and the result is nonlinearly transformed using an activation function. The initial values ​​of each filter component and the bias are arbitrary, and in this embodiment, they are determined by random numbers. The activation function can be, for example, a ReLU (Rectified Linear Unit) or a sigmoid function. The multidimensional array output from each layer except the final layer is a feature map.

[0039] A feature map is a four-dimensional array with dimensions of batch, length, width, and channel. In the skip connection 23, feature maps output from discontinuous layers are combined. In this case, the feature maps can be combined by adding each element together or by concatenation in the channel direction.

[0040] In addition, the elements (blocks or modules) within the dotted line frame in Figure 6 represent residual blocks. A network with multiple layers of residual blocks is called a residual network, and is widely used in DL image processing.

[0041] In this embodiment, the neural network configuration shown in Figure 6 is used, but the present invention is not limited to this. For example, an inception module may be used in which convolution layers with different convolution filter sizes are arranged side by side and the resulting feature maps are integrated to form a final feature map. Furthermore, a network may be configured by layering other elements, such as dense blocks with dense skip connections.

[0042] First, in step S101, the acquisition unit 101b acquires a blurred patch 21, which is a training image, and a corresponding sharp, low-blur ground truth patch 20. In this embodiment, a patch is a small image having a predetermined number of pixels. For example, the blurred patch is 128 × 128 pixels, and the corresponding ground truth patch is 128 × 128 pixels. The blurred patch and the corresponding ground truth patch may be acquired by capturing images of the same subject using an optical system with good optical performance and low blur and an optical system with poor optical performance and high blur, and then cropping corresponding portions of the two images. Alternatively, the influence of the optical system 102a (aberration and diffraction) on the low-blur ground truth patch may be applied by numerical calculation to generate a blurred patch equivalent to that acquired by the imaging device 102. In this embodiment, the blurred patch and the corresponding ground truth patch are generated by numerical calculation, but the present invention is not limited to this.

[0043] Next, in step S102, the learning unit 101c uses a neural network to generate a blur-reduced patch 22 from the blur patch 21. Ideally, the blur-reduced patch 22 and the correct patch 20 will match. Furthermore, image information may be input to the neural network along with the blur patch to perform blur reduction that takes the image information into consideration. For example, if the focal length and aperture of the optical system 102a are used as information related to the shooting conditions, the effects of aberration and diffraction, which are blurs specific to these, can be accurately recovered.

[0044] Next, in step S103, the learning unit 101c updates the weights of the neural network based on the error between the ground truth patch 20 and the blur reduction patch 22. Here, the weights include the filter components and biases of each layer. Backpropagation is used to update the weights, but the present invention is not limited to this. In mini-batch learning, the error between multiple ground truth patches 20 and their corresponding estimated blur reduction patches 22 is calculated, and the weights are updated. For example, the L2 norm or the L1 norm may be used as the loss function. The weight update method (learning method) is not limited to mini-batch learning, and batch learning or online learning may also be used.

[0045] Next, in step S104, the learning unit 101c determines whether learning of the neural network is complete. Completion of learning can be determined by, for example, whether the number of iterations of weight update has reached a specified value, or whether the amount of change in weight during update is smaller than a specified value. If it is determined that learning is incomplete, the process returns to step S101, and multiple new blur patches 21 and corresponding correct patches 20 are obtained. On the other hand, if it is determined that learning is complete, the learning device 101 ends learning and saves weight information in the storage unit 101a.

[0046] Next, the generation of a blur-reduced image executed by the image estimation device 103 in this embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart related to the generation of a blur-reduced image. The steps in Fig. 8 are mainly executed by the acquisition unit 103b, the addition unit 103c, the division unit 103d, the adjustment unit 103e, the processing unit 103f, and the synthesis unit 103g of the image estimation device 103.

[0047] First, in step S201, the acquisition unit 103b acquires a captured image. The captured image in this embodiment is a blurred image, similar to that used in learning. In this embodiment, the captured image is transmitted from the imaging device 102, but is not limited to this. In this embodiment, the size of the captured image is 256 x 256 pixels. Image information may also be acquired together with the captured image and used in steps described below.

[0048] In step S202, the adding unit 103c generates a padded image (first image) having pixels (additional region) in which a predetermined number of padding pixels are added around the captured image. In this embodiment, the predetermined number of padding pixels is six pixels in each of the vertical and horizontal directions, and the same number of pixels is added to the four sides of the captured image by mirroring. Therefore, the image size (first size) after padding is 268 × 268 pixels.

[0049] In this embodiment, a predetermined fixed value is used as the predetermined number of padding pixels, but the present invention is not limited to this. For example, the predetermined number of padding pixels may be determined using a predetermined numerical table based on the number of pixels of the image sensor 102b in the image information acquired together with the captured image. Furthermore, if image information cannot be acquired, a value specified by the user may be used as the predetermined number of padding pixels.

[0050] In step S203, the dividing unit 103d divides the padding image into multiple tiles (second images) each having a predetermined number of pixels. In this embodiment, the predetermined size (second size) of the tiles is 64 x 64 pixels. The multiple tiles also have overlapping portions (overlapping areas) with a predetermined number of pixels between adjacent tiles. In this embodiment, this predetermined number of overlapping pixels is 12 pixels (first number of pixels) for horizontally adjacent tiles and 12 pixels for vertically adjacent tiles.

[0051] In this embodiment, predetermined fixed values ​​are used as the predetermined number of tile pixels and the predetermined number of overlapping pixels, but the present invention is not limited to this. For example, values ​​may be determined using a predetermined numerical table based on the number of pixels of the image sensor 102b in the image information acquired together with the captured image. Furthermore, if image information cannot be acquired, values ​​specified by the user may be used as the predetermined number of tile pixels and the predetermined number of overlapping pixels.

[0052] In this embodiment, the padding image is divided into tiles horizontally (horizontally) starting from the upper left corner, similar to the order shown in Figure 1(d), and then the padding image is divided into tiles in a left-to-right order, starting from the upper left corner, then moving down one level to the left corner, and the same process is repeated. Therefore, as a result of the tile division, tiles extend beyond the right and bottom edges of the padding image by four pixels. Furthermore, each tile obtained by the processing of the division unit 103d has one to four overlapping areas of a predetermined number of pixels, depending on its position in the padding image.

[0053] Note that the tile division method in this embodiment is an example, and the present invention is not limited to this. For example, the upper right corner of the padding image may be the starting point for tile division, or the tiles may be divided in the order of right-to-left writing. In this case, tiles will extend beyond the bottom and left edges of the padding image.

[0054] In step S204, the adjustment unit 103e divides the padding image into multiple tiles by a predetermined number of pixels. If a tile extends beyond the edge of the padding image, the adjustment unit 103e adjusts the number of overlapping pixels to maintain a constant number of tiles. In this embodiment, as shown in FIG. 1B, the tile division results in tiles extending beyond the right and bottom edges of the padding image by four pixels. Therefore, the number of overlapping pixels of the tiles corresponding to the right and bottom edges of the padding image is increased from the predetermined number of 12 pixels by the four pixels that extend beyond the padding image, to 16 pixels. For example, tile T5, which extends beyond the right edge of the padding image in FIG. 1, overlaps with the adjacent tile T4 on the left, so the number of overlapping pixels is set to 16 pixels.

[0055] In this way, by the processing of the dividing unit 103d and the adjusting unit 103e, tiles that include one or more overlapping areas with a predetermined number of pixels and tiles that have overlapping areas with more than the predetermined number of pixels are generated. By using this configuration, it is possible to make the size of tiles input to the machine learning model described later constant.

[0056] Note that the method for adjusting the number of overlapping pixels between adjacent tiles in this embodiment is merely an example, and the present invention is not limited thereto. For example, if a tile extends beyond the edge of the padding image due to tile division, the number of overlapping pixels between adjacent tiles may be adjusted by dividing the tile starting from the edge of the padding image where the tile extends. Alternatively, if a tile extends beyond the edge of the padding image due to tile division, the number of overlapping pixels between tiles corresponding to the edge and center of the padding image and adjacent tiles may be adjusted. Furthermore, after the division process, a process of adding an additional area to tiles with a pixel count smaller than a predetermined pixel count may be performed.

[0057] In step S205, the processing unit 103f uses a machine learning model to sequentially process multiple tiles to generate blur-reduced tiles (third images). To generate the blur-reduced tiles, a machine learning model with the same configuration as that shown in FIG. 6 is used. When blur reduction is performed using image information, the method described in step S102 is used. The weight information is transmitted from the learning device 101 and stored in the storage unit 103a.

[0058] In step S206, the composition unit 103g combines the blur reduction tiles to generate a blur reduction image (output image). Here, the composition is performed using the weighted average shown in Fig. 3(b). Alternatively, the blur reduction tiles may be combined after removing pixels corresponding to the padding pixels added to the periphery of the captured image in step S202.

[0059] The order of the steps in this embodiment is merely an example and is not limited to this. The order of the steps may be changed as necessary, and the processing of each step may be performed as an integrated configuration.

[0060] With the above configuration, according to this embodiment, image processing using a machine learning model can be performed with high accuracy on images of various sizes.

[0061] Although the above example illustrates a tile with the same number of pixels in both the vertical and horizontal directions (specifically, 64 × 64 pixels), the present invention is not limited to this example, and the number of pixels in the vertical and horizontal directions of a tile may not be the same. Furthermore, the predetermined number of pixels in the divided areas included in a tile may be different in the vertical and horizontal directions. For example, the predetermined number of overlapping pixels (first number of pixels) in the overlapping area with an image adjacent in the vertical direction may be different from the predetermined number of overlapping pixels (second number of pixels) in the overlapping area with an image adjacent in the horizontal direction. In this case, the adjustment unit 103e described above can adjust the number of overlapping pixels protruding from the padding image in at least one of the vertical and horizontal directions by increasing the number of overlapping pixels so that the number of overlapping pixels becomes a third pixel number. [Example]

[0062] Next, an image processing system 200 according to a second embodiment will be described with reference to Fig. 9 and Fig. 10. The image processing system 200 of this embodiment differs from that of the first embodiment in that a captured image (low-resolution image) is acquired by an imaging device and image processing is performed by a processing unit in the imaging device.

[0063] Fig. 9 is a block diagram of an image processing system 200 according to this embodiment, and Fig. 10 is an external view of the image processing system 200.

[0064] The image processing system 200 includes a learning device 201 and an imaging device 202 connected via a network 203. Furthermore, the learning device 201 and the imaging device 202 do not need to be connected via the network 203 all the time.

[0065] The learning device 201 has a storage unit (storage means) 211, an acquisition unit (acquisition means) 212, and a learning unit (learning means) 213. These are used to perform image processing for generating a high-resolution image from a low-resolution image, and to learn a machine learning model. Each component of the learning device 201 is equivalent to each component of the learning device 101 in the first embodiment.

[0066] The imaging device 202 captures an image of a subject space, acquires a captured image (low-resolution image), and generates an upscaled image from the captured image. Details of the image processing executed by the imaging device 202 will be described later. The imaging device 202 has an optical system 221 and an image sensor 222. The image estimation unit 223 has an acquisition unit (acquisition means) 223a, an addition unit (addition means) 223b, a division unit (division means) 223c, an adjustment unit (adjustment means) 223d, a processing unit (estimation means) 223e, and a synthesis unit 223f.

[0067] The method of training the machine learning model executed by the learning device 201 differs from that of Example 1 in that low-resolution patches and corresponding high-resolution answer patches are used as training images. The learning method, except for the combination of training images, is the same as that of Example 1, and therefore a detailed description thereof will be omitted.

[0068] The weight information of the machine learning model is learned in advance by the learning device 201 and stored in the storage unit 211. The imaging device 202 reads the weight information from the storage unit 211 via the network 203 and stores it in the storage unit 224. The image estimation unit 223 generates a high-resolution image (output image) by upscaling the low-resolution image. The image estimation unit 223 generates a high-resolution image using the weight information of the trained machine learning model stored in the storage unit 224, and the low-resolution image (captured image) acquired by the acquisition unit 223a and its image information. The generated upscaled image is stored in the recording medium 225a.

[0069] Furthermore, when a user issues an instruction to display an upscaled image, the stored image is read out and displayed on the display unit 225b. Note that the captured image and its image information stored in the recording medium 225a may be read out, and the upscaled image may be generated by the image estimation unit 223. The above series of controls are performed by the system controller 227.

[0070] Next, a description will be given of generation of an upscaled image executed by the image estimation unit 223 in this embodiment. The steps of image processing are mainly executed by an acquisition unit 223a (acquisition means), an addition unit (addition means) 223b, a division unit (division means) 223c, an adjustment unit (adjustment means) 223d, a processing unit (processing means) 223e, and a synthesis unit (synthesis means) 223f.

[0071] Note that steps S303 and S305 in this embodiment are similar to steps S203 and S205 in the first embodiment, respectively, and therefore will not be described here.

[0072] First, in step S301, the acquisition unit 223a acquires a low-resolution image (captured image). Note that in this embodiment, the captured image is acquired by the imaging device 202 and stored in the storage unit 224, but the present invention is not limited to this. Also, in this embodiment, for convenience of explanation using FIG. 1, the size of the captured image is set to 256×256 pixels. The size of the captured image in this embodiment is not limited to this.

[0073] Next, in step S302, the adding unit 223b generates a padded image (first image) having a portion (additional region) in which a predetermined number of padding pixels are added to the periphery of the captured image. In this embodiment, the predetermined number of padding pixels is six, and the same number of pixels is added to all four sides of the captured image. Note that the padding method used in this embodiment is a method called replication. In replica padding, pixels at the edge of the captured image are repeatedly arranged six pixels outward to add padding pixels to the periphery of the captured image. Therefore, the image size after padding is 268 × 268 pixels. In this embodiment, the predetermined number of padding pixels is a predetermined fixed value, but the present invention is not limited to this. For example, the predetermined number of padding pixels may be determined using a predetermined numerical table based on the number of pixels of the image sensor 222, which is included in the image information acquired together with the captured image. Alternatively, if image information cannot be acquired, a value specified by the user may be used as the predetermined number of padding pixels.

[0074] In step S304, the adjustment unit 223d divides the padding image into multiple tiles. If a tile extends beyond the edge of the padding image, the adjustment unit 223d adjusts the number of overlapping pixels to maintain a constant number of pixels in the tile. In this embodiment, the tile division causes the tile to extend beyond the right and bottom edges of the padding image by four pixels. Therefore, the number of overlapping pixels for the tile located in the center of the padding image is increased from the specified 12 pixels by the four pixels that extend beyond it, to 16 pixels. For example, the number of overlapping pixels between tile T'3 and tile T'2, adjacent to it on the left, in Figure 1(d) is increased to 16 pixels.

[0075] In step S306, the composition unit 223f combines the upscaled tiles to generate an upscaled image (output image). Here, the tile composition is performed by spatial concatenation as shown in FIG. 3(a). Note that the tiles are composed taking into consideration the number of overlapping pixels that increases with upscaling. For example, when the applescale magnification is 2x, the predetermined number of overlapping pixels increases from 12 to 24, and therefore the number of pixels to be removed for concatenation is 12, half of 24. Also, as mentioned above, the number of pixels to be removed for concatenation is not limited to half the number of overlapping pixels. Furthermore, the upscaled tiles may be composed after removing pixels corresponding to the padding pixels added around the captured image in step S302.

[0076] With the above configuration, according to this embodiment, image processing using a machine learning model can be performed with high accuracy on images of various sizes. [Example]

[0077] Next, an image processing system 300 according to a third embodiment will be described with reference to Fig. 11 and Fig. 12. This embodiment differs from the first and second embodiments in that it includes a processing device (computer) that transmits a captured image (noisy image) to be subjected to image processing to an image estimation device and receives a processed output image (noise-reduced image) from the image estimation device. Note that a noisy image in this embodiment is an image with many high-frequency components (graininess in the image), such as an image taken with a high ISO sensitivity.

[0078] Fig. 11 is a block diagram of the image processing system 300 in this embodiment. Fig. 12 is a flowchart relating to learning.

[0079] Image processing system 300 includes a learning device 301, an imaging device 302, an image estimation device 303, and a processing device (computer) 304. Learning device 301 and image estimation device 303 are, for example, servers. Computer 304 is, for example, a user terminal (a personal computer or a smartphone). Computer 304 is connected to image estimation device 303 via network 305. Image estimation device 303 is connected to learning device 301 via network 306. That is, computer 304 and image estimation device 303 are configured to be able to communicate with each other, and image estimation device 303 and learning device 301 are configured to be able to communicate with each other.

[0080] The method of training the neural network executed by the training device 301 differs in that noisy patches and corresponding low-noise correct patches are used as training images. The training method, except for the combination of training images, is the same as in the first embodiment, so a detailed explanation will be omitted.

[0081] The configuration of the image capturing device 302 is the same as that of the image capturing device 102 of the first embodiment, and therefore a description thereof will be omitted.

[0082] The image estimation device 303 includes a storage unit 303a, an acquisition unit 303b, an addition unit 303c, a division unit 303d, an adjustment unit 303e, a processing unit 303f, a synthesis unit 303g, and a communication unit (reception means) 303h. The communication unit 303h has a function of receiving a request transmitted from the computer 304 and a function of transmitting an output image generated by the image estimation device 303 to the computer 304.

[0083] The computer 304 has a communication unit (transmission means) 304a, a display unit 304b, an input unit 304c, a processing unit 304d, and a storage unit 304e. The communication unit 304a has a function of transmitting a request to the image estimation device 303 to cause the image estimation device 303 to execute processing on a captured image, and a function of receiving an output image processed by the image estimation device 303. The display unit 304b has a function of displaying various information.

[0084] Information displayed by the display unit 304b includes, for example, a captured image to be transmitted to the image estimation device 303 and an output image received from the image estimation device 303. The input unit 304c receives an instruction from a user to start image processing, etc. The processing unit 304d has a function of performing image processing including sharpening on the output image received from the image estimation device 303. The storage unit 304e stores the captured image acquired from the imaging device 302, the output image received from the image estimation device 303, etc.

[0085] Next, image processing in this embodiment will be described. Image processing according to this embodiment differs from embodiment 1 in that it is started when a user issues an instruction to start image processing via the computer 304, and is executed by the image estimation device 303. First, the operation of the computer 304 will be described below.

[0086] First, in step S401, the computer 304 transmits a request for processing a captured image to the image estimation device 303. Note that the method for transmitting the captured image to be processed to the image estimation device 303 does not matter. For example, the captured image may be uploaded to the image estimation device 303 simultaneously with S301, or may be uploaded to the image estimation device 303 before S401. Furthermore, the captured image may be an image stored on a server different from the image estimation device 303. Furthermore, in S401, the computer 304 may transmit an ID for authenticating a user, image information, and the like together with the request for processing the captured image.

[0087] In step S 402 , the computer 304 receives the output image generated in the image estimation device 303 .

[0088] Next, the operation of the image estimation device 303 will be described.

[0089] First, in step S501, the image estimation device 303 receives a request for processing a captured image transmitted from the computer 304. The image estimation device 303 determines that processing for the captured image has been instructed, and executes the processing from step S502 onwards.

[0090] In step S502, the acquisition unit 303b acquires a captured image. In this embodiment, the captured image is transmitted from the computer 304. Note that image information may also be acquired together with the captured image and used in steps described below.

[0091] Note that steps S503 to S507 are the same as steps S202 to S206 in the first embodiment, and therefore the explanation will be omitted.

[0092] Next, in step S508, the image estimation device 303 transmits the output image to the computer 304.

[0093] With the above configuration, according to this embodiment, image processing using a machine learning model can be performed with high accuracy on images of various sizes.

[0094] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0095] According to each embodiment, it is possible to provide an image processing method, an image processing device, a program, and a storage medium that are capable of performing image processing using a machine learning model with high accuracy on images of various sizes. The image processing device may be any device that has the image processing function of the present invention, and may be realized in the form of an imaging device or a personal computer.

[0096] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0097] The embodiments of the present invention include the following methods, configurations, and programs.

[0098] (Method 1) a first step of dividing a first image to generate a plurality of second images smaller than the first image; a second step of generating a plurality of third images by inputting the plurality of second images into a machine learning model; each of the plurality of second images includes an overlap region that overlaps with an adjacent second image; An image processing method characterized in that the multiple second images are generated to include an image including an overlapping area with a first number of pixels and an image including an overlapping area with a second number of pixels different from the first number of pixels.

[0099] (Method 2) 2. The image processing method according to method 1, further comprising the step of generating the first image by adding an additional region adjacent to the periphery of the input image to the input image.

[0100] (Method 3) In the first step, the overlapping region in the second image located at the outermost edge of the plurality of second images is set to the first pixel number; The image processing method according to Method 1 or 2, wherein the second number of pixels is set to the overlapping area in the second image other than the second image located at the outermost edge.

[0101] (Method 4) The image processing method according to any one of methods 1 to 3, wherein the first number of pixels is a predetermined fixed value, a value based on instructions from a user, or a value based on the imaging device used to acquire the first image or the shooting mode of the imaging device.

[0102] (Method 5) The image processing method described in any one of Methods 1 to 4, characterized in that the second number of pixels is either a predetermined fixed value, a value based on instructions from a user, or a value based on the imaging device used to acquire the first image or the shooting mode of the imaging device.

[0103] (Method 6) The image processing method described in Method 2, characterized in that the number of pixels in the additional area is one of a predetermined fixed value, a value based on instructions from a user, or a value based on the imaging device used to acquire the input image and the shooting mode of the imaging device.

[0104] (Method 7) 7. The image processing method according to any one of Methods 1 to 6, further comprising a third step of generating a fourth image by combining the plurality of third images.

[0105] (Method 8) The image processing method described in Method 7, characterized in that in the third step, the plurality of third images are synthesized by removing at least a portion of the overlapping regions in the plurality of third images and then connecting adjacent third images in the spatial direction.

[0106] (Method 9) 8. The image processing method according to method 7, wherein in the third step, the plurality of third images are synthesized by taking a weighted average of the plurality of third images.

[0107] (Method 10) a first step of dividing a first image to generate a plurality of second images smaller than the first image; a second step of generating a plurality of third images by inputting the plurality of second images into a machine learning model; each of the plurality of second images includes an overlap region that overlaps with an adjacent second image; In the first step, the plurality of second images are generated, each of which includes an image including one or more overlapping regions with a horizontally adjacent image having a first number of pixels and one or more overlapping regions with a vertically adjacent image having a second number of pixels, and an image including one or more overlapping regions with a horizontally or vertically adjacent image having a third number of pixels; An image processing method, wherein the first number of pixels, the second number of pixels, and the third number of pixels are different from one another.

[0108] (Method 11) a first step of dividing a first image to generate a plurality of second images smaller than the first image; adjusting the plurality of second images; a second step of generating a plurality of third images by inputting the adjusted plurality of second images into a machine learning model; each of the plurality of second images includes an overlap region that overlaps with an adjacent second image; the plurality of second images include an image with a predetermined number of pixels and an image with a number of pixels smaller than the predetermined number of pixels; an image processing method, wherein in the adjusting step, an image having a number of pixels smaller than the predetermined number of pixels is adjusted to have the predetermined number of pixels;

[0109] (Method 12) The image processing method described in Method 11, characterized in that in the adjusting step, the predetermined number of pixels is achieved by adding an adjacent additional area around the image having a number of pixels smaller than the predetermined number of pixels.

[0110] (Method 13) 12. The image processing method according to claim 11, wherein in the adjusting step, the predetermined number of pixels is achieved by increasing overlapping areas in the image having a number of pixels smaller than the predetermined number of pixels.

[0111] (Configuration 1) a first means for dividing a first image to generate a plurality of second images smaller than the first image; and second means for generating a plurality of third images by inputting the plurality of second images into a machine learning model; each of the plurality of second images includes an overlap region that overlaps with an adjacent second image; 10. An image processing device, wherein the plurality of second images include an image including an overlapping region with a first number of pixels and an image including an overlapping region with a second number of pixels.

[0112] (Program 1) A program causing a computer to execute the image processing method according to any one of Methods 1 to 13. [Explanation of symbols]

[0113] S204 First step S205 Second process

Claims

1. a first step of dividing a first image to generate a plurality of second images smaller than the first image; a second step of generating a plurality of third images by inputting the plurality of second images into a machine learning model; each of the plurality of second images includes an overlap region that overlaps with an adjacent second image; An image processing method characterized in that the multiple second images are generated to include an image including an overlapping area with a first number of pixels and an image including an overlapping area with a second number of pixels different from the first number of pixels.

2. 2. The image processing method according to claim 1, further comprising the step of generating the first image by adding an additional region adjacent to the periphery of the input image to the input image.

3. In the first step, the overlapping region in the second image located at the outermost edge of the plurality of second images is set to the first pixel number; 2. The image processing method according to claim 1, wherein the second number of pixels is set to the overlapping area in the second image other than the second image located at the outermost edge.

4. 2. The image processing method according to claim 1, wherein the first number of pixels is one of a predetermined fixed value, a value based on instructions from a user, and a value based on an imaging device used to acquire the first image or a shooting mode of the imaging device.

5. 2. The image processing method according to claim 1, wherein the second number of pixels is one of a predetermined fixed value, a value based on instructions from a user, and a value based on the imaging device used to acquire the first image or the shooting mode of the imaging device.

6. The image processing method according to claim 2, characterized in that the number of pixels in the additional area is one of a predetermined fixed value, a value based on instructions from a user, or a value based on the imaging device used to acquire the input image or the shooting mode of the imaging device.

7. 2. The image processing method according to claim 1, further comprising a third step of generating a fourth image by combining the plurality of third images.

8. 8. The image processing method according to claim 7, wherein in the third step, the plurality of third images are synthesized by removing at least a portion of the overlapping regions in the plurality of third images and then connecting adjacent third images in the spatial direction.

9. 8. The image processing method according to claim 7, wherein in the third step, the plurality of third images are synthesized by taking a weighted average of the plurality of third images.

10. a first step of dividing a first image to generate a plurality of second images smaller than the first image; a second step of generating a plurality of third images by inputting the plurality of second images into a machine learning model; each of the plurality of second images includes an overlap region that overlaps with an adjacent second image; In the first step, the plurality of second images are generated, each of which includes an image including one or more overlapping regions with a horizontally adjacent image having a first number of pixels and one or more overlapping regions with a vertically adjacent image having a second number of pixels, and an image including one or more overlapping regions with a horizontally or vertically adjacent image having a third number of pixels; The image processing method, wherein the first number of pixels, the second number of pixels, and the third number of pixels are different from one another.

11. a first step of dividing a first image to generate a plurality of second images smaller than the first image; adjusting the plurality of second images; a second step of generating a plurality of third images by inputting the adjusted plurality of second images into a machine learning model; each of the plurality of second images includes an overlap region that overlaps with an adjacent second image; the plurality of second images include an image having a predetermined number of pixels and an image having a number of pixels smaller than the predetermined number of pixels; an image processing method, wherein in the adjusting step, an image having a number of pixels smaller than the predetermined number of pixels is adjusted to have the predetermined number of pixels;

12. 12. The image processing method according to claim 11, wherein in the adjusting step, the predetermined number of pixels is adjusted by adding an additional area adjacent to the periphery of the image having a number of pixels smaller than the predetermined number of pixels.

13. 12. The image processing method according to claim 11, wherein in the adjusting step, the predetermined number of pixels is achieved by increasing an overlapping area in an image having a number of pixels smaller than the predetermined number of pixels.

14. a first means for dividing a first image to generate a plurality of second images smaller than the first image; and second means for generating a plurality of third images by inputting the plurality of second images into a machine learning model; each of the plurality of second images includes an overlap region that overlaps with an adjacent second image; 10. An image processing apparatus according to claim 9, wherein the plurality of second images include an image including an overlapping region with a first number of pixels and an image including an overlapping region with a second number of pixels.

15. A program causing a computer to execute the image processing method according to any one of claims 1 to 13.

Images