Image processing method, image processing program, and image processing apparatus
The image processing method generates a first map based on pixel and resampling information to enhance CNN-based sharpening accuracy for cropped or resampled images, addressing precision issues in existing CNN-based sharpening techniques.
Patent Information
- Application Number
- JP2023072727
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-26
- Publication Date
- 2024-11-08
AI Technical Summary
Existing image sharpening methods using convolutional neural networks (CNNs) are less accurate when the captured image is generated by cropping or resampling, as the pixel position maps no longer match the actual image positions, leading to decreased sharpening precision.
An image processing method that includes generating a first map based on the number of pixels and resampling information, and inputting this map along with the image to a machine learning model to sharpen blur caused by optical systems, using CNNs to accurately adjust sharpening based on image plane coordinates and other factors.
The method achieves high-precision sharpening of blurred images even after cropping or resampling, ensuring accurate blur correction by aligning the CNN's sharpening process with the actual image positions and optical system characteristics.
Smart Images

Figure 2024157989000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an image processing technique for sharpening a captured image. [Background technology]
[0002] Non-Patent Document 1 discloses a method of sharpening the blur caused by the aberration of the optical system in a captured image using a CNN (convolutional neural network). Since the aberration of the optical system changes depending on the position in the captured image, the blur caused by the aberration can be sharpened with high precision by inputting a map indicating the position (coordinates) of each pixel to the CNN together with the captured image. Each pixel in the map indicates the horizontal and vertical coordinates of the corresponding pixel in the captured image. The horizontal and vertical coordinates are expressed so that the coordinates of the upper left pixel in the captured image are (-1,-1) and the coordinates of the lower right pixel are (1,1). [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] S. Kim, et al., “Non-stationary deep network for restoration of non-Stationary lens blur”, Pattern Recognition Letters, Volume 108, 1 June 2018, Pages 62-69. Summary of the Invention [Problem to be solved by the invention]
[0004] The method disclosed in Patent Document 1 has a problem in that the accuracy of sharpening decreases when the captured image to be sharpened is an image generated by cropping using only some of the pixels of the image sensor, or when the image is an image that has been resampled with enlargement or reduction. This is because cropping or resampling causes the map indicating the positions of pixels to no longer match the actual positions of the pixels in the captured image.
[0005] The present invention provides an image processing method and the like that makes it possible to sharpen blur with high precision in a captured image obtained by performing cropping and resampling. [Means for solving the problem]
[0006] An image processing method according to one aspect of the present invention includes a step of acquiring a first image generated by performing resampling on an image obtained by imaging using an optical system and an image sensor, a map generating step of generating a first map based on the number of pixels of the first image and first information related to resampling, and an image generating step of generating a second image in which the blur caused by the optical system included in the first image is sharpened by inputting input data based on the first image and the first map into a machine learning model. The value of each pixel in the first map is a value related to sharpening performed on the corresponding pixel of the first image. Note that a program for causing a computer to execute processing according to the above image processing method also constitutes another aspect of the present invention.
[0007] An image processing device according to another aspect of the present invention includes an acquisition means for acquiring a first image generated by performing resampling on an image obtained by imaging using an optical system and an imaging element, a map generation means for generating a first map based on the number of pixels of the first image and first information related to the resampling, and an image generation means for generating a second image in which the blur caused by the optical system included in the first image has been sharpened by inputting input data based on the first image and the first map into a machine learning model. The value of each pixel in the first map is a value related to sharpening performed on the corresponding pixel of the first image. Note that an imaging device or an image processing system including the image processing device also constitute another aspect of the present invention.
[0008] A method for manufacturing a machine learning model according to another aspect of the present invention includes the steps of acquiring a ground truth image and a training image generated by performing resampling on the image, generating a first map based on the number of pixels of the training image and first information related to resampling, inputting input data based on the training image and the first map into the machine learning model to generate an estimated image in which blurring included in the training image is sharpened, and updating weights of the machine learning model based on the ground truth image and the estimated image. The value of each pixel in the first map is a value related to sharpening performed on the corresponding pixel of the first image. A training device according to another aspect of the present invention includes a means for acquiring a ground truth image and a training image generated by resampling the image, a means for generating a first map based on the number of pixels of the training image and first information related to the resampling, a means for generating an estimated image in which blurring included in the training image is sharpened by inputting input data based on the training image and the first map to a machine learning model, and a means for updating weights of the machine learning model based on the ground truth image and the estimated image. The value of each pixel in the first map is a value related to sharpening performed on the corresponding pixel of the first image. Effect of the Invention
[0009] According to the present invention, it is possible to sharpen blur with high accuracy in a first image obtained by performing cropping and resampling. [Brief description of the drawings]
[0010] [Figure 1] 4 is a flowchart showing image processing in the first embodiment. [Diagram 2] FIG. 1 is a block diagram showing a configuration of an image processing system according to a first embodiment. [Diagram 3] FIG. 1 is a diagram showing the appearance of an image processing system according to a first embodiment. [Figure 4] 5A to 5C are diagrams for explaining imaging modes in the first embodiment. [Diagram 5] 1 is a flowchart showing a flow of training of a machine learning model in the first and second embodiments. [Figure 6] FIG. 4 is a diagram showing an example of a first map in the first embodiment. [Figure 7] FIG. 2 is a diagram showing the configuration of a machine learning model in the first embodiment. [Figure 8] 5 is a graph showing the relationship between the image plane coordinate and the PSF in the first embodiment. [Figure 9] FIG. 11 is a block diagram showing the configuration of an image processing system according to a second embodiment. [Figure 10] FIG. 11 is a diagram showing the appearance of an image processing system according to a second embodiment. [Figure 11] FIG. 11 is a diagram showing a color filter array in the second embodiment. [Figure 12] FIG. 11 is a graph showing the relationship between the resolution performance of the optical system and a first map in the second embodiment. [Figure 13] FIG. 11 is a diagram illustrating a configuration of a machine learning model in a second embodiment. [Figure 14] 10 is a flowchart showing image processing in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0012] Before describing specific Examples 1 and 2, an overview of each embodiment will be described. Images captured by various imaging devices such as digital still cameras, video cameras, broadcast cameras, and security cameras contain blurs, which are image components corresponding to blurs caused by aberrations and diffraction in the optical system. The shapes of these blurs differ depending on the position on the captured image. A machine learning model that performs blur sharpening can be obtained by training the machine learning model using multiple pairs of blurred images (training images) containing blurs of various shapes and ground truth images that do not contain blurs (or contain only blurs smaller than those of the blurred images).
[0013] However, if a machine learning model is trained by inputting only blurred images, the model is trained to sharpen the blur of various shapes on an average basis, without sharpening the blur in accordance with each shape. When using such a machine learning model, the blur is over-sharpened in areas with small blur, such as near the optical axis of the optical system (the center of the captured image), while the blur is not sufficiently sharpened in areas with large blur, such as the periphery.
[0014] This problem can be solved by inputting the following first map to the machine learning model together with the blurred image. The first map has, for example, values related to positions (image plane coordinates) on the image sensor corresponding to each pixel in the captured image. The machine learning model learns sharpening by associating the value of each pixel in the first map with the blur given to each pixel (one or more pixels) of the blurred image corresponding to the pixel in the map. In this way, the trained machine learning model performs sharpening suited to the shapes of various shapes of blur present in the captured image. As a result, a second image can be obtained as an output image in which the blur has been sharpened with high accuracy.
[0015] On the other hand, the imaging device may be provided with a plurality of imaging modes that differ in whether or not a process of cropping or resampling with enlargement or reduction is performed. A correct first map (a map related to image plane coordinates in the above example) corresponding to a captured image cannot be generated only from a captured image on which a process of cropping or resampling has been performed. For this reason, in each embodiment, the first map is generated based on the number of pixels of the first image, which is at least a part of the captured image on which resampling has been performed, and first information related to the resampling performed on the first image. By using the first information, a correct first map can be generated even if cropping or resampling has been performed on the captured image. By inputting input data based on the first map and the first image generated in this way into a trained machine learning model, it is possible to perform high-precision blur sharpening even on the first image on which cropping or resampling has been performed. In the following description, information about XX and values related to XX include not only information or values that directly indicate XX, but also information or values that can be converted to XX or that correspond to XX. Values related to XX also include cases where XX is determined using a value as one of the arguments. For this reason, XX also changes when the value changes, but also includes cases where multiple different values become the same XX. EXAMPLES
[0016] An image processing system in the first embodiment will be described. In the first embodiment, each pixel of the first map has a value representing the position in the image sensor of the corresponding pixel of the first image (an image obtained by performing cropping or resampling on a captured image). This will be described in detail later. Fig. 2 shows the configuration of the image processing system in the first embodiment of the present invention. Fig. 3 shows the external appearance of the image processing system 100.
[0017] The image processing system 100 includes a training device 101, a sharpening device 102, an imaging device 103, and a lens device 104. In this embodiment, the imaging device 103 is an interchangeable lens imaging device, and multiple types of lens devices 104 can be connected. The lens device 104 collects light incident from a subject space using an optical system 141 to form a subject image. The subject image is photoelectrically converted by an imaging element 131 in the imaging device 103, and a calculation unit 134 generates a captured image using an imaging signal from the imaging element 131.
[0018] The subject image formed by the optical system 141 contains blur due to aberration and diffraction of the optical system 141, and the captured image also contains a blur component (hereinafter simply referred to as blur), which is an image component corresponding to this blur. The effective pixel area of the image sensor 131 is, for example, 36 [mm] horizontal × 24 [mm] vertical, and contains 6720 × 4480 imaging pixels. Therefore, the number of pixels of the captured image is also 6720 × 4480 pixels. However, the size and number of pixels of the effective pixel area are not limited to these. The captured image may be either a still image or a frame image of a moving image.
[0019] A plurality of imaging modes can be set in the imaging device 103. In an imaging mode in which cropping or resampling is performed, a captured image or a part thereof that has been cropped or resampled by the calculation unit 134 becomes the first image, and in other imaging modes, a captured image that has not been cropped or resampled becomes the first image. The first image is stored in the storage unit 132.
[0020] Specifically, the imaging device 103 has the following three imaging modes. In the full pixel imaging mode, the entire captured image without cropping and resampling is used as the first image. In the 4K crop imaging mode, resampling is not performed and a 3840 x 2160 pixel area in the center of the captured image is cropped to use as the first image. In the 4K oversampling imaging mode, cropping and resampling are performed on the captured image to use as the first image.
[0021] The 4K oversampling imaging mode will be described in more detail with reference to Figs. 4(A) and (B). In the 4K oversampling imaging mode, the calculation unit 134 crops a 5760 x 3240 pixel region (crop region) 202 in the center of the captured image 201 as shown in Fig. 4(A). In Fig. 4(B), each pixel in the crop region 202 is represented by a broken line. The calculation unit 134 performs resampling on the crop region 202 at a resampling rate of 2 / 3 to obtain a first image of 3840 x 2160 pixels. The solid lines in Fig. 4(B) indicate each pixel of the first image obtained by resampling. The calculation unit 134 generates the first image by binning a 1.5 x 1.5 pixel region in the crop region 202 as one pixel. The binning of non-integer pixels is performed by weighted averaging. The S / N ratio can be improved more than in the 4K crop imaging mode by binning. It should be noted that the imaging modes are not limited to the above three, and the number of crop pixels and the resampling rate may also be other values.
[0022] The first image may be an undeveloped RAW image or a developed image. Processing required for development is executed by the calculation unit 134. The image sensor 131 has color filters in a Bayer array of RGB (Red, Green, Blue). However, this is not limited to this, and a honeycomb or Quad Bayer array (which will be described in the second embodiment) may also be used. When resampling is executed on a RAW image in a Bayer array, demosaicing is performed before resampling. If necessary, the image may be re-Bayerized after resampling.
[0023] The first image is transmitted from the imaging device 103 to the sharpening device 102 as an image processing device constituted by a computer via the communication unit 133 and the communication unit 122. As described above, the first image has blur caused by aberration and diffraction of the optical system 141. The sharpening device 102 sharpens the blur of the first image to generate a second image. Specifically, the calculation unit 124 in the sharpening device 102 uses a machine learning model to perform sharpening that reflects the cropping and resampling performed on the first image to generate a second image. The machine learning model is trained by the training device 101, and is acquired in advance by the sharpening device 102 via the communication unit 112 and the communication unit 122 and stored in the storage unit 121. The training of the machine learning model and the sharpening of blur by the trained machine learning model will be described in detail later. The second image is displayed on the display unit 125 and stored in the storage unit 121.
[0024] The sharpening device 102 described in this embodiment is realized by a computer owned or used by a user who uses the imaging device 103. However, it may be realized by a so-called cloud-based computer owned or used by a provider who provides a service for sharpening a blurred image using a machine learning model via a network such as the Internet. Also, an image processing system may be configured that includes the sharpening device 102 and a control device capable of communicating with the sharpening device 102. In this image processing system, the control device may function as a transmission means that transmits a request to cause the sharpening device 102 to execute processing on a first image, and the sharpening device 102 may function as a processing means that executes processing on the first image in response to the request.
[0025] Using the flowchart of FIG. 5, the training of the machine learning model (determination of the set of weights) executed by the training device 101, that is, the manufacturing method of the machine learning model, will be described. The training device 101 is configured by a computer having a storage unit 111, a communication unit 112, an acquisition unit 113, a calculation unit 114, and an update unit 115, and executes the processing of each step shown in FIG. 5 according to a program. Here, a CNN (Convolutional Neural Network) is used as the machine learning model, but other machine learning models may be used. Usable machine learning models include neural networks, genetic programming, Bayesian networks, etc. In addition to the above CNN, neural networks include MLP (Multi-Layer Perceptron), Transformer, GAN (Generative Adversarial Network), RNN (Recurrent Neural Network), etc. The set of weights is a plurality of variables determined by training in the machine learning model. For example, the filter and bias of the convolution layer, and the weight and bias of the full connection layer.
[0026] In step S101, the training device 101 (acquisition unit 113) acquires one or more pairs of training input data and a correct answer image from the storage unit 111. The training input data is data to be input to the CNN during training. The training input data includes a blurred image and the first to fourth maps.
[0027] A blurred image is a degraded image that contains blur to be sharpened. A correct image is an image that contains the same subject as the corresponding blurred image and has less blur (or no blur) than the blurred image. The blurred image and the correct image can be prepared by real photography, imaging simulation, CG (Computer Graphics), etc.
[0028] In this embodiment, sharpening for all blurs occurring in the optical system 141 is learned all at once. For this reason, blurred images including various blurs occurring in the optical system 141 and corresponding answer images are stored in the storage unit 111. The blur of the optical system 141 is determined by the state of the optical system 141 (focal length, aperture value, focus distance, etc.) and the horizontal and vertical image plane coordinates. The focus distance is the distance between the subject on which the optical system 141 is focused and the image sensor 131 or the optical system 141.
[0029] Furthermore, since the imaging device 103 of this embodiment is of an interchangeable lens type, the optical system 141 may be connected to imaging devices having imaging elements of different sizes and numbers of pixels. This may result in changes in the colors acquired by the imaging elements, the pixel pitch, and the action of the optical low-pass filter. In this embodiment, as an example, the imaging elements of imaging devices that can be connected to the optical system 141 all have an RGB Bayer array, and the colors are fixed to RGB.
[0030] The multiple blurred images stored in the memory unit 111 include blurs that vary in various ways within the range of values that can be taken by the image plane coordinates, the pixel pitch of the image sensor 131, and the optical low-pass filter, in addition to the focal length, aperture value, and focus distance of the optical system 141.
[0031] In the first embodiment, the first map is a map having values related to image plane coordinates corresponding to the blur in the blurred image. Since the image sensor 131 is disposed on the image plane, the image plane coordinates are synonymous with the position on the image sensor 131. The first map has, as the value of each pixel thereof, a value indicating the image plane coordinates corresponding to the blur in the blurred image. Since the image plane coordinates are coordinates in two directions, horizontal and vertical, the first map has two channel values for each pixel thereof.
[0032] 6(A) and (B) show examples of the first map near the optical axis of the optical system 141 when the image plane coordinates on the optical axis are set as the origin (0,0). Fig. 6(A) shows the first map when the pixel pitch of the image sensor 131 is 5.4 [um], and Fig. 6(B) shows the first map when the pixel pitch is 8.1 [um]. The x marks indicate the position of the optical axis. The image plane coordinates may be expressed in polar coordinates.
[0033] The value of each pixel in the first map may be normalized by a value such as the maximum image height (18 [mm], 12 [mm]). The number of pixels in the first map is determined based on the number of pixels in the blurred image. If the number of pixels in the blurred image is W x H x C pixels in horizontal x vertical x channel notation (C = 1 if the blurred image is a RAW image in a Bayer array, and C = 3 if the image is after demosaicing), the number of pixels in the first map is, for example, W x H x 2 pixels. In addition, if the blurred image is rearranged later to (W / 2) x (H / 2) x 4C pixels, the number of pixels in the first map may be (W / 2) x (H / 2) x 2. In this case, the first map has the image plane coordinates of every two pixels of the blurred image before rearrangement as the value of each pixel.
[0034] Furthermore, when the blurred image is generated, for example, by an imaging simulation in which blur is added to an original image that is a subject, there is no need to add shift-variant blur as in the actual optical system 141. It is sufficient to generate a plurality of blurred images including blurs at different image plane coordinates by regarding one blurred image as including shift-invariant blur at a specific image plane coordinate. In this case, unlike Figs. 6(A) and (B), the first map is a map having uniform values in all pixels.
[0035] The second map is a map having values related to the noise intensity of the blurred image. The noise here refers to shot noise and the like generated in the imaging element (various imaging elements that can be combined with the optical system 141) 131. The multiple blurred images stored in the storage unit 111 contain noise of various intensities that may occur when imaging is performed using the optical system 141. When the blurred image is acquired by actual imaging, the noise intensity can be obtained from the imaging element 131 used in the imaging and the ISO sensitivity at the time of imaging. Also, when the blurred image is generated by imaging simulation, the known noise intensity given to the blurred image can be obtained. The noise intensity can be expressed by the standard deviation of noise for a specific luminance, etc. Also, a second map having each coefficient when the variance of noise for luminance is approximated by an n-th order equation (n is a natural number) as the noise intensity for each channel may be generated. By inputting the second map into the CNN, there is an advantage that the CNN can easily distinguish between noise and the subject. The number of pixels of the second map is also determined based on the number of pixels of the blurred image, similar to the first map. Note that if the ground truth image is noise-free, the CNN is trained to perform denoising as well as sharpening. If the ground truth image contains noise that is similar in intensity to and correlated with the noise in the blurred image, the CNN is trained to perform blur sharpening with suppressed noise variation.
[0036] The third map is a map having values related to the size of one pixel of the blurred image in real space (pixel pitch). The size of the blur of the optical system 141 for the pixel varies depending on the pixel pitch. Therefore, when the image pickup devices that can be combined with the optical system 141 have different pixel pitches or when resampling can be performed on the captured image, the accuracy of the blur sharpening can be improved by inputting the third map into the CNN. The multiple blurred images stored in the storage unit 111 include various pixel pitches. The number of pixels of the third map is also determined based on the number of pixels of the blurred image, similar to the first map.
[0037] The fourth map is a map having values related to the state of the optical system 141 corresponding to the blur of the blurred image. As described above, the state of the optical system 141 includes at least one of the focal length, the aperture value, and the focus distance. The blur caused by aberration or diffraction changes depending on the state of the optical system 141. For this reason, by inputting the fourth map into the CNN, the CNN is trained to perform sharpening suitable for the shape of the blur. The fourth map in this embodiment has values indicating the focal length, the aperture value, and the focus distance for each channel. The number of pixels in the fourth map is also determined based on the number of pixels in the blurred image, similar to the first map. Note that the fourth map does not need to be used under conditions such as the optical system 141 being a fixed-focus lens or the aperture value being fixed.
[0038] In step S102, the training device 101 (the calculation unit 114) inputs training input data to a machine learning model (CNN) to generate a blur-sharpened image in which the blur of the blurred image is sharpened. FIG. 7 shows a flow of generating a blur-sharpened image. The calculation unit 114 connects the blurred image 210, the first map 211, the second map 212, the third map 213, and the fourth map 214 in the channel direction and inputs them to the CNN. However, the input method is not limited to this. For example, the blurred image 210, the first map 211, the second map 212, the third map 213, and the fourth map 214 may be input to individual convolution layers and converted into feature maps, and then the feature maps may be connected in the channel direction.
[0039] By inputting not only the blurred image 210 but also the first map 211 into the CNN for training, it becomes possible to sharpen the blur caused by the aberration and diffraction of the optical system 141, which changes depending on the image plane coordinate, with high accuracy. This principle will be described in detail with reference to Figures 8(A) and (B).
[0040] Fig. 8(A) shows image plane coordinates, and Fig. 8(B) shows the PSF (Point Spread Function) of the optical system 141 for several sample points of the image plane coordinates. Since the PSF changes depending on the state (focal length, aperture value, and focus distance) of the optical system 141, the PSF for a certain specific state is shown here. In addition, the PSF actually differs for each RGB color, but here, the PSF for white light is shown for simplicity.
[0041] Since the shape of the PSF changes depending on the image plane coordinates, CNN cannot achieve high-precision sharpening of blur unless it performs sharpening corresponding to the blur (PSF) for each image plane coordinate. However, if the first map is not present in the input to CNN, CNN cannot identify the shape of the blur present in the blurred image, and high-precision sharpening cannot be achieved. In contrast, if the first map is input to CNN, CNN can identify the blur (PSF) present in the blurred image from the values of the first map (image plane coordinates), and therefore it becomes possible to achieve high-precision sharpening suitable for the shape of the blur.
[0042] As shown in FIG. 8B, the PSF on the optical axis (0,0) has an isotropic spread in all directions, but the PSF at image heights other than the optical axis has a different spread depending on the direction. Since the PSF is linearly symmetric with respect to the axis in the meridional direction of the image plane coordinates, the spread of the PSF in each direction differs depending on the azimuth of the image plane coordinates even at the same image height. This property appears in the PSFs at the image plane coordinates (12,0), (8.5,8.5), and (0,12) in FIG. 8B. The fact that the spread of the PSF differs in each direction means that sharpening with different strengths must be performed in each direction in order to sharpen the blur with high accuracy. The strength of the sharpening in each of the multiple directions is linked to the shape of the PSF, and the shape of the PSF is specified by the value of the first map as described above. In other words, the value of each pixel in the first map is a value related to the strength of the sharpening in each of the multiple directions performed on the corresponding pixel in the blurred image.
[0043] The CNN to which the training input data is input generates a residual component 215 for sharpening the blur of the blurred image 210, and generates a blur-sharpened image 216 by taking the sum of the residual component 215 and the blurred image 210. However, the blur-sharpened image 216 may be generated directly from the CNN without generating the residual component 215.
[0044] In step S103, the training device 101 (update unit 115) updates the set of weights of the CNN based on the difference (error) between the blurred and sharpened image (estimated image) 216 and the correct image. In this embodiment, the MSE (Mean Square Error) between the blurred and sharpened image 216 and the correct image is used. However, the difference is not limited to this, and MAE (Mean Absolute Error) or the like may be used. In addition, backpropagation or the like is used for updating.
[0045] In step S104, the training device 101 (update unit 115) determines whether the training of the CNN is complete, and if not, returns to step S101 to acquire one or more new sets of training input data and ground truth images. If the training is complete, the set of weights is stored in the storage unit 111.
[0046] Next, a process of sharpening the blur of the first image using a trained CNN (machine learning model) executed by the sharpening device 102 will be described with reference to the flowchart in Fig. 1. As shown in Fig. 2, the sharpening device 102 has a storage unit 121, a communication unit 122, an acquisition unit (acquisition means) 123, a calculation unit (map generation means, image generation means) 124, and a display unit 125, and executes the process of each of the following steps according to a program.
[0047] In step S201, the sharpening device 102 (acquisition unit 123) acquires a first image in which blurring occurs due to aberration or diffraction of the optical system 141. The first image is a result of cropping or resampling performed on a captured image according to the imaging mode of the imaging device 103.
[0048] In step S202, the sharpening device 102 (acquisition unit 123) acquires a CNN (a network structure and a set of weights) used to sharpen the blur contained in the first image. In this embodiment, multiple sets of CNN weights are prepared for each type of lens device, and the network structure is the same regardless of the type of lens device. However, the weight set is not limited to this, and may be prepared in common for multiple types of lens devices, or multiple sets of weights may be prepared for one type of lens device. In addition, the network structure may be changed depending on the type of lens device.
[0049] The acquisition unit 123 acquires, from the storage unit 121, the structure of the CNN network and a set of weights corresponding to the lens device 104 used at the time of capturing an image to capture the first image. Information about the lens device 104 used at the time of capturing an image may be acquired, for example, from metadata of the first image. The storage unit 142 stores identification information of the lens device 104. When the lens device 104 and the imaging device 103 are connected or when imaging is performed by the imaging device 103, the imaging device 103 acquires the identification information via the communication units 133 and 143 and writes the identification information as metadata in the first image.
[0050] The following steps S203 to S207 correspond to a map generating step. In step S203, the sharpening device 102 (acquisition unit 123) acquires the following first information, second information, third information, and fourth information.
[0051] The first information is information related to resampling performed on the first image. In this embodiment, the acquisition unit 123 acquires information on the resampling rate as the first information. However, the number of pixels of the captured image used to generate the first image (the number of pixels of the crop area 202 shown in FIG. 4(A)) may be acquired as the first information. The resampling rate can be calculated from the ratio of the number of pixels to the number of pixels of the first image. Note that a resampling rate of 1 indicates that no resampling has been performed.
[0052] The second information is information about sampling of the image sensor 131 used for imaging to obtain the first image. In this embodiment, the acquisition unit 123 acquires, as the second information, information about the pixel pitch of the image sensor 131. However, as the second information, information about the Nyquist frequency of the image sensor 131 may be acquired, or information about the horizontal and vertical lengths and the number of pixels of the effective pixel area of the image sensor 131 may be acquired.
[0053] The third information is information about noise of the image sensor 131 used for capturing an image to obtain the first image. In this embodiment, the acquisition unit 123 acquires information about the type of image sensor and the ISO sensitivity at the time of capturing an image as the third information. However, the third information may also be information (such as signal dispersion) about an optical black area outside the effective pixel area of the captured image corresponding to the first image.
[0054] The fourth information is information about the state of the optical system 141 used in imaging to obtain the first image. In this embodiment, the acquisition unit 123 acquires information about the focal length, aperture value, and focus distance of the optical system 141 as the fourth information. However, the fourth information may include other elements that change the blur, for example, the type of a wide converter, a teleconverter, and a half-neutral-reduction filter attached to the lens device 104. In addition, information about elements that have no degree of freedom in the optical system 141 (for example, the focal length of a single-focus lens) is not necessary. The acquisition unit 123 acquires the above first to fourth information, for example, from metadata added to the first image.
[0055] In step S204, the sharpening device 102 (the calculation unit 124) generates a first map based on the number of pixels in the first image, the first information, and the second information. As in training the CNN, each pixel in the first map has a value related to a position (image plane coordinates) on the image sensor 131 that corresponds to the corresponding pixel in the first image.
[0056] In this embodiment, the calculation unit 124 divides the pixel pitch of the image sensor 131 obtained from the second information by the resampling rate of the resampling performed on the first image obtained from the first information. This determines the effective pixel pitch (length on the image plane) of one pixel in the first image in real space. For example, the pixel pitch of the image sensor 131 is about 5.4 [μm], and the resampling rate in the full pixel imaging mode is 1, so the effective pixel pitch of the first image is also 5.4 [μm]. On the other hand, the resampling rate in the 4K oversampling imaging mode is 2 / 3, so the effective pixel pitch of the first image is 8.1 [μm]. Due to the resampling, the blur caused by the aberration and diffraction of the optical system 141 present in the first image is smaller in size relative to the pixel compared to the captured image before resampling. If the captured image was upsampled (high pixelization) to generate the first image, the resampling rate would be greater than 1, so the effective pixel pitch of the first image would be smaller than 5.4 [μm]. In this case, the blur present in the first image is greater than that in the captured image.
[0057] The calculation unit 124 generates the first map in the same manner as in training the CNN, i.e., in the same manner as in the method shown in Fig. 6. Specifically, the position of the optical axis of the optical system 141 is set as the origin, and horizontal and vertical image plane coordinates (positions on the image sensor 131) corresponding to each pixel of the first image are obtained based on the effective pixel pitch of the first image. As a result, even if resampling is performed on the first image, each pixel of the first map has a value related to the position on the image sensor 131 of the corresponding pixel in the first image.
[0058] It should be noted that the second information is not necessarily required when the pixel pitch of an image sensor that can be combined with the lens device 104 is constant. In this case, a first map corresponding to the pixel pitch is stored in advance, and the values of the first map are corrected based on the first information, so that a first map corresponding to a first image can be generated.
[0059] Since the input data input to the CNN is the same as that during training shown in FIG. 7, the first map has the same number of horizontal and vertical pixels as the first image. If the blurred image is rearranged from W×H×C pixels to (W / 2)×(H / 2)×4C pixels during training, the calculation unit 124 rearranges the first image to W 1 ×H 1 ×C 1 From pixels (W 1 / 2)×(H 1 / 2)×4C 1 Pixels are rearranged, where W 1 , H 1 and C 1 are the number of pixels of the first image in the horizontal, vertical and channel directions, respectively. With this sorting, the first map is also sorted by (W 1 / 2)×(H 1 / 2) × 2 pixels. In this way, the number of pixels in the first map is determined based on the number of pixels in the first image.
[0060] The first information and the second information are not limited to the resampling rate and pixel pitch, respectively. For example, the first information may be the number of pixels of the captured image used to generate the first image, and the second information may be the horizontal and vertical lengths (36 [mm] x 24 [mm]) and the number of pixels (6720 x 4480 pixels) of the effective pixel area of the image sensor 131. In the case of the 4K oversampling imaging mode, the number of pixels of the captured image used to generate the first image is 5760 x 3240 pixels. The maximum image height in the first image can be obtained by multiplying the ratio of the number of pixels of the captured image used to generate the first image divided by the number of pixels of the effective pixel area of the captured image by the horizontal and vertical lengths of the effective pixel area. This allows the value of the pixel located at the edge of the first map to be obtained. Since the number of pixels of the first map is determined by the number of pixels of the first image, if the values of the pixels at the origin and the edge are known, the values of all the pixels of the first map can be obtained.
[0061] Each pixel of the first map has a value related to the image plane coordinates of a corresponding pixel in the first image. Therefore, the first map has values that are not spatially uniform (have a distribution) for multiple pixels in the first image. Also, each pixel of the first map has a value that is not dependent on the structure (edges, etc.) of the object included in the first image.
[0062] In step S205, the sharpening device 102 (the calculation unit 124) generates a second map based on the number of pixels of the first image, the first information, and the third information. The second map has values related to the noise intensity of the first image, as in training. From the third information, the noise intensity of the captured image is identified, and if resampling has not been performed on the first image, the noise intensity is the same as that of the captured image. However, if the resampling rate of the first image is not 1, the noise intensity deviates from that of the captured image, and therefore it is necessary to generate the second map based on the first information.
[0063] When the resampling rate is less than 1, noise intensity is reduced by binning. In 4K oversampling imaging mode, the resampling rate is 2 / 3, so the pixel pitch of 5.4 [μm] is equivalent to 8.1 [μm], and the standard deviation of the noise in this case is 2 / 3 of that of the captured image before resampling.
[0064] On the other hand, when the resampling rate of the first image is greater than 1, that is, when upsampling is performed on the captured image, the change in noise intensity depends on the upsampling method. When bilinear interpolation, bicubic interpolation, or the like is used as the upsampling method, the noise is blurred and weakened by the interpolation process. In addition, in upsampling using a machine learning model, the noise intensity of the first image may increase depending on the training method. For this reason, it is necessary to correct the noise intensity according to the upsampling method used. The horizontal and vertical pixel counts of the second map are determined based on the pixel count of the first image, similar to the first map.
[0065] In step S206, the sharpening device 102 (the calculation unit 124) generates a third map based on the number of pixels of the first image, the first information, and the second information. The third map has a value related to the effective pixel pitch of the first image. The magnitude of the blur caused by the aberration and diffraction of the optical system 141 for the pixels of the first image changes depending on the magnitude of the effective pixel pitch. The effective pixel pitch of the first image is a value obtained by dividing the pixel pitch of the image sensor 131 by the resampling rate. Note that, if the pixel pitch of the image sensor that can be combined with the lens device 104 is constant, the third map may be generated only from the resampling rate. Also, the effective pixel pitch of the first image may be obtained from the spatial differentiation of the first map. The horizontal and vertical pixel numbers of the third map are determined based on the pixel number of the first image, similar to the first map.
[0066] In step S207, the sharpening device 102 (the calculation unit 124) generates a fourth map based on the number of pixels of the first image and the fourth information. The fourth map has values related to the state of the optical system 141 at the time of capturing the first image. When the blur varies greatly depending on the state of the optical system 141, the fourth map is added to the input of the CNN, so that the CNN can identify the blur given to each pixel of the first image, and the accuracy of the sharpening of the blur is improved. The horizontal and vertical pixel numbers of the fourth map are determined based on the number of pixels of the first image, similar to the first map. Note that the order of executing the processes of steps S204 to S207 may be any order between steps S203 and S208.
[0067] In step S208 (image generation step), the sharpening device 102 (the calculation unit 124) inputs the input data to the trained CNN to generate a second image. The input data is based on the first image, the first map, the second map, the third map, and the fourth map. Specifically, the calculation unit 124 connects the first image and the first to fourth maps in the channel direction and inputs them to the CNN, similar to the training of the CNN shown in FIG. 7. The CNN generates a residual component that sharpens the blur caused by the aberration and diffraction of the optical system 141 for the first image, and generates a second image by taking the sum of the residual component and the first image. The second image is an image in which the blur caused by the aberration and diffraction of the optical system 141 in the first image has been sharpened.
[0068] If the entire first image cannot be processed at once due to memory restrictions in the calculation unit 124, the first image may be divided into a plurality of partial regions and each partial region may be processed sequentially. In this case, the first to fourth maps are also divided into partial regions for use.
[0069] In addition, after the captured image is cropped or resampled according to the imaging mode, it may be divided into a plurality of partial regions that can be collectively processed by the calculation unit 124, and one of the partial regions may be set as the first image and sharpened sequentially by CNN. In this case, since it is not necessary to hold the high-pixel first to fourth maps before division into partial regions, the required memory is reduced.
[0070] Note that the expression "inputting data to CNN" does not include the process of selecting a set of weights to be used based on the data from a set of multiple weights of CNN. In other words, the expression "inputting data to CNN" refers to the case where data is input to a layer (such as a convolutional layer) that constitutes CNN and a calculation is performed based on the data.
[0071] In addition, upsampling can be performed with CNN along with blur sharpening. By preparing pairs of images in which the blurred image has lower pixel count than the ground truth image when training the CNN, it is possible to train a CNN that performs upsampling along with blur sharpening.
[0072] Also, fifth information related to the optical low pass filter of the image sensor 131 may be acquired, a fifth map having values related to the function of the optical low pass filter may be generated based on the fifth information, and the fifth map may be added to the input data of the CNN. Since multiple types of image sensors can be combined with the optical system 141, the presence or absence of an optical low pass filter, the separation method (horizontal / vertical four-point separation, horizontal two-point separation, etc.), and the cutoff frequency differ depending on the type. By adding a fifth map related to the function of the optical low pass filter, the accuracy of blur sharpening can be further improved.
[0073] The fifth map may have, for example, values of two channels, each of which represents a cutoff frequency in the horizontal and vertical directions. In this case, the fifth map has a value obtained by dividing the Nyquist frequency by the cutoff frequency. If there is no optical low-pass filter, the value of the fifth map will be 0 since the cutoff frequency is infinite. In the case of horizontal two-point separation where the cutoff frequency is 1.5 times the Nyquist frequency, the value of the two channels of the fifth map will be (2 / 3, 0). If it is desired to add cutoff frequencies in directions other than the above two directions, a channel can be added to the fifth map.
[0074] However, the method of generating the fifth map is not limited to this, and for example, the fifth map may be generated so that the class value representing the separation method is the first channel and the second and subsequent channels have values representing the cutoff frequency. When the fifth map is included in the input data, the fifth map must also be included in the training input data when training the CNN.
[0075] According to the above configuration, blur can be sharpened with high accuracy even in the first image obtained by performing cropping and resampling. EXAMPLES
[0076] An image processing system according to a second embodiment of the present invention will be described below. In the second embodiment, a sharpening device serving as an image processing device is built into an imaging device.
[0077] In the second embodiment, each pixel of the first map has a value related to the strength of the sharpening to be performed on the corresponding pixel of the first image, and the value is based on the resolution performance of the optical system used for imaging.
[0078] Fig. 9 shows the configuration of an image processing system 300 of Example 2. Fig. 10 shows the external appearance of the image processing system 300. The image processing system 300 has a training device 301 and an imaging device 302. Fig. 10 shows the external appearance of both the front and back sides of the imaging device 302.
[0079] The imaging device 302 generates (acquires) a captured image by receiving an object image formed by an optical system 321 with an image sensor 322. The optical system 321 is a single-focus lens, and the aperture value is also fixed. Therefore, the change in blur due to the focus distance is small, and the blur changes depending on the image plane coordinates. The image sensor 322 has an effective pixel area with horizontal and vertical lengths of 9.6 [mm] × 7.2 [mm], and the number of pixels is 8000 × 6000. The pixel pitch is 1.2 [μm], and it has a color filter with a Quad Bayer array as shown in FIG. 11.
[0080] The imaging device 302 has the following two imaging modes. The high pixel imaging mode is a mode in which an image is read out from the 8000×6000 pixels of the imaging element 322 without performing resampling. The high sensitivity imaging mode is a mode in which adjacent 2×2 pixels of the same color in the imaging element 322 are resampled by binning, and an image of 4000×3000 pixels is read out. Note that the 2×2 pixel binning may be performed after the image is read out from the imaging element 322.
[0081] An image read out in any one of the above imaging modes is subjected to development processing by the calculation unit 326 to become a first image, and is stored in the storage unit 323. However, a RAW image may also be used as the first image.
[0082] The acquisition unit (acquisition means) 325 acquires the machine learning model and the first image from the storage unit 323. The calculation unit (map generation means, image generation means) 326 generates a second image in which the blur of the first image is sharpened using the machine learning model. The machine learning model is trained by the training device 301, and is acquired in advance by the imaging device 302 via the communication units 312 and 324. The second image is stored in the storage unit 323 of the imaging device 302, and is displayed on the display unit 327. The acquisition unit 325 and the calculation unit 326 constitute a sharpening device.
[0083] In this embodiment, the training of the machine learning model executed by the training device 301 will be described using the flowchart in Fig. 5. The training device 301 is configured by a computer having a storage unit 311, a communication unit 312, an acquisition unit 313, a calculation unit 314, and an update unit 315, and executes the processing of each of the following steps according to a program.
[0084] In step S101, the training device 301 (acquisition unit 313) acquires one or more sets of training input data and a ground truth image from the storage unit 311. In this embodiment, the training input data includes a blurred image, a first map, and a second map. As described above, each pixel of the first map in this embodiment has a value related to the strength of sharpening to be performed on the corresponding pixel in the blurred image. In this embodiment, the first map has two channel values, each of which has a value related to the strength of sharpening in the horizontal and vertical directions. However, the first map is not limited to this, and the number of channels may be further increased to represent the strength of sharpening in various directions. In addition, the strength of sharpening in multiple directions may be averaged and represented by one channel, so that sharpening of the same strength is performed for various directions. Furthermore, the two directions may be the meridional direction and the sagittal direction. When expressing the strength of sharpening in the meridional and sagittal directions, the direction changes depending on the azimuth of the image plane coordinates, so a channel representing the azimuth is added to the first map.
[0085] The value of each pixel in the first map is determined based on the blur added to the corresponding pixel of the blurred image. The blur added to the blurred image is caused by aberration and diffraction of the optical system 321. If the image sensor 322 has an optical system low-pass filter, the blur also includes the effect of the optical system low-pass filter.
[0086] A method for determining pixel values of the first map in this embodiment will be described with reference to Fig. 12. Fig. 12(A) and (B) respectively show horizontal and vertical MTFs (Modulation Transfer Functions) corresponding to the PSF (blur) of white light at three mutually different image plane coordinates. The solid line shows the MTF at the image plane coordinates (0,0) on the optical axis of the optical system 321, the dashed line shows the MTF at the image plane coordinates (3,0), and the dashed two-dot line shows the MTF at the image plane coordinates (4.8,0).
[0087] The pixels of the first map are assigned an evaluation frequency f eval The horizontal and vertical MTFs at the evaluation frequency feval is a frequency obtained by multiplying the Nyquist frequency of the image sensor 322 by a predetermined coefficient, which is 0.8 times the Nyquist frequency in this embodiment. However, the evaluation frequency is not limited to this.
[0088] Evaluation frequency f eval The MTF for image plane coordinate (0,0) in is 0.5 in both the horizontal and vertical directions. Therefore, the value of the pixel in the first map corresponding to the blur at image plane coordinate (0,0) is (0.5,0.5). The MTF for image plane coordinate (3,0) is 0.25 in the horizontal direction and 0.15 in the vertical direction, so the value of the pixel in the first map corresponding to the blur at image plane coordinate (3,0) is (0.25,0.15). The MTF for image plane coordinate (4.8,0) is 0.1 in the horizontal direction and 0 in the vertical direction, so the value of the pixel in the first map corresponding to the blur at image plane coordinate (4.8,0) is (0.1,0).
[0089] In this way, the value of a pixel in the first map is closer to 1 as the resolution performance is higher (the blur is smaller), and closer to 0 as the resolution performance is lower (the blur is larger). Therefore, a CNN trained by inputting the first map together with a blurred image changes the strength of sharpening according to the value of the first map, and the strength of sharpening is increased as the value of the first map is smaller. However, the value of a pixel in the first map may be determined by a method other than the above. For example, the MTF for each color of RGB may be used. The value of a pixel in the first map may also be determined from the peak value of the PSF, the spread of the PSF, the integral value of the MTF, or the like.
[0090] Furthermore, the blur contained in the blurred image does not necessarily have to be blur based on the optical system 321. For example, it may be blur based on Gaussian or the like, which has different variances in the horizontal and vertical directions. In this case, a plurality of blurred images are prepared so as to include blur ranging from very small blur to blur with the same resolution performance as the blur at which the resolution performance of the optical system 321 is minimized.
[0091] The second map has values related to the noise intensity of the blurred image, similar to Example 1. In this example, only one set of CNN weights is trained, rather than multiple sets of weights as in Example 1.
[0092] In step S102, the training device 301 (the calculation unit 314) inputs training input data to the machine learning model to generate a blur-sharpened image. In this embodiment, the blur-sharpened image is generated by the configuration shown in Fig. 13. Specifically, the blurred image 400, the first map 401, and the second map 402 are linked in the channel direction and input to the CNN, which is a machine learning model, to generate a blur-sharpened image 403.
[0093] In step S103, the update unit 315 updates the set of weights of the CNN based on the difference between the blur-sharpened image 403 and the correct image.
[0094] In step S104, the training device 301 (update unit 315) determines whether the CNN training is complete, and if not, returns to step S101 to acquire one or more new sets of training input data and ground truth images. If the training is complete, the set of weights is stored in the storage unit 311.
[0095] 14 shows a process of sharpening the blur of the first image using a trained CNN, which is executed by the imaging device 302 of this embodiment. The image processing device is composed of the acquisition unit 325 and the calculation unit 326 of the imaging device 302, and these execute the processes of the following steps according to a program.
[0096] In step S301, the imaging device 302 (acquisition unit 325) acquires a first image and a trained CNN. The configuration of the CNN network and a set of weights are acquired from the storage unit 323.
[0097] In step S302, the imaging device 302 (acquisition unit 325) acquires first information and third information. The first information is information related to resampling performed on the first image. If the set imaging mode is a high pixel imaging mode, the resampling rate is 1, and if the set imaging mode is a high sensitivity imaging mode, the resampling rate is 0.5. The third information is information related to noise of the image sensor 322 during imaging to acquire the first image, and is ISO sensitivity in this embodiment.
[0098] The following steps S303 and S304 correspond to a map generation step. In step S303, the imaging device 302 (the calculation unit 326) generates a first map based on the number of pixels in the first image and the first information. In this embodiment, each pixel in the first map has a value related to the strength of sharpening to be performed on the corresponding pixel in the first image. Specifically, the horizontal and vertical MTFs shown in FIGS. 12A and 12B are stored for each image plane coordinate, and an evaluation frequency f eval In the high pixel imaging mode, the value of each pixel in the first map is determined by the evaluation frequency f eval is 0.8 times the Nyquist frequency, and in high sensitivity imaging mode it is 0.4 times the Nyquist frequency.
[0099] Alternatively, only the MTF at 0.8 times the Nyquist frequency of the image sensor 322 may be retained, and the MTF (which is 1 when the frequency is 0) may be linearly approximated to calculate the values of the first map when the resampling rate is not 1. If the MTF becomes negative due to the effect of linear interpolation when the resampling rate is greater than 1, the values of the first map are clipped at 0.
[0100] As in the first embodiment, the number of pixels in the first map is based on the number of pixels in the first image. In addition, the first map has spatially non-uniform values because the resolution performance of the optical system 321 varies depending on the image plane coordinate. Furthermore, the values in the first map are determined by the resolution performance of the optical system 321, and therefore do not depend on the structure of the object present in the first image.
[0101] In this embodiment, the fourth information on the state of the optical system 321 used in capturing the first image is not used to generate the first map. This is because the optical system 321 is a single-focus lens and has a fixed aperture value, so that the variation in blur due to the focus distance is small. If the focal length, aperture value, and focus distance change to multiple values and the blur changes due to these values, it is desirable to obtain the fourth information and generate the first map by determining the resolution performance of the optical system 321 based on the fourth information.
[0102] If the imaging device 302 is an interchangeable lens imaging device as in the first embodiment, the first image may be acquired through a plurality of types of imaging elements. In this case, since the pixel pitch varies depending on the type of imaging element, the first map is generated based on second information related to the sampling of the imaging element as well. In addition, if the optical low pass filter differs depending on the imaging element, fifth information related to the optical low pass filter is acquired, and the first map is generated based on the fifth information.
[0103] Furthermore, under conditions where the resolution is very low, subject information is significantly attenuated, so even if the blur is sharpened, problems such as undershooting are likely to occur. For this reason, under conditions where the resolution is lower than a default value, the value of the first map may be set to be greater than the value determined by the resolution, and the strength of the sharpening may be reduced. By reducing the strength of the sharpening, the above problems can be made less likely to occur.
[0104] In step S304, the image capturing device 302 (the calculation unit 326) generates a second map based on the number of pixels of the first image, the first information, and the third information. In the high sensitivity imaging mode, the resampling rate is 0.5, so the standard deviation of noise is 0.5 times that in the high pixel imaging mode.
[0105] In step S305 (image generation step), the imaging device 302 (the calculation unit 326) inputs the input data to the CNN to generate a second image. At this time, similar to the training time shown in FIG. 13, the first image, the first map, and the second map are linked in the channel direction and input to the CNN.
[0106] According to the above configuration, blur can be sharpened with high accuracy even in the first image obtained by performing cropping and resampling.
[0107] The above embodiments include the following methods.
[0108] (Method 1) Obtaining a first image generated by performing resampling on an image obtained by imaging using an optical system and an image sensor; a map generating step of generating a first map based on a pixel count of the first image and the first information regarding the resampling; and an image generating step of generating a second image in which the blur caused by the optical system included in the first image is sharpened by inputting input data based on the first image and the first map into a machine learning model, 11. A method of image processing, comprising the steps of: a) selecting a first image pixel from a second image pixel; b) selecting a second image pixel from the first image pixel; c) selecting a second image pixel from the first image pixel; (Method 2) The image processing method described in configuration 1, characterized in that the sharpening value for each pixel of the first map is a value related to the position on the imaging element of the corresponding one or more pixels of the first image. (Method 3) 2. The image processing method of claim 1, wherein the value related to the sharpening at each pixel of the first map is a value related to the strength of the sharpening performed on one or more corresponding pixels of the first image. (Method 4) The image processing method described in configuration 3, characterized in that the value relating to the strength of the sharpening at each pixel of the first map is a value based on the resolution performance of the optical system at the position of the corresponding one or more pixels of the first image. (Method 5) The sharpening is performed for a plurality of directions of the first image; 4. The image processing method according to claim 3, wherein the sharpening value for each pixel of the first map is a value related to the strength of the sharpening in each of the plurality of directions. (Method 6) 6. The image processing method according to any one of configurations 1 to 5, wherein the first information includes a resampling rate for the resampling. (Method 7) The image processing method according to any one of configurations 1 to 6, characterized in that the sharpening value for each pixel of the first map is not a uniform value in the first map and is a value independent of the structure of the subject contained in the first image. (Method 8) The image processing method according to any one of configurations 1 to 7, characterized in that, in the map generation step, the first map is generated based on the number of pixels of the first image, the first information, and second information regarding sampling of the imaging element. (Method 9) In the map generating step, a second map regarding noise intensity of the first image is generated based on a number of pixels of the first image, the first information, and third information regarding noise of the imaging element; The image processing method according to any one of configurations 1 to 7, characterized in that in the image generation step, the second image is generated by inputting the input data including the first image, the first map, and the second map into the machine learning model. (Method 10) In the map generating step, a third map is generated regarding a size of a pixel of the first image in a real space based on a number of pixels of the first image and the first information; The image processing method according to any one of configurations 1 to 7, characterized in that in the image generation step, the second image is generated by inputting the input data including the first image, the first map, and the third map into the machine learning model. (Method 11) In the map generating step, a fourth map regarding a state of the optical system is generated based on a number of pixels of the first image and fourth information regarding a state of the optical system during the imaging; The image processing method according to any one of configurations 1 to 7, characterized in that in the image generation step, the second image is generated by inputting the input data including the first image, the first map, and the fourth map into the machine learning model. (Method 12) generating a fifth map relating to an operation of the optical low pass filter based on the number of pixels of the first image and fifth information relating to an optical low pass filter of the image sensor, in the map generating step; The image processing method according to any one of configurations 1 to 7, characterized in that in the image generation step, the second image is generated by inputting the input data including the first image, the first map, and the fifth map into the machine learning model. (Method 13) The image processing method according to any one of configurations 1 to 7, characterized in that in the map generation step, the first map is generated based on the number of pixels of the first image, the first information, and fourth information relating to the state of the optical system during the imaging. (Method 14) 8. The image processing method according to claim 1, wherein in the map generating step, the first map is generated based on the number of pixels of the first image, the first information, and fifth information related to an optical low pass filter of the image sensor.
[0109] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.
[0110] The embodiments described above are merely representative examples, and various modifications and alterations are possible for each embodiment when implementing the present invention. [Explanation of symbols]
[0111] 102 Sharpening device (image processing device) 123,325 Acquisition Department 124,326 calculation section 131,322 Image sensor 141,321 Optical system
Claims
1. acquiring a first image generated by performing resampling on an image obtained by imaging using an optical system and an image sensor; a map generating step of generating a first map based on a pixel count of the first image and the first information regarding the resampling; and an image generating step of generating a second image in which blur caused by the optical system included in the first image is sharpened by inputting input data based on the first image and the first map into a machine learning model, 11. A method of image processing, comprising: a step of: extracting a first image from a first map; extracting a second image from a first map; and extracting a third image from the first map.
2. 2. The image processing method according to claim 1, wherein the sharpening value for each pixel of the first map is a value related to a position on the imaging element of one or more corresponding pixels of the first image.
3. 2. The method of claim 1, wherein the value related to the sharpening at each pixel of the first map is a value related to the strength of the sharpening performed on one or more corresponding pixels of the first image.
4. 4. The image processing method according to claim 3, wherein the value relating to the strength of the sharpening at each pixel of the first map is a value based on the resolution performance of the optical system at the position of the corresponding one or more pixels of the first image.
5. The sharpening is performed for a plurality of directions of the first image; 4. The image processing method according to claim 3, wherein the sharpening value for each pixel of the first map is a value related to the strength of the sharpening in each of the plurality of directions.
6. 2. The image processing method according to claim 1, wherein the first information includes a resampling rate in the resampling.
7. The image processing method according to claim 1, characterized in that the sharpening value for each pixel of the first map is not a uniform value in the first map and is a value independent of the structure of the subject contained in the first image.
8. 8. The image processing method according to claim 1, wherein in the map generation step, the first map is generated based on the number of pixels of the first image, the first information, and second information regarding sampling of the imaging element.
9. In the map generating step, a second map regarding noise intensity of the first image is generated based on a number of pixels of the first image, the first information, and third information regarding noise of the imaging element; The image processing method according to any one of claims 1 to 7, characterized in that, in the image generation step, the second image is generated by inputting the input data including the first image, the first map, and the second map into the machine learning model.
10. In the map generating step, a third map is generated regarding a size of a pixel of the first image in a real space based on a number of pixels of the first image and the first information; The image processing method according to any one of claims 1 to 7, characterized in that, in the image generation step, the second image is generated by inputting the input data including the first image, the first map, and the third map into the machine learning model.
11. In the map generating step, a fourth map regarding a state of the optical system is generated based on a number of pixels of the first image and fourth information regarding a state of the optical system during the imaging; The image processing method according to any one of claims 1 to 7, characterized in that in the image generation step, the second image is generated by inputting the input data including the first image, the first map, and the fourth map into the machine learning model.
12. generating a fifth map relating to an operation of the optical low pass filter based on the number of pixels of the first image and fifth information relating to an optical low pass filter of the image sensor, in the map generating step; The image processing method according to any one of claims 1 to 7, characterized in that in the image generation step, the second image is generated by inputting the input data including the first image, the first map, and the fifth map into the machine learning model.
13. 8. The image processing method according to claim 1, wherein in the map generation step, the first map is generated based on the number of pixels of the first image, the first information, and fourth information relating to the state of the optical system during the imaging.
14. 8. The image processing method according to claim 1, wherein in the map generating step, the first map is generated based on the number of pixels of the first image, the first information, and fifth information related to an optical low pass filter of the image sensor.
15. A program for causing a computer to execute a process according to the image processing method according to any one of claims 1 to 7.
16. an acquisition means for acquiring a first image generated by performing resampling on an image acquired by imaging using an optical system and an image sensor; a map generating means for generating a first map based on the number of pixels of the first image and the first information regarding the resampling; and an image generating means for generating a second image in which blur caused by the optical system included in the first image is sharpened by inputting input data based on the first image and the first map into a machine learning model, 13. An image processing apparatus comprising: an image processor configured to process an image of a first image; a first map having a first pixel value; a first image having a second pixel value; a second image having a first pixel value;
17. The optical system; The imaging element; An imaging device comprising: the image processing device according to claim 16.
18. 17. An image processing system comprising the image processing device according to claim 16 and a control device capable of communicating with the image processing device, the control device has a transmission means for transmitting a request for causing the image processing device to execute processing on the first image; The image processing system according to claim 1, wherein the image processing device has a processing means for executing processing on the first image in response to the request.
19. obtaining a ground truth image and a training image generated by performing resampling on the image; generating a first map based on a pixel count of the training images and the first information regarding resampling; inputting input data based on the training image and the first map into a machine learning model to generate an estimated image in which blur contained in the training image is sharpened; updating weights of a machine learning model based on the ground truth image and the estimated image; A method for producing a machine learning model, characterized in that the value at each pixel of the first map is a value related to the sharpening performed on the corresponding pixel of the first image.
20. A means for obtaining a ground truth image and a training image generated by performing resampling on the image; means for generating a first map based on the number of pixels in the training images and the first information regarding the resampling; a means for generating an estimated image in which blur contained in the training image is sharpened by inputting input data based on the training image and the first map into a machine learning model; means for updating weights of a machine learning model based on the correct image and the estimated image; 11. A training device according to claim 10, wherein the value at each pixel of said first map is a value for said sharpening performed on a corresponding pixel of said first image.
Citation Information
Cited By
Dark light image enhancement method
CN119850450A
