Information processing device, image processing device, program, image processing system, information processing method, and image processing method

Abstract

An information processing device (110) includes an optical computation unit (112) that computes a PSF of an optical element in each of a plurality of regions obtained by dividing an image-capturing surface of an image-capturing element that performs image capturing using the optical element, and generates an aberration image that has influence of aberration by the optical element on an ideal image by applying a corresponding PSF to each of a plurality of region images obtained by dividing the ideal image in accordance with the plurality of regions, a noise image generation unit (113) that generates a noise image representing noise at the time of image capturing, an image computation unit (114) that generates a learning image by superimposing the noise image on the aberration image, and a labeled training data generation unit (115) that generates labeled training data with the learning image being used as training data and the ideal image being used as correct answer data.

Description

Information processing device, image processing device, program, image processing system, information processing method, and image processing method

[0001] The present disclosure relates to an information processing device, an image processing device, a program, an image processing system, an information processing method, and an image processing method.

[0002] In recent years, image correction techniques using machine learning have made it possible to correct aberrations and noise components contained in images with high accuracy. In order to build a machine learning model for such image correction, it is important to prepare a large amount of training data, each consisting of a set of ideal images with little aberration and noise and training images that contain these images.

[0003] Patent Literature 1 describes an image processing device that uses a multi-layer neural network to learn how to correct blur components and noise components derived from the optical characteristics of an imaging system. Specifically, the image processing device acquires a single point spread function (PSF) based on an optical simulation and generates a blurred image by multiplying it by an ideal image. Training data including these is then generated.

[0004] The image processing device also adds noise to the ideal image and the blurred image, respectively, to generate a noise-loaded ideal image and a noise-loaded blurred image. Then, it generates training data including these. The image processing device corrects the aberration component and the noise component separately using a machine learning model trained using the training data generated in this manner.

[0005] Japanese Patent Application Laid-Open No. 2022-161503

[0006] In conventional techniques, a blurred image is generated from an ideal image based on a single PSF, and therefore, it is not possible to generate an appropriate blurred image when the influence of aberrations that depend on the field of view, such as coma aberration, is significant.

[0007] Therefore, one or more aspects of the present disclosure aim to make it possible to appropriately generate an aberration image even when the influence of aberrations that depend on the field of view, such as coma aberration, is large.

[0008] An information processing device according to one aspect of the present disclosure includes an optical calculation unit that calculates a point spread function (PSF) of an optical element for each of a plurality of regions obtained by dividing an imaging surface of an imaging element that captures an image using the optical element, and applies the corresponding PSF to each of a plurality of region images obtained by dividing an ideal image desired as a corrected image corresponding to the plurality of regions, thereby generating an aberration image in which the ideal image is affected by the aberration caused by the optical element; a noise image generation unit that generates a noise image representing noise at the time of imaging; an image calculation unit that generates a training image by superimposing the noise image on the aberration image; and a teacher data generation unit that generates teacher data in which the training image is used as training data and the ideal image is used as corrected data.

[0009] An image processing device according to one aspect of the present disclosure includes: a target image acquisition unit that acquires a target image that is an image to be corrected; and an inference unit that calculates a point spread function (PSF) of an optical element for each of a plurality of regions obtained by dividing an imaging surface of an imaging element that captures images using the optical element, and applies the corresponding PSF to each of a plurality of region images obtained by dividing an ideal image desired as a corrected image corresponding to the plurality of regions, thereby superimposing a noise image representing noise at the time of imaging on an aberration image in which the ideal image is affected by aberration caused by the optical element, as training data, and inputs the target image into a learning model generated using teacher data in which the ideal image is used as corrected data, thereby inferring a corrected image obtained by correcting the target image.

[0010] A program according to one aspect of the present disclosure causes a computer to function as an optical calculation unit that calculates a point spread function (PSF) of an optical element for each of a plurality of regions obtained by dividing an imaging surface of an imaging element that captures an image using the optical element, and applies the corresponding PSF to each of a plurality of region images obtained by dividing an ideal image desired as a corrected image corresponding to the plurality of regions, thereby generating an aberration image in which the ideal image is affected by the aberration caused by the optical element; a noise image generation unit that generates a noise image representing noise at the time of imaging; an image calculation unit that generates a training image by superimposing the noise image on the aberration image; and a teacher data generation unit that generates teacher data in which the training image is used as training data and the ideal image is used as corrected data.

[0011] A program according to one aspect of the present disclosure causes a computer to function as a target image acquisition unit that acquires a target image that is an image to be corrected, and an inference unit that calculates a point spread function (PSF) of an optical element for each of a plurality of regions obtained by dividing an imaging surface of an imaging element that captures an image using the optical element, and applies the corresponding PSF to each of a plurality of region images obtained by dividing an ideal image that is desired as a corrected image corresponding to the plurality of regions, thereby superimposing a noise image that represents noise at the time of imaging on an aberration image that is affected by aberration caused by the optical element on the ideal image, and inputs the target image into a learning model generated using teacher data in which the ideal image is used as corrected data, thereby inferring a corrected image obtained by correcting the target image.

[0012] an image processing system according to one aspect of the present disclosure, the image processing system comprising: an optical calculation unit that calculates a point spread function (PSF) of an optical element for each of a plurality of regions obtained by dividing an imaging surface of an image sensor that captures an image using the optical element, and applies the corresponding PSF to each of a plurality of region images obtained by dividing an ideal image desired as a corrected image corresponding to the plurality of regions, thereby generating an aberration image in which the ideal image is affected by the aberration caused by the optical element; a noise image generation unit that generates a noise image representing noise at the time of imaging; an image calculation unit that generates a learning image by superimposing the noise image on the aberration image; a teacher data generation unit that generates teacher data using the learning image as training data and the ideal image as ground truth data; a learning unit that generates learning models corresponding to a plurality of types and a plurality of temperatures by learning using the teacher data; a target image acquisition unit that acquires a target image that is an image to be corrected; and an inference unit that infers a corrected image obtained by correcting the target image by inputting the target image to the learning model.

[0013] An information processing method according to one aspect of the present disclosure includes: calculating a point spread function (PSF) of an optical element for each of a plurality of regions obtained by dividing an imaging surface of an imaging element that captures an image using the optical element; applying the corresponding PSF to each of a plurality of region images obtained by dividing an ideal image desired as a corrected image corresponding to the plurality of regions; generating an aberration image in which the ideal image is affected by aberration caused by the optical element; generating a noise image representing noise at the time of imaging; generating a learning image by superimposing the noise image on the aberration image; and generating teacher data in which the learning image is used as training data and the ideal image is used as corrected data.

[0014] An image processing method according to one aspect of the present disclosure includes acquiring a target image that is an image to be corrected, calculating a point spread function (PSF) of an optical element for each of a plurality of regions obtained by dividing an imaging surface of an imaging element that captures images using the optical element, and applying the corresponding PSF to each of a plurality of region images obtained by dividing an ideal image desired as a corrected image corresponding to the plurality of regions. A learning image is generated by superimposing a noise image representing noise at the time of imaging on an aberration image in which the ideal image is affected by aberration due to the optical element, and the target image is input into a learning model generated using teacher data in which the ideal image is used as corrected data, thereby inferring a corrected image obtained by correcting the target image.

[0015] According to one or more aspects of the present disclosure, even when the influence of aberrations that depend on the field of view, such as coma aberration, is large, it is possible to generate an aberration image appropriately.

[0016] FIG. 1 is a block diagram schematically showing the configuration of an image processing system according to Embodiments 1 and 2. FIG. 2 is a block diagram schematically showing the configuration of an information processing device according to Embodiments 1 and 2. FIG. 3 is a schematic diagram for explaining the process of calculating a PSF in each of a plurality of divided regions. FIG. 4 is a block diagram schematically showing the configuration of a PC. FIG. 5 is a block diagram schematically showing the configuration of an image processing device according to Embodiments 1 and 2. FIG. 6 is a flowchart showing the operation of an information processing device. FIG. 7 is a flowchart showing the operation of an image processing device. FIG. 8 is a schematic diagram for explaining a noise image generated in Embodiment 2.

[0017] 1 is a block diagram showing a schematic configuration of an image processing system 100 according to embodiment 1. The image processing system 100 includes an information processing device 110 and an image processing device 130. The information processing device 110 and the image processing device 130 can communicate with each other via a network 101.

[0018] The information processing device 110 functions as a learning device that learns a learning model for correcting image blur and noise, and the image processing device 130 functions as an inference device that uses the learning model to infer a corrected image in which the image blur and noise have been corrected.

[0019] 2 is a block diagram schematically illustrating the configuration of information processing device 110 according to embodiment 1. Information processing device 110 includes ideal image acquisition unit 111, optical calculation unit 112, noise image generation unit 113, image calculation unit 114, teacher data generation unit 115, learning unit 116, learning model storage unit 117, and communication unit 118.

[0020] The ideal image acquisition unit 111 acquires an ideal image, which is an image to be treated as correct answer data of the training data. The ideal image may be an image captured using a high-quality lens and sensor with little blur and noise, or an image generated by CG (Computer Graphics). In other words, the ideal image may be any image as long as it is an image that is preferable as a corrected image. The acquired ideal image is provided to the optical calculation unit 112.

[0021] In addition, the ideal image acquisition unit 111 may acquire the ideal image from another device, for example, via the communication unit 118, or may acquire the ideal image from another device or recording medium, for example, via a connector (not shown) compatible with USB (Universal Serial Bus).

[0022] The optical calculation unit 112 calculates a PSF (Point Spread Function) of an optical element for each of a plurality of regions obtained by dividing the imaging surface of the imaging element that captures an image using the optical element. The optical calculation unit 112 then applies a corresponding PSF to each of a plurality of region images obtained by dividing an ideal image desired as a corrected image into the plurality of regions, thereby generating an aberration image in which the ideal image is affected by the aberration caused by the optical element. Here, the optical element is, for example, a condenser lens, a reflecting mirror, or the like. The sensor that is the imaging element is, for example, a CCD (Charge-Coupled Device) sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, a thermal diode infrared sensor, or the like.

[0023] Specifically, the optical calculation unit 112 uses optical simulation to calculate a PSF that indicates blur, which is the effect of aberration caused by an optical system that simulates the optical elements used to capture the image to be corrected (hereinafter also referred to as the target image). As shown in FIG. 3 , the optical calculation unit 112 divides the sensor surface SF, which is an imaging element of an imaging device that uses optical elements, into multiple regions and calculates a PSF for each of the multiple divided regions. Here, as shown in FIG. 3 , the sensor surface SF is divided into nine regions using a grid. However, the first embodiment is not limited to this example. It is sufficient that the sensor surface SF be divided into multiple regions so that the characteristics of the blur of the optical element can be easily identified. For example, one or more of the regions included in the multiple regions may have different sizes. The number of divided regions may also be selected appropriately, taking into account factors such as the computational load.

[0024] Furthermore, the optical calculation unit 112 receives settings of parameters indicating alignment errors of the optical elements, and calculates the PSF so that the influence of the alignment errors is taken into account. For example, if at least one of the tilt, decenter, and thickness of the optical elements is input as a tolerance in the optical simulation, the optical calculation unit 112 can calculate the deterioration of the PSF due to alignment errors that occur during the manufacture of the optical elements and the imaging device that uses the optical elements.

[0025] Furthermore, the optical calculation unit 112 receives settings of parameters indicating the temperatures of the optical elements, and calculates the PSF so that the influence of temperature on the optical elements is taken into account. For example, by inputting temperature information in the optical simulation, the optical calculation unit 112 can calculate deterioration of the PSF due to changes in the shapes of optical elements such as reflecting mirrors and lenses, or changes in the refractive index of lenses, caused by temperature changes, or due to thermal expansion of the housing that houses the optical elements.

[0026] The optical calculation unit 112 then divides the ideal image into the same regions as those for which the PSFs have been calculated, and generates an aberration image by superimposing the corresponding PSFs on the divided region images. Specifically, the optical calculation unit 112 generates an aberration image by multiplying the pixel values ​​of each of the region images divided from the ideal image by the PSF calculated from the corresponding region. The ideal image and the generated aberration image are provided to the image calculation unit 114.

[0027] The noise image generating unit 113 generates a noise image that represents noise that occurs during imaging. Specifically, the noise image generating unit 113 generates a noise image that represents noise that is predicted to occur in the imaging element.

[0028] The noise image generating unit 113 can generate an image representing random noise as a noise image. For example, the noise image generating unit 113 may generate a noise image according to a Gaussian distribution so as to reproduce random noise that occurs randomly in a captured image due to dark current noise, readout noise, or the like.

[0029] Furthermore, when specific noise other than random noise occurs in the captured image due to the characteristics of the imaging device, the noise image generation unit 113 can use an image captured by the imaging device with the shutter closed as the noise image. For example, the noise image generation unit 113 can also generate, as the noise image, an image captured by the sensor of the imaging device with the shutter closed.

[0030] Since dark current is temperature dependent, if noise fluctuations due to temperature changes occur in the imaging device, the noise image generation unit 113 can use images taken with the shutter closed at different temperatures as noise images.

[0031] As described above, since noise is generated depending on the environment surrounding the imaging device, it is desirable for the noise image generation unit 113 to generate a plurality of noise images. The noise images generated in this manner are provided to the image calculation unit 114.

[0032] The image calculation unit 114 generates a training image by superimposing a noise image on the aberration image. Specifically, the image calculation unit 114 can generate the training image by multiplying the pixel value of each pixel in the aberration image by the pixel value of the corresponding pixel in the normalized noise image. The ideal image and the generated training image are provided to the teacher data generation unit 115.

[0033] The teacher data generating unit 115 generates teacher data from the learning images as training data and the ideal images as correct answer data. The generated teacher data is provided to the learning unit 116.

[0034] The learning unit 116 generates a learning model by learning using the training data. For example, the learning unit 116 inputs the training data into a neural network and performs learning to generate the learning model. The learning model storage unit 117 stores the learning model learned by the learning unit 116.

[0035] The communication unit 118 communicates with the image processing device 130 via the network 101. For example, the communication unit 118 transmits the learning model stored in the learning model storage unit 117 to the image processing device 130.

[0036] The information processing device 110 described above can be realized by, for example, a computer such as the PC 150 shown in Fig. 4. The PC 150 includes a storage 151 such as a hard disk drive (HDD) and a solid state drive (SSD), a memory 152, a processor 153 such as a central processing unit (CPU), a communication interface (I / F) 154 such as a network interface card (NIC), an input interface 155 such as a keyboard and a mouse, and a display 156.

[0037] For example, the learning model storage unit 117 can be realized by the storage 151 or the memory 152. The ideal image acquisition unit 111, the optical calculation unit 112, the noise image generation unit 113, the image calculation unit 114, the teacher data generation unit 115, and the learning unit 116 can be realized by loading a program stored in the storage 151 into the memory 152 and having the processor 153 execute the program. The communication unit 118 can be realized by the communication I / F 154.

[0038] The above program may be downloaded to storage 151 from a recording medium (not shown) via a reader / writer (not shown) or from a network via communication I / F 154, and then loaded onto memory 152 and executed by processor 153. Alternatively, the program may be directly loaded onto memory 152 from a recording medium (not shown) via a reader / writer or from a network via communication I / F 154, and then executed by processor 153. In other words, the program may be provided by a program product such as a recording medium.

[0039] 5 is a block diagram schematically illustrating the configuration of the image processing device 130 according to Embodiment 1. The image processing device 130 includes a communication unit 131, a learning model storage unit 132, a target image acquisition unit 133, and an inference unit 134.

[0040] The communication unit 131 performs communication via the network 101. For example, the communication unit 131 receives a learning model from the information processing device 110. The learning model storage unit 132 stores the learning model received by the communication unit 131.

[0041] The target image acquisition unit 133 acquires a target image that is an image to be corrected. For example, the target image acquisition unit 133 may acquire the target image from another device via the communication unit 131, or may acquire the target image from another device or recording medium via a USB-compatible connector (not shown) or the like. The acquired target image is provided to the inference unit 134.

[0042] The inference unit 134 retrieves the learning model from the learning model storage unit 132 and inputs the target image into the learning model, thereby inferring a corrected image that is an image obtained by correcting the target image. Here, blur and noise in the target image are corrected.

[0043] The image processing device 130 described above can also be realized by a computer such as the PC 150 shown in Fig. 4. For example, the learning model storage unit 132 can be realized by the storage 151 or the memory 152. The target image acquisition unit 133 and the inference unit 134 can be realized by loading a program stored in the storage 151 into the memory 152 and having the processor 153 execute the program. The communication unit 131 can be realized by the communication I / F 154.

[0044] 6 is a flowchart showing the operation of the information processing device 110. First, the ideal image acquisition unit 111 acquires an ideal image, which is an image to be treated as correct answer data of the training data (S10).

[0045] Next, the optical calculation unit 112 divides the sensor surface of the imaging device into multiple regions, calculates the PSF of the optical element in each of the multiple divided regions using optical simulation, and generates an aberration image by multiplying the PSF by the ideal image (S11).

[0046] Next, the noise image generating unit 113 generates a noise image indicating noise that is predicted to occur in the imaging device (S12).

[0047] Next, the image calculation unit 114 generates a learning image by superimposing the aberration image generated by the optical calculation unit 112 and the noise image generated by the noise image generation unit 113 (S13).

[0048] Next, the teacher data generating unit 115 generates teacher data in which the learning image generated by the image calculation unit 114 is used as training data and the ideal image is used as correct answer data (S14).

[0049] Next, the learning unit 116 generates a learning model by learning a model using the training data generated by the training data generating unit 115 (S15).

[0050] 7 is a flowchart showing the operation of the image processing device 130. First, the target image acquisition unit 133 acquires a target image that is an image to be corrected (S20).

[0051] Next, the inference unit 134 infers a corrected image, which is an image in which the blur and noise of the target image have been corrected, by inputting the target image into the learning model stored in the learning model storage unit 132 (S21).

[0052] As described above, according to the first embodiment, the PSF is calculated by dividing the sensor surface into a grid (field of view), so that the influence of aberrations that depend on the field of view, such as coma aberration, can be accurately calculated. Also, blurring due to alignment errors of optical elements can be appropriately corrected. Furthermore, blurring due to changes in the shape of optical elements such as lenses or mirrors, the optical path length, and the refractive index of lenses that occur due to temperature changes in the optical system can be appropriately corrected. Furthermore, noise specific to sensors can be appropriately corrected. Furthermore, because training data is created based on optical simulations, a large amount of training data required for building a learning model can be easily prepared.

[0053] 1, an image processing system 200 according to the second embodiment includes an information processing device 210 and an image processing device 230. The information processing device 210 and the image processing device 230 can communicate with each other via a network 101.

[0054] 2 , the information processing device 210 according to the second embodiment includes an ideal image acquisition unit 111, an optical calculation unit 112, a noise image generation unit 213, an image calculation unit 214, a teacher data generation unit 215, a learning unit 216, a learning model storage unit 217, and a communication unit 118. The ideal image acquisition unit 111, the optical calculation unit 112, and the communication unit 118 of the information processing device 210 according to the second embodiment are similar to the ideal image acquisition unit 111, the optical calculation unit 112, and the communication unit 118 of the information processing device 110 according to the first embodiment.

[0055] The noise image generation unit 213 generates a noise image that indicates noise that is predicted to occur in the imaging device. In the second embodiment, the noise image generation unit 213 generates noise for each combination of the type of sensor used in the imaging device and the temperature of the sensor. In other words, the noise image generation unit 213 generates a noise image for each combination of one type of sensor selected from multiple types of image sensors and one temperature selected from multiple sensor temperatures. Note that the temperature of the sensor is preferably the temperature of the sensor itself, but if it is difficult to obtain the temperature of the sensor itself, the temperature around the imaging device when capturing an image (environmental temperature) may be used.

[0056] For example, if the sensor s has 1 to n types (n is an integer of 2 or greater) and the temperature t that generates noise is 1 to m types (m is an integer of 2 or greater), the noise image generation unit 213 generates noise images (s1, t1) to (sn, tm) for all combinations of these, as shown in Figure 8.

[0057] Here, the noise image generating unit 213 may generate, as a noise image, an image captured by one type of sensor in the combination with the shutter closed at one temperature in the combination.

[0058] The image calculation unit 214 generates training images (s1, t1) to (sn, tm) by superimposing the aberration image and each of the noise images (s1, t1) to (sn, tm). The ideal image and the generated training images (s1, t1) to (sn, tm) are provided to the teacher data generation unit 215.

[0059] The teacher data generation unit 215 generates teacher data from each of the learning images (s1, t1) to (sn, tm) as training data and the ideal image as ground truth data. The generated teacher data is provided to the learning unit 216.

[0060] The learning unit 216 generates a learning model by learning the model using teacher data. In the second embodiment, the learning unit 216 performs learning using one of the following three patterns depending on the learning images (s1, t1) to (sn, tm) based on the noise images (s1, t1) to (sn, tm).

[0061] In pattern 1, the learning unit 216 generates multiple learning models corresponding to each of multiple sensor types by learning using corresponding teacher data for each of the multiple sensor types. Specifically, as shown by symbols P11 to P1n in FIG. 8 , the learning unit 216 generates a learning model for each sensor type by learning using corresponding teacher data for each sensor type. Here, for example, in the case of sensor s1, the corresponding teacher data is teacher data including learning images (s1, t1) to (s1, tm) generated using noise images (s1, t1) to (s1, tm) of sensor s1. In the example shown in FIG. 8 , n learning models are generated. The n learning models thus learned are stored in the learning model storage unit 217.

[0062] In pattern 2, the learning unit 216 generates multiple learning models corresponding to each of the multiple temperatures by learning using corresponding teacher data for each of the multiple temperatures. Specifically, as shown by symbols P21 to P2m in FIG. 8 , the learning unit 216 generates a learning model for each temperature by learning using the corresponding teacher data for each temperature. Here, for example, for temperature t1, the corresponding teacher data is teacher data including learning images (s1, t1) to (sn, t1) generated using noise images (s1, t1) to (sn, t1) at temperature t1. In the example shown in FIG. 8 , m learning models are generated. The m learning models thus learned are stored in the learning model storage unit 217.

[0063] In pattern 3, the learning unit 216 generates learning models corresponding to multiple sensor types and multiple temperatures by learning using the teacher data generated as described above. Specifically, as shown by symbol P3 in FIG. 8 , the learning unit 216 generates one learning model by learning using teacher data including learning images (s1, t1) to (sn, tm) generated using all noise images (s1, t1) to (sn, tm). The one learning model learned in this way is stored in the learning model storage unit 217.

[0064] 5 , the image processing device 230 according to the second embodiment includes a communication unit 131, a learning model storage unit 232, a target image acquisition unit 133, and an inference unit 234. The communication unit 131 and the target image acquisition unit 133 of the image processing device 230 according to the second embodiment are similar to the communication unit 131 and the target image acquisition unit 133 of the image processing device 130 according to the first embodiment.

[0065] The learning model storage unit 232 stores the learning model received by the communication unit 131. As described above, in the second embodiment, a learning model is generated using one of patterns 1 to 3, and therefore the learning model storage unit 232 stores the learning model corresponding to the pattern.

[0066] The inference unit 234 inputs the target image into the learning model stored in the learning model storage unit 232, and thereby infers a corrected image, which is an image obtained by correcting blur and noise in the target image.

[0067] For example, when a learning model is generated using Pattern 1, the inference unit 234 infers multiple corrected images by inputting the target image into each of all learning models generated corresponding to all sensors. The inference unit 234 then selects an optimal corrected image from the multiple corrected images as an inference result based on predetermined criteria corresponding to the intended use of the corrected images. For example, by selecting a corrected image with the smallest standard deviation of brightness values, an image with minimal brightness variation can be obtained. The criteria for selecting a single corrected image can be set appropriately depending on the intended use of the corrected image. Note that, if the type of sensor can be identified from the metadata of the target image, the inference unit 234 may infer the corrected image by inputting the target image into a learning model corresponding to the identified sensor. The inference unit 234 may also use multiple corrected images output from multiple learning models as inference results. In this case, the user of the image processing device 230 simply selects the optimal corrected image from the multiple corrected images.

[0068] Furthermore, if a learning model is generated using pattern 2, the inference unit 234 may select one learning model generated at a temperature corresponding to the temperature of the sensor of the imaging device when the target image was captured and input the target image into that one learning model to infer a corrected image. In this case, if the metadata of the target image includes a temperature, that temperature can be used as the temperature of the sensor of the imaging device when the target image was captured. The inference unit 234 may also accept input of the temperature of the sensor of the imaging device when the target image was captured via an input unit (not shown). Note that the temperature of the sensor of the imaging device when the target image was captured is preferably the temperature of the sensor itself, but may also be the temperature around the imaging device (environmental temperature). Furthermore, if a learning model with a temperature matching the temperature of the sensor of the imaging device when the target image was captured has not been generated, a learning model with a temperature closest to the temperature of the sensor of the imaging device when the target image was captured may be used.

[0069] Furthermore, if a learning model is generated using pattern 3, the inference unit 234 may infer a corrected image by inputting the target image into the learning model.

[0070] In the first and second embodiments described above, the image processing systems 100 and 200 are configured by the information processing devices 110 and 210 and the image processing devices 130 and 230, but the first and second embodiments are not limited to such examples. For example, the information processing devices 110 and 210 may be directly connected to the image processing devices 130 and 230 without using the network 101. Furthermore, the image processing systems 100 and 200 may be configured by one device or three or more devices. In other words, the functional units of the information processing devices 110 and 210 and the image processing devices 130 and 230 may be appropriately arranged in one or more devices such as computers.

[0071] Specifically, the information processing devices 110 and 210 may perform processing up to the generation of training data, and transmit the training data to another device, thereby generating a learning model in the other device.

[0072] Alternatively, the learning model generated by the information processing device 110, 210 may be stored in another server such as a cloud server, and the image processing device 130, 230 may transmit the target image and necessary information acquired therefrom to the server, thereby allowing the server to infer the corrected image. In this case, the image processing device 130, 230 may obtain the inference result from the server.

[0073] Note that learning in the information processing devices 110 and 210 may be continued even after processing in the image processing devices 130 and 230 has started. For example, if a sensor that did not yet exist when the learning model was learned appears, learning may be performed later using a noise image of that sensor. Furthermore, if an optical element that did not yet exist when the learning model was learned appears, learning may be performed later by calculating a PSF for that optical element.

[0074] In addition, in the first and second embodiments, the information processing devices 110 and 210 are described as devices that generate learning models, but they may also be configured as devices that generate training data. Even in such cases, it is possible to easily prepare a large amount of training data that reflects the influence of aberrations that depend on the field of view, such as coma aberration.

[0075] 100, 200 Image processing system, 110, 210 Information processing device, 111 Ideal image acquisition unit, 112 Optical calculation unit, 113, 213 Noise image generation unit, 114, 214 Image calculation unit, 115, 215 Teacher data generation unit, 116, 216 Learning unit, 117, 217 Learning model storage unit, 118 Communication unit, 130, 230 Image processing device, 131 Communication unit, 132, 232 Learning model storage unit, 133 Target image acquisition unit, 134, 234 Inference unit.

Claims

1. An information processing device comprising: an optical calculation unit that calculates a PSF (Point Spread Function) of an optical element for each of a plurality of regions obtained by dividing an imaging surface of an imaging element that captures an image using the optical element, and applies the corresponding PSF to each of a plurality of region images obtained by dividing an ideal image desired as an image after correction corresponding to the plurality of regions, thereby generating an aberration image in which the ideal image is affected by the aberration caused by the optical element; a noise image generation unit that generates a noise image representing noise at the time of imaging; an image calculation unit that generates a learning image by superimposing the noise image on the aberration image; and a teacher data generation unit that generates teacher data in which the learning image is used as training data and the ideal image is used as corrective data.

2. The information processing device according to claim 1, wherein the optical calculation unit receives a parameter indicating an alignment error of the optical element, and calculates the PSF so that the effect of the alignment error is included.

3. The information processing device according to claim 1 or 2, wherein the optical calculation unit receives a parameter indicating the temperature of the optical element, and calculates the PSF so as to include the effect of the temperature on the optical element.

4. An information processing device according to any one of claims 1 to 3, wherein the noise image generating unit generates an image representing random noise as the noise image.

5. An information processing device according to any one of claims 1 to 4, wherein the noise image generating section generates, as the noise image, an image captured by the imaging element with a shutter closed.

6. An information processing device according to any one of claims 1 to 5, further comprising a learning unit that generates a learning model by learning using the teacher data.

7. An information processing device according to any one of claims 1 to 5, characterized in that the noise image generating unit generates the noise image for each combination of one type selected from a plurality of types of the imaging element and one temperature selected from a plurality of temperatures of the imaging element.

8. The information processing device according to claim 7, wherein the noise image generating unit generates as the noise image an image captured by the imaging element of the one type in the combination with a shutter closed at the one temperature in the combination.

9. An information processing device as described in claim 7 or 8, further comprising a learning unit that generates a plurality of learning models corresponding to each of the plurality of types by learning using the corresponding teacher data for each of the plurality of types.

10. An information processing device as described in claim 7 or 8, further comprising a learning unit that generates a plurality of learning models corresponding to each of the plurality of temperatures by learning using the corresponding teacher data for each of the plurality of temperatures.

11. An information processing device as described in claim 7 or 8, further comprising a learning unit that generates a learning model corresponding to the multiple types and the multiple temperatures by learning using the teacher data.

12. An image processing device comprising: a target image acquisition unit that acquires a target image that is an image to be corrected; and an inference unit that calculates a PSF (Point Spread Function) of an optical element in each of a plurality of regions obtained by dividing an imaging surface of an imaging element that captures images using the optical element, and applies the corresponding PSF to each of a plurality of region images obtained by dividing an ideal image desired as an image after correction corresponding to the plurality of regions, thereby inferring a corrected image obtained by correcting the target image by inputting the target image into a learning model generated using teacher data in which the ideal image is used as corrective data and a learning image generated by superimposing a noise image representing noise at the time of imaging on an aberration image in which the ideal image is affected by the aberration caused by the optical element as training data.

13. The image processing device described in claim 12, characterized in that the noise image is generated for each combination of one type selected from the multiple types of the imaging element and one temperature selected from the multiple temperatures of the imaging element, the learning model is generated by learning for each of the multiple types using the corresponding teacher data, thereby generating multiple learning models corresponding to each of the multiple types, and the inference unit inputs the target image to each of the multiple learning models and infers the corrected image, thereby inferring multiple corrected images from each of the multiple learning models.

14. The image processing device according to claim 13, wherein the inference unit selects, from the plurality of corrected images, one corrected image that is optimal according to a predetermined criterion as an inference result.

15. The image processing device described in claim 12, characterized in that the noise image is generated for each combination of one type selected from the multiple types of the imaging element and one temperature selected from the multiple temperatures of the imaging element, and the learning model is generated by learning for each of the multiple temperatures using the corresponding teacher data, thereby generating multiple learning models corresponding to each of the multiple temperatures, and the inference unit selects one learning model from the multiple learning models that corresponds to the temperature of the imaging element when the corrected image was captured, and inputs the target image to the one learning model.

16. A program causing a computer to function as: an optical calculation unit that calculates a PSF (Point Spread Function) of an optical element for each of a plurality of regions obtained by dividing an imaging surface of an imaging element that captures an image using an optical element, and applies the corresponding PSF to each of a plurality of region images obtained by dividing an ideal image desired as an image after correction corresponding to the plurality of regions, thereby generating an aberration image in which the ideal image is affected by the aberration caused by the optical element; a noise image generation unit that generates a noise image representing noise at the time of imaging; an image calculation unit that generates a learning image by superimposing the noise image on the aberration image; and a teacher data generation unit that generates teacher data using the learning image as training data and the ideal image as corrective data.

17. A program that causes a computer to function as: a target image acquisition unit that acquires a target image that is an image to be corrected; and an inference unit that calculates a PSF (Point Spread Function) of an optical element for each of a plurality of regions obtained by dividing an imaging surface of an imaging element that captures images using an optical element, and applies the corresponding PSF to each of a plurality of region images obtained by dividing an ideal image desired as an image after correction corresponding to the plurality of regions, thereby superimposing a noise image representing noise at the time of imaging on an aberration image in which the ideal image is affected by the aberration caused by the optical element, and inputs the target image into a learning model generated using teacher data in which the ideal image is used as corrective data, thereby inferring a corrected image obtained by correcting the target image.

18. An image processing system comprising: an optical calculation unit that calculates a PSF (Point Spread Function) of an optical element for each of a plurality of regions obtained by dividing an imaging surface of an imaging element that captures an image using an optical element, and applies the corresponding PSF to each of a plurality of region images obtained by dividing an ideal image desired as an image after correction corresponding to the plurality of regions, thereby generating an aberration image in which the ideal image is affected by the aberration caused by the optical element; a noise image generation unit that generates a noise image representing noise at the time of imaging; an image calculation unit that generates a learning image by superimposing the noise image on the aberration image; a teacher data generation unit that generates teacher data using the learning image as training data and the ideal image as corrective data; a learning unit that generates learning models corresponding to a plurality of types and a plurality of temperatures by learning using the teacher data; a target image acquisition unit that acquires a target image that is an image to be corrected; and an inference unit that infers a corrected image obtained by correcting the target image by inputting the target image into the learning model.

19. An information processing method comprising: calculating a PSF (Point Spread Function) of an optical element for each of a plurality of regions obtained by dividing an imaging surface of an imaging element using the optical element; applying the corresponding PSF to each of a plurality of region images obtained by dividing an ideal image desired as a corrected image corresponding to the plurality of regions, thereby generating an aberration image in which the ideal image is affected by the aberration caused by the optical element; generating a noise image representing noise at the time of imaging; generating a learning image by superimposing the noise image on the aberration image; and generating teacher data in which the learning image is used as training data and the ideal image is used as corrective data.

20. An image processing method comprising the steps of: acquiring a target image to be corrected; calculating a PSF (Point Spread Function) of an optical element for each of a plurality of regions obtained by dividing an imaging surface of an imaging element that captures images using an optical element; and applying the corresponding PSF to each of a plurality of region images obtained by dividing an ideal image desired as a corrected image corresponding to the plurality of regions. A learning image is generated by superimposing a noise image representing noise at the time of imaging on an aberration image in which the ideal image is affected by the aberration caused by the optical element, and inputting the target image into a learning model generated using teacher data in which the ideal image is used as corrected data, thereby inferring a corrected image obtained by correcting the target image.

Images