Information processing device, image processing device, image processing system, information processing method, image processing method, information processing program, and image processing program

Abstract

An information processing device (110) comprises: an optical calculation unit (112) that calculates each of the optical path of light from an imaging field of view in an imaging device that performs imaging using an optical element and the optical path of light from outside the imaging field of view that is outside the imaging field of view, calculates stray light produced by the optical element and a point spread function (PSF) of the optical element, and generates a degraded image in which the stray light and the PSF are applied to an ideal image that is handled as correct answer data in teacher data; a noise image generation unit (113) that generates a noise image including noise predicted to occur in an imaging element of the imaging device during imaging; an image calculation unit (114) that superimposes the noise image on the degraded image to generate a learning image; and a teacher data generation unit (115) that generates teacher data in which the learning image is employed as training data and an image corresponding to the imaging field of view of the imaging device within the ideal image is employed as correct answer data.

Description

Information processing apparatus, image processing apparatus, image processing system, information processing method, image processing method, information processing program, and image processing program

[0004]

[0001] The present disclosure relates to an information processing apparatus, an image processing apparatus, an image processing system, an information processing method, an image processing method, an information processing program, and an image processing program.

[0002] In recent years, image correction technology using machine learning has made it possible to accurately correct stray light components such as aberration components, noise components, flare, or ghost included in an image. In order to construct such a machine learning model for image correction, it is necessary to prepare a large amount of teacher data in which an ideal image with less aberration, noise, and stray light and a learning image including aberration, noise, and stray light are paired.

[0003] Patent Document 1 describes an information processing apparatus that provides teacher data for machine learning according to a user's use case. Here, the use case is noise reduction, person detection, or the like. The information processing apparatus imparts deterioration due to aberration, transmittance, or stray light to an ideal image from information on a sensor input by the user. Further, the information processing apparatus imparts deterioration due to dark current noise or shot noise to an ideal image from information on an image sensor input by the user.

[0004] The information processing apparatus described in Patent Document 1 generates a learning image in which optical deterioration caused by a lens such as aberration or stray light and deterioration caused by noise caused by an image sensor are imparted to an ideal image according to a use case as described above. Then, teacher data composed of the ideal image and the learning image generated from the ideal image is generated.

[0005] International Publication No. 2024 / 029349

[0006] The technology described in Patent Document 1 uses lens information and image sensor information to add degradation caused by the lens and image sensor to an input ideal image the same size as the field of view of the imaging device, thereby generating a training image. However, the technology described in Patent Document 1 has the problem that light from outside the field of view of the imaging device can also reach the image sensor by following an unexpected optical path, and in such cases, it is not possible to generate a training image with appropriately added stray light.

[0007] The purpose of this disclosure is to provide an information processing device, an image processing device, an image processing system, an information processing method, an image processing method, an information processing program, and an image processing program that can generate training images with aberration components, noise components, and stray light components added by taking into account the effects of stray light from outside the field of view.

[0008] The information processing device of the present disclosure is characterized by comprising: an optical calculation unit that calculates the optical path of light from the imaging field of view and the optical path of light from outside the imaging field of view, which is outside the imaging field of view, in an imaging device that takes images using an optical element, and calculates stray light caused by the optical element; and a PSF calculation unit that calculates the PSF (Point Spread Function) of the optical element; an optical calculation unit that generates a degraded image by applying the stray light and the PSF to an ideal image which is treated as correct answer data for training data; a noise image generation unit that generates a noise image which includes noise that is predicted to occur in the image sensor of the imaging device during imaging; an image calculation unit that generates a training image by superimposing the noise image onto the degraded image; and a training data generation unit that uses the training image as training data and generates training data in which the image corresponding to the imaging field of view of the imaging device from the ideal image is used as correct answer data.

[0009] The image processing apparatus of the present disclosure comprises: a target image acquisition unit that acquires a target image which is an image to be corrected; and an inference unit that calculates the optical path of light from the imaging field of view of an imaging device that takes images using an optical element, and the optical path of light from outside the imaging field of view which is outside the imaging field of view, calculates stray light from the optical element and the PSF (Point Spread Function) of the optical element, and uses a training image as training data, which is a degraded image obtained by applying the stray light and the PSF to an ideal image which is treated as the correct answer data for the training data, and uses a training image which is generated by superimposing a noise image which includes noise that is predicted to be generated by the image sensor of the imaging device during imaging, and inputs the target image into a learning model for inferring a corrected image which is a corrected image which is a corrected image which is a corrected image which is an image to be corrected from the target image.

[0010] The image processing system disclosed herein includes a stray light calculation unit that calculates the optical path of light from the imaging field of view and the optical path of light from outside the imaging field of view in an imaging device that uses optical elements to take images, and calculates stray light caused by the optical elements, and a PSF (Point Spread) of the optical elements. The device is characterized by comprising: an optical calculation unit that calculates PSF (Function), an optical calculation unit that generates a degraded image by applying the stray light and the PSF to an ideal image treated as correct answer data for training data; a noise image generation unit that generates a noise image including noise that is predicted to occur in the image sensor of the imaging device during imaging; an image calculation unit that generates a training image by superimposing the noise image onto the degraded image; a training data generation unit that generates training data using the training image as training data and using an image from the ideal image corresponding to the imaging field of view of the imaging device as correct answer data; a learning unit that generates a learning model for inferring a corrected image using the training data; an image acquisition unit that acquires an image that is to be corrected; and an inference unit that infers a corrected image by inputting the image into the learning model.

[0011] The information processing method disclosed herein is an information processing method performed by a computer, comprising the steps of: calculating the optical path of light from the imaging field of view and the optical path of light from outside the imaging field of view in an imaging device that takes images using an optical element, and calculating stray light from the optical element; calculating the Point Spread Function (PSF) of the optical element; generating a degraded image by applying the stray light and the PSF to an ideal image which is treated as correct answer data for training data; generating a noise image which includes noise that is predicted to occur in the image sensor of the imaging device during imaging; generating a training image by superimposing the noise image onto the degraded image; and generating training data which uses the training image as training data and the image from the ideal image corresponding to the imaging field of view of the imaging device as correct answer data.

[0012] The image processing method of the present disclosure is an image processing method performed by a computer, comprising the steps of: acquiring a target image which is an image to be corrected; calculating the optical path of light from the imaging field of view of an imaging device that takes images using an optical element, and the optical path of light from outside the imaging field of view which is outside the imaging field of view; calculating stray light from the optical element and the PSF (Point Spread Function) of the optical element; superimposing a noise image containing noise predicted to occur at the imaging element of the imaging device during imaging onto a degraded image obtained by applying the stray light and the PSF to an ideal image which is treated as the correct answer data for the training data, and using the training image generated by superimposing a noise image containing noise predicted to occur at the imaging element of the imaging device during imaging as training data, and inputting the target image into a learning model for inferring a corrected image which is a corrected image obtained by correcting the target image.

[0013] The information processing program of this disclosure causes a computer to perform the following steps: calculate the optical path of light from the imaging field of view and the optical path of light from outside the imaging field of view in an imaging device that takes images using an optical element, and calculate stray light caused by the optical element; calculate the Point Spread Function (PSF) of the optical element; generate a degraded image by applying the stray light and the PSF to an ideal image which is treated as the correct answer data for training data; generate a noise image which includes noise that is predicted to occur in the image sensor of the imaging device during imaging; generate a training image by superimposing the noise image onto the degraded image; and generate training data which uses the training image as training data and the image from the ideal image corresponding to the imaging field of view of the imaging device as the correct answer data.

[0014] The image processing program of this disclosure causes a computer to perform the following steps: acquire a target image which is an image to be corrected; calculate the optical path of light from the imaging field of view of an imaging device that takes images using an optical element, and the optical path of light from outside the imaging field of view which is outside the imaging field of view; calculate the stray light from the optical element and the PSF (Point Spread Function) of the optical element; use the training image generated by superimposing a noise image containing noise expected to occur at the imaging element of the imaging device during imaging onto a degraded image obtained by applying the stray light and the PSF to an ideal image which is treated as the correct answer data for the training data; and input the target image into a learning model for inferring a corrected image which is generated using training data which is the image corresponding to the imaging field of view of the imaging device from the ideal image which is the correct answer data.

[0015] According to this disclosure, it is possible to provide an information processing device, an image processing device, an image processing system, an information processing method, an image processing method, an information processing program, and an image processing program that can generate training images with aberration components, noise components, and stray light components added by taking into account the effects of stray light from outside the field of view.

[0016] This is a block diagram schematically showing the configuration of the image processing system according to this embodiment. This is a block diagram schematically showing the configuration of the information processing device in this embodiment. This is a block diagram showing an example of the configuration of the optical calculation unit according to this embodiment. This is a schematic diagram showing the results of ray tracing for an imaging device created using optical simulation in stray light calculation unit according to this embodiment. This is an explanatory diagram showing an example of an ideal image input to the optical calculation unit according to this embodiment and a degraded image output. This is an explanatory diagram showing an example of training data generated in this embodiment. This is a block diagram showing an example of the information processing device hardware configuration of the information processing device according to this embodiment. This is a block diagram schematically showing the configuration of the image processing device in this embodiment. This is a flowchart showing an example of processing in the information processing device according to this embodiment. This is a flowchart showing an example of processing in the image processing device according to this embodiment.

[0017] The image processing system according to the embodiment will be described below with reference to the drawings. The following embodiment is merely an example, and it is possible to combine the embodiments as appropriate and modify each embodiment as appropriate.

[0018] Figure 1 is a schematic block diagram showing the configuration of the image processing system 100 according to this embodiment. The image processing system 100 comprises an information processing device 110 and an image processing device 130. The information processing device 110 and the image processing device 130 can communicate with each other via a network 101.

[0019] The information processing device 110 functions as a learning device that constructs a learning model by machine learning for inferring a corrected image that corrects for degradation due to image aberrations and at least one of stray light, as well as noise. The image processing device 130 functions as an inference device that uses the constructed learning model to infer a corrected image that corrects for degradation due to image aberrations and at least one of stray light, as well as noise.

[0020] Figure 2 is a schematic block diagram showing the configuration of the information processing device 110 in this embodiment. The information processing device 110 comprises an ideal image acquisition unit 111, an optical calculation unit 112, a noise image generation unit 113, an image calculation unit 114, a training data generation unit 115, a learning unit 116, a learning model storage unit 117, and a communication unit 118.

[0021] The ideal image acquisition unit 111 acquires an ideal image, which is an image to be treated as the correct answer data for the training data. The ideal image may be an image captured using a high-quality lens and sensor with little blur and noise, or it may be an image generated by computer graphics (CG). In other words, the ideal image can be any image that is preferable as a corrected image after correction inferred by the learning model. The acquired ideal image is supplied to the optical calculation unit 112.

[0022] The ideal image acquisition unit 111 may acquire an ideal image from another device via, for example, the communication unit 118, or it may acquire an ideal image from another device or recording medium via a USB (Universal Serial Bus) compatible connector (not shown).

[0023] Figure 3 is a block diagram showing an example of the configuration of the optical calculation unit 112. As shown in Figure 3, the optical calculation unit 112 consists of a stray light calculation unit 119 and a PSF (Point Spread Function) calculation unit 120. PSF is a function that represents the response of the optical system to a point light source.

[0024] Figure 4 is a schematic diagram showing the results of ray tracing for the imaging device 200, created using optical simulation in the optical calculation unit 112. Here, a Fresnel lens 203 is used as the optical element. A Fresnel lens is used in this embodiment of the present invention because its special shape makes it prone to generating stray light, but the optical element in this embodiment is not limited to a Fresnel lens. Examples of optical elements include spherical focusing lenses, reflecting mirrors, or diffracting lenses.

[0025] The image sensor 204 is a CCD (Charge-Coupled Device) sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, or a thermal diode infrared sensor, etc.

[0026] The stray light calculation unit 119 of the optical calculation unit 112 uses optical simulation to calculate the optical path of light rays passing through an optical system that simulates the optical elements used to capture the image to be corrected (hereinafter also referred to as the target image). At this time, as shown in Figure 4, the stray light calculation unit 119 calculates the stray light that unintentionally reaches the image sensor 204 due to reflection, scattering, diffraction, etc., from the light rays emitted from the field of view 201 of the imaging device 200, which is determined by the size of the image sensor 204 and the focal length of the Fresnel lens 203, and the field of view 202, which is outside the field of view 201 and larger than the field of view 201. Note that the field of view 201 is an example of the imaging field of view, and the field of view 202 is an example of an area outside the imaging field of view.

[0027] Furthermore, the stray light calculation unit 119 calculates stray light based on the settings of the mask and lens barrel information of the imaging device, so as to be affected by them. For example, in the optical simulation shown in Figure 5, if the position and size of the mask (for example, a focusing mask provided on the objective side of the Fresnel lens 203) and the lens barrel (for example, a housing in which the Fresnel lens 203 is placed inside the imaging device) 205 are input to the stray light calculation unit 119, it can calculate stray light from each of the fields of view 201 and 202 in accordance with the actual imaging device.

[0028] The PSF calculation unit 120 uses optical simulation to calculate a PSF that shows the effect of aberrations caused by an optical system that simulates the optical elements used to capture the target image. Furthermore, the PSF calculation unit 120 calculates the image degradation due to aberrations by superimposing the calculated PSF onto the ideal image.

[0029] In this embodiment, the PSF is not limited to a single PSF within the imaging plane of the image sensor. For example, since coma aberration differs depending on the observation area, its effect can be taken into consideration. Specifically, the PSF calculation unit 120 can calculate the PSF of the optical element in each of the multiple regions obtained by dividing the imaging plane of the image sensor. The PSF calculation unit 120 then applies the corresponding PSF to each of the multiple region images obtained by dividing the desired ideal image, which is the corrected image, according to the multiple regions, thereby calculating the image degradation due to aberrations according to the observation area in the ideal image.

[0030] The optical calculation unit 112 calculates image degradation by calculating stray light and PSF, respectively, based on the setting of parameters indicating the manufacturing tolerance and alignment error of the optical elements, so as to include the effects of the manufacturing tolerance and alignment error of the optical elements.

[0031] For example, in an optical simulation, the stray light calculation unit 119 of the optical calculation unit 112 can calculate stray light affected by the manufacturing tolerance of the optical element and the alignment error that occurs during the manufacturing of the imaging device 200 using the optical element, if at least one of the following is input as a tolerance: the shape of the optical element, the tilt indicating the inclination of the optical element, the decentering indicating the deviation from the design position in a plane perpendicular to the optical axis of the optical element, and the thickness indicating the deviation from the design position in the optical axis direction of the optical element.

[0032] Similarly, the PSF calculation unit 120 of the optical calculation unit 112 can calculate the deterioration of PSF due to the manufacturing tolerance of the optical element and the alignment error that occurs during the manufacturing of the imaging device 200 using the optical element, if at least one of the shape, tilt, decentering, and thickness of the optical element is input as a tolerance in the optical simulation.

[0033] Furthermore, the optical calculation unit 112 calculates image degradation by calculating stray light and PSF respectively, based on the setting of a parameter indicating the temperature of the optical element, so as to include the effect of temperature on the optical element.

[0034] For example, the stray light calculation unit 119 of the optical calculation unit 112 can calculate stray light affected by changes in the shape of optical elements, changes in the refractive index of lenses, or thermal expansion of the housing that houses the optical elements, by inputting temperature information during optical simulation.

[0035] Similarly, the PSF calculation unit 120 of the optical calculation unit 112 can calculate the deterioration of PSF due to changes in the shape of the optical elements, changes in the refractive index of the lenses, or thermal expansion of the housing that houses the optical elements, by inputting temperature information during optical simulation.

[0036] As described above, the optical calculation unit 112 generates a degraded image, which is a degraded image due to stray light and aberrations, for an ideal image larger than the field of view 201 of the imaging device 200, as shown in Figure 6. The region acquired as the degraded image is the same observation region as the field of view 201 of the imaging device 200. Therefore, the degraded image is calculated for an ideal image larger than the field of view 201 of the imaging device 200, including rays from the field of view 201 and the field of view 202 outside the field of view 201, and is acquired in the same observation region as the field of view 201. The ideal image and the degraded image generated by the optical calculation unit 112 are then supplied to the image calculation unit 114.

[0037] The noise image generation unit 113 generates a noise image that shows the noise expected to be generated by the image sensor of the imaging device during imaging. The noise image generation unit 113 can generate an image representing random noise as a noise image.

[0038] For example, the noise image generation unit 113 may generate a noise image according to a Gaussian distribution in order to reproduce random noise that randomly occurs in the captured image due to dark current noise or readout noise, etc. Noise according to a Gaussian distribution is generated as statistical noise having a probability density function equal to that of a Gaussian distribution.

[0039] Furthermore, if a specific type of noise other than random noise occurs in the captured image due to the characteristics of the imaging device 200, the noise image generation unit 113 can use an image taken with the shutter of the imaging device 200 closed as a noise image. Since dark current is temperature-dependent, if noise fluctuations due to temperature changes occur in the imaging device 200, the noise image generation unit 113 can use an image taken with the shutter closed at a different temperature as a noise image. Alternatively, the noise image generation unit 113 may use an image taken with the shutter closed at a temperature equal to the same value as the parameter indicating the temperature of the optical element as a noise image. The noise image generated in this manner is then supplied to the image processing unit 114.

[0040] The image processing unit 114 generates a training image by superimposing a noise image onto a degraded image. Specifically, the image processing unit 114 can generate a training image by multiplying the pixel value of each pixel in the degraded image by the pixel value of the corresponding pixel in the normalized noise image.

[0041] The ideal image and the generated training image are supplied to the training data generation unit 115. As shown in Figure 7, the training data generation unit 115 generates training data, which consists of pairs of training images 206 as training data and images from the ideal image corresponding to the field of view 201 of the imaging device 200 as ground truth data. The generated training data is supplied to the learning unit 116. The image from the ideal image corresponding to the field of view 201 of the imaging device 200 as ground truth data is obtained by cutting out the same area from the ideal image as the field of view 201 of the imaging device 200 and the same area as the degraded image. Note that the same size and area here includes not only cases where the set of pixels is exactly the same, but also cases where there is a difference of several to a dozen pixels, but the size and area are approximately the same. This is because the pixel unit is very small, and such a difference is within an acceptable range for training data. Furthermore, the process of cutting out the image corresponding to the field of view 201 of the imaging device 200 from the ideal image may be performed by the training data generation unit 115, or it may be performed by other functional blocks such as the image processing unit 114. In other words, the training data generation unit 115 only needs to generate training data by pairing training data with correct answer data.

[0042] The learning unit 116 generates a learning model for inferring a corrected image by machine learning using teacher data. For example, the learning unit 116 generates a learning model by inputting teacher data into a neural network for learning. Then, the generated learning model is stored in the learning model storage unit 117.

[0043] The communication unit 118 communicates with the image processing device 130 via the network 101. For example, the communication unit 118 transmits the learning model stored in the learning model storage unit 117 to the image processing device 130.

[0044] The information processing device 110 described above can be realized by a computer such as the PC 150 shown in FIG. 8. The PC 150 may be composed of a plurality of computers connected by a network, or may be composed of a dedicated hardware processing circuit such as a single circuit or a composite circuit.

[0045] The PC 150 includes a storage 151 such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), a memory 152, a processor 153 such as a CPU (Central Processing Unit), a communication I / F (Interface) 154 such as a NIC (Network Interface Card), an input I / F 155 such as a keyboard and a mouse, and a display 156. For example, the learning model storage unit 117 can be realized by the storage 151 or the memory 152.

[0046] The ideal image acquisition unit 111, the optical arithmetic unit 112, the noise image generation unit 113, the image arithmetic unit 114, the teacher data generation unit 115, and the learning unit 116 can be realized by loading an information processing program stored in the storage 151 into the memory 152 and having the processor 153 execute the information processing program. As a result, an information processing method is realized that includes steps of generating a degraded image, generating a noise image, generating a learning image, generating teacher data, and generating a learning model by machine learning using the teacher data. Also, the communication unit 118 can be realized by the communication I / F 154.

[0047] The above information processing program may be loaded onto the memory 152 and executed by the processor 153 after being input from a recording medium (not shown) into the storage 151 via a reader / writer (not shown), or after being downloaded into the storage 151 via the network via the communication I / F 154. Also, it may be directly loaded onto the memory 152 from a recording medium (not shown) via the reader / writer or from the network via the communication I / F 154 and executed by the processor 153. In other words, the information processing program may be provided by a program product such as a recording medium.

[0048] FIG. 9 is a block diagram schematically showing the configuration of the image processing apparatus 130 in the present embodiment. The image processing apparatus 130 includes a communication unit 131, a learning model storage unit 132, a target image acquisition unit 133, and an inference unit 134.

[0049] The communication unit 131 communicates via the network 101. For example, the communication unit 131 receives a learning model from the information processing apparatus 110. The learning model received by the communication unit 131 is stored in the learning model storage unit 132.

[0050] The target image acquisition unit 133 acquires the target image, which is the image to be corrected. For example, the target image acquisition unit 133 may acquire the target image from another device via the communication unit 131, or it may acquire the target image from another device or recording medium via a USB-compatible connector (not shown), etc. The acquired target image is then supplied to the inference unit 134.

[0051] The inference unit 134 retrieves a learning model from the learning model storage unit 132 and inputs the target image into the learning model stored in the learning model storage unit 132, thereby inferring a corrected image, which is the target image corrected. Through this inference, the inference unit 134 infers and outputs a corrected image in which stray light, aberrations, and noise have been corrected from the target image.

[0052] The image processing apparatus 130 described above can be implemented by a computer such as the PC 150 shown in Figure 8. For example, the learning model storage unit 132 can be implemented by storage 151 or memory 152. The target image acquisition unit 133 and the inference unit 134 can be implemented by loading an image processing program stored in storage 151 into memory 152, and having the processor 153 execute the image processing program. As a result, an image processing method is realized that includes the steps of acquiring a target image which is the image to be corrected, and correcting stray light, aberrations, and noise in the target image. The communication unit 131 can be implemented by a communication interface 154.

[0053] Figure 10 is a flowchart showing an example of processing in the information processing device 110. In step S10, the ideal image acquisition unit 111 acquires an ideal image, which is an image to be treated as the correct answer data for the training data.

[0054] In step S11, the optical calculation unit 112 tracks the optical paths of light rays from an area larger than the field of view of the imaging device 200, calculates the optical paths of light from the field of view 201 and the field of view 202 which is outside the field of view, calculates stray light using optical simulation, and calculates the PSF of optical elements using optical simulation, and generates a degraded image by applying the stray light and PSF to the ideal image, respectively.

[0055] In step S12, the noise image generation unit 113 generates a noise image that shows the noise expected to be generated by the imaging device 200.

[0056] In step S13, the image processing unit 114 generates a training image by superimposing the degraded image generated by the optical processing unit 112 and the noise image generated by the noise image generation unit 113.

[0057] In step S14, the training data generation unit 115 uses the learning images generated by the image processing unit 114 as training data and generates training data in which the images corresponding to the imaging field of view of the imaging device 200 from among the ideal images are used as correct answer data.

[0058] In step S15, the learning unit 116 generates a learning model and terminates the process by training a model for inferring corrected images using the training data generated by the training data generation unit 115.

[0059] Figure 11 is a flowchart showing an example of processing in the image processing device 130. In step S20, the target image acquisition unit 133 acquires the target image, which is the image to be corrected.

[0060] In step S21, the inference unit 134 inputs the target image into the learning model stored in the learning model storage unit 132, and then infers a corrected image, which is an image in which stray light, aberrations, and noise have been corrected from the target image, and then terminates the process.

[0061] As described above, according to this embodiment, by calculating the optical path of light from the imaging field of view of the imaging device and the optical path of light from outside the imaging field of view that is larger than the imaging field of view, and by calculating stray light outside the imaging field of view in addition to stray light within the imaging field of view, the effect of stray light generated by optical elements with special shapes such as Fresnel lenses can be accurately calculated. In other words, by taking into account the effect of stray light from outside the field of view, it becomes possible to generate training images to which aberration components, noise components, and stray light components are added.

[0062] Furthermore, according to this embodiment, when setting parameters indicating information about the mask and lens barrel, manufacturing tolerances and alignment errors of the optical elements, the effects of stray light and aberrations due to the presence of the mask and lens barrel, as well as the manufacturing tolerances and alignment errors of the optical elements, can be appropriately corrected. In addition, when setting parameters for the temperature of the optical elements, it becomes possible to appropriately correct not only stray light and aberrations caused by changes in the shape of the optical elements, optical path length, and refractive index of the lenses due to temperature changes in the optical system, but also temperature-dependent sensor-derived noise such as dark current.

[0063] In the embodiment described above, the image processing system 100 is configured with an information processing device 110 and an image processing device 130, but the system is not limited to this configuration. For example, the information processing device 110 and the image processing device 130 may be directly connected without going through the network 101. Also, the image processing system 100 may consist of one device or three or more devices. In other words, the functional units of the information processing device 110 and the image processing device 130 can be appropriately arranged in one or more devices such as computers. Specifically, the information processing device 110 may perform processing up to the generation of training data and transmit that training data to another device, thereby generating a learning model in the other device. Alternatively, the learning model generated by the information processing device 110 may be stored in another server such as a cloud, and the image processing device 130 may transmit the target image acquired and necessary information to the server, and the server may infer a corrected image. In such a case, the image processing device 130 obtains the inference result from the server.

[0064] Furthermore, the learning process in the information processing device 110 may be carried out continuously after the processing in the image processing device 130 has started. For example, if a sensor that was not present during the learning model's training appears, learning may be carried out using noise images from that sensor after its appearance. Also, if an optical element that was not present during the learning model's training appears, learning may be carried out by calculating stray light and PSF from that optical element after its appearance. In addition, although the information processing device 110 is described as a device for generating a learning model in this embodiment, it may also be configured as a device for generating training data.

[0065] 100 Image processing system, 110 Information processing device, 111 Ideal image acquisition unit, 112 Optical calculation unit, 113 Noise image generation unit, 114 Image calculation unit, 115 Training data generation unit, 116 Learning unit, 117 Learning model storage unit, 118 Communication unit, 119 Stray light calculation unit, 120 PSF calculation unit, 130 Image processing device, 131 Communication unit, 132 Learning model storage unit, 133 Target image acquisition unit, 134 Inference unit, 200 Imaging device, 201 Field of view of imaging device, 202 Outside of field of view of imaging device, 203 Fresnel lens, 204 Image sensor, 205 Lens barrel, 206 Training image.

Claims

1. An information processing device comprising: an optical calculation unit that calculates the optical path of light from the imaging field of view and the optical path of light from outside the imaging field of view, which is outside the imaging field of view, in an imaging device that takes images using optical elements, and calculates stray light caused by the optical elements, and a PSF calculation unit that calculates the PSF (Point Spread Function) of the optical elements, wherein the optical calculation unit generates a degraded image by applying the stray light and the PSF to an ideal image which is treated as correct answer data for training data; a noise image generation unit that generates a noise image which includes noise that is predicted to be generated by the image sensor of the imaging device during imaging; an image calculation unit that generates a training image by superimposing the noise image onto the degraded image; and a training data generation unit that generates training data which uses the training image as training data and uses an image from the ideal image which corresponds to the imaging field of view of the imaging device as correct answer data.

2. The information processing apparatus according to claim 1, characterized in that the optical calculation unit calculates the stray light and the PSF respectively such that the influence of the alignment error is included in each of the stray light and the PSF, based on the setting of a parameter indicating the alignment error of the optical element.

3. The information processing apparatus according to claim 1 or 2, characterized in that the optical calculation unit calculates the stray light and the PSF, respectively, based on the setting of parameters indicating the manufacturing tolerance of the optical element, such that the stray light and the PSF are each affected by the manufacturing tolerance.

4. The information processing apparatus according to any one of claims 1 to 3, characterized in that the optical calculation unit calculates the stray light and the PSF, respectively, based on the setting of a parameter indicating the temperature of the optical element, such that the temperature influence is included in each of the stray light and the PSF.

5. The information processing apparatus according to any one of claims 1 to 4, characterized in that the optical calculation unit calculates the stray light based on the settings of the lens barrel so as to include the influence of the lens barrel.

6. The information processing apparatus according to any one of claims 1 to 5, characterized in that the noise image generation unit generates an image representing random noise as the noise image.

7. The information processing apparatus according to any one of claims 1 to 6, characterized in that the noise image generation unit closes the shutter of the imaging device and generates the image captured by the image sensor as the noise image.

8. The information processing apparatus according to any one of claims 1 to 7, further comprising a learning unit that generates a learning model for inferring a corrected image using the training data.

9. An image processing apparatus comprising: an image acquisition unit that acquires a target image which is an image to be corrected; an inference unit that calculates the optical path of light from the imaging field of view of an imaging device that takes images using optical elements, and the optical path of light from outside the imaging field of view which is outside the imaging field of view, calculates stray light from the optical elements and the PSF (Point Spread Function) of the optical elements, and uses a training image generated by superimposing a noise image containing noise predicted to occur at the imaging element of the imaging device during imaging onto a degraded image obtained by applying the stray light and the PSF to an ideal image which is treated as the correct answer data for the training data, and inputs the target image into a learning model for inferring a corrected image which is generated using training data in which the image corresponding to the imaging field of view of the imaging device from the ideal images is the correct answer data; and inference unit that infers a corrected image which is target image which is an image to be corrected from the target image.

10. An imaging device that uses optical elements to take images has an optical calculation unit that calculates the optical path of light from the imaging field of view and the optical path of light from outside the imaging field of view, and calculates stray light caused by the optical elements, and a PSF calculation unit that calculates the PSF (Point Spread Function) of the optical elements, and generates a degraded image by applying the stray light and the PSF to an ideal image which is treated as the correct answer data for training data; a noise image generation unit that generates a noise image which includes noise that is predicted to occur in the image sensor of the imaging device during imaging; an image calculation unit that superimposes the noise image on the degraded image to generate a training image; a training data generation unit that uses the training image as training data and generates training data which includes an image from the ideal image which corresponds to the imaging field of view of the imaging device as the correct answer data; a learning unit that uses the training data to generate a learning model for inferring a corrected image; and a target image acquisition unit that acquires a target image which is the image to be corrected. An image processing system characterized by comprising: an inference unit that infers a corrected image obtained by correcting the target image by inputting the target image to the learning model; 11. An information processing method performed by a computer, comprising: a step of calculating the optical path of light from the imaging field of view and the optical path of light from outside the imaging field of view in an imaging device that takes images using an optical element, and calculating stray light from the optical element; a step of calculating the Point Spread Function (PSF) of the optical element; a step of generating a degraded image by applying the stray light and the PSF to an ideal image which is treated as correct answer data for training data; a step of generating a noise image which includes noise that is predicted to occur in the image sensor of the imaging device during imaging; a step of superimposing the noise image onto the degraded image to generate a training image; and a step of generating training data which uses the training image as training data and the image from the ideal image corresponding to the imaging field of view of the imaging device as correct answer data.

12. An image processing method performed by a computer, comprising: a step of acquiring a target image which is an image to be corrected; a step of calculating the optical path of light from the imaging field of view of an imaging device that takes images using optical elements, and the optical path of light from outside the imaging field of view which is outside the imaging field of view, calculating stray light from the optical elements and the PSF (Point Spread Function) of the optical elements, and using a training image generated by superimposing a noise image containing noise expected to occur at the imaging element of the imaging device during imaging onto a degraded image obtained by applying the stray light and the PSF to an ideal image which is treated as the correct answer data for the training data, and inputting the target image into a learning model for inferring a corrected image which is generated using training data which is the image corresponding to the imaging field of view of the imaging device from the ideal images which is the correct answer data; 13. An information processing program that causes a computer to perform the following steps:

13. Calculate the optical path of light from the imaging field of view and the optical path of light from outside the imaging field of view in an imaging device that takes images using an optical element, and calculate stray light caused by the optical element; 13. Calculate the Point Spread Function (PSF) of the optical element; 24. Generate a degraded image by applying the stray light and the PSF to an ideal image which is treated as the correct answer data for training data; 24. Generate a noise image which includes noise that is predicted to occur in the image sensor of the imaging device during imaging; 35. Generate a training image by superimposing the noise image onto the degraded image; 46. Generate training data which uses the training image as training data and the image from the ideal image corresponding to the imaging field of view of the imaging device as the correct answer data.

14. An image processing program that causes a computer to perform the following steps:

14. Obtain a target image which is an image to be corrected; calculate the optical path of light from the imaging field of view of an imaging device that takes images using optical elements, and the optical path of light from outside the imaging field of view which is outside the imaging field of view; calculate the stray light from the optical elements and the PSF (Point Spread Function) of the optical elements; apply the stray light and the PSF to an ideal image which is treated as the correct answer data for the training data; superimpose a noise image containing noise that is expected to occur at the imaging sensor of the imaging device during imaging onto this degraded image, using the training image as training data; input the target image into a learning model for inferring a corrected image which is a corrected image which is a corrected image which is a corrected image which is a corrected image which is a target image which is an image to be corrected from the ideal image which is treated as the correct answer data for the training data;

Images