Image processing apparatus, image processing method, and program

Abstract

To acquire data of an image reproducing an SPAD image.SOLUTION: An image processing apparatus 100 according to the present disclosure sets, for every pixel of a first image, a pixel value corresponding to the first image to a parameter representing the number of photons incident per unit time on an image pick-up device performing photoelectric conversion based on photon counting, in the probability distribution representing the stochastic variation of a pixel value, due in part to statistical fluctuation of the number of photons incident on the image pick-up device, and generates data of a second image on the basis of a random number derived on the basis of the probability distribution.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to image data generation techniques. [Background technology]

[0002] SPAD (Single Photon Avalanche Diode) sensors have been attracting attention as imaging elements different from conventional imaging elements such as CCD (Charge Coupled Device) sensors or CMOS (Complementary Metal Oxide Semiconductor) sensors. SPAD sensors perform photoelectric conversion based on photon counting, which counts the number of photons (hereinafter referred to as "incident photons") incident on the sensor. That is, in an image (hereinafter referred to as "SPAD image") obtained by imaging with a SPAD sensor, the number of incident photons is converted to become the pixel value of the SPAD image. Therefore, in SPAD sensors, the statistical fluctuation of the number of counted photons (hereinafter referred to as "statistical fluctuation") is the main source of noise, and the readout noise that exists in conventional imaging elements does not occur. For these reasons, SPAD sensors have the characteristic of high dark performance and are expected to be used for surveillance applications, etc., but SPAD sensors are still in the development stage and have not yet become widespread.

[0003] As with conventional imaging elements, by applying a predetermined image processing to the output from the SPAD sensor, it is possible to obtain effects such as reducing noise and improving sharpness. In particular, in recent years, the above-mentioned effects can be further improved by using a learning model such as a neural network. For example, Patent Document 1 discloses a technology in which processing based on a neural network is applied to the output of a photoelectric conversion element based on photon counting. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-119380 A Summary of the Invention [Problem to be solved by the invention]

[0005] In order to realize the neural network disclosed in Patent Document 1, it is necessary to collect a large amount of image data (SPAD image) that is the output of a photoelectric conversion element based on photon counting as training data and train the neural network. In addition, a large amount of SPAD image data may be required when a developer of an image processing circuit or program to be incorporated into an imaging device including a SPAD sensor tests the image processing to be developed on data of various images. However, since the SPAD sensor is a device that has not yet been widely used as described above, it is difficult to collect a large amount of SPAD image data.

[0006] In view of the above-mentioned problems, an object of the present disclosure is to provide a technology capable of acquiring data of an image that reproduces a SPAD image. [Means for solving the problem]

[0007] The image processing device according to the present disclosure has a setting means for setting, for each pixel of a first image, a parameter representing the number of photons incident on an imaging element per unit time of a probability distribution representing probabilistic fluctuations in pixel values ​​caused in part by statistical fluctuations in the number of photons incident on the imaging element that performs photoelectric conversion based on photon counting, to a corresponding pixel value in the first image, and a generating means for generating data of a second image based on a random number derived based on the probability distribution. Effect of the Invention

[0008] According to the present disclosure, image data that reproduces a SPAD image can be acquired. [Brief description of the drawings]

[0009] [Figure 1]1 is a block diagram showing an example of a functional configuration of an image processing device according to a first embodiment. [Diagram 2] 1 is a block diagram showing an example of a hardware configuration of an image processing device according to a first embodiment. [Diagram 3] 5 is a flowchart showing an example of a processing flow in the image processing device according to the first embodiment. [Figure 4] FIG. 11 is a block diagram showing an example of a functional configuration of an image processing device according to a second embodiment. [Diagram 5] 10 is a flowchart showing an example of a processing flow in an image processing device according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The following embodiments do not limit the present disclosure, and not all of the combinations of features described in the following embodiments are necessarily essential to the solution of the present disclosure. The configurations of the following embodiments may be appropriately modified or changed depending on the specifications of the device to which the present disclosure is applied, or various conditions such as the conditions of use or the environment of use. In addition, the configurations may be appropriately combined with parts of the embodiments described below. In the following embodiments, the same components are described with the same reference numerals.

[0011] [First embodiment] <Configuration of image processing device> The configuration of an image processing device 100 will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is a block diagram showing an example of a functional configuration of the image processing device 100 according to the first embodiment. The image processing device 100 has, as its functional configuration, an image acquisition unit 110, a probability distribution parameter setting unit 120, an image generation unit 130, a linearization correction unit 140, a learning unit 150, and a trained model output unit 160. The processing of each unit that the image processing device 100 has as its functional configuration will be described later.

[0012] 2 is a block diagram showing an example of a hardware configuration of the image processing device 100 according to the first embodiment. The image processing device 100 has, as its hardware configuration, a random access memory (RAM) 201, a central processing unit (CPU) 202, a hard disk drive (HDD) 203, and a graphics processing unit (GPU) 204. The RAM 201, the CPU 202, the HDD 203, and the GPU 204 are connected to each other via a data bus 205 so as to be able to transmit and receive information to and from each other.

[0013] The CPU 202 performs various controls by reading out a control program stored in the HDD 203, loading it in the RAM 201, and executing the control program. The CPU 202 also performs processing of each part related to the above-mentioned functional configuration by reading out an image processing program according to the present embodiment stored in the HDD 203, loading it in the RAM 201, and executing the image processing program. The GPU 204 cooperates with the CPU 202 to perform part of the processing of the above-mentioned image processing program executed by the CPU 202.

[0014] The image processing program according to the present embodiment includes a probability distribution parameter setting processing program. The processing in the probability distribution parameter setting unit 120 can be realized by the CPU 202 executing the probability distribution parameter setting processing program stored in the HDD 203. The image processing program includes an image generation processing program, and the processing in the image generation unit 130 can be realized by the CPU 202 executing the image generation processing program stored in the HDD 203. The image processing program includes a linearization correction processing program, and the processing in the linearization correction unit 140 can be realized by the CPU 202 executing the linearization correction processing program stored in the HDD 203. The image processing program includes a learning processing program, and the processing in the learning unit 150 can be realized by the CPU 202 or the GPU 204 executing the learning processing program stored in the HDD 203. The RAM 201 is used as a temporary storage area for expanding the control program and the image processing program executed by the CPU 202 and the GPU 204.

[0015] A program for realizing one or more functions of the image processing device 100 according to this embodiment may be supplied to a system or device via a network or a storage medium. Then, one or more processors in a computer of the system or device to which the program is supplied read and execute the program, thereby realizing one or more of the above-mentioned functions. Also, the above-mentioned one or more functions may be realized by a circuit such as an ASIC (application specific integrated circuit) that realizes one or more of the above-mentioned functions. The present embodiment merely shows examples of the embodiment of the present disclosure, and the technical scope of the present disclosure should not be interpreted as being limited by these. In other words, the present disclosure may be implemented in various forms without departing from its technical concept or main features.

[0016] The image processing device 100 first acquires data of a predetermined image in which luminance is expressed linearly (hereinafter referred to as a "luminance linear image"). Next, the image processing device 100 sets the corresponding pixel value in the luminance linear image to a parameter representing the number of incident photons per unit time included in the probability distribution for each pixel value of the acquired luminance linear image. Next, the image processing device 100 generates image data based on a random number derived based on this probability distribution. Here, this probability distribution represents a probabilistic fluctuation of pixel values ​​caused in part by the statistical fluctuation of the number of counted photons. Hereinafter, as an example, a form in which the data of the generated image is used as learning data for learning a learning model will be described, but the use of the data of the generated image is not limited to learning a learning model. For example, the data of the image generated by the image processing device 100 according to this embodiment can be used for various other processes or steps requiring data of an image (SPAD image) obtained by imaging with a SPAD sensor by applying appropriate corrections or changes.

[0017] <Statistical fluctuations in the number of photons> Before describing the processing of each unit that the image processing device 100 has as a functional configuration, we will explain noise in a SPAD image caused by statistical fluctuations in the number of counted photons (hereinafter referred to as "optical shot noise") and a method for mathematically modeling the noise. The light incident on the imaging element is composed of particles called photons. Since there is randomness in the timing at which each photon arrives at the imaging element, fluctuations occur in the number of photons arriving at the imaging element even with a constant exposure time, and this is observed in the form of optical shot noise in the SPAD image. Optical shot noise is particularly noticeable in SPAD images when the incident light intensity is low. When the average number of incident photons per unit time is λ p and the number of incident photons per unit time is k, the probability distribution P of the number of incident photons k can be expressed by a Poisson distribution as shown in the following equation (1). JPEG2025025972000002.jpg16148

[0018] Among image sensors that output signals based on photon counting, in the case of an image sensor that outputs a signal proportional to the number of photons arriving within a predetermined period, the light shot noise can be expressed based on formula (1). On the other hand, in the case of an image sensor that outputs a signal based on clock recharge drive or event recharge drive, the light shot noise can be expressed based on a formula obtained by incorporating a formula obtained based on formula (1) into a formula showing another probability distribution. For example, in the case of an image sensor that outputs a signal based on clock recharge drive, the light shot noise can be expressed based on the following formula (2) obtained by incorporating a formula based on formula (1) into a formula showing a binomial distribution. Here, n represents the number of divisions of a unit time. JPEG2025025972000003.jpg17147

[0019] Ideally, in photon counting, it is preferable that all photons incident on the image sensor are counted, and that no other photons are counted. However, in reality, only a portion of the incident photons may be counted. The ratio of the number of incident photons to the number of all incident photons is called quantum efficiency or detection efficiency. In addition, photons may be counted as having been input even though they are not incident due to heat or other factors. Such counting is called dark counting. If the quantum efficiency is q and the dark count per unit time is d, the probability distribution P of the number of photons (hereinafter referred to as the "photoelectric conversion number") k that are generated per unit time in a clock recharge driven image sensor can be expressed by the following formula (3). JPEG2025025972000004.jpg17145

[0020] <Processing of each unit in the functional configuration of the image processing device> The processing of each unit that the image processing device 100 has as a functional configuration will be described below. The image acquisition unit 110 acquires data of a luminance linear image. Here, the luminance linear image data is, for example, image data output from an image input device such as an imaging device, and image processing such as gamma correction processing has not been performed on the image. The luminance linear image data can be collected by any method, for example, by using an imaging device equipped with a conventional imaging element such as a general CMOS sensor, or by collecting via the Internet. Depending on the collection method, the collected image data may be a luminance linear image to which gamma correction processing such as sRGB has been applied. In that case, the collected image data may be converted into luminance linear image data by applying de-gamma correction processing to the collected image data. Furthermore, the linearity of the luminance linear image is maintained even if the pixel value is multiplied by a constant. Therefore, the image acquisition unit 110 may acquire data in which each pixel value of the luminance linear image collected in advance is multiplied by a predetermined value as luminance linear image data, or may multiply each pixel value of the acquired luminance linear image by a predetermined value.

[0021] Furthermore, the image acquiring unit 110 acquires, in addition to the luminance linear image data, information necessary for learning a learning model, which will be described later, as necessary. For example, the image acquiring unit 110 acquires information indicating the presence or absence of an abnormality to be used as a correct answer label for learning, together with the luminance linear image data output from an imaging device for monitoring. Also, for example, the image acquiring unit 110 acquires information indicating the position of a lesion site to be used as a correct answer label for learning, together with medical luminance linear image data obtained by a CT (Computed Tomography) scan or the like. The information acquired together with the luminance linear image data is not limited to the above-mentioned information for monitoring or medical use, and may be information suited to the application field of the image processing device 100.

[0022] The probability distribution parameter setting unit 120 sets, for each pixel of the luminance linear image acquired by the image acquisition unit 110, a corresponding pixel value in the luminance linear image to a parameter representing the number of incident photons per unit time included in the probability distribution P shown in equations (1) to (3) etc. Since the parameter representing the number of incident photons per unit time included in the probability distribution P needs to be proportional to the luminance, it is preferable to use data such as a RAW image obtained by capturing an image using a conventional imaging element such as a CMOS sensor as the luminance linear image data. The image generation unit 130 generates data of a new image by determining a pixel value for each pixel based on the probability distribution P.

[0023] The linearization correction unit 140 applies linearization correction processing for removing nonlinearity resulting from dark counts or photon saturation, or the driving method, to the image data generated by the image generation unit 130. For example, when clock recharge driving is adopted as the driving method and image data is generated based on the probability distribution P shown in equation (3), the expected value of the number k of photoelectric conversions per unit time is expressed by the following equation (4). JPEG2025025972000005.jpg19145

[0024] Therefore, in order to remove the nonlinearity caused by the clock recharge driving contained in the image, it is sufficient to apply the processing f shown in the following equation (5), which is the inverse function of equation (4). JPEG2025025972000006.jpg19145

[0025] The learning unit 150 updates the weight parameters of the learning model using the image data after the linearization correction process is performed by the linearization correction unit 140 and the information required for learning the learning model acquired by the image acquisition unit 110. Here, the learning model may be a statistical machine learning model or a deep learning model. The learned model output unit 160 outputs the learning model in which the weight parameters are optimized by the learning unit 150 as a learned model. The output learned model is acquired by an external image processing device or information processing device (not shown) and is used for inference processing in these devices.

[0026] <About the operation of the image processing device> FIG. 3 is a flowchart showing an example of a process flow in the image processing device 100 according to the first embodiment. In each of the following flowcharts, the reference symbol "S" represents a process step (process). First, in S301, the image acquisition unit 110 acquires data of a luminance linear image to be processed. The luminance linear image data acquired in S301 is transmitted to the probability distribution parameter setting unit 120 and the image generating unit 130. In addition, in S301, the image acquisition unit 110 also acquires information necessary for learning the learning model, as necessary, that corresponds to the data of the luminance linear image to be processed. The information necessary for learning the learning model acquired in S301 is transmitted to the learning unit 150.

[0027] Next, in S302, the probability distribution parameter setting unit 120 calculates, for each pixel of the luminance linear image acquired in S301, λ, which is a parameter of the probability distribution P shown in the equations (1) to (3), etc. pIn step S304, the linearization correction unit 140 executes linearization correction processing to remove nonlinearity contained in the image by executing processing f shown in equation (5) on the image data generated in step S303. In step S304, the linearization correction unit 140 executes linearization correction processing to remove nonlinearity contained in the image by executing processing f shown in equation (5) on the image data generated in step S303. In step S304, the linearization correction unit 140 executes linearization correction processing to remove nonlinearity contained in the image by executing processing f shown in equation (5) on the image data generated in step S303. In step S304, the linearization correction unit 140 executes linearization correction processing to remove nonlinearity contained in the image by executing processing f shown in equation (5). ...).

[0028] Next, in S305, the learning unit 150 updates the weight parameters of the learning model using the image data after the linearization correction process in S304 and, if necessary, the information required for learning the learning model acquired in S301 as learning data. In this embodiment, the learning method is an online learning method in which the weight parameters of the learning model are updated for each piece or set of learning data, but the learning method is not limited to online learning. For example, it may be mini-batch learning in which the weight parameters of the learning model are updated for each piece or set of learning data, or batch learning in which the weight parameters of the learning model are updated using all learning data.

[0029] In the present embodiment, the image data that has been subjected to linearization correction processing is used as is as learning data, but the image data used as learning data does not necessarily need to be luminance linear, and can be changed in various ways depending on the conditions of use of the trained model. For example, when the input to the trained model at the time of inference is SPAD image data after applying sRGB gamma correction processing, the image data after applying sRGB gamma characteristics to the image data that has been subjected to linearization correction processing may be used as learning data to execute the above-mentioned learning process.

[0030] After S305, in S306, the trained model output unit 160 determines whether the learning end condition is satisfied. Specifically, for example, the trained model output unit 160 determines whether the learning end condition is satisfied by determining whether a predetermined number of learnings have been performed, or whether the weight parameters of the learning model are more stable than a predetermined standard. If it is determined in S306 that the learning end condition is not satisfied, the image processing device 100 returns to the process of S301 and repeatedly executes the processes from S301 to S306 until it is determined in S306 that the learning end condition is satisfied. If it is determined in S306 that the learning end condition is satisfied, in S307, the trained model output unit 160 outputs the learning model to be subjected to the learning process as a trained model. After S307, the image processing device 100 ends the process of the flowchart shown in FIG. 3.

[0031] According to the image processing device 100 configured as above, it is possible to generate image data reproducing a SPAD image using readily available image data such as image data obtained using a conventional imaging element such as a CMOS sensor. Furthermore, it is possible to use the image data generated in this way to learn weight parameters of a learning model for a predetermined application to the SPAD image data.

[0032] [Second embodiment] In the first embodiment, an embodiment in which image data reproducing a SPAD image is generated, and the generated image data is used to learn weight parameters of a learning model, as an example. In the second embodiment, an embodiment in which image data reproducing a plurality of SPAD images having different amounts of optical shot noise is generated is described. In addition, as an example, an embodiment in which weight parameters of a learning model that reduces optical shot noise contained in a SPAD image are learned by using the generated image data as a learning data pair is described. Note that in the second embodiment, an embodiment in which image data reproducing two types of SPAD images having different amounts of optical shot noise is generated from one input image data is described.

[0033] FIG. 4 is a block diagram showing an example of a functional configuration of an image processing device 100 according to a second embodiment. In the functional configuration shown in FIG. 4, the description of functional units that are the same or substantially the same as those in the functional configuration of FIG. 1 referred to in the description of the first embodiment will be omitted as appropriate. The image processing device 100 according to the second embodiment (hereinafter, simply referred to as "image processing device 100") has a gain division unit 470 and a gain multiplication unit 480 in addition to the functional configuration of the image processing device 100 according to the first embodiment shown in FIG. 1. Also, the number of paths of the data flow from the image acquisition unit 110 to the learning unit 150 is increased to two, and one of the two paths includes the gain division unit 470 and the gain multiplication unit 480.

[0034] The gain division unit 470 and the gain multiplication unit 480 can be realized by the CPU 202 executing an image processing program. Specifically, the image processing program includes a gain division processing program, and the processing in the gain division unit 470 can be realized by the CPU 202 executing the gain division processing program stored in the HDD 203. The image processing program includes a gain multiplication processing program, and the gain multiplication processing in the gain multiplication unit 480 can be realized by the CPU 202 executing the gain multiplication processing program stored in the HDD 203.

[0035] The gain division unit 470 executes a gain division process to divide each pixel value of the luminance linear image acquired by the image acquisition unit 110 by a first gain value. The gain multiplication unit 480 executes a gain multiplication process to multiply each pixel value of the image after the linearization correction process has been performed by the linearization correction unit 140 by a second gain value. The learning unit 150 acquires data of an image (hereinafter referred to as a "first image") that has passed through a path including the gain division process and the gain multiplication process, and data of an image (hereinafter referred to as a "second image") that has passed through a path not including the gain division process and the gain multiplication process. The learning unit 150 learns weight parameters of the learning model using the data of the first image and the second image, and information required for learning the learning model acquired by the image acquisition unit 110.

[0036] Applying division by the first gain value to the luminance linear image corresponds to reproducing imaging with changed exposure conditions such as aperture or exposure time. Therefore, image data in which a SPAD image with different exposure conditions is reproduced is generated between a path including gain division processing and a path not including gain division processing. Therefore, by adding the gain division unit 470 and adding image data that has passed through a path including gain division processing as learning data for learning weight parameters of a learning model, it becomes possible to learn about conversion between two images with different exposure conditions.

[0037] Multiplying each pixel value by the second gain value at the end of the pass including the gain division process has the effect of adjusting the brightness of the image by the digital gain. When the first gain value is sufficiently larger than 1, the brightness of the image greatly differs between the pass including the gain division process and the pass not including the gain division process, but the gain multiplication process by the gain multiplication unit 480 can compensate for the difference in brightness between the images. In particular, when the first gain value and the second gain value are approximately the same, image data with the same exposure is generated in the pass including the gain division process and the pass not including the gain division process. Therefore, in this case, data of two images having the same brightness and different amounts of noise are obtained, and these are used as a learning data pair for learning the learning model. For example, by learning a learning model that converts the data of the image with the larger amount of noise of the two image data constituting the learning data pair into the data of the other image, a learned model that reduces noise contained in a SPAD image can be obtained.

[0038] Fig. 5 is a flowchart showing an example of a process flow in the image processing device 100 according to the second embodiment. Note that in the flowchart of Fig. 5, explanations of steps that are the same or substantially the same as those in the flowchart shown in Fig. 3, which were referred to in explaining the first embodiment, will be omitted as appropriate. The flowchart shown in Fig. 5 includes, in addition to steps similar to those in the flowchart shown in Fig. 3, S501 for executing a gain division process and S505 for executing a gain multiplication process. Also, in the flowchart shown in Fig. 5, processes from S502 to S504 are added as processes similar to those in the process flow from S302 to S304, as in Fig. 4 showing an example of the functional configuration of the image processing device 100.

[0039] First, the image processing device 100 executes the processes from S301 to S304. After S304, in S501, the gain division unit 470 executes a gain division process of dividing each pixel value of the luminance linear image acquired in S301 by a first gain value. After S501, the image processing device 100 executes the processes from S502 to S504. The processes from S502 to S504 are similar to the processes from S302 to S304, and therefore description thereof will be omitted. Note that the process in S502 is performed based on the data of the luminance linear image after the gain division process in S501 has been performed.

[0040] After S504, in S505, the gain multiplication unit 480 executes a gain multiplication process of multiplying each pixel value of the image after the linearization correction process in S504 by a second gain value. After S505, the image processing device 100 executes a process of S506. The process of S506 is similar to the process of S305, and therefore a description thereof will be omitted. After S506, the image processing device 100 repeatedly executes the processes of S301 to S306 until it is determined in S306 that the learning end condition is satisfied. If it is determined in S306 that the learning end condition is satisfied, the image processing device 100 executes a process of S307. After S307, the image processing device 100 ends the process of the flowchart shown in FIG. 5. In this embodiment, the processing of S501 to S505 is described as being executed after S304, but the processing of S302 to S304 may be executed after S505, or S501 to S505 and S302 to S304 may be processed in parallel.

[0041] According to the image processing device 100 configured as above, it is possible to generate image data reproducing a plurality of types of SPAD images with different amounts of noise by using one easily available image data such as image data obtained by using a conventional imaging element. In addition, by using a pair of learning data in which one of the plurality of image data thus generated with a larger amount of noise is used as input data and the other is used as output data, it is possible to learn weight parameters of a learning model that reduces noise contained in a SPAD image. That is, according to the image processing device 100, by adding an additional configuration to the image processing device 100 according to the first embodiment, it is possible to realize learning of a learning model that reduces noise contained in a SPAD image, among learning models to be applied to SPAD image data.

[0042] [Other embodiments] The present disclosure can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0043] It should be noted that, within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0044] [Configuration of this disclosure] The present disclosure includes the following configurations, methods, and programs.

[0045] <Configuration 1> a setting means for setting, for each pixel of a first image, a parameter representing the number of photons incident on an image sensor per unit time, which parameter represents a probabilistic variation in pixel value caused in part by a statistical fluctuation in the number of photons incident on the image sensor that performs photoelectric conversion based on photon counting; and a generating means for generating data of a second image based on a random number derived based on the probability distribution; 13. An image processing device comprising:

[0046] <Configuration 2> the first image is an image in which luminance is expressed linearly; 2. The image processing device according to claim 1,

[0047] <Configuration 3> a learning means for learning a learning model by using the data of the second image as learning data;

[0033] 3. The image processing device according to configuration 1 or 2,

[0048] <Configuration 4> a correction means for performing a correction for removing nonlinearity in luminance on the data of the second image;

[0033] 4. The image processing device according to any one of configurations 1 to 3,

[0049] <Component 5> A learning means for learning a learning model by using the data of the second image corrected by the correction means as learning data;

[0033] 5. The image processing device according to configuration 4,

[0050] <Component 6> a division means for dividing each pixel value in the first image by a first gain value; Further comprising: the setting means sets, for each pixel in a third image that is the first image after the division process by the division means, a corresponding pixel value in the third image to the parameter of the probability distribution; the generating means generates data of the second image corresponding to the third image based on a random number derived based on the probability distribution; 6. The image processing device according to any one of configurations 1 to 5,

[0051] <Component 7> a multiplication means for multiplying each pixel value in the second image by a second gain value; 7. The image processing device according to configuration 6,

[0052] <Component 8> a correction means for performing a correction for removing nonlinearity in luminance on the data of the second image; Further comprising: the multiplication means multiplying each pixel value in the second image after the correction process by the correction means by the second gain value; 8. The image processing device according to configuration 7,

[0053] <Component 9> the first gain value and the second gain value are substantially equal; 9. The image processing device according to configuration 7 or 8,

[0054] <Component 10> A learning means for learning a learning model by using the data of the second image after the multiplication process by the multiplication means as learning data;

[0033] 10. The image processing device according to any one of configurations 7 to 9,

[0055] <Component 11> the learning means uses the data of the second image and the data of the second image after the multiplication process by the multiplication means as a learning data pair to learn the learning model; 11. The image processing device according to claim 10,

[0056] <Method> a setting step of setting, for each pixel of the first image, a parameter representing the number of photons incident on an image sensor per unit time, which parameter represents a probabilistic variation in pixel value caused in part by a statistical fluctuation in the number of photons incident on the image sensor that performs photoelectric conversion based on photon counting; A generating step of generating data of a second image based on a random number derived based on the probability distribution; 13. An image processing method comprising:

[0057] <Program> A program for causing a computer to function as the image processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0058] 100 Image processing device 120 Probability distribution parameter setting unit 130 Image Generation Unit

Claims

1. a setting means for setting, for each pixel of a first image, a parameter representing the number of photons incident on an image sensor per unit time, the parameter representing a probabilistic variation in pixel value caused in part by statistical fluctuation in the number of photons incident on the image sensor that performs photoelectric conversion based on photon counting; and a generating means for generating data of a second image based on a random number derived based on the probability distribution; 13. An image processing device comprising:

2. the first image is an image in which luminance is expressed linearly; 2. The image processing device according to claim 1,

3. a learning means for learning a learning model by using the data of the second image as learning data; [0033] 2. The image processing device according to claim 1,

4. a correction means for performing a correction for removing nonlinearity of luminance on the data of the second image; [0033] 2. The image processing device according to claim 1,

5. A learning means for learning a learning model by using the data of the second image corrected by the correction means as learning data; [0033] 5. The image processing device according to claim 4,

6. a division means for dividing each pixel value in the first image by a first gain value; and the setting means sets, for each pixel in a third image that is the first image after the division process by the division means, a corresponding pixel value in the third image to the parameter of the probability distribution; the generating means generates data of the second image corresponding to the third image based on a random number derived based on the probability distribution; 2. The image processing device according to claim 1,

7. a multiplication means for multiplying each pixel value in the second image by a second gain value; 7. The image processing device according to claim 6,

8. a correction means for performing a correction for removing nonlinearity of luminance on the data of the second image; and the multiplication means multiplying each pixel value in the second image after the correction process by the correction means by the second gain value; 8. The image processing device according to claim 7,

9. the first gain value and the second gain value are substantially equal; The image processing device according to claim 7 ,

10. A learning means for learning a learning model by using the data of the second image after the multiplication process by the multiplication means as learning data; [0033] The image processing device according to claim 7 ,

11. the learning means uses the data of the second image and the data of the second image after the multiplication process by the multiplication means as a learning data pair to learn the learning model; The image processing device according to claim 10 .

12. a setting step of setting, for each pixel of the first image, a parameter representing the number of photons incident on an image sensor per unit time, which parameter represents a probabilistic variation in pixel value caused in part by a statistical fluctuation in the number of photons incident on the image sensor that performs photoelectric conversion based on photon counting; A generating step of generating data of a second image based on a random number derived based on the probability distribution; 13. An image processing method comprising:

13. A program for causing a computer to function as the image processing device according to any one of claims 1 to 11.

Images