Learning device, learning method, and program

Abstract

A learning device provided with: a learning data acquisition unit which acquires learning data including image data of an imaging target captured through a filter and filter status information indicating the status of the filter; and a learning unit which implements a mathematical model including both a fidelity process for solving an inverse problem and thereby generating tensors that are closest to tensors to be processed and that serve as solutions, and a regularization process for generating image data of an image having characteristics similar to statistical characteristics satisfied by an image of the imaging target. The number of tensors to be processed in the fidelity process is larger than the number of tensors to be processed in the regularization process; the size of each tensor to be processed in the fidelity process is smaller than the size of the tensors to be processed in the regularization process; the tensors to be processed in the regularization process are combinations of tensors generated in the fidelity process; and the fidelity process and the regularization process are executed alternately.

Description

[Technical field]

[0001] The present invention relates to a learning device, a learning method, and a program. [Background technology]

[0002] Even when there is little information about the image to be estimated, there are techniques that use the results of Bayesian estimation to generate the original image. One such technique is compressed sensing. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Wagadarikar, Ashwin A., et al. “Video Rate Spectral Imaging Using a Coded Aperture Snapshot Spectral Imager.” Optics Express, vol. 17, no. 8, 2009, pp. 6368-6388. [Non-Patent Document 2] Lu Gan, “BLOCK COMPRESSED SENSING OF NATURAL IMAGES”, Proc. of the 2007 15th Intl. Conf. on Digital Signal Processing (DSP 2007), 403-406. [Non-Patent Document 3] Zhang, Jian, and Bernard Ghanem. “ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing.” 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 1828-1837. [Non-Patent Document 4] Wang, Lizhi, et al. “Hyperspectral Image Reconstruction Using a Deep Spatial-Spectral Prior.” 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 8032-8041. Summary of the Invention [Problem to be solved by the invention]

[0004] However, such techniques sometimes increase the amount of calculations in proportion to the power of the input information (for example, Non-Patent Documents 3 and 4). In addition, when trying to suppress the increase in the amount of calculations, the accuracy of the generated image may be low. In other words, it is difficult to suppress the increase in the amount of calculations and to increase the accuracy of the generated image at the same time. This is a common issue not only for images but also for signals.

[0005] In view of the above circumstances, an object of the present invention is to provide a technique for achieving both an increase in the amount of calculation required for signal generation and an increase in the accuracy of signal generation. [Means for solving the problem]

[0006] One aspect of the present invention is a learning device comprising: a learning data acquisition unit that acquires learning data including image data of an image of a subject captured through a filter and filter state information indicating the state of the filter; a fidelity processing that is a process of generating, based on the learning data, a tensor whose solution is a tensor that is closest to a tensor to be processed by solving an inverse problem; and a regularization processing that is a process of generating, based on the learning data, image data of an image having properties close to statistical properties satisfied by the image of the subject, wherein the number of tensors to be processed by the fidelity processing is greater than the number of tensors to be processed by the regularization processing, the size of each tensor to be processed by the fidelity processing is smaller than the size of the tensor to be processed by the regularization processing, the tensor to be processed by the regularization processing is a tensor formed by combining the tensors generated by the fidelity processing, and the fidelity processing and the regularization processing are executed alternately.

[0007] One aspect of the present invention is a learning device that includes a learning data acquisition unit that acquires learning data including a signal obtained by imaging a subject through a filter and filter state information indicating the state of the filter, a fidelity process that is a process of generating a tensor based on the learning data by solving an inverse problem, the tensor having a solution that is a tensor that is closest to the tensor to be processed, and a regularization process that is a process of generating, based on the learning data, a signal having properties close to the statistical properties satisfied by the signal obtained by imaging the subject, wherein the number of tensors to be processed by the fidelity process is greater than the number of tensors to be processed by the regularization process, the size of each tensor to be processed by the fidelity process is smaller than the size of the tensor to be processed by the regularization process, the tensor to be processed by the regularization process is a tensor formed by combining the tensors generated by the fidelity process, and the fidelity process and the regularization process are executed alternately.

[0008] One aspect of the present invention is a learning method comprising: a learning data acquisition step of acquiring learning data including image data of an image of a subject photographed through a filter and filter state information indicating a state of the filter; a fidelity processing step of generating, based on the learning data, a tensor having a solution that is a tensor that is closest to a tensor to be processed by solving an inverse problem; and a regularization processing step of generating, based on the learning data, image data of an image having properties close to statistical properties satisfied by the image of the subject, wherein the number of tensors to be processed by the fidelity processing is greater than the number of tensors to be processed by the regularization processing, the size of each tensor to be processed by the fidelity processing is smaller than the size of the tensor to be processed by the regularization processing, the tensor to be processed by the regularization processing is a tensor formed by combining the tensors generated by the fidelity processing, and the fidelity processing and the regularization processing are executed alternately.

[0009] One aspect of the present invention is a program for causing a computer to function as the above-mentioned learning device. Effect of the Invention

[0010] According to the present invention, it is possible to achieve both an increase in the amount of calculation required for signal generation and an increase in the accuracy of signal generation. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of an image generating system according to an embodiment. [Diagram 2] FIG. 2 is an explanatory diagram for explaining a reconstructed neural network in the image generating system according to the embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a control device according to an embodiment. [Figure 4] FIG. 2 is a diagram showing an example of a functional configuration of a control unit in the embodiment. [Diagram 5] FIG. 4 is a diagram showing an example of a processing flow executed by the image generating system according to the embodiment. [Figure 6]FIG. 2 is a diagram showing an example of a hardware configuration of a learning device according to an embodiment. [Figure 7] FIG. 2 is a diagram showing an example of a functional configuration of a control unit in the embodiment. [Figure 8] 1 is a flowchart showing an example of a flow of a process executed by a learning device in an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (Embodiment) The following description will be given taking an image as an example, but the following description is not limited to images and is applicable to signals in general.

[0013] FIG. 1 is an explanatory diagram for explaining an overview of an image generation system 100 according to an embodiment. First, an overview of the image generation system 100 will be explained. The image generation system 100 is a system that generates image data of an image of a subject to be photographed. The image is, for example, a photograph. The image generation system 100 includes at least a control device 1, a photographing device 2, and a filter 3. The control device 1 controls the image generation system 100.

[0014] The imaging device 2 may be any sensor using one or more photodiodes. Examples of the sensor include any camera or X-ray camera. If the imaging device 2 is an X-ray camera, the image generating system 100 may be used for X-ray photography, for example. The imaging device 2 may be composed of one photodiode or a two-dimensional array of photodiodes. The imaging device 2 may include a photodiode for each primary color of RGB, for example.

[0015] The image capturing device 2 outputs information indicating the frequency and intensity of the incident electromagnetic waves (hereinafter referred to as "electromagnetic wave information"). If it is composed of a single photodiode, the electromagnetic wave information is frequency and intensity, and if it is composed of a two-dimensional array of photodiodes, the electromagnetic wave information is information indicating the spatial distribution of frequency and intensity. From here on, we will explain the case where the image capturing device 2 is composed of a single photodiode.

[0016] The filter 3 is a medium in which the spatial distribution of the optical constants changes due to a predetermined action such as application of a voltage, application of a magnetic field, application of heat, application of a load, etc. The filter 3 is, for example, a thin film formed of a photorefractive material. The filter 3 may be, for example, a photonic crystal whose dielectric constant or structure changes due to application of an action. The spatial distribution of the optical constants of the filter 3 is controlled by the control device 1. Note that the spatial distribution of the optical constants may be, for example, a spatial distribution of the positions of the apertures (i.e., the aperture pattern).

[0017] The image capturing device 2 captures an image of a subject through the filter 3. When the image capturing device 2 is composed of one photodiode, the image capturing device 2 converts an incident signal through the filter 3 into an electrical signal. Note that the process of converting an incident signal into an electrical signal is image capturing. Therefore, the content indicated by the electromagnetic wave information depends on the subject and the state of the filter 3. The state of the filter 3 is specifically the spatial distribution of the optical constants of the filter 3.

[0018] In the image generating system 100, the generation of electromagnetic wave information by the photographing device 2 and the filter state change process are repeatedly executed until a predetermined end condition is satisfied. The filter state change process is a process for changing the state of the filter 3 based on the filter state information and electromagnetic wave information obtained until the predetermined end condition is satisfied. The filter state information is information indicating the spatial distribution of the optical constants of the filter 3. Information obtained by solving the inverse problem based on the electromagnetic wave information and the filter state information obtained until the predetermined end condition (hereinafter referred to as the "filter state change end condition") is satisfied is image data (hereinafter referred to as the "final image data") that is output to a predetermined output destination such as a storage device. That is, the inverse problem is obtained using one or more pieces of electromagnetic wave information. The filter state change process is executed by the control device 1.

[0019] The filter state change ending condition is, for example, a condition that the filter state change process has been executed a predetermined number of times.

[0020] More specifically, the filter state change process is a process in which the state of the filter 3 is changed so as not to capture an image that satisfies a predetermined concealment condition. The concealment condition is a condition that is satisfied by an image of a target whose information leakage due to photography should be suppressed. Therefore, an image that satisfies the concealment condition is not captured in the image shown by the final image data (hereinafter referred to as the "final image"). An image that satisfies the concealment condition is, for example, a face. The filter state change process may be a process whose contents are determined by a machine learning method, or may be a process whose contents are determined in advance by a method other than machine learning.

[0021] The image generating system 100 generates image data for an image that meets the concealing condition so as not to obtain any more information from the time when it is determined that the image meets the concealing condition. In other words, the image generating system 100 generates image data for an image that meets the concealing condition so that the amount of information is less than that of an image that does not meet the concealing condition. That is, the image generating system 100 generates image data for an image that meets the concealing condition by controlling the optical system, with less information than that of an image that does not meet the concealing condition. The image generating system 100 does not process the obtained image to delete the image that meets the concealing condition from the image, and obtain image data with less information than that of an image that does not meet the concealing condition as a result of the deletion. Therefore, the image generating system 100 can suppress the leakage of information due to shooting.

[0022] <About filter state change processing> The filter state change process will be described. The filter state change process includes an image reconstruction process and an optimization process. The image reconstruction process is a process of solving an inverse problem based on electromagnetic wave information and filter state information. The image reconstruction process estimates image data of an image of a subject by solving an inverse problem. Hereinafter, image data estimated by the image reconstruction process is referred to as reconstructed image data. Hereinafter, an image indicated by the reconstructed image data is referred to as a reconstructed image. Note that in this embodiment, the image is reconstructed from a small number of samples by the image reconstruction process. The small number of samples refers to, for example, a portion of the pixels that constitute the image. By reconstructing an image from a small amount of information in this way, the image generating system 100 can estimate what is being photographed even from an amount of information that is insufficient to estimate the image being photographed.

[0023] The method for solving the inverse problem may be, for example, compressed sensing. The method for solving the inverse problem may be, for example, a convex optimization method such as a method for solving the inverse problem by using ADMM (alternating direction method of multiplier) with total variation minimization as a regularization term. The method for solving the inverse problem may be, for example, a method using a trained model obtained by using deep learning.

[0024] The optimization process is a process of changing the state of the filter 3 based on the reconstructed image data so as not to capture an image that satisfies a predetermined concealment condition. Therefore, the optimization process is a process of controlling the spatial distribution of the optical constants of the filter 3 based on the reconstructed image data. Therefore, the optimization process is a process of controlling the aperture pattern of the filter 3 based on, for example, the reconstructed image data.

[0025] <Optimization processing> The optimization process will be described in more detail. The optimization process includes a filter state information update process and an update information application process. The filter state information update process is a process for updating the filter state information based on a reconstructed image. The updated filter state information indicates the state of the filter 3 so as to satisfy an update condition. The update condition is a condition that the state of the filter 3 indicated by the updated filter state information obtained by the filter state information update process is a state that does not capture an image that satisfies a predetermined occlusion condition, compared to the state before the update.

[0026] The filter state information update process may use any method to update the filter state information as long as the method is capable of updating the filter state information so as to satisfy the update condition based on the reconstructed image.

[0027] The update information application process is a process for controlling the state of the filter 3 so that the state of the filter 3 is the state indicated by the post-update filter state information updated by the filter state information update process.

[0028] <First example of updating filter status information> The filter state information update process may update the filter state information by executing, for example, a random number update process, a detection process, and a mask process.

[0029] The random number update process is a process of updating the filter state information using random numbers of a Gaussian distribution. The random number update process updates the spatial distribution of the optical constants of the filter 3 indicated by the filter state information. The detection process is a process of detecting an image that satisfies the concealment condition based on a reconstructed image. The detection of an image that satisfies the concealment condition may be a process of detecting an image that satisfies the concealment condition by detecting a feature that the image that satisfies the concealment condition has, for example. When the image that satisfies the concealment condition is, for example, a face, the feature that the image that satisfies the concealment condition has is, for example, the eyes, nose, or mouth.

[0030] The mask process is a process for updating some of the optical constants at each position of the filter 3 indicated by the filter state information after update by the random number update process. The targets of update by the mask process are the optical constants at the positions of the filter 3 corresponding to the image detected by the detection process. Specifically, the corresponding positions are positions on the filter 3 through which electromagnetic waves coming from the subject and entering the imaging device 2 pass in an optical system formed by the subject, the filter 3, and the imaging device 2. The mask process changes the optical constants to be updated to values ​​that reduce the transmittance of the electromagnetic waves coming from the subject in the optical system formed by the subject, the filter 3, and the imaging device 2.

[0031] In this manner, the filter state information is updated so as to satisfy the update condition by executing the random number update process, the detection process, and the mask process.

[0032] <Second example of updating filter status information> The filter state information update process may be a process (hereinafter referred to as "machine learning process") that updates the filter state information based on reconstructed image data using a trained model obtained by performing training so as to satisfy the following filter loss condition by a machine learning method. For the training performed so as to satisfy the filter loss condition, a large amount of image data is prepared as simulated image data of a subject to be photographed. For the training, the image data of the subject to be photographed (hereinafter referred to as "first training data") and reconstructed image data (hereinafter referred to as "second training data") obtained through an image reconstruction process when the image data of the subject to be photographed is input are used.

[0033] The filter loss condition includes a first overall image condition and a local image difference increase condition. The first overall image condition is a condition that the difference between the image indicated by the first learning data and the image indicated by the second learning data is reduced. The local image difference increase condition is a condition that the difference between an image satisfying the concealment condition in the image of the subject to be photographed and an image satisfying the concealment condition in the reconstructed image data is increased. In this way, in the learning, a process is executed in which the entire image of the subject to be photographed is accurately acquired while detailed information is not acquired for the image satisfying the concealment condition in the image of the subject to be photographed. That is, in the learning, in order to prevent an image satisfying a predetermined concealment condition from being captured in the image of the subject to be photographed, the filter state information is updated so as to satisfy the condition that the difference between the image of the subject to be photographed and the reconstructed image is increased for the image satisfying the concealment condition. Note that the machine learning method is, for example, deep learning.

[0034] In this manner, the filter state information is updated so as to satisfy the update condition by executing the machine learning process.

[0035] <Adversarial Learning> Incidentally, as described above, the image reconstruction process may be a process using a trained model obtained by a machine learning method, and the filter state information update process may also be a trained model obtained by a machine learning method as described above. Thus, the acquisition of the trained model used in the image reconstruction process and the acquisition of the trained model used in the filter state information update process may be obtained independently of each other, but may not be independent of each other. Specifically, the trained model may be obtained by adversarial learning.

[0036] For ease of explanation, hereinafter, the mathematical model before a predetermined termination condition for the end of learning is satisfied and which estimates reconstructed image data based on electromagnetic wave information and filter state information is referred to as an image reconstruction model. The image reconstruction model at the point in time when a predetermined termination condition for the end of learning is satisfied is a trained image reconstruction model, and the trained image reconstruction model is a trained model used in image reconstruction processing.

[0037] For ease of explanation, the mathematical model before a predetermined termination condition regarding the end of learning is satisfied and which updates the filter state information based on the reconstructed image data is referred to as the filter state information update model. The filter state information update model at the time when the predetermined termination condition regarding the end of learning is satisfied is the trained filter state information update model, and the trained filter state information update model is the trained model used in the filter state information update process. Note that the above-mentioned machine learning process can also be said to be a process of executing the filter state information update model.

[0038] A method for obtaining a trained model used in the image reconstruction process and a trained model used in the filter state information update process using an image reconstruction model based on adversarial learning will be specifically described. In adversarial learning, the image reconstruction model and the filter state information update model are alternately updated according to a predetermined rule. The filter state information update model is updated so as to satisfy the above-mentioned filter loss condition. The image reconstruction model is updated so as to satisfy the reconstruction loss condition. The learning data used for updating (i.e., learning) the image reconstruction model includes electromagnetic wave information and filter state information.

[0039] The reconstruction loss condition includes a second overall image condition and a local image difference reduction condition. The second overall image condition is a condition that reduces the difference between the entire reconstructed image obtained based on the electromagnetic wave information and the filter state information (i.e., the entire image indicated by the second learning data) and the image indicated by the first learning data. The local image difference reduction condition is a condition that reduces the difference between an image that satisfies the occlusion condition in the image of the subject and an image that satisfies the occlusion condition in the reconstructed image data.

[0040] In this way, in the adversarial learning, the filter state information update model is trained to increase the difference between an image that satisfies the occlusion condition in the image obtained as a result of the processing and an image that satisfies the occlusion condition in the image of the photographed subject. On the other hand, the image reconstruction model is trained to decrease the difference between an image that satisfies the occlusion condition in the image obtained as a result of the processing and an image that satisfies the occlusion condition in the image of the photographed subject. Therefore, the learning of the filter state information update model and the learning of the image reconstruction model are adversarial learning.

[0041] <An example of an image reconstruction model> Here, we will explain a more specific example of an image reconstruction model. As mentioned above, a reconstructed image can be obtained by compressed sensing, and one method for obtaining a solution using compressed sensing is the convex optimization method. In other words, compressed sensing is reduced to a convex optimization problem. The convex optimization method is a method in which the mathematical expression to be solved is expressed as the sum of two terms, a faithfulness term and a regularization term, using auxiliary variables, and the faithfulness term and regularization term are optimized alternately. In other words, the convex optimization method is a method for solving the faithfulness term and the regularization term self-consistently.

[0042] The image reconstruction model is represented by, for example, a neural network that reflects this convex optimization method in machine learning. More specifically, the image reconstruction model is represented by a neural network that includes a plurality of high-fidelity neural networks and regularization neural networks. The high-fidelity neural network is a neural network that expresses the processing represented by the high-fidelity term. The regularization neural network is a neural network that expresses the processing represented by the regularization term. Hereinafter, the neural network that expresses the image reconstruction model is referred to as the image reconstruction neural network.

[0043] A neural network is a mathematical model whose processing content is updated through learning. A mathematical model is a set of one or more processes whose execution timing is predetermined. Therefore, executing a mathematical model means executing each process included in the mathematical model according to a predetermined rule.

[0044] In the learning of the image reconstruction model, each of the fidelity neural networks and each of the regularization neural networks included in the image reconstruction neural network is updated by learning. This means that the content of the process of self-consistently calculating the solution of the fidelity term and the regularization term, such as the manner of changing the parameters in the process of self-consistently calculating the solution of the fidelity term and the regularization term, is updated by learning.

[0045] As described above, the faithfulness term and the regularization term are mathematically one and the other of the two terms when the equation to be solved is divided into two terms using an auxiliary variable. However, the faithfulness term does not simply mean one of the two terms mathematically divided, but is a term defined so that the optimization process of the faithfulness term can be interpreted as a process of obtaining a value that minimizes the difference from a predetermined standard. The regularization term is a term defined so that the optimization process of the regularization term can be interpreted as a process of obtaining a value that minimizes the difference from a quantity indicated by a priori information.

[0046] In the image generating system 100, a faithful neural network is used instead of the faithful term, and a regularization neural network is used instead of the regularization term, but the faithful neural network and the regularization neural network are defined so that they can be physically interpreted. This will be explained below with reference to FIG.

[0047] 2 is an explanatory diagram for explaining the reconstructed neural network in the image generating system 100 according to the embodiment. The reconstructed neural network of the image generating system 100 alternately includes a fidelity neural network and a regularization neural network. That is, the fidelity process and the regularization process are alternately executed.

[0048] In the image generation system 100, image data represented by a tensor of first or higher order is input to the high fidelity neural network in a state in which the image data is divided into a plurality of tensors of smaller size. Hereinafter, the plurality of tensors of smaller size are referred to as block tensors. When the image data before division is expressed as a matrix, the blocks are minor matrices. The division process in FIG. 2 is a process of dividing the image data to be processed, which is represented by a tensor of first or higher order, into a plurality of block tensors. In the division process, information indicating the arrangement of the block tensors (hereinafter referred to as "block tensor arrangement information") is also generated.

[0049] In the fidelity neural network, a process (hereinafter referred to as "fidelity process") is executed to output a tensor (hereinafter referred to as "first interim optimization block tensor") that minimizes the difference between each input block tensor and a predetermined standard. Therefore, the fidelity neural network outputs the same number of first interim optimization block tensors as the input block tensors. In the fidelity process, filter state information is also used.

[0050] The faithful processing is a processing represented by the faithful neural network and a processing represented by the faithful term. The predetermined standard in the faithful processing is the input block tensor itself. In general, when the result inferred from the input information is used to infer the information on which the inference is based, the information on which the inference is based cannot necessarily be obtained.

[0051] In the fidelity process, the input block tensor is inferred using the result obtained from the input block tensor, and the inferred result is compared with the input block tensor. In the fidelity process, the image that minimizes the result of the comparison is output as the first interim optimization block tensor. In other words, in the fidelity process, a tensor whose solution is the tensor that is closest to the input block tensor when solving the inverse problem is output as the first interim optimization block tensor.

[0052] In the image generating system 100, a plurality of first interim optimization block tensors output from the preceding high-fidelity neural network are input in a combined state to the regularization neural network. The process indicated as a combining process in FIG. 2 is a process of combining a plurality of first interim optimization block tensors.

[0053] In the combining process, the optimized block tensors corresponding to each block tensor are combined in the arrangement of each block tensor indicated by the block tensor arrangement information. Hereinafter, the tensor generated by the combining process is called a combined tensor. Since the combined tensor is a combination of block tensors, the size of the combined tensor is larger than the size of each block tensor.

[0054] In the regularization neural network, a process (hereinafter referred to as "regularization process") is executed to output a tensor (hereinafter referred to as "second interim optimization block tensor") that minimizes the difference between the input combined tensor and the amount indicated by the predetermined prior information. The regularization process is the process represented by the regularization neural network and the process represented by the regularization term.

[0055] The quantity indicated by the predetermined a priori information is a criterion that is updated by learning. The quantity indicated by the a priori information is specifically a quantity that indicates the statistical properties of each pixel of the image of the subject to be photographed. The statistical properties are, for example, a property in which many coefficients become 0 when a discrete cosine transform is performed. The property in which many coefficients become 0 when a discrete cosine transform is performed is called sparsity. The smaller the difference between the image and the quantity indicated by the a priori information, the closer it is to the image of the subject to be photographed. In this way, the regularization process is a process of generating image data (i.e., the second provisional optimization block tensor) of an image having properties close to the statistical properties satisfied by the image of the subject to be photographed.

[0056] The following formula (1) is an example of a formula expressing the faithful processing, and formula (2) is an example of a formula expressing the regularization processing.

[0057]

number

[0058]

number

[0059] In equations (1) and (2), f is a vector indicating image data of the image of the subject. When the subscript k is used, f and h represent the provisional reconstructed image data of the image of the subject in the kth iterative calculation. In equations (1) and (2), u represents an auxiliary variable. k is the number of iterations of the self-consistent process. f is a vector indicating image data of the image of the subject. g represents electromagnetic wave information. Φ is a tensor indicating filter state information. ε represents the step size in the gradient method. η represents the weight of the penalty term. τ represents a weight parameter. I represents a unit matrix. R represents a regularization term. The function H is a function of equation (2) with f and u as input.

[0060] As described above, in the fidelity processing, the input block tensor is inferred using the result obtained from the input block tensor, and the inferred result is compared with the input block tensor. In the fidelity processing, as a result of the comparison, a tensor having a solution that is the tensor closest to the input block tensor is output as a first provisional optimization block tensor. Therefore, the fidelity processing executes a process of obtaining a result using filter state information based on the input block tensor, and a process of estimating the input block tensor by solving an inverse problem from the obtained result using the filter state information. More specifically, the filter state information used in solving the inverse problem is an inverse image of a mapping Φ representing the filter state information. Since the elements of the mapping representing the filter state information are real numbers, the inverse image is a matrix proportional to the transposed matrix of the mapping Φ. Therefore, the fidelity processing that executes a process of estimating the input tensor using the result obtained using the filter state information from the input tensor requires an amount of calculation that is a power of the size of the tensor representing the mapping Φ.

[0061] That is, in the fidelity processing, the provisional reconstructed image data (h (k) ) and the observed electromagnetic wave distribution information g, and converts the input provisional reconstructed image data of the image of the subject in the kth image so that it is consistent with the observed electromagnetic wave distribution information, and outputs it. When solving this problem using a gradient method with a step size of ε, a matrix product of the transposed matrix of the mapping Φ representing the filter state information and the mapping Φ representing the filter state information is required. Therefore, the faithfulness processing requires an amount of calculation that is a power of the size of the tensor representing the mapping Φ.

[0062] As the first term of formula (1) indicates, the fidelity process handles the matrix product of Φ and the transposed matrix of Φ, where Φ indicates filter state information. On the other hand, as the function H indicates, the regularization process executes a process of convolving K types of filters with the input tensor. In other words, unlike the fidelity process, the amount of calculations in the regularization process does not increase exponentially.

[0063] <Effects of division and combination processes> As mentioned above, the fidelity process is performed for each block tensor. In the operation for each block tensor, the size of Φ only needs to be the same as the size of the block tensor to be operated on. Because R (L2×L2) Φ belongs to T Φ and R (L2×1) The product of f belonging to R (l2×l2) belongs to T Φ' and R (l2×1)This is because Φ is divided into f' and f' belonging to f'. Φ is filter state information, and can have a structure that can be divided. Being divisible means that the results are the same even if the product of Φ and a block tensor is calculated after each division. If the division process is not performed, the faithful processing requires calculations of N to the fourth power according to the size N of the input tensor, but if p block tensors of size M are generated by the division process, the amount of calculations required is p times the fourth power of M. The input tensor is the tensor to be processed. Non-Patent Document 1 and Non-Patent Document 2 are examples of Φ having a structure that can be divided.

[0064] As described above, in the regularization process, a combining process is executed. In the regularization process, it is important to generate an image in which the entire image is close to the entire image of the subject. Therefore, not only the properties of each block tensor but also the information on the arrangement between the block tensors is important. Therefore, unlike the faithful processing, the regularization process uses the tensor obtained in the combining process. Also, as described above, in the regularization process, unlike the faithful processing, the amount of calculation does not increase exponentially. Therefore, it is better to use the tensor obtained in the combining process rather than using the block tensor.

[0065] In this way, the image reconstruction model executes a fidelity process, which is a process of performing an operation proportional to the power of the tensor to be processed and generates, based on the learning data, a tensor whose solution is a tensor closest to the tensor to be processed when solving an inverse problem.Furthermore, the image reconstruction model executes a regularization process, which is a process of performing an operation proportional to the tensor to be processed and generates, based on the learning data, image data of an image having properties close to the statistical properties satisfied by the image of the captured object.

[0066] Furthermore, a combination of block tensors being a combined tensor means that the number of tensors to be processed in the fidelity process is greater than the number of tensors to be processed in the regularization process, and the size of each tensor to be processed in the fidelity process is smaller than the size of the tensor to be processed in the regularization process.

[0067] In addition, in the learning of the image reconstruction model, the regularization neural network is updated so that the generated tensor is closer to the tensor to be processed when solving the inverse problem. In addition, in the learning of the image reconstruction model, the regularization neural network is updated so that the generated tensor generates image data of an image having properties closer to the statistical properties satisfied by the image of the photographed object.

[0068] 3 is a diagram showing an example of a hardware configuration of the control device 1 in the embodiment. The control device 1 includes a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) and a memory 92 connected by a bus, and executes a program. The control device 1 functions as a device including the control unit 11, an input unit 12, a communication unit 13, a storage unit 14, an output unit 15, and a filter control circuit 16 by executing the program.

[0069] More specifically, the processor 91 reads out a program stored in the storage unit 14, and stores the read out program in the memory 92. The processor 91 executes the program stored in the memory 92, whereby the control device 1 functions as a device including a control unit 11, an input unit 12, a communication unit 13, a storage unit 14, an output unit 15, and a filter control circuit 16.

[0070] The control unit 11 controls the operation of various functional units included in the control device 1. The control unit 11 executes, for example, a filter state change process. The control unit 11 controls the operation of, for example, the filter control circuit 16, thereby controlling the state of the filter 3. The control unit 11 records, for example, various pieces of information generated by the execution of the filter state change process in the storage unit 14.

[0071] The input unit 12 includes input devices such as a mouse, a keyboard, a touch panel, etc. The input unit 12 may be configured as an interface that connects these input devices to the control device 1. The input unit 12 accepts input of various information to the control device 1.

[0072] The communication unit 13 includes a communication interface for connecting the control device 1 to an external device. The communication unit 13 communicates with the external device via wired or wireless communication. The external device is, for example, the photographing device 2. The communication unit 13 acquires electromagnetic wave information by communicating with the photographing device 2.

[0073] The storage unit 14 is configured using a non-transitory computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 14 stores various information related to the control device 1. The storage unit 14 stores information inputted via the input unit 12 or the communication unit 13, for example. The storage unit 14 stores various information generated by, for example, execution of a filter state change process. The storage unit 14 stores, for example, filter state information.

[0074] The output unit 15 outputs various information. The output unit 15 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The output unit 15 may be configured as an interface that connects these display devices to the control device 1. The output unit 15 outputs, for example, information input to the input unit 12. The output unit 15 outputs, for example, final image data.

[0075] The filter control circuit 16 is a circuit that applies an action to the filter 3 to change the state of the filter 3. The filter control circuit 16 is a circuit that applies a voltage to the filter 3, for example.

[0076] 4 is a diagram showing an example of a functional configuration of the control unit 11 in the embodiment. The control unit 11 includes an electromagnetic wave information acquisition unit 111, a filter state control unit 112, and a storage control unit 113. The electromagnetic wave information acquisition unit 111 acquires electromagnetic wave information generated by the imaging device 2. The filter state control unit 112 executes a filter state change process and an end determination process.

[0077] The end determination process is a process for determining whether or not a filter state change end condition is satisfied. The memory control unit 113 records various information in the memory unit 14. For example, every time the filter state control unit 112 changes the state of the filter 3, the memory control unit 113 records filter state information indicating the state of the filter 3 resulting from the control in the memory unit 14.

[0078] 5 is a diagram showing an example of a flow of processing executed by the image generating system 100 of the embodiment. The imaging device 2 generates electromagnetic wave information (step S101). Next, the electromagnetic wave information acquisition unit 111 acquires the electromagnetic wave information generated in step S101 via the communication unit 13 (step S102). Next, the filter state control unit 112 executes image reconstruction processing (step S103). Reconstructed image data is generated by executing the reconstruction image processing.

[0079] The filter state control unit 112 executes an end determination process (step S104). If the filter state change end condition is satisfied (step S104: YES), the process ends. The reconstructed image data at the time the process ends is the final image data.

[0080] On the other hand, if the filter state change end condition is not satisfied (step S104: NO), the filter state control unit 112 executes a filter state information update process (step S105). More specifically, the filter state control unit 112 updates the filter state information based on the filter state information stored in the storage unit 14 and the reconstructed image data acquired in step S103.

[0081] Next, the filter state control unit 112 executes an update information application process (step S106). By executing the update information application process, the filter state control unit 112 controls the operation of the filter control circuit 16 to control the state of the filter 3 so that the state of the filter 3 is the state indicated by the filter state information after update by the filter state information update process. Next, the process returns to step S101.

[0082] The image reconstruction model is generated by, for example, a learning device 4 shown in Fig. 6 below. Fig. 6 is a diagram showing an example of a hardware configuration of the learning device 4 in an embodiment. The learning device 4 includes a control unit 41 including a processor 93 such as a CPU and a memory 94 connected by a bus, and executes a program. The learning device 4 functions as a device including the control unit 41, an input unit 42, a communication unit 43, a storage unit 44, and an output unit 45 by executing the program.

[0083] More specifically, the processor 93 reads out the program stored in the storage unit 44 and stores the read out program in the memory 94. The processor 93 executes the program stored in the memory 94, whereby the learning device 4 functions as a device including a control unit 41, an input unit 42, a communication unit 43, a storage unit 44, and an output unit 45.

[0084] The control unit 41 controls the operation of various functional units included in the learning device 4. The control unit 41 executes, for example, an image reconstruction model learning process. The image reconstruction model learning process is a process of updating the image reconstruction model based on the electromagnetic wave information and the filter state information until a predetermined termination condition (hereinafter referred to as a "learning termination condition") is satisfied.

[0085] The learning end condition is, for example, a condition that learning has been performed a predetermined number of times. The learning end condition may be, for example, a condition that a change in the image reconstruction model due to learning is smaller than a predetermined change. The image reconstruction model at the time when the learning end condition is satisfied is a trained image reconstruction model.

[0086] The control unit 41 records, for example, various pieces of information generated by the execution of the image reconstruction model learning process in the storage unit 44.

[0087] The input unit 42 includes input devices such as a mouse, a keyboard, and a touch panel. The input unit 42 may be configured as an interface that connects these input devices to the learning device 4. The input unit 42 accepts input of various information to the learning device 4. For example, a pair of electromagnetic wave information and filter state information is input to the input unit 42 as learning data used in the image reconstruction model learning process.

[0088] The communication unit 43 includes a communication interface for connecting the learning device 4 to an external device. The communication unit 43 communicates with the external device via a wired or wireless connection.

[0089] The storage unit 44 is configured using a non-transitory computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 44 stores various information related to the learning device 4. The storage unit 44 stores information inputted via the input unit 42 or the communication unit 43, for example. The storage unit 44 stores various information generated by, for example, execution of an image reconstruction model learning process. The storage unit 44 stores, for example, pairs of electromagnetic wave information and filter state information as learning data. The storage unit 44 stores in advance an image reconstruction model to be updated.

[0090] The output unit 45 outputs various information. The output unit 45 includes a display device such as a CRT display, a liquid crystal display, or an organic EL display. The output unit 45 may be configured as an interface that connects these display devices to the learning device 4. The output unit 45 outputs information input to the input unit 42, for example.

[0091] 7 is a diagram showing an example of a functional configuration of the control unit 41 in the embodiment. The control unit 41 includes a learning data acquisition unit 411, a learning unit 412, and a memory control unit 413. The learning data acquisition unit 411 acquires pairs of electromagnetic wave information and filter state information as learning data.

[0092] The learning unit 412 executes an image reconstruction model learning process. By executing the image reconstruction model learning process, the learning unit 412 learns the image reconstruction model based on the learning data acquired by the learning data acquisition unit 411. During learning, the image reconstruction model to be updated is executed, and the image reconstruction model is updated based on the result of the execution of the image reconstruction model. Since the image reconstruction model includes a fidelity process and a regularization process, the execution of the image reconstruction model also means the execution of the fidelity process and the regularization process.

[0093] The memory control unit 413 records various information in the memory unit 44.

[0094] 8 is a flowchart showing an example of the flow of processing executed by the learning device 4 in the embodiment. The learning data acquisition unit 411 acquires pairs of electromagnetic wave information and filter state information as learning data (step S201). Next, the learning unit 412 updates the image reconstruction model based on the learning data acquired in step S201 (step S202). Next, the learning unit 412 determines whether or not a learning end condition is satisfied (step S203). If the learning end condition is satisfied (step S203: YES), the processing ends. On the other hand, if the learning end condition is not satisfied (step S203: NO), the processing returns to step S201.

[0095] The control device 1 in the embodiment configured in this manner generates image data of an image that does not capture an image that satisfies the concealment condition by controlling the filter 3. That is, the control device 1 generates image data of an image that does not capture an image that satisfies the concealment condition by controlling the optical system. The control device 1 does not process the obtained image to delete the image that satisfies the concealment condition from the image, and obtains an image that does not capture an image that satisfies the concealment condition as a result of the deletion. Therefore, the control device 1 can suppress the leakage of information due to shooting.

[0096] The learning device 4 in the embodiment configured as described above executes a fidelity process and a regularization process. The fidelity process is a process of performing an operation proportional to the power of an input tensor, and is a process of generating a tensor that has as its solution a tensor that is closest to the input tensor when solving an inverse problem.

[0097] The regularization process is a process of performing a calculation proportional to the input tensor and generating image data of an image having a property close to the statistical property satisfied by the image of the subject. The processing target of the fidelity process is a block tensor, and the processing target of the regularization process is a combined tensor. Therefore, the learning device 4 can achieve both an increase in the amount of calculation required for image generation and an increase in the accuracy of image generation.

[0098] (Modification) The image reconstruction model learning process may be executed by the control device 1. That is, the control unit 11 may include a learning data acquisition unit 411 and a learning unit 412.

[0099] In addition, the faithful neural network and the regularized neural network may be the neural network described in Reference 1 below, except for the difference in the input data.

[0100] Reference 1: Yoko Sogabe, Shiori Sugimoto, Takayuki Kurozumi, and Hideaki Kimata “ADMM-INSPIRED RECONSTRUCTION NETWORK FOR COMPRESSIVE SPECTRAL IMAGING” ICIP 2020, 2865-2869

[0101] The control device 1 may be implemented using a plurality of information processing devices communicably connected via a network. In this case, each functional unit of the control device 1 may be distributed and implemented in the plurality of information processing devices.

[0102] The learning device 4 may be implemented using a plurality of information processing devices communicably connected via a network. In this case, each functional unit included in the learning device 4 may be distributed and implemented in the plurality of information processing devices.

[0103] The control device 1 is an example of an image generating device. The electromagnetic wave information is an example of image data of an image of a subject captured through the filter 3. When the processing target of the control device 1 and the learning device 4 is a signal instead of an image, the control device 1 and the learning device 4 use a signal obtained by capturing the subject through the filter 3 instead of the electromagnetic wave information. In such a case, the regularization process generates a signal having a property close to the statistical property satisfied by the signal obtained by capturing the subject instead of image data of an image having a property close to the statistical property satisfied by the image of the subject. The filter 3 is an example of an acquisition unit. The imaging device 2 is an example of a conversion unit. The signal that has passed through the filter 3, that is, the signal that is incident on the imaging device 2, is an example of an observation signal. The electrical signal output from the photodiode is an example of a partial image signal. An image that satisfies a predetermined occlusion condition is an example of a subject belonging to a predetermined attribute. The image of the subject is an example of a region that constitutes an image.

[0104] All or part of the functions of the control device 1 and the learning device 4 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The program may be transmitted via a telecommunications line.

[0105] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0106] 100...image generating system, 1...control device, 2...imaging device, 3...filter, 4...learning device, 11...control unit, 12...input unit, 13...communication unit, 14...memory unit, 15...output unit, 16...filter control circuit, 111...electromagnetic wave information acquisition unit, 112...filter state control unit, 113...memory control unit, 41...control unit, 42...input unit, 43...communication unit, 44...memory unit, 45...output unit, 411...learning data acquisition unit, 412...learning unit, 413...memory control unit, 91...processor, 92...memory, 93...processor, 94...memory

Claims

1. a learning data acquisition unit that acquires learning data including image data of an image of a subject captured through a filter and filter state information indicating a state of the filter; a learning unit that executes an image reconstruction model, which is a mathematical model including: a faithful processing that is a processing of generating a tensor based on the learning data, the tensor having a solution that is the closest to the tensor to be processed by solving an inverse problem; and a regularization processing that is a processing of generating image data of an image having properties close to the statistical properties satisfied by the image of the photographed object based on the learning data; Equipped with the number of tensors to be processed in the fidelity processing is greater than the number of tensors to be processed in the regularization processing, the size of each tensor to be processed in the fidelity processing is smaller than the size of the tensor to be processed in the regularization processing, and the tensor to be processed in the regularization processing is a tensor formed by combining the tensors generated in the fidelity processing, The faithfulness process and the regularization process are performed alternately. Learning device.

2. The state of the filter is the spatial distribution of the optical constants of the filter. The learning device according to claim 1 .

3. A learning data acquisition unit that acquires learning data including a signal obtained by capturing an image of a subject through a filter and filter state information indicating a state of the filter; a learning unit that executes an image reconstruction model, which is a mathematical model including: a faithful processing that is a processing of generating a tensor based on the learning data by solving an inverse problem, the tensor having a solution that is closest to the tensor to be processed; and a regularization processing that is a processing of generating a signal having properties close to the statistical properties satisfied by a signal obtained by imaging the imaging target based on the learning data; Equipped with the number of tensors to be processed in the fidelity processing is greater than the number of tensors to be processed in the regularization processing, the size of each tensor to be processed in the fidelity processing is smaller than the size of the tensor to be processed in the regularization processing, and the tensor to be processed in the regularization processing is a tensor formed by combining the tensors generated in the fidelity processing, The faithfulness process and the regularization process are performed alternately. Learning device.

4. a learning data acquisition step of acquiring learning data including image data of an image of a subject photographed through a filter and filter state information indicating a state of the filter; A learning step of executing an image reconstruction model, which is a mathematical model including: a faithful processing, which is a processing of generating a tensor based on the learning data, the tensor having a solution that is the closest to the tensor to be processed by solving an inverse problem; and a regularization processing, which is a processing of generating image data of an image having properties close to the statistical properties satisfied by the image of the photographed object based on the learning data; having the number of tensors to be processed in the fidelity processing is greater than the number of tensors to be processed in the regularization processing, the size of each tensor to be processed in the fidelity processing is smaller than the size of the tensor to be processed in the regularization processing, and the tensor to be processed in the regularization processing is a tensor formed by combining the tensors generated in the fidelity processing, The faithfulness process and the regularization process are performed alternately. How to learn.

5. A program for causing a computer to function as the learning device according to any one of claims 1 to 3.

Images