Information processing device, information processing method, and program

Abstract

To more quickly perform information processing using a machine learning model.SOLUTION: An information processing apparatus performs inference processing using a machine learning model having a first processing layer and a second processing layer, and stores an intermediate output from the first processing layer. The first processing layer receives a first input and outputs a first intermediate output corresponding to the first input. The second processing layer receives the first intermediate output and a second intermediate output from the first processing layer corresponding to a second input previous to the first input, outputs a result of the inference, and stores the first intermediate output.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, machine learning has been applied to various information processing application programs. Examples of information processing that uses machine learning include noise reduction processing that restores degraded images, super-resolution processing that increases image resolution and enhances high-frequency components to improve visibility, and Debayer processing that develops captured RAW images.

[0003] There are machine learning models with a multi-stage network structure. Patent Document 1 and Non-Patent Document 1 disclose a method for removing noise from a video using a two-stage model consisting of a first noise removal unit and a second noise removal unit. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 1,151,694 [Non-patent literature]

[0005] [Non-Patent Document 1] “FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation”,Matias, Tassano; Julie, Delon; Thomas, Veit, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 1354-1363 Summary of the Invention [Problem to be solved by the invention]

[0006] In Patent Document 1 and Non-Patent Document 1, a time-series of multiple frames of a video is input and noise removal is performed. At this time, a first noise removal process is performed on a part of the input multiple frames as a set, and the results are input and a second noise removal process is performed on the input. For example, for five input frames, the first noise removal process is performed three times on a total of three sets of three frames, and the resulting three intermediate outputs are input and a second noise removal process is performed. However, in the technology described in Patent Document 1 and Non-Patent Document 1, if the same process is performed on the next five input frames that are input sequentially, the same first noise process as the previous one will be performed on two sets of input frames.

[0007] An object of the present invention is to perform information processing using a machine learning model at higher speed. [Means for solving the problem]

[0008] In order to achieve the object of the present invention, for example, an information processing device according to one embodiment has the following configuration: That is, the information processing device includes a processing means for performing inference processing using a machine learning model having a first processing layer and a second processing layer, and a storage means for storing an intermediate output by the first processing layer, in which a first input is input to the first processing layer and a first intermediate output corresponding to the first input is output, and the second processing layer stores the first intermediate output and a second intermediate output from the first processing layer corresponding to a second input earlier than the first input. and a third intermediate output from the first processing layer corresponding to a third input that is earlier than the second input. is input, and an inference result is output, and the storage means stores the first intermediate output the second intermediate output; and the third intermediate output. Store A first interval between the first input and the second input inputted to the first processing layer is different from a second interval between the second input and the third input inputted to the first processing layer. . Effect of the Invention

[0009] Process information faster using machine learning models. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an information processing system according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a functional configuration of the information processing device according to the first embodiment. [Diagram 3] FIG. 2 is a diagram for explaining a learning process in the information processing device according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of the configuration of a model used by the information processing device according to the first embodiment. [Diagram 5] FIG. 4 is a diagram for explaining the degradation adding process according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of a network structure of a model according to the first embodiment. [Figure 7A] 5 is a flowchart showing an example of a learning process according to the first embodiment. [Figure 7B] 5 is a flowchart showing an example of a restoration process according to the first embodiment. [Figure 7C] 5 is a flowchart showing an example of a model switching process according to the first embodiment. [Figure 8A] FIG. 11 is a block diagram showing an example of input / output in a model according to a second embodiment. [Figure 8B] FIG. 11 is a block diagram showing an example of input / output in a model according to a second embodiment. [Figure 8C] FIG. 11 is a block diagram showing an example of input / output in a model according to a second embodiment. [Figure 9] FIG. 13 is a diagram showing an example of a network structure of a model for performing preprocessing according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] [Embodiment 1] The information processing device of this embodiment, in an inference process using a machine learning model having a first processing layer and a second processing layer, outputs a first intermediate output from a first input to the first processing layer, and inputs the first intermediate output and a second intermediate output from the first processing layer corresponding to a second input earlier than the first input to the second processing layer, thereby outputting an inference result.

[0013] [About CNN] First, a description will be given of a Convolutional Neural Network (CNN) used in information processing technology generally using deep learning, which is used in the following embodiments. CNN is a mathematical model that repeats convolution of a filter generated by training with image data and then nonlinear operation. In CNN, data obtained by convolving input image data with a filter and then nonlinear operation is called a feature map. In addition, CNN training is performed using training data (training images or data sets) consisting of pairs of input image data and output image data. In other words, generating filter values ​​that can convert input image data to corresponding output image data with high accuracy from training data is called learning.

[0014] When image data has RGB color channels or when a feature map is composed of multiple pieces of image data, the filter used for convolution also has multiple channels accordingly. In CNN, the process of performing nonlinear calculations after convolving a filter with image data (or feature map) is expressed in units of layers, such as the nth layer feature map and the nth layer filter. For example, a CNN with a three-layer network structure repeats the filter convolution and nonlinear calculation three times. This type of nonlinear calculation process can be formulated as the following equation (1).

number

[0015] In formula (1), W n is the nth layer filter, b n is the bias of the nth layer, f is a nonlinear operator, and X n represents the n-th layer feature map, and * represents the convolution operator. Note that (l) indicates that this is the l-th filter or feature map. This filter and bias are generated by learning, and are collectively called "network parameters." For example, a sigmoid function or ReLU (Rectified Linear Unit) is used as the nonlinear operation. When ReLU is used, the nonlinear operation process in CNN is given by, for example, the following formula (2).

number

[0016] As shown in equation (2), negative elements of the input vector X will be set to zero, and elements greater than or equal to 0 will remain unchanged.

[0017] Examples of networks using CNN include Residual Network (ResNet) in the field of image recognition and RED-Net, which is an application of ResNet in the field of super-resolution. Both of these enable high-precision recognition by making CNN multi-layered and performing filter convolutions multiple times. For example, ResNet features a network structure with paths that shortcut the convolutional layer, which allows for a multi-layer network of 152 layers, achieving high-precision recognition approaching the recognition rate of humans. Multi-layer CNNs achieve high-precision recognition by expressing nonlinear relationships between input and output by repeating nonlinear calculations multiple times.

[0018] [CNN learning] Next, we will explain CNN learning. CNN learning is performed by minimizing the objective function expressed by, for example, the following formula (3) for learning data consisting of a set of input image data and corresponding correct output image (teacher image) data.

number

[0019] In equation (3), L is a loss function that measures the error between the correct answer and the inference result. Y i is the i-th correct output image data, X i is the i-th input image data. F is a function that collectively represents the calculations (equation (1)) performed in each layer of CNN. θ is the network parameters (filter and bias). ||Z||2 is the L2 norm, that is, the square root of the sum of the squares of the elements of vector Z. n is the number of data sets in the training data. Since the amount of training data is generally large, in the Stochastic Gradient Descent (SGD) method, a portion of the training data is randomly selected and used for training. In addition, various methods are known for minimizing (optimizing) the objective function, such as the momentum method, AdaGrad method, AdaDelta method, and Adam method. The Adam method is given by the following equation (4).

number

[0020] In formula (4), t represents the value of the tth iteration. θ i t is the i-th network parameter at iteration t, and g is θ i tis the gradient of the loss function L with respect to m. m and v are moment vectors, α is the base learning rate, β1 and β2 are hyperparameters, and ε is a small constant. Note that the optimization method in learning is not limited to this, and any known optimization technique can be applied. It is known that the convergence of each method differs, resulting in differences in learning time, and it is possible to adopt an optimization method according to desired conditions.

[0021] In this embodiment, information processing (image processing) is performed to restore a degraded image using the above-mentioned CNN. Examples of image degradation factors in this degraded image include degradation such as noise, blur, aberration, compression, low resolution, and loss, and degradation such as contrast reduction due to weather conditions such as fog, haze, snow, and rain at the time of shooting. Examples of image processing to restore a degraded image include noise removal, blur removal, aberration correction, correction of degradation due to compression, super-resolution processing for low-resolution images, loss completion, and processing to correct contrast reduction due to weather conditions at the time of shooting. The degraded image restoration processing according to this embodiment is a process of performing restoration by generating an image with no degradation (or very little degradation) from an image with degradation, and will be referred to as image restoration (processing) in the following description. That is, image restoration according to this embodiment includes not only a case where an image with no degradation (or little degradation) itself is restored but also a case where degradation caused by subsequent amplification, compression / expansion, or other image processing is restored, but also a case where degradation contained in the original image itself can be reduced.

[0022] Here, when using an image restoration process using a neural network, it is generally expected that the expressive power of the network will be improved and image restoration with higher accuracy will be performed when a neural network with a larger number of parameters is used. On the other hand, a larger number of parameters leads to an increase in the number of calculations, which increases the inference time and the amount of memory required for inference. Therefore, the model to be used in the image restoration process differs depending on the amount of calculation resources and the usage state of the information processing device that performs the inference.

[0023] [Embodiment 1] In the first embodiment, an information processing device is described that stores intermediate output by a first processing layer in a machine processing model that performs image restoration processing, and performs inference by inputting the current intermediate output and the previous intermediate output to a second processing layer. In the following embodiment, noise is taken as an example of an image degradation element, and a process of performing noise reduction processing as image restoration processing is described.

[0024] [Example of information processing system configuration] FIG. 1 is an example of an information processing system configuration including an information processing device according to the first embodiment. In the information processing system shown in FIG. 1, a cloud server 200 that generates learning data and learns restoration of image quality degradation, and an information processing device 100 (edge ​​device 100) that performs image restoration for a processing target image are communicably connected via the Internet or the like. Hereinafter, the generation of learning data and the learning of restoration of image quality degradation in the cloud server 200 will be referred to as restoration learning, and the restoration of a degraded image in the information processing device 100 will be referred to as restoration inference. Note that, in this embodiment, the cloud server 200 and the information processing device 100 will be described as separate devices, but the restoration learning process described as being performed by the cloud server 200 may be performed by the information processing device 100.

[0025] [Hardware configuration of information processing device] The information processing device 100 acquires an image to be processed, and sets the acquired image as an input image to a machine learning model that performs restoration inference. In this embodiment, the information processing device 100 may acquire RAW image data (Bayer array) input from the imaging device 10 as an input image to be subjected to image restoration processing. This image is not particularly limited in terms of image format such as color space, and any image format can be used. For example, this image may be image data of other color filter arrays, a demosaiced RGB image, or an image after YUV conversion.

[0026] The information processing device 100 performs restoration processing of a degraded image by a machine learning model using an input image to be processed as an input. The information processing device 100 according to the present embodiment can perform restoration inference using a trained neural network model provided by the cloud server 200. That is, the information processing device 100 can be an information processing device that reduces noise in RAW image data by using the model provided by the cloud server 200 and executing a pre-installed information processing application program.

[0027] The information processing device 100 has a CPU 101, a RAM 102, a ROM 103, a large-capacity storage device 104, a general-purpose interface (I / F) 105, and a network I / F 106, and each component is connected to each other via a system bus 107. The information processing device 100 is also connected to an imaging device 10, an input device 20, an external storage device 30, and a display device 40 via the general-purpose I / F 105.

[0028] The CPU 101 uses the RAM 102 as a work memory, executes a program stored in the ROM 103, and generally controls each component of the information processing device 100 via a system bus 107. The mass storage device 104 is, for example, an HDD or an SSD, and stores various data and image data handled by the information processing device 100. The CPU 101 writes data to the mass storage device 104 via the system bus 107, and reads data stored in the mass storage device 104. The general-purpose I / F 105 is, for example, a serial bus interface such as USB, IEEE1394, or HDMI (registered trademark). The information processing device 100 acquires data from an external storage device 30 (for example, various storage media such as a memory card, a CF card, an SD card, or a USB memory) via the general-purpose I / F 105.

[0029] The information processing device 100 receives user instructions from an input device 20 such as a mouse or a keyboard, or a mobile terminal such as a smartphone, via a general-purpose I / F 105. The information processing device 100 also outputs image data processed by the CPU 101 to a display device 40 (for example, various image display devices such as a liquid crystal display) via the general-purpose I / F 105. The display device 40 can also function as the input device 20 as a display device integrated with a touch panel. The information processing device 100 also acquires data of a captured image (RAW image) to be subjected to noise reduction processing from the imaging device 10 via the general-purpose I / F 105. The network I / F 106 is an interface for connecting to the Internet. The information processing device 100 can acquire a trained model for restoration inference by accessing a cloud server 200 via an installed web browser.

[0030] [Cloud server hardware configuration] The cloud server 200 provides the information processing device 100 with a machine learning model that performs restoration processing on an input degraded image. The cloud server 200 according to this embodiment is a server that provides cloud services on the Internet. More specifically, the cloud server 200 generates learning data and performs restoration learning, and generates a trained machine learning model (trained model) that stores network parameters and a network structure of the learning result. Then, the cloud server 200 provides the trained model in response to a request from the information processing device 100. Hereinafter, when simply referred to as a "model", this refers to the trained model.

[0031] The cloud server 200 has a CPU 201, a ROM 202, a RAM 203, a mass storage device 204, and a network I / F 205, and each component is connected to each other by a system bus 206. The CPU 201 controls the overall operation by reading out a control program stored in the ROM 202 and executing various processes. The RAM 203 is used as a temporary storage area such as a main memory or work area of ​​the CPU 201. The mass storage device 204 is a large-capacity secondary storage device such as an HDD or SSD that stores image data or various programs. The network I / F 205 is an interface for connecting to the Internet, and provides a trained model that stores the above-mentioned network parameters and network structure in response to a request from a web browser of the information processing device 100.

[0032] In this embodiment, it is assumed that the information processing device 100 downloads a trained model, which is a result of performing training data generation and restoration learning, from the cloud server 200, and performs restoration inference on input image data to be processed. The components of the information processing device 100 and the cloud server 200 that perform such processing may be realized by a configuration different from the configuration described above. For example, the functions of the cloud server 200 may be subdivided, and the generation of training data and the restoration learning may be performed by separate devices. Alternatively, the imaging device 10 that has both the functions of the information processing device 100 and the cloud server 200 may be configured to perform all of the generation of training data, restoration learning, and restoration inference.

[0033] [System-wide functional blocks] Next, the functional configuration of the entire information processing system according to this embodiment will be described with reference to Fig. 2. As shown in Fig. 2, the information processing device 100 has an inference unit 112 and a model storage unit 119. The model storage unit 119 acquires and stores a trained model 220 from a cloud server 200. The inference unit 112 has a function of image restoration processing for restoring a degraded image. The inference unit 112 includes an image restoration unit 115 and a storage unit 113 for inference.

[0034] The image restoration unit 115 inputs a degraded image to the trained model 220 and restores the image. Here, the processing performed by the image restoration unit 115 will be described with reference to FIG. 3(A). The image restoration unit 115 acquires the input image data 116 and executes pre-processing 301. The pre-processing 301 is a processing performed before inputting the input image data 116 to the trained model 220, and is, for example, a processing of converting each data to a floating point number and normalizing the value to 0.0 to 1.0 when the input image data is a 14-bit integer value of a Bayer array. Next, the image restoration unit 115 executes model 302 using the trained model 220 with the data after pre-processing 301 as input. The image restoration unit 115 executes post-processing 303 on the result of model execution 302 to obtain the output image 118. The post-processing 303 is a processing performed on the output of the trained model 220, and is, for example, a processing of converting the value to a 14-bit integer value when the output result data of the model is a floating point number.

[0035] FIG. 4 shows an example of the structure of the model. In this example, the model is composed of a plurality of convolution layers 401 and a concatenation layer 402. In the convolution layer 401, the convolution operation by the filter represented by the above-mentioned formula (1) and the nonlinear operation represented by formula (2) are repeated a plurality of times. The image restoration unit 115 sequentially applies the convolution layers 401 to the input data of the model to calculate a feature map. Next, the image restoration unit 115 connects the feature map and the input data in the channel direction by the concatenation layer 402. Furthermore, the image restoration unit 115 sequentially applies the convolution layers 401 to the concatenation result and outputs the result from the final layer. The process shown in FIG. 4 is a process generally performed by CNN, and a detailed description will be omitted. The image restoration unit 115 performs restoration inference of the input image data 116 using the trained model 220 received from the cloud server 200.

[0036] The trained model 220 according to this embodiment has a first processing layer and a second processing layer, and outputs a model inference result from the second processing layer. The storage unit 113 stores intermediate outputs corresponding to inputs of the first processing layer of the trained model 220. Here, the intermediate output from the first processing layer corresponding to the nth input (n is a natural number) input to the first processing layer is referred to as the nth intermediate output.

[0037] As described above, the image restoration unit 115 outputs a first intermediate output for a first input to the first processing layer. Then, the image restoration unit 115 inputs the first intermediate output and a second intermediate output corresponding to a second input that is earlier than the first input to the second processing layer, and outputs an inference result. Examples of the structures of the first processing layer and the second processing layer and the processes performed in these processing layers will be described later with reference to FIG. 6.

[0038] The cloud server 200 includes a degradation adding unit 211 and a learning unit 212. The degradation adding unit 211 generates a degraded learning image for input to be used for learning restoration inference. The degradation adding unit 211 can generate a degraded learning image by adding at least one type of degradation element to teacher image data extracted from a group of teacher images without degradation, for example. In the example of this embodiment, noise is used as the degradation element, and the degradation adding unit 211 adds noise as a degradation element to teacher image data to generate degraded learning image data. In this embodiment, the degradation adding unit 211 may generate input image data by analyzing the physical characteristics of the imaging device and adding noise corresponding to a wider range of degradation amount than the degradation amount that can occur in the imaging device as a degradation element to the teacher image data. Since the range of degradation amount that can occur varies depending on the individual difference of the imaging device, by adding a wider range of degradation amount than the analysis result, a margin can be provided and robustness can be increased. Here, the teacher image may be acquired from, for example, the imaging device 10, or may be acquired from an external device (not shown) via the Internet.

[0039] Next, the degradation addition will be described with reference to FIG. 5. The degradation addition unit 211 generates degraded learning image data 504 by adding 503 noise based on the physical property analysis result 218 of the imaging device as degradation element 502 to teacher image data 501 extracted from the teacher image group 217. The degradation addition unit 211 adds a pair of teacher image data 501 and degraded learning image data 504 to learning data 505. The degradation addition unit 211 generates a degraded learning image group consisting of multiple degraded learning image data 504 by adding the degradation element 502 to each teacher image data of the teacher image group 217, and the learning data 505 is generated from the generated degraded learning image group. Note that, although noise is given as an example in this embodiment, the degradation addition unit 211 may add any one of multiple types of degradation elements such as blur, aberration, compression, low resolution, loss, and contrast reduction caused by the weather at the time of shooting to the teacher image data, or a combination of multiple types of degradation elements.

[0040] The teacher image group 217 stores various types of image data, such as nature photos including landscapes and animals, portraits or sports photos, and man-made photos such as architecture and products. The teacher image data according to this embodiment is RAW image data in which each pixel has a pixel value corresponding to one of the colors of RGB, as with the input image data 116. The physical property analysis result 218 of the imaging device includes, for example, the amount of noise for each sensitivity generated by an imaging sensor built into a camera (imaging device), or the amount of aberration generated by a lens. By using these, it is possible to estimate the degree of image quality degradation that occurs for each shooting condition. In other words, by adding the degradation estimated under a certain shooting condition to the teacher image data, it is possible to generate an image equivalent to the image obtained during shooting.

[0041] Returning to the explanation of FIG. 2, the learning unit 212 includes an image restoration unit 214 for learning, an error calculation unit 215, and a model update unit 216. The learning unit 212 acquires learning parameters 219, and performs restoration learning using the learning data generated by the degradation adding unit 211. The learning parameters 219 include initial values ​​of parameters of the neural network model, and hyperparameters indicating the network structure and optimization method of the model. The image restoration unit 214 performs image restoration processing on the input image. The error calculation unit 215 calculates the error between the image restoration result image output by the image restoration unit 214 and the teacher image. The model update unit 216 updates the parameters of the neural network model of the image restoration unit 214 based on the calculated error.

[0042] FIG. 3B is a diagram showing the flow of the learning process performed by the learning unit 212. The image restoration unit 214 performs image restoration processing by model execution 305 using data obtained by performing preprocessing 310 on the input image data 308 as input. In the model execution 305, the image restoration unit 214 repeats convolution calculation and nonlinear calculation using filters represented by equations (1) and (2) multiple times on the input data of the model, and outputs the restoration result 313. Next, the error calculation unit 215 calculates the error between the data obtained by performing preprocessing 311 on the teacher image data 309 and the restoration result 313 by loss calculation 314. After that, the model update unit 216 executes model update 315 based on the error calculated by the error calculation unit 215, and updates the network parameters of the model so that the error is reduced (minimized).

[0043] 3 to 5 are merely examples, and other processes may be performed as long as inference and learning can be performed similarly using a general CNN. In this embodiment, the degraded learning image is generated by the degradation adding unit 211, but the learning image may be prepared by other means, such as by acquiring the learning image from an external device (not shown).

[0044] The configuration shown in Fig. 2 can be modified or changed as appropriate. For example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. The configuration shown in Fig. 2 may be realized by two or more devices. In this case, the devices are connected via a circuit or a wired or wireless network, and perform data communication with each other to perform cooperative operations, thereby realizing each process according to this embodiment.

[0045] In this embodiment, the storage unit 113 stores the intermediate output from the first processing layer, and the image restoration unit 115 inputs the first intermediate output and the second intermediate output to the second processing layer, thereby outputting an inference result. Fig. 6 is a schematic diagram of a network configuration of such a model candidate. Hereinafter, an example of a network structure of a trained model 220 executed by the image restoration unit 115 will be described with reference to Fig. 6.

[0046] In the example of FIG. 6, input data 601 input to the network of the trained model is data obtained by pre-processing 301 of input image 116 at each time (represented by t). The input image 116 is an example in which a plurality of frames input sequentially are arranged in a time series, and output data (t=0) for an input image at t=0 is output. In addition, t is expressed as a negative value for a time relatively earlier than the image at t=0, and t is expressed as a positive value for a time relatively later. In FIG. 6, for the output image at t=0, output is performed for five frames of input data from t=-2 to 2.

[0047] The network in FIG. 6 receives five frames of input data 601 and outputs output data 605, which is the inference result for one frame. The network structure illustrated in FIG. 6 has a two-stage configuration in which a first inference process 602 is performed in a first processing layer, and a second inference process 604 is performed in a second processing layer. Here, the image restoration unit 115 inputs three chronologically consecutive input data 601 (t=0 to 2) as one set to the first processing layer, and performs the first inference process 602. Here, the storage unit 113 stores an intermediate output 603 (t=0 to 2) which is the result of the first inference process. The intermediate output 603 is, for example, data in an image format, but may be any data format that can be input to the subsequent second processing layer. The intermediate output may differ from the input data 601 in image characteristics such as width, height, and channel.

[0048] Here, the image restoration unit 115 inputs the output intermediate output 603 (t=0 to 2) and two previously output intermediate outputs 603 (t=-2 to 0, t=-1 to 1) stored in the storage unit 113 to the second processing layer, and outputs output data 605 as an inference result. That is, the intermediate outputs 603 (t=-2 to 0, t=-1 to 1) which are past intermediate outputs are stored in the storage unit 113, and the inference process in the second processing layer is performed by reusing these past intermediate outputs 603 in addition to the current intermediate output 603 (t=0 to 2). Hereinafter, the intermediate output (and the process of outputting it) that is not being reused (using the input data 601 of t=0 to 2) may be referred to as the "current" intermediate output (processing), and the intermediate output that is being reused may be referred to as the "previous" intermediate output. In addition, with respect to the current intermediate output, the intermediate output of the unit frame immediately before (here, data from t=-1 to 1 is input) may be referred to as the "previous (one before)" intermediate output, and the intermediate output two frames before (here, data from t=-2 to 0 is input) may be referred to as the "second before (two before)" intermediate output. This unit frame is the smallest unit of multiple frames arranged in time series that are input sequentially to the trained model 220.

[0049] In the example of Fig. 6, there are a plurality of input data 601 and one output data 605. However, this embodiment is not limited to this example, and the number of input data and the number of output data may each be one or more, and the output data 605 may be an output corresponding to another time other than t=0 of the input data. Also, in Fig. 6, the input data 601 is continuous data in a time series, but it does not have to be continuous data in a time series.

[0050] [Processing flow of the entire system] Next, various processes performed in the information processing system according to this embodiment will be described with reference to Figs. 7A to 7C. Figs. 7A to 7C are flowcharts showing an example of processes performed in the information processing system according to this embodiment. Each functional unit shown in Fig. 2 is realized by CPU 101 or 201 executing an information processing computer program according to this embodiment. However, all or part of the functional units shown in Fig. 2 may be implemented in hardware. Below, a description will be given along with the flowchart in Fig. 7. In the following description, the symbol "S" means a processing step.

[0051] First, an example of the flow of restoration learning performed by the cloud server 200 will be described with reference to the flowchart of Fig. 7A. In S701, the cloud server 200 inputs a teacher image group 217 prepared in advance and a physical property analysis result 218 of the imaging device 10, such as the characteristics of the imaging sensor and the sensitivity at the time of shooting, the subject distance, the focal length and F value of the lens, and the exposure value, to the cloud server 200. Note that the teacher image data here is a RAW image of a Bayer array, and is obtained by capturing an image with the imaging device 10. However, the method of acquiring the teacher image group 217 is not particularly limited to this, and for example, images captured in advance may be stored in a HDD or the like and then uploaded.

[0052] In S702, the degradation adding unit 211 adds noise based on the physical property analysis result 218 of the imaging device as a degradation element to the teacher image data of the teacher image group 217 input in S701, thereby generating degraded learning image data. Here, the degradation adding unit 211 adds noise of an amount measured in advance based on the physical property analysis result 218 of the imaging device to the teacher image data in a preset order or in a random order.

[0053] In S703, the learning unit 212 acquires the learning parameters 219 to be applied to the restoration learning. As described above, the learning parameters include the initial values ​​of the parameters of the neural network model, and the hyperparameters that indicate the network structure and optimization method of the model. In S704, the image restoration unit 214 performs the initial setting of the model using the received network parameters.

[0054] In S705, the image restoration unit 214 performs restoration processing of the degraded learning image data by inputting the degraded learning image data to the model. The image restoration unit 214 performs restoration processing by inputting a first (current) intermediate output from the first processing layer and a second intermediate output prior to the first intermediate output to the second processing layer using a machine learning model having a first processing layer and a second processing layer as shown in Fig. 6, for example.

[0055] In S706, the error calculation unit 215 calculates the error between the restoration result in S705 and the teacher image data according to the loss function shown in formula (3). In S707, the model update unit 216 updates the network parameters of the model so that the error obtained in S706 becomes smaller (minimum). Here, the model update unit 216 updates the parameters not only for the current intermediate output, but also using previous intermediate outputs. By learning in this manner, it becomes possible to update the model parameters more efficiently without wasting the calculation time required for outputting the previous intermediate output.

[0056] In S708, the learning unit 212 determines whether or not to end the learning. For example, the learning unit 212 determines to end the learning when the number of updates of the network parameters reaches a predetermined number, or when a predetermined period has passed since the start of the learning. If the learning is to be ended, the process proceeds to S709, and if the learning is not to be ended, the process returns to S705. In S705 to S708 from the second time onwards, the learning is continued using unprocessed degraded learning image data and teacher image data. In S709, the learning unit 212 stores the learned model in the mass storage device 204 or the like.

[0057] Next, a flow of an example of the restoration inference process performed in the information processing device 100 will be described with reference to the flowchart in Fig. 7B. In S710, the image restoration unit 115 determines whether or not the initial settings of the network model used for inference have been completed. If the initial settings have been completed, the process proceeds to S713, and if not, the process proceeds to S711. In S711, the model storage unit 119 acquires the trained model 220 from the cloud server 200 and stores it.

[0058] In S712, the image restoration unit 115 performs initial settings of a model to be used for inference in order to start inference using the trained model 220 stored in the model storage unit 119. The image restoration unit 214 performs restoration processing by using a machine learning model having a first processing layer and a second processing layer as shown in Fig. 6, for example, by inputting a first (current) intermediate output from the first processing layer and a second intermediate output prior to the first intermediate output to the second processing layer.

[0059] In S713, the information processing device 100 acquires input image data 116, which is a RAW image in a Bayer array to be subjected to image restoration processing. The RAW image may be directly input after being captured by the imaging device 10, or may be read out after being captured in advance and stored in the large-capacity storage device 104. Here, the input image data 116 acquired is at least the number of frames to be input to the first processing layer in S714.

[0060] In S714, the image restoration unit 115 inputs the input image data 116 to the first processing layer of the trained model 220, and outputs an intermediate output. In S715, the storage unit 113 stores the intermediate output output in S714 together with information corresponding to the input. Here, the storage unit 113 can store the intermediate output together with information indicating the timing of the input, for example, t=0 to 2, but this information is not limited to this as long as the image restoration unit 115 can determine which input the intermediate output corresponds to.

[0061] In S716, the image restoration unit 115 acquires the previous intermediate output stored by the storage unit 113. In the example of FIG. 6, the intermediate output corresponding to the input image data 116 of t=0 to 2 is output in S714, and the intermediate output corresponding to the input data 116 corresponding to t=-2 to 0 and t=-1 to 1 is acquired in S716. In S717, the image restoration unit 115 inputs the intermediate output output in S714 and the intermediate output acquired in S716 to the second processing layer of the trained model 220, and outputs the inference result of the model. In S717, the image data restored by the image restoration unit 115 is output as output image data 118, and the processing of FIG. 7B ends.

[0062] The above is the overall flow of the processing performed in the information processing system of this embodiment. According to this processing, it is possible to perform inference by storing the intermediate output by the first processing layer in the machine processing model and inputting the current intermediate output and the previous intermediate output to the second processing layer. Therefore, by performing processing by reusing the past intermediate output, it is possible to speed up the inference processing.

[0063] The information processing device 100 may store a plurality of trained models 220 in the model storage unit 119, and switch the model to be used for inference when a predetermined condition (switching condition) is satisfied. For example, the information processing device 100 may satisfy the switching condition when the resource usage state during inference is a predetermined state (for example, the resource usage rate exceeds 90%). In this way, it is possible to select a more appropriate model and perform inference processing by switching the model to be used to a model that consumes less resources during inference, for example, depending on whether the resource usage state is a predetermined state. Therefore, for example, the image restoration unit 115 may set a ranking for each characteristic, such as the resource consumption amount during inference, for each stored trained model 220, and switch the model to be used according to the set ranking when the switching condition for the characteristic is satisfied. Such model switching will be described with reference to FIG. 7(C).

[0064] The flowchart in Fig. 7C is a flowchart showing an example of a model switching process performed by the image restoration unit 115. The process in Fig. 7C can be performed, for example, before S713.

[0065] In S720, the image restoration unit 115 acquires the resource usage state. In S721, the image restoration unit 115 acquires the model characteristics of the trained model 220 stored in the model storage unit 119. In addition to the resource consumption during inference described above, various items such as the accuracy of inference or memory usage can be set as the model characteristics. In S722, the switching unit 113 selects a model to be used for inference next (a model to be switched to) from the trained models 220 stored in the model storage unit 119 based on the resource usage state acquired in S720 and the model characteristics acquired in S721. Here, the image restoration unit 115 may set a ranking between the models for the model characteristic "resource consumption during inference", for example, and switch to a model one rank lower than the currently used model when the resource usage rate exceeds a predetermined threshold.

[0066] In S723, the switching unit 113 determines whether the model selected in S722 is different from the model currently being used. If they are different, the process proceeds to S724, and if they match, the process ends. In S724, the switching unit 113 switches the model used by the image restoration unit 115 to the model selected in S722, and ends the process.

[0067] In this embodiment, the learning data is generated in S702, but it may be generated later. Specifically, it may be configured to generate input image data corresponding to the teacher image data in the subsequent restoration learning. In addition, in this embodiment, the cloud server 200 has been described as performing learning from scratch using data of a group of teacher images prepared in advance, but it is also possible to obtain a machine learning model having trained network parameters and perform each process using the model.

[0068] In the present embodiment, the processing is performed using noise as a degradation factor, but similar processing can be performed when a degradation factor other than noise is used. For example, the degradation factor may be blur, aberration, compression, low resolution, loss, or contrast reduction due to the influence of fog, haze, snow, or rain at the time of shooting, as described above, or a combination of a plurality of these.

[0069] In the present embodiment, an example has been described in which the information processing device 100 performs restoration only from the input image data using a trained model, but parameters that assist image restoration may also be used. For example, the information processing device 100 may hold a lookup table that estimates in advance the degree of image quality degradation that occurs depending on shooting conditions such as the distance to the subject, the focal length, the sensor size, or the exposure, and may adjust the amount of restoration by referring to the lookup table during image restoration. That is, the inference unit 112 of the information processing device 100 may adjust the restoration strength of the degradation based on the shooting conditions when the image of the input image data was captured.

[0070] In this embodiment, an example of using a machine learning model that performs image restoration processing has been described, but the task of the model is not limited to image restoration processing, and similar processing can be performed even when using a model that performs image recognition, for example.

[0071] [Embodiment 2] In the first embodiment, an example of an information processing device that reuses a predetermined number of previous intermediate outputs as input to the second processing layer has been described. Compared with the configuration of the first embodiment, the information processing device according to the second embodiment changes the interval or number of intermediate outputs input to the second processing layer in inference. Note that the basic configuration of the information processing system and the functional configuration of the information processing device 100 are the same as those in the first embodiment, and a description of the common contents will be omitted.

[0072] Hereinafter, a model in which intermediate outputs to be reused are discretely used from the past at regular intervals, a model in which intermediate outputs to be reused are discretely used from the past at irregular intervals, and a model in which the number of intermediate outputs to be reused is variable according to this embodiment will be described with reference to Figs. 8A to 8C. Note that the inference processing performed by the model of this embodiment is assumed to be super-resolution processing.

[0073] The model of FIG. 8A to FIG. 8C is a network having a two-stage configuration of a first processing layer and a second processing layer, similar to the model shown in FIG. 6. In the model shown in FIG. 8A to FIG. 8C of this embodiment, similar to the model shown in FIG. 6, three frames of input data 801 are input to the first processing layer as a first input, and a first intermediate output 803 is output. The intermediate output 803 may be data having a larger number of channels than the input image, for example. In the network of FIG. 8A to FIG. 8C, in addition to the first intermediate output, the previous intermediate output is input to the second processing layer, and output data 805 is output as an inference result. Hereinafter, the difference between the network model of FIG. 6 and the network shown in FIG. 8A to FIG. 8C will be described.

[0074] In the network of FIG. 6, inference is performed by using the previous intermediate output (t=-1 to 1) and the intermediate output before last (t=-2 to 0) as inputs to the second processing layer in addition to the current intermediate output (t=0 to 2). Here, the interval between each intermediate output in the network of FIG. 6 is one unit frame. On the other hand, the network of FIG. 8A is a model that uses the intermediate output to be reused discretely from a certain interval in the past. That is, in the network of FIG. 8A, inference is performed by using the intermediate output before last (two times ago) (t=-2 to 0) and the intermediate output four times ago (t=-4 to -2) as inputs to the second processing layer in addition to the current intermediate output (t=0 to 2). In this way, the information processing device 100 according to the present embodiment can set an interval between the first input and the second input, and input the second intermediate output corresponding to the second input specified by the set interval to the second processing layer for the first input. For example, by lengthening the interval between intermediate outputs input to the second processing layer, as in the network of Fig. 8A for the network of Fig. 6, inference processing can be performed using information of a longer time even when the same number of intermediate outputs are used. Note that, hereinafter, the interval between intermediate outputs and the inputs input to the first processing layer to output those intermediate outputs will be referred to as the "intermediate output interval."

[0075] In the network of FIG. 6, two stored intermediate outputs are acquired at one frame intervals for the first input, and in the network of FIG. 8A, stored intermediate outputs are acquired at two frame intervals for the first input. However, the image restoration unit 115 does not need to set the intervals of the acquired intermediate outputs (second intermediate outputs) to a constant interval. For example, the image acquisition unit 115 may acquire a second intermediate output corresponding to a second input inputted a first interval before the first input, and a third intermediate output corresponding to a third input inputted a second interval before the second input (different from the first interval). The image restoration unit 115 can input the second intermediate output and the third intermediate output acquired in this way, and the first intermediate output, to the second processing layer, and output an inference result.

[0076] FIG. 8B illustrates a model in which the current intermediate output 803, the stored intermediate output 803 acquired at such unequal intervals, and the intermediate output 803 are input to the second processing layer. In the network of FIG. 8B, in addition to the current intermediate output, the previous (one previous) intermediate output (t=-1 to 1) and the intermediate output four times previous (t=-4 to -2) are input to the second processing layer to perform inference. In this way, by setting the interval for acquiring the intermediate output to be closer to the current (processing target) frame than to other frames, it becomes possible to perform inference processing using intermediate outputs that are more closely related to the processing target frame. Note that, although an example in which the first interval is shorter than the second interval has been described here, it is also possible to implement in a different manner, for example, by setting the second interval to be shorter than the first interval when information immediately before the processing target frame is relatively unnecessary.

[0077] In addition, in all of the models illustrated in Fig. 6, Fig. 8A, and Fig. 8B, a total of three intermediate outputs are input to the second processing layer (i.e., two stored intermediate outputs are obtained and reused). However, the number of intermediate outputs reused here is not limited, and may be changeable depending on conditions. That is, the image restoration unit 115 according to this embodiment can change the number of intermediate outputs to be reused depending on whether a predetermined condition (change condition) is satisfied.

[0078] Fig. 8C illustrates a model in which a total of five intermediate outputs (four intermediate outputs are reused) are input to the second processing layer to perform inference. In the model of Fig. 6, three intermediate outputs are input to the second processing layer, whereas by inputting five intermediate outputs to the second processing layer, it is possible to increase information and improve the accuracy of inference. In addition, when the number of intermediate outputs input to the second processing layer is reduced, it is possible to reduce the information to be input and reduce the processing load of inference.

[0079] The above-mentioned change condition may be, for example, whether or not the user has input a change in the settings. For example, the image restoration unit 115 may acquire various user inputs of settings, such as whether the user prioritizes speed or accuracy in inference, via the input device 20, and change the number of intermediate outputs acquired (repurposed) to be input to the second processing layer according to the settings. In this example, the image restoration unit 115 may reduce the number of intermediate outputs to be diverted when the settings are changed to prioritize speed in inference, and increase the number of intermediate outputs to be diverted when the settings are changed to prioritize accuracy in inference. In addition, for example, the image restoration unit 115 may evaluate the processing load in inference, and change the number of intermediate outputs to be diverted when the evaluation of the processing load reaches a predetermined state. This means, for example, reducing the number of intermediate outputs to be diverted when a high-load state occurs, such as when the resource usage rate exceeds a predetermined threshold, and increasing the number of intermediate outputs to be diverted when a low-load state occurs, such as when the resource usage rate falls below a predetermined threshold.

[0080] Furthermore, for example, the image restoration unit 115 may increase the number of intermediate outputs to be used when a condition for improving accuracy is met, such as when a specific subject set as a subject for which high-precision super-resolution processing is desired is detected. Here, since detection of the specific subject can be performed by a known detection technique, detailed description will be omitted.

[0081] In this embodiment, even if the number of intermediate outputs to be reused is changed, the information on the intermediate outputs stored in the storage unit 113 is kept, and even after the change, the inference process is performed by the second processing layer using the stored intermediate outputs. According to this process, even if the number of intermediate outputs to be reused is changed, the process can be continued without re-storing the intermediate outputs, and the processing time can be reduced compared to the case where the intermediate outputs are re-stored.

[0082] [Embodiment 3] With reference to FIG. 9, a case where the result of the intermediate output up to the previous time cannot be reused will be described. The network shown in FIGS. 9A and 9B has a three-stage configuration with an additional processing layer before the first processing layer in the first embodiment, and in this embodiment, this additional processing layer is called the "0th processing layer". In the 0th processing layer, a plurality of image data 901 are input as one set, and inference input data 903 is output. In this embodiment, in the 0th processing layer, preprocessing is performed before information processing by the first processing layer and the second processing layer. Here, the inference input data 903 output from the 0th processing layer is auxiliary information different from the image, which is effective for performing the target image processing (Debayer). In FIG. 9A, image data for four frames (t=-1 to 2) is input as one set to the 0th processing layer, and corresponding inference output data is output as four pieces (t=-1 to 2). Also, in FIG. 9B, image data for three frames (t=0 to 2) is input as one set to the 0th processing layer, and corresponding inference output data is output as three pieces (t=0 to 2). The first processing layer receives a set of three pieces of inference input data, and outputs one intermediate output 905 for each set. That is, in Fig. 9(A), two intermediate outputs 905 corresponding to t = -1 to 1 and t = 0 to 2 are output, and in Fig. 9(B), one intermediate output 905 corresponding to t = 0 to 2 is output.

[0083] Here, in the example of FIG. 9(A), since the intermediate outputs corresponding to t=-1 to 1 are different between the previous process and the current process, the previous intermediate outputs cannot be used in the second processing layer as in the first embodiment. This is because the inference input data 903 corresponding to t=-1 to 1 output from the 0th output layer when image data for four frames from t=-2 to 1 is input to the 0th processing layer in the previous process is different from the current process. On the other hand, according to the structure of the network model shown in FIG. 9(B), the inference process can be speeded up by performing the process by using the past intermediate outputs as in the first embodiment while performing the preprocessing (0th process) for outputting the same number of data as the number of input data to the trained model 220.

[0084] Note that the processing in the first processing layer and the second processing layer according to this embodiment is the same as that in embodiment 1, and therefore the description thereof will be omitted here. Also, the trained model 220 according to this embodiment performs Debayer processing, which converts information input to the machine learning model into an image as an inference process, but this process is not particularly limited to this.

[0085] The disclosure of this specification includes the following information processing device, information processing method, and program.

[0086] (Item 1) A processing means for performing inference processing using a machine learning model having a first processing layer and a second processing layer; a storage means for storing intermediate outputs from the first processing layer; In the processing means, a first processing layer receives a first input and outputs a first intermediate output corresponding to the first input; the second processing layer receives the first intermediate output and a second intermediate output from the first processing layer corresponding to a second input earlier than the first input, and outputs an inference result; The storage means stores the first intermediate output. An information processing device comprising: (Item 2) the second processing layer receives as input the first intermediate output, and a third intermediate output from the first processing layer corresponding to the second intermediate output and a third input preceding the second input; 2. The information processing device according to item 1, characterized in that a first interval at which the first input and the second input are input to the first processing layer is equal to a second interval at which the second input and the third input are input to the first processing layer. (Item 3) 3. The information processing device according to item 2, further comprising a first setting means for setting the first interval and the second interval. (Item 4) Item 4. The information processing device according to item 3, wherein the first setting means sets the first interval and the second interval to an interval of a unit frame of image data sequentially input to the machine learning model. (Item 5) the second processing layer receives as input the first intermediate output, and a third intermediate output from the first processing layer corresponding to the second intermediate output and a third input preceding the second input; 2. The information processing device according to item 1, characterized in that a first interval at which the first input and the second input are input to the first processing layer is different from a second interval at which the second input and the third input are input to the first processing layer. (Item 6) 6. The information processing device according to item 5, wherein the first interval is shorter than the second interval. (Item 7) 7. The information processing device according to any one of items 1 to 6, further comprising a second setting means for setting the number of the second intermediate outputs to be input to the second processing layer. (Item 8) The machine learning model further includes a detection unit for detecting a predetermined object from an image input to the machine learning model, 8. The information processing device according to item 7, wherein the second setting means increases the number of the second intermediate outputs input to the second processing layer when the specified subject is detected, compared to when the specified subject is not detected. (Item 9) An evaluation means for evaluating a processing load in the inference is further provided, 8. The information processing device according to item 7, characterized in that, when the evaluation of the processing load reaches a predetermined state, the number of the second intermediate outputs input to the second processing layer is changed. (Item 10) The machine learning model further comprises a third processing layer before the first processing layer; The information processing device according to any one of items 1 to 9, characterized in that the processing means outputs the first input by inputting a third input, which is the same number of inputs as the first inputs, to the third processing layer. (Item 11) An update means for updating parameters of the machine learning model is further provided, The information processing device described in any one of items 1 to 10, characterized in that the update means updates the parameters based on a third intermediate output for a fourth input from the first processing layer and a fourth intermediate output from the first processing layer corresponding to a fifth input earlier than the fourth input. (Item 12) 12. The information processing device according to any one of items 1 to 11, wherein the inference is a process of restoring an input degraded image. (Item 13) The information processing device according to any one of items 1 to 11, characterized in that the inference is a super-resolution process that restores information of an image input to the machine learning model. (Item 14) The information processing device according to any one of items 1 to 11, characterized in that the inference is a Debayer process that converts information input to the machine learning model into an image. (Item 15) A processing step of performing inference processing using a machine learning model having a first processing layer and a second processing layer; and storing an intermediate output from the first processing layer; In the processing step, a first processing layer receives a first input and outputs a first intermediate output corresponding to the first input; the second processing layer receives the first intermediate output and a second intermediate output from the first processing layer corresponding to a second input earlier than the first input, and outputs an inference result; In the storing step, the first intermediate output is stored. 1. An information processing method comprising: (Item 16) A program for causing a computer to function as each of the means of the information processing device according to any one of items 1 to 14.

[0087] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0088] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0089] 100: information processing device, 101: CPU, 102: RAM, 103: ROM, 104: mass storage device, 105: general-purpose interface, 106: network interface, 107: system bus

Claims

1. A processing means for performing inference processing using a machine learning model having a first processing layer and a second processing layer; a storage means for storing intermediate outputs from the first processing layer; In the processing means, a first processing layer receives a first input and outputs a first intermediate output corresponding to the first input; the second processing layer receives as input the first intermediate output, a second intermediate output from the first processing layer corresponding to a second input earlier than the first input, and a third intermediate output from the first processing layer corresponding to a third input earlier than the second input, and outputs an inference result; the storage means stores the first intermediate output, the second intermediate output, and the third intermediate output; A first interval between the first input and the second input inputted to the first processing layer is different from a second interval between the second input and the third input inputted to the first processing layer. An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, further comprising a first setting means for setting the first interval and the second interval.

3. The information processing device according to claim 2 , wherein the first setting means sets the first interval and the second interval to intervals based on unit frames of image data sequentially input to the machine learning model.

4. 2. The information processing apparatus according to claim 1, wherein the first interval is shorter than the second interval.

5. 2. The information processing apparatus according to claim 1, further comprising a second setting means for setting the number of intermediate outputs to be input to said second processing layer.

6. The machine learning model further includes a detection unit for detecting a predetermined object from an image input to the machine learning model, 6. The information processing device according to claim 5, wherein the second setting means increases the number of the intermediate outputs input to the second processing layer when the specified subject is detected, compared to when the specified subject is not detected.

7. An evaluation means for evaluating a processing load in the inference is further provided, 6. The information processing apparatus according to claim 5, further comprising: a step of changing a number of said intermediate outputs input to said second processing layer when said evaluation of said processing load reaches a predetermined state.

8. The machine learning model further comprises a third processing layer before the first processing layer; 2 . The information processing device according to claim 1 , wherein the processing means outputs the first input by inputting third inputs, which are the same number of inputs as the first inputs, to the third processing layer.

9. An update means for updating parameters of the machine learning model is further provided, 2. The information processing device according to claim 1, wherein the update means updates the parameters based on a fourth intermediate output for a fourth input from the first processing layer and a fifth intermediate output from the first processing layer corresponding to a fifth input earlier than the fourth input.

10. 2. The information processing apparatus according to claim 1, wherein the inference is a process of restoring an input degraded image.

11. The information processing device according to claim 1 , wherein the inference is a super-resolution process for restoring information of an image input to the machine learning model.

12. The information processing apparatus according to claim 1 , wherein the inference is a Debayer process that converts information input to the machine learning model into an image.

13. A processing step of performing inference processing using a machine learning model having a first processing layer and a second processing layer; and storing an intermediate output from the first processing layer, In the processing step, a first processing layer receives a first input and outputs a first intermediate output corresponding to the first input; the second processing layer receives as input the first intermediate output, a second intermediate output from the first processing layer corresponding to a second input earlier than the first input, and a third intermediate output from the first processing layer corresponding to a third input earlier than the second input, and outputs an inference result; a first interval between the first input and the second input inputted to the first processing layer is different from a second interval between the second input and the third input inputted to the first processing layer; In the storing step, the first intermediate output, the second intermediate output, and the third intermediate output are stored.

23. An information processing method executed by a computer.

14. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 12.

Images