Information processing apparatus, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing method, and a program, configured to switch machine learning models faster.SOLUTION: An information processing apparatus performs inference on an input using a first machine learning model, and selects a second machine learning model in which at least some of configuration units of a network structure of the first machine learning model have been modified, as a machine learning model used for the inference, in response to a predetermined condition being satisfied. The first machine learning model includes a first configuration unit and a second configuration unit. The second machine learning model has a configuration unit in which a part of a parameter or a path between nodes in the first configuration unit or the second configuration unit has been modified.SELECTED DRAWING: None

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, machine learning has been applied to various information processing application programs. Information processing using machine learning is diverse, such as image processing for restoring a degraded image or class classification for determining the type of object in an image.

[0003] Non-Patent Document 1 discloses a method of removing noise from a video using a two-stage model consisting of a first noise removal unit and a second noise removal unit. Patent Document 1 also discloses a method of performing recognition processing using multiple devices with different computing capabilities. In Patent Document 1, the response time required to recognize data is calculated, and if the response time exceeds a threshold, the model is switched to a faster model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-77129 [Non-patent literature]

[0005] [Non-Patent Document 1] “FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation”,Matias, Tassano; Julie, Delon; Thomas, Veit, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 1354-1363 Summary of the Invention [Problem to be solved by the invention]

[0006] There is a need to switch models depending on the execution environment. For example, there may be a case where a high-precision but high-load and low-speed model is switched to a low-load and high-speed but low-precision model depending on the resource usage state of the CPU / GPU of the model execution environment. However, the method disclosed in Patent Document 1 has a problem in that the model to be used needs to be initialized when switching models, and the switching may take a long time.

[0007] The present invention aims to switch machine learning models more quickly. [Means for solving the problem]

[0008] In order to achieve the object of the present invention, for example, an information processing device according to one embodiment has the following configuration: That is, the information processing device has an inference means for performing inference on an input using a first machine learning model, and a selection means for selecting, in response to a predetermined condition being satisfied, a machine learning model to be used for the inference, which is a second machine learning model in which at least a part of a component of a network structure of the first machine learning model has been changed, the first machine learning model has a first component and a second component, and the second machine learning model has at least a component in which a part of a parameter or a path between nodes in the first component or the second component has been changed. Effect of the Invention

[0009] Switch between machine learning models faster. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an information processing system according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a functional configuration of the information processing device according to the first embodiment. [Diagram 3]FIG. 2 is a diagram for explaining a learning process in the information processing device according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of the configuration of a model used by the information processing device according to the first embodiment. [Diagram 5] FIG. 4 is a diagram for explaining the degradation adding process according to the first embodiment. [Figure 6A] FIG. 2 is a diagram showing an example of a network structure of a model according to the first embodiment. [Figure 6B] FIG. 2 is a diagram showing an example of a network structure of a model according to the first embodiment. [Figure 6C] FIG. 2 is a diagram showing an example of a network structure of a model according to the first embodiment. [Figure 6D] FIG. 2 is a diagram showing an example of a network structure of a model according to the first embodiment. [Figure 6E] FIG. 2 is a diagram showing an example of a network structure of a model according to the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of characteristics of a model according to the first embodiment. [Figure 8] FIG. 2 is a diagram for explaining a process of performing inference using multiple GPUs according to the first embodiment. [Figure 9A] 5 is a flowchart showing an example of a learning process according to the first embodiment. [Figure 9B] 5 is a flowchart showing an example of a restoration process according to the first embodiment. [Figure 9C] 5 is a flowchart showing an example of a model switching process according to the first embodiment. [Figure 10] FIG. 11 is a block diagram showing an example of a functional configuration of an information processing device according to a second embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a setting screen according to the second embodiment. [Figure 12A] 10 is a flowchart showing an example of a model switching process according to the second embodiment. [Figure 12B] 6 is a flowchart showing an example of a setting process for user settings according to the first embodiment. [Figure 13] FIG. 11 is a block diagram showing an example of a functional configuration of an information processing device according to a third embodiment. [Figure 14]FIG. 13 is a diagram showing an example of a network structure of a model according to a third embodiment. [Figure 15A] 5 is a flowchart showing an example of a learning process according to the first embodiment. [Figure 15B] 5 is a flowchart showing an example of a class classification process according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] [Embodiment 1] The information processing device according to the present embodiment performs inference on an input using a first machine learning model. Next, when a predetermined condition is satisfied, the information processing device switches the machine learning model used for inference to a second machine learning model by switching components of the network structure of the first machine learning model.

[0013] [About CNN] First, a description will be given of a Convolutional Neural Network (CNN) used in information processing technology generally using deep learning, which is used in the following embodiments. CNN is a mathematical model that repeats convolution of a filter generated by training with image data and then nonlinear operation. In CNN, data obtained by convolving input image data with a filter and then nonlinear operation is called a feature map. In addition, CNN training is performed using training data (training images or data sets) consisting of pairs of input image data and output image data. In other words, generating filter values ​​that can convert input image data to corresponding output image data with high accuracy from training data is called learning.

[0014] When image data has RGB color channels or when a feature map is composed of multiple pieces of image data, the filter used for convolution also has multiple channels accordingly. In CNN, the process of performing nonlinear calculations after convolving a filter with image data (or feature map) is expressed in units of layers, such as the nth layer feature map and the nth layer filter. For example, a CNN with a three-layer network structure repeats the filter convolution and nonlinear calculation three times. This type of nonlinear calculation process can be formulated as the following equation (1).

number

[0015] In formula (1), W n is the nth layer filter, b n is the bias of the nth layer, f is a nonlinear operator, and X nrepresents the n-th layer feature map, and * represents the convolution operator. Note that (l) indicates that this is the l-th filter or feature map. This filter and bias are generated by learning, and are collectively called "network parameters." For example, a sigmoid function or ReLU (Rectified Linear Unit) is used as the nonlinear operation. When ReLU is used, the nonlinear operation process in CNN is given by, for example, the following formula (2).

number

[0016] As shown in equation (2), negative elements of the input vector X will be set to zero, and elements greater than or equal to 0 will remain unchanged.

[0017] Examples of networks using CNN include Residual Network (ResNet) in the field of image recognition and RED-Net, which is an application of ResNet in the field of super-resolution. Both of these enable high-precision recognition by making CNN multi-layered and performing filter convolutions multiple times. For example, ResNet features a network structure with paths that shortcut the convolutional layer, which allows for a multi-layer network of 152 layers, achieving high-precision recognition approaching the recognition rate of humans. Multi-layer CNNs achieve high-precision recognition by expressing nonlinear relationships between input and output by repeating nonlinear calculations multiple times.

[0018] [CNN learning] Next, we will explain CNN learning. CNN learning is performed by minimizing the objective function expressed by, for example, the following formula (3) for learning data consisting of a set of input image data and corresponding correct output image (teacher image) data.

number

[0019] In equation (3), L is a loss function that measures the error between the correct answer and the inference result. Y i is the i-th correct output image data, X i is the i-th input image data. F is a function that collectively represents the calculations (equation (1)) performed in each layer of CNN. θ is the network parameters (filter and bias). ||Z||2 is the L2 norm, that is, the square root of the sum of the squares of the elements of vector Z. n is the number of data sets in the training data. Since the amount of training data is generally large, in the Stochastic Gradient Descent (SGD) method, a portion of the training data is randomly selected and used for training. In addition, various methods are known for minimizing (optimizing) the objective function, such as the momentum method, AdaGrad method, AdaDelta method, and Adam method. The Adam method is given by the following equation (4).

number

[0020] In formula (4), t represents the value of the tth iteration. θ i t is the i-th network parameter at iteration t, and g is θ i t is the gradient of the loss function L with respect to m. m and v are moment vectors, α is the base learning rate, β1 and β2 are hyperparameters, and ε is a small constant. Note that the optimization method in learning is not limited to this, and any known optimization technique can be applied. It is known that the convergence of each method differs, resulting in differences in learning time, and it is possible to adopt an optimization method according to desired conditions.

[0021] In this embodiment, information processing (image processing) is performed to restore a degraded image using the above-mentioned CNN. Examples of image degradation factors in this degraded image include degradation such as noise, blur, aberration, compression, low resolution, and loss, and degradation such as contrast reduction due to weather conditions such as fog, haze, snow, and rain at the time of shooting. Examples of image processing to restore a degraded image include noise removal, blur removal, aberration correction, correction of degradation due to compression, super-resolution processing for low-resolution images, loss completion, and processing to correct contrast reduction due to weather conditions at the time of shooting. The degraded image restoration processing according to this embodiment is a process of performing restoration by generating an image with no degradation (or very little degradation) from an image with degradation, and will be referred to as image restoration (processing) in the following description. That is, image restoration according to this embodiment includes not only a case where an image with no degradation (or little degradation) itself is restored but also a case where degradation caused by subsequent amplification, compression / expansion, or other image processing is restored, but also a case where degradation contained in the original image itself can be reduced.

[0022] Here, when using an image restoration process using a neural network, it is generally expected that the expressive power of the network will be improved and image restoration with higher accuracy will be performed when a neural network with a larger number of parameters is used. On the other hand, a larger number of parameters leads to an increase in the number of calculations, which increases the inference time and the amount of memory required for inference. Therefore, the model to be used in the image restoration process differs depending on the amount of calculation resources and the usage state of the information processing device that performs the inference.

[0023] [Embodiment 1] In the first embodiment, a method for switching the model for performing image restoration processing without reinitialization will be described. In the following embodiment, noise will be taken as an example of an image degradation factor, and a process for performing noise reduction processing as image restoration processing will be described.

[0024] [Example of information processing system configuration] FIG. 1 is an example of an information processing system configuration including an information processing device according to the first embodiment. In the information processing system shown in FIG. 1, a cloud server 200 that generates learning data and learns restoration of image quality degradation, and an information processing device 100 (edge ​​device 100) that performs image restoration for a processing target image are communicably connected via the Internet or the like. Hereinafter, the generation of learning data and the learning of restoration of image quality degradation in the cloud server 200 will be referred to as restoration learning, and the restoration of a degraded image in the information processing device 100 will be referred to as restoration inference. Note that, in this embodiment, the cloud server 200 and the information processing device 100 will be described as separate devices, but the restoration learning process described as being performed by the cloud server 200 may be performed by the information processing device 100.

[0025] [Hardware configuration of information processing device] The information processing device 100 acquires an image to be processed, and sets the acquired image as an input image to a machine learning model that performs restoration inference. In this embodiment, the information processing device 100 may acquire RAW image data (Bayer array) input from the imaging device 10 as an input image to be subjected to image restoration processing. This image is not particularly limited in terms of image format such as color space, and any image format can be used. For example, this image may be image data of other color filter arrays, a demosaiced RGB image, or an image after YUV conversion.

[0026] The information processing device 100 performs restoration processing of a degraded image by a machine learning model using an input image to be processed as an input. The information processing device 100 according to the present embodiment can perform restoration inference using a trained neural network model provided by the cloud server 200. That is, the information processing device 100 can be an information processing device that reduces noise in RAW image data by using the model provided by the cloud server 200 and executing a pre-installed information processing application program.

[0027] The information processing device 100 has a CPU 101, a RAM 102, a ROM 103, a large-capacity storage device 104, a general-purpose interface (I / F) 105, and a network I / F 106, and each component is connected to each other via a system bus 107. The information processing device 100 is also connected to an imaging device 10, an input device 20, an external storage device 30, and a display device 40 via the general-purpose I / F 105.

[0028] The CPU 101 uses the RAM 102 as a work memory, executes a program stored in the ROM 103, and generally controls each component of the information processing device 100 via a system bus 107. The mass storage device 104 is, for example, an HDD or an SSD, and stores various data and image data handled by the information processing device 100. The CPU 101 writes data to the mass storage device 104 via the system bus 107, and reads data stored in the mass storage device 104. The general-purpose I / F 105 is, for example, a serial bus interface such as USB, IEEE1394, or HDMI (registered trademark). The information processing device 100 acquires data from an external storage device 30 (for example, various storage media such as a memory card, a CF card, an SD card, or a USB memory) via the general-purpose I / F 105.

[0029] The information processing device 100 receives user instructions from an input device 20 such as a mouse or a keyboard, or a mobile terminal such as a smartphone, via a general-purpose I / F 105. The information processing device 100 also outputs image data processed by the CPU 101 to a display device 40 (for example, various image display devices such as a liquid crystal display) via the general-purpose I / F 105. The display device 40 can also function as the input device 20 as a display device integrated with a touch panel. The information processing device 100 also acquires data of a captured image (RAW image) to be subjected to noise reduction processing from the imaging device 10 via the general-purpose I / F 105. The network I / F 106 is an interface for connecting to the Internet. The information processing device 100 can acquire a trained model for restoration inference by accessing a cloud server 200 via an installed web browser.

[0030] [Cloud server hardware configuration] The cloud server 200 provides the information processing device 100 with a machine learning model that performs restoration processing on an input degraded image. The cloud server 200 according to this embodiment is a server that provides cloud services on the Internet. More specifically, the cloud server 200 generates learning data and performs restoration learning, and generates a trained machine learning model (trained model) that stores network parameters and a network structure of the learning result. Then, the cloud server 200 provides the trained model in response to a request from the information processing device 100. Hereinafter, when simply referred to as a "model", this refers to the trained model.

[0031] The cloud server 200 has a CPU 201, a ROM 202, a RAM 203, a mass storage device 204, and a network I / F 205, and each component is connected to each other by a system bus 206. The CPU 201 controls the overall operation by reading out a control program stored in the ROM 202 and executing various processes. The RAM 203 is used as a temporary storage area such as a main memory or work area of ​​the CPU 201. The mass storage device 204 is a large-capacity secondary storage device such as an HDD or SSD that stores image data or various programs. The network I / F 205 is an interface for connecting to the Internet, and provides a trained model that stores the above-mentioned network parameters and network structure in response to a request from a web browser of the information processing device 100.

[0032] In this embodiment, it is assumed that the information processing device 100 downloads a trained model, which is a result of performing training data generation and restoration learning, from the cloud server 200, and performs restoration inference on input image data to be processed. The components of the information processing device 100 and the cloud server 200 that perform such processing may be realized by a configuration different from the configuration described above. For example, the functions of the cloud server 200 may be subdivided, and the generation of training data and the restoration learning may be performed by separate devices. Alternatively, the imaging device 10 that has both the functions of the information processing device 100 and the cloud server 200 may be configured to perform all of the generation of training data, restoration learning, and restoration inference.

[0033] [System-wide functional blocks] Next, the functional configuration of the entire information processing system according to this embodiment will be described with reference to Fig. 2. As shown in Fig. 2, the information processing device 100 has an inference unit 112 and a model storage unit 119. The model storage unit 119 acquires and stores a trained model 220 from a cloud server 200. The inference unit 112 has a function of image restoration processing for restoring a degraded image. The inference unit 112 includes an image restoration unit 115 and a switching unit 113 for inference.

[0034] The image restoration unit 115 inputs a degraded image to the trained model 220 and restores the image. Here, the processing performed by the image restoration unit 115 will be described with reference to FIG. 3(A). The image restoration unit 115 acquires the input image data 116 and executes pre-processing 301. The pre-processing 301 is a processing performed before inputting the input image data 116 to the trained model 220, and is, for example, a processing of converting each data to a floating point number and normalizing the value to 0.0 to 1.0 when the input image data is a 14-bit integer value of a Bayer array. Next, the image restoration unit 115 executes model 302 using the trained model 220 with the data after pre-processing 301 as input. The image restoration unit 115 executes post-processing 303 on the result of model execution 302 to obtain the output image 118. The post-processing 303 is a processing performed on the output of the trained model 220, and is, for example, a processing of converting the value to a 14-bit integer value when the output result data of the model is a floating point number.

[0035] FIG. 4 shows an example of the structure of the model. In this example, the model is composed of a plurality of convolution layers 401 and a concatenation layer 402. In the convolution layer 401, the convolution operation by the filter represented by the above-mentioned formula (1) and the nonlinear operation represented by formula (2) are repeated a plurality of times. The image restoration unit 115 sequentially applies the convolution layers 401 to the input data of the model to calculate a feature map. Next, the image restoration unit 115 connects the feature map and the input data in the channel direction by the concatenation layer 402. Furthermore, the image restoration unit 115 sequentially applies the convolution layers 401 to the concatenation result and outputs the result from the final layer. The process shown in FIG. 4 is a process generally performed by CNN, and a detailed description will be omitted. The image restoration unit 115 performs restoration inference of the input image data 116 using the trained model 220 received from the cloud server 200.

[0036] When a predetermined condition is satisfied, the switching unit 113 switches the trained model 220 on which the image restoration unit 115 executes inference from a first machine learning model to a second machine learning model by switching components of a network structure. Hereinafter, the switching process of the components of the network structure performed by the switching unit 113 may be simply referred to as "(model) switching." As the first machine learning model and the second machine learning model, for example, models as shown in Figs. 6A to 6E described later can be used, and the configurations of these models will be described later.

[0037] The cloud server 200 includes a degradation adding unit 211 and a learning unit 212. The degradation adding unit 211 generates a degraded learning image for input to be used for learning restoration inference. The degradation adding unit 211 can generate a degraded learning image by adding at least one type of degradation element to teacher image data extracted from a group of teacher images without degradation, for example. In the example of this embodiment, noise is used as the degradation element, and the degradation adding unit 211 adds noise as a degradation element to teacher image data to generate degraded learning image data. In this embodiment, the degradation adding unit 211 may generate input image data by analyzing the physical characteristics of the imaging device and adding noise corresponding to a wider range of degradation amount than the degradation amount that can occur in the imaging device as a degradation element to the teacher image data. Since the range of degradation amount that can occur varies depending on the individual difference of the imaging device, by adding a wider range of degradation amount than the analysis result, a margin can be provided and robustness can be increased. Here, the teacher image may be acquired from, for example, the imaging device 10, or may be acquired from an external device (not shown) via the Internet.

[0038] Next, the degradation addition will be described with reference to FIG. 5. The degradation addition unit 211 generates degraded learning image data 504 by adding 503 noise based on the physical property analysis result 218 of the imaging device as degradation element 502 to teacher image data 501 extracted from the teacher image group 217. The degradation addition unit 211 adds a pair of teacher image data 501 and degraded learning image data 504 to learning data 505. The degradation addition unit 211 generates a degraded learning image group consisting of multiple degraded learning image data 504 by adding the degradation element 502 to each teacher image data of the teacher image group 217, and the learning data 505 is generated from the generated degraded learning image group. Note that, although noise is given as an example in this embodiment, the degradation addition unit 211 may add any one of multiple types of degradation elements such as blur, aberration, compression, low resolution, loss, and contrast reduction caused by the weather at the time of shooting to the teacher image data, or a combination of multiple types of degradation elements.

[0039] The teacher image group 217 stores various types of image data, such as nature photos including landscapes and animals, portraits or sports photos, and man-made photos such as architecture and products. The teacher image data according to this embodiment is RAW image data in which each pixel has a pixel value corresponding to one of the colors of RGB, as with the input image data 116. The physical property analysis result 218 of the imaging device includes, for example, the amount of noise for each sensitivity generated by an imaging sensor built into a camera (imaging device), or the amount of aberration generated by a lens. By using these, it is possible to estimate the degree of image quality degradation that occurs for each shooting condition. In other words, by adding the degradation estimated under a certain shooting condition to the teacher image data, it is possible to generate an image equivalent to the image obtained during shooting.

[0040] Returning to the explanation of FIG. 2, the learning unit 212 includes an image restoration unit 214 for learning, an error calculation unit 215, and a model update unit 216. The learning unit 212 acquires learning parameters 219, and performs restoration learning using the learning data generated by the degradation adding unit 211. The learning parameters 219 include initial values ​​of parameters of the neural network model, and hyperparameters indicating the network structure and optimization method of the model. The image restoration unit 214 performs image restoration processing on the input image. The error calculation unit 215 calculates the error between the image restoration result image output by the image restoration unit 214 and the teacher image. The model update unit 216 updates the parameters of the neural network model of the image restoration unit 214 based on the calculated error.

[0041] FIG. 3B is a diagram showing the flow of the learning process performed by the learning unit 212. The image restoration unit 214 performs image restoration processing by model execution 305 using data obtained by performing preprocessing 310 on the input image data 308 as input. In the model execution 305, the image restoration unit 214 repeats convolution calculation and nonlinear calculation using filters represented by equations (1) and (2) multiple times on the input data of the model, and outputs the restoration result 313. Next, the error calculation unit 215 calculates the error between the data obtained by performing preprocessing 311 on the teacher image data 309 and the restoration result 313 by loss calculation 314. After that, the model update unit 216 executes model update 315 based on the error calculated by the error calculation unit 215, and updates the network parameters of the model so that the error is reduced (minimized).

[0042] 3 to 5 are merely examples, and other processes may be performed as long as inference and learning can be performed similarly using a general CNN. In this embodiment, the degraded learning image is generated by the degradation adding unit 211, but the learning image may be prepared by other means, such as by acquiring the learning image from an external device (not shown).

[0043] The configuration shown in Fig. 2 can be modified or changed as appropriate. For example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. The configuration shown in Fig. 2 may be realized by two or more devices. In this case, the devices are connected via a circuit or a wired or wireless network, and perform data communication with each other to perform cooperative operations, thereby realizing each process according to this embodiment.

[0044] The switching unit 113 according to this embodiment stores a plurality of candidate models for performing restoration inference, and when a predetermined condition is satisfied, switches the components of the network structure so that inference is performed by a model selected from the candidates. Figures 6A to 6E are schematic diagrams of the components of a network of such candidate models. Hereinafter, an example of a network structure of a trained model 220 executed by the image restoration unit 115 will be described with reference to Figures 6A to 6E.

[0045] In the examples of FIGS. 6A to 6E, input data 601 input to the network of the trained model is data obtained by pre-processing 301 of input image 116 at each time (represented by t). The input image 116 is a chronological sequence of multiple frames, and an example is shown in which output data (t=0) for an input image at t=0 is output. In addition, t is expressed as a negative value for a time relatively earlier than the image at t=0, and t is expressed as a positive value for a time relatively later. In FIGS. 6A to 6E, the output image at t=0 is output using five frames of input data from t=-2 to 2.

[0046] The network in FIG. 6A receives five frames of input data 601 and outputs one frame of output data 605. This network structure has a two-stage configuration, a first inference unit that performs a first inference process 602, and a second inference unit that performs a second inference process 604. First, the image restoration unit 115 performs the first inference process 602 for three sets (t=2 to 0, 1 to 1, 0 to 3) of three sets of chronologically consecutive input data 601 as one set. Next, the image restoration unit 115 inputs intermediate outputs 603, which are the outputs of the three sets of the first inference process 602, to the second inference unit, performs the second inference process 604, and outputs output data 605. In the network shown in FIG. 6A, the three sets of network parameters in the first inference process 602 are different from each other.

[0047] The network shown in Fig. 6B has a similar network structure to that shown in Fig. 6A, and is a network that similarly receives five frames of input data 601. In the network shown in Fig. 6B, three sets of network parameters in the first inference process 602 are common.

[0048] The network shown in FIG. 6C is a network having a common component in the network structure with the network shown in FIG. 6B. In the network shown in FIG. 6C, one of the outputs from the first inference process and an output output from a past first inference process are input to a second inference unit. In the example of FIG. 6C, only the last set of the three sets of first inference processes 602 shown in FIG. 6B is performed. Next, of the intermediate outputs 603 input to the second stage, two sets except the last set are made by using the inference results output in the past (for example, the previous and the time before last) first inference processes. That is, the image restoration unit 115 can store the intermediate output t=-1 to 1 obtained as the output of the first stage in the previous inference and the intermediate output t=-2 to 0 obtained as the output of the first stage in the time before last, and use these intermediate outputs.

[0049] The network shown in Fig. 6D is a network having only a part of the nodes in the network structure shown in Fig. 6B. In the network shown in Fig. 6D, the image restoration unit 115 inputs three frames of input data 601 (here, one set of t=-1 to 1) and performs a first inference process 602. Next, the image restoration unit 115 inputs an intermediate output 603, which is the output of the first inference process 602, to a second inference process 604 and outputs output data 605 for one frame.

[0050] 6A to 6D are one-stage networks having a first inference unit and a second inference unit, whereas the network shown in FIG. 6E is one-stage. In the network shown in FIG. 6E, the image restoration unit 115 inputs three frames of input data 601 (here, one set of t=-1 to 1), and outputs one frame of output data 605 by inference processing 602. In the example of FIG. 6, there are a plurality of input data 601, and one output data 605. However, this embodiment is not limited to this example, and the number of input data and the number of output data may be one or more, and the output data 605 may be an output corresponding to another time other than t=0 of the input data. In addition, although the input data 601 is chronologically continuous data in FIG. 6, it is not necessary that the input data 601 is chronologically continuous data.

[0051] The five types of networks described in FIG. 6 differ in processing time and amount of resources required for processing. Below, these network structures will be compared with reference to FIG. 7. In the table shown in FIG. 7, models (A) to (E) are defined as corresponding to FIGS. 6A to 6E, respectively, and an evaluation of the characteristics of these models for each comparison item (or simply "item") is given. Hereinafter, when "models (A) to (E)" are written, this refers to the models shown in FIG. 6 and FIG. 7.

[0052] In this example, for each item of models (A) to (E), ◯ is marked if it is relatively good compared to other models, △ is marked if it is medium, and × is marked if it is bad. Here, the items used are "(inference) accuracy," "memory usage," "learning time," "inference calculation load," and "dependency on previous inference." In the network structure shown in FIG. 6A, the parameters of the first inference process 602 are different. Therefore, the configuration of FIG. 6A has a higher expressive power and higher accuracy than the configuration of FIG. 6B. On the other hand, the configuration of FIG. 6A has a larger number of parameters than the configuration of FIG. 6B, so the memory usage is larger, the learning time is longer, and the inference calculation load is higher. Thus, each network configuration of FIG. 6A to FIG. 6E has relative advantages and disadvantages compared to other network configurations, and the evaluation is performed so that there is no model in FIG. 7 that has all items marked as ◯.

[0053] By switching between models having different performance as shown in FIG. 7, it is possible to use an appropriate model depending on the resource usage state of the CPU, GPU, etc. For example, it is possible to appropriately switch between a model with high accuracy but high load and a model with low load but relatively low accuracy. Here, the information processing device 100 according to the present embodiment does not initialize the model when switching the model, but changes the components of the network structure of the model (for example, parameters between nodes or part of the path), thereby making it possible to switch the model at a higher speed. The switching process will be described later.

[0054] Here, the dependency on the previous inference, which is one of the comparison items in FIG. 7, will be described with reference to FIG. 8. In this embodiment, the dependency on the previous inference is information indicating whether the previous inference and the current inference can be executed independently. The configuration in FIG. 6C reuses intermediate outputs from the previous and previous-previous inferences, so there is a dependency on the previous inference (i.e., the current inference cannot be performed without using the results of the previous and previous-previous inferences).

[0055] FIG. 8 is a diagram showing an example of which process is performed at what timing when the model of FIG. 6C is executed using multiple GPUs. In FIG. 8(A), a first inference process 602 and a second inference process 604 are alternately executed using GPU1 and GPU2. In the example of FIG. 8(A), the input data 601 at times t=0 and t=2 is image restored by GPU1, and the input data 601 at times t=1 and t=3 is image restored by GPU2. Here, another GPU process 801 is inserted between the image restoration process at time t=0 and the image restoration process at time t=2 in GPU1, causing a waiting time in the image restoration process at t=2. Due to the influence of this waiting time, the start of the first inference process 602 at t=2 is delayed, and the transfer of the intermediate output 603 to GPU2 is accordingly delayed. As a result, the start of the second inference process 604 at t=3 in GPU2 is delayed. 8A, even if no other GPU processing 801 occurs in GPU2, a delay in the processing of GPU1 causes a waiting time in GPU2, deteriorating the processing efficiency of GPU2. Similarly, if another GPU processing 801 occurs in GPU2, the transfer of the intermediate output 603 to GPU1 becomes slow, causing a waiting time in GPU1, deteriorating the processing efficiency of GPU1. In other words, GPU1 and GPU2 are affected by the delay in both directions.

[0056] Fig. 8(B) is a diagram showing an example of processing in each GPU when the influence of delay in Fig. 8(A) is changed from two-way to one-way. In Fig. 8(B), a first inference process 602 is executed by GPU1, and a second inference process 604 is executed by GPU2. Since the second inference process 604 requires an intermediate output 603 which is an output of the first inference process 602, the intermediate output 603 is transferred from GPU1 to GPU2. This transfer of the intermediate output 603 is executed between the first inference process 602 and the first inference process 604 for the input data 601 at the same time (e.g., t=0). That is, when executing a model, the inter-GPU transfer time of the intermediate output 603 is added to the model execution time.

[0057] In addition, as shown in FIG. 8B, when the first inference process 602 and the second inference process 604 are executed by different GPUs, if the processing times of these inference processes are different, the idle time during which the other GPU does not perform processing becomes long. In the example of FIG. 8B, the processing time of the second inference process 604 is longer than that of the first inference process 602, so the idle time of the GPU1 performing the first inference process 602 is long, and processing efficiency becomes poor. In this way, since delays may occur when performing inference using multiple GPUs, it may be preferable to use a model that does not depend on the previous inference. Therefore, in such a case, the switching unit 113 may switch the model used for inference from a model that depends on the previous inference to a model that does not depend on the previous inference.

[0058] The model switching process performed by the switching unit 113 will be described below. As described above, the information processing device 100 according to this embodiment switches between models having different performance as shown in FIG. 6 depending on whether a predetermined condition (switching condition) is satisfied. The switching unit 113 periodically judges whether the switching condition is satisfied (model switching judgment), and when it judges that the condition is satisfied (switching is necessary), switches the model executed by the image restoration unit 115. Here, the frequency of the model switching judgment can be set to, for example, every fixed time such as every 10 minutes, or every inference. In this embodiment, information indicating each model as shown in FIG. 6A to FIG. 6E is stored in the model storage unit 119, and when switching the model, the information is read from the model storage unit 119.

[0059] In this embodiment, the switching condition may be, for example, whether the operating state of the information processing device 100 is in a predetermined state. As described in FIG. 7, each of the above-mentioned models requires a different amount of resources for inference. Therefore, the switching unit 113 may use the resource usage state as the operating state of the information processing device 100, and switch the model used for inference when the resource usage state is in a predetermined state. For example, the switching unit 113 may switch the model used for inference to a model consuming less resources during inference when the resource usage rate during inference exceeds a predetermined rate (for example, 90%). Also, for example, the switching unit 113 may switch the model used for inference to a model consuming more resources during inference (having higher inference accuracy) when the resource usage rate during inference is less than a predetermined rate (for example, 30%) as the resource usage state. In this way, the switching condition used by the switching unit 113 according to this embodiment may be set according to, for example, the resource usage rate of the information processing device 100, or the characteristics of other models as described in each item shown in FIG. 7. Hereinafter, the description will be given using the usage rate as the resource usage state.

[0060] Here, the switching unit 113 acquires the resource usage rate (computation resource usage state) during inference of the information processing device 100. For example, the resource usage rate can be a CPU usage rate, a RAM usage rate, a GPU computation unit usage rate, or a GPU memory usage rate. Here, for example, an API (Application Programming Interface) for acquiring the CPU usage rate, the RAM usage rate, the GPU computation unit usage rate, or the GPU memory usage rate provided by an OS (Operating System) is used to acquire the computation resource usage state. The switching unit 113 can determine whether or not a switching condition is satisfied based on the acquired resource usage rate. For example, when the resource usage rate such as the CPU usage rate, the RAM usage rate, the GPU computation unit usage rate, or the GPU memory usage rate exceeds a first threshold value (90%) while the model of (A) in FIG. 7 is being used, the switching unit 113 may switch the model used for inference to a model with a lower inference computation load (here, to any of (B) to (E) shown in FIG. 7, for example, (B)). Furthermore, when the resource usage rate is less than a second threshold (30%) while using the model (E) in FIG. 7, the switching unit 113 may switch the model used for inference to a model with higher inference accuracy (here, to any of (A) to (D) shown in FIG. 7, for example (B)). In this way, the switching unit 113 according to this embodiment switches the model used for inference based on the model characteristics (such as accuracy, memory usage, inference calculation load, or dependency on the previous inference) and the resource usage rate of the information processing device 100.

[0061] In this embodiment, the model is not reinitialized when switching the model. When initializing the network model used for inference, the image restoration unit 115 constructs a model having a network structure that can implement all registered models (for example, those shown in Figs. 6A to 6E) by switching the components of the network structure. In the model switching process according to this embodiment, the model area before switching is reused, and the first machine learning model is switched to the second machine learning model by switching the model parameters and paths, and the model reconstruction process is not performed.

[0062] For example, model switching from FIG. 6A to FIG. 6B is performed by changing the parameters of the first inference process 602 and the second inference process 604. In this switching, only the process of changing the parameters is performed, and the process of initializing (performing initial settings) and reconstructing the network structure of the trained model 220 is not performed. Also, model switching from FIG. 6B to FIG. 6C is performed by changing the parameters of the first inference process 602 and the second inference process 604, and by changing the paths between nodes. That is, although only one set of inputs is illustrated in FIG. 6C, the model area before switching is also used internally in the model of FIG. 6C, and the model remains constructed with a structure in which the first inference process 602 can be performed three times, as in FIG. 6B. However, in the model of FIG. 6C, that path is not used, and the path illustrated in FIG. 6C is used. Compared to the case where the network structure is initialized and then the model is reconstructed when switching models, the processing according to this embodiment makes it possible to speed up model switching by reusing the model area before switching models, thereby eliminating the need for model reconstruction processing.

[0063] [Processing flow of the entire system] Next, various processes performed in the information processing system according to this embodiment will be described with reference to Figs. 9A to 9C. Figs. 9A to 9C are flowcharts showing an example of processes performed in the information processing system according to this embodiment. Each functional unit shown in Fig. 2 is realized by CPU 101 or 201 executing an information processing computer program according to this embodiment. However, all or part of the functional units shown in Fig. 2 may be implemented in hardware. Below, a description will be given along with the flowchart in Fig. 9. In the following description, the symbol "S" means a processing step.

[0064] First, an example of the flow of restoration learning performed by the cloud server 200 will be described with reference to the flowchart of Fig. 9A. In S901, the cloud server 200 inputs a teacher image group 217 prepared in advance and a physical property analysis result 218 of the imaging device 10, such as the characteristics of the imaging sensor and the sensitivity at the time of shooting, the subject distance, the focal length and F value of the lens, and the exposure value, to the cloud server 200. Note that the teacher image data here is a RAW image of a Bayer array, and is obtained by capturing an image with the imaging device 10. However, the method of acquiring the teacher image group 217 is not particularly limited to this, and for example, images captured in advance may be stored in a HDD or the like and then uploaded.

[0065] In S902, the degradation adding unit 211 adds noise based on the physical property analysis result 218 of the imaging device as a degradation element to the teacher image data of the teacher image group 217 input in S901, thereby generating degraded learning image data. Here, the degradation adding unit 211 adds noise of an amount measured in advance based on the physical property analysis result 218 of the imaging device to the teacher image data in a preset order or in a random order.

[0066] In S903, the learning unit 212 acquires the learning parameters 219 to be applied to the restoration learning. As described above, the learning parameters include the initial values ​​of the parameters of the neural network model, and the hyperparameters indicating the network structure and optimization method of the model.

[0067] In S904, the image restoration unit 214 performs initial setting of the model using the received network parameters. In S905, the image restoration unit 214 performs restoration processing of the degraded training image data, using the degraded training image data as an input to the model.

[0068] In S906, the error calculation unit 215 calculates the error between the restoration result in S905 and the teacher image data according to the loss function shown in Equation (3). In S907, the model update unit 216 updates the network parameters of the model so that the error obtained in S906 becomes small (minimum).

[0069] In S908, the learning unit 212 determines whether or not to end the learning. For example, the learning unit 212 determines to end the learning when the number of updates of the network parameters reaches a predetermined number, or when a predetermined period of time has passed since the start of the learning. If the learning is to be ended, the process proceeds to S909, and if the learning is not to be ended, the process returns to S905. In S905 to S908 from the second time onwards, the learning is continued using unprocessed degraded learning image data and teacher image data. In S909, the learning unit 212 stores the learned model in the mass storage device 204 or the like.

[0070] Next, referring to the flowchart of Fig. 9B, a flow of an example of the restoration inference process performed in the information processing device 100 will be described. In S910, the image restoration unit 115 determines whether or not the initial settings of the network model used for inference have been completed. If the initial settings have been completed, the process proceeds to S913, and if not, the process proceeds to S911.

[0071] In S911, the model storage unit 119 acquires and stores the trained model 220 from the cloud server 200. In S912, the image restoration unit 115 performs initial setting of the model to be used for inference in order to start inference using the trained model 220 stored in the model storage unit 119.

[0072] In S913, the switching unit 113 performs a model switching process. This process will be described later with reference to Fig. 9C. In S914, the information processing device 100 acquires input image data 116, which is a RAW image of a Bayer array to be subjected to image restoration processing. The RAW image may be, for example, an image captured by the imaging device 10 and directly input, or an image captured in advance and stored in the large-capacity storage device 104 may be read out. The input image data 116 is acquired in the number required for image restoration in S915.

[0073] In S915, the image restoration unit 115 restores the input image data 116. The image data restored by the image restoration unit 115 in S915 is output as output image data 118, and the processing in Fig. 9B ends. In addition, when there is data to be reused in the next or subsequent inferences, such as intermediate output 603 of the model in Fig. 6C, the data is stored in the image restoration unit 115 and reused in the next or subsequent inferences.

[0074] Next, a description will be given of the flow of an example of model switching processing performed by switching unit 113 of information processing device 100, with reference to the flowchart of Fig. 9C. In S920, switching unit 113 acquires the usage state of resources.

[0075] In S921, the switching unit 113 acquires model characteristics of the trained model 220 stored in the model storage unit 119. The model characteristics are values ​​related to items such as accuracy and memory usage as exemplified in Fig. 7. In S922, the switching unit 113 selects a model to be used next for inference (a model to be switched to) from among the trained models 220 stored in the model storage unit 119, based on the resource usage state acquired in S920 and the model characteristics acquired in S921.

[0076] Hereinafter, the switching unit 113 switches the model when the resource usage rate exceeds the first threshold, for example, as described above. Here, the switching unit 113 may set a ranking among the models (A) to (E) for the item "inference calculation load" of the model characteristics, and switch to a model that is one rank lower than the currently used model when the resource usage rate exceeds the first threshold. The switching unit 113 may also switch to a model that is one rank higher than the currently used model when the resource usage rate is less than the second threshold. The switching unit 113 may also perform different processing based on the ranking, such as setting a third threshold that is greater than the first threshold in addition to the first threshold, and switching to a model that is two ranks lower than the currently used model when the resource usage rate exceeds the third threshold. Here, it is assumed that the currently used model is selected as the model to be switched to when the resource usage rate is equal to or less than the first threshold and equal to or greater than the second threshold.

[0077] Also, here, the model is switched once it is confirmed that the resource usage rate exceeds the first threshold. However, in order to prevent the frequency of model switching from becoming too high, the resource usage rate in a predetermined period may be evaluated multiple times, and based on the evaluation result, it may be determined whether or not to switch the model. For example, the switching unit 113 may acquire the model characteristics in S921 for a predetermined period (for example, a predetermined number of times at a predetermined interval), and switch the model if the period during which the model characteristics exceed the first threshold is equal to or greater than a predetermined ratio. Also, the switching unit 113 may hold information on the model characteristics acquired in S921, and switch the model if the model characteristics exceeding the first threshold in the most recent predetermined period are equal to or greater than a predetermined ratio. Here, the predetermined ratio may be 100% or 80%, and may be arbitrarily set by the user according to desired conditions.

[0078] In S923, the switching unit 113 determines whether the model selected in S922 is different from the model currently being used. If they are different, the process proceeds to S924, and if they match, the process ends. In S924, the switching unit 113 switches the model used by the image restoration unit 115 to the model selected in S922, and ends the process.

[0079] The above is the overall flow of the processing performed in the information processing system of this embodiment. According to this processing, when switching models, the model area before switching is reused and only parameters and paths are switched, eliminating the need for model reconstruction processing and enabling faster model switching.

[0080] In this embodiment, the learning data is generated in S902, but it may be generated later. Specifically, it may be configured to generate input image data corresponding to the teacher image data in the subsequent restoration learning. In addition, in this embodiment, the cloud server 200 has been described as performing learning from scratch using data of a group of teacher images prepared in advance, but it is also possible to obtain a machine learning model having trained network parameters and perform each process using the model.

[0081] In the present embodiment, the processing is performed using noise as a degradation factor, but similar processing can be performed when a degradation factor other than noise is used. For example, the degradation factor may be blur, aberration, compression, low resolution, loss, or contrast reduction due to the influence of fog, haze, snow, or rain at the time of shooting, as described above, or a combination of a plurality of these.

[0082] In the present embodiment, an example has been described in which the information processing device 100 performs restoration only from the input image data using a trained model, but parameters that assist image restoration may also be used. For example, the information processing device 100 may hold a lookup table that estimates in advance the degree of image quality degradation that occurs depending on shooting conditions such as the distance to the subject, the focal length, the sensor size, or the exposure, and may adjust the amount of restoration by referring to the lookup table during image restoration. That is, the inference unit 112 of the information processing device 100 may adjust the restoration strength of the degradation based on the shooting conditions when the image of the input image data was captured.

[0083] In this embodiment, an example using a machine learning model that performs image restoration processing has been described, but the task of the model is not limited to image restoration processing, and similar processing can be performed even when using a model that performs image recognition, etc. Furthermore, the machine learning algorithm according to this embodiment is not limited to a neural network, and other algorithms may be used.

[0084] [Embodiment 2] In the first embodiment, an example of an information processing device that performs model switching determination based on the resource usage state has been described. The information processing device 1000 according to the second embodiment acquires a user setting and performs model switching determination based on the acquired user setting. That is, the information processing device 1000 according to the present embodiment switches the model used for inference from a first model to a second model when the user setting satisfies a predetermined condition. Note that a description of the basic configuration of the information processing system and other contents common to the configuration listed in the first embodiment will be omitted, and the following description will focus on the differences from the first embodiment.

[0085] Fig. 10 is a block diagram showing the functional configuration of the entire information processing system according to embodiment 2. In the information processing system of Fig. 10, the functional units of the cloud server 200 according to embodiment 2 are similar to those of embodiment 1, and therefore the description thereof will be omitted. Also, an information processing device 1000 according to embodiment 2 has a setting unit 1001 and a setting storage unit 1002, and has the same configuration as the information processing device 100 in Fig. 2 except that it has a switching unit 1003 instead of the switching unit 113, and therefore the duplicated description will be omitted.

[0086] The setting unit 1001 receives input of user settings related to inference via the input device 20 or the network I / F 106, and stores the input user settings in the setting storage unit 1002. The switching unit 1003 reads the user settings from the setting storage unit 1002, and switches the trained model 220 to be executed by the image restoration unit 115 in response to the user settings satisfying predetermined conditions.

[0087] The user settings include settings related to inference, such as a setting to prioritize speed or accuracy in inference, a setting of image quality in restoration, a setting of the degree of restoration, a setting of whether the input degraded image is captured indoors or outdoors, etc. Hereinafter, the processing by the information processing device 1000 in response to these user settings will be described with reference to FIG.

[0088] Fig. 11 is an example of a setting screen for acquiring user settings. This setting screen is displayed on the display device 40, and values ​​are selected via the input device 20. 1100 in Fig. 11(A) is an example of an image restoration setting screen, and current user settings for each item are displayed. In Fig. 11(A), "Accuracy / Speed", "Restoration Strength", "Camera Model", "Color Vividness", "Sharpness of Edges", and "Indoor / Outdoor" are displayed as example items for image restoration setting, and the user inputs settings for each setting item.

[0089] The "Accuracy vs. Speed" setting item specifies whether to prioritize speed or accuracy in inference. If a setting is made to prioritize speed, the switching unit 1003 will prioritize a model with a low inference calculation load (for example, a model one rank lower than the model currently being used in the "Inference Calculation Load" item).

[0090] In the "restoration strength" setting item, the degree of restoration is set. Here, the restoration strength is input in five stages from 1 to 5, and the model to be switched to is selected according to the input value. For example, when the value of the restoration strength exceeds a predetermined threshold (for example, 4), the switching unit 1003 may switch to a model with a restoration strength higher than the model currently being used, or may select a model corresponding to the value of the restoration strength as the model to be used for inference.

[0091] In the "indoor / outdoor" setting item, whether the input image is captured indoors or outdoors is set. For example, the switching unit 1003 may classify the stored models into models that are good at restoring indoor images and models that are good at restoring outdoor images, and when the "indoor / outdoor" setting is changed, the model may be switched to correspond to the setting.

[0092] In addition, when a model suitable for the camera model is set and a specific camera model is set in the "camera model" item, the switching unit 1003 refers to the setting and selects a model that performs restoration suitable for images captured by that camera model.

[0093] Screen 1120 shown in Fig. 11(B) is an example of a screen for setting the priority order of setting items. By determining the order of priority among the setting items, the switching unit 1003 can perform model switching judgment using the setting item with the highest priority order. On screen 1120, the camera model setting is given priority, and the outline clarity setting is not given priority. The values ​​set on the screens of Fig. 11(A) and Fig. 11(B) are stored in the setting storage unit 1002.

[0094] 11, an example in which the user settings are input using the display device 40 and the input device 20 has been described, but the settings may be input by other methods. For example, the user settings may be input via the network I / F 106.

[0095] [Processing flow of the entire system] Next, various processes performed in the information processing system according to this embodiment will be described with reference to Fig. 12A to Fig. 12B. Fig. 12A to Fig. 12B are flowcharts showing an example of processes performed in the information processing system according to this embodiment. Each functional unit shown in Fig. 10 is realized by the CPU 101 or 201 executing the information processing computer program according to this embodiment.

[0096] Fig. 12A is a flowchart showing an example of the restoration inference processing performed by the information processing device 1000. The processing shown in Fig. 12A is a model switching processing performed in S913, and is performed in the same manner as in Fig. 9C of the first embodiment except that S1200 and S1201 are performed instead of S920 and S922, respectively, and therefore a duplicated description will be omitted.

[0097] In S1200, the switching unit 1003 acquires the user settings stored in the setting storage unit 1002. The acquired user settings are, for example, the image restoration settings and the priority order of the image restoration settings illustrated in FIG.

[0098] In S1201, the switching unit 1003 selects a model to switch to from among the trained models 220 stored in the model storage unit 119, based on the settings acquired in S1200 and the model characteristics acquired in S921. Here, the model is selected using the settings of the item with the highest (e.g., highest) priority in the image restoration settings. For example, as shown in FIG. 11(B), when the camera model setting has the highest priority, a model suitable for the model set as the camera model is preferentially selected.

[0099] 12B is a flowchart showing an example of a setting process of user settings performed by the setting unit 1001 according to this embodiment. In S1220, the setting unit 1001 displays a menu screen on the display device 40. This menu screen is a screen that displays a menu for opening the screen of FIG. 11, for example, and displays an image restoration setting start button and an image restoration setting priority setting start button. Hereinafter, "a button is pressed" means that a user performs a button pressing operation, which may be an operation on a touch panel or an operation via a mechanical button.

[0100] In S1221, the setting unit 1001 determines whether or not the image restoration setting start button has been pressed. If it has been pressed, the process proceeds to S1222, and if it has not been pressed, the process proceeds to S1225. In S1222, the setting unit 1001 reads the current settings from the setting storage unit 1002, and displays an image restoration setting screen on the display device 40, for example, as shown in FIG. 11(A).

[0101] In S1223, the setting unit 1001 determines whether or not the setting completion button has been pressed by the user. If the setting completion button has been pressed, the process proceeds to S1224, and if not, the process returns to S1223. In S1224, the setting unit 1001 saves the setting in the setting saving unit 1002, and the process returns to S1220. In S1225, the setting unit 1001 determines whether the priority setting start button for image restoration settings has been pressed. If it has been pressed, the process proceeds to S1226, and if it has not been pressed, the process proceeds to S1227. In S1226, the setting unit 1001 reads the current settings from the setting storage unit 1002, and displays a priority setting screen for image restoration settings on the display device 40, for example, as shown in Fig. 11(B). In S1227, the setting unit 1001 returns to S1221 and repeats the process unless there is a request to stop.

[0102] The above is the overall flow of the processing performed by the information processing system of this embodiment. According to this processing, it is possible to acquire user settings and determine whether or not to switch the model used for inference depending on the acquired user settings.

[0103] Although the user settings according to the present embodiment have been described as being input via the input device 20, some or all of the user settings may be set automatically. For example, the setting items of "camera model" or "indoor / outdoor" may be automatically acquired and set from information added to the input image data. Also, the setting item of "restoration strength" may be automatically set based on the amount of noise obtained by analyzing the input image data.

[0104] [Embodiment 3] In the first embodiment, a process using a machine learning model that performs image restoration processing has been described. In the third embodiment, an example of an information processing device using a machine learning model that performs class classification processing to determine the type of object in an image, instead of the machine learning model that performs image restoration processing in the first embodiment, will be described. Note that a description of the basic configuration of the information processing system and other contents that are common to the configuration given in the first embodiment will be omitted, and the following description will focus on the differences from the first embodiment. For example, Random forest is used as a machine learning algorithm in the class classification in this embodiment. Random forest is a machine learning technology based on a decision tree, and is a known technology, so a description here will be omitted.

[0105] Fig. 13 is a block diagram showing the functional configuration of the entire information processing system according to embodiment 3. Moreover, the information processing device 1300 according to embodiment 3 has the same configuration as the information processing device 100 in Fig. 2 except that it includes an inference unit 1301 having a class classification unit 1302 for inference instead of the image restoration unit 115, and therefore a duplicated description will be omitted.

[0106] The model storage unit 119 stores the trained model 220 received from the cloud server 1350. The inference unit 1301 has a class classification function that determines the type of an object based on the feature amount of the object in an image.

[0107] The inference classifier 1302 includes a model execution unit of a decision tree for classifier. The classifier 1302 receives object features 1303 in an image as an input, and outputs a classifier result 1304 that is a result of classifying the object. The classes classified in the classifier according to this embodiment are types of objects, such as humans, ships, or mountains, and the classifier result 1304 is the likelihood of each class. The object features 1303 are features extracted from an image in which an object is captured, and in this embodiment, the features are extracted outside the information processing device 1300, but the information processing device 1300 may acquire an image and extract the object features 1303.

[0108] The cloud server 1350 includes a learning unit 1354. The learning unit 1354 includes a class classification unit 1353 for learning, an error calculation unit 1355, and a model update unit 1356. The learning unit 1354 acquires learning parameters 1352 and performs class classification learning using the learning data 1351. The learning data 1351 according to the present embodiment includes a plurality of pairs of object features and teacher data representing object classes. The learning parameters 1352 include parameters such as the tree depth of the decision tree model and a method of generating random numbers. The class classification unit 1353 receives the object features included in the learning data 1351 as input, and outputs a class classification result. The error calculation unit 1355 calculates the error between the class classification result and the teacher data. The model update unit 1356 updates the parameters of the decision tree model of the class classification unit 1353 based on the calculated error.

[0109] Here, an example of the structure of the trained model 220 executed by the classifier 1302 will be described with reference to Figs. 14(A) to (C). Fig. 14 is a schematic diagram of a network of a model that performs class classification using a random forest. The network structure of the model that uses a random forest is composed of multiple decision trees. The models shown in Figs. 14(A) to (C) use any or all of decision trees from decision tree 1 to decision tree 4.

[0110] The model in Figure 14(A) uses all decision trees, and has higher class classification accuracy but a higher processing load compared to the models in Figures 14(B) and 14(C). On the other hand, the models in Figures 14(B) and 14(C) use fewer decision trees than Figure 14(A), so the processing is relatively lighter, but the class classification accuracy is relatively low.

[0111] In this embodiment, a model having a network structure that can be implemented by changing the components of the network structure is constructed during initial setting of the classifier 1302. In the model iteration process according to this embodiment, similarly to the first embodiment, a process of switching the first machine learning model to the second machine learning model is performed by switching the parameters and paths of the model, and a process of reconstructing the model is not performed.

[0112] Next, various processes performed in the information processing system according to this embodiment will be described with reference to Fig. 15. Figs. 5(A) to 15B are flowcharts showing an example of processes performed in the information processing system according to this embodiment. Each functional unit shown in Fig. 13 is realized by the CPU 101 or 201 executing the information processing computer program according to this embodiment. Hereinafter, with reference to the flowchart in Fig. 15A, the flow of an example of class classification learning performed in the cloud server 1350 will be described.

[0113] In S1501, the cloud server 1350 acquires learning data 1351 to be applied to class classification learning. In S1502, the cloud server 1350 acquires learning parameters 1352 to be applied to class classification learning.

[0114] In S1503, the classifying unit 1353 performs initial setting of the model using the learning parameters acquired in S1502. In S1504, the classifying unit 1353 performs classifying of objects using the object feature amount included in the learning data 1351 as an input to the model.

[0115] In S1505, the error calculation unit 1355 calculates the error between the classification result and the training data according to the loss function. In S1506, the model update unit 1356 updates the model so that the error obtained in S1505 becomes small (minimum).

[0116] In S1507, the learning unit 1354 determines whether or not to end the learning. For example, the learning unit 1354 determines to end the learning when the number of times the model has been updated reaches a predetermined number of times, or when a predetermined period of time has passed since the start of the learning. If the learning is to be ended, the process proceeds to S1508, and if the learning is not to be ended, the process returns to S1504. In S1504 to S1507 from the second time onwards, the learning is continued using unprocessed object features and teacher data. In S1508, the learning unit 1354 stores the learned model in the mass storage device 204 or the like.

[0117] Next, an example of the flow of class classification performed by the information processing device 1300 will be described with reference to the flowchart in Fig. 15B. In S1550, the class classification unit 1302 determines whether or not the initial settings of the network model used for inference have been completed. If the initial settings have been completed, the process proceeds to S1553, and if not, the process proceeds to S1551.

[0118] In S1551, the model storage unit 119 acquires and stores the trained model 220 from the cloud server 1350. In S1552, the classifier unit 1302 performs initial setting of the model to be used for inference in order to start inference using the trained model 220 stored in the model storage unit 119.

[0119] In S1553, the switching unit 113 performs a model switching process. This process is similar to that described in FIG. 9C, and therefore will not be described here. In S1554, the information processing device 1300 acquires an object feature 1303 to be subjected to class classification. In S1555, the class classification unit 1302 performs class classification on the object feature 1303, and outputs a class classification result 1304.

[0120] The above is the overall flow of the processing performed in the information processing system of this embodiment. According to this processing, even when inference is performed using a model that performs class classification processing, the model area before switching is reused when switching models, and only parameters and paths are switched, eliminating the need for model reconstruction processing, and making it possible to speed up model switching. Note that the machine learning algorithm according to this embodiment is not limited to a decision tree, and other algorithms may be used.

[0121] The disclosure of this specification includes the following information processing device, information processing method, and program.

[0122] (Item 1) inference means for performing inference on an input using a first machine learning model; a selection means for selecting a second machine learning model, in which at least a part of a configuration part of a network structure of the first machine learning model is changed, as a machine learning model to be used for the inference in response to a predetermined condition being satisfied; Equipped with the first machine learning model has a first component and a second component; The information processing device, characterized in that the second machine learning model has at least a component part in which a parameter or a part of a path between nodes in the first component part or the second component part is changed. (Item 2) The first machine learning model includes a first inference unit having a first parameter and a second parameter different from the first parameter as the first component, and a second inference unit having an output of the first inference unit as the second component, The information processing device described in item 1, characterized in that the second machine learning model comprises a third inference unit that replaces the second parameters in the first inference unit with the first parameters, and a fourth inference unit that has the same network structure as the second inference unit and receives an output from the third inference unit as an input. (Item 3) The first machine learning model includes a first inference unit as the first component that outputs a first output by a first inference process and a second output by a second inference process, and a second inference unit as the second component that receives the first output and the second output as inputs; The information processing device described in item 1, characterized in that the second machine learning model includes a fifth inference unit that outputs a first output by a first inference process, and a sixth inference unit that receives as input the first output and the second output that was output in the past. (Item 4) 4. The information processing device according to any one of items 1 to 3, wherein the predetermined condition is satisfied when an operating state of the information processing device is a predetermined state. (Item 5) 5. The information processing device according to item 4, wherein the operating state includes a resource usage state in the inference. (Item 6) The information processing device described in item 5, characterized in that the selection means switches the machine learning model used for the inference to the second machine learning model, which is a machine learning model that consumes less resources during inference than the first machine learning model, when the period during which the resource usage rate in a specified period is higher than a specified threshold is equal to or greater than a specified percentage. (Item 7) Item 6. The information processing device according to item 5, wherein the selection means selects the second machine learning model having a smaller number of layers than the first machine learning model when a period during which the resource usage rate in a predetermined period is higher than a predetermined threshold is equal to or greater than a predetermined percentage. (Item 8) The inference setting unit further includes an acquisition unit for acquiring a user setting related to the inference. 8. The information processing device according to any one of items 1 to 7, wherein the predetermined condition is satisfied when the user setting is in a predetermined state. (Item 9) the acquiring means acquires, as the user setting, a setting as to whether speed or accuracy is to be prioritized in the inference; Item 9. The information processing device according to item 8, wherein the selection means switches the machine learning model used in the inference to the second machine learning model, which is a machine learning model having a faster inference speed than the first machine learning model, when it is set to prioritize speed in the inference, and switches the machine learning model used in the inference to the second machine learning model, which is a machine learning model having a higher inference accuracy than the first machine learning model, when it is set to prioritize accuracy in the inference. (Item 10) The inference is a process of restoring an input degraded image, The acquisition means acquires, as a user setting, a setting of image quality in the restoration, Item 9. The information processing device according to item 8, wherein the selection means switches the machine learning model used for the inference to the second machine learning model when the image quality setting in the restoration has been changed and the predetermined condition is satisfied. (Item 11) The inference is a process of restoring an input degraded image, The acquiring means acquires a setting of the degree of restoration as a user setting, Item 9. The information processing device according to item 8, wherein the selection means switches the machine learning model used for the inference to the second machine learning model when the setting of the degree of restoration has been changed and the predetermined condition is satisfied. (Item 12) The inference is a process of restoring an input degraded image, The acquiring means acquires, as a user setting, a setting as to whether the input degraded image has been captured indoors or outdoors, The information processing device according to item 8, wherein the selection means determines that the predetermined condition is satisfied when a setting is changed to indicate whether the input degraded image is captured indoors or outdoors, and switches the machine learning model used for the inference to the second machine learning model. (Item 13) The inference is a process of restoring an input degraded image, The acquiring means acquires, as a user setting, a setting related to an imaging device that captured the input degraded image, Item 9. The information processing device according to item 8, wherein the selection means switches the machine learning model used for the inference to the second machine learning model when the input degraded image is captured by a predetermined imaging device and the predetermined condition is satisfied. (Item 14) 14. The information processing device according to any one of items 1 to 13, wherein the inference is a process of restoring an input degraded image. (Item 15) Item 15. The information processing device according to item 14, wherein the degraded image is an image degraded due to noise, compression, low resolution, blur, aberration, loss, or reduced contrast. (Item 16) 14. The information processing device according to any one of items 1 to 13, wherein the inference is a process of classifying a subject in an input image. (Item 17) performing inference on an input using a first machine learning model; selecting a second machine learning model, in which at least a part of a configuration part of a network structure of the first machine learning model is changed, as the machine learning model to be used for the inference in response to a predetermined condition being satisfied; Equipped with the first machine learning model has a first component and a second component; An information processing method, characterized in that the second machine learning model has at least a component part in which a parameter or a part of a path between nodes in the first component part or the second component part is changed. (Item 18) A program for causing a computer to function as each of the means of the information processing device according to any one of items 1 to 16.

[0123] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0124] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0125] 100: information processing device, 101: CPU, 102: RAM, 103: ROM, 104: mass storage device, 105: general-purpose interface, 106: network interface, 107: system bus

Claims

1. inference means for performing inference on an input using a first machine learning model; a selection means for selecting a second machine learning model, in which at least a part of a configuration part of a network structure of the first machine learning model is changed, as a machine learning model to be used for the inference in response to a predetermined condition being satisfied; Equipped with The first machine learning model has a first component and a second component; An information processing device, characterized in that the second machine learning model has at least a component part in which a parameter or a path between nodes in the first component part or the second component part is changed.

2. The first machine learning model includes a first inference unit having a first parameter and a second parameter different from the first parameter as the first component, and a second inference unit having an output of the first inference unit as the second component, 2. The information processing device according to claim 1, wherein the second machine learning model comprises: a third inference unit that replaces the second parameters in the first inference unit with the first parameters; and a fourth inference unit that has the same network structure as the second inference unit and receives an output from the third inference unit as an input.

3. The first machine learning model includes a first inference unit as the first component that outputs a first output by a first inference process and a second output by a second inference process, and a second inference unit as the second component that receives the first output and the second output as inputs; 2. The information processing device according to claim 1, wherein the second machine learning model comprises a fifth inference unit that outputs a first output by a first inference process, and a sixth inference unit that receives as input the first output and the second output that was previously output.

4. The information processing device according to claim 1 , wherein the predetermined condition is satisfied when the operating state of the information processing device is a predetermined state.

5. The information processing apparatus according to claim 4 , wherein the operation state includes a usage state of resources in the inference.

6. The information processing device according to claim 5, characterized in that the selection means switches the machine learning model used for the inference to the second machine learning model, which is a machine learning model that consumes less resources during inference than the first machine learning model, when the period during which the resource usage rate in a specified period is higher than a specified threshold is equal to or greater than a specified percentage.

7. The information processing device according to claim 5, characterized in that the selection means selects the second machine learning model having a smaller number of layers than the first machine learning model when a period during which the resource usage rate in a specified period is higher than a specified threshold is equal to or greater than a specified percentage.

8. The inference setting unit further includes an acquisition unit for acquiring a user setting related to the inference. The information processing apparatus according to claim 1 , wherein the predetermined condition is satisfied when the user setting is in a predetermined state.

9. the acquiring means acquires, as the user setting, a setting as to whether speed or accuracy is to be prioritized in the inference; 9. The information processing device according to claim 8, wherein the selection means switches the machine learning model used for the inference to the second machine learning model, which is a machine learning model having a faster inference speed than the first machine learning model, when speed is set to be prioritized in the inference, and switches the machine learning model used for the inference to the second machine learning model, which is a machine learning model having a higher inference accuracy than the first machine learning model, when accuracy is set to be prioritized in the inference.

10. The inference is a process of restoring an input degraded image, The acquisition means acquires, as a user setting, an image quality setting for the restoration, The information processing device according to claim 8 , wherein the selection means switches the machine learning model used for the inference to the second machine learning model when the image quality setting for the restoration has been changed and the predetermined condition is satisfied.

11. The inference is a process of restoring an input degraded image, The acquisition means acquires a setting of the degree of restoration as a user setting, The information processing device according to claim 8 , wherein the selection means switches the machine learning model used for the inference to the second machine learning model when the setting of the degree of restoration has been changed and the predetermined condition is satisfied.

12. The inference is a process of restoring an input degraded image, The acquiring means acquires, as a user setting, a setting as to whether the input degraded image has been captured indoors or outdoors, The information processing device according to claim 8, wherein the selection means determines that the specified condition is satisfied when a setting has been changed to indicate whether the input degraded image has been captured indoors or outdoors, and switches the machine learning model used for the inference to the second machine learning model.

13. The inference is a process of restoring an input degraded image, The acquiring means acquires, as a user setting, a setting related to an imaging device that captured the input degraded image, The information processing device according to claim 8 , characterized in that the selection means switches the machine learning model used for the inference to the second machine learning model when the input degraded image is captured by a predetermined imaging device and the predetermined condition is satisfied.

14. The information processing apparatus according to claim 1 , wherein the inference is a process of restoring an input degraded image.

15. The information processing apparatus according to claim 14 , wherein the degraded image is an image degraded due to noise, compression, low resolution, blur, aberration, loss, or reduced contrast.

16. The information processing apparatus according to claim 1 , wherein the inference is a process of classifying objects in an input image.

17. performing inference on an input using a first machine learning model; selecting a second machine learning model, in which at least a part of a configuration part of a network structure of the first machine learning model is changed, as the machine learning model to be used for the inference in response to a predetermined condition being satisfied; Equipped with The first machine learning model has a first component and a second component; An information processing method, characterized in that the second machine learning model has at least a component part in which a part of a parameter or path between nodes in the first component part or the second component part is changed.

18. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 16.

Images