Information processing apparatus and program

Abstract

To perform inference about images in a more suitable manner even in a situation where data of the images to be inferred may have various sizes.SOLUTION: A setting unit 121 sets, in an input image, a ROI to be inferred by a model that is built on the basis of machine learning. A model determination unit 113 determines, according to the size of the ROI set in the input image, a model to be applied to inference using the input image as input data, from a plurality of models different from each other in the size of input data and subjected to initialization to enable execution of inference about the input data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and a program. [Background technology]

[0002] In recent years, learning models (hereinafter, simply referred to as models) constructed based on machine learning have been applied to various image processing and image recognition application programs. One example of image processing using a model is noise removal processing that reduces (and eventually removes) the effects of degradation such as noise that manifests itself in a target image. Another example of image recognition using a model is object detection processing that detects a desired object captured as a subject in an image. Some of the models described above have different sizes of image data that can be accepted as input. In light of this, Patent Document 1 proposes a technology in which the size of the target image data is changed to a size of image data that can be accepted as input by the object detection model before being input. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-71757 Summary of the Invention [Problem to be solved by the invention]

[0004] On the other hand, as in the technology disclosed in Patent Document 1, in a situation where the size of the input image data is changed, the characteristics of the image (e.g., the characteristics of noise manifested in the image) may change, and the accuracy of the image processing using the model (e.g., noise removal accuracy) may decrease. To address this issue, a method of applying a model capable of accepting the target image data as input may be considered. However, it may take time to initialize the model so that it can perform inference on the image data (e.g., image processing such as noise removal), and a situation may be envisioned in which the time related to this initialization becomes apparent as a waiting time.

[0005] In view of the above problems, the present invention realizes inference on images in a more suitable manner even in a situation where image data of various sizes may be the target. [Means for solving the problem]

[0006] The information processing device of the present invention is characterized in having a setting means for setting an inference target area in an input image that is to be subject to inference by a model constructed based on machine learning, and a determination means for determining a model to be applied to inference using the input image as input data from a plurality of models having different input data sizes that have been initialized to make the model capable of performing inference on input data, depending on the size of the inference target area set by the setting means. Effect of the Invention

[0007] According to the present invention, even in a situation where image data of various sizes may be the target, it becomes possible to realize inference on the image in a more suitable manner. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of a system configuration of an image processing system. [Diagram 2] FIG. 1 is a diagram illustrating an example of a functional configuration of an image processing system. [Diagram 3]FIG. 2 is a diagram showing an example of a processing flow of the image processing system. [Figure 4] FIG. 13 is a diagram showing an example of a model structure. [Diagram 5] FIG. 11 is a diagram showing an example of a process of adding degradation to an image. [Figure 6] FIG. 1 is a diagram showing an outline of an inference process in which an image is input. [Figure 7] FIG. 11 is a diagram showing an example of a method for determining a model. [Figure 8] 11 is a flowchart showing an example of processing of the image processing system. [Figure 9] FIG. 1 illustrates an example of a functional configuration of an image processing system. [Figure 10] FIG. 13 is a diagram showing an example of a setting screen of the image processing system. [Figure 11] FIG. 11 is a diagram showing an example of processing related to automatic setting of an ROI. [Figure 12] 11 is a flowchart showing an example of processing of the image processing system. [Figure 13] FIG. 1 illustrates an example of a model network structure. [Figure 14] FIG. 13 is a diagram showing an example of a setting screen of the image processing system. [Figure 15] 11 is a flowchart showing an example of processing of the image processing system. [Figure 16] FIG. 11 is a diagram showing an example of processing related to automatic setting of an ROI. [Figure 17] FIG. 13 is a diagram showing an example of a setting screen of the image processing system. [Figure 18] 11 is a flowchart showing an example of processing of the image processing system. [Figure 19] FIG. 13 is a diagram showing an example of a setting screen of the image processing system. [Figure 20] 11 is a flowchart showing an example of processing of the image processing system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0010] <CNNについて> First, an overview of a convolutional neural network (CNN) used in information processing techniques that apply deep learning in general, including information processing devices according to the embodiments of the present disclosure, will be described. CNN is a technique that repeatedly performs nonlinear operations after convolving a filter generated by training with image data. Data obtained by convolving a filter with image data and performing nonlinear operations thereon is also called a feature map. Furthermore, learning is performed using training data (training images or data sets) consisting of pairs of input image data and output image data. Simply put, learning is the generation of filter values ​​that can convert input image data to corresponding output image data with high accuracy from training data.

[0011] When image data has RGB color channels or when a feature map is composed of multiple pieces of image data, the filter used for convolution also has multiple channels corresponding to them. The process of performing nonlinear calculations after convolving a filter with image data (or feature map) is expressed in units of layers. Therefore, when explicitly indicating each feature map and filter corresponding to each layer, they are expressed as, for example, the nth layer feature map and the nth layer filter. In addition, for example, a CNN that repeats filter convolution and nonlinear calculations three times has a three-layer network structure. Such nonlinear calculation processing can be formulated as the relational equation shown below as Equation (1).

[0012]

number

[0013] In Equation (1), W n is the filter of the n-th layer, b n is the bias of the n-th layer, f is a non-linear operator, X n is the feature map of the n-th layer, and * is a convolution operator. Also, (1) attached as a superscript to each operator indicates that it is the first filter or feature map. The above-mentioned filters and biases are generated by learning and are collectively referred to as "network parameters". As the non-linear operation, for example, a sigmoid function or ReLU (Rectified Linear Unit) is used. For example, ReLU is represented by the conditional expression shown as Equation (2) below.

[0014]

Equation

[0015] As shown in Equation (2), zero is applied to the negative elements of the input vector X, and the original values are applied to the elements of 0 or more.

[0016] As networks using CNN, ResNet in the field of image recognition and its application RED-Net in the field of super-resolution are well-known. In any of these technologies, by stacking multiple layers of CNN and performing convolution of filters multiple times, high-precision processing is achieved. For example, ResNet features a network structure with a path that shortcuts the convolutional layer, thereby realizing a multi-layer network of 152 layers and achieving high-precision recognition approaching the human recognition rate. The reason why the processing is made more accurate by a multi-layer CNN is simply that by repeating the non-linear operation multiple times, a non-linear relationship between the input and output can be expressed.

[0017] <Learning of CNN> Next, we will explain CNN learning. CNN learning is generally performed by minimizing the objective function shown below as formula (3) for learning data consisting of a set of input image data and corresponding correct output image (teacher image) data.

[0018]

number

[0019] In equation (3), L is a loss function that measures the error between the correct answer and the estimated result. Y i is the i-th correct output image data, X i is the i-th input image data. F is a function that collectively represents the calculations (equation (1)) performed in each layer of CNN. θ is the network parameters (filter and bias). ||Z||2 is the L2 norm, or more simply, the square root of the sum of the squares of the elements of vector Z. n is the number of data sets in the training data. Since the number of training data is generally large, in the Stochastic Gradient Descent (SGD) method, a portion of the training data is randomly selected and used for training. In addition, various methods are known for minimizing (optimizing) the objective function, such as the momentum method, AdaGrad method, AdaDelta method, and Adam method. The Adam method is given by the relational equation shown in the following equation (4).

[0020]

number

[0021] In formula (4), t represents the tth iteration. i t is the i-th network parameter at the t-th iteration, and g is θ i tis the gradient of the loss function L with respect to m. m and v are moment vectors, α is the base learning rate, β1 and β2 are hyperparameters, and ε is a tiny constant to prevent division by zero. Note that there are no specific guidelines for selecting an optimization method in learning, so essentially any method can be used, but it is known that there are differences in convergence between methods, resulting in differences in learning time.

[0022] <First embodiment> A first embodiment of the present disclosure will be described. In this embodiment, an example of an information processing device (image processing device) that applies information processing (image processing) to a target image by using the above-mentioned CNN to reduce degradation that appears in the image will be described. Examples of image degradation factors include noise, blur, aberration, compression, low resolution, loss, and contrast reduction due to weather conditions such as fog, haze, snow, and rain at the time of image capture. Image processing for reducing degradation that is apparent in an image includes noise removal, blur removal, aberration correction, loss completion, correction of degradation due to compression, super-resolution processing for low-resolution images, and processing for correcting contrast reduction due to weather conditions at the time of image capture. The image degradation reduction process in this embodiment is a process for generating or restoring an image without degradation (or with very little degradation) from an image in which degradation is evident, and is also referred to as an image restoration process in the following description. That is, the image restoration process according to this embodiment is not limited to a process for restoring an image that was without degradation (or had very little degradation) itself but has been degraded by subsequent amplification, compression / expansion, other image processing, etc., but also includes a process for reducing degradation contained in the original image itself. For the sake of convenience, the following description will focus on an example in which noise is taken as an example of an image degradation factor and noise reduction processing is performed as image restoration processing.

[0023] (Image processing system configuration example) An example of a system configuration of an image processing system to which an image processing device according to this embodiment is applied will be described with reference to Fig. 1. The image processing system shown in Fig. 1 includes a cloud server 200 that generates learning data and learns restoration of image quality degradation, and an edge device 100 that performs degradation restoration for an image to be processed. The cloud server 200 and the edge device 100 are connected via a network such as the Internet. Hereinafter, the process of generating learning data and learning restoration of image quality degradation in the cloud server 200 is also referred to as degradation restoration learning, and the process of degradation restoration in the edge device 100 is also referred to as degradation restoration inference.

[0024] (Edge device hardware configuration) The edge device 100 according to this embodiment acquires RAW image data (Bayer array) input from the imaging device 10 as an input image to be subjected to image restoration processing. Then, the edge device 100 performs degradation restoration inference on the input image to be processed using a trained neural network model (a model constructed based on so-called machine learning) provided by the cloud server 200. In other words, the edge device 100 corresponds to an image processing device that reduces noise in RAW image data by using the model provided by the cloud server 200 and executing a pre-installed image processing application program.

[0025] The edge device 100 has a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, and a large-capacity storage device 104. The edge device 100 also has a general-purpose interface (I / F) 105 and a network I / F 106. The components of the edge device 100 are connected to each other via a system bus 107. The edge device 100 is also connected to an imaging device 10, an input device 20, an external storage device 30, and a display device 40 via the general-purpose I / F 105.

[0026] The CPU 101 uses the RAM 102 as a work memory, executes programs stored in the ROM 103, and performs overall control of each component of the edge device 100 via a system bus 107. The mass storage device 104 is realized by a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various data and image data handled by the edge device 100. The CPU 101 writes data to the mass storage device 104 via the system bus 107, and reads data stored in the mass storage device 104.

[0027] The general-purpose I / F 105 is, for example, a serial bus interface based on a standard such as USB, IEEE1394, or HDMI (registered trademark). The edge device 100 may acquire data from an external storage device 30 (for example, various storage media such as a memory card, a CF card, an SD card, and a USB memory) via the general-purpose I / F 105. The edge device 100 also accepts user instructions from an input device 20 such as a mouse or a keyboard via the general-purpose I / F 105. The edge device 100 also outputs image data processed by the CPU 101 to a display device 40 (for example, various image display devices such as a liquid crystal display) via the general-purpose I / F 105. The display device 40 may be a display device integrated with a touch panel. The edge device 100 may also acquire data of a captured image (RAW image) to be subjected to noise reduction processing from the imaging device 10 via the general-purpose I / F 105.

[0028] The network I / F 106 is an interface for connecting to a network such as the Internet. The edge device 100 accesses the cloud server 200 via, for example, an installed web browser, and acquires a trained model for degradation restoration inference.

[0029] (Cloud server hardware configuration) The cloud server 200 according to this embodiment is an information processing device (e.g., an image processing device) that provides a network service, such as a cloud service, on a network such as the Internet. More specifically, the cloud server 200 generates learning data and performs degradation restoration learning, and generates a trained model in which network parameters and a network structure as the learning result are stored. Then, the cloud server 200 provides the generated trained model in response to a request from the edge device 100.

[0030] The cloud server 200 includes a CPU 201, a ROM 202, a RAM 203, a mass storage device 204, and a network I / F 205. These components of the cloud server 200 are connected to each other via a system bus 206.

[0031] The CPU 201 reads out a control program stored in the ROM 202 and executes various processes to control the overall operation of the cloud server 200. The RAM 203 is used as a temporary storage area such as the main memory and work area of ​​the CPU 201. The mass storage device 204 is realized by a storage device (for example, a large-capacity secondary storage device) such as an HDD or SSD, and stores image data and various programs. The network I / F 205 is an interface for connecting to a network such as the Internet. In response to a request from the edge device 100 using a web browser or the like, the network I / F 205 provides the edge device 100 with a trained model in which the above-mentioned network parameters and network structure are stored.

[0032] It should be noted that the edge device 100 and the cloud server 200 may include other components in addition to the components described above, but detailed description of these other components will be omitted.

[0033] In this embodiment, it is assumed that the cloud server 200 downloads a trained model, which is the result of generating training data and performing degradation restoration training, to the edge device 100, and the edge device 100 performs degradation restoration inference on the input image data to be processed. The above-mentioned system configuration is merely an example, and does not limit the configuration of the image processing system according to the present embodiment. For example, the functions of the cloud server 200 may be subdivided, and the generation of learning data and degradation restoration learning may be performed by separate devices. As another example, a configuration may be applied in which the generation of learning data, degradation restoration learning, and degradation restoration inference are performed in a device (e.g., the imaging device 10) that has both the functions of the edge device 100 and the cloud server 200.

[0034] (System overall functional configuration) An example of the functional configuration of the image processing system according to this embodiment will be described with reference to FIG. First, a description will be given of an example of the configuration of the edge device 100. As shown in Fig. 2, the edge device 100 has an inference unit 112 and a model storage unit 119. A trained model 220 received from the cloud server 200 is stored in the model storage unit 119. The inference unit 112 has a function of image restoration processing to reduce image degradation. The inference unit 112 has a degradation restoration unit for inference 115, a model determination unit 113, and a model control unit 114. The degradation restoration unit for inference 115 also includes a model execution unit of a neural network for degradation restoration.

[0035] Here, an example of the processing of the inference degradation restoration unit 115 will be described with reference to FIG. The degradation restoration unit for inference 115 acquires input image data 116 and executes pre-processing 301 on the input image data 116. For example, the pre-processing 301 may include a process of converting the format of the input image data 116 into a data format that can be accepted as an input by a trained model to be applied to image processing. As a specific example, the pre-processing 301 may include a process of converting each data into a floating-point number and normalizing the value to 0.0 to 1.0 when the input image data is a 14-bit integer value of a Bayer array. Next, the degradation restoration unit for inference 115 performs model execution 302 using the trained model 220 with the data after preprocessing 301 as input. Then, the degradation restoration unit for inference 115 executes post-processing 303 on the result of model execution 302, and obtains the output image 118 as a result. For example, the post-processing 303 may include a process of converting the format of data output by the trained model applied to the image processing into a preset output data format (for example, the same format as the input image data 116). As a specific example, the post-processing 303 may include a process of converting the value into a 14-bit integer value when the output result data of the model is a floating-point number.

[0036] An example of the structure of a model applied to image processing of input image data will be described with reference to Fig. 4. In the example shown in Fig. 4, the model is composed of a plurality of convolution layers 401 and a concatenation layer 402. In the convolution layer 401, a convolution operation using a filter represented by the above-mentioned formula (1) and a nonlinear operation represented by formula (2) are repeatedly executed a plurality of times. The degradation restoration unit for inference 115 sequentially applies the convolutional layers 401 to the input data of the model to calculate a feature map. Next, the degradation restoration unit for inference 115 connects the feature map and the input data in the channel direction by the connection layer 402. Furthermore, the degradation restoration unit for inference 115 sequentially applies the convolutional layers 401 to the connection result, and outputs the result of the series of processes from the final layer.

[0037] 2 again, the degradation restoration unit for inference 115 performs degradation restoration inference for the input image data 116 using the trained model 220 received from the cloud server 200. The model control unit 114 switches the trained model 220 to be applied to the degradation restoration inference by the inference degradation restoration unit 115. At this time, the model to be switched to is determined by the model determination unit 113. The process of switching the trained model 220 by the model determination unit 113 and the model control unit 114 will be described in detail later.

[0038] Next, a description will be given of an example of the configuration of the cloud server 200. The cloud server 200 includes a degradation adding unit 211 and a learning unit 212. The degradation adding unit 211 adds at least one degradation element to teacher image data extracted from a group of undegraded teacher images to generate input image data. As described above, in this embodiment, a case where noise is applied as a degradation element will be described, so that the degradation adding unit 211 adds noise as a degradation element to teacher image data to generate input image data. At this time, for example, the degradation adding unit 211 may analyze the physical characteristics of the imaging device and add noise corresponding to a wider range of degradation amount than the degradation amount that can occur in the imaging device to the teacher image data as a degradation element to generate input image data. In this way, by adding degradation corresponding to a wider range of degradation amount than the analysis result, it is possible to provide a margin and increase robustness so that the influence of the individual difference can be absorbed even in a situation where the range of degradation amount varies depending on the individual difference of the imaging device.

[0039] Here, the degradation adding process for a target image (for example, a teacher image) will be described with reference to FIG. 5. The degradation adding unit 211 generates input image data 504 by adding noise based on the physical property analysis result 218 of the imaging device as a degradation element 502 to teacher image data 501 extracted from the teacher image group 217. For example, in the example shown in FIG. 5, the process related to adding the degradation element 502 to the teacher image data 501 is shown as an addition process 503 for convenience. Then, the degradation adding unit 211 adds a pair of the teacher image data 501 and the input image data 504 generated from the teacher image data 501 to the learning data 505. In this manner, the degradation adding unit 211 generates an input image group consisting of a plurality of input image data 504 by adding the degradation element 502 to each teacher image data included in the teacher image group 217, and generates learning data 505 from the input image group and the teacher image group. In this embodiment, noise is used as an example, but the degradation adding unit 211 may add any one or a combination of multiple types of degradation elements to the teacher image data, such as the blur, aberration, compression, low resolution, loss, and reduced contrast due to the weather at the time of shooting.

[0040] The teacher image group 217 may store various types of image data, such as nature photos including landscapes and animals, portraits or sports photos, and man-made photos such as architecture and products. In this embodiment, the teacher image data is RAW image data in which each pixel has a pixel value corresponding to one of the colors of RGB, like the input image data 116. The physical property analysis result 218 of the imaging device includes, for example, the amount of noise for each sensitivity generated by an imaging sensor built into a camera (imaging device), the amount of aberration generated by a lens, and the like. By using information included in such physical property analysis result 218, it is possible to estimate the degree of image degradation that may occur for each imaging condition. In other words, by adding the degradation estimated for a certain imaging condition to the teacher image data, it is possible to generate an image that is substantially equivalent to the image obtained during imaging.

[0041] 2 again, the learning unit 212 includes a learning degradation restoration unit 214, an error calculation unit 215, and a model update unit 216. The learning unit 212 acquires learning parameters 219 and performs degradation restoration learning using the learning data generated by the degradation adding unit 211. The learning parameters 219 include hyperparameters indicating initial values ​​of parameters of the neural network model, the network structure of the model, and an optimization method. The learning-purpose degradation restoration unit 214 generates a degradation-restored image by performing image restoration processing on the input image. The error calculation unit 215 calculates the error between the teacher image and the deteriorated restored image generated as a result of the deterioration restoration process by the learning deterioration restoration unit 214. The model update unit 216 updates the parameters of the neural network model applied to the learning degradation restoration unit 214 based on the error calculated by the error calculation unit 215 .

[0042] Now, with reference to FIG. 3(B), an example of the processing in learning section 212 will be described. The learning degradation restoration unit 214 receives data obtained by performing preprocessing 310 on input image data 308 as input for the model execution process 305 , and executes degradation restoration processing on the data as the model execution process 305 . In the model execution process 305, the learning degradation restoration unit 214 repeats convolution calculations and nonlinear calculations using filters expressed by equations (1) and (2) on the input data of the model multiple times, and outputs the results as restoration results 313. Next, the error calculation unit 215 receives the data obtained by performing preprocessing 311 on the teacher image data 309 and the restoration result 313 as input to a loss calculation process 314, and executes the loss calculation process 314 to calculate the error therebetween. Thereafter, the model update unit 216 executes a model update process 315 based on the error calculated by the error calculation unit 215, thereby updating the network parameters of the model so that the error becomes smaller (eventually, minimized).

[0043] The configuration shown in FIG. 2 is merely an example, and can be modified or changed as appropriate as long as it is possible to realize the functions of each component. For example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. The configuration shown in FIG. 2 may also be realized by two or more devices. In this case, each device is connected via a circuit or a wired or wireless network, and performs data communication with each other to perform cooperative operations, thereby realizing each process according to this embodiment.

[0044] The degradation restoration unit for inference 115 initializes the trained model 220 and performs inference by providing input data to the trained model 220 after initialization. In the initialization, various processes are performed to make the target model capable of executing inference on the input data, such as securing memory to be used by the model and fixing variable parameters in the model. The variable parameters in the model include, for example, the size of the input data (for example, the size indicating the size of the image indicated by the image data).

[0045] The inference-use degradation restoration unit 115 performs inference using the inference-use model. Here, an overview of the inference process executed by the inference-use degradation restoration unit 115 will be described with reference to FIG. To perform inference on an ROI 602 (region to be inferred) set in an input image 601, the degradation restoration unit for inference 115 cuts out a partial region 603 that includes the ROI 602 and has a data size that allows the model for inference to accept input, from the input image 601. Reference numeral 604 indicates the cutting process of the partial region 603 from the input image 601. The degradation restoration unit for inference 115 inputs a partial image 605 cut out from the input image 601 by the cutting process 604 to a model for inference 606, and obtains an inference result image 607 as an output of the model for inference 606. The degradation restoration unit for inference 115 superimposes an ROI result image 608 corresponding to a partial region in the inference result image 607, the position and size of which correspond to the ROI 602, on the position of the input image 601 corresponding to the ROI 602. Reference numeral 609 denotes a process of superimposing the ROI result image 608 on the position corresponding to the ROI 602 in the input image 601. If the image 601 after the ROI result image 608 is superimposed is then used as an output image, it becomes possible to obtain an image in which the inference result after degradation restoration is reflected only in the region corresponding to the ROI 602. By applying the above-described processing, it becomes possible to have the inference model 606 perform inference, for example, even if the size of the ROI 602 does not match the input data size of the inference model 606 (even if the size of the input data is larger than the size of the ROI 602). In the example shown in FIG. 6, the ROI result image 608 is superimposed on the input image 601, but in order to preserve the original input image 601, the ROI result image 608 may also be superimposed on a copy of the input image 601.

[0046] Next, the setting and switching of the model by the model determination unit 113 and the model control unit 114 will be described. The setting and switching of the model by the model determination unit 113 and the model control unit 114 is performed according to the size of the ROI, with each of a plurality of initialized models having different input data sizes as candidates. Specifically, when an ROI is set for an input image by setting a new ROI or updating an existing ROI, the model determination unit 113 determines whether or not it is necessary to set or switch an inference model for the inference degradation restoration unit 115 to perform degradation restoration inference. As a more specific example, the model determination unit 113 may determine whether or not the size of the ROI has been changed, and when it is determined that the size of the ROI has been changed, determine whether or not it is necessary to switch a model applied to degradation restoration inference by the inference degradation restoration unit 115.

[0047] Here, an example of a method for determining an inference model to be applied to degradation restoration inference by the inference degradation restoration unit 115 will be described with reference to FIG. The model determination unit 113 determines, as the model to be set, a model having input data with a smaller size among a series of models having input data with a larger size than the size of the ROI, among a plurality of models that are candidates for the setting target. As a specific example, the model determination unit 113 according to the present embodiment determines whether data of the size of the target ROI can be accepted as an input (i.e., whether inference is possible) in order from the model with the smallest input data size. Then, the model determination unit 113 may determine the model that is first determined to be able to accept data of the size of the ROI as an input as the model to be set.

[0048] In the example shown in FIG. 7, an ROI 702 (width 400 pixels, height 400 pixels) is set in an input image 701 (width 1920 pixels, height 1080 pixels). In addition, three types of initialized inference models 706, 707, and 708 are prepared as multiple models that are candidates for setting. In addition, reference numerals 703, 704, and 705 shown in FIG. 7 diagrammatically indicate the sizes of data that the inference models 706, 707, and 708 can accept as input. Specifically, it is assumed that the inference model 706 is constructed so as to be able to accept data of size 703 (width 200 pixels, height 100 pixels) as input. In addition, it is assumed that the inference model 707 is constructed so as to be able to accept data of size 704 (width 600 pixels, height 400 pixels) as input. In addition, it is assumed that the inference model 708 is constructed so as to be able to accept data of size 705 (width 1920 pixels, height 1080 pixels) as input.

[0049] First, the inference model 706 has the smallest input data size, and therefore the inference model 706 is to be determined. At this time, the inference model 706 is not selected as a model to be set, since the input data size 703 of the inference model 706 is smaller than the size of the ROI 702 in both width and height. Next, the inference model 707, which has the next smallest input data size after the inference model 706, is to be determined. At this time, the inference model 707 has the input data size 704 of the inference model 707 that is equal to or larger than the size of the ROI 702 in both width and height, and therefore is selected as a model to be set. Note that the inference model 708 also has the input data size 705 that is equal to or larger than the size of the ROI 702 in both width and height. However, the inference model 707, which has the smaller input data size, is selected as a model to be set first, and therefore the inference model 708 is not selected as a model to be set.

[0050] The model control unit 114 sets the model determined as the setting target by the model determination unit 113 as an inference model for the inference degradation restoration unit 115 to perform degradation restoration inference. Specifically, when an inference model for the inference degradation restoration unit 115 to perform degradation restoration inference is not set (for example, at the time of initial setting of ROI), the model control unit 114 sets the model determined as the setting target by the model determination unit 113 as the inference model. In addition, when the model determined as the setting target by the model determination unit 113 is different from the inference model currently used by the inference degradation restoration unit 115 for degradation restoration inference, the model control unit 114 switches the inference model to the model determined as the setting target. Note that, at the timing when the model control unit 114 sets the inference model to be used by the inference degradation restoration unit 115 for degradation restoration inference, the model to be set is not initialized. This is because the initialization of each of the multiple models that are candidates for the setting target has already been performed when the inference degradation restoration unit 115 is initialized. Therefore, according to this embodiment, even if the size of the ROI is changed, it is possible to quickly complete switching to a model that corresponds to the changed size of the ROI.

[0051] (process) An example of processing of the image processing system according to this embodiment will be described with reference to Fig. 8. The series of processing shown in Fig. 8 is realized, for example, by CPU 101 executing an image processing computer program according to this embodiment, which causes each functional unit shown in Fig. 2 to function. However, all or part of the functional units shown in Fig. 2 may also be realized by being executed by the hardware of CPU 101.

[0052] An example of the processing of the edge device 100 will be described with reference to FIG. 8(A), focusing in particular on the processing related to initialization of the inference-use degradation restoration unit 115. In S801, the inference degradation restoration unit 115 acquires one model to be subjected to the initialization process from among a series of models stored in the model storage unit 119 (for example, a model that has not yet been initialized). In S802, the inference-use degradation restoration unit 115 initializes the model acquired in S801. In S803, the inference degradation restoration unit 115 determines whether or not a termination condition for the process related to the initialization of the model is satisfied. As a specific example, when a series of initializations of the model stored in the model storage unit 119 is completed, it may be determined that the termination condition for the process related to the initialization of the model is satisfied. If the inference degradation restoration unit 115 determines in S803 that the termination condition for the process related to the initialization of the model is not satisfied, the process proceeds to S801. In this case, the process from S801 onward is executed again for the model that has not yet been initialized. Then, if the inference degradation restoration unit 115 determines in S803 that the end condition for the process related to the initialization of the model is satisfied, it advances the process to S804. In S804, the inference degradation restoration unit 115 sets the model set as the default among the series of initialized models as the inference model to be applied to the degradation restoration inference process. Then, the inference degradation restoration unit 115 ends the series of processes shown in FIG.

[0053] An example of the processing of the edge device 100 will be described with reference to FIG. 8(B), focusing in particular on an example in which degradation restoration inference processing is executed. In S810, the inference degradation restoration unit 115 acquires input image data. In S811 , the inference degradation restoration unit 115 , the model determination unit 113 , and the model control unit 114 acquire the ROI setting from the setting storage unit 120 . In S812, the model determination unit 113 determines whether or not there has been a change in the setting of the ROI. If the model determination unit 113 determines in S812 that the setting of the ROI has been changed, the process proceeds to S813. In S813, the model determination unit 113 and the model control unit 114 execute a process related to switching of the inference model applied to the degradation restoration inference process. The process of S813 will be described in detail later with reference to FIG. 8(C). On the other hand, if the model determination unit 113 determines in S812 that there has been no change in the setting of the ROI, the process proceeds to S814, in which case the process of S813 is skipped. In S814, the degradation restoration unit for inference 115 prepares input data for the inference model. Specifically, if an ROI is set in the input image data based on the setting of the ROI acquired in S811, the degradation restoration unit for inference 115 cuts out an area corresponding to the ROI from the input image data and sets it as input data for the inference model. In addition, if an ROI is not set in the input image data, the degradation restoration unit for inference 115 sets the input image data as it is as input data for the inference model. In S815, the degradation restoration unit for inference 115 inputs the input data prepared in S814 to the model for inference, and executes inference (degradation restoration inference) targeting the input data.

[0054] Referring to FIG. 8(C), an example of processing related to switching of an inference model by the edge device 100, shown as the processing of S813 in FIG. 8(B), will be described. In S820, the model determination unit 113 reads out a list of inference models that are candidates for application to the degradation restoration inference process. In this embodiment, the list of inference models is assumed to be sorted in ascending order of input data size, assuming that an inference model to be applied to the degradation restoration inference process is determined according to the method described with reference to Fig. 7. However, the method of managing the list of inference models is not particularly limited, and the list of inference models may be managed in a state sorted based on other conditions, for example.

[0055] In S821, the model determination unit 113 judges whether or not a judgment has been made as to whether or not the series of inference models read out in S820 are to be applied to the degradation restoration inference process. If the model determination unit 113 determines in S821 that a determination has been made as to whether or not a series of inference models are to be applied to the degradation restoration inference process, the model determination unit 113 advances the process to S826. On the other hand, if the model determination unit 113 determines in S821 that it has not been determined whether or not the series of inference models are to be applied to the degradation restoration inference process, the model determination unit 113 advances the process to S822.

[0056] In S822, the model determination unit 113 reads out information on one model for which a determination has not yet been made, which will be described later in S823 and S824, from the series of inference models read out in S820. In S823, the model determination unit 113 determines whether the width of the target ROI is less than or equal to the width of the input image of the target model (the model whose information was read out in S822) based on the ROI setting acquired in S811 of FIG. 8(B). If the model determination unit 113 determines in S823 that the width of the target ROI is equal to or smaller than the width of the input image of the target model, the model determination unit 113 advances the process to S824. On the other hand, if the model determination unit 113 determines in S823 that the width of the target ROI is not equal to or less than the width of the input image of the target model, it advances the process to S821.

[0057] In S824, the model determination unit 113 determines whether or not the height of the target ROI is less than or equal to the height of the input image of the target model (the model whose information was read out in S822) based on the ROI setting acquired in S811 of FIG. 8(B). If the model determination unit 113 determines in S824 that the height of the target ROI is equal to or less than the height of the input image of the target model, the model determination unit 113 advances the process to S825. On the other hand, if the model determination unit 113 determines in S824 that the height of the target ROI is not equal to or less than the height of the input image of the target model, the model determination unit 113 advances the process to S821.

[0058] In S825, the model determination unit 113 selects the model currently being judged as an inference model to be applied to the degradation restoration inference process. Then, the model control unit 114 sets the model selected by the model determination unit 113 as an inference model to be applied to the degradation restoration inference process by the inference degradation restoration unit 115. As a result, the inference model set for the inference degradation restoration unit 115 is switched to the model selected by the model determination unit 113 (i.e., the model determined as the application target for the restoration inference process).

[0059] In S826, the model determination unit 113 selects the model set as the default as an inference model to be applied to the degradation restoration inference process. Then, the model control unit 114 sets the model selected by the model determination unit 113 (i.e., the default model) as an inference model to be applied to the degradation restoration inference process by the inference degradation restoration unit 115. As a result, the inference model set for the inference degradation restoration unit 115 is switched to the default model selected by the model determination unit 113.

[0060] An example of the processing of the image processing system according to the present embodiment has been described above. As described above, in the image processing system according to the present embodiment, a model that has been initialized in advance is applied when setting a model accompanying the initial setting of an ROI or when setting (switching) a model accompanying a change of an ROI. By applying such control, when setting a model (for example, switching a model), no waiting time occurs due to the initialization of the target model, so that it is possible to quickly complete the setting of the model used for inference. Due to such characteristics, the image processing system according to the present embodiment has a high affinity with a use case in which the size of an ROI set in an image is changed successively and inference is performed on the changed ROI.

[0061] In this embodiment, an example of a case where noise is applied as a degradation factor is described, but the degradation factor is not necessarily limited to noise. As described above, the degradation factor includes blur, aberration, compression, low resolution, loss, and contrast reduction due to weather such as fog, haze, snow, rain, etc. at the time of image capture, and any one of these or a combination of two or more of these may be applied. In addition, in the present embodiment, an example of a case where restoration is performed from input image data using a trained model on the edge device side has been described, but a parameter that assists degradation restoration may be used. For example, information that estimates the degree of image quality degradation that occurs in advance depending on imaging conditions such as the distance to the subject, the focal length, the sensor size, and the exposure may be stored as a lookup table. Then, for example, when applying the degradation restoration inference process to the input image data, the amount of restoration may be adjusted according to the information stored in the lookup table. In this way, the inference unit 112 of the edge device 100 may adjust the restoration strength of degradation targeted at the input image data based on the imaging conditions when the image indicated by the input image data was captured. Furthermore, in the present embodiment, an example in which the model performs degradation restoration processing has been described, but the task of the model is not limited to degradation restoration processing.

[0062] <Second embodiment> A second embodiment of the present disclosure will be described below. In the first embodiment described above, an example of a case where a model applied to inference is determined based on the setting of an ROI was described. In the first embodiment, by applying such control, when the setting of an ROI is changed, the model applied to inference can be switched to a model corresponding to the setting of the ROI. In contrast, in this embodiment, an example of control that suppresses model switching even if the setting of an ROI is changed, and an example of control that automatically sets an ROI using the detection result of an object in an image, etc. will be described. Note that in this embodiment, the description will be given focusing on parts that are particularly different from the first embodiment described above, and detailed description of parts that are substantially similar to the first embodiment will be omitted.

[0063] An example of the functional configuration of the image processing system according to this embodiment will be described with reference to Fig. 9. As shown in Fig. 9, the edge device according to this embodiment has a setting unit 901 and a setting storage unit 902. The setting unit 901 accepts setting input via the input device 20 or the network I / F 106, and stores the input setting in the setting storage unit 902.

[0064] Here, an example of a setting screen of the image processing system according to this embodiment will be described with reference to Fig. 10. The setting screen 1000 shown in Fig. 10 is presented to the user by being displayed on the display device 40, for example, and accepts instructions from the user (e.g., instructions related to value selection, etc.) via the input device 20. The setting screen 1000 corresponds to an example of a setting screen for accepting designations related to the settings of an inference model from the user. The settings of the inference model designated via the setting screen 1000 are referred to by the model determination unit 903 when determining a model to be applied to the degradation restoration inference process by the inference degradation restoration unit 905.

[0065] The setting items of the inference model include, for example, a setting of whether or not to fix the size of the inference model. Specifically, when the setting is made to fix the size of the inference model, switching of the inference model is suppressed even if the size of the ROI is changed. Also, when the setting is made to change the size of the inference model, a determination is made regarding switching of the inference model when the size of the ROI is changed. The values ​​specified via the setting screen 1000 are stored in the setting storage unit 902. This allows the model determination unit 903 to access the setting storage unit 902 and refer to the values ​​stored in the setting storage unit 902.

[0066] 10, an example of the case where the settings are input using the display device 40 and the input device 20 has been described, but the settings may be input by other methods. As a specific example, the settings may be input to the edge device 900 by information transmitted via a network to which the network I / F 106 is connected.

[0067] The object detection unit 906 performs various analyses (e.g., image analysis) using the input image data 116 as input to detect the position and size of an object (e.g., an object to be detected) in the image represented by the input image data 116, and stores the detection result in the setting storage unit 902.

[0068] An example of a process for automatically setting an ROI using the result of detecting an object in an image will be described with reference to Fig. 11. Fig. 11 shows an example in which a ship captured as a subject in an image is detected, and an ROI is set in an area in the image corresponding to the ship.

[0069] 11(A), as a result of object detection performed on an image 1100, a ship is detected in the image, and an area 1101 corresponding to the ship is specified, and an ROI 1102 is set for the area 1101. Then, degradation restoration inference processing is performed on an area including at least the area 1101 corresponding to the ship in the image in which the ROI 1102 is set, and the result is superimposed on the area 1101. For example, Fig. 11(B) shows an example of the result of degradation restoration inference processing performed on a region 1101 in which an ROI 1102 is set in an image 1100 shown in Fig. 11(A). Specifically, in an image 1110 after degradation restoration inference processing shown in Fig. 11(B), a result of noise removal performed on the region 1101 corresponding to the region 1101 in Fig. 11(A) is superimposed on the region 1111.

[0070] 11(A) is changed, the inference model applied to the degradation restoration inference process is switched according to the changed size of the ROI, as in the first embodiment. Then, the degradation restoration inference process is executed for the area corresponding to the changed ROI using the switched inference model, and the result is superimposed on the area corresponding to the changed ROI.

[0071] 11(A) and 11(B), for convenience, the areas 1101, 1102, and 1111 are indicated by dashed lines, but when the image is actually displayed, information indicating these areas does not necessarily need to be visualized and displayed. As a specific example, when presenting these areas to the user, the areas in the image may be visualized.

[0072] (process) With reference to Fig. 12, an example of processing of the image processing system according to this embodiment will be described, focusing in particular on the degradation restoration inference processing of the edge device 900. Note that the example shown in Fig. 12 will be described, focusing on parts that are particularly different from the first embodiment, and detailed description of parts that are substantially the same as the first embodiment will be omitted.

[0073] In S1201, the object detection unit 906 performs image analysis related to object detection on input image data to detect an object in the image, and attempts to identify the position and size of the object in the image. In S1202, the object detection unit 906 determines whether or not an object was detected in the image in S1201. If the object detection unit 906 determines in S1202 that an object has been detected in the image, the process proceeds to S1203. In S1203, the setting unit 901 automatically sets an ROI for a region in the image corresponding to the object, based on the result of the object detection in the image in S1201. On the other hand, if the object detection unit 906 determines in S1202 that an object has not been detected in the image, the process proceeds to S1204. In S1204, the setting unit 901 sets the ROI according to a default setting. As a specific example, the setting unit 901 may set the ROI for an area corresponding to the entire target image. Of course, this control is merely an example, and the type of control to be applied as the default setting may be appropriately set according to the use case.

[0074] In S1205, the model determination unit 903 determines whether or not there has been a change in the setting of the ROI. If the model determination unit 903 determines in S1205 that the setting of the ROI has been changed, the process proceeds to S1206. On the other hand, if the model determination unit 903 determines in S1205 that there has been no change in the setting of the ROI, the process proceeds to S814. In this case, the processes of S1206 and S813 are skipped.

[0075] In S1206, the model determination unit 903 reads the setting value of the setting stored in the setting storage unit 902 indicating whether or not to fix the size of the inference model, and determines whether the setting value indicates that the size of the inference model is variable. If the model determination unit 903 determines that the setting value read in S1206 indicates that the size of the inference model is variable, it proceeds to S813. On the other hand, if the model determination unit 903 determines that the setting value read in S1206 indicates that the size of the inference model is not variable (is fixed), the process proceeds to S814. In this case, the process of S813 is skipped.

[0076] As described above, in the image processing system according to the present embodiment, the edge device 900 suppresses switching of the inference model when the size of the inference model is set to be fixed even if the setting of the ROI is changed. With this mechanism, the user can intentionally control whether or not to apply switching of the inference model accompanying the change of the ROI. In addition, the image processing system according to the present embodiment makes it possible to automatically set the ROI using the detection result of an object from within the image. This is expected to reduce the user's efforts compared to when the ROI is set manually.

[0077] <Third embodiment> As a third embodiment of the present disclosure, an example of a network structure of the trained model 220 applied to the degradation restoration inference process by the inference degradation restoration unit 905 will be described, focusing on a case where a recurrent configuration in which a previous inference result is used is applied. Note that in this embodiment, the description will focus on parts that are particularly different from the first and second embodiments described above, and detailed descriptions of parts that are substantially similar to the first and second embodiments will be omitted.

[0078] With reference to FIG. 13, an example of the network structure of the trained model 220 applied to the degradation restoration inference process by the inference degradation restoration unit 905 according to this embodiment will be described. In the example shown in FIG. 13, the input data (hereinafter also referred to as inference input data 1302) to the trained model (neural network) is data obtained by performing preprocessing 301 on the input image data 116. In addition, the neural network applied as the inference model 1303 shown in FIG. 13 is a neural network with a recurrent configuration. The recurrent configuration is a configuration in which a part of the previous inference result is reused as the current input data. By reusing the previous inference result as the current input data, it is expected to have the effect of improving the consistency and continuity of the inference result in the time direction.

[0079] A previous recurrent output 1301 and inference input data 1302 are input to an inference model 1303. A previous recurrent output 1304 is data that is output as part of a previous inference result. Note that in this embodiment, image data having the same image size and number of channels as the inference input data 1302 is applied as the previous recurrent output 1304, but the type of data applied as the previous recurrent output 1304 is not limited. Moreover, the inference model 1303 outputs a recurrent output 1304 and inference output data 1305 as inference results. The inference output data 1305 is image data that has been subjected to degradation restoration inference processing and is output as an inference result. The recurrent output 1304 corresponds to a part of the image data (image data that has been subjected to degradation restoration inference processing) that is output as an inference result, and is used as a previous recurrent output 1301 that corresponds to a part of the input data in the next inference.

[0080] Note that, at the time of the first inference or at the time of inference immediately after switching the model, the previous recurrent output 1301 does not exist. In such a case, for example, initial data such as data equivalent to a black image (in other words, dummy data) may be used as the previous recurrent output 1301. As another example, the recurrent output 1304, which is a part of the data output as the previous inference result by the inference model before switching, may be used as the previous recurrent output 1301. Note that, in this embodiment, for control when the previous recurrent output 1301 does not exist, any of the above-mentioned operations are configured to be switchable by setting.

[0081] Here, an example of a setting screen of the image processing system according to the embodiment will be described with reference to Fig. 14. The setting screen 1400 shown in Fig. 14 is presented to the user by being displayed on the display device 40, for example, and accepts instructions from the user (e.g., instructions related to value selection, etc.) via the input device 20. The setting screen 1400 corresponds to an example of a setting screen for accepting designations related to the settings of an inference model from the user. The settings of the inference model designated via the setting screen 1400 are referred to when the model determination unit 903 determines a model to be applied to the degradation restoration inference process by the inference degradation restoration unit 905.

[0082] Examples of inference model settings include whether or not to reuse the output of the inference model before switching as recurrent information that is input to the inference model when the inference model is switched. Specifically, it is possible to specify a setting to reuse the output of the inference model before switching as the recurrent information to be input to the inference model. In this case, when the inference model is switched, if the recurrent information before the switch is usable, the recurrent information is reused as the input to the inference model after the switch. It is also possible to specify a setting to create recurrent information to be input to the inference model. In this case, the inference models before and after the switching are caused to perform inference in parallel, and recurrent information is generated based on the results of a predetermined number of inferences performed in the inference model after the switching, and the inference model is switched after the recurrent information is generated. In this case, since the inference models before and after the switching perform inference in parallel, the usage of computer resources such as the CPU and GPU of the image processing device tends to be greater than when only one of them performs inference. Therefore, for example, when the inference models before and after the switching perform inference in parallel, the frame rate related to the inference may be limited depending on the usage of the computer resources used for the inference. By applying such control, it is also possible to limit the usage of computer resources associated with the execution of inference. It is also possible to specify a setting to use initial data as recurrent information to be input to the inference model. In this case, when the inference model is switched, initial data such as a black image is used as recurrent information to be input to the inference model after the switch. The values ​​specified via the setting screen 1400 are stored in the setting storage unit 902. This allows the model determination unit 903 to access the setting storage unit 902 and refer to the values ​​stored in the setting storage unit 902.

[0083] 14, an example of the case where the settings are input using the display device 40 and the input device 20 has been described, but the settings may be input by other methods. As a specific example, the settings may be input to the edge device 900 by information transmitted via a network to which the network I / F 106 is connected.

[0084] (process) With reference to Fig. 15, an example of processing of the image processing system according to this embodiment will be described, focusing in particular on the degradation restoration inference processing of the edge device 900. Note that the example shown in Fig. 15 will be described, focusing on parts that are particularly different from the first embodiment, and detailed description of parts that are substantially the same as the first embodiment will be omitted.

[0085] In S1501, the degradation restoration unit for inference 115 prepares input data for the inference model. Specifically, if an ROI is set in the input image data based on the setting of the ROI acquired in S811, the degradation restoration unit for inference 115 cuts out an area corresponding to the ROI from the input image data and sets it as input data for the inference model. In addition, if an ROI is not set in the input image data, the degradation restoration unit for inference 115 sets the input image data as it is as input data for the inference model. Furthermore, the degradation restoration unit for inference 115 also sets the previous recurrent output held as input data for the inference model.

[0086] Here, when the setting of the recurrent information at the time of model switching read from the setting storage unit 902 indicates reuse of the recurrent output before the switching, the degradation restoration unit for inference 115 reuses the recurrent output before the model switching as the previous recurrent output. Also, when the setting of the recurrent information indicates use of initial data, the degradation restoration unit for inference 115 sets predetermined initial data such as data equivalent to a black image as the previous recurrent output. In these cases, in S1502, the degradation restoration unit for inference 115 inputs the input data prepared in S1501 to the inference model after the switching, and executes inference (degradation restoration inference) targeting the input data. Furthermore, when the setting of the recurrent information indicates the creation of recurrent information, the inference degradation restoration unit 115 performs inference a predetermined number of times with the inference model after switching, generates recurrent information based on the results of the predetermined number of inferences, and then switches the model. In this case, in S1502, the inference degradation restoration unit 115 causes the inference model before switching to perform inference in parallel until the inference model after switching has performed a predetermined number of inferences. By applying such control, even in a state before a recurrent output based on the inference result by the inference model after switching is created, it is possible to perform degradation restoration of the target image using the inference result of the inference model before switching.

[0087] As described above, in the image processing system according to the present embodiment, in a situation where a network structure is applied in which the previous recurrent output is used as an input to an inference model, the user can set a method for preparing the previous recurrent output. With this mechanism, the behavior of the system when switching models can be appropriately switched so as to be the behavior desired by the user.

[0088] <Fourth embodiment> As the fourth embodiment of the present disclosure, an example of a case where inference is performed on an area including multiple ROIs and an example of a case where an initialized model can be added by setting will be described. Note that in this embodiment, the description will be focused on parts that are particularly different from the first to third embodiments described above, and detailed descriptions of parts that are substantially similar to the first to third embodiments will be omitted.

[0089] Fig. 16 shows an example of a situation where multiple objects are detected in an image and an ROI is automatically set for each of the multiple objects. Fig. 16 shows an example of a situation where multiple ships captured as subjects in an image are detected and an ROI is set in an area in the image corresponding to each of the multiple ships. Specifically, as shown in FIG. 16(A), as a result of object detection performed on an image 1600, two ships are detected in the image, and regions 1601 and 1602 corresponding to the two ships are specified. Then, in the example shown in FIG. 16(A), ROIs are not set individually for the detection results of the two objects (two ships), but one ROI 1603 including the two objects is set. By applying such control, it is possible to limit the number of inferences using an inference model to one inference targeting the ROI 1603. Then, degradation restoration inference processing is performed on the region including at least the regions 1601 and 1602 corresponding to the two ships in the image in which the ROI 1603 is set, and the result is superimposed on the regions 1601 and 1602. For example, Fig. 16(B) shows an example of the result of degradation restoration inference processing performed on regions 1601 and 1602 in which ROI 1603 is set in image 1600 shown in Fig. 16(A). In Fig. 16(B), the region indicated by reference numeral 1613 corresponds to the region corresponding to ROI 1603 shown in Fig. 16(A). Also, in Fig. 16(B), the regions indicated by reference numerals 1611 and 1612 correspond to regions 1601 and 1602 in Fig. 16(A). That is, in image 1610 after degradation restoration inference processing shown in Fig. 16(B), the result of noise removal performed on regions 1601 and 1602 is superimposed on regions 1611 and 1612 included in region 1613.

[0090] 16(A) is changed, the inference model applied to the degradation restoration inference process is switched according to the changed size of the ROI, as in the first embodiment. Then, the degradation restoration inference process is executed for the area corresponding to the changed ROI using the switched inference model, and the result is superimposed on the area corresponding to the changed ROI.

[0091] In this embodiment, an example of setting an ROI for an area including each of a plurality of objects detected in an image has been described, but an ROI may be set for each of the areas of the plurality of objects individually. In this case, inference is performed using an inference model for each ROI individually. 16(A) and 16(B), for convenience, areas 1601, 1602, 1611, and 1612 are indicated by dashed lines, but when an image is actually displayed, information indicating these areas does not necessarily need to be visualized and displayed. As a specific example, when presenting these areas to a user, the areas in the image may be visualized.

[0092] Here, an example of a setting screen of the image processing system according to the embodiment will be described with reference to Fig. 17. The setting screen 1700 shown in Fig. 17 is presented to the user by being displayed on the display device 40, for example, and accepts instructions from the user (e.g., instructions related to value selection, etc.) via the input device 20. The setting screen 1700 corresponds to an example of a setting screen for accepting specifications related to the setting of an inference model from the user. Examples of inference model settings include settings regarding how to apply inference using the inference model when multiple objects are detected in an image. Specifically, it is possible to specify a setting for performing inference on the detection results of multiple objects in an image at once. In this case, one ROI is set for the detection results of multiple objects in an image so as to include the areas corresponding to each of the multiple objects, and inference is performed using the inference model for that one ROI. It is also possible to specify settings for performing inference separately for each of the results of detecting multiple objects in an image. In this case, an ROI is set separately for each of the results of detecting multiple objects in an image, and inference is performed separately for each ROI using an inference model.

[0093] Another example of an item for setting an inference model is setting operations related to adding an inference model to be used for inference. For example, when an ROI is set or updated, if there is no initialized model corresponding to the condition (e.g., size) of the ROI, it is possible to specify a setting to automatically initialize a new inference model according to the ROI. In addition, a condition for such automatic initialization of an inference model may be specified. As a specific example, it may be possible to specify the above-mentioned automatic initialization setting for an ROI that is set or updated at a frequency equal to or greater than a threshold within a predetermined time. In this case, if there is no initialized model corresponding to the size of an ROI that is set or updated at a frequency equal to or greater than a threshold within a predetermined time, an inference model that can accept data of the size of the ROI as an input is newly initialized. Note that, for example, by executing the process related to the initialization of the model in this case in parallel with the inference by the model before switching, it is possible to prevent the occurrence of a waiting time associated with initialization when switching models. It is also possible to manually specify settings for adding an inference model. In this case, the specification of the conditions of the inference model to be added (e.g., the size of the input data, etc.) is accepted via the setting screen 1700, and the inference model to be added is initialized based on the specification. It is also possible to specify a setting to restrict the addition of inference models. In this case, for example, even if there is no initialized model that corresponds to the ROI condition (e.g., size), a new inference model will not be added. The values ​​specified via the setting screen 1700 are stored in the setting storage unit 902. This allows the model determination unit 903 to access the setting storage unit 902 and refer to the values ​​stored in the setting storage unit 902.

[0094] 17, an example of the case where the settings are input using the display device 40 and the input device 20 has been described, but the settings may be input by other methods. As a specific example, the settings may be input to the edge device 900 by information transmitted via a network to which the network I / F 106 is connected.

[0095] (process) An example of the processing of the image processing system according to this embodiment will be described with reference to Fig. 18, focusing in particular on the degradation restoration inference processing of the edge device 900. Note that the example shown in Fig. 18 will be described with a focus on parts that are particularly different from the first to third embodiments, and detailed descriptions of parts that are substantially the same as the first to third embodiments will be omitted.

[0096] In S1800, the object detection unit 906 determines whether or not multiple objects were detected in the image in S1201. If the object detection unit 906 determines in S1800 that a plurality of objects have been detected in the image, the process proceeds to S1801. On the other hand, if the object detection unit 906 determines in S1800 that multiple objects have not been detected in the image (that only one object has been detected), the process proceeds to S1203.

[0097] In S1801, the setting unit 901 determines whether or not a setting has been made regarding the method of applying inference using an inference model, such that inference is performed collectively for the detection results of multiple objects in an image. If the setting unit 901 determines in S1801 that a setting is made to perform inference collectively on the detection results of a plurality of objects in an image, the process proceeds to S1803. In S1803, the setting unit 901 sets an ROI for an area including areas corresponding to each of the detected objects, and proceeds to S1804. On the other hand, if the model determination unit 903 determines in S1801 that the setting is not made to perform collective inference on the detection results of multiple objects from the image, the process proceeds to S1802. In S1802, the setting unit 901 sets ROIs individually for the regions corresponding to the multiple detected objects, and the process proceeds to S1804.

[0098] In S1804, the setting unit 901 determines whether or not settings have been made for automatic initialization of an inference model targeted at an ROI that has been set or updated frequently. If the setting unit 901 determines in S1804 that settings have been made for automatic initialization of an inference model targeted at an ROI that has been set or updated frequently, it proceeds to S1805. On the other hand, if the setting unit 901 determines in S1804 that no settings have been made for automatic initialization of an inference model targeting an ROI that has been set or updated frequently, it proceeds to S1205.

[0099] In S1805, the setting unit 901 determines whether or not the target ROI corresponds to an ROI that has been set or updated with a frequency equal to or greater than a threshold within a predetermined time period. If the setting unit 901 determines in S1805 that the target ROI corresponds to an ROI that has been set or updated with a frequency equal to or greater than a threshold within a predetermined time, the setting unit 901 advances the process to S1806. On the other hand, if the setting unit 901 determines in S1805 that the target ROI is not an ROI that has been set or updated with a frequency equal to or greater than the threshold within the predetermined time, the process proceeds to S1205.

[0100] In S1806, the model determination unit 903 determines whether the difference between the size of the target ROI and the size of the input data of the inference model (initialized model) is within a predetermined range. If the model determination unit 903 determines in S1806 that the difference between the size of the target ROI and the size of the input data of the inference model is within a predetermined range, it proceeds to S1205. On the other hand, if the model determination unit 903 determines in S1806 that the difference between the size of the target ROI and the size of the input data of the inference model is not within the predetermined range, it proceeds to S1807. In S1807, the model determination unit 903 newly initializes a model that can accept data of the size of the target ROI as an input. In this case, in the process of S813, the inference model applied to the degradation restoration inference process is switched to the model newly initialized in S1807.

[0101] As described above, in the image processing system according to the present embodiment, it is possible to apply control for automatically setting an ROI so that inference is performed collectively for multiple regions in an image according to a specified setting. By applying such control, it is possible to reduce the number of inferences to be performed. In addition, when there is no initialized model that matches the conditions of the set or updated ROI, it is also possible to apply control for automatically initializing a new model according to the conditions. In particular, by applying such control for automatic model initialization to an ROI that is relatively frequently set or updated, it is expected to have the effect of speeding up the process related to inference using an inference model.

[0102] <Fourth embodiment> As a fourth embodiment of the present disclosure, an example will be described in which, when an ROI is set or updated, if the size of the ROI is close to the size of data that an initialized model can accept as input, the size of the ROI is adjusted to match the model. Note that, in this embodiment, the description will be focused on parts that are particularly different from the first to fourth embodiments described above, and detailed descriptions of parts that are substantially similar to the first to fourth embodiments will be omitted.

[0103] Fig. 19 is a diagram showing an example of a screen for setting an ROI in a target image. The setting screen 1900 shown in Fig. 19 is presented to a user by being displayed on the display device 40, for example, and accepts instructions from the user via the input device 20 (e.g., instructions related to value selection, etc.). The setting of the ROI specified via the screen 1900 shown in Fig. 19 is used for degradation restoration inference processing by the inference degradation restoration unit 115 and processing related to the determination of a model to be set by the model determination unit 113.

[0104] Fig. 19(A) shows a schematic diagram of a situation in which a user specifies an ROI in a target image via a screen 1900. Specifically, in the example shown in Fig. 19(A), an input from the user is accepted via a touch panel, and an ROI 1901 is specified so as to include an object 1903 (a house) captured as a subject in the image. Fig. 19(B) shows a schematic diagram of a state in which the size of ROI1901 specified in Fig. 19(A) is automatically adjusted to match the size of the input data of the initialized model. Specifically, the setting unit 121 judges whether or not there is an initialized model in which the difference in width and height between the size of the input data and the size of ROI1901 is within a predetermined range. If there is an initialized model in which the difference in width and height between the size of ROI1901 and the size of ROI1901 is within a predetermined range, the setting unit 121 adjusts the size of ROI1901 to match the size of the input data of the model. ROI1902 shown in Fig. 19(B) shows the ROI after the size adjustment. 19B, the size of ROI 1902 is adjusted to be smaller than the unadjusted size of ROI 1901. By applying such control, the area to be inferred becomes smaller, and the effect of further improving the processing speed related to the inference can be expected. The button 1904 is an input interface for receiving an instruction from the user regarding the reflection of the ROI settings specified via the screen 1900. For example, when the user accepts the ROI settings that have been automatically adjusted as shown in Fig. 19(B), the user may reflect the settings by pressing the button 1904. In addition, the ROI that has been automatically adjusted may be updated or reset by a user's operation again. In addition, in this case, if the ROI is updated or reset within a predetermined time after the automatic adjustment of the ROI, the automatic adjustment of the ROI may be suppressed again. By applying such control, it is possible to prevent the occurrence of a situation in which the adjustment result of the ROI is overwritten by the automatic adjustment of the ROI when the user intentionally adjusts the size of the ROI. The values ​​specified via the setting screen 1900 are stored in the setting storage unit 120. This allows the model determination unit 113 to access the setting storage unit 120 and refer to the values ​​stored in the setting storage unit 120.

[0105] 19 has been described as an example in which the settings are input using the display device 40 and the input device 20, but the settings may be input by other methods. As a specific example, the settings may be input to the edge device 100 by information transmitted via a network to which the network I / F 106 is connected.

[0106] (process) An example of processing of the image processing system according to this embodiment will be described with reference to Fig. 20, focusing in particular on processing when an ROI is set by an operation via the edge device 900. Note that the example shown in Fig. 20 will be described with a focus on parts that are particularly different from the first to fourth embodiments, and detailed description of parts that are substantially similar to the first to fourth embodiments will be omitted.

[0107] In S2000, the setting unit 121 displays a menu screen on the display device 40. On the menu screen, as exemplified with reference to FIG. 19, a display area for an input image and a setting completion button 1904 are displayed. In S2001, the setting unit 121 updates the image displayed in the input image display area with the latest input image. In S2002, the setting unit 121 determines whether or not an ROI has been set. If the setting unit 121 determines in S2002 that the ROI has not been set, the process proceeds to S2003. On the other hand, if the setting unit 121 determines in S2002 that the ROI has been set, the process proceeds to S2005. Note that the setting of the ROI is performed by the user specifying an area in the input image displayed on the menu screen by an operation such as drag and drop via a touch panel, as described with reference to Fig. 19. Of course, the method of setting the ROI described with reference to Fig. 19 is merely an example, and the method is not particularly limited as long as it is possible to set the ROI for an area in a target image.

[0108] In S2005, the setting unit 121 determines whether or not a predetermined time has elapsed since the previous automatic adjustment of the ROI. If the setting unit 121 determines in S2005 that the predetermined time has not elapsed since the previous automatic adjustment of the ROI, the setting unit 121 advances the process to S2003. On the other hand, if the setting unit 121 determines in S2005 that the predetermined time has elapsed since the previous automatic adjustment of the ROI, the process proceeds to S2006. In S2006, the setting unit 121 reads out information on a series of initialized models that are candidates for inference models for performing degradation restoration inference.

[0109] In S2007, the setting unit 121 judges whether or not a series of judgment processes, which will be described later as S2008 to S2010, have been executed for all models whose information has been read out in S2006. If the setting unit 121 determines in S2007 that the series of determination processes has been executed for all models, it advances the process to S2003. On the other hand, if the setting unit 121 determines in S2007 that the series of determination processes have not been executed for all models (that is, there are models remaining for which the determination processes have not been executed), the process proceeds to S2008.

[0110] In S2008, the setting unit 121 reads out information on models that have not yet been subjected to judgment, from among the series of models whose information has been read out in S2006. In S2009, the setting unit 121 determines whether or not the difference between the width of the set ROI and the width of the input image of the model whose information has been read out in S2008 is within a predetermined range. If the setting unit 121 determines in S2009 that the difference between the width of the ROI and the width of the input image of the model is within the predetermined range, the process proceeds to S2010. On the other hand, if the setting unit 121 determines in S2009 that the difference between the width of the ROI and the width of the input image of the model is not within the predetermined range, it advances the process to S2007. In S2010, the setting unit 121 determines whether or not the difference between the height of the set ROI and the height of the input image of the model whose information was read out in S2008 is within a predetermined range. If the setting unit 121 determines in S2010 that the difference between the height of the ROI and the height of the input image of the model is within the predetermined range, it advances the process to S2011. On the other hand, if the setting unit 121 determines in S2010 that the difference between the height of the ROI and the height of the input image of the model is not within the predetermined range, it advances the process to S2007. In S2011, the setting unit 121 adjusts the width and height of the set ROI to match the width and height of the model whose information was read out in S2008.

[0111] In S2003, the setting unit 121 determines whether or not a condition related to completion of the setting of the ROI is satisfied. As a specific example, the setting unit 121 may determine that a condition related to completion of the setting of the ROI is satisfied when the setting completion button 1904 is pressed on the screen 1900 illustrated in FIG. If the setting unit 121 determines in S2003 that the condition for completing the setting of the ROI is not satisfied, the process returns to S2001. In this case, the process from S2001 onwards is executed again. On the other hand, if the setting unit 121 determines in S2003 that the condition related to the completion of the setting of the ROI is satisfied, the process proceeds to S2004. In S2004, the setting unit 121 saves the setting of the ROI in the setting saving unit 120, and then ends the series of processes shown in FIG.

[0112] As described above, in the image processing system according to the present embodiment, when the size of the ROI is close to the size of data that the initialized model can accept as input, the size of the ROI is adjusted to match the model. By applying such control, it becomes possible to automatically select a model that is more suitable for executing an inference process targeting an area corresponding to the ROI set in an image (for example, a model that can perform inference at a higher speed).

[0113] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0114] The disclosure of this embodiment also includes the following configuration and program. (Configuration 1) An information processing device characterized by having: a setting means for setting an inference target area in an input image that is to be subject to inference by a model constructed based on machine learning; and a determination means for determining a model to be applied to inference using the input image as input data from a plurality of models having different input data sizes that have been initialized to make the model capable of performing inference on input data, in accordance with the size of the inference target area set by the setting means. (Configuration 2) The information processing device described in configuration 1, characterized in that the determination means determines, among a series of models among the plurality of models, a model having an input data size larger than the inference target area set by the setting means, a model having an input data size smaller than the input data size, as a model to be applied to inference using the input image as input data. (Configuration 3) An information processing device as described in configuration 1 or 2, characterized in that it has a switching means for switching, when the size of the inference target area is changed, the model to be applied to inference using the input image as input data to a model determined by the determination means in accordance with the size. (Configuration 4) The information processing device described in configuration 3, characterized in that the switching means suppresses switching of the model to be applied to inference using the input image as input data when a setting is made to restrict model switching, even if the size of the inference target area is changed. (Configuration 5) The information processing device according to any one of configurations 1 to 4, wherein the model is a model that performs image processing on a target image to restore degradation that has become apparent in the image. (Configuration 6) The information processing device according to configuration 5, wherein the degradation includes at least one of noise, compression, low resolution, blur, aberration, loss, and reduced contrast. (Configuration 7) An information processing device described in any one of configurations 1 to 6, characterized in that the multiple models that have been initialized include a recurrent configuration model that uses the results of previously executed inference as input. (Configuration 8) An information processing device as described in configuration 7, characterized in that when the size of the inference target area is changed and the model to be applied to inference using the input image as input data is switched to a model of the recurrent configuration, inference is performed in parallel using the model before switching and the model after switching, and after a predetermined number of inferences have been performed using the model after switching, the model is switched. (Configuration 9) An information processing device as described in configuration 8, characterized in that, when switching the model to be applied to inference using the input image as input data, during a period in which inference is performed in parallel using the model before the switch and the model after the switch, the frame rate of the processing related to the inference is limited depending on the usage status of the resources used for the inference. (Configuration 10) An information processing device as described in any one of configurations 7 to 9, characterized in that when the size of the inference target area is changed and the model applied to inference using the input image as input data is switched to a model of the recurrent configuration, the model after the switch uses as input the result of inference previously performed by the model before the switch. (Configuration 11) The information processing device described in any one of configurations 1 to 10, characterized in that when multiple inference target areas are set by the setting means, the determination means determines a model to be applied to inference using the input image as input data from the multiple models depending on the size of the area containing the multiple inference target areas. (Configuration 12) An information processing device as described in any one of configurations 1 to 11, further comprising an initialization means for newly initializing a model whose input data size is larger than the size of the inference target area set by the setting means, and wherein the determination means switches the method of determining a model to be applied to inference using the input image as input data between a method of determining from the multiple models and a method of determining a model newly initialized by the initialization means, depending on a setting related to a model determination method. (Configuration 13) The information processing device described in configuration 12, characterized in that the initialization means, when the frequency of setting the inference target area is equal to or greater than a threshold and the difference between the size of the inference target area and the input data size of the initialized model is outside a predetermined range, initializes a new model whose input data size is larger than the size of the inference target area whose setting frequency is equal to or greater than the threshold. (Configuration 14) The information processing device according to any one of configurations 1 to 13, characterized in that when the set inference target area is changed and the difference between the size of the inference target area after the change and the input data size of the initialized model is within a predetermined range, the setting means adjusts the size of the inference target area after the change to the input data size. (Configuration 15) The information processing device described in Configuration 14, characterized in that the setting means limits the adjustment of the size of the changed inference target area when the inference target area is changed again within a predetermined time after adjusting the size of the inference target area. (Configuration 16) The information processing device according to any one of configurations 1 to 15, characterized in that the setting means sets the inference target area in the input image based on a detection result of an object in the input image. (Program 1) A program for causing a computer to function as an information processing device, characterized by having a setting means for setting an inference target area in an input image that is to be subject to inference using a model constructed based on machine learning, and a determination means for determining a model to be applied to inference using the input image as input data from a plurality of models having different input data sizes that have been initialized to make the model capable of performing inference on input data, in accordance with the size of the inference target area set by the setting means. [Explanation of symbols]

[0115] 100 Edge Devices 112 Reasoning part 113 Model Determination Unit 114 Model Control Unit 121 Setting section

Claims

1. A setting means for setting an inference target area in an input image that is to be inferred by a model constructed based on machine learning; A determination means for determining a model to be applied to inference using the input image as input data from a plurality of models having different input data sizes, the models being initialized to be in a state in which inference can be performed on the input data according to the size of the inference target area set by the setting means; An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein the determination means determines, among the plurality of models, a model having an input data size smaller than the inference target area set by the setting means, as the model to be applied to inference using the input image as input data.

3. 2. The information processing device according to claim 1, further comprising a switching means for switching, when the size of the inference target area is changed, a model to be applied to inference using the input image as input data to a model determined by the determination means in accordance with the size.

4. The information processing device according to claim 3, characterized in that, when a setting is made to restrict model switching, the switching means suppresses switching of the model to be applied to inference using the input image as input data, even if the size of the inference target area is changed.

5. The information processing apparatus according to claim 1 , wherein the model is a model for performing image processing on a target image to restore degradation that has become evident in the target image.

6. The information processing apparatus according to claim 5 , wherein the degradation includes at least one of noise, compression, low resolution, blur, aberration, loss, and reduced contrast.

7. The information processing apparatus according to claim 1 , wherein the plurality of initialized models includes a model of a recurrent configuration that uses a result of a previously executed inference as an input.

8. The information processing device according to claim 7, characterized in that when the size of the inference target area is changed and the model to be applied to inference using the input image as input data is switched to the recurrent configuration model, inference is performed in parallel using the model before the switch and the model after the switch, and after a predetermined number of inferences have been performed using the model after the switch, the model is switched.

9. 9. The information processing device according to claim 8, characterized in that, when switching the model to be applied to inference using the input image as input data, during a period in which inference is performed in parallel using the model before the switch and the model after the switch, a frame rate of the processing related to the inference is limited depending on the usage status of resources used for the inference.

10. The information processing device according to claim 7, characterized in that when the size of the inference target area is changed and the model applied to inference using the input image as input data is switched to the recurrent configuration model, the model after the switch uses as input the result of the inference previously performed by the model before the switch.

11. 2. The information processing device according to claim 1, wherein, when a plurality of inference target areas are set by the setting means, the determination means determines a model to be applied to inference using the input image as input data from the plurality of models depending on a size of an area including the plurality of inference target areas.

12. an initialization means for newly initializing a model in which the size of input data is larger than the size of the inference target area set by the setting means; The determination means switches a method of determining a model to be applied to inference using the input image as input data between a method of determining from the plurality of models and a method of determining a model newly initialized by the initialization means, depending on a setting related to a method of determining a model.

2. The information processing device according to claim 1.

13. 13. The information processing device according to claim 12, wherein the initialization means initializes a new model whose input data size is larger than the size of the inference target area whose setting frequency is equal to or greater than the threshold when the frequency of setting the inference target area is equal to or greater than a threshold and the difference between the size of the inference target area and the input data size of the initialized model is outside a predetermined range.

14. 2. The information processing device according to claim 1, wherein when the set inference target area is changed and a difference between the size of the inference target area after the change and the input data size of the initialized model is within a predetermined range, the setting means adjusts the size of the inference target area after the change to the input data size.

15. 15. The information processing device according to claim 14, wherein the setting means limits adjustment of the size of the inference target area after the change when the inference target area is changed again within a predetermined time after the adjustment of the size of the inference target area.

16. 2. The information processing apparatus according to claim 1, wherein said setting means sets said inference target region in said input image based on a result of detection of an object in said input image.

17. Computer, A setting means for setting an inference target area in an input image that is to be inferred by a model constructed based on machine learning; A determination means for determining a model to be applied to inference using the input image as input data from a plurality of models having different input data sizes, the models being initialized to be in a state in which inference can be performed on the input data according to the size of the inference target area set by the setting means; A program for causing an information processing device to function, comprising:

Images