Information processing device and program

Abstract

To perform inference about images in a more suitable manner even in a situation where data of the images to be inferred may have various sizes.SOLUTION: A setting unit 121 sets, in an input image, a ROI to be inferred by a model that is built on the basis of machine learning. A model determination unit 113 determines, according to the size of the ROI set in the input image, a model to be applied to inference using the input image as input data, from a plurality of models different from each other in the size of input data and subjected to initialization to enable execution of inference about the input data.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device and a program. [Background technology]

[0002] In recent years, learning models (hereinafter simply referred to as models) constructed based on machine learning have been applied to a variety of image processing and image recognition application programs. One example of image processing using models is noise removal processing, which reduces (and ultimately removes) the effects of degradation such as noise that appears in a target image. Another example of image recognition using models is object detection processing, which detects a desired object captured as a subject in an image. Some of the models described above have different sizes of image data that can be accepted as input. In light of this, Patent Document 1 proposes a technology in which the size of the target image data is changed to a size of image data that can be accepted as input by the object detection model before being input. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-71757 Summary of the Invention [Problem to be solved by the invention]

[0004] On the other hand, as in the technology disclosed in Patent Document 1, when the size of input image data is changed, the characteristics of the image (e.g., the characteristics of noise manifested in the image) change, and the accuracy of image processing using the model (e.g., noise removal accuracy) may decrease. To address this issue, a method of applying a model that can accept the target image data as input may be considered. However, it may take time to initialize the model so that it can perform inference on the image data (e.g., image processing such as noise removal), and it is possible to imagine a situation in which the time related to this initialization manifests as waiting time.

[0005] In view of the above problems, the present invention realizes inference on images in a more suitable manner even in situations where image data of various sizes may be the target. [Means for solving the problem]

[0006] The information processing device according to the present invention includes a setting means for setting an inference target area in an input image that is to be inferred by a model constructed based on machine learning, and a determination means for determining a model to be applied to inference using the input image as input data from a plurality of models having different input data sizes that have been initialized to be in a state where inference on input data can be executed, according to the size of the inference target area set by the setting means. The determining means determines, from among the plurality of models, a series of models having input data with a size larger than the inference target region set by the setting means, a model with a smaller input data size as a model to be applied to inference using the input image as input data. It is characterized by: [Effects of the Invention]

[0007] According to the present invention, even in a situation where image data of various sizes may be the target, it becomes possible to realize inference on the image in a more suitable manner. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of an image processing system. [Figure 2] FIG. 1 is a diagram illustrating an example of a functional configuration of an image processing system. [Figure 3] FIG. 2 is a diagram illustrating an example of a processing flow of an image processing system. [Figure 4] FIG. 10 is a diagram illustrating an example of a model structure. [Figure 5] 10A and 10B are diagrams illustrating an example of a process for adding degradation to an image. [Figure 6] FIG. 1 is a diagram showing an outline of an inference process using an image as an input. [Figure 7] FIG. 10 is a diagram illustrating an example of a method for determining a model. [Figure 8] 10 is a flowchart showing an example of processing of the image processing system. [Figure 9] FIG. 1 is a diagram illustrating an example of a functional configuration of an image processing system. [Figure 10] FIG. 10 is a diagram showing an example of a setting screen of the image processing system. [Figure 11] FIG. 10 is a diagram showing an example of processing related to automatic setting of an ROI. [Figure 12] 10 is a flowchart showing an example of processing of the image processing system. [Figure 13] FIG. 1 is a diagram illustrating an example of a network structure of a model. [Figure 14] FIG. 10 is a diagram showing an example of a setting screen of the image processing system. [Figure 15] 10 is a flowchart showing an example of processing of the image processing system. [Figure 16] FIG. 10 is a diagram showing an example of processing related to automatic setting of an ROI. [Figure 17] FIG. 10 is a diagram showing an example of a setting screen of the image processing system. [Figure 18] 10 is a flowchart showing an example of processing of the image processing system. [Figure 19] FIG. 10 is a diagram showing an example of a setting screen of the image processing system. [Figure 20] 10 is a flowchart showing an example of processing of the image processing system. DETAILED DESCRIPTION OF THE INVENTION

[0009] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0010] <CNNについて> First, an overview of a convolutional neural network (CNN) used in general information processing technologies that apply deep learning, including information processing devices according to embodiments of the present disclosure, will be provided. CNN is a technology that repeatedly convolves image data with a filter generated through training, followed by nonlinear computation. Data obtained by convolving image data with a filter and then performing nonlinear computation is also called a feature map. Furthermore, training is performed using training data (training images or data sets) consisting of pairs of input image data and output image data. Simply put, learning is the process of generating, from the training data, filter values that can convert input image data into corresponding output image data with high accuracy.

[0011] When image data has RGB color channels or when a feature map is composed of multiple image data, the filter used for convolution also has multiple channels corresponding to those channels. The process of performing nonlinear operations after convolving a filter with image data (or feature map) is expressed in units called layers. Therefore, when explicitly indicating the feature map and filter corresponding to each layer, they are expressed as, for example, the nth layer feature map and the nth layer filter. Also, for example, a CNN that repeats filter convolution and nonlinear operations three times has a three-layer network structure. Such nonlinear operation processing can be formulated as the relational expression shown below as Equation (1).

[0012]

number

[0013] In Equation (1), W n is the filter of the n-th layer, b n is the bias of the n-th layer, f is a non-linear operator, X n is the feature map of the n-th layer, and * is a convolution operator. Also, the (1) attached as a superscript to each operator indicates that it is the first filter or feature map. The above-mentioned filters and biases are generated by learning and are collectively also referred to as "network parameters". As the non-linear operation, for example, a sigmoid function or ReLU (Rectified Linear Unit) is used. For example, ReLU is represented by the conditional expression shown as Equation (2) below.

[0014]

Equation

[0015] As shown in Equation (2), zero is applied to the negative elements of the input vector X, and the original values are applied to the elements of 0 or more.

[0016] As networks using CNN, ResNet in the field of image recognition and its application RED-Net in the field of super-resolution are well-known. Any of these technologies aims to improve the accuracy of processing by stacking multiple layers of CNN and performing convolution of filters multiple times. For example, ResNet features a network structure with a path that shortcuts the convolutional layer, thereby realizing a multi-layer network of 152 layers and achieving high-precision recognition approaching the human recognition rate. Incidentally, the reason why the processing is made more accurate by multi-layer CNN is simply that by repeating the non-linear operation multiple times, a non-linear relationship between the input and output can be expressed.

[0017] <Learning of CNN> Next, we will explain CNN learning. CNN learning is generally performed by minimizing the objective function shown below as equation (3) for training data consisting of a set of input image data and corresponding correct output image (teacher image) data.

[0018]

number

[0019] In equation (3), L is a loss function that measures the error between the correct answer and the estimated result. Y i is the i-th correct output image data, X i is the i-th input image data. F is a function that collectively represents the calculations (equation (1)) performed in each layer of CNN. θ is the network parameters (filter and bias). ||Z||2 is the L2 norm, or simply the square root of the sum of the squares of the elements of vector Z. n is the number of data sets in the training data. Since the amount of training data is generally large, in stochastic gradient descent (SGD), a portion of the training data is randomly selected and used for training. In addition, various methods are known for minimizing (optimizing) the objective function, such as the momentum method, AdaGrad method, AdaDelta method, and Adam method. The Adam method is given by the relationship shown in equation (4) below.

[0020]

number

[0021] In equation (4), t represents the tth iteration. i t is the i-th network parameter at iteration t, and g is θ i tis the gradient of the loss function L with respect to m. m and v are moment vectors, α is the base learning rate, β1 and β2 are hyperparameters, and ε is a tiny constant to prevent division by zero. Note that there are no specific guidelines for selecting an optimization method in learning, so any method can basically be used, but it is known that there are differences in convergence between methods, which can result in differences in learning time.

[0022] First Embodiment A first embodiment of the present disclosure will be described. In this embodiment, an example of an information processing device (image processing device) that applies information processing (image processing) to a target image using the above-described CNN to reduce degradation that is apparent in the image will be described. Examples of image degradation factors include noise, blur, aberration, compression, low resolution, loss, and contrast reduction due to weather conditions such as fog, haze, snow, and rain at the time of image capture. Image processing for reducing degradation that is apparent in an image includes noise removal, blur removal, aberration correction, loss completion, correction of compression-induced degradation, super-resolution processing for low-resolution images, and processing for correcting contrast reduction due to weather conditions at the time of image capture. The image degradation reduction process in this embodiment is a process of generating or restoring an image with no (or very little) degradation from an image in which degradation is evident, and is also referred to as image restoration process in the following explanation. That is, the image restoration process according to this embodiment is not limited to a process of restoring an image that was originally free of (little) degradation in itself but has been degraded by subsequent amplification, compression / expansion, other image processing, etc., but also includes a process of reducing degradation contained in the original image itself. For convenience, the following description will be given by taking noise as an example of an image degradation factor, and focusing on an example in which noise reduction processing is performed as image restoration processing.

[0023] (Image processing system configuration example) An example of the system configuration of an image processing system to which an image processing device according to this embodiment is applied will be described with reference to Fig. 1. The image processing system shown in Fig. 1 includes a cloud server 200 that generates learning data and learns restoration of image quality degradation, and an edge device 100 that performs degradation restoration for images to be processed. The cloud server 200 and the edge device 100 are connected via a network such as the Internet. Hereinafter, the processing related to the generation of learning data and learning restoration of image quality degradation in the cloud server 200 will also be referred to as degradation restoration learning, and the processing of degradation restoration in the edge device 100 will also be referred to as degradation restoration inference.

[0024] (Edge device hardware configuration) The edge device 100 according to this embodiment acquires RAW image data (Bayer array) input from the imaging device 10 as an input image to be subjected to image restoration processing. Then, the edge device 100 performs degradation restoration inference on the input image to be processed using a trained neural network model (a model constructed based on so-called machine learning) provided by the cloud server 200. In other words, the edge device 100 corresponds to an image processing device that reduces noise in RAW image data by using the model provided by the cloud server 200 and executing a pre-installed image processing application program.

[0025] The edge device 100 has a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, and a large-capacity storage device 104. The edge device 100 also has a general-purpose interface (I / F) 105 and a network I / F 106. The components of the edge device 100 are connected to each other via a system bus 107. The edge device 100 is also connected to an imaging device 10, an input device 20, an external storage device 30, and a display device 40 via the general-purpose I / F 105.

[0026] The CPU 101 uses the RAM 102 as a work memory, executes programs stored in the ROM 103, and performs overall control of each component of the edge device 100 via a system bus 107. The mass storage device 104 is realized by a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various data and image data handled by the edge device 100. The CPU 101 writes data to the mass storage device 104 via the system bus 107, and reads data stored in the mass storage device 104.

[0027] The general-purpose I / F 105 is a serial bus interface based on standards such as USB, IEEE 1394, and HDMI (registered trademark). The edge device 100 may acquire data from an external storage device 30 (e.g., various storage media such as a memory card, a CF card, an SD card, and a USB memory) via the general-purpose I / F 105. The edge device 100 also accepts user instructions from an input device 20 such as a mouse or a keyboard via the general-purpose I / F 105. The edge device 100 also outputs image data processed by the CPU 101 to a display device 40 (e.g., various image display devices such as a liquid crystal display) via the general-purpose I / F 105. The display device 40 may be a display device integrated with a touch panel. The edge device 100 may also acquire data of a captured image (RAW image) to be subjected to noise reduction processing from the imaging device 10 via the general-purpose I / F 105.

[0028] The network I / F 106 is an interface for connecting to a network such as the Internet. The edge device 100 accesses the cloud server 200 via, for example, an installed web browser, and acquires a trained model for degradation restoration inference.

[0029] (Cloud server hardware configuration) The cloud server 200 according to this embodiment is an information processing device (e.g., an image processing device) that provides a network service, typically a cloud service, over a network such as the Internet. More specifically, the cloud server 200 generates training data and performs degradation restoration training, and generates a trained model that stores network parameters and a network structure resulting from the training. The cloud server 200 then provides the generated trained model in response to a request from the edge device 100.

[0030] The cloud server 200 includes a CPU 201, a ROM 202, a RAM 203, a mass storage device 204, and a network I / F 205. These components of the cloud server 200 are connected to each other via a system bus 206.

[0031] The CPU 201 reads out a control program stored in the ROM 202 and executes various processes to control the overall operation of the cloud server 200. The RAM 203 is used as a temporary storage area such as the main memory and work area of the CPU 201. The mass storage device 204 is realized by a storage device (for example, a mass secondary storage device) such as an HDD or SSD, and stores image data and various programs. The network I / F 205 is an interface for connecting to a network such as the Internet. In response to a request from the edge device 100 using a web browser or the like, the network I / F 205 provides the edge device 100 with a trained model in which the above-mentioned network parameters and network structure are stored.

[0032] The edge device 100 and the cloud server 200 may include other components in addition to the components described above, but detailed description of these other components will be omitted.

[0033] In this embodiment, it is assumed that the cloud server 200 downloads a trained model, which is the result of generating training data and performing degradation restoration learning, to the edge device 100, and the edge device 100 performs degradation restoration inference on the input image data to be processed. Note that the above-described system configuration is merely an example and does not limit the configuration of the image processing system according to this embodiment. For example, a configuration may be applied in which the functions of the cloud server 200 are subdivided and the generation of learning data and the degradation restoration learning are performed by separate devices. As another example, a configuration may be applied in which the generation of learning data, degradation restoration learning, and degradation restoration inference are performed in a device (e.g., the imaging device 10) that combines the functions of the edge device 100 and the cloud server 200.

[0034] (System-wide functional configuration) An example of the functional configuration of the image processing system according to this embodiment will be described with reference to FIG. First, a description will be given of an example of the configuration of the edge device 100. As shown in Fig. 2, the edge device 100 has an inference unit 112 and a model storage unit 119. The model storage unit 119 stores a trained model 220 received from the cloud server 200. The inference unit 112 has the function of image restoration processing to reduce image degradation. The inference unit 112 has a degradation restoration unit for inference 115, a model determination unit 113, and a model control unit 114. The degradation restoration unit for inference 115 also includes a model execution unit of a neural network for degradation restoration.

[0035] Here, an example of the processing of the inference degradation restoration unit 115 will be described with reference to FIG. The degradation restoration unit for inference 115 acquires input image data 116 and executes preprocessing 301 on the input image data 116. For example, preprocessing 301 may involve converting the format of the input image data 116 into a data format that can be accepted as input by a trained model to be applied to image processing. As a specific example, preprocessing 301 may involve converting each piece of data into a floating-point number and normalizing the value to 0.0 to 1.0 when the input image data is a 14-bit integer value in a Bayer array. Next, the degradation restoration unit for inference 115 performs model execution 302 using the trained model 220 with the data after preprocessing 301 as input. Then, the degradation restoration unit for inference 115 performs post-processing 303 on the result of model execution 302, and obtains an output image 118 as a result. For example, post-processing 303 may include converting the format of data output by the trained model applied to image processing into a preset output data format (for example, the same format as the input image data 116). As a specific example, post-processing 303 may include converting the value of the model output result data, which is a floating-point number, into a 14-bit integer value.

[0036] An example of the structure of a model applied to image processing of input image data will be described with reference to Fig. 4. In the example shown in Fig. 4, the model is composed of a plurality of convolution layers 401 and a concatenation layer 402. In the convolution layer 401, the convolution operation using the filter expressed by the above-mentioned formula (1) and the nonlinear operation expressed by formula (2) are repeatedly executed a plurality of times. The degradation restoration unit for inference 115 sequentially applies convolutional layers 401 to the input data of the model to calculate a feature map. Next, the degradation restoration unit for inference 115 concatenates the feature map and the input data in the channel direction using the concatenation layer 402. Furthermore, the degradation restoration unit for inference 115 sequentially applies convolutional layers 401 to the concatenation result, and outputs the results of the series of processes from the final layer.

[0037] 2, the inference-use degradation restoration unit 115 uses the trained model 220 received from the cloud server 200 to perform degradation restoration inference on the input image data 116. The model control unit 114 switches the trained model 220 to be applied to the degradation restoration inference by the degradation restoration for inference unit 115. At this time, the model to be switched to is determined by the model determination unit 113. The process of switching the trained model 220 by the model determination unit 113 and the model control unit 114 will be described in detail later.

[0038] Next, a description will be given of an example of the configuration of the cloud server 200. The cloud server 200 includes a degradation applying unit 211 and a learning unit 212. The degradation adding unit 211 adds at least one degradation element to teacher image data extracted from a group of degradation-free teacher images to generate input image data. As described above, in this embodiment, a case where noise is applied as a degradation element is described, so the degradation adding unit 211 adds noise as a degradation element to the teacher image data to generate input image data. In this case, for example, the degradation adding unit 211 may analyze the physical characteristics of the imaging device and add noise corresponding to a wider range of degradation levels than the amount of degradation that can occur in the imaging device to the teacher image data as a degradation element to generate input image data. In this way, by adding degradation levels corresponding to a wider range of degradation levels than the analysis results, it is possible to provide a margin and increase robustness so that the influence of individual differences can be absorbed even in situations where the range of degradation levels varies depending on the individual differences of the imaging device.

[0039] Here, the process of adding degradation to a target image (e.g., a teacher image) will be described with reference to FIG. 5 . The degradation adding unit 211 adds noise based on the physical property analysis result 218 of the imaging device as a degradation element 502 to teacher image data 501 extracted from the teacher image group 217, thereby generating input image data 504. For example, in the example shown in FIG. 5 , the process of adding the degradation element 502 to the teacher image data 501 is schematically shown as an addition process 503 for convenience. Then, the degradation adding unit 211 adds a pair of the teacher image data 501 and input image data 504 generated from the teacher image data 501 to the training data 505. In this way, the degradation adding unit 211 adds the degradation element 502 to each piece of teacher image data included in the teacher image group 217, thereby generating an input image group consisting of multiple pieces of input image data 504, and generates training data 505 from this input image group and the teacher image group. Although noise is used as an example in this embodiment, the degradation adding unit 211 may add any one or a combination of multiple types of degradation elements to the teacher image data, such as the blur, aberration, compression, low resolution, loss, and reduced contrast due to the weather at the time of shooting, as mentioned above.

[0040] The teacher image group 217 may store various types of image data, such as nature photos including landscapes and animals, portraits of people such as portraits or sports photos, and photos of man-made objects such as architecture and products. In this embodiment, the teacher image data is RAW image data, similar to the input image data 116, in which each pixel has a pixel value corresponding to one of the RGB colors. The physical property analysis result 218 of the imaging device includes, for example, the amount of noise generated for each sensitivity of the imaging sensor built into the camera (imaging device) and the amount of aberration generated by the lens. Using information contained in the physical property analysis result 218, it is possible to estimate the degree of image degradation that may occur for each imaging condition. In other words, by applying the degradation estimated for a certain imaging condition to the teacher image data, it is possible to generate an image that is substantially equivalent to the image obtained during imaging.

[0041] 2, the learning unit 212 includes a learning degradation restoration unit 214, an error calculation unit 215, and a model update unit 216. The learning unit 212 acquires learning parameters 219 and performs degradation restoration learning using the learning data generated by the degradation adding unit 211. The learning parameters 219 include initial values of parameters of the neural network model, the network structure of the model, and hyperparameters indicating an optimization method. The learning restoration unit 214 performs image restoration processing on the input image to generate a restored image. The error calculation unit 215 calculates the error between the teacher image and the degraded restored image generated as a result of the degradation restoration process by the learning degradation restoration unit 214. The model update unit 216 updates the parameters of the neural network model applied to the learning degradation restoration unit 214 based on the error calculated by the error calculation unit 215 .

[0042] Here, an example of the processing in the learning unit 212 will be described with reference to FIG. The degradation restoration unit for learning 214 receives data obtained by performing preprocessing 310 on input image data 308 as input to the model execution process 305, and executes degradation restoration processing on the data as the model execution process 305. In the model execution process 305, the learning degradation restoration unit 214 repeats convolution operations and nonlinear operations using the filters expressed by equations (1) and (2) on the input data of the model multiple times, and outputs the results as restoration results 313. Next, the error calculation unit 215 uses the data obtained by performing preprocessing 311 on the teacher image data 309 and the restoration result 313 as inputs to the loss calculation process 314, and executes the loss calculation process 314 to calculate the error therebetween. Thereafter, the model update unit 216 executes the model update process 315 based on the error calculated by the error calculation unit 215, thereby updating the network parameters of the model so that the error becomes smaller (eventually, minimized).

[0043] The configuration shown in FIG. 2 is merely an example, and can be modified or changed as appropriate as long as the functions of each component can be realized. For example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. The configuration shown in FIG. 2 may also be realized by two or more devices. In this case, the devices are connected via a circuit or a wired or wireless network, and perform data communication with each other to perform cooperative operations, thereby realizing each process according to this embodiment.

[0044] The degradation restoration unit for inference 115 initializes the trained model 220 and performs inference by providing input data to the trained model 220 after initialization. The initialization involves various processes to make the target model ready to perform inference on the input data, such as allocating memory to be used by the model and fixing variable parameters in the model. Variable parameters in the model include, for example, the size of the input data (for example, the size indicating the size of the image represented by the image data).

[0045] The inference-use degradation restoration unit 115 performs inference using the inference-use model. Here, an overview of the inference processing executed by the inference-use degradation restoration unit 115 will be described with reference to FIG. To perform inference on an ROI 602 (region to be inferred) set in an input image 601, the degradation restoration unit for inference 115 cuts out a partial region 603 from the input image 601 that includes the ROI 602 and has a data size that allows the inference model to accept input. Reference numeral 604 indicates the process of cutting out the partial region 603 from the input image 601. The degradation restoration unit for inference 115 inputs a partial image 605 cut out from the input image 601 by the cutout process 604 to an inference model 606, and obtains an inference result image 607 as the output of the inference model 606. The degradation restoration unit for inference 115 superimposes an ROI result image 608 corresponding to a partial region in the inference result image 607, the position and size of which correspond to the ROI 602, on the position of the input image 601 corresponding to the ROI 602. Reference numeral 609 indicates the process of superimposing the ROI result image 608 on the position corresponding to the ROI 602 in the input image 601. If the image 601 after the ROI result image 608 has been superimposed is then used as the output image, it becomes possible to obtain an image in which the inference result after degradation restoration is reflected only in the region corresponding to the ROI 602. By applying the above processing, it becomes possible to have the inference model 606 perform inference even if the size of the ROI 602 does not match the input data size of the inference model 606 (even if the size of the input data is larger than the size of the ROI 602). In the example shown in FIG. 6, the ROI result image 608 is superimposed on the input image 601, but in order to preserve the original input image 601, the ROI result image 608 may also be superimposed on a copy of the input image 601.

[0046] Next, the setting and switching of models by the model determination unit 113 and the model control unit 114 will be described. The setting and switching of models by the model determination unit 113 and the model control unit 114 are performed according to the size of the ROI, with each of a plurality of initialized models with different input data sizes as candidates. Specifically, when an ROI is set for an input image by setting a new ROI or updating an existing ROI, the model determination unit 113 determines whether or not it is necessary to set or switch an inference model for the inference degradation restoration unit 115 to perform degradation restoration inference. As a more specific example, the model determination unit 113 may determine whether or not the size of the ROI has been changed, and, if it is determined that the size of the ROI has been changed, determine whether or not it is necessary to switch the model to be applied to degradation restoration inference by the inference degradation restoration unit 115.

[0047] Here, an example of a method for determining an inference model to be applied to degradation restoration inference by the inference degradation restoration unit 115 will be described with reference to FIG. The model determination unit 113 determines, as the model to be set, a model having a smaller input data size from among a series of models having input data sizes larger than the size of the ROI, among a plurality of models that are candidates for the setting target. As a specific example, the model determination unit 113 according to this embodiment determines whether data of the size of the target ROI can be accepted as an input (i.e., whether inference is possible) in order from the model with the smallest input data size. Then, the model determination unit 113 may determine the model that is first determined to be able to accept data of the size of the ROI as an input as the model to be set.

[0048] In the example shown in FIG. 7, an ROI 702 (400 pixels wide, 400 pixels high) is set within an input image 701 (1920 pixels wide, 1080 pixels high). Three initialized inference models 706, 707, and 708 are prepared as candidate models for setting. Reference numerals 703, 704, and 705 in FIG. 7 schematically indicate the sizes of data that the inference models 706, 707, and 708 can accept as input. Specifically, the inference model 706 is configured to be able to accept data of size 703 (200 pixels wide, 100 pixels high). The inference model 707 is configured to be able to accept data of size 704 (600 pixels wide, 400 pixels high). The inference model 708 is configured to be able to accept data of size 705 (1920 pixels wide, 1080 pixels high).

[0049] First, inference model 706 has the smallest input data size, and therefore this inference model 706 is the target for determination. At this time, inference model 706 is not selected as the model to be set because input data size 703 is smaller than the size of ROI 702 in both width and height. Next, inference model 707, which has the next smallest input data size after inference model 706, is the target for determination. At this time, inference model 707 is selected as the model to be set because input data size 704 is equal to or larger than the size of ROI 702 in both width and height. Note that inference model 708 also has input data size 705 that is equal to or larger than the size of ROI 702 in both width and height. However, inference model 707, which has the smaller input data size, is selected as the model to be set first, and therefore inference model 708 is not selected as the model to be set.

[0050] The model control unit 114 sets the model determined by the model determination unit 113 as the inference model for use by the inference degradation restoration unit 115 to perform degradation restoration inference. Specifically, when an inference model for use by the inference degradation restoration unit 115 to perform degradation restoration inference has not been set (for example, during initial ROI setting), the model control unit 114 sets the model determined by the model determination unit 113 as the inference model. Furthermore, when the model determined by the model determination unit 113 as the setting model is different from the inference model currently used by the inference degradation restoration unit 115 for degradation restoration inference, the model control unit 114 switches the inference model to the model determined as the setting model. Note that when the model control unit 114 sets the inference model to be used by the inference degradation restoration unit 115 for degradation restoration inference, the model to be set is not initialized. This is because the initialization of each of the multiple models that are candidates for the setting model has already been performed when the inference degradation restoration unit 115 is initialized. Therefore, according to this embodiment, even if the size of the ROI is changed, it is possible to quickly complete the switching to a model that corresponds to the changed size of the ROI.

[0051] (process) An example of processing of the image processing system according to this embodiment will be described with reference to Fig. 8. The series of processing shown in Fig. 8 is realized, for example, by CPU 101 executing an image processing computer program according to this embodiment, causing the respective functional units shown in Fig. 2 to function. However, all or part of the functional units shown in Fig. 2 may also be realized by being executed by the hardware of CPU 101.

[0052] An example of the processing of the edge device 100 will be described with reference to FIG. 8(A), focusing particularly on the processing related to initialization of the inference degradation restoration unit 115. In S801, the inference degradation restoration unit 115 acquires one model to be subjected to initialization processing from the series of models stored in the model storage unit 119 (for example, a model that has not yet been initialized). In S802, the degradation restoration unit for inference 115 initializes the model acquired in S801. In S803, the inference degradation restoration unit 115 determines whether the termination condition for the process related to the initialization of the model has been satisfied. As a specific example, when the initialization of a series of models stored in the model storage unit 119 has been completed, it may be determined that the termination condition for the process related to the initialization of the model has been satisfied. If the inference degradation restoration unit 115 determines in S803 that the termination condition for the process related to the initialization of the model is not satisfied, the process proceeds to S801. In this case, the process from S801 onwards is executed again for the model that has not yet been initialized. Then, if the inference degradation restoration unit 115 determines in S803 that the termination condition for the process related to the initialization of the model is satisfied, it advances the process to S804. In S804, the inference degradation restoration unit 115 sets the model set as the default among the series of initialized models as the inference model to be applied to the degradation restoration inference process. Then, the inference degradation restoration unit 115 ends the series of processes shown in FIG.

[0053] An example of the processing of the edge device 100 will be described with reference to FIG. 8(B), focusing particularly on an example in which degradation restoration inference processing is executed. In S810, the inference degradation restoration unit 115 acquires input image data. In S811 , the inference degradation restoration unit 115 , the model determination unit 113 , and the model control unit 114 acquire the ROI settings from the setting storage unit 120 . In S812, the model determination unit 113 determines whether or not there has been a change in the setting of the ROI. If the model determination unit 113 determines in S812 that the setting of the ROI has been changed, the process proceeds to S813. In S813, the model determination unit 113 and the model control unit 114 execute a process for switching the inference model to be applied to the degradation restoration inference process. Details of the process of S813 will be described separately later with reference to FIG. 8(C). On the other hand, if the model determination unit 113 determines in S812 that there has been no change in the setting of the ROI, the process proceeds to S814, in which case the process of S813 is skipped. In S814, the degradation restoration unit for inference 115 prepares input data for the inference model. Specifically, if an ROI is set in the input image data based on the ROI setting acquired in S811, the degradation restoration unit for inference 115 cuts out an area corresponding to the ROI from the input image data and sets it as input data for the inference model. On the other hand, if an ROI is not set in the input image data, the degradation restoration unit for inference 115 sets the input image data as is as input data for the inference model. In S815, the degradation restoration unit for inference 115 inputs the input data prepared in S814 to the model for inference, and executes inference (degradation restoration inference) targeting the input data.

[0054] Referring to FIG. 8(C), an example of the processing related to switching of the inference model by the edge device 100, shown as the processing of S813 in FIG. 8(B), will be described. In S820, the model determination unit 113 reads out a list of inference models that are candidates for application to the degradation restoration inference process. Note that in this embodiment, the list of inference models is assumed to be sorted in ascending order of input data size, assuming that an inference model to be applied to the degradation restoration inference process is determined according to the method described with reference to Fig. 7. However, the method for managing the list of inference models is not particularly limited, and the list of inference models may be managed in a state sorted based on other conditions, for example.

[0055] In S821, the model determination unit 113 determines whether or not a determination has been made as to whether or not the series of inference models read out in S820 are to be applied to the degradation restoration inference process. If the model determination unit 113 determines in S821 that a determination has been made as to whether or not a series of inference models are to be applied to the degradation restoration inference process, the model determination unit 113 advances the process to S826. On the other hand, if the model determination unit 113 determines in S821 that it has not yet been determined whether or not a series of inference models are to be applied to the degradation restoration inference process, the model determination unit 113 advances the process to S822.

[0056] In S822, the model determination unit 113 reads out information on one model for which a determination has not yet been made, which will be described later in S823 and S824, from the series of inference models read out in S820. In S823, the model determination unit 113 determines whether the width of the target ROI is equal to or less than the width of the input image of the target model (the model whose information was read in S822) based on the ROI settings acquired in S811 of Figure 8 (B). If the model determination unit 113 determines in S823 that the width of the target ROI is equal to or less than the width of the input image of the target model, the process proceeds to S824. On the other hand, if the model determination unit 113 determines in S823 that the width of the target ROI is not equal to or less than the width of the input image of the target model, the process proceeds to S821.

[0057] In S824, the model determination unit 113 determines whether the height of the target ROI is equal to or less than the height of the input image of the target model (the model whose information was read out in S822) based on the ROI setting acquired in S811 of FIG. 8(B). If the model determination unit 113 determines in S824 that the height of the target ROI is equal to or less than the height of the input image of the target model, the process proceeds to S825. On the other hand, if the model determination unit 113 determines in S824 that the height of the target ROI is not equal to or less than the height of the input image of the target model, the process proceeds to S821.

[0058] In S825, the model determination unit 113 selects the model currently being judged as the inference model to be applied to the degradation restoration inference process. Then, the model control unit 114 sets the model selected by the model determination unit 113 as the inference model to be applied to the degradation restoration inference process by the inference degradation restoration unit 115. As a result, the inference model set for the inference degradation restoration unit 115 is switched to the model selected by the model determination unit 113 (i.e., the model determined to be applied to the restoration inference process).

[0059] In S826, the model determination unit 113 selects the model set as the default as the inference model to be applied to the degradation restoration inference process. Then, the model control unit 114 sets the model selected by the model determination unit 113 (i.e., the default model) as the inference model to be applied to the degradation restoration inference process by the inference degradation restoration unit 115. As a result, the inference model set for the inference degradation restoration unit 115 is switched to the default model selected by the model determination unit 113.

[0060] An example of processing by the image processing system according to this embodiment has been described above. As described above, in the image processing system according to this embodiment, a pre-initialized model is applied when setting a model in accordance with the initial setting of an ROI or when setting (switching) a model in accordance with a change in ROI. By applying such control, no waiting time is required for initializing the target model when setting a model (for example, switching models), so it is possible to quickly complete setting of the model to be used for inference. Due to these characteristics, the image processing system according to this embodiment is particularly compatible with use cases in which the size of an ROI set in an image is successively changed and inference is performed using the changed ROI as the target.

[0061] In this embodiment, an example in which noise is used as a degradation factor has been described, but degradation factors are not necessarily limited to noise. As described above, degradation factors include blurring, aberration, compression, low resolution, defects, and contrast reduction due to weather conditions such as fog, haze, snow, and rain at the time of image capture, and any one of these or a combination of two or more of these may be applied. Furthermore, in the present embodiment, an example of restoration from input image data using a trained model on the edge device side has been described. However, parameters that assist in degradation restoration may also be used. For example, information that estimates the degree of image quality degradation that will occur depending on imaging conditions such as the distance to the subject, focal length, sensor size, and exposure may be stored as a lookup table. Then, for example, when applying the degradation restoration inference process to the input image data, the amount of restoration may be adjusted according to the information stored in the lookup table. In this way, the inference unit 112 of the edge device 100 may adjust the restoration strength for the input image data based on the imaging conditions under which the image represented by the input image data was captured. Furthermore, in this embodiment, an example in which the model performs degradation restoration processing has been described, but the task of the model is not limited to degradation restoration processing.

[0062] <Second embodiment> A second embodiment of the present disclosure will be described below. In the first embodiment described above, an example of a case where a model to be applied to inference is determined based on the setting of an ROI was described. In the first embodiment, by applying such control, when the setting of an ROI is changed, the model to be applied to inference can be switched to a model corresponding to the setting of the ROI. In contrast, in the present embodiment, an example of control that suppresses model switching even when the setting of an ROI is changed, and an example of control that automatically sets an ROI using the detection results of an object in an image, etc. will be described. Note that in the description of this embodiment, attention will be focused on parts that are particularly different from the first embodiment described above, and detailed description of parts that are substantially similar to the first embodiment will be omitted.

[0063] An example of the functional configuration of the image processing system according to this embodiment will be described with reference to Fig. 9. As shown in Fig. 9, the edge device according to this embodiment has a setting unit 901 and a setting storage unit 902. The setting unit 901 accepts setting input via the input device 20 or the network I / F 106, and stores the input setting in the setting storage unit 902.

[0064] Here, an example of a setting screen of the image processing system according to this embodiment will be described with reference to Fig. 10. The setting screen 1000 shown in Fig. 10 is presented to the user by being displayed on the display device 40, for example, and accepts instructions from the user (e.g., instructions related to value selection, etc.) via the input device 20. The setting screen 1000 corresponds to an example of a setting screen for accepting specifications regarding the settings of an inference model from the user. The settings of the inference model specified via the setting screen 1000 are referenced by the model determination unit 903 when determining a model to be applied to the degradation restoration inference process by the inference degradation restoration unit 905.

[0065] An example of an item for setting the inference model is whether or not to fix the size of the inference model. Specifically, if the setting is made to fix the size of the inference model, switching of the inference model is suppressed even if the size of the ROI is changed. Furthermore, if the setting is made to allow the size of the inference model to be variable, a determination is made regarding switching of the inference model when the size of the ROI is changed. The values specified via the setting screen 1000 are stored in the setting storage unit 902. This allows the model determination unit 903 to access the setting storage unit 902 and refer to the values stored in the setting storage unit 902.

[0066] 10 illustrates an example in which settings are input using the display device 40 and the input device 20, but settings may be input by other methods. As a specific example, settings may be input to the edge device 900 by information transmitted via a network to which the network I / F 106 is connected.

[0067] The object detection unit 906 performs various analyses (e.g., image analysis) using the input image data 116 as input to detect the position and size of an object (e.g., an object to be detected) in the image represented by the input image data 116, and stores the detection results in the setting storage unit 902.

[0068] An example of processing for automatically setting an ROI using the result of detecting an object in an image will be described with reference to Fig. 11. Fig. 11 shows an example in which a ship captured as a subject in an image is detected, and an ROI is set in an area in the image corresponding to the ship.

[0069] 11(A), as a result of object detection performed on an image 1100, a ship is detected in the image, thereby identifying an area 1101 corresponding to the ship, and an ROI 1102 is set for the area 1101. Then, a degradation restoration inference process is performed on an area including at least the area 1101 corresponding to the ship in the image in which the ROI 1102 is set, and the result is superimposed on the area 1101. For example, Fig. 11(B) shows an example of the result of performing degradation restoration inference processing on an area 1101 in which an ROI 1102 is set in the image 1100 shown in Fig. 11(A). Specifically, in an image 1110 after degradation restoration inference processing shown in Fig. 11(B), the result of performing noise removal on the area 1101 corresponding to the area 1101 in Fig. 11(A) is superimposed on the area 1111.

[0070] 11(A) is changed, the inference model applied to the degradation restoration inference process is switched according to the changed size of the ROI, as in the first embodiment. Then, the degradation restoration inference process is executed for the area corresponding to the changed ROI using the switched inference model, and the result is superimposed on the area corresponding to the changed ROI.

[0071] 11(A) and 11(B), for convenience, areas 1101, 1102, and 1111 are indicated by dashed lines, but when an image is actually displayed, information indicating these areas does not necessarily need to be visualized and displayed. As a specific example, when presenting these areas to the user, the areas in the image may be visualized.

[0072] (process) 12, an example of processing of the image processing system according to this embodiment will be described, focusing particularly on the degradation restoration inference processing of the edge device 900. Note that the example shown in Fig. 12 will be described focusing on parts that are particularly different from the first embodiment, and detailed description of parts that are substantially the same as those in the first embodiment will be omitted.

[0073] In S1201, the object detection unit 906 performs image analysis related to object detection on input image data to detect an object in the image, and attempts to identify the position and size of the object in the image. In S1202, the object detection unit 906 determines whether or not an object was detected in the image in S1201. If the object detection unit 906 determines in S1202 that an object has been detected in the image, the process proceeds to S1203. In S1203, the setting unit 901 automatically sets an ROI for a region in the image corresponding to the object, based on the result of detecting the object in the image in S1201. On the other hand, if the object detection unit 906 determines in S1202 that an object has not been detected in the image, the process proceeds to S1204. In S1204, the setting unit 901 sets the ROI according to default settings. As a specific example, the setting unit 901 may set the ROI for an area corresponding to the entire target image. Of course, this control is merely an example, and the type of control to be applied as the default setting may be set appropriately depending on the use case.

[0074] In S1205, the model determination unit 903 determines whether or not there has been a change in the setting of the ROI. If the model determination unit 903 determines in S1205 that the setting of the ROI has been changed, the process proceeds to S1206. On the other hand, if the model determination unit 903 determines in S1205 that there has been no change in the setting of the ROI, the process proceeds to S814, in which case the processes of S1206 and S813 are skipped.

[0075] In S1206, the model determination unit 903 reads the setting value of the setting stored in the setting storage unit 902 indicating whether the size of the inference model is to be fixed, and determines whether the setting value indicates that the size of the inference model is to be variable. If the model determination unit 903 determines that the setting value read in S1206 indicates that the size of the inference model is variable, it proceeds to S813. On the other hand, if the model determination unit 903 determines that the setting value read in S1206 indicates that the size of the inference model is not variable (is fixed), the process proceeds to S814. In this case, the process of S813 is skipped.

[0076] As described above, in the image processing system according to this embodiment, the edge device 900 suppresses switching of the inference model when the size of the inference model is set to be fixed, even if the setting of the ROI is changed. This mechanism allows the user to intentionally control whether or not to apply switching of the inference model in response to a change in the ROI. Furthermore, the image processing system according to this embodiment makes it possible to automatically set an ROI using the results of object detection from within an image. This is expected to reduce the user's effort compared to when the ROI is set manually.

[0077] <Third embodiment> As a third embodiment of the present disclosure, an example of a network structure of the trained model 220 applied to the degradation restoration inference process by the inference degradation restoration unit 905 will be described, focusing on the case where a recurrent configuration in which the previous inference result is used is applied. Note that in this embodiment, the description will focus on parts that are particularly different from the first and second embodiments described above, and detailed description of parts that are substantially the same as the first and second embodiments will be omitted.

[0078] An example of the network structure of the trained model 220 applied to the degradation restoration inference process by the inference degradation restoration unit 905 according to this embodiment will be described with reference to FIG. 13. In the example shown in FIG. 13, the input data (hereinafter also referred to as inference input data 1302) to the trained model (neural network) is assumed to be data obtained by performing preprocessing 301 on the input image data 116. The neural network applied as the inference model 1303 shown in FIG. 13 is assumed to be a neural network with a recurrent configuration. A recurrent configuration is a configuration in which part of the previous inference result is reused as the current input data. By reusing the previous inference result as the current input data, it is expected that the consistency and continuity of the inference results in the time direction can be improved.

[0079] A previous recurrent output 1301 and inference input data 1302 are input to an inference model 1303. The previous recurrent output 1304 is data that is output as part of the previous inference result. Note that in this embodiment, image data having the same image size and number of channels as the inference input data 1302 is applied as the previous recurrent output 1304, but the type of data applied as the previous recurrent output 1304 is not limited. Furthermore, the inference model 1303 outputs a recurrent output 1304 and inference output data 1305 as inference results. The inference output data 1305 is image data that has been subjected to degradation restoration inference processing and is output as an inference result. The recurrent output 1304 corresponds to a portion of the image data that has been output as an inference result (image data that has been subjected to degradation restoration inference processing), and is used as the previous recurrent output 1301 that corresponds to a portion of the input data for the next inference.

[0080] Note that there is no previous recurrent output 1301 during the initial inference or during inference immediately after switching models. In such cases, for example, initial data (in other words, dummy data) such as data equivalent to a black image may be used as the previous recurrent output 1301. As another example, there may be used, as the previous recurrent output 1301, a recurrent output 1304 that is part of the data output as the previous inference result by the inference model before switching. Note that in this embodiment, with regard to control when there is no previous recurrent output 1301, one of the above-mentioned operations is configured to be switchable by setting.

[0081] Here, an example of a setting screen of the image processing system according to the embodiment will be described with reference to Fig. 14. The setting screen 1400 shown in Fig. 14 is presented to the user by being displayed on the display device 40, for example, and accepts instructions from the user (e.g., instructions related to value selection, etc.) via the input device 20. The setting screen 1400 corresponds to an example of a setting screen for accepting specifications regarding the settings of an inference model from the user. The settings of the inference model specified via the setting screen 1400 are referenced by the model determination unit 903 when determining a model to be applied to the degradation restoration inference process by the inference degradation restoration unit 905.

[0082] Examples of inference model settings include whether or not to reuse the output of the inference model before switching as recurrent information that is input to the inference model when the inference model is switched. Specifically, it is possible to specify a setting to reuse the output of the inference model before switching as the recurrent information that is input to the inference model. In this case, when the inference model is switched, if the recurrent information before switching is available, the recurrent information is reused as input to the inference model after switching. It is also possible to specify a setting to create recurrent information to be input to the inference model. In this case, the inference models before and after the switch are caused to perform inference in parallel, and recurrent information is generated based on the results of a predetermined number of inferences performed with the inference model after the switch. After the recurrent information is generated, the inference model is switched. In this case, since the inference models before and after the switch perform inference in parallel, the usage of computer resources such as the CPU and GPU of the image processing device tends to be greater than when only one of the models performs inference. Therefore, for example, when the inference models before and after the switch perform inference in parallel, the frame rate related to the inference may be limited depending on the usage status of the computer resources used for the inference. By applying such control, it is also possible to limit the usage of computer resources associated with the execution of inference. It is also possible to specify a setting to use initial data as recurrent information to be input to the inference model. In this case, when the inference model is switched, initial data such as a black image is used as recurrent information to be input to the inference model after the switch. The values specified via the setting screen 1400 are stored in the setting storage unit 902. This allows the model determination unit 903 to access the setting storage unit 902 and refer to the values stored in the setting storage unit 902.

[0083] 14 illustrates an example in which settings are input using the display device 40 and the input device 20, but settings may be input by other methods. As a specific example, settings may be input to the edge device 900 by information transmitted via a network to which the network I / F 106 is connected.

[0084] (process) 15, an example of processing of the image processing system according to this embodiment will be described, focusing particularly on the degradation restoration inference processing of the edge device 900. Note that the example shown in Fig. 15 will be described focusing on parts that are particularly different from the first embodiment, and detailed description of parts that are substantially the same as those in the first embodiment will be omitted.

[0085] In S1501, the degradation restoration unit for inference 115 prepares input data for the inference model. Specifically, if an ROI is set in the input image data based on the ROI setting acquired in S811, the degradation restoration unit for inference 115 cuts out an area corresponding to the ROI from the input image data and uses it as input data for the inference model. On the other hand, if an ROI is not set in the input image data, the degradation restoration unit for inference 115 uses the input image data as is as input data for the inference model. Furthermore, the degradation restoration unit for inference 115 also uses the stored previous recurrent output as input data for the inference model.

[0086] Here, if the setting of the recurrent information at the time of model switching read from the setting storage unit 902 indicates reuse of the recurrent output before the switching, the degradation restoration unit for inference 115 reuses the recurrent output before the model switching as the previous recurrent output. Also, if the setting of the recurrent information indicates use of initial data, the degradation restoration unit for inference 115 sets predetermined initial data, such as data equivalent to a black image, as the previous recurrent output. In these settings, in S1502, the degradation restoration unit for inference 115 inputs the input data prepared in S1501 to the inference model after the switching, and performs inference (degradation restoration inference) targeting the input data. Furthermore, if the setting of the recurrent information indicates the creation of recurrent information, the inference degradation restoration unit 115 performs inference a predetermined number of times with the inference model after switching, generates recurrent information based on the results of the predetermined number of inferences, and then switches the model. In this case, in S1502, the inference degradation restoration unit 115 causes inference to be performed in parallel with the inference model before switching until inference has been performed a predetermined number of times with the inference model after switching. By applying such control, it is possible to restore degradation of the target image using the inference results of the inference model before switching, even in a state before a recurrent output based on the inference results of the inference model after switching is created.

[0087] As described above, in the image processing system according to this embodiment, in a situation where a network structure is applied in which the previous recurrent output is used as input to an inference model, the user can set the method for preparing the previous recurrent output. This mechanism makes it possible to appropriately switch the system behavior when switching models so that it becomes the behavior desired by the user.

[0088] <Fourth embodiment> As a fourth embodiment of the present disclosure, an example of a case where inference is performed on an area including multiple ROIs and an example of a case where an initialized model can be added by setting will be described. Note that in this embodiment, the description will focus on parts that are particularly different from the first to third embodiments described above, and detailed description of parts that are substantially the same as the first to third embodiments will be omitted.

[0089] Fig. 16 shows an example of a situation where multiple objects are detected in an image and an ROI is automatically set for each of the multiple objects. Fig. 16 shows an example of a situation where multiple ships captured as subjects in an image are each detected and an ROI is set in an area in the image corresponding to each of the multiple ships. Specifically, as shown in FIG. 16(A), object detection is performed on image 1600, and two ships are detected in the image, thereby identifying regions 1601 and 1602 corresponding to the two ships. Then, in the example shown in FIG. 16(A), instead of setting ROIs individually for the detection results of the two objects (the two ships), a single ROI 1603 encompassing the two objects is set. By applying this control, it becomes possible to limit the number of inferences using the inference model to a single inference targeting ROI 1603. Then, a degradation restoration inference process is performed on a region in the image in which ROI 1603 is set, the region including at least regions 1601 and 1602 corresponding to the two ships, and the result is superimposed on the regions 1601 and 1602. For example, Fig. 16(B) shows an example of the results of degradation restoration inference processing performed on regions 1601 and 1602 in which ROI 1603 is set in image 1600 shown in Fig. 16(A). In Fig. 16(B), the region indicated by reference numeral 1613 corresponds to the region corresponding to ROI 1603 shown in Fig. 16(A). Also, in Fig. 16(B), the regions indicated by reference numerals 1611 and 1612 correspond to regions 1601 and 1602 in Fig. 16(A). That is, in image 1610 after degradation restoration inference processing shown in Fig. 16(B), the results of noise removal performed on regions 1601 and 1602 are superimposed on regions 1611 and 1612 included in region 1613.

[0090] 16(A) is changed, the inference model applied to the degradation restoration inference process is switched according to the changed size of the ROI, as in the first embodiment. Then, the degradation restoration inference process is executed for the area corresponding to the changed ROI using the switched inference model, and the result is superimposed on the area corresponding to the changed ROI.

[0091] In this embodiment, an example in which an ROI is set for an area including each of a plurality of objects detected in an image has been described, but an ROI may be set for each of the areas of the plurality of objects individually. In this case, inference using an inference model is performed for each ROI individually. 16(A) and 16(B), for convenience, areas 1601, 1602, 1611, and 1612 are indicated by dashed lines, but when an image is actually displayed, information indicating these areas does not necessarily need to be visualized and displayed. As a specific example, when presenting these areas to the user, the areas in the image may be visualized.

[0092] Here, an example of a setting screen of the image processing system according to the embodiment will be described with reference to Fig. 17. The setting screen 1700 shown in Fig. 17 is presented to the user by being displayed on the display device 40, for example, and accepts instructions from the user (e.g., instructions related to value selection, etc.) via the input device 20. The setting screen 1700 corresponds to an example of a setting screen for accepting specifications related to the setting of an inference model from the user. Examples of inference model settings include settings regarding how to apply inference using the inference model when multiple objects are detected in an image. Specifically, it is possible to specify a setting to perform inference on the detection results of multiple objects from an image at once. In this case, one ROI is set to include the area corresponding to each of the multiple objects, and inference is performed using the inference model for that single ROI. It is also possible to specify settings to perform inference individually for each of the detection results of multiple objects in an image. In this case, an ROI is set individually for each of the detection results of multiple objects in an image, and inference is performed individually using an inference model for each ROI.

[0093] Another example of an item for setting an inference model is setting operations related to adding an inference model to be used for inference. For example, when an ROI is set or updated and there is no initialized model corresponding to the ROI conditions (e.g., size), it is possible to specify a setting to automatically initialize a new inference model to match the ROI. Furthermore, conditions for such automatic initialization of the inference model may be specified. As a specific example, the above-described automatic initialization setting may be specified for ROIs that are set or updated with a frequency equal to or greater than a threshold within a predetermined time. In this case, if there is no initialized model corresponding to the size of the ROI set or updated with a frequency equal to or greater than a threshold within a predetermined time, a new inference model capable of accepting data of the ROI size as input is initialized. Note that the process related to model initialization in this case can be executed in parallel with inference using the model before switching, for example, to prevent waiting time associated with initialization when switching models. It is also possible to manually specify settings for adding an inference model. In this case, specifications for the conditions of the inference model to be added (e.g., the size of the input data) are accepted via the setting screen 1700, and the inference model to be added is initialized based on the specifications. It is also possible to specify a setting that restricts the addition of inference models. In this case, even if there is no initialized model that matches the ROI conditions (e.g., size), a new inference model will not be added. The values specified via the setting screen 1700 are stored in the setting storage unit 902. This allows the model determination unit 903 to access the setting storage unit 902 and refer to the values stored in the setting storage unit 902.

[0094] 17 illustrates an example in which settings are input using the display device 40 and the input device 20, but settings may be input by other methods. As a specific example, settings may be input to the edge device 900 by information transmitted via a network to which the network I / F 106 is connected.

[0095] (process) 18, an example of processing of the image processing system according to this embodiment will be described, focusing particularly on the degradation restoration inference processing of the edge device 900. Note that the example shown in FIG. 18 will be described focusing on parts that are particularly different from the first to third embodiments, and detailed description of parts that are substantially the same as the first to third embodiments will be omitted.

[0096] In S1800, the object detection unit 906 determines whether or not multiple objects were detected in the image in S1201. If the object detection unit 906 determines in S1800 that multiple objects have been detected in the image, the process proceeds to S1801. On the other hand, if the object detection unit 906 determines in S1800 that multiple objects have not been detected in the image (that one object has been detected), the process proceeds to S1203.

[0097] In S1801, the setting unit 901 determines whether or not a setting has been made regarding the method of applying inference using an inference model, such that inference is performed collectively on the detection results of multiple objects from an image. If the setting unit 901 determines in S1801 that a setting has been made to perform inference collectively on the detection results of a plurality of objects in an image, the process proceeds to S1803. In S1803, the setting unit 901 sets ROIs for regions that include regions corresponding to each of the plurality of detected objects, and the process proceeds to S1804. On the other hand, if the model determination unit 903 determines in S1801 that the setting to perform collective inference on the detection results of multiple objects from the image is not made, the process proceeds to S1802. In S1802, the setting unit 901 sets ROIs individually for the regions corresponding to the multiple detected objects, and the process proceeds to S1804.

[0098] In S1804, the setting unit 901 determines whether or not settings have been made for automatic initialization of an inference model targeted at an ROI that has been set or updated frequently. If the setting unit 901 determines in S1804 that settings have been made for automatic initialization of an inference model targeted at an ROI that has been set or updated frequently, it proceeds to S1805. On the other hand, if the setting unit 901 determines in S1804 that no settings have been made for automatic initialization of an inference model targeting an ROI that has been set or updated frequently, it proceeds to S1205.

[0099] In S1805, the setting unit 901 determines whether or not the target ROI corresponds to an ROI that has been set or updated with a frequency equal to or greater than a threshold within a predetermined time period. If the setting unit 901 determines in S1805 that the target ROI corresponds to an ROI that has been set or updated with a frequency equal to or greater than a threshold within a predetermined time, the process proceeds to S1806. On the other hand, if the setting unit 901 determines in S1805 that the target ROI is not an ROI that has been set or updated with a frequency equal to or greater than the threshold within the predetermined time, the process proceeds to S1205.

[0100] In S1806, the model determination unit 903 determines whether the difference between the size of the target ROI and the size of the input data of the inference model (initialized model) is within a predetermined range. If the model determination unit 903 determines in S1806 that the difference between the size of the target ROI and the size of the input data of the inference model is within a predetermined range, it proceeds to S1205. On the other hand, if the model determination unit 903 determines in S1806 that the difference between the size of the target ROI and the size of the input data of the inference model is not within the predetermined range, it proceeds to S1807. In S1807, the model determination unit 903 newly initializes a model that can accept data of the size of the target ROI as input. In this case, in the process of S813, the inference model applied to the degradation restoration inference process is switched to the model newly initialized in S1807.

[0101] As described above, the image processing system according to this embodiment can apply control for automatically setting ROIs so that inference is performed simultaneously on multiple regions in an image according to specified settings. By applying such control, it is possible to reduce the number of inferences performed. Furthermore, if there is no initialized model that matches the conditions of a set or updated ROI, it is also possible to apply control for automatically initializing a new model according to those conditions. In particular, applying such control for automatic model initialization to ROIs that are set or updated relatively frequently can be expected to further speed up processing related to inference using an inference model.

[0102] <Fourth embodiment> As a fourth embodiment of the present disclosure, an example will be described in which, when an ROI is set or updated, if the size of the ROI is close to the size of data that an initialized model can accept as input, the size of the ROI is adjusted to match the model. Note that in this embodiment, the description will focus on parts that are particularly different from the first to fourth embodiments described above, and detailed description of parts that are substantially the same as the first to fourth embodiments will be omitted.

[0103] Fig. 19 is a diagram showing an example of a screen for setting an ROI in a target image. The setting screen 1900 shown in Fig. 19 is presented to the user by being displayed on the display device 40, for example, and accepts instructions from the user (e.g., instructions related to value selection, etc.) via the input device 20. The setting of the ROI specified via the screen 1900 shown in Fig. 19 is used in the degradation restoration inference process by the inference degradation restoration unit 115 and in the process related to the determination of the model to be set by the model determination unit 113.

[0104] Fig. 19(A) shows a schematic diagram of a situation in which a user specifies an ROI in a target image via a screen 1900. Specifically, in the example shown in Fig. 19(A), an input from the user is received via a touch panel, and an ROI 1901 is specified so as to include an object 1903 (a house) captured as a subject in the image. 19(B) schematically illustrates a situation in which the size of ROI 1901 specified in FIG. 19(A) is automatically adjusted to match the size of the input data of the initialized model. Specifically, the setting unit 121 determines whether there is an initialized model whose input data size differs from the size of ROI 1901 by a predetermined range in width and height. If there is an initialized model whose width and height differ from the size of ROI 1901 by a predetermined range, the setting unit 121 adjusts the size of ROI 1901 to match the size of the input data of the model. ROI 1902 shown in FIG. 19(B) illustrates the ROI after the size adjustment. 19(B), the size of ROI 1902 is adjusted to be smaller than the unadjusted ROI 1901. By applying such control, the area to be inferred becomes smaller, which is expected to further improve the processing speed related to the inference. The button 1904 is an input interface for receiving instructions from the user regarding the reflection of the ROI settings specified via the screen 1900. For example, when the user accepts the automatically adjusted ROI settings as shown in Fig. 19(B), the user can reflect the settings by pressing the button 1904. Furthermore, the ROI that has been automatically adjusted may be updated or reset by a new user operation. Furthermore, if the ROI is updated or reset within a predetermined time after the automatic adjustment of the ROI, the automatic adjustment of the ROI may be suppressed. By applying such control, it becomes possible to prevent a situation in which the result of the ROI adjustment is overwritten by the automatic adjustment of the ROI when the user intentionally adjusts the size of the ROI. The values specified via the setting screen 1900 are stored in the setting storage unit 120. This allows the model determination unit 113 to access the setting storage unit 120 and refer to the values stored in the setting storage unit 120.

[0105] 19 illustrates an example in which settings are input using the display device 40 and the input device 20, but settings may be input by other methods. As a specific example, settings may be input to the edge device 100 by information transmitted via a network to which the network I / F 106 is connected.

[0106] (process) 20, an example of processing of the image processing system according to this embodiment will be described, focusing particularly on processing when an ROI is set by an operation via the edge device 900. Note that the example shown in FIG. 20 will be described focusing on parts that are particularly different from the first to fourth embodiments, and detailed description of parts that are substantially the same as the first to fourth embodiments will be omitted.

[0107] In S2000, the setting unit 121 displays a menu screen on the display device 40. On the menu screen, as illustrated with reference to FIG. 19, a display area for the input image and a setting completion button 1904 are displayed. In S2001, the setting unit 121 updates the image displayed in the display area of the input image with the latest input image. In S2002, the setting unit 121 determines whether or not an ROI has been set. If the setting unit 121 determines in S2002 that the ROI has not been set, the process proceeds to S2003. On the other hand, if the setting unit 121 determines in S2002 that the ROI has been set, the process proceeds to S2005. Note that the setting of an ROI is performed by the user specifying an area in the input image displayed on the menu screen by performing an operation such as drag and drop via a touch panel, as described with reference to Fig. 19. Of course, the method of setting an ROI described with reference to Fig. 19 is merely an example, and the method is not particularly limited as long as it is possible to set an ROI for an area in a target image.

[0108] In S2005, the setting unit 121 determines whether a predetermined time has passed since the previous automatic adjustment of the ROI. If the setting unit 121 determines in S2005 that the predetermined time has not elapsed since the previous automatic adjustment of the ROI, the process proceeds to S2003. On the other hand, if the setting unit 121 determines in S2005 that the predetermined time has elapsed since the previous automatic adjustment of the ROI, the process proceeds to S2006. In S2006, the setting unit 121 reads information on a series of initialized models that are candidates for inference models for performing degradation restoration inference.

[0109] In S2007, the setting unit 121 determines whether or not a series of determination processes, which will be described later as S2008 to S2010, have been executed for all models whose information has been read out in S2006. If the setting unit 121 determines in S2007 that a series of determination processes has been executed for all models, the process proceeds to S2003. On the other hand, if the setting unit 121 determines in S2007 that the series of determination processes have not been executed for all models (that is, there are still models for which the determination processes have not been executed), the process proceeds to S2008.

[0110] In S2008, the setting unit 121 reads out information on models that have not yet been subjected to judgment, from among the series of models whose information has been read out in S2006. In S2009, the setting unit 121 determines whether or not the difference between the width of the set ROI and the width of the input image of the model whose information was read out in S2008 is within a predetermined range. If the setting unit 121 determines in S2009 that the difference between the width of the ROI and the width of the input image of the model is within the predetermined range, the process proceeds to S2010. On the other hand, if the setting unit 121 determines in S2009 that the difference between the width of the ROI and the width of the input image of the model is not within the predetermined range, the process proceeds to S2007. In S2010, the setting unit 121 determines whether or not the difference between the height of the set ROI and the height of the input image of the model whose information was read out in S2008 is within a predetermined range. If the setting unit 121 determines in S2010 that the difference between the height of the ROI and the height of the input image of the model is within the predetermined range, the process proceeds to S2011. On the other hand, if the setting unit 121 determines in S2010 that the difference between the height of the ROI and the height of the input image of the model is not within the predetermined range, the process proceeds to S2007. In S2011, the setting unit 121 adjusts the width and height of the set ROI to match the width and height of the model whose information was read out in S2008.

[0111] In S2003, the setting unit 121 determines whether or not the conditions for completing the setting of the ROI are satisfied. As a specific example, the setting unit 121 may determine that the conditions for completing the setting of the ROI are satisfied when the setting completion button 1904 is pressed on the screen 1900 illustrated in Fig. 19 . If the setting unit 121 determines in S2003 that the condition for completing the setting of the ROI is not satisfied, the process proceeds to S2001. In this case, the process from S2001 onwards is executed again. On the other hand, if the setting unit 121 determines in S2003 that the condition for completing the setting of the ROI is satisfied, the process proceeds to S2004. In S2004, the setting unit 121 saves the setting of the ROI in the setting saving unit 120, and then ends the series of processes shown in FIG.

[0112] As described above, in the image processing system according to this embodiment, when the size of an ROI is close to the size of data that an initialized model can accept as input, the size of the ROI is adjusted to match the model. By applying such control, it becomes possible to automatically select a model that is more suitable for executing inference processing targeting an area corresponding to an ROI set in an image (for example, a model that enables faster inference).

[0113] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0114] The disclosure of this embodiment also includes the following configuration and program. (Configuration 1) An information processing device characterized by having: a setting means for setting an inference target area in an input image that is to be the subject of inference by a model constructed based on machine learning; and a determination means for determining a model to be applied to inference using the input image as input data from a plurality of models with different input data sizes that have been initialized to make inference on input data executable, according to the size of the inference target area set by the setting means. (Configuration 2) The information processing device described in Configuration 1, characterized in that the determination means determines, from among the plurality of models, a series of models whose input data size is larger than the inference target area set by the setting means, a model whose input data size is smaller as the model to be applied to inference using the input image as input data. (Configuration 3) An information processing device according to configuration 1 or 2, characterized in that it has a switching means for switching the model to be applied to inference using the input image as input data to a model determined by the determination means in accordance with the size when the size of the inference target area is changed. (Configuration 4) The information processing device described in Configuration 3 is characterized in that, when a setting is made to restrict model switching, the switching means suppresses switching of the model to be applied to inference using the input image as input data, even if the size of the inference target area is changed. (Configuration 5) The information processing device according to any one of configurations 1 to 4, wherein the model is a model that performs image processing on a target image to restore degradation that has become apparent in the image. (Configuration 6) The information processing device according to configuration 5, wherein the degradation includes at least one of noise, compression, low resolution, blur, aberration, loss, and reduced contrast. (Configuration 7) An information processing device described in any one of configurations 1 to 6, characterized in that the multiple models that have been initialized include a recurrent configuration model that uses the results of previously executed inference as input. (Configuration 8) An information processing device according to configuration 7, characterized in that when the size of the inference target area is changed and the model to be applied to inference using the input image as input data is switched to a model of the recurrent configuration, inference is performed in parallel using the model before switching and the model after switching, and after a predetermined number of inferences have been performed using the model after switching, the model is switched. (Configuration 9) An information processing device as described in configuration 8, characterized in that when switching the model to be applied to inference using the input image as input data, during a period in which inference is performed in parallel using the model before the switch and the model after the switch, the frame rate of the processing related to the inference is limited depending on the usage status of the resources used for the inference. (Configuration 10) An information processing device described in any one of configurations 7 to 9, characterized in that when the size of the inference target area is changed and the model applied to inference using the input image as input data is switched to a model of the recurrent configuration, the model after the switch uses as input the results of inference previously performed by the model before the switch. (Configuration 11) An information processing device described in any one of configurations 1 to 10, characterized in that when multiple inference target areas are set by the setting means, the determination means determines a model to be applied to inference using the input image as input data from the multiple models depending on the size of the area containing the multiple inference target areas. (Configuration 12) An information processing device described in any one of configurations 1 to 11, characterized in that it has an initialization means for newly initializing a model whose input data size is larger than the size of the inference target area set by the setting means, and the determination means switches the method of determining a model to be applied to inference using the input image as input data between a method of determining from the multiple models and a method of determining a model newly initialized by the initialization means, depending on the setting related to the model determination method. (Configuration 13) The information processing device described in Configuration 12, characterized in that the initialization means, when the frequency of setting the inference target area is equal to or greater than a threshold and the difference between the size of the inference target area and the input data size of the initialized model is outside a predetermined range, newly initializes a model whose input data size is larger than the size of the inference target area whose setting frequency is equal to or greater than a threshold. (Configuration 14) An information processing device described in any one of configurations 1 to 13, characterized in that when the set inference target area is changed and the difference between the size of the inference target area after the change and the input data size of the initialized model is within a predetermined range, the setting means adjusts the size of the inference target area after the change to the input data size. (Configuration 15) The information processing device described in Configuration 14, characterized in that the setting means limits the adjustment of the size of the inference target area after the change if the inference target area is changed again within a predetermined time after adjusting the size of the inference target area. (Configuration 16) The information processing device according to any one of configurations 1 to 15, characterized in that the setting means sets the inference target area in the input image based on the detection result of an object in the input image. (Program 1) A program for causing a computer to function as an information processing device, characterized by having a setting means for setting an inference target area in an input image that is the subject of inference using a model constructed based on machine learning, and a determination means for determining a model to be applied to inference using the input image as input data from a plurality of models with different input data sizes that have been initialized to make inference on input data possible, depending on the size of the inference target area set by the setting means. [Explanation of symbols]

[0115] 100 Edge Devices 112 Reasoning Department 113 Model Determination Unit 114 Model Control Unit 121 Settings

Claims

1. A setting means for setting an inference target region in an input image that is to be inferred by a model constructed based on machine learning; a determination means for determining a model to be applied to inference using the input image as input data from a plurality of models having different input data sizes, the models having been initialized to be in a state where inference on input data can be performed, in accordance with the size of the inference target area set by the setting means; and The determining means determines, from among the plurality of models, a series of models having input data with a size larger than the inference target region set by the setting means, a model with a smaller input data size as a model to be applied to inference using the input image as input data. An information processing device characterized by:

2. A setting means for setting an inference target area in an input image that is to be inferred by a model constructed based on machine learning; a determination means for determining a model to be applied to inference using the input image as input data from a plurality of models having different input data sizes, the models having been initialized to be in a state where inference on input data can be performed, in accordance with the size of the inference target area set by the setting means; and The initialized models include a recurrent configuration model that uses the results of a previously performed inference as input. An information processing device characterized by:

3. 3. The information processing device according to claim 1, further comprising a switching means for switching, when the size of the inference target area is changed, the model to be applied to inference using the input image as input data to the model determined by the determination means in accordance with the size.

4. 4. The information processing device according to claim 3, wherein the switching means suppresses switching of the model to be applied to inference using the input image as input data, even if the size of the inference target area is changed, when a setting to restrict model switching is set.

5. 3. The information processing apparatus according to claim 1, wherein the model is a model that performs image processing on a target image to restore degradation that has become apparent in the image.

6. The information processing device according to claim 5 , wherein the degradation includes at least one of noise, compression, low resolution, blur, aberration, loss, and reduced contrast.

7. 3. The information processing device according to claim 2, wherein when the size of the inference target area is changed and the model to be applied to inference using the input image as input data is switched to the recurrent configuration model, inference is performed in parallel using the model before the switch and the model after the switch, and the model is switched after a predetermined number of inferences have been performed using the model after the switch.

8. 8. The information processing device according to claim 7, characterized in that, when switching the model to be applied to inference using the input image as input data, during a period in which inference is performed in parallel using the model before the switch and the model after the switch, the frame rate of the processing related to the inference is limited depending on the usage status of resources used for the inference.

9. 3. The information processing device according to claim 2, characterized in that when the size of the inference target area is changed and the model applied to inference using the input image as input data is switched to the recurrent configuration model, the switched model uses as input the results of inference previously performed by the model before the switch.

10. 3. The information processing device according to claim 1, wherein when a plurality of inference target regions are set by the setting means, the determining means determines a model to be applied to inference using the input image as input data from the plurality of models in accordance with the size of an area that includes the plurality of inference target regions.

11. an initialization means for newly initializing a model in which the size of input data is larger than the size of the inference target area set by the setting means; The determination means switches the method of determining a model to be applied to inference using the input image as input data between a method of determining from the plurality of models and a method of determining a model newly initialized by the initialization means, depending on a setting related to a method of determining a model.

3. The information processing device according to claim 1, wherein the information processing device is a computer.

12. 12. The information processing device according to claim 11, wherein the initialization means, when the frequency of setting the inference target area is equal to or greater than a threshold and the difference between the size of the inference target area and the input data size of the initialized model is outside a predetermined range, newly initializes a model whose input data size is larger than the size of the inference target area whose setting frequency is equal to or greater than the threshold.

13. 3. The information processing device according to claim 1, wherein when the set inference target area is changed and a difference between the size of the inference target area after the change and the input data size of the initialized model is within a predetermined range, the setting means adjusts the size of the inference target area after the change to the input data size.

14. 14. The information processing device according to claim 13, wherein the setting means limits adjustment of the size of the inference target area after the change if the inference target area is changed again within a predetermined time after the size of the inference target area is adjusted.

15. 3. The information processing apparatus according to claim 1, wherein the setting means sets the inference target region in the input image based on a result of detecting an object in the input image.

16. Computer, A setting means for setting an inference target region in an input image that is to be inferred by a model constructed based on machine learning; a determination means for determining a model to be applied to inference using the input image as input data from a plurality of models having different input data sizes, the models having been initialized to be in a state where inference on input data can be performed, in accordance with the size of the inference target area set by the setting means; and The determining means determines, from among the plurality of models, a series of models having input data with a size larger than the inference target region set by the setting means, a model with a smaller input data size as a model to be applied to inference using the input image as input data. A program for causing an information processing device to function as the information processing device.

17. A computer, A setting means for setting an inference target region in an input image that is to be inferred by a model constructed based on machine learning; a determination means for determining a model to be applied to inference using the input image as input data from a plurality of models having different input data sizes, the models having been initialized to be in a state where inference on input data can be performed, in accordance with the size of the inference target area set by the setting means; and The initialized models include a recurrent configuration model that uses the results of a previously performed inference as input. A program for causing an information processing device to function as the information processing device.

Images