Information processor and method for controlling information processor

Abstract

To realize additional learning for improving performance of a learned model in a further preferable form.SOLUTION: A deterioration restoration performance determination processing unit 115 specifies a weak patch corresponding to a region in an image where performance of an inference by CNN performing a prescribed inference on an input image becomes equal to or less a threshold. A cloud server 200 collects a weak patch proximity image similar to the specified weak patch. A learning unit 212 improves the result of the inference by the CNN, targeting a test environment different from the learning environment of the CNN, using an additional image including the collected weak patch proximity image.SELECTED DRAWING: Figure 2A

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and a control method for an information processing device. [Background technology]

[0002] In recent years, with the development of machine learning and deep learning technologies, various image recognition and image generation technologies using AI (trained models) have been attracting attention. One example of an image generation technology using AI is image restoration, which reduces image quality degradation when a degraded image is input. Examples of image degradation factors include noise, blur, low resolution, and defects. In addition, there are a wide variety of processes for reducing image degradation, such as noise removal, blur removal, super-resolution, and defect completion.

[0003] In addition, by utilizing the mechanism of image degradation restoration as described above and calculating the difference between the degraded image and the restored image, it is also possible to realize a function of detecting an image with a larger difference as an abnormal image. Patent Document 1 discloses a technology in which normal images are learned by an autoencoder or the like, and a network capable of stably restoring normal image patterns is acquired in advance, thereby detecting an unrestorable region as an abnormal region. In a situation where the function for detecting abnormal images using the mechanism of image degradation restoration as exemplified above is applied, a situation may occur in which a normal area unrelated to image degradation is erroneously detected as an abnormal area when the degree of restoration is low. In such a case, the performance of anomaly detection may be improved by performing additional learning on images with a low degree of restoration. By adopting this type of technique, it is expected that even if the test environment and the learning environment are somewhat different, the performance of anomaly detection in the test environment can be improved through additional learning using images in the test environment or similar images. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-86382 Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, when a label indicating normality or abnormality has been assigned in advance, additional learning can be performed based on the inconsistency of the label. However, as in the case of an image restoration example, a situation in which a label indicating normality or abnormality has not been assigned can be assumed. In such a situation, it may be difficult to obtain a basis for determining whether the performance of the applied trained model (for example, the performance of image restoration) is insufficient. In this context, there is a demand for realizing a technology that enables additional learning to improve the performance of the model, even in a situation in which it is difficult to determine whether the performance of the trained model is insufficient using conventional methods.

[0006] In view of the above problems, the present invention has an object to make it possible to realize additional learning for improving the performance of a trained model in a more suitable manner. [Means for solving the problem]

[0007] The information processing device of the present invention is characterized in having an identification means for identifying a partial image corresponding to an area in an image where the performance of inference by a learning model that performs a predetermined inference on an input image is below a threshold, a collection means for collecting similar images that are similar to the partial image identified by the identification means, and a learning means for improving the results of the inference by the learning model, targeting a test environment different from the learning environment of the learning model, using additional images including the similar images collected by the collection means. Effect of the Invention

[0008] According to the present invention, it is possible to realize additional learning for improving the performance of a trained model in a more suitable manner. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system. [Figure 2A] FIG. 2 is a functional block diagram showing an example of a functional configuration of the information processing system. [Figure 2B] FIG. 2 is a functional block diagram showing an example of a functional configuration of the information processing system. [Diagram 3] FIG. 13 is a diagram showing an example of a method for obtaining a low-order metric space. [Figure 4] 11 is a flowchart showing an example of processing of an information processing system. [Diagram 5] FIG. 11 is a diagram showing an example of a process related to generation of degraded image data. [Figure 6] FIG. 13 is a diagram illustrating an example of a process related to model learning. [Figure 7A] 11 is a flowchart showing an example of processing of an information processing system. [Figure 7B] 11 is a flowchart showing an example of processing of an information processing system. [Figure 8] 11 is a flowchart showing an example of processing of an information processing system. [Figure 9] FIG. 2 is a functional block diagram showing an example of a functional configuration of the information processing system. [Figure 10] 11 is a flowchart showing an example of processing of an information processing system. [Figure 11] FIG. 2 is a functional block diagram showing an example of a functional configuration of the information processing system. [Figure 12A] 11 is a flowchart showing an example of processing of an information processing system. [Figure 12B] 11 is a flowchart showing an example of processing of an information processing system. [Figure 13] FIG. 13 is a diagram showing an example of a case where human detection and tracking fails. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (conditions of use, environment of use, etc.). In addition, the embodiments may be configured by appropriately combining parts of the embodiments described below. In the following embodiments, the same configurations are described with the same reference symbols.

[0011] <cnn> First, a description will be given of a Convolutional Neural Network (CNN) that is applied to information processing technology in general that applies deep learning and is used in each embodiment described later. CNN is a technology that repeats nonlinear calculation after convolution of a filter generated by training or learning with image data. The filter is also called a local receptive field. Two-dimensional data obtained by nonlinear calculation after convolution of the filter with image data is called a feature map. Moreover, the learning is performed using training data (training images or data sets) consisting of a pair of input image data and output image data. Simply put, the learning can correspond to generating, from the training data, a value of a filter that can convert input image data into corresponding output image data with high accuracy. The details of this will be described later.

[0012] When image data has RGB color channels or when a feature map is composed of multiple pieces of image data, the filter used for convolution also has multiple channels accordingly. That is, a convolution filter is expressed as a four-dimensional array that includes the number of channels in addition to the vertical and horizontal sizes and number of sheets. The process of performing nonlinear operations after convolving a filter with image data or a feature map is expressed in units of layers, such as the nth layer feature map or the nth layer filter. In addition, for example, a CNN that repeats filter convolution and nonlinear operations three times has a three-layer network structure. Such nonlinear operation processing can be formulated as shown below in Equation (1).

[0013]

number

[0014] In formula (1), W n is the nth layer filter, b n is the bias of the n-th layer, f is a non-linear operator, and X n is the feature map of the n-th layer, and * represents the convolution operator respectively. Also, (l) indicates that it is the l-th filter or feature map. The filter (weights) and bias are generated by the learning described later. Hereinafter, the weights and bias obtained by learning are collectively referred to as "model parameters". As the non-linear operation, for example, a sigmoid function or ReLU (Rectified Linear Unit) is used. In the case of ReLU, it is given by the following formula (2). As shown in the following formula (2), among the elements of the input vector X, negative ones become zero and positive ones remain the same value.

[0015]

Equation

[0016] As a network using CNN, for example, ResNet in the field of image recognition and its application RED-Net in the field of super-resolution are well-known. In both cases, by stacking multiple layers of CNN and performing convolution of filters multiple times, the processing accuracy is improved. For example, ResNet features a network structure with a path that shortcuts the convolutional layer, thereby realizing a 152-layer multi-layer network and achieving high-precision recognition approaching the human recognition rate. The reason why the processing accuracy is improved by multi-layer CNN is simply that by repeating the non-linear operation multiple times, the non-linear relationship between the input and output can be expressed.

[0017] <Learning of CNN> Next, the learning of CNN will be explained. The learning of CNN is generally performed by minimizing the objective function represented by the following formula (3) for the training data consisting of a pair of input training image data and corresponding output training image (correct answer image) data.

[0018]

Equation

[0019] In equation (3), L is a loss function that measures the error between the correct answer and its estimation. i is the i-th output training image data, X i indicates the i-th input training image data. F is a function that collectively represents the calculations (equation (1)) performed in each layer of CNN. θ is the model parameter (filter and bias). ∥Z∥2 is the L2 norm, or more simply, the square root of the sum of the squares of the elements of vector Z. n is the total number of training data used for training.

[0020] Generally, the total number of training data is large, so in the Stochastic Gradient Descent (SGD) method, a portion of the training image data is randomly selected and used for training. This makes it possible to reduce the computational load in training using a large amount of training data. In addition, various methods are known for minimizing (optimizing) objective functions, such as the momentum method, AdaGrad method, AdaDelta method, and Adam method. The Adam method is given by the following equation (4).

[0021]

number

[0022] In equation (4), θ i t is the i-th model parameter at iteration t, and g is θ i t It shows the gradient of the loss function L with respect to m and v. m and v are moment vectors, α is the base learning rate, β1 and β2 are hyperparameters, and ε is a small constant. Note that since there are no guidelines for selecting an optimization method in learning, essentially any method can be used, but it is known that there are differences in convergence between methods, which results in differences in learning time.

[0023] In this embodiment, information processing (image processing) that reduces image degradation is performed using the above-mentioned CNN. Image degradation factors include, for example, noise, blur, aberration, compression, low resolution, loss, and degradation such as contrast reduction due to weather influences such as fog, haze, snow, and rain at the time of image capture. Image processing that reduces image degradation includes noise removal, blur removal, aberration correction, loss completion, correction of degradation due to compression, super-resolution processing for low-resolution images, and processing to correct contrast reduction due to weather at the time of image capture. The image degradation reduction process in this embodiment is a process for generating or restoring an image with no degradation (or very little degradation) from an image with degradation, and will be referred to as an image restoration process in the following description. That is, the image restoration according to this embodiment includes not only the restoration of an image with no degradation (or very little degradation) that has been degraded by subsequent amplification, compression / expansion, other image processing, etc., but also the reduction of degradation contained in the original image itself.

[0024] Here, in the case of image restoration processing using a neural network, it is possible to realize image restoration performance superior to conventional processing that does not use a neural network for image degradation that can be expressed by specific parameters. On the other hand, even if the image degradation level is the same, when a scene with a different tendency from that at the time of learning is captured, the level of image restoration may deteriorate. Therefore, in order to obtain good results in a test environment that differs from the tendency during learning, a correct answer image is originally obtained in the test environment, and a learning data set is created by a pair of a degraded image and a correct answer image that is given a known image degradation pattern for the correct answer image, and additional learning is performed. By performing such additional learning, it becomes possible to improve the performance level of image restoration in the test environment. However, in a test environment, only degraded images can be obtained, and it may be difficult to obtain a truly correct image.

[0025] <First embodiment> As a first embodiment of the present disclosure, an example of a method for performing a process of reducing noise in an input image using a neural network will be described below.

[0026] (Example of information processing system configuration) An example of a system configuration to which an information processing device according to a first embodiment is applied will be described with reference to Fig. 1. In the information processing system shown in Fig. 1, a cloud server 200 that generates learning data and estimates image quality degradation and learns restoration, and an edge device 100 that performs degradation restoration for a processing target image are connected via the Internet or the like. Hereinafter, the generation of learning data and the estimation and restoration of image quality degradation in the cloud server 200 will also be referred to as degradation restoration learning, and the degradation restoration in the edge device 100 will also be referred to as degradation restoration inference.

[0027] (Edge device hardware configuration) The edge device 100 according to this embodiment acquires RAW image data (Bayer array) input from the imaging device 10 as an input image to be subjected to image restoration processing. Then, the edge device 100 performs degradation restoration inference by applying learned model parameters provided by the cloud server 200 to the input image to be processed. That is, the edge device 100 is an information processing device that reduces noise in RAW image data by using a neural network provided by the cloud server 200 and executing a pre-installed information processing application program. The edge device 100 has a CPU 101, a RAM 102, a ROM 103, a large-capacity storage device 104, a general-purpose interface (I / F) 105, and a network I / F 106, and each component is connected to each other by a system bus 107. The edge device 100 is also connected to the imaging device 10, the input device 20, the external storage device 30, and the display device 40 via the general-purpose I / F 105.

[0028] The CPU 101 uses the RAM 102 as a working memory, executes programs stored in the ROM 103, and performs overall control of each component of the edge device 100 via a system bus 107. The mass storage device 104 is realized by, for example, an HDD or SSD, and stores various data and image data handled by the edge device 100. The CPU 101 writes data to the mass storage device 104 via the system bus 107, and reads data stored in the mass storage device 104. The general-purpose I / F 105 may be realized by a serial bus interface based on a standard such as USB, IEEE1394, or HDMI (registered trademark). The edge device 100 acquires data from an external storage device 30 (such as various storage media such as a memory card, a CF card, an SD card, or a USB memory) via the general-purpose I / F 105. The edge device 100 also accepts user instructions from an input device 20 such as a mouse or a keyboard via the general-purpose I / F 105. The edge device 100 also outputs image data processed by the CPU 101 to a display device 40 (such as various image display devices such as a liquid crystal display) via the general-purpose I / F 105. The edge device 100 also acquires data of a captured image (RAW image) to be subjected to noise reduction processing from the imaging device 10 via the general-purpose I / F 105. The network I / F 106 is an interface for connecting to various networks such as the Internet. The edge device 100 may access the cloud server 200 via, for example, an installed web browser to obtain model parameters for degradation restoration inference.

[0029] (Cloud server hardware configuration) The cloud server 200 according to this embodiment is an information processing device that provides a network service, such as a so-called cloud service, on a network such as the Internet. More specifically, the cloud server 200 generates training data and performs degradation restoration training, and generates a trained model that stores model parameters and a network structure as a result of the training. Then, the cloud server 200 provides the generated trained model in response to a request from the edge device 100. The cloud server 200 includes a CPU 201 , a ROM 202 , a RAM 203 , a mass storage device 204 , and a network I / F 205 , and these components are connected to each other via a system bus 206 .

[0030] The CPU 201 reads out a control program stored in the ROM 202 and executes various processes to control the overall operation of the cloud server 200. The RAM 203 is used as a temporary storage area such as the main memory and work area of ​​the CPU 201. The mass storage device 204 is a large-capacity secondary storage device that stores image data and various programs, and can be realized by, for example, an HDD or SSD. The network I / F 205 is an interface for connecting to various networks such as the Internet. The network I / F 205 provides a trained model that stores the above-mentioned model parameters and network structure in response to a request from, for example, a web browser of the edge device 100.

[0031] The edge device 100 and the cloud server 200 may include components other than those described above, but a description of those components will be omitted here. In this embodiment, it is assumed that the cloud server 200 transmits a trained model, which is the result of generating training data and performing degradation restoration training, to the edge device 100, and the edge device 100 performs degradation restoration inference on the input image data to be processed. The above-mentioned system configuration is an example, and does not necessarily limit the configuration of the system to which the information processing device according to the present embodiment is applied. For example, the functions of the cloud server 200 may be subdivided, and a configuration may be applied in which the generation of learning data and the degradation restoration learning are performed by separate devices. As another example, a configuration may be applied in which the generation of learning data, the degradation restoration learning, and the degradation restoration inference are all performed by a device (e.g., the imaging device 10) that has both the functions of the edge device 100 and the cloud server 200.

[0032] (System overall functional configuration) An example of the functional configuration of the entire information processing system according to this embodiment will be described with reference to Fig. 2A. As shown in Fig. 2A, the edge device 100 has an image acquisition unit 111, an inference degradation restoration unit 112, and a degradation restoration performance determination processing unit 115. The cloud server 200 has a degradation assignment unit 211 and a learning unit 212. The learning unit 212 has a learning degradation restoration unit 213, an error calculation unit 214, and a model update unit 215. The configuration shown in FIG. 2A is merely an example, and can be modified or changed as appropriate. For example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. The configuration shown in FIG. 2A may be realized by two or more devices. In this case, the devices are connected via a circuit or a wired or wireless network, and perform data communication with each other to perform cooperative operations, thereby realizing each process according to this embodiment.

[0033] First, each functional unit of the edge device 100 will be described in detail. The image acquisition unit 111 acquires an input image 113 to be processed. The input image 113 may be one image, or multiple images that are successive in time series. Each input image may be, for example, RAW image data in which each pixel has a pixel value corresponding to one of the RGB colors. Note that the RAW image data in this disclosure is image data captured using a Bayer array color filter in which each pixel has information for one color. The degradation restoration unit for inference 112 performs degradation restoration inference by inputting one or more received input image data to the trained model 219 transmitted from the cloud server 200, and outputs an output image 114 according to the result to a predetermined output destination. The output image 114 may also be one image or multiple images consecutive in time series.

[0034] The degradation restoration performance judgment processing unit 115 receives the degraded image input from the image acquisition unit 111 and an image equivalent to the output image 114 whose degradation has been restored by the inference degradation restoration unit 112. The degradation restoration performance judgment processing unit 115 judges whether the degradation restoration of the image in the test environment (for example, the environment in which the edge device 100 is used) is performed well or not based on the received degraded image and the image whose degradation has been restored. Then, the degradation restoration performance judgment processing unit 115 determines the poor patch image group 116 that is the cause of the judgment that the degradation restoration in the test environment is not good as a judgment result, and transmits the poor patch image group 116 to the cloud server 200. The poor patch image group 116 is converted into the feature amount of the learned low-order feature space by the low-order feature space conversion 221. Then, the poor patch image group converted into the low-order feature amount is input as a query to the large image group 220. The query is a query for searching for a similar image patch in order to additionally learn an image patch adjacent to the poor patch generated in the test environment. Then, according to the query, image patches having characteristics close to the poor patch image group are added to the correct image group 216. By applying this correct image group 216 when implementing model learning in the cloud server 200 described below, a model with better degradation restoration performance in a test environment can be obtained.

[0035] Next, each functional unit of the cloud server 200 will be described. An image to be applied to learning is uploaded to the cloud server 200. A data set of images to be applied to learning, including this image, is held in the cloud server 200 as a group of correct answers 216. A more suitable group of correct answers 216 is, for example, an image of a subject close to the subject to be captured in a test environment where noise removal is applied, and is desirably composed of images without degradation. In the case of this embodiment, the correct answer image data is RAW image data in which each pixel has a pixel value corresponding to one of the RGB colors, like the input image 113. The physical characteristic analysis result 217 of the imaging device includes, for example, the amount of noise for each sensitivity generated by the imaging sensor built into the camera (imaging device), the amount of aberration generated by the lens, etc. Using these, it is possible to estimate the degree of image quality degradation that occurs for each imaging condition. In other words, by adding the degradation estimated under certain imaging conditions to the correct answer image data, it is possible to generate an image equivalent to the image obtained at the time of imaging.

[0036] The degradation adding unit 211 adds at least one degradation element to the correct answer image data extracted from the correct answer image group 216 with no degradation (or very little degradation) to generate degraded image data. In the example of this embodiment, noise is taken as an example of the degradation element, so the degradation adding unit 211 generates degraded image data by adding noise as a degradation element to the correct answer image data. Specifically, in the case of this embodiment, the degradation adding unit 211 generates degraded image data by adding noise corresponding to a wider range of degradation amount than the degradation amount that can occur in the imaging device, as a degradation element, to the correct answer image data, reflecting the physical property analysis result 217 of the imaging device. The reason for adding a wider range of degradation amount than the analysis result is to provide a margin and increase robustness, since the range of degradation amount varies depending on the individual difference of the imaging device. For example, Fig. 5 is an explanatory diagram for explaining a process related to generation of deteriorated image data by the deterioration adding unit 211. As shown in Fig. 5, the deterioration adding unit 211 generates deteriorated image data 503 by adding noise based on the physical property analysis result 217 of the imaging device as a deterioration element to the correct answer image data 501 extracted from the correct answer image group 216 by addition processing 502. Then, the deterioration adding unit 211 sets the pair of the correct answer image data 501 and the deteriorated image data 503 as learning data. The deterioration adding unit 211 generates a deteriorated image group consisting of a plurality of deteriorated image data by adding a deterioration element to each correct answer image data of the correct answer image group 216, and thereby generates learning data 504. In this embodiment, noise is used as an example, but the degradation adding unit 211 may add any one of several types of degradation elements, such as the blur, aberration, compression, low resolution, loss, and reduced contrast due to the weather at the time of image capture, or a combination of several of these, to the correct image data.

[0037] The learning unit 212 acquires model parameters 218 to be applied to the CNN (model) in degradation restoration learning, initializes the weights of the CNN using the model parameters 218, and then performs degradation restoration learning on the degraded image group generated by the degradation adding unit 211. In other words, the learning unit 212 learns image restoration processing by updating parameters of a learning model (CNN) for improving the image quality of an input image using learning data. The model parameters 218 include hyperparameters indicating initial values ​​of parameters of the neural network, and the structure and optimization method of the neural network. The degradation restoration learning in the learning unit 212 is performed by the learning degradation restoration unit 213, the error calculation unit 214, and the model update unit 215.

[0038] FIG. 6 is a diagram showing an example of the flow of processing in learning section 212. As shown in FIG. The degradation restoration unit for learning 213 receives learning data 504, which is a set of the degraded image group created by the degradation adding unit 211 and the correct image group 216, and restores degradation of the degraded image data 503. Specifically, the degradation restoration unit for learning 213 inputs the degraded image data 503 to a CNN 601, and repeats convolution calculations and nonlinear calculations using filters expressed by equations (1) and (2) multiple times, and outputs degradation-restored image data 602 as a result. The error calculation unit 214 inputs the correct image data 501 and the degraded restored image data 602 to a Loss 603 (a calculation unit that calculates the error) and calculates the error therebetween. Here, it is assumed that the correct image data 501 and the degraded restored image data 602 have the same number of pixels. Next, the model update unit 215 inputs the error calculated by the error calculation unit 214 to the update process 604, and updates the model parameters related to the CNN 601 so as to reduce the error. Note that the CNN used in the learning unit 212 is the same neural network as the CNN used in the inference degradation restoration unit 112.

[0039] 2A, the degradation restoration process of the input image in the edge device 100 and the learning of the degradation restoration process in the cloud server 200 have been described. In this mechanism, an example of a functional configuration for further improving the performance of the degradation restoration process in the edge device 100 will be described below. When the edge device 100 performs inference that reflects the results learned by the cloud server 200, the image acquisition unit 111 on the edge device 100 side may receive an input image 113 that differs from the tendency learned by the cloud server 200. In such a case, the degree of degradation restoration of the output image 114 that is the output of the inference degradation restoration unit 112 may not reach a level that satisfies the user. Therefore, the accuracy is improved (in other words, performance is improved) by the method described below.

[0040] The degradation restoration performance judgment processing unit 115 receives the input image acquired by the image acquisition unit 111 and the image output by the degradation restoration unit for inference 112, identifies a group of difficult patch images 116 whose restoration performance is equal to or lower than a threshold, and transmits the group of difficult patch images 116 to the cloud server 200. In this way, the degradation restoration performance judgment processing unit 115 acquires a model with good degradation restoration performance in the edge environment (in other words, the test environment) from the cloud server 200. The weak patch image group 116 transmitted from the edge device 100 to the cloud server 200 is converted into features in a low-level feature space using a learned distance measure by a presented feature space transformation 221. Based on these features, a comparison is made with a huge number of image patches held in a large image group 220 stored in the cloud server 200. The large image group is assumed to be composed only of images with good image quality, with no (or very little) degradation.

[0041] In order to easily search for images similar to the poor patch image group 116, it is desirable that, for example, the images in the large image group 220 are already assigned coordinates in a low-order feature space that reflect the similarity between image patches according to a predetermined criterion. On the other hand, at the timing when the poor patch image group 116 is input, learning of a low-order distance space using the large image group 220 and the poor patch image group 116 may be performed. A nearby image of the weak patch image group 116 input from the large image group 220 is selected, and the selected image is added to the correct image group 216, thereby updating the model parameters already learned in the learning unit 212 in the same manner as in the above learning. In addition, the updated trained model 219 is transmitted to the degradation restoration unit 112 for inference of the edge device 100, whereby the edge device 100 can restore the degraded image by a good degradation restoration model that matches the tendency of each input image.

[0042] (Processing flow of the entire system) An example of the processing of the information processing system according to this embodiment will be described with reference to Figures 7A and 7B. Each of Figures 7A and 7B is a flowchart showing an example of the processing flow of the information processing system according to this embodiment. The processing shown in Figures 7A and 7B is realized, for example, by each of the CPUs 101 and 201 described with reference to Figure 2A executing the information processing computer program according to this embodiment. However, all or part of the functional units shown in Figure 2A may be implemented by hardware. In this case, for example, at least part of the processing shown in Figures 7A and 7B may be realized by the hardware.

[0043] First, an example of a flow of a series of processes for degradation restoration learning by cloud server 200 will be described with reference to FIG. 7A. In S701, the cloud server 200 accepts input of a group of correct images 216 prepared in advance and a physical property analysis result 217 of the imaging device such as the characteristics of the imaging sensor, the sensitivity at the time of imaging, the subject distance, the focal length and f-number of the lens, and the exposure level. The correct image data is a RAW image of a Bayer array, and can be obtained by imaging with the imaging device 10, for example. In addition, not limited to the RAW image of the Bayer array, an image captured with the imaging device may be uploaded as it is, or an image captured in advance and stored in a storage device such as a HDD may be uploaded. Then, the data of the group of correct images 216 input to the cloud server 200 and the physical property analysis result 217 of the imaging device are transmitted to the degradation adding unit 211.

[0044] In S702, the degradation adding unit 211 generates degraded image data by adding noise based on the physical property analysis result 217 of the imaging device to the supervised image data of the supervised image group 216 input in S701. At this time, the degradation adding unit 211 may add, for example, an amount of noise measured in advance based on the physical property analysis result 217 of the imaging device to the supervised image data in a preset order or in a random order. The degraded image data generated in S702 is used as learning data for learning a model by a process described separately below.

[0045] In S703, the cloud server 200 accepts input of model parameters to be applied to the CNN for degradation restoration learning. The model parameters here include the weights of the neural network that will result in optimal results after learning as described above, and the initial values ​​of parameters that summarize the bias. The input model parameters are transmitted to the learning unit 212.

[0046] In S704, the learning restoration unit 213 initializes the weights of the CNN using the received model parameters, and then performs restoration of the degraded image data generated in S702. In S705, the error calculation unit 214 calculates the difference between the degraded restored image data and the correct image data in accordance with the loss function (objective function) shown in equation (3). In S706, the model update unit 215 updates the model parameters so that the difference between the degraded restored image data obtained in S705 and the correct image data becomes smaller (for example, becomes minimum).

[0047] In S707, the learning unit 212 determines whether or not a learning convergence condition is satisfied. The learning convergence condition is not particularly limited and may be set appropriately depending on the use case. As a specific example, whether or not the number of updates of the model parameters of the learning unit 212 reaches a predetermined number may be set as the learning convergence condition. If the learning unit 212 determines in S707 that the learning convergence condition is not satisfied, the process proceeds to S704. In this case, learning is performed using other degraded image data and correct image data in the processes from S704 onward. Then, if the learning unit 212 determines in S707 that the learning convergence condition is satisfied, the learning unit 212 ends the series of processes shown in FIG. 7A.

[0048] Next, an example of the flow of degradation restoration inference processing by the edge device 100 will be described with reference to FIG. 7B. In S708, the edge device 100 acquires the trained model 219 trained in the cloud server 200. In S709, the image acquisition unit 111 selects input image data of N frames (N≧1) from the input image 113 to be subjected to degradation restoration processing, and generates input linked image data linked in the channel direction. For the input image 113, for example, an image captured by the imaging device 10 may be directly input, or an image captured in advance and stored in the large-capacity storage device 104 may be read out. In this embodiment, the N frames referred to here represent N frames that are consecutive in time series.

[0049] In S710, the degradation restoration unit for inference 112 constructs a CNN similar to that used in learning by the learning unit 212, and performs degradation restoration of the input linked image data. At this time, the degradation restoration unit for inference 112 initializes the existing model parameters with the updated model parameters received from the cloud server 200. In this manner, the degradation restoration unit for inference 112 inputs the input linked image data to the CNN to which the updated model parameters have been applied, and performs degradation restoration on the input image data in a manner similar to that performed by the learning unit 212, thereby obtaining output image data.

[0050] In S711, the edge device 100 outputs the output image data obtained in S710 to a predetermined output destination as the output image 114. As a specific example, the edge device 100 may output the output image 114 to the display device 40 shown in FIG. 1 to display the output image 114 on the display device 40. As another example, the edge device 100 may output the output image 114 to the external storage device 30 to store the output image 114 in the external storage device 30.

[0051] In S712, the degradation restoration performance judgment processing unit 115 judges the degree of restoration when the restoration result is output in S711. As a specific example, in the output image output as the restoration result, there may be an area where the degradation restoration is good and an area where it is not. If there is an area where the degree of degradation restoration is poor in the test environment (for example, an area where the degree of degradation restoration is not sufficient to meet the required degree), it can be said that the currently learned degradation restoration model is weak at performing the degradation restoration process on the image of that area. Therefore, in this embodiment, the area where the degree of degradation restoration is poor is specified as a weak area by a threshold value set in advance, and a process of improving the image restoration process accuracy in the test environment is performed. The process of improving the image restoration process accuracy will be described as the process of S712 to S718.

[0052] Here, an example of the degradation restoration performance determination process in S712 will be described with reference to FIG. In S801, the degradation restoration performance judgment processing unit 115 acquires r images (r is, for example, about 100) in chronological order. In S802, the degradation restoration performance judgment processing unit 115 divides the r consecutive images acquired in S801 at a constant interval vertically and horizontally into small regions of a size equivalent to an image patch, and judges whether each region is a still region without movement. The simplest processing for judging movement is a method of judging based on the difference between previous and next frames of an input image. For example, by obtaining different difference images when noise occurs and when the object moves, it becomes possible to use only the non-still regions of each of the r images.

[0053] In S803, the degradation restoration performance judgment processing unit 115 creates a cumulative average image for each small region judged to be a still region in S802. Usually, the denoising result of the cumulative average image in the still region is very good. In S804, the degradation restoration performance judgment processing unit 115 calculates the difference between the denoising result in the region judged to be a still region and the cumulative average image. In S805, the degradation restoration performance judgment processing unit 115 judges that a frame in which the difference calculated in S804 exceeds a threshold is an image patch with low denoising quality. If there is no image patch in S805 in which the difference exceeds the threshold, the degradation restoration performance is judged to be good, and in this case, there is no need to update the learned degradation restoration inference model. On the other hand, if there is an image patch in S805 in which the difference exceeds the threshold, the degradation restoration performance is judged to be low (for example, does not satisfy the required accuracy), and the cumulative average image of the corresponding area is set as a poor patch image.

[0054] An example of the degradation restoration performance judgment process shown in S712 of Fig. 7B has been described above with reference to Fig. 8. The degradation restoration performance judgment processing unit 115 judges whether or not to update the learned degradation restoration inference model in S713 based on the result. Specifically, when a low-quality image patch exists, the degradation restoration performance judgment processing unit 115 judges that the learned degradation restoration inference model needs to be updated. If the degradation restoration performance judgment processing unit 115 judges in S713 that the learned degradation restoration inference model is to be updated, it advances the process to S715. On the other hand, if the degradation restoration performance judgment processing unit 115 determines in S713 that the learned degradation restoration inference model is not to be updated, it advances the process to S714.

[0055] In S715, the degradation restoration performance judgment processing unit 115 identifies a poor patch and transmits the poor patch to the cloud server 200. Note that the poor patch identified in S715 (i.e., an image patch corresponding to at least a part of an area in the target image) corresponds to an example of a partial image corresponding to an area in the image where the performance of inference by the inference means (e.g., a learned degradation restoration inference model) is equal to or lower than a threshold. In S716, the cloud server 200 compares the huge image patches stored in the large image group 220 with the poor patch transmitted in S715, and selects a patch image having similar characteristics from the large image group 220. In S716, it is possible to select a nearby patch by expressing the difficult patch in a similar metric space from a large image group defined in a learned low-order metric space, but in order to do so, a mechanism is required to properly learn the metric space. Therefore, an example of a method for acquiring the low-order metric space using a large image group 220 stored in advance will be described below with reference to Fig. 3.

[0056] In order to compare the similarity between a huge number of image patches in the large image group 220 and each image patch in the poor patch image group 116, it is desirable to have a CNN capable of converting the image patches into a low-level feature space in which the similarity can be calculated as a distance. As shown by reference numeral 300 in Fig. 3, the CNN is a neural network that receives an image patch as input and outputs m-dimensional coordinates (m≧1) in a trained metric space that reflects the similarity between the image patches. The CNN shown by reference numeral 300 in Fig. 3 is the same as the CNN that performs low-level feature space conversion shown by reference numeral 221 in Fig. 2A. Without the CNN (hereinafter also referred to as trained image patch metric space transformation CNN), the large image set 220 is simply a data set of image patches, and even if the poor patch is obtained, it is difficult to select a patch similar to the poor patch. Therefore, in this embodiment, the large image set 220 is specified to be accompanied by the trained image patch metric space transformation CNN.

[0057] Here, an example of a learning flow of the image patch metric space transformation CNN will be described with reference to Fig. 4. The metric learning illustrated in Fig. 4 is realized by two types of image patch metric learning. Therefore, for simplicity, these are referred to as image patch similarity-based metric learning (S402 to S405) and image patch transformation processing-based metric learning (S406 to S409). First, the input in the image patch distance space conversion CNN is an input image defined by the image patch size held in the currently targeted large image group 220. The larger the size of the input image, the higher the dimension, but it is desirable to set it to the same size as the image patch input to the learning degradation restoration unit 213 and the inference degradation restoration unit 112 in FIG. 2A. In addition, the output of the image patch distance space conversion CNN is a feature space dimension vector with a sufficiently low dimension compared to the input image. Basically, in this learning flow, the similarity between images is calculated as a one-dimensional score and embedded in the feature space by referring to each distance, so a low-dimensional space of approximately 10 dimensions or less is assumed. In addition, at the beginning of learning, the model parameters of the image patch distance space transformation CNN are initialized with random values.

[0058] In S401, the learning unit 212 determines the subsequent processing depending on whether or not image patch similarity-based metric learning is to be performed (image patch conversion processing-based metric learning). That is, when the learning unit 212 determines in S401 to perform image patch similarity-based metric learning, the process proceeds to S402. On the other hand, if the learning unit 212 determines in S401 that image patch similarity-based metric learning is not to be performed, the process proceeds to S406. In this embodiment, the above two processes are performed alternately based on conditions such as a certain learning converging below a preset threshold, and the process first proceeds to image patch similarity-based metric learning.

[0059] In S402, the learning unit 212 samples three image patches from the image patches stored in the large image group 220. Although the method of sampling the three image patches is not particularly limited, in this embodiment, the three image patches are randomly sampled. FIG. 3(A) shows an image of inputting three image patches to the image patch metric space conversion CNN in S402 and calculating the score by the loss function. The three image patches are denoted as image patches Xc (302), Xd (301), and Xe (303), respectively. In S403, the learning unit 212 calculates feature quantities Fc (305), Fd (304), and Fe (306) in the current feature space by the current image patch distance space transformation CNN (300). In S404, the learning unit 212 calculates a loss (Xc, Xd, Xe) based on the distance in the feature space of the above feature amounts Fc, Fd, and Fe and the image patch similarity, as indicated by reference numeral 307 in Fig. 3(A). This loss is defined as follows so as to be reflected in the low-level feature space by referring to the distance relationship based on a separately defined image similarity standard.

[0060] First, the learning unit 212 determines whether Xd or Xe is closer to Xc. For example, image similarity is used as an index for determining the proximity. In this embodiment, the image similarity is calculated by the score SSIM (structural similarity) defined below. Specifically, if the image patches for which similarity is compared are expressed as x and y, the score SSIM is calculated by the relational expression shown as equation (5).

[0061]

number

[0062] In equation (5), l(x,y) is the luminance, c(x,y) is the contrast, and s(x,y) is the comparison term for the structure. x is the average brightness of x[i], μ y indicates the average brightness of y[i], and σ x is the standard deviation of x[i], σ y indicates the standard deviation of y[i]. Also, σ xy indicates the covariance of x[i] and y[i]. Also, L indicates the maximum amplitude, which is 255 in the case of 8-bit representation. In addition, when α = β = γ = 1, it is often used in the form of the following equation (6), so from here on, we will explain it as if the score calculated based on equation (6) is used.

[0063]

number

[0064] The method of calculating the similarity is not limited to the method based on the relational expression shown as formula (5), and various methods can be applied as long as the similarity can be calculated as a one-dimensional score. As a specific example, a method of calculating the similarity as a one-dimensional score by finding the correspondence of feature points using AKAZE, SIFT, or SURF may be applied, or a method of defining the similarity by a simple square error or the like may be applied.

[0065] In the case of a method using SSIM, it is possible to determine that the images are similar if the value is large. For example, when determining whether Xd or Xe is similar to Xc, the image patch with the larger score calculated by formula (5) may be determined to be the image patch more similar to Xc. In the following, the image patch similar to the target (e.g., Xc) is also referred to as Xp, and the other image patch is also referred to as Xp. In addition, in accordance with this designation, the low-level feature vector that is the output of the image patch distance space conversion CNN corresponding to Xp is referred to as Fp, and the low-level feature vector corresponding to Xn is referred to as Fn. An example of the loss score (307) of the above three image patches calculated in S404 is expressed by the relational expression shown below as formula (7).

[0066]

number

[0067] In the above formula (7), it is possible to calculate Loss(Xc, Xd, Xe) by determining which of Xd and Xe is closer based on the distance from Xc. That is, the one closer to Xc is designated as Xp, and the other is designated as Xn, and the loss is calculated by finding the distance d in the feature space of each of the outputs Fc, Fp, and Fn by the image patch metric space conversion CNN. Note that the distance d(A, B) is assumed to be the Euclidean distance between A and B in the above image patch metric space, but other distances may be used. The loss is calculated by performing the above-mentioned loss score calculation for many combinations of Xc, Xd, and Xe. In this manner, in S405, the learning unit 212 updates the model parameters of the image patch metric space transformation CNN in a direction to reduce the above-mentioned loss.

[0068] Next, in S410, the learning unit 212 determines whether or not a condition for terminating the distance learning is satisfied. As a specific example, the learning unit 212 may determine that the condition for terminating the distance learning is satisfied when the magnitude of the reduction in the loss by the loss function falls below a preset threshold. If the learning unit 212 determines in S410 that the end condition of the distance learning is not satisfied, the process proceeds to S401. In this case, the process from S401 onwards, that is, the image patch similarity-based metric learning or the image patch transformation processing-based metric learning is continued again. Then, if it is determined in S410 that the end condition for the distance learning is satisfied, the learning unit 212 ends the series of processes shown in FIG.

[0069] Next, the image patch transformation processing-based metric learning (S406 to S409) will be described. In the case of the image patch similarity-based metric learning (S402 to S405) described above, the similarity of three randomly extracted image patches is treated as a one-dimensional distance, and calculations are performed to learn a metric space in which the relationships between the three randomly extracted image patches are embedded in a low-dimensional distance space. In contrast, in the case of the image patch transformation processing-based metric learning shown in S406 to S409, an image patch is intentionally created by performing image transformation on one of two types of randomly extracted image patches. In this way, distance learning is performed by utilizing the relative relationship that a known similar image patch is in the vicinity compared to an image patch with a lower relationship (eventually completely unrelated).

[0070] In S406, the learning unit 212 randomly extracts an image patch Xc (302) and an image patch Xh (313) as shown in Fig. 3(B) from the large image group 220, and generates an image patch Xc' (311) by performing a random image transformation on the image patch Xc. For example, the affine transformation T may be used for the transformation process at this time. In Fig. 3B, image patch Xc' (311) is an image obtained by performing a simple affine transformation on image patch Xc (302). Therefore, in S407, the learning unit 212 calculates feature quantities Fc (315), Fc' (314), and Fh (316) in the current feature space by using a CNN (300 shown in Fig. 3 and 221 shown in Fig. 2A) that embeds three target image patches in a low-order metric space.

[0071] Further, the learning unit 212 performs learning so as to acquire a distance space in which the feature amounts Fc’(314) and Fc(315) are close to each other in the feature space and the feature amount Fh(316) is located farther away in each relationship. At this time, an example of the loss score LOSS(317) based on the above three image patches calculated in S408 is shown as Equation (8) below.

[0072]

Equation

[0073] In the above (8), based on the distance from Xc, a loss considering the distances to Xc’ and Xh is calculated. Equation (8) is calculated by obtaining the distance d in the feature space of each of the outputs Fc, Fc’, and Fh by the image patch distance space conversion CNN. Note that as the distance d, the Euclidean distance between A and B in the above-described image patch distance space is assumed in the same manner as Equation (7) of d(A,B) described above, but other distances may be used. Along with the loss score calculation as described above, losses are calculated for a large number of combinations of Xc, Xc’, and Xh. Comparing with Equation (7), since an image patch that is always in the vicinity is input to Equation (8), it is preferable that s < q as the magnitude relationship between s and q in Equation (7). However, in this case, it is preferable that the control is performed so that the transformation at the level that significantly impairs the properties of the original input image is not performed in the above affine transformation T. As described above, in S409, the learning unit 212 updates the model parameters of the image patch distance space conversion CNN in the direction of reducing the above loss.

[0074] Then, in S410, the learning unit 212 determines whether or not the end condition of the distance learning is satisfied. Since the process of S410 has been described above, a detailed description thereof is omitted.

[0075] We have described an example of the learning flow of image patch distance space transformation CNN by implementing both image patch similarity-based metric learning and image patch transformation processing-based metric learning. As a result, two types of learning are implemented: distance learning based on an existing image similarity-based distance measure called SSIM for comparing different image patches, and similar image patches that can be generated by transforming the own patch are embedded closer to the different image patches.

[0076] The above-mentioned process is continued until the learning satisfies the termination condition, and as a result, an image patch data set is finally obtained in which coordinates in the low-level feature space in which the proximity relationships of the image patches in the large image set 220 have been learned are held. Note that it does not matter whether the similar patch generated in S406 is included in the large image set in which the distance relationships are defined here.

[0077] In addition, while an example of a method using CNN to obtain the distance space of image patches of the large image group 220 has been described above, the method is not limited to the above example, and various methods for performing distance learning have been proposed. For example, it is possible to use FDA (Fisher Discriminant Analysis), which is a linear dimensionality reduction method. In this case, different image patches can be regarded as different classes, and image patches generated by affine transformation can be regarded as the same class. Then, a linear projection matrix that reduces the intra-class scatter and increases the inter-class scatter can be obtained, and the obtained projection matrix can be used as the low-order feature space transformation 221 shown in FIG. 2A. In addition, a nonlinear transformation to a low-order feature space may be applied. Specifically, when the space is newly defined by the weak patch image group 116 newly input each time the low-order feature space transformation 221 is performed, the purpose can be achieved even if Isomap (Isometric mapping) or the like is performed each time.

[0078] By the method described above, the distance space between the image patches stored in the large image group 220 is predefined. Accordingly, since the distance relationship between all image patches is predefined, it becomes possible to quickly select a nearby image in the large image group 220 for the weak patch image group 116 input as a query. The poor patch image group 116 is converted into low-order features in the learned distance space by a low-order feature space transformation 221 in order to select nearby images that are close to the poor patch image. A predetermined number of image patches that exist in the vicinity of the low-order features are then selected. In this selection, for example, if it is necessary to collect P nearby image patches, P patches are selected in order from the nearest feature point in the distance space, and image patches associated with the P feature points are selected. As for the distance, the simplest method is to apply a Euclidean distance criterion, but this is not limiting. Above, an example of a method for learning a low-order distance space in advance in order to select an image close to the poor patch group from the large image group in S716 has been described. In the following, the processing flow following S716 will be described with reference to FIG. 7B.

[0079] In S717, the cloud server 200 adds the patch selected in S716 (hereinafter, also referred to as the weak patch adjacent image) to the learning supervised image group 216. Note that an image added to the learning supervised image group 216, which includes at least the weak patch adjacent image, corresponds to an example of an added image. After that, the cloud server 200 executes a process similar to the learning flow already described after adding the weak patch adjacent image to the learning supervised image group 216. Then, when the learning converges, the cloud server 200 updates the inference degradation model in S718 and transmits the updated model to the inference degradation restoration unit 112 of the edge device 100. As a result, the model of the inference degradation restoration unit 112 is updated according to the tendency of the input image in the test environment. As a result, improvement in degradation restoration accuracy in the test environment (in other words, improvement in degradation restoration performance) can be expected.

[0080] In S714, the edge device 100 determines whether to end the image restoration process. The determination of whether to end the image restoration process in S714 may be determined based on whether an instruction to end the image restoration process is received from the user via the input device 20, for example. If the edge device 100 determines in S714 not to end the image restoration process, it advances the process to S709. In this case, the process from S709 onwards is executed again for the next frame. As described above, the process from S709 onwards is repeatedly executed until it is determined in S714 that the image restoration process is to end. In this way, the area where the degradation restoration performance is poor in the degradation restoration performance determination process S712 is determined as needed, and additional learning is continued to be executed each time, thereby maintaining the degradation restoration inference model adapted to the change in the test environment. Then, if the edge device 100 determines in S714 to end the image restoration process, it ends the series of processes shown in FIG. 7B.

[0081] 7A, the example in which the learning data is generated in S702 has been described, but the learning data may be generated at a timing after S702. Specifically, in the subsequent degradation restoration learning, the degraded image data corresponding to the correct image data may be generated. In addition, in this embodiment, an example of performing learning from scratch using data of a group of correct images prepared in advance has been described, but the processing of this embodiment may also be performed based on learned model parameters.

[0082] In this embodiment, the case where RAW image data captured with a Bayer array color filter is described as the target is described, but other color filter arrays may be applied. Also, although the RAW image data has one channel, the pixels may be rearranged in the order of R, G1, G2, B in the color filter array. In this case, the data structure is H×W×4, and when N=3, the data structure becomes H×W×12 by concatenating the RAW image data. Also, the data format of the image is not limited to a RAW image, and may be, for example, a demosaiced RGB image or an image after YUV conversion. In addition, in the present embodiment, the degradation factor is explained by taking noise as an example, but the degradation factor is not limited to this. The degradation factor may include any of the above-mentioned blur, aberration, compression, low resolution, defects, and contrast reduction due to the influence of fog, haze, snow, or rain at the time of image capture, or a combination of these, and the like, but the degradation factor can be solved by the same method. In addition, the image quality improvement AI according to the present embodiment has been described on the assumption that a model learned by deep learning / machine learning is applied, but the method for realizing the AI ​​is not necessarily limited. In other words, other methods that can improve image quality may also be applied.

[0083] <Second embodiment> As a second embodiment of the present disclosure, a modified form of the first embodiment will be described below. Specifically, in the first embodiment, a case was described in which image patches in an area where the accuracy of image quality restoration is below a threshold in a test environment (i.e., an area where the accuracy of restoration performance is poor) are selected and additional learning is performed, using restoration of a deteriorated image as an example. In this embodiment, an example of a method for solving the problem by using a deteriorated restored image to select a poor patch adjacent image having characteristics close to the image patch in the area where the accuracy of image quality restoration is below a threshold. In this embodiment, the hardware configuration of the edge device is substantially similar to that of the first embodiment. Therefore, in this embodiment, the description will focus on the processing different from the first embodiment and the functional configuration of the information processing system, and detailed description of the parts substantially similar to those of the first embodiment will be omitted.

[0084] In the first embodiment, a method was described in which the large image group 220 in Fig. 2A is an image group made up of images with less degradation (eventually no degradation), and images close to the test environment are searched for and added from the image group. In contrast, in this embodiment, the functional configuration of the entire information processing system is as shown in Fig. 2B, and a series of large image groups are managed separately as a large image group (no degradation) 2201 and a large image group (degraded restored images) 2202.

[0085] The large image group (no degradation) 2201 shown in FIG. 2B is substantially the same as the large image group 220 shown in FIG. 2A. In contrast, the large image group (degraded restored image) 2202 is degraded by the degradation imparting unit 211 that imparts degradation to the large image group 2201 (no degradation) during normal learning, and stores an image group in which the degradation-imparted images are restored by the learning degradation restoration unit 213. Naturally, it is desirable that the number of images stored in the large image group (no degradation) 2201 and the large image group (degraded restored image) 2202 is the same. Also, each series of degraded restored images in the large image group (degraded restored image) 2202 and each non-degraded image in the large image group (no degradation) 2201 that is the source of the degraded restored image are linked one-to-one and managed.

[0086] The model used in the learning degradation restoration unit 213 for degraded image restoration is the same as the model used in the inference degradation restoration unit 112 for image degradation restoration in the test environment at that time. Also, the images in the large image group (degraded restored images) 2202 are updated when the trained model 219 is updated in the subsequent additional learning flow. Therefore, a series of large image groups are separated into a large image group (without degradation) 2201 and a large image group (degraded restored image) 2202, and thus a search for a poor image adjacent image patch is performed in the large image group (degraded restored image) 2202 composed of degraded restored images. That is, the images in the large image group (degraded restored image) 2202 composed of degraded restored images are held as an image group to which a distance relationship defined in a low-level feature space is given by the distance learning shown in the processing flow shown in FIG. 4 described in the first embodiment. Then, in the processing of S712 in FIG. 7B, a judgment is made as to whether the degradation restoration performance in the test environment is good or bad. Then, when it is judged that a model update is necessary by the degradation restoration performance judgment processing unit 115 in S713, the degradation restoration patch itself of the image patch whose level of image quality degradation restoration was not high in S715 is transmitted to the cloud server 200 as the poor patch image group 116. In addition, in S716, the low-level feature space coordinates of the large image group (degraded restored image) 2202 are calculated in the low-level feature space transformation 221, and an image close to the poor patch is selected by the same process as in the first embodiment. On the other hand, since the weak patch adjacent image selected in S716 is a degraded restored image, it is not desirable to add this image group as learning data as it is. Therefore, an original image with less degradation (eventually no degradation) of the selected weak patch adjacent image is specified from the large image group 2201 and added to the correct image group 216. Note that the subsequent processing flow (for example, additional learning flow, etc.) is the same as that of the first embodiment.

[0087] In addition, in the present embodiment, the degradation factor is explained by taking noise as an example, but the degradation factor is not limited to this. The degradation factor may include any of the above-mentioned blur, aberration, compression, low resolution, defects, and contrast reduction due to the influence of fog, haze, snow, or rain at the time of image capture, or a combination of these, and the like, but the degradation factor can be solved by the same method. In addition, the image quality improvement AI according to the present embodiment has been described on the assumption that a model learned by deep learning / machine learning is applied, but the method for realizing the AI ​​is not necessarily limited. In other words, other methods that can improve image quality may also be applied.

[0088] <Third embodiment> As the third embodiment of the present disclosure, a modified form of the first embodiment will be described below. Specifically, in the first embodiment, a case where image patches in an area with poor image quality restoration accuracy in a test environment are selected and additional learning is performed is described using restoration of a degraded image as an example. In this embodiment, an example of a method for identifying an area where learning is insufficient not in a test environment but in the learning process, and selecting learning patches for improving image restoration performance due to insufficient learning to expand the learning data set will be described. In addition, since this embodiment is an embodiment that focuses only on learning, no distinction is made between a cloud server / edge device, etc. Therefore, in this embodiment, for convenience, when no distinction is made between a cloud server / edge device, etc., various explanations will be made assuming that the subject of each process is an information processing system.

[0089] In the following, the functional blocks and the processing flow will be described in association with each other. FIG. 0 is a functional block diagram showing an example of the functional configuration of the information processing system according to this embodiment. FIG. 10 is a flowchart showing an example of the processing of the information processing system according to this embodiment. In the functional block diagram shown in FIG. 9, the functional components substantially similar to those shown in FIG. 2A are given the same reference numerals. Therefore, detailed description of the functional components given the same reference numerals as those shown in FIG. 2A will be omitted. As shown in FIG. 9, the information processing system according to this embodiment is particularly different from the first embodiment in that the degradation restoration performance determination processing unit 915 is interposed between the error calculation unit 214 and the presentation feature space conversion 221 because the corresponding processing is performed in the learning process.

[0090] In the training of image generation-related CNNs, it is difficult to predict in advance what kind of training images will provide sufficient performance. In normal training, learning may be influenced by biases that potentially exist in the training dataset. As a result, even if the training dataset contains images with insufficient image degradation restoration, if the number of such images is small, it is difficult to reflect further information in the training, even if the image is a weak image patch. Therefore, under such circumstances, training on a specific small number of weak image patches may not proceed actively. In the case where an image similar to a difficult image patch for which degradation restoration was insufficient during learning is input in a test environment, the restoration level will remain similarly insufficient. Therefore, in this embodiment, an example of a method for improving the overall image degradation restoration performance by adding an image patch for which image restoration was insufficient, which was found in the learning process, at an appropriate timing during learning, will be described.

[0091] As shown in FIG. 9, the information processing system according to this embodiment includes a degradation adding unit 211 and a learning unit 212. The learning unit 212 includes a learning degradation restoration unit 213, an error calculation unit 214, and a model update unit 215. The configuration shown in FIG. 9 can be modified or changed as appropriate, as in the example shown in FIG. 2A. For example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. The configuration shown in FIG. 9 may be realized by two or more devices. In this case, the devices are connected via a circuit or a wired or wireless network, and perform data communication with each other to perform cooperative operations, thereby realizing each process according to this embodiment.

[0092] In S1001, the information processing system acquires a correct answer image input from the correct answer image group 216 and a physical property analysis result 217 of the imaging device. In S1002, the information processing system performs a learning data generation process on a set of a degraded image to which degradation has been added by the degradation adding unit 211, which reflects the physical property analysis result 217 of the imaging device, and the correct image. In S1003, the learning degradation restoration unit 213 acquires the model parameters 218 to be applied to the CNN for degradation restoration learning. In S1004, the learning degradation restoration unit 213 performs image degradation restoration processing defined by the model parameters 218 (initial model parameters) acquired in S1003. In S1005, the error calculation unit 214 calculates the error between the image after the degradation restoration process and the correct image.

[0093] In S1006, the degradation restoration performance determination processing unit 915 determines whether or not there is an image patch in which the error calculated in S1005 exceeds a threshold value. If the degradation restoration performance judgment processing unit 915 judges in S1006 that there is an image patch whose error exceeds the threshold, the process proceeds to S1007. In this case, in S1007, the degradation restoration performance judgment processing unit 915 holds the image patch whose error exceeds the threshold as a poor image patch group, and proceeds to S1008. On the other hand, if the degradation restoration performance determination processing unit 915 determines in S1006 that there is no image patch whose error exceeds the threshold, the process proceeds to S1008. In this case, the process of S1007 is skipped. In S1008, when the error calculation for the series of learning data is completed, the model update unit 215 updates the inference model parameters so as to minimize the image degradation restoration error due to the current model parameters.

[0094] In S1009, the information processing system determines whether or not the learning convergence condition is satisfied. If it is determined in S1009 that the learning convergence condition is satisfied, the information processing system ends the series of processes shown in FIG. On the other hand, if the information processing system determines in S1009 that the learning convergence condition is not satisfied, it proceeds to S1010.

[0095] In S1010, the information processing system determines whether or not there is a poor image patch group that was held in S1007 due to the error calculated in S1005 exceeding a threshold value in the learning process up to this point. If the information processing system determines in S1010 that a group of poor image patches is stored, the process proceeds to S1011. On the other hand, if the information processing system determines in S1010 that there is no poor image patch group stored, the process proceeds to S1004. In this case, the restoration of the degraded image and the learning process associated with the restoration result from S1004 onward are executed again.

[0096] In S1011, the information processing system selects an image similar to the weak image patch from the image patch data set of the large image group 220. The image similar to the weak image patch is transformed into a learned low-order metric space by the low-order feature space transformation 221, It is possible to select the image as a nearby image in the learned low-order metric space. The low-order metric space is substantially the same as that described in the first embodiment, and therefore a detailed description thereof will be omitted. In S1012, the information processing system adds image patches similar to the poor image patches selected in S1011 to the correct image group, and then proceeds to S1004. In this case, the restoration of the degraded image and the learning process associated with the restoration results from S1004 onwards are executed again based on the update result of the learning dataset.

[0097] In the case of learning aimed at image restoration, it is possible to give a degradation pattern to the image patch held in the correct answer image group 216 during the learning process by the degradation adding unit 211. Therefore, if a patch that is difficult to restore can be identified during the learning process, it is possible to proceed with detailed learning even for image patches that were difficult to restore due to insufficient learning by selecting and adding images close to the image patch from the large image group 220 at any time. In addition, the processing of the degradation restoration performance judgment processing unit 915 can be performed more easily than the processing performed on the test image in the first embodiment.

[0098] The difference between a correct answer image acquired from the correct answer image group 216 in the learning process and an image restored by the learning degradation restoration unit 213 after the correct answer image is degraded by the degradation adding unit 211 is calculated by the error calculation unit 214, and the error is input to the degradation restoration performance judgment processing unit 915. Therefore, the image patch information of the area where the performance of the image degradation restoration is equal to or lower than the threshold value is determined by the error image input to the degradation restoration performance judgment processing unit 915. Since the corresponding correct answer image patch has already been acquired, this correct answer image patch is automatically input to the low-order feature space transformation 221 as a poor image patch, thereby achieving the purpose. In this embodiment, as in the first embodiment, the degradation factors are not limited to noise alone. As described above, the degradation factors may include blur, aberration, compression, low resolution, defects, contrast reduction due to the influence of fog, haze, snow, or rain during imaging, or a combination of these, and the like, and these can be solved by the same method. In addition, the image quality improvement AI according to the present embodiment has been described on the assumption that a model learned by deep learning / machine learning is applied, but the method for realizing the AI ​​is not necessarily limited. In other words, other methods that can improve image quality may also be applied.

[0099] <Fourth embodiment> As the fourth embodiment of the present disclosure, a modified form of the first embodiment will be described below. Specifically, in the first embodiment, a case where restoration of a degraded image is taken as an example, and additional learning is performed by selecting image patches from an area where the accuracy of image quality restoration is below a threshold in a test environment (i.e., an area where the accuracy of restoration performance is poor) is described. In this embodiment, a specific embodiment for realizing detection and tracking of a person will be described. A system configuration to which the information processing device according to the fourth embodiment is applied may be substantially the same as that of the first embodiment, that is, may be represented as a configuration as shown in Fig. 1. Thus, in the information processing system shown in Fig. 1, a cloud server 200 that performs estimation and learning of person detection and tracking, and an edge device 100 that performs estimation of person detection and tracking for a processing target image are connected via a network such as the Internet. In the person detection and tracking of this embodiment, when an image is input, a person is detected from the image, and the same person is tracked as long as it continues to be captured in the camera, and the same person ID is continually assigned to the person, thereby performing person monitoring.

[0100] First, an overview of the algorithm for detecting and tracking people in this embodiment will be described. In the information processing system according to this embodiment, when a video image is input, person detection is performed on one frame of the image by the person detection CNN, and the person detection is performed continuously for each frame. After that, in order to utilize the time-series context information of the images, multiple frame images and the result of the detection map by the person detection CNN are input to a person tracking CNN, and the same ID is assigned to the same person by the person tracking CNN. The above algorithm makes it possible to realize functions such as counting the number of people within the duration of a target video for surveillance purposes, or searching retroactively for people who have engaged in abnormal behavior.

[0101] Here, referring to FIG. 13, as in the first embodiment, an example of a case where person detection and tracking fails due to input of a person image that tends to be absent or present in only a small number of images in the learning dataset in the test environment will be described.

[0102] FIG. 13(A) shows a situation where the same person is walking from the upper left area to the lower right area in a time series, superimposed on one image as detection and tracking results for four frames of a video. In this case, when a person who has not been detected before in the upper left area enters the angle of view for the first time and is detected, a rectangular area 130 corresponding to the person (in other words, a detection frame of the target person) is specified according to the detection result. In this case, an ID different from the ID given to a person detected in the past is given to the person indicated by the rectangular area 130. For example, in the example shown in FIG. 13(A), a new ID of 57 is given to the person indicated by the rectangular area 130. The same person is also detected in the next frame, and a rectangular area 131 corresponding to the person is specified. At this time, the tracking CNN determines whether the person represented by each rectangular area is the same person or not, depending on whether the feature amount of the rectangular area 130 in the previous frame is similar to the feature amount of the rectangular area 131. In the example shown in FIG. 13(A), the persons represented by the rectangular areas 130 and 131 are determined to be the same person, and as a result, 57 is assigned as an ID to the person represented by the rectangular area 131.

[0103] In person tracking, if the person in the detection frame is the same in the chronological order, it may be required to continue assigning the same ID for surveillance purposes. However, there are several cases in which person detection and tracking can fail. In the example shown in FIG. 13(A), there is an area 134 in which the lighting environment changes significantly compared to other areas in the same angle of view, and a situation is diagrammatically shown in which a person to be detected and tracked enters the area. Specifically, the person 135 corresponds to the person indicated by the rectangular area 132 in the previous frame, and enters the area 134 with subsequent movement. In this case, although the person 135 could be detected as the person indicated by the rectangular area 132 in the previous frame, the features in the image are lost with the person's entry into the area 134, and as a result, the person is not detected. As a cause of such a situation, a situation in which there is a shortage of images similar to the image of the person 135 that entered the area 134 in the learning data of the CNN for person detection. Generally, a situation in which a person disappears within the same angle of view cannot be assumed, so it is possible to consider that the problem can be solved by adding an image with an appearance similar to the image within the rectangle inferred from the trajectory 133 of person tracking from the previous frame to the learning image.

[0104] In addition, the example shown in FIG. 13(B) shows a schematic diagram of a situation in which a target person who was correctly tracked as the same person in the past three frames moves into an area where the lighting environment has changed and is determined to be a different person. Specifically, in the past three frames, the target person for person detection and tracking is considered to be the same person, and the same ID 57 is assigned to each person. On the other hand, the person shown in the rectangular area 136 is the same person as the person who was assigned ID 57 in the past three frames, but his / her appearance has changed by entering an area where the lighting environment has changed (i.e., the features that are apparent on the image have changed). As a result, the person tracking CNN cannot determine that the person shown in the rectangular area 136 and the person who was assigned ID 57 in the past three frames are different people, and a new ID of 58 is assigned. Even in such a case, it is possible to consider that the problem can be solved by adding an image with an appearance similar to the image in the rectangle inferred from the trajectory of person tracking from the previous frame to the learning image.

[0105] As illustrated above, an example of a configuration and process for improving the performance of a CNN by adding appropriate image patches to a test environment at a predetermined timing during test execution will be described below with reference to FIGS. 11 and 12.

[0106] (System overall functional configuration) An example of a functional configuration of the information processing system according to this embodiment will be described with reference to Fig. 11. The information processing system according to this embodiment includes an edge device 1100 and a cloud server 1130. The edge device 1100 includes an image acquisition unit 1111, an inference-use detection and tracking unit 1112, and a poor patch image identification unit 1115. The cloud server 1130 includes a learning unit 1132. The learning unit 1132 includes a learning-use detection and tracking unit 1133, an error calculation unit 1134, and a model update unit 1135. The configuration shown in FIG. 11 is merely an example, and can be modified or changed as appropriate. For example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. The configuration shown in FIG. 11 may be realized by two or more devices. In this case, the devices are connected via a circuit or a wired or wireless network, and perform data communication with each other to perform cooperative operations, thereby realizing each process according to this embodiment.

[0107] First, each functional unit of the edge device 1100 will be described in detail. The image acquisition unit 1111 acquires input image data 1113 to be processed. The input image data 1113 is a plurality of surveillance images that are successive in time series. The detection and tracking for inference unit 1112 causes the trained model 1139 transmitted from the cloud server 1130 to perform person detection for inference for each input image, and determines whether the person detected by performing tracking for inference is the same person in previous and subsequent frames. After that, the detection and tracking for inference unit 1112 outputs an output image 1114 to which a person ID is assigned and a detection rectangle is attached to a predetermined output destination.

[0108] The weak patch image identifying unit 1115 judges the quality of the estimation result in the test environment by the inference detection and tracking unit 1112 using the trained model transmitted to the edge device 1100. Then, based on the result of the judgment, the weak patch image identifying unit 1115 identifies a weak patch where the accuracy of the estimation performance of the trained model is equal to or lower than a threshold. Specifically, the weak patch image specifying unit 1115 receives an image input from the image acquiring unit 1111 and an image equivalent to the output image 1114 to which a person ID is assigned and a detection rectangle is added by the inference detection and tracking unit 1112. Based on these images, the weak patch image specifying unit 1115 judges whether or not the detection and tracking of the target person is performed well in the test environment. Then, as a result of the above judgment, the weak patch image specifying unit 1115 determines a weak patch image group 1116 that is the cause of the judgment that the estimation result in the test environment is poor (the estimation performance is below a threshold). Then, the weak patch image specifying unit 1115 transmits the weak patch image group 1116 to the cloud server 1130.

[0109] The poor patch image group 1116 is converted into features in a learned low-level feature space by a low-level feature space transformation 1141. The poor patch image group converted into the low-level feature is input to a large image group 1140 as a query. Images in which a person is captured and stored in the large image group 1140 are given rectangular information in the same format as a rectangular area generally taught in a person detection task, as illustrated with reference to Fig. 13. In addition, in a scene with a relatively short continuous image, rectangular areas corresponding to the same person between previous and subsequent frames are given the same ID and stored. In addition, the large image group 1140 may include images of scenes in which no person is captured, and such images may be given information indicating that the rectangular area (i.e., an area corresponding to a person) is not included.

[0110] The above query is used when searching for similar image patches in order to additionally learn image patches adjacent to the weak patches generated in the test environment. Then, a learning scene including an image patch having characteristics adjacent to the weak patch image group is selected according to the query, and an image corresponding to the learning scene is added to the learning image group 1136. This learning image group 1136 is applied to model learning in the cloud server 1130 described below, making it possible to obtain a model with better performance in human detection and tracking in the test environment.

[0111] Next, each functional unit of the cloud server 1130 will be described. Images to be applied to learning are uploaded to the cloud server 1130. A data set of images to be applied to learning, including this image, is stored in the cloud server 1130 as a learning image group with correct answer information 1136. Each image in the learning image group with correct answer information 1136 is an image composed of a relatively short scene, similar to the format of the images stored in the large image group 1140 described above, and is accompanied by rectangular information indicating an area in which a person is captured and ID information assigned to the person. In addition, it is preferable that the learning image group with correct answer information 1136 contains sufficient data of images that satisfy conditions such as a background, angle of view, and person size that are closer to the test environment.

[0112] The learning unit 1132 acquires model parameters 1138 to be applied to the CNN for human detection and tracking learning, initializes the weights of the CNN using the model parameters 1138, and then learns the CNN. The model parameters 1138 include hyperparameters indicating initial values ​​of parameters of the neural network, the structure of the neural network, and an optimization method. The person detection and tracking learning in the learning unit 1132 is performed by a learning detection and tracking unit 1133, an error calculation unit 1134, and a model update unit 1135. The CNN 1133 used in the learning unit 1132 is the same neural network as the CNN used in the inference detection and tracking unit 1112. The simplest configuration is based on the premise that two types of CNNs are used: a CNN for person detection and a CNN for person tracking. The CNN for person detection is a CNN that detects a person area by inputting one frame. The CNN for person tracking is a CNN that inputs a detected person area rectangle in multiple fixed frames in the time series direction, performs a same person determination, and assigns an ID. On the other hand, the above processing may be realized by one CNN. In this case, the detection and tracking unit for learning 1133 and the detection and tracking unit for inference 1112 may be realized as one functional block.

[0113] (Processing flow of the entire system) An example of processing of the information processing system according to this embodiment will be described with reference to Fig. 12A and Fig. 12B. Each of Fig. 12A and Fig. 12B is a flowchart showing an example of the flow of processing of the information processing system according to this embodiment.

[0114] First, an example of a series of processing flows for person detection and tracking learning by cloud server 1130 will be described with reference to FIG. 12A. In S1201, the cloud server 1130 accepts input of a group of learning images to which rectangular information indicating a region of a person prepared in advance and an ID given to the person are assigned as correct answer information. In S1203, the cloud server 1130 accepts input of model parameters to be applied to the CNN used for human detection and tracking. The model parameters here are initial values ​​of parameters summarizing the weights and biases of the neural network that will result in optimal results after learning, as described above. The input model parameters are transmitted to the learning unit 1132. In S1204, the learning detection and tracking unit 1133 initializes the weights of the CNN using the received model parameters, and then performs person detection and tracking. In S1205, the error calculation unit 1134 calculates the difference between the estimated result of human detection and tracking and correct answer data (a group of learning images to which correct answer information has been added). In S1206, the model update unit 1135 updates the model parameters so that the difference calculated in S1205 becomes smaller (eventually, becomes minimum), as described above.

[0115] In S1207, the learning unit 1132 determines whether or not a learning convergence condition is satisfied. The learning convergence condition is not particularly limited and may be set appropriately depending on the use case. As a specific example, whether or not the number of updates of the model parameters of the learning unit 1132 reaches a predetermined number may be set as the learning convergence condition. If the learning unit 1132 determines in S1207 that the learning convergence condition is not satisfied, the process proceeds to S1204. In this case, learning is performed using other learning image data and supervised image data by the processes from S1204 onward. Then, if it is determined in S1207 that the learning convergence condition is satisfied, the learning unit 1132 ends the series of processes shown in FIG. 12A.

[0116] Next, an example of the flow of the person detection and tracking inference process by the edge device 1100 will be described with reference to FIG. 12B. In S1208, the edge device 1100 acquires the trained model 1139 trained by the cloud server 1130. In S1209, the image acquisition unit 1111 selects input image data of N frames (N≧1) from the input image data 1113 that is to be subjected to person detection and tracking processing, and generates input linked image data linked in the channel direction.

[0117] In S1210, the detection and tracking unit for inference 1112 constructs a CNN similar to that used in learning by the learning unit 1132, and performs person detection and tracking from the input linked image data. At this time, the detection and tracking unit for inference 1112 initializes existing model parameters with the updated model parameters received from the cloud server 1130. In this manner, the detection and tracking unit for inference 1112 inputs the input linked image data to the CNN to which the updated model parameters have been applied, and performs person detection and tracking on the input image data in a similar manner to that performed by the learning unit 1132, thereby obtaining output image data.

[0118] In S1211, the edge device 1100 outputs the output image data obtained in S1210 to a predetermined output destination as an output image 1114. As a specific example, the edge device 1100 may output the output image 1114 to the display device 40 shown in FIG. 1 to display the output image 1114 on the display device 40. As another example, the edge device 1100 may output the output image 1114 to the external storage device 30 to store the output image 1114 in the external storage device 30.

[0119] In S1212, the weak patch image specification unit 1115 judges whether the person detection and tracking results in S1211 are good or bad. The judgment of the person detection and tracking results can be performed by the method described below. Specifically, in normal operation, the same ID is not updated from when a person enters the screen until the person leaves the screen. On the other hand, behavior in which a situation occurs in which a target person is not detected due to reasons such as the conditions of a part of the screen being different from those at the time of learning is clearly unstable and can be easily distinguished.

[0120] 13(A), the target person is not detected during the period when passing through area 134, and the display of the detection frame showing the rectangular area corresponding to the person is suppressed, resulting in the detection frame disappearing from the image. In this way, when a frame is detected in which an event that cannot be expected in reality occurs, such as the sudden disappearance of a person who should be present on the screen, it is possible to determine whether or not person detection and tracking for the area of ​​the person in that frame is successful. In general applications, even if a person's detection frame disappears, it is unlikely that the person in question will suddenly disappear, so a process is often used in which the detection frame is drawn at a position predicted by rule-based processing based on the rectangular coordinates and movement speed of the past few frames. Therefore, when the result of the person detection and tracking is judged to be negative in S1212, the area to be retained as a poor patch may be an image of a rectangular area cut out using the rule-based processing exemplified above, where the person area that could not be detected by CNN is the person area.

[0121] Another example of a case where the person detection and tracking results can be judged as bad is when the appearance of the same person changes suddenly, as in the case of detecting a person in the example shown in FIG. 13(B), and the CNN for person tracking judges that the person is a different person and changes the ID. In this case, an event that would not occur in a typical surveillance scene occurs, where a person who did not exist in the previous frame suddenly appears in a part of the image. In this way, when a frame is detected in which an event that cannot be expected in reality occurs, such as a person who did not exist in the screen suddenly appears, it is possible to judge whether the person detection and tracking for the area of ​​the person in that frame is good or bad. In this case, it is also possible to store the image of the person who suddenly appeared in the screen and was given a new ID as a weak patch image.

[0122] As described above, in S1212, the poor patch image identifying unit 1115 judges the quality of the person detection and tracking result in 1211 by, for example, comparing the person detection and tracking result with a preset threshold, and identifies and holds the poor patch according to the result. In other words, the poor patch image identifying unit 1115 compares the person detection and tracking results for each frame with each other, and identifies and holds the image patch corresponding to a frame where the tendency of the person detection and tracking result is different from other frames (for example, the difference is equal to or greater than a threshold).

[0123] In S1213, the poor patch image identifying unit 1115 determines whether to update the learned human detection and tracking inference model. Specifically, when a poor patch is held, the poor patch image identifying unit 1115 determines that the learned human detection and tracking inference model needs to be updated. If the poor patch image specifying unit 1115 determines in S1213 that the learned human detection and tracking inference model is to be updated, the process proceeds to S1215. On the other hand, if the poor patch image specifying unit 1115 determines in S1213 that the learned human detection and tracking inference model should not be updated, it proceeds to S1214.

[0124] In S1215, the poor patch image specifying unit 1115 transmits the held poor patch to the cloud server 1130. In S716, the cloud server 1130 compares the huge number of image patches stored in the large image group 1140 with the poor patch transmitted in S1215, and selects a patch image having a close feature from the large image group 1140. Note that the method of obtaining the low-order distance feature space for selecting a close image is substantially similar to that in the first embodiment, and therefore a detailed description thereof will be omitted.

[0125] In S1217, the cloud server 1130 adds the weak patch adjacent image having features adjacent to the weak patch selected in S1216 to the training image group 1136 with correct answer information. After that, the cloud server 1130 executes the same process as the training flow already described after adding the weak patch adjacent image to the training image group 1136 with correct answer information. Then, when the training converges, the cloud server 1130 updates the inference human detection and tracking model in S1218 and transmits the updated model to the inference detection and tracking unit 1112 of the edge device 1100. As a result, the model of the inference detection and tracking unit 1112 is updated according to the tendency of the input image in the test environment. This is expected to improve the accuracy of human detection and tracking in the test environment.

[0126] In S1214, the edge device 1100 determines whether or not to end the person detection and tracking process. The determination of whether or not to end the person detection and tracking process in S1214 may be determined based on whether or not an instruction to end the process has been received from the user via the input device 20, for example. If the edge device 1100 determines in S1214 not to end the person detection and tracking process, it advances the process to S1209. In this case, the processes from S1209 onwards are executed again for the next frame. As described above, the processes from S1209 onwards are repeatedly executed until it is determined in S1214 that the person detection and tracking process is to end. In this way, the person detection and tracking inference model is maintained in accordance with changes in the test environment by continuously determining areas where the person detection and tracking performance is poor by the person detection and tracking inference performance determination process S1212 and executing additional learning each time. Then, if the edge device 1100 determines in S1214 to end the person detection and tracking process, it ends the series of processes shown in FIG. 12B.

[0127] In this embodiment, an example of a case where inference is performed for detection and tracking of a person has been described, but the subject of the inference is not necessarily limited to a person. That is, even in a situation where detection or identification is performed with an object other than a person as a detection target, a situation where the detection or identification fails is assumed, and the performance of the detection or identification can be improved (improved) by the additional learning described in this embodiment. In addition, various explanations have been given on the assumption that the AI ​​performing the inference in this embodiment is implemented using a model trained by deep learning / machine learning, but the method is not particularly limited as long as the same purpose can be achieved.

[0128] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0129] The disclosure of this embodiment also includes the following configurations and methods. (Configuration 1) A means for identifying a partial image corresponding to an area in an image where the performance of inference by a learning model that performs a predetermined inference on an input image is below a threshold, a collection means for collecting similar images similar to the partial image identified by the identification means, and a learning means for improving the result of the inference by the learning model in a test environment different from the learning environment of the learning model, using additional images including the similar images collected by the collection means. An information processing device comprising: (Configuration 2) The information processing device described in Configuration 1, characterized in that the identification means identifies the partial image in which the inference performance of the learning model for the input image is below a threshold during the learning process of the learning model. (Configuration 3) The information processing device according to configuration 1 or 2, wherein the additional image includes the similar image collected by the collection means and the partial image identified by the identification means. (Configuration 4) The information processing device described in any one of configurations 1 to 3, characterized in that the collection means collects images that are closer to the partial image identified by the identification means in a predefined low-order distance space as similar images that are similar to the partial image. (Configuration 5) The information processing device described in any one of configurations 1 to 4, characterized in that the learning model performs restoration on an input image and outputs a restored image as a result of the inference, and the identification means identifies the partial image for which the performance of the inference is below a threshold by comparing a cumulative average image of images of a stationary area from a plurality of images consecutive in a chronological order with the restored image output as a result of the inference by the learning model. (Configuration 6) The information processing device described in any one of configurations 1 to 4, characterized in that the learning model detects a specific detection target in an input image and outputs a detection result for the detection target as a result of the inference, and the identification means compares the detection results for the detection target from multiple images that are consecutive in a chronological order with each other, and identifies a partial image corresponding to the detection result for the detection target from an image that has a different tendency from the detection result for the detection target from other images. (Configuration 7) The information processing device according to any one of configurations 1 to 6, characterized in that the learning means learns the specified inference by updating parameters of the learning model using the additional image. (Configuration 8) An information processing device characterized by having an acquisition means for acquiring a partial image corresponding to an area in an image identified based on a cumulative average image of a plurality of images, where the performance of inference by a learning model that performs a predetermined inference on an input image is below a threshold value, a collection means for collecting similar images that are similar to the partial image acquired by the acquisition means, and a learning means for improving the results of the inference by the learning model, targeting a test environment different from the learning environment of the learning model, using additional images including the similar images collected by the collection means. (Method 1) A control method for an information processing device, comprising: a determination step for determining a partial image corresponding to an area in an image where the performance of inference by a learning model that performs a predetermined inference on an input image is below a threshold; a collection step for collecting similar images that are similar to the partial image determined in the determination step; and an additional learning step for improving the results of the inference by the learning model in a test environment different from the learning environment of the learning model, using additional images including the similar images collected in the collection step. [Explanation of symbols]

[0130] 100 Edge Devices 115 Degradation restoration performance judgment processing unit 200 Cloud Servers 212 Learning Department< / cnn>

Claims

1. a means for identifying a partial image corresponding to a region in an image input to a learning model where the performance of the inference is equal to or less than a threshold when the inference by the learning model that performs a predetermined inference on an input image is performed in a test environment different from the learning environment of the learning model; a collection means for collecting similar images similar to the partial image identified by the identification means; a learning means for updating parameters of the learning model using additional images including the similar images collected by the collection means; An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein the specifying means specifies the partial image in which the inference performance of the learning model for the input image is equal to or lower than a threshold during the learning process of the learning model.

3. The information processing apparatus according to claim 1 , wherein the additional images include the similar images collected by the collecting means and the partial images identified by the identifying means.

4. 2. The information processing device according to claim 1, wherein the collection means collects, in a predefined low-order distance space, images that are closer to the partial image identified by the identification means as the similar images that are similar to the partial image.

5. The learning model performs restoration on the input image, and outputs a restored image as a result of the inference; The specifying means compares a cumulative average image of images of a region in a stationary state from a plurality of images consecutive in a time series direction with the restored image output as a result of the inference by the learning model, and specifies the partial image for which the performance of the inference is equal to or lower than a threshold.

2. The information processing device according to claim 1, wherein:

6. the learning model detects a predetermined detection target in an input image and outputs a detection result of the detection target as a result of the inference; The specifying means compares the detection results of the detection target from a plurality of images that are consecutive in a time series direction with each other, and specifies a partial image according to the detection result of the detection target from an image that has a different tendency from the detection result of the detection target from another image.

2. The information processing device according to claim 1, wherein:

7. An acquisition means for acquiring a partial image corresponding to a small area in an image input to a learning model, when the difference between a cumulative average image generated based on a still area determined to have no subject movement in a plurality of images acquired successively in a time series direction and an area corresponding to the still area in a restored image restored by inputting an image selected from the plurality of images into the learning model exceeds a predetermined threshold; a collection means for collecting similar images similar to the partial images acquired by the acquisition means; a learning means for updating parameters of the learning model using additional images including the similar images collected by the collection means; An information processing device comprising:

8. A control method for an information processing device, comprising: a step of identifying a partial image corresponding to a region in an image input to a learning model where the performance of the inference is equal to or less than a threshold when the inference by the learning model that performs a predetermined inference on an input image is performed in a test environment different from the learning environment of the learning model; a collection step of collecting similar images similar to the partial image identified in the identification step; an additional learning step of updating parameters of the learning model using additional images including the similar images collected in the collection step; 10. A method for controlling an information processing device, comprising:

Images