Information processing device, information processing method, and program

Abstract

This technology provides information about anomalies in the output of machine learning models. [Solution] The information processing device comprises a first image processing means having a first machine learning model that performs first image processing to improve the quality of an input image, and a first anomaly detection means that generates a first anomaly detection result as information about anomalies included in the first output result of the first machine learning model, based on a first output result that includes at least one of the image and feature quantities output by the first machine learning model.

Description

Technical Field

[0001] The present invention relates to technologies related to machine learning models.

Background Art

[0002] In recent years, as image processing technologies for improving the image quality of images and videos, methods using machine learning have been actively developed. In particular, technologies for realizing high-image-quality image processing such as noise removal, blur removal, and super-resolution using neural networks have been developed.

[0003] On the other hand, high-image-quality image processing realized by machine learning may cause extreme deterioration or abnormalities in the image processing results. For example, artifacts may occur in the image processing results, a part of the image may be missing, or the image may not be processed. Such phenomena occur, for example, when data in a domain not learned in a learned machine learning model (hereinafter also referred to as a model) is input, and the machine learning model cannot perform appropriate processing on data that it does not assume as input. In addition to this, in a machine learning model that maintains a state, it is conceivable that the output becomes abnormal due to the model being in an incorrect state. For example, consider a machine learning model that inputs temporally continuous images and updates and maintains the state each time an image is input to the model. At this time, assume that a temporally discontinuous image is input to the model. In that case, the internal state maintained by the model and the images input to the model become temporally discontinuous. As a result, a state not assumed by the model occurs, and thus an abnormality may occur in the output. Such abnormal output is a major problem because it leads to a decrease in the reliability of the system and the evidentiary ability of videos.

[0004] As a technique for detecting data from domains that have not been trained, as described above, machine learning-based out-of-distribution detection methods have been proposed. For example, Variational Autoencoders (VAEs) described in Patent Document 1 and Non-Patent Document 1 are sometimes used for out-of-distribution detection. By using such VAE-based out-of-distribution detection methods, it is possible to detect input data from domains that have not been trained by the machine learning model. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Special Publication No. 2023-515367 [Non-patent literature]

[0006] [Non-Patent Document 1] An Introduction to Variational Autoencoders, Diederik P. Kingma and Max Welling., Foundations and Trends in Machine Learning,Vol. 12,2019. [Non-Patent Document 2] Toward Convolutional Blind Denoising of Real Photographs, Shi Guo et al., CVPR2019. [Non-Patent Document 3] Image Super-Resolution Using Deep Convolutional Networks, Chao Dong et al., IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016. [Overview of the project] [Problems that the invention aims to solve]

[0007] However, it is not possible to generate information about anomalies in the output of machine learning models.

[0008] Therefore, the present invention provides a technique for generating information regarding anomalies in the output of machine learning models. [Means for solving the problem]

[0009] To solve this problem, for example, the information processing apparatus of the present invention has the following configuration. That is, A first image processing means having a first machine learning model that performs first image processing to improve the image quality of an input image, A first anomaly detection means that generates a first anomaly detection result as information about anomalies included in the first output result of the first machine learning model, based on a first output result that includes at least one of an image and features output by the first machine learning model, It is equipped with. [Effects of the Invention]

[0010] According to the present invention, it is possible to generate information regarding anomalies in the output of a machine learning model. [Brief explanation of the drawing]

[0011] [Figure 1] A block diagram showing an example of the hardware configuration of an information processing device according to the embodiment. [Figure 2] A block diagram showing an example of the functional configuration of the information processing device and learning device according to the first embodiment. [Figure 3] Anomaly detection processing flow of the information processing device of the first embodiment. [Figure 4] A diagram showing the configuration of the neural network that performs noise reduction according to the first embodiment. [Figure 5] A diagram showing the configuration of the autoencoder used in the first embodiment. [Figure 6] This diagram shows the relationship between the inverse-phase noise image of the noise reduction model's output and the inverse-phase noise image restored by the autoencoder, and the resulting anomaly score. [Figure 7] Learning processing flow of the learning device according to the first embodiment. [Figure 8] Configuration diagrams of a machine learning model and an autoencoder that execute super-resolution processing in a modification example. [Figure 9] Block diagram showing a functional configuration example of the information processing device according to the second embodiment. [Figure 10] Abnormality detection processing flow of the information processing device according to the second embodiment. [Figure 11] Block diagram showing a functional configuration example of the information processing device according to the third embodiment. [Figure 12] Abnormality type discrimination processing flow of the information processing device according to the third embodiment. [Figure 13] Outline of the inference processing of the neural network that performs noise removal according to the third embodiment. [Figure 14A] Diagram showing the structure of the neural network of the first stage according to the third embodiment. [Figure 14B] Diagram showing the structure of the neural network of the internal state generation module according to the third embodiment. [Figure 14C] Diagram showing the structure of the neural network of the second stage according to the third embodiment.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and duplicate explanations are omitted.

[0013] <First Embodiment> Hereinafter, based on the first embodiment, it will be described in detail with reference to the attached drawings. Note that the configurations shown in the following embodiments are merely examples and are not limited to the illustrated configurations.

[0014] Figure 1 is a block diagram showing an example of the hardware configuration of the information processing device 100 in this embodiment. The information processing device 100 detects abnormalities in the output, such as a decrease in the performance of a machine learning model.

[0015] The information processing device 100 may be a general-purpose computer. The information processing device 100 comprises a CPU 101, memory 102, input unit 103, storage unit 104, display unit 105, communication unit 106, and bus 107. The CPU 101, memory 102, input unit 103, storage unit 104, display unit 105, and communication unit 106 are connected to each other via the bus 107 so that they can send and receive data.

[0016] CPU101 stands for Central Processing Unit and is a processor. CPU101 is responsible for the overall control of the information processing device 100.

[0017] The information processing device 100 may have other processors such as an MPU (Micro Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), and QPU (Quantum Processing Unit) in place of or in addition to the CPU 101. Furthermore, the information processing device 100 may have multiple processors of the same type, each performing a different function.

[0018] Some or all of the functions of the information processing device 100 are realized by one or more processors, including the CPU 101, reading computer programs (hereinafter also referred to as programs) stored in the storage unit 104, expanding them in the memory 102, and executing them. Alternatively, some or all of the functions of the information processing device 100 may be realized by one or more circuits, such as an ASIC (Application Specific Integrated Circuit) and a PLD (Programmable Logic Device) including an FPGA (Field Programmable Gate Array).

[0019] Memory 102 may be, for example, RAM (Random Access Memory). Memory 102 may be a high-speed read and write storage medium. Memory 102 functions as a working area when the CPU 101 executes a program. Memory 102 stores image data that the program will process, as well as parameters necessary for program execution.

[0020] The input unit 103 receives input from a user or other source and outputs it to the CPU 101. The input unit 103 may be a keyboard, mouse, touch panel, or the like.

[0021] The memory unit 104 may be a non-volatile storage device such as a ROM (Read Only Memory), HDD (Hard Disk Drive), or SSD (Solid State Drive). The memory unit 104 stores the program to be read by the CPU 101, the data to be processed by the program, and the parameters necessary for the execution of the program.

[0022] The display unit 105 may be a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 105 displays an image using image data acquired from the CPU 101.

[0023] The communication unit 106 may be an interface for realizing communication to send and receive information with external devices, etc. The communication unit 106 may realize communication via a LAN (Local Area Network) and a WAN (Wide Area Network), etc.

[0024] Figure 2 is a block diagram showing an example of the functional configuration of an information processing device and a learning device according to the first embodiment. Figure 2(a) is a block diagram showing an example of the functional configuration of an information processing device 100 according to the first embodiment.

[0025] The information processing device 100 detects abnormalities in the output of a machine learning model that performs high-resolution image processing. The information processing device 100 performs high-resolution image processing on the image captured by the imaging device 120 and determines whether there are any abnormalities in the image processing results. The imaging device 120 may be, for example, a so-called digital camera that outputs image data obtained by converting light from a subject into an electrical signal.

[0026] The information processing device 100 includes an image acquisition unit 110, an image processing unit 111, an anomaly detection unit 112, a display unit 113, a notification unit 114, and a recording unit 115. The processor, including the CPU 101, may implement all or part of the functions of the image acquisition unit 110, image processing unit 111, anomaly detection unit 112, display unit 113, notification unit 114, and recording unit 115 by reading and executing a program stored in the storage unit 104.

[0027] The image acquisition unit 110 acquires images captured by the imaging device 120 as input images and outputs them to the image processing unit 111.

[0028] The image processing unit 111 performs image quality enhancement processing on the input image received from the image acquisition unit 110, using a machine learning model that has been trained to perform image quality enhancement processing using machine learning techniques to improve image quality.

[0029] The anomaly detection unit 112 detects anomalies in the image processing result and determines whether or not an anomaly has occurred, based on the output result which includes at least one of the image and feature quantities, which are the image processing results output by the image processing unit 111's high-resolution image processing. The anomaly detection unit 112 also generates and outputs an anomaly detection result as a detection result, which includes information about the anomaly, such as the location of the anomaly within the image where the anomaly has occurred. Furthermore, the anomaly detection unit 112 may generate an anomaly detection result by treating the anomaly in the image processing result as an anomaly in the machine learning model.

[0030] The display unit 113 displays information about the abnormality, such as the location of the abnormality, as indicated by the abnormality detection result detected by the abnormality detection unit 112, to the user on a monitor or the like.

[0031] The notification unit 114 notifies the user that an anomaly has been detected when the anomaly detection unit 112 detects an anomaly.

[0032] The recording unit 115 records information about anomalies based on the detection results detected by the anomaly detection unit 112. For example, the recording unit 115 may record the presence or absence of an anomaly in association with the results of image processing.

[0033] In this embodiment, as an example of image processing, we will describe a noise reduction process that removes noise from a noisy image containing noise and generates a noise-free image. Note that the same configuration and processing can be applied not only to noise reduction but also to other image enhancement processes such as super-resolution, blur reduction, and de-blurring.

[0034] With respect to this embodiment configured as described above, a method for detecting abnormalities present in the high-resolution image processing results using the information processing device 100 will be explained according to the example of the functional configuration in Figure 2(a) and the processing flow in Figure 3. Figure 3 is the abnormality detection processing flow of the information processing device of the first embodiment.

[0035] In S301, the image acquisition unit 110 acquires the image captured by the imaging device 120. The image acquisition unit 110 outputs the acquired image to the image processing unit 111.

[0036] In S302, the image processing unit 111 performs noise reduction image processing on the image acquired in S301. At this time, the image processing unit 111 outputs an inverted-phase noise image generated during the noise reduction process to the anomaly detection unit 112.

[0037] Convolutional Neural Networks (CNNs) are sometimes used as machine learning techniques to achieve high-resolution image processing. A CNN is a neural network composed of numerous convolutional layers and activation functions. In particular, a network called a U-Net, which has a U-shaped structure, is used as a CNN to achieve high-resolution image processing such as noise reduction and super-resolution. The network in Non-Patent Document 2 also uses a U-Net to perform noise reduction. This embodiment also uses a U-Net.

[0038] Figure 4 shows the configuration of the neural network that performs noise reduction in the first embodiment. The processing of the neural network used in this embodiment will be explained in detail using Figure 4. The neural network has an encoder that generates feature quantities while compressing the image, and a decoder that reconstructs the image from the compressed feature quantities.

[0039] The neural network encoder generates multiple feature vectors with different resolutions and channel counts from the input image 401. In the encoder's processing, the neural network generates feature vectors 412 by repeatedly applying convolution and the ReLU function 411 to the input image 401. The neural network reduces the resolution of the generated feature vectors 412 through pooling 413. The neural network then increases the channel count of feature vectors 412 by repeatedly applying convolution and the ReLU function to the reduced-resolution feature vectors. Furthermore, during the image reconstruction process described later, the neural network skip-combines 414 the feature vectors 412 generated at this time with other feature vectors generated through upsampling.

[0040] Next, the neural network decoder performs a deconvolution operation 415 on the compressed feature vectors 412, thereby reducing the number of channels in the feature vectors 412 and increasing the resolution of the feature vectors 412, while restoring the feature vectors 412 to an image. In the deconvolution operation 415, the neural network skip-combines the upsampled feature vectors and the feature vectors generated by the encoder, and repeats the process 411, which applies multiple convolution operations and the ReLU function, and the deconvolution operation 415.

[0041] The neural network repeats the series of processes described above to ultimately output an image with the desired resolution and number of channels. The output image is a noise image 402 which is the inverse of the noisy image, which is the input image 401. Therefore, the neural network performs an addition process 416 which adds the inverse noise image 402 and the noisy image, which is the input image 401 of the neural network, to output a denoised image 403, which is an image from which the noise has been removed. The inverse noise image 402 is used as input to the anomaly detection unit 112 in the next process S303. This embodiment uses the neural network described above, but is not limited to a neural network; any image processing method based on machine learning techniques that can obtain an intermediate output of the image processing is acceptable.

[0042] Furthermore, in image processing other than noise reduction, intermediate features output during the image processing process, such as the inverted-phase noise image 402, may be used as input to the anomaly detection unit 112 in the next process S303. Alternatively, instead of the inverted-phase noise image output from the network, the final processing result, the noise-reduced image 403, may be used as input to the next anomaly detection unit 112. In the above case, the input image and output image of the machine learning-based model used in the anomaly detection unit 112 will be the intermediate features output during the image processing process and the final processing result, the noise-reduced image 403.

[0043] The neural network in this embodiment may use a pre-trained model that has acquired parameters capable of performing high-resolution image processing. The pre-trained model is trained based on a supervised learning method. In the supervised learning method, a large set of images is prepared, consisting of noisy images containing noise and clean images without noise for a given image. When a noisy image is given as input to the neural network, it obtains a denoised image as output by removing the noise from the noisy image. Based on backpropagation, which calculates the error using the denoised image and the clean image, the parameters of the neural network, such as weights and biases, are repeatedly updated. This results in a neural network that performs denoising.

[0044] In S303, the anomaly detection unit 112 compresses the inverted-phase noise image acquired in S302 into features, and then performs a process to restore it to the original inverted-phase noise image that was input. The anomaly detection unit 112 may perform this process using a machine learning-based model.

[0045] The anomaly detection unit 112 uses an autoencoder as a machine learning-based model. The autoencoder compresses the input data into features and then reconstructs the compressed features back into the input data. As described in Non-Patent Literature 1, the autoencoder acquires a feature representation of the input data projected onto the latent space through learning. However, the machine learning-based model used by the anomaly detection unit 112 is not limited to an autoencoder; any model that compresses the input data into features and then reconstructs the compressed data back into the input data is acceptable.

[0046] Figure 5 shows the configuration of the autoencoder used in the first embodiment. The autoencoder includes an encoder that generates feature quantities while compressing an image, and a decoder that reconstructs the image from the compressed feature quantities.

[0047] The autoencoder repeatedly performs multiple convolution operations and ReLU activation function processing 511, and pooling operations 512. This increases the number of channels in the inverse-phase noise image 402 of the input denoising model's output, compressing it into a feature vector with reduced resolution. Finally, the autoencoder obtains the feature vector 501 via a fully connected layer 513.

[0048] Next, the autoencoder's decoder repeatedly performs the deconvolution operation 514 and the convolution operation and activation function relu processing 511 to reconstruct the features to the same resolution and number of channels as the input inverted-phase noise image 402. As a result, the autoencoder obtains the reconstructed inverted-phase noise image 502.

[0049] In S304, the anomaly detection unit 112 calculates the degree of anomaly using the inverted-phase noise image output by the image processing unit 111 and the inverted-phase noise image restored by the anomaly detection unit 112 itself.

[0050] The anomaly detection unit 112 may calculate the degree of anomaly as the error between the inverted-phase noise image output by the image processing unit 111 and the inverted-phase noise image restored by the anomaly detection unit 112 itself. The error referred to here may be the difference in signal intensity at each pixel between the images. In the following explanation, the error may be called the degree of anomaly, and the degree of anomaly may be called the error. For example, the anomaly detection unit 112 may calculate the degree of anomaly E based on formula (1). Here, the value N is the value of each pixel in the inverted-phase noise image output by the noise reduction model. The value Nrec is the value of each pixel in the inverted-phase noise image restored by the autoencoder. The pixel value may be the signal intensity of the pixel or the signal intensity of the noise of the pixel. As shown in formula (1), the anomaly detection unit 112 may calculate the degree of anomaly E by the absolute value of the difference between the value N of each pixel in the inverted-phase noise image and the value Nrec of each pixel in the inverted-phase noise image.

number

[0051] An autoencoder can reconstruct the original input data if the input data is projectable into the latent space using parameters acquired through training. This means that the features are similar to those of the training data used to train the autoencoder. On the other hand, if the input data is difficult to represent in the latent space, that is, if it is data not present in the training data, the autoencoder cannot successfully reconstruct the input data. Using this property, the degree of anomaly (i.e., error) will be small if the inverted noise image output based on the neural network's image processing is similar to the features of the inverted noise used to train the autoencoder. Conversely, the degree of anomaly will be large if the noise features of the inverted noise image input to the autoencoder differ from those of the inverted noise image used to train the autoencoder.

[0052] Figure 6 shows the relationship between the degree of anomaly calculated from the inverse-phase noise image of the output of the denoising model and the inverse-phase noise image restored by the autoencoder. In the example shown in Figure 6, the denoising model is given an input image 601 that includes a noise region 605 with noise characteristics that the denoising model has not learned. In this case, the denoising model often cannot correctly denoise noise with characteristics that it has not learned. Therefore, in the inverse-phase noise image 602 output by the denoising model, a region 606 of inverse-phase noise with characteristics different from those learned occurs. Similarly, the autoencoder is unable to correctly restore the region 606 of inverse-phase noise with characteristics different from those learned. Therefore, in the inverse-phase noise image 603 restored by the autoencoder, a region 607 with low restoration accuracy occurs. The anomaly detection unit 112 calculates the degree of anomaly 604 of the autoencoder using the generated inverse-phase noise image 602 of the output of the denoising model and the inverse-phase noise image 603 restored by the autoencoder. As a result, a region 608 with a high degree of abnormality occurs in the abnormality degree 604 calculated by the abnormality detection unit 112, relative to the noise region 605 which has different characteristics from the input image during training. In addition to the above example, it is also conceivable that the degree of abnormality of the autoencoder may increase when a CG image is input to an autoencoder that was trained using images of the real world during training.

[0053] In S305, the anomaly detection unit 112 detects anomalies, including abnormal parts in the output of the image processing unit 111, based on the degree of anomaly calculated in S304.

[0054] The anomaly detection unit 112 detects anomalies in the image based on the degree of anomaly calculated in S304. For example, the anomaly detection unit 112 determines that a pixel whose degree of anomaly exceeds a threshold is an anomaly and detects it as such. The threshold may be set in advance for the degree of anomaly. The threshold may be set based on the statistical value of the reconstruction error, calculated from the error calculated using the same method as the calculation method for the degree of anomaly when training the autoencoder. For example, the anomaly detection unit 112 may calculate the mean and variance of the reconstruction error of the autoencoder during training, and determine that it is not an anomaly if the degree of anomaly calculated in S305 falls within a predetermined range of the variance of the reconstruction error during training. On the other hand, the anomaly detection unit 112 may determine it is an anomaly if the degree of anomaly is outside the range of the variance. Furthermore, the anomaly detection unit 112 may determine the machine learning model to be abnormal if an anomaly exists, or if a certain number or proportion of anomalies exist in the image.

[0055] The following processing assumes a camera system in which the user monitors the video from which noise has been removed using the information processing device 100. In this assumption, if the noise removal processing result contains an abnormal output value, the anomaly detection unit 112 performs the following processing.

[0056] In S306, the notification unit 114 notifies the user of an anomaly if an anomaly is detected by the anomaly detection unit 112. Specifically, the notification unit 114 notifies the user that the image processing result contains an abnormal output value. The notification unit 114 may notify the user by displaying a screen showing the anomaly in the image processing result on a monitoring monitor or by sending an email. The notification unit 114 may also notify the user that there is an anomaly in the machine learning model based on the anomaly.

[0057] In S307, the display unit 113 displays the anomaly detection result to the user if an anomaly is detected by the anomaly detection unit 112. For example, the display unit 113 displays the anomaly detection result to the user by superimposing the image processing result and the detected anomaly. The display unit 113 may also display a message indicating that there is an anomaly in the machine learning model based on the anomaly.

[0058] The display unit 113 may display an image in which the image processing result and an image created according to the magnitude of the anomaly level are superimposed, as an anomaly detection result in the image processing result. For example, the display unit 113 may generate an image as an anomaly detection result by alpha blending a noise-removed image and an image indicating the anomaly detection location created according to the anomaly level value, and display it on the monitoring monitor.

[0059] In S308, the recording unit 115 tags the image processing result with whether or not an anomaly is present based on the anomaly detection result from the anomaly detection unit 112, and records it in the storage unit 104 along with the image processing result. The recording unit 115 may also store the location of the anomaly in the storage unit 104 along with whether or not an anomaly is present, linking it to the image processing result.

[0060] The recording unit 115 stores the recorded denoised image and whether or not anomalies have been tagged, allowing the user to refer to the recorded denoised image and see if the denoising result contains any abnormal output values.

[0061] Next, a learning device 130 for learning the autoencoder used in the anomaly detection unit 112 in this embodiment will be described. Figure 2(b) is a block diagram showing an example of the functional configuration of the learning device 130 according to the first embodiment. The hardware configuration of the learning device 130 is the same as that of the information processing device 100 shown in Figure 1, so its description will be omitted. The learning device 130 may also be mounted on the information processing device 100 that detects anomalies.

[0062] The learning device 130 in Figure 2(b) is used to train an autoencoder that compresses and restores the output of a machine learning model that performs high-resolution image processing.

[0063] The learning device 130 comprises a database unit 140, an image acquisition unit 110, an image processing unit 111, an anomaly detection unit 112, and a learning unit 141. The processor, including the CPU 101, may implement all or part of the functions of the database unit 140, the image acquisition unit 110, the image processing unit 111, the anomaly detection unit 112, and the learning unit 141 by reading and executing a program stored in the storage unit 104.

[0064] The database unit 140 stores the image set used when training the denoising model employed by the image processing unit 111. This image set consists of a set of images: a noisy image in which noise has been added to the image, and a clean image in which the same image does not contain noise.

[0065] The image acquisition unit 110 acquires an arbitrary image from the image collection stored in the database unit 140. The image processing unit 111 takes an image as input, performs image processing, and outputs the image or image features generated during the process to the anomaly detection unit 112.

[0066] The anomaly detection unit 112 uses an autoencoder to compress the input image or image features, and then performs a process to restore the input image or image features.

[0067] The learning unit 141 updates the learning parameters of the neural network that constitutes the autoencoder used in the anomaly detection unit 112 based on the backpropagation method.

[0068] With respect to this embodiment configured as described above, a learning method for the autoencoder used in the anomaly detection unit 112 of the information processing device 100 will be explained according to the example of the functional configuration in Figure 2(b) and the learning process flow in Figure 7. Figure 7 is the learning process flow of the learning device of the first embodiment.

[0069] In S701, the learning device 130 performs training on the model used by the autoencoder used by the anomaly detection unit 112. The learning device 130 repeatedly updates the parameters, including the weights and biases of each layer that make up the autoencoder model, by repeating the training process. At the start of training, initial parameter values ​​are given, and thereafter, the learning device 130 updates the parameters of each layer through repeated training. Training may be terminated after a predetermined number of repetitions.

[0070] In S702, the image acquisition unit 110 acquires an arbitrary noisy image from the image group stored in the database unit 140. The image acquisition unit 110 outputs the acquired noisy image to the image processing unit 111.

[0071] In S703, the image processing unit 111 performs image processing on the noisy image acquired in S702. The image processing unit 111 generates an inverted noise image using the same image processing as in S302, and outputs the inverted noise image to the anomaly detection unit 112 and the learning unit 141.

[0072] In S704, the anomaly detection unit 112 uses an autoencoder to compress the inverted noise image into features using the same processing as in S303, and then reconstructs the original inverted noise image using the features. The anomaly detection unit 112 outputs the reconstructed inverted noise image to the learning unit 141.

[0073] In S705, the anomaly detection unit 112 calculates the error using the inverted-phase noise image restored by the autoencoder and the inverted-phase noise image output by the noise reduction model of the image processing unit 111. The anomaly detection unit 112 calculates the error in the same way as the degree of anomaly E shown in equation (1), and the average value of the errors of all elements may be used as the final restoration error. The entire element refers, for example, to all pixels of the image. Therefore, the restoration error may be the arithmetic mean obtained by dividing the sum of all errors in the image by the number of errors.

[0074] In the explanation of S305, it was stated that the threshold for the degree to which the anomaly detection unit 112 determines an anomaly may be determined based on the statistical value of the restoration error when learning the autoencoder. The anomaly detection unit 112 accumulates the restoration error calculated as described above each time the processes from S702 to S706, which are repeatedly executed, and calculates the mean and variance of all the restoration errors calculated in the learning process. The anomaly detection unit 112 may use at least one of the calculated mean and variance of the restoration error as the threshold for determining an anomaly, as described in the explanation of the process in S305.

[0075] In S706, the learning unit 141 updates the learning parameters of the autoencoder network constituting the anomaly detection unit 112 based on the backpropagation method, using the error or recovery error calculated in S705.

[0076] The learning device 130 may terminate by repeating the neural network learning process, which is executed repeatedly from S701 to S706, a predetermined number of times.

[0077] The learning device 130 acquires an autoencoder that realizes the function of the anomaly detection unit 112 of the information processing device 100 by executing a learning process.

[0078] As described above, the first embodiment can detect abnormal output included in the output of a machine learning model that performs high-resolution image processing and generate an anomaly detection result as information about the anomaly. Therefore, the first embodiment can use the anomaly detection result to realize functions such as notifying, displaying, and recording the presence or absence of an anomaly to a user of the information processing device 100.

[0079] In the first embodiment, the difference in signal intensity between the pixels of the out-of-phase noise image and the pixels of the restored reverse-flow noise image is used as the degree of abnormality, so that abnormalities can be detected according to the degree of abnormality.

[0080] In the first embodiment, anomalies are detected based on a degree threshold, which is a statistical value of the degree of anomaly calculated during learning, thus enabling more accurate detection of anomalies.

[0081] <Variation> In the first embodiment, noise reduction processing was described as an example of image processing for improving image quality. As a modification, an example of applying the first embodiment to super-resolution processing is described. In super-resolution processing as well, the invention can be realized by a machine learning model that performs super-resolution processing and an autoencoder that takes the output of the machine learning model as input, compresses it into features, and restores the original input from the compressed features.

[0082] Figure 8 shows a diagram of the machine learning model and autoencoder that perform super-resolution processing. The machine learning model that performs super-resolution processing in Figure 8 is realized using the same method as in Non-Patent Document 3. When a low-resolution image 801 is given as input, the machine learning model performs convolution operations and processing with the ReLU activation function multiple times on the low-resolution image 801, and the processing is realized by a neural network that outputs a high-resolution image 803.

[0083] An autoencoder 805 is connected to this neural network to compress the intermediate feature vector 802, which is output during the super-resolution processing, into a feature vector and generate the input intermediate feature vector 806. The autoencoder 805 uses the same configuration as the autoencoder in Figure 5. During training, the autoencoder 805 calculates the error between the intermediate feature vector 802 output by the machine learning model that performs the super-resolution processing and the reconstructed intermediate feature vector 806. The learning device updates the learning parameters of the autoencoder using a backpropagation method based on the calculated error. In a modified example, the degree of anomaly in the output of the machine learning model that performs super-resolution can be calculated in the same manner as in the first embodiment using the trained autoencoder 805. In this modified example, by using a machine learning model that performs super-resolution processing and an autoencoder, it is possible to detect anomalies that occur in the output results even in super-resolution processing.

[0084] <Second Embodiment> In the first embodiment, a method for detecting anomalies in the output of a machine learning model that performs high-resolution image processing was described. In the second embodiment, a process for switching to the optimal model from among multiple machine learning models, or for suggesting the optimal model to the user, is described based on the degree of anomaly detected.

[0085] Figure 9 is a block diagram showing an example of the functional configuration of the information processing device 900 according to the second embodiment. The hardware configuration of the information processing device 900 is the same as that of the information processing device 100 shown in Figure 1, so a description is omitted.

[0086] The information processing device 900 includes an image acquisition unit 110, a first image processing unit 910, a first anomaly detection unit 911, a second image processing unit 912, a second anomaly detection unit 913, a model selection unit 914, and a notification unit 114. The processor, including the CPU 101, may implement all or part of the functions of the image acquisition unit 110, the first image processing unit 910, the first anomaly detection unit 911, the second image processing unit 912, the second anomaly detection unit 913, the model selection unit 914, and the notification unit 114 by reading and executing a program stored in the storage unit 104.

[0087] The first image processing unit 910 and the second image processing unit 912 each have machine learning models trained to perform image enhancement processing using machine learning techniques, and they perform image enhancement processing on the input image. In other words, the first image processing unit 910 and the second image processing unit 912 perform the same type of image processing.

[0088] The first anomaly detection unit 911 uses the image or feature quantities output by the image processing of the first image processing unit 910 to detect anomalies in the image processing results based on the degree of anomaly, and generates and outputs an anomaly detection result that includes at least one of the degree of anomaly, the anomaly location, and the average value of the degree of anomaly.

[0089] The second anomaly detection unit 913 detects anomalies in the image processing results of the second image processing unit 912 based on the degree of anomaly, and generates and outputs an anomaly detection result that includes at least one of the degree of anomaly, the anomaly location, and the average value of the degree of anomaly.

[0090] The model selection unit 914 performs a selection process to select the optimal image processing unit from among multiple image processing units under the operating environment of the information processing device 900. For example, the model selection unit 914 may compare at least one of the anomaly detection results and the average value of the degree of anomaly output by each anomaly detection unit 911, 913 and select either the image processing unit 910 or 912 as the optimal image processing unit.

[0091] The other functional units have the same functions as the information processing device 100 described in the first embodiment, so they are given the same reference numerals and their descriptions are omitted.

[0092] The second embodiment, configured as described above, will be explained in detail with reference to the example of the functional configuration in Figure 9 and the flowchart in Figure 10. Figure 10 is an anomaly detection processing flow of the information processing device according to the second embodiment. In the second embodiment, as with the first embodiment, noise reduction processing will be used as an example of high-resolution image processing to be explained. In the second embodiment, processing similar to that in the first embodiment will be simplified or omitted from the explanation.

[0093] In S1001, the image acquisition unit 110 acquires the image captured by the imaging device 120. The image acquisition unit 110 outputs the acquired image to the first image processing unit 910 and the second image processing unit 912.

[0094] In S1002, the first image processing unit 910 performs noise reduction image processing on the image acquired in S1001. The first image processing unit 910 outputs the inverse-phase noise image generated during the noise reduction process to the first anomaly detection unit 911.

[0095] The first image processing unit 910 performs noise reduction using a machine learning model trained to perform noise reduction on an input image, similar to the first embodiment. However, the machine learning model of the first image processing unit 910 may be a pre-trained model that differs from the machine learning model of the second image processing unit 912 in either its training data or at least its training method. Furthermore, the model structure of the first image processing unit 910 may differ from the model structure of the pre-trained model used in the second image processing unit 912. For example, the machine learning model used in the first image processing unit 910 is a model trained to remove noise from an image containing Gaussian noise that follows a Gaussian distribution.

[0096] In S1003, the first anomaly detection unit 911 takes the inverted noise image acquired in S1002 as input, compresses the inverted noise image into features, and then performs compression and restoration processing to restore the features to the original inverted noise image. The first anomaly detection unit 911 performs this processing using a machine learning-based model.

[0097] The first anomaly detection unit 911 may compress the input data into features using an autoencoder, as in the first embodiment, and then restore the compressed features to the input data. However, the autoencoder used in the first anomaly detection unit 911 may be a model that takes the output of a machine learning model capable of performing high-quality image processing used in the first image processing unit 910 as input and has been trained to restore the original input. Therefore, the autoencoder used in the first anomaly detection unit 911 is an autoencoder that has learned the characteristics of the inverse-phase noise output when removing Gaussian noise from an image.

[0098] In S1004, the first anomaly detection unit 911 calculates the first degree of anomaly using the inverted-phase noise image output by the first image processing unit 910 and the inverted-phase noise image restored by the first anomaly detection unit 911 itself. Furthermore, the first anomaly detection unit 911 calculates the average value of the first degree of anomaly (hereinafter also referred to as the first average degree of anomaly) and outputs information including the first average degree of anomaly (an example of the first anomaly detection result) to the model selection unit 914. The first anomaly detection unit 911 may calculate the first average degree of anomaly using the degree of anomaly shown in formula (1) as the first degree of anomaly, and calculate the first average degree of anomaly for the entire element. The entire element may be all pixels of the image. The first anomaly detection unit 911 may calculate the arithmetic mean calculated from the degree of anomaly E of all pixels of the image as the first average degree of anomaly. The first anomaly detection unit 911 may calculate the average value of the degree of anomaly using the same calculation method as the restoration error described above.

[0099] In S1005, the second image processing unit 912 performs noise reduction image processing on the image acquired in S1001. The second image processing unit 912 outputs the inverse-phase noise image generated during the noise reduction image processing process to the second anomaly detection unit 913.

[0100] The second image processing unit 912 performs noise reduction using a pre-trained machine learning model, similar to the first image processing unit 910. However, the machine learning model of the second image processing unit 912 may be a pre-trained model that differs from the machine learning model of the first image processing unit 910 in at least one of its training data, training method, and model structure. For example, the machine learning model used in the second image processing unit 912 may be a model trained to remove noise from images containing Poisson noise that follows a Poisson distribution.

[0101] In S1006, the second anomaly detection unit 913 takes the inverted noise image acquired in S1005 as input, compresses the inverted noise image into features, and then performs compression and restoration processing to restore the features to the original inverted noise image. The second anomaly detection unit 913 performs this processing using a machine learning-based model.

[0102] The second anomaly detection unit 913 compresses the input data into features using an autoencoder, similar to the first anomaly detection unit 911, and then restores the compressed features to the input data. However, the autoencoder used in the second anomaly detection unit 913 may be a model that has been trained to restore the original input by taking the output of a machine learning model capable of performing high-resolution image processing used in the second image processing unit 912 as input. Therefore, the autoencoder used in the second anomaly detection unit 913 is an autoencoder that has learned the characteristics of the inverse-phase noise output when removing Poisson noise from an image.

[0103] In S1007, the second anomaly detection unit 913 calculates a second average anomaly degree (hereinafter also referred to as the second average anomaly degree) using the inverted-phase noise image output by the second image processing unit 912 and the inverted-phase noise image restored by the second anomaly detection unit 913 itself. The second anomaly detection unit 913 outputs information including the calculated second average anomaly degree (an example of the second anomaly detection result) to the model selection unit 914. The second anomaly detection unit 913 may calculate the second average anomaly degree in the same way as the first average anomaly degree.

[0104] In S1008, the model selection unit 914 selects the optimal machine learning model for high-resolution image processing. For example, the model selection unit 914 compares the first average anomaly degree and the second average anomaly degree to determine their relative magnitudes. The model selection unit 914 selects the image processing unit that performed the image processing with the smallest average anomaly degree among the multiple average anomaly degrees as the optimal image processing unit for processing the input image. In other words, the model selection unit 914 selects either the machine learning model of the first image processing unit 910 or the second image processing unit 912 as the optimal machine learning model for the input image.

[0105] As explained in the first embodiment, the smaller the degree of anomaly, the more the machine learning model can obtain image processing results that are similar in properties to the image processing results obtained during training. If the image processing results are similar in properties to the image processing results obtained during training, it means that the machine learning model's image processing results are likely to be good. On the other hand, if the image processing results are different in properties from the image processing results obtained during training, it means that the machine learning model's image processing results are less likely to be good. Therefore, an image processing unit that outputs image processing results with a small degree of anomaly has higher reliability in image processing. In other words, the image processing unit with a small average degree of anomaly selected by the model selection unit 914 is most likely to be able to perform the best possible image processing on the input image.

[0106] In S1009, the model selection unit 914 changes the machine learning model of the image processing unit to the optimal machine learning model for high-quality image processing. Specifically, the model selection unit 914 sets the optimal image processing unit selected in S1008 as the image processing unit to be used by the information processing device 900.

[0107] Through the above process, the information processing device 900 of the second embodiment can select a machine learning model that performs appropriate image processing on images acquired in the operating environment and then perform the image processing.

[0108] In S1010, the notification unit 114 proposes or notifies the user of the information processing device 900 of information regarding the optimal machine learning model for the image processing unit that has been selected and set by the model selection unit 914.

[0109] Through the above process, the information processing device 900 proposes to the user a machine learning model that performs optimal high-resolution image processing on images acquired in the operating environment. Alternatively, if the information processing device 900 is using an image processing unit that is not suitable for the input image, it notifies the user that the image processing unit is not suitable for the operating environment and that the operation should be reviewed.

[0110] As described above, the second embodiment can detect the optimal machine learning model for image processing among multiple machine learning models based on the average value of the degree of anomaly. As a result, the second embodiment can suggest the optimal machine learning model to the user, allowing the user to select the best machine learning model.

[0111] <Third Embodiment> In the third embodiment, we describe a process for determining the cause of an anomaly, such as whether the detected anomaly is caused by the input image data or by an invalid state of the machine learning model. We also describe an application example in which the third embodiment initializes the state of the model when the anomaly is caused by an invalid state of the machine learning model.

[0112] Figure 11 is a block diagram showing an example of the functional configuration of the information processing device 1100 according to the third embodiment.

[0113] The information processing device 1100 includes an image acquisition unit 110, an image processing unit 111, an anomaly detection unit 112, a notification unit 114, an input image anomaly detection unit 1110, an anomaly type determination unit 1111, and a state initialization unit 1112. The processor, including the CPU 101, may implement all or part of the functions of the image acquisition unit 110, image processing unit 111, anomaly detection unit 112, notification unit 114, input image anomaly detection unit 1110, anomaly type determination unit 1111, and state initialization unit 1112 by reading and executing a program stored in the storage unit 104.

[0114] The input image anomaly detection unit 1110 determines whether the image input to the image processing unit 111 is an anomaly as input to the image processing unit 111, and generates and outputs the determination result.

[0115] The anomaly type determination unit 1111 uses the detection result of the anomaly detection unit 112 and the determination result of the input image anomaly detection unit 1110 to determine the type of anomaly in the image output by the image processing unit 111. The types of anomalies include, for example, anomalies caused by the machine learning model of the image processing unit 111 and anomalies caused by the input image.

[0116] The state initialization unit 1112 initializes the internal state of the model when an abnormality occurs due to the internal state of the model.

[0117] The other functional units have the same functions as the information processing device 100 described in the first embodiment, so they are given the same reference numerals and their descriptions are omitted.

[0118] The third embodiment will be described in detail with reference to the example of the functional configuration in Figure 11 and the flowchart in Figure 12. Figure 12 shows the abnormality type determination processing flow of the information processing device of the third embodiment. In the third embodiment, as in the first embodiment, noise reduction processing will be described as an example of high-quality image processing.

[0119] In S1201, the image acquisition unit 110 acquires the image captured by the imaging device 120. The image acquisition unit 110 outputs the acquired image to the image processing unit 111 and the input image anomaly detection unit 1110.

[0120] The processing flow from S302 to S305 is the same as the processing flow described in the first embodiment, so its explanation is omitted. However, the image processing unit 111 performs high-resolution image processing using a neural network that has a structure that maintains a state within the machine learning model. In addition, the anomaly detection unit 112 outputs the detected anomaly location to the anomaly type determination unit 1111.

[0121] Figure 13 shows an overview of the inference process of the neural network used by the image processing unit 111 of the third embodiment to perform noise reduction. This neural network is a recurrent neural network (RNN) that maintains its internal state and uses the internal state as input, and is used in a system that performs image processing continuously over time. Here, let s be the time (or timing) at which the image processing is performed, and assume that the image processing is performed at s=0,1...

[0122] The neural network of the third embodiment includes a first stage 1310, an internal state generation module 1312, and a second stage 1311. The internal state generation module 1312 generates intermediate features from the image features output by the first stage, which become the input for the second stage of image processing at the next time step. The second stage 1311 takes the image features output by the first stage 1310 and the intermediate features generated by the internal state generation module 1312 as input, and outputs an image that has undergone image processing on the input image of the first stage 1310.

[0123] We will explain this using the image processing performed at the initial time s=0 and the next time s=1 as examples.

[0124] First, in the image processing at time s=0, the first stage 1310 obtains image 1301 at time t=0 as input to the neural network. The first stage 1310 outputs image feature vector 1302 corresponding to the input image. Next, the second stage 1311 is given image feature vector 1302 and a dummy intermediate feature vector 1303. Normally, the second stage 1311 would obtain the internal state feature vector generated by the internal state generation module 1312 in the image processing at the previous time as input. However, in the processing at the start time s=0, there are no internal state feature vectors generated in the processing at the previous time. Therefore, at the start time s=0, the second stage 1311 obtains the dummy intermediate feature vector 1303 as input.

[0125] The dummy intermediate feature 1303 is also provided as input when training the neural network, and training is performed. Therefore, the dummy intermediate feature 1303 during inference may be the dummy feature defined during training. The dummy intermediate feature may be, for example, an intermediate feature with all values ​​of 0, or an intermediate feature that replicates the intermediate feature output by the first stage 1310. The second stage 1311 takes the image feature 1302 and the dummy intermediate feature 1303 as input and applies image processing to the image 1301 corresponding to the image feature 1302. As a result, the second stage 1311 outputs the denoised image 1304. The internal state generation module 1312 takes the image feature 1302 output from the first stage 1310 as input and generates the internal state feature 1307 to be used in the next image processing at time s=1.

[0126] Next, at time s=1, the first stage 1310 takes image 1305 at time t=1 as input. The first stage 1310 outputs image feature vector 1306 as a result. Next, the second stage 1311 takes image feature vector 1306 and internal state feature vector 1307, which was generated by the internal state generation module 1312 in the previous processing at time s=0, as input. The internal state feature vector 1307 used here is a feature vector generated from image 1301 at time t=0. Image feature vector 1306 is a feature vector generated from image 1305 at time t=1. Therefore, the internal state feature vector 1307 and the image feature vector 1306 are time-sequential. The second stage 1311 takes these feature vectors as input and applies image processing to image 1305, which corresponds to image feature vector 1306. The second stage 1311 outputs a denoised image 1308 as a result.

[0127] Figure 14 shows the structure of the neural network in the third embodiment. Figure 14A shows the structure of the neural network in the first stage 1310. Figure 14B shows the structure of the neural network in the internal state generation module 1312. Figure 14C shows the structure of the neural network in the second stage 1311.

[0128] The layer structure of the first stage 1310 shown in Figure 14A and the second stage 1311 shown in Figure 14C may be a UNet with the same configuration as the neural network in Figure 4 described in the first embodiment. However, the first stage 1310 outputs image features 1402 when given an input image 1401. On the other hand, the second stage 1311 takes as input a feature obtained by combining the image features 1402, which is the output of the first stage 1310, and the internal state features 1404 output by the internal state generation module 1312 in the channel direction, and outputs an out-of-phase noise image 1405. However, the internal state features 1404 output by the internal state generation module 1312 and the image features 1402 are generated from temporally consecutive images, respectively.

[0129] The internal state generation module 1312, shown in Figure 14B, generates internal state features 1404 by performing multiple convolution operations and processing with the ReLU activation function on the intermediate features 1403, which is the output of the first stage 1310. In the second stage 1311, the denoised image 1406 is obtained by adding the inverse-phase noise image 1405, which is the final output of the internal state generation module 1312, with the image features 1402.

[0130] In S1202, the input image anomaly detection unit 1110 uses the image acquired in S1201 as the input image, compresses the input image into features, and then performs compression and restoration processing to restore the features to the original input image. The input image anomaly detection unit 1110 performs the compression and restoration processing using a machine learning-based model.

[0131] The input image anomaly detection unit 1110 may be implemented using an autoencoder. This autoencoder is trained using the input images used to train the machine learning model that performs high-quality image processing, which is provided by the image processing unit 111. The autoencoder is trained to compress the input noisy image into features, and then reconstruct the original noisy image from the features.

[0132] The input image anomaly detection unit 1110, as in the first embodiment, uses an autoencoder to compress the input image into features and then restores the compressed features to the original input image. However, the autoencoder used in the input image anomaly detection unit 1110 is trained to restore the original input image using the input image of the machine learning model that performs high-quality image processing used in the image processing unit 111 as input. In other words, the autoencoder compresses the image that becomes the input to the image processing unit 111 into features, and then performs the process of restoring the original image from the features.

[0133] Such an autoencoder can be realized by training it in the same way using the learning device 130 described in the first embodiment. However, the input for the autoencoder to learn and the true value for calculating the degree of anomaly (error) are not the out-of-phase noise image output by the machine learning model of the image processing unit 111, but the input image used when training the machine learning model. In other words, the autoencoder takes the noisy image used to train the image processing unit 111, which is stored in the database unit 140, as input and learns to restore it to a noisy image again.

[0134] An autoencoder learns using the input images used to train a machine learning model, and therefore learns the feature representations of the input images used during the model's training. This autoencoder can reconstruct an image well if the input image closely resembles the properties of the training images. On the other hand, if the input image differs significantly from the training images, it cannot reconstruct the image properly. In other words, this autoencoder can be used to determine whether an input image is suitable for the machine learning model when performing image processing on an image input to the aforementioned machine learning model.

[0135] In S1203, the input image anomaly detection unit 1110 calculates the degree of anomaly in the input image using the input image acquired from the image acquisition unit 110 and the image reconstructed by the input image anomaly detection unit 1110 itself. The input image anomaly detection unit 1110 may calculate the degree of anomaly using the input image of the autoencoder and the reconstructed image using the same calculation method as in formula (1).

[0136] In S1204, the input image anomaly detection unit 1110 detects anomalies in the input image based on the degree of anomaly calculated in S1203. The input image anomaly detection unit 1110 detects anomalies in the input image in the same way as in S305 and outputs them to the anomaly type determination unit 1111.

[0137] The degree of abnormality calculated by the input image anomaly detection unit 1110 is synonymous with the degree of suitability, which indicates whether the image is suitable as input for a machine learning model that performs high-resolution image processing. If the input image anomaly detection unit 1110 determines that the input image is abnormal, it means that it is not suitable as input for a machine learning model that performs high-resolution image processing.

[0138] In S1205, the abnormality type determination unit 1111 determines the type of abnormality using the abnormal location obtained from the abnormality detection unit 112 and the abnormal location obtained from the input image abnormality detection unit 1110. Here, the abnormality type determination unit 1111 determines whether the abnormality is caused by the machine learning model that performs image processing or by the image input to the machine learning model.

[0139] The anomaly type determination unit 1111 may determine the type of anomaly as follows: The anomaly type determination unit 1111 may determine that anomalies in locations where the input image anomaly detection unit 1110 has detected an anomaly, and where the anomaly detection unit 112 has determined that there is no anomaly, are anomalies caused by the model. On the other hand, the anomaly type determination unit 1111 may determine that anomalies in locations where the input image anomaly detection unit 1110 has detected an anomaly, and where the anomaly detection unit 112 has also detected an anomaly, are anomalies caused by the input image.

[0140] Here, there are two types of causes for anomalies that occur in the output of a machine learning model. One is anomalies that originate from the input image, and the other is anomalies that originate from the machine learning model.

[0141] Anomalies caused by input images refer to situations where, for example, an image containing noise with different properties than the noise learned by the machine learning model is input, preventing the machine learning model from obtaining the correct output after performing image processing.

[0142] Anomalies originating from machine learning models refer to situations where, for example, an invalid state occurs within the machine learning model, preventing it from obtaining the correct output when performing image processing. An invalid state can arise, for instance, when using an RNN (Resonant Network Nucleotide) that maintains internal state, due to a state not anticipated as an internal state. Consider an RNN that assumes a time-series continuous image as input and maintains time-series features continuous with the input image as its internal state. In this assumption, if a time-series discontinuous image is input, an inconsistency occurs with the RNN's internal state, preventing it from processing correctly. Therefore, the correct image processing result may not be obtained. Other invalid states include quantization errors that occur when quantizing or integerizing the parameters of a trained machine learning model, and errors caused by model transformations such as limitations on the parameter value range, which can ultimately lead to abnormal output.

[0143] The anomaly detected by the anomaly detection unit 112 is determined based on the output of the machine learning model, so it is unclear whether the anomaly is caused by the model or by the input image. Therefore, the anomaly type determination unit 1111 uses the result of the input image anomaly detection unit 1110 to determine the cause of the anomaly. If the input image anomaly detection unit 1110 detects an anomaly in the input image, and the anomaly detection unit 112 does not detect an anomaly in the image processing result, the anomaly type determination unit 1111 can determine that the anomaly is caused by the machine learning model that performs high-resolution image processing. On the other hand, if the input image anomaly detection unit 1110 detects an anomaly in the input image, and the anomaly detection unit 112 also detects an anomaly in the image processing result, the anomaly type determination unit 1111 can determine that the anomaly is caused by the input image.

[0144] In S1206, if the notification unit 114 detects an anomaly in the model due to the input image, it notifies the user of the information processing device 1100 of the determination result that the input image is an anomaly that does not conform to the learned model. The notification unit 114 may notify the user by means of an email, a message displayed on a monitor or other screen, or other notification method.

[0145] In S1207, the state initialization unit 1112 initializes the internal state of the machine learning model used by the image processing unit 111 when an anomaly caused by the machine learning model is detected.

[0146] The state initialization unit 1112 initializes the internal state of the machine learning model by setting it to a state that the internal state of the machine learning model can take. In the neural network of this embodiment, it was stated that in the processing at the first time step when no time-series consecutive images are provided, image processing is performed using dummy intermediate features with all values ​​of 0. Therefore, when intermediate features with all values ​​of 0 are provided, the inconsistency between the time-series consecutive input and the internal state can be reset.

[0147] As described above, the third embodiment can determine whether an anomaly occurring in the output of the machine learning model is caused by the input image fed into the machine learning model or by the internal state of the machine learning model itself. As a result, the third embodiment can notify the user of the type of anomaly, so that the user can know not only whether there is an anomaly, but also the type of anomaly.

[0148] Furthermore, in the third embodiment, in a machine learning model that maintains its internal state, an incorrect internal state can be initialized to allow the machine learning model to correctly perform image processing.

[0149] (Other examples) In the embodiments described above, a configuration in which the machine learning model outputs an image such as an inverted-phase noise image was used as an example. However, the embodiments described above may also be applied to configurations in which the machine learning model outputs other information such as features.

[0150] The embodiments described above may be combined as appropriate. When embodiments are combined, the system may be configured so that the user can select the processing method of any of the embodiments.

[0151] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0152] The disclosures herein include the following information processing devices, information processing methods, and programs. (Item 1) A first image processing means having a first machine learning model that performs first image processing to improve the image quality of an input image, A first anomaly detection means that generates a first anomaly detection result as information about anomalies included in the first output result of the first machine learning model, based on a first output result that includes at least one of an image and features output by the first machine learning model, An information processing device characterized by comprising: (Item 2) The first anomaly detection means includes a second machine learning model that compresses the first output result of the first machine learning model into features and reconstructs the second output result from the compressed features. The information processing device described in item 1, characterized by the features described herein. (Item 3) The first anomaly detection means calculates a first anomaly degree, which indicates the degree of anomaly in the first output result of the first machine learning model, based on the first output result and the second output result. The information processing device described in item 2, characterized by the features described herein. (Item 4) The first anomaly detection means detects the anomaly based on the statistical value of the first anomaly degree calculated during the training of the second machine learning model. The information processing device described in item 3, characterized by the features described herein. (Item 5) The information processing device according to any one of items 1 to 4, characterized in that it includes a display means for displaying an anomaly detected by the first anomaly detection means to the user. (Item 6) The information processing device according to any one of items 1 to 5, characterized in that when the first anomaly detection means detects an anomaly, it includes a notification means for notifying the user that the anomaly has been detected. (Item 7) The information processing apparatus according to any one of items 1 to 6, characterized in that it includes a recording means for recording the presence or absence of an abnormality and the first output result in association with the abnormality detection result of the first abnormality detection means. (Item 8) The first image processing of the first machine learning model is noise reduction, The first anomaly detection means acquires the out-of-phase noise image output by the first machine learning model as the first output result. An information processing device according to any one of items 1 to 7, characterized by the features described in item 1 to 7. (Item 9) The system includes a learning means for training the aforementioned second machine learning model, The learning means takes the first output result as input and uses the second output result to update the parameters of the second machine learning model through learning. The information processing device described in item 2, characterized by the features described herein. (Item 10) A second image processing means having a third machine learning model that performs a second image processing of the same type as the first image processing, A second anomaly detection means generates a second anomaly detection result as information about anomalies included in the third output result of the third machine learning model, based on a third output result that includes at least one of an image and features output by the third machine learning model. A model selection means that compares the first anomaly detection result with the second anomaly detection result and performs a selection process to select either the first image processing means or the second image processing means, An information processing device according to any one of items 1 to 9, characterized by comprising: (Item 11) The first anomaly detection means is The machine learning model comprises a second machine learning model that compresses the first output result of the first machine learning model into features and reconstructs the second output result from the compressed features. Based on the first output result and the second output result, the first anomaly detection result is generated, which includes the average value of the first anomaly degree indicating the degree of anomaly in the first output result of the first machine learning model. The second anomaly detection means is The machine learning model comprises a fourth machine learning model that compresses the third output result of the third machine learning model into features and reconstructs the fourth output result from the compressed features, Based on the third output result and the fourth output result, a second anomaly detection result is generated, which includes the average value of the second anomaly degree indicating the degree of anomaly in the second output result of the second machine learning model. The model selection means performs the selection process based on the average value of the first degree of abnormality and the average value of the second degree of abnormality. The information processing device according to item 10, characterized in that it is a processing device. (Item 12) The information processing apparatus according to item 10 or item 11, characterized in that it uses the image processing means selected by the model selection means for image processing. (Item 13) The information processing apparatus according to any one of items 10 to 12, characterized by comprising a notification means for notifying information regarding the selected image processing means. (Item 14) An input image anomaly detection means determines whether the first input image, which is an image input to the first image processing means, is abnormal or not, and generates a determination result. An abnormality type determination means for determining the type of abnormality included in the first output result based on the first abnormality detection result and the determination result, An information processing device according to any one of items 1 to 13, characterized by comprising: (Item 15) The aforementioned input image anomaly detection means is The machine learning model comprises a fifth machine learning model that compresses the first input image into features and reconstructs the second input image from the compressed features. Based on the first input image and the second input image, a third anomaly degree indicating the degree of anomaly in the first input image is calculated. Based on the third degree of abnormality, it is determined whether the first input image is suitable or not. The information processing device described in item 14, characterized by the features described herein. (Item 16) The abnormality type determination means is The input image anomaly detection means determines that the first input image is not suitable for input, and, If the first anomaly detection means determines that there is an anomaly in the first output result, The abnormality in the first output result is determined to have been caused by the first input image. An information processing device as described in item 14 or item 15, characterized by the above. (Item 17) The abnormality type determination means is The input image anomaly detection means determines that the first input image is suitable for input, and, If the first anomaly detection means determines that there is an anomaly in the first output result, The abnormality in the first output result is determined to have been caused by the first machine learning model. An information processing device according to any one of items 14 to 16, characterized in that it is an information processing device. (Item 18) The aforementioned first machine learning model is a structure that maintains an internal state, and is a model that has been trained to maintain a dummy internal state during training. If it is determined that the anomaly included in the first output result is an anomaly caused by the first machine learning model, the system includes a state initialization means to set the internal state to the dummy internal state. The information processing device described in item 17, characterized by the features described herein. (Item 19) The first anomaly detection means detects anomalies by comparing the first anomaly degree calculated for each pixel with a predetermined degree threshold. The information processing device described in item 3, characterized by the features described herein. (Item 20) A first image processing step having a first machine learning model that performs first image processing to improve the image quality of the input image, A first anomaly detection step, which generates a first anomaly detection result as information about anomalies included in the first output result of the first machine learning model, based on a first output result that includes at least one of an image and features output by the first machine learning model, An information processing method having (Item 21) A program to cause a computer to function as one of the information processing devices described in any one of items 1 through 19.

[0153] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0154] 100, 900, 1100... Information processing unit, 110... Image acquisition unit, 111, 910... Image processing unit, 112, 911... Anomaly detection unit, 113... Display unit, 114... Notification unit, 115... Recording unit, 130... Learning device, 141... Learning unit, 805... Autoencoder, 910... First image processing unit, 911... First anomaly detection unit, 912... Second image processing unit, 913... Second anomaly detection unit, 914... Model selection unit, 1110... Input image anomaly detection unit, 1111... Anomaly type discrimination unit, 1112... State initialization unit.

Claims

1. A first image processing means having a first machine learning model that performs first image processing to improve the image quality of an input image, A first anomaly detection means that generates a first anomaly detection result as information about anomalies included in the first output result of the first machine learning model, based on a first output result that includes at least one of an image and features output by the first machine learning model, An information processing device characterized by comprising:

2. The first anomaly detection means includes a second machine learning model that compresses the first output result of the first machine learning model into features and reconstructs the second output result from the compressed features. The information processing apparatus according to feature 1.

3. The first anomaly detection means calculates a first anomaly degree, which indicates the degree of anomaly in the first output result of the first machine learning model, based on the first output result and the second output result. The information processing apparatus according to feature 2.

4. The first anomaly detection means detects the anomaly based on the statistical value of the first anomaly degree calculated during the training of the second machine learning model. The information processing apparatus according to claim 3.

5. The information processing apparatus according to claim 1, further comprising a display means for displaying an anomaly detected by the first anomaly detection means to the user.

6. The information processing apparatus according to claim 1, characterized in that when the first anomaly detection means detects an anomaly, it includes a notification means for notifying the user that the anomaly has been detected.

7. The information processing apparatus according to claim 1, further comprising a recording means for recording the presence or absence of an abnormality and the first output result in association with the abnormality detection result of the first abnormality detection means.

8. The first image processing of the first machine learning model is noise reduction, The first anomaly detection means acquires the out-of-phase noise image output by the first machine learning model as the first output result. The information processing apparatus according to feature 1.

9. The system includes a learning means for training the aforementioned second machine learning model, The learning means takes the first output result as input and uses the second output result to update the parameters of the second machine learning model through learning. The information processing apparatus according to feature 2.

10. A second image processing means having a third machine learning model that performs a second image processing of the same type as the first image processing, A second anomaly detection means generates a second anomaly detection result as information about anomalies included in the third output result of the third machine learning model, based on a third output result that includes at least one of an image and features output by the third machine learning model. A model selection means that compares the first anomaly detection result with the second anomaly detection result and performs a selection process to select either the first image processing means or the second image processing means, The information processing apparatus according to claim 1, characterized by comprising:

11. The first anomaly detection means is The machine learning model comprises a second machine learning model that compresses the first output result of the first machine learning model into features and reconstructs the second output result from the compressed features. Based on the first output result and the second output result, the first anomaly detection result is generated, which includes the average value of the first anomaly degree indicating the degree of anomaly in the first output result of the first machine learning model. The second anomaly detection means is The machine learning model comprises a fourth machine learning model that compresses the third output result of the third machine learning model into features and reconstructs the fourth output result from the compressed features, Based on the third output result and the fourth output result, a second anomaly detection result is generated, which includes the average value of the second anomaly degree indicating the degree of anomaly in the second output result of the second machine learning model. The model selection means executes the selection process based on the average value of the first degree of abnormality and the average value of the second degree of abnormality. The information processing apparatus according to feature 10.

12. The information processing apparatus according to claim 10, characterized in that the image processing means selected by the model selection means is used for image processing.

13. The information processing apparatus according to claim 10, further comprising a notification means for notifying information regarding the selected image processing means.

14. An input image anomaly detection means determines whether the first input image, which is an image input to the first image processing means, is abnormal or not, and generates a determination result. An abnormality type determination means for determining the type of abnormality included in the first output result based on the first abnormality detection result and the determination result, The information processing apparatus according to claim 1, characterized by comprising:

15. The aforementioned input image anomaly detection means is The machine learning model comprises a fifth machine learning model that compresses the first input image into features and reconstructs the second input image from the compressed features. Based on the first input image and the second input image, a third anomaly degree indicating the degree of anomaly in the first input image is calculated. Based on the third degree of abnormality, it is determined whether the first input image is suitable or not. The information processing apparatus according to feature 14.

16. The aforementioned abnormality type determination means is The input image anomaly detection means determines that the first input image is not suitable for input, and, If the first anomaly detection means determines that there is an anomaly in the first output result, The abnormality in the first output result is determined to have been caused by the first input image. The information processing apparatus according to feature 14.

17. The aforementioned abnormality type determination means is The input image anomaly detection means determines that the first input image is suitable for input, and, If the first anomaly detection means determines that there is an anomaly in the first output result, The abnormality in the first output result is determined to have been caused by the first machine learning model. The information processing apparatus according to feature 14.

18. The aforementioned first machine learning model is a structure that maintains an internal state, and is a model that has been trained to maintain a dummy internal state during training. If it is determined that the anomaly included in the first output result is an anomaly caused by the first machine learning model, the system includes a state initialization means to set the internal state to the dummy internal state. The information processing apparatus according to feature 17.

19. The first anomaly detection means detects anomalies by comparing the first anomaly degree calculated for each pixel with a predetermined degree threshold. The information processing apparatus according to claim 3.

20. A first image processing step having a first machine learning model that performs first image processing to improve the image quality of the input image, A first anomaly detection step, which generates a first anomaly detection result as information about anomalies included in the first output result of the first machine learning model, based on a first output result that includes at least one of an image and features output by the first machine learning model, An information processing method having

21. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 19.

Images