Information processing device and its control method, learning device

Abstract

Generate a dataset for use in machine learning. [Solution] The information processing device includes: acquisition means for acquiring captured images obtained by an imaging device capable of controlling the field of view; generation means for generating image pairs including a noisy image containing noise and a clean image in which the noise in the image has been reduced, based on a plurality of captured images taken at the same field of view; control means for controlling the generation of a plurality of image pairs corresponding to a plurality of different fields of view; and assignment means for assigning a label to each of the plurality of image pairs that can uniquely identify each image pair.

Description

Technical Field

[0001] The present invention relates to a technique for generating a dataset used in machine learning.

Background Art

[0002] In recent years, in image processing techniques for improving the image quality of images and videos, methods using neural networks (NNs) have been actively developed. For example, in noise removal for removing noise included in an image and generating a clean image without noise, many methods using NNs have been proposed. In such an NN, learning is performed using a pair of a noisy image (an image including noise) and a clean image (an image without noise) corresponding to the noisy image. For example, a noisy image is given as an input, and the NN is learned so that the output approaches the clean image.

[0003] In the learning of an NN, it is easier to obtain a high-quality noise removal model by using images taken in an actual operation environment. Patent Document 1 discloses a system that separates a photographed image obtained by actual shooting into a foreground image and a background image, stores them in a database, and generates a composite image by combining the foreground image and the background image. Non-Patent Document 1 discloses a method of generating a clean image by averaging a large number of noisy images photographed by actual shooting, and using the pair of the noisy image and the clean image as learning data.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

[0006] In surveillance cameras, the subjects being monitored are generally moving objects. Therefore, when training a neural network for noise reduction, it is desirable to use real-world images that also capture moving objects. However, Patent Document 1 generates a composite image by processing a foreground image and pasting it onto a background image, and this does not necessarily result in an image that can be considered a real-world image. Furthermore, the method in Non-Patent Document 1 assumes that the subject is stationary in the numerous noisy images that undergo averaging in order to generate a suitable clean image, making it difficult to apply to moving objects.

[0007] This invention has been made in view of these problems and aims to provide a technology for generating datasets used in machine learning. [Means for solving the problem]

[0008] To solve the above-mentioned problems, the information processing apparatus according to the present invention has the following configuration. That is, the information processing apparatus is An acquisition means for acquiring an image captured by an imaging device capable of controlling the field of view, A generation means that generates an image pair including a noisy image containing noise and a clean image with reduced noise, based on multiple captured images taken at the same angle of view, A control means for controlling the generation means to generate multiple image pairs corresponding to multiple different field angles, A means for assigning a label to each of the plurality of image pairs that can uniquely identify each image pair, It is equipped with. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a technique for generating a dataset used for machine learning.

Brief Description of the Drawings

[0010] [Figure 1] It is a diagram showing the hardware configuration of an information processing apparatus. [Figure 2] It is a diagram showing the functional configuration of the system (First Embodiment). [Figure 3] It is a flowchart of data generation processing (First Embodiment). [Figure 4] It is a diagram showing an outline of data generation processing. [Figure 5] It is a diagram showing the functional configuration of the system (Second Embodiment). [Figure 6] It is a flowchart of data generation processing (Second Embodiment). [Figure 7] It is a diagram for explaining the mask creation process. [Figure 8] It is a diagram showing the functional configuration of the system (Third Embodiment). [Figure 9] It is a flowchart of data generation processing (Third Embodiment). [Figure 10] It is a diagram for explaining the generation process of a noisy image by applying image processing. [Figure 11] It is a diagram for explaining the addition of motion blur. [Figure 12] It is a diagram showing the functional configuration of the system (Fourth Embodiment). [Figure 13] It is a flowchart of additional learning (Fourth Embodiment). [Figure 14] It is a diagram for explaining the operation of a neural network.

Modes for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] (First Embodiment) As a first embodiment of the information processing apparatus according to the present invention, an information processing apparatus that generates an image pair (noisy image and clean image) that can be used for learning a neural network that performs noise removal will be described below as an example. In particular, a method for generating a plurality of image pairs suitable for learning a moving subject (moving body) will be described.

[0013] <System Configuration> FIG. 2 is a diagram showing the functional configuration of the system in the first embodiment. The data generation system is for generating a data set that can be used for learning a noise removal model that realizes noise removal by machine learning.

[0014] The data generation system includes an information processing apparatus 100, an imaging apparatus 201, and a database 202. The imaging apparatus 200 includes an imaging unit 210 and a drive unit 211. The imaging unit 210 can capture an image using an optical system, an image sensor, and the like. The drive unit 211 can control the shooting angle of the imaging unit 210 by performing pan, tilt, and zoom operations (PTZ operations).

[0015] Here, the data generation system is assumed to acquire and process images captured in a real-world operating environment. For example, in nighttime surveillance using a surveillance camera system, images captured by the camera (corresponding to the imaging device 200) tend to be dark. Therefore, the sensor sensitivity of the camera is increased to make the captured images brighter. On the other hand, increasing the sensor sensitivity also amplifies noise, so visible noise will be included in the image. Therefore, the data generation system is assumed to acquire and process images that contain noise.

[0016] The information processing device 100 comprises an image generation unit 220 and a labeling unit 221. The image generation unit 220 acquires images captured by the imaging device 200 and generates a "noisy image" containing noise and a "clean image" of the same scene as the noisy image but with reduced noise. The labeling unit 221 assigns a uniquely identifiable label to the image pair of the noisy image and the corresponding clean image generated by the image generation unit 220. The database 202 acquires and stores the image pairs to which the information processing device 100 has assigned labels.

[0017] Figure 1 shows the hardware configuration of the information processing device 100. The information processing device 100 can be configured as a general-purpose information processing device equipped with a CPU 101, memory 102, input unit 103, storage unit 104, display unit 105, communication unit 106, etc. For example, the CPU 101 realizes the image generation unit 220 and the labeling unit 221 by executing a program stored in the storage unit 104. Note that part or all of the image generation unit 220 and / or the labeling unit 221 may be realized by hardware such as an application-specific integrated circuit (ASIC).

[0018] <Device Operation> Figure 3 is a flowchart of the data generation process in the first embodiment.

[0019] In S301, the image generation unit 220 starts the process of acquiring moving images (multiple frame images) obtained by the imaging device 201. Thereafter, the loop processing in S302 to S305 is repeatedly executed to generate a dataset (multiple image pairs) to be used for training.

[0020] In S302, the image generation unit 220 controls the field of view to the drive unit 211 to change the field of view (shooting range) of the imaging unit 210. As a result, the drive unit 211 changes the field of view of the imaging unit 210 through operations such as panning, tilting, and zooming. At this time, based on the field of view information of the imaging unit 210 (information on pan and tilt angles and / or zoom magnification) that can be obtained from the drive unit 211 in advance, the amount of movement of the drive unit 211 in changing the field of view is associated with the amount of movement on the captured image. A limit value is set in advance for the amount of movement on the captured image, and the amount of movement of the drive unit 211 is also limited in accordance with the limit value for the amount of movement on the captured image. The field of view is moved (for example, randomly) within the range of the limited amount of movement.

[0021] In S303, the image generation unit 220 controls the imaging unit 210 to continuously capture images at the angle of view changed in S302, and acquires a large number of images (noisy images 401) obtained by the capture. Here, as an example, it is assumed that 1000 images will be acquired. Note that the number of noisy images to be acquired may be configured to be changed according to the magnitude of the noise. For example, since noise generally increases as the sensor sensitivity increases, the number of noisy images to be captured may be increased in proportion to the sensor sensitivity.

[0022] In S304, the image generation unit 220 generates a single clean image 402 with reduced noise using the 1000 noisy images acquired in S303. For example, the image generation unit 221 generates a single clean image by averaging the 1000 noisy images. If the subject in the 1000 noisy images is stationary, pixels at the same position in multiple images represent the same position of the same subject. Therefore, by averaging the variation in pixel values ​​caused by noise, the original pixel values ​​without noise can be obtained.

[0023] In S305, the image generation unit 220 creates an image pair by associating one clean image 402 and one noisy image 401 generated in S304, and outputs it to the labeling unit 305. For example, the noisy image 401 used in the image pair may be randomly selected from 1000 noisy images. However, a set may also be created by associating multiple noisy images with one clean image.

[0024] In S305, the labeling unit 221 assigns labels to the image pairs created in S304 and saves the image pairs in the database 202. The assigned labels are capable of uniquely identifying each image pair created in the loop processing from S302 to S305. For example, information about the time the images were taken can be assigned as a label.

[0025] Figure 4 shows a schematic diagram of the data generation process (loop processing from S302 to S305). For example, in the first loop (t=0), the imaging unit 210 takes an image at the initial field of view. The initial field of view may be a pre-specified field of view. In the second loop (t=1), the drive unit 211 changes the field of view of the imaging unit 210 by PTZ operation, and the imaging unit 210 takes an image at the changed field of view. The field of view of the imaging unit 210 is changed and images are taken in the same manner for the third and subsequent loops.

[0026] In the explanation of S302 above, the field of view is changed within a limited range of motion, but it may also be changed without any restrictions on the amount of motion. In this case, the image generation unit 220 acquires the field of view information (pan / tilt angle information and / or zoom magnification information) of the imaging unit 210, which can be obtained from the drive unit 211. Then, when assigning labels in S305, it is preferable to assign the field of view information as a label.

[0027] As described above, according to the first embodiment, one image pair (a noisy image and a clean image) is generated based on multiple images captured by the imaging device at a certain angle of view. Then, each time an image pair is generated, the angle of view of the imaging device is controlled to change. As a result, the position of the subject in the image changes in different image pairs. Therefore, a dataset obtained by arranging multiple image pairs generated in this way in a time series can be considered as a dataset of images of moving subjects (motion objects). Therefore, by using this dataset to train a noise reduction model, it becomes possible to obtain a model that can suitably remove noise from frame images constituting moving images of motion objects.

[0028] (Second Embodiment) In the second embodiment, when generating a clean image based on multiple noisy images, a mask is also generated that indicates areas where the subject's position does not coincide in the multiple noisy images (areas where the subject is moving).

[0029] In the first embodiment described above, a single clean image was generated by averaging multiple noisy images. Ideally, the subject in each image should be in the same position (i.e., stationary). If an image contains a moving subject, simply averaging the images will not restore the correct pixel values. While aligning and averaging multiple images is an option, image alignment is generally difficult when noise is present. Therefore, mask information is generated for the regions containing moving subjects within the image. Then, when training the denoising model, the regions indicated by this mask are excluded from training.

[0030] <System Configuration> Figure 5 shows the functional configuration of the system in the second embodiment. The data generation system comprises an information processing device 500, an imaging device 201, and a database 202. The information processing device 500 includes a mask creation unit 501 that creates a mask representing the region in the captured image where a moving subject is present. The other devices and functional units have the same functions as those described in the first embodiment, so their description is omitted.

[0031] <Device Operation> Figure 6 is a flowchart of the data generation process in the second embodiment.

[0032] In S601, the mask creation unit 501 starts the process of obtaining a mask to be used for training the noise reduction model. Thereafter, the loop processing in S602 to S605 is repeatedly executed to generate the mask to be used for training.

[0033] In S602, the mask creation unit 501 controls the drive unit 211 to change the field of view (shooting range) of the imaging unit 210. As a result, the drive unit 211 changes the field of view of the imaging unit 210 through operations such as panning, tilting, and zooming. In other words, it is the same as S302 in the first embodiment. The field of view information of the imaging unit at this time is output to the labeling unit 221.

[0034] In S603, the mask creation unit 501 controls the imaging unit 210 to continuously capture images with the field of view changed by S602, and acquires a large number of images obtained through the capture. Here, as an example, we assume that 1000 images will be acquired. Furthermore, it is assumed that the processing in S603 will be performed during a time of day when the subject is bright and less likely to generate noise in the captured images (such as daytime).

[0035] In S604, the mask creation unit 501 uses the numerous images acquired in S603 to detect areas with movement in the images and creates a mask representing the areas with movement. The created mask is output to the labeling unit 221.

[0036] Figure 7 illustrates the mask creation process (S604). First, using a large number of captured images (1000 in this case) 700, the variance of pixel values ​​at each pixel position in the image is calculated. By generating an image with the calculated variance values ​​as pixel values, a variance map 701 is obtained. In this map, areas with little movement become areas with small variance values ​​702, and areas with a lot of movement (for example, the sea with strong waves) become areas with large variance values ​​703. A threshold for the variance value is set in advance, and a mask is created by setting the value to 0 for pixel positions where the variance value exceeds the threshold, and the value to 1 for pixel positions where the variance value falls below the threshold. This generates a mask 704 corresponding to the magnitude of the movement.

[0037] In S605, the labeling unit 221 assigns labels to the masks created in S604 and saves the labels in the database 202. The assigned labels are such that they can uniquely identify each mask created in the loop processing from S602 to S605. For example, a label may be assigned that represents the field of view (shooting range) of the imaging unit 210 when the image used to create the mask was captured.

[0038] By repeatedly executing the loop processing steps S602 to S605 described above, it is possible to create masks (regions with movement in the image) in images taken at various angles of view.

[0039] In S606, the information processing device 500 generates an image pair consisting of a noisy image and a clean image. That is, the process is the same as in the first embodiment (Figure 3). However, the pair generated at this time is generated from an image captured with the same field of view as the imaging unit 210 when the mask was acquired. As a result, an image pair with the same field of view as the created mask is acquired.

[0040] In S607, the labeling unit 221 assigns a label to the image pair acquired in S606. At this time, it assigns a label similar to that of a mask with the same field of view stored in the database 202. This assigns a label that can uniquely identify the image pair (noisy image and clean image) and the mask as a set. The labeled image pair is then stored in the database 202.

[0041] As explained above, according to the second embodiment, mask information is generated for regions where the subject positions do not match (regions of moving subjects) in the multiple images used to generate one image pair. The image pair and the mask are then associated and saved. When training a denoising model using the image pair, a suitable denoising model can be obtained by not training the regions of the corresponding masks.

[0042] In the second embodiment, a method for identifying the region of a moving subject was described as calculating the variance of pixel values ​​at each pixel position. However, other methods may be used to identify the region of a moving subject. For example, a pre-trained object recognition model may be used to identify the image region of a moving object, and the region encompassing that region may be used as the mask region.

[0043] (Third embodiment) In the third embodiment, a method for generating image pairs consisting of a noisy image and a clean image to which image processing has been applied will be described. Below, an example will be described in which a noisy image is generated by adding noise to a clean image by applying motion blur.

[0044] <System Configuration> Figure 8 shows the functional configuration of the system in the third embodiment. The data generation system comprises an information processing device 800, an imaging device 201, and a database 202. The information processing device 800 includes an image processing unit 801 that performs image processing on images. The other devices and functional units have the same functions as those described in the first embodiment, so their descriptions are omitted.

[0045] <Device Operation> Figure 9 is a flowchart of the data generation process in the third embodiment. Figure 10 is a diagram illustrating the process of generating noisy images by applying image processing. Here, it is assumed that multiple image pairs have been stored in the database 202 by the processing of the first embodiment.

[0046] In S901, the image processing unit 801 retrieves one image pair (noisy image 1001, clean image 1002) stored in the database 202.

[0047] In S902, the image processing unit 801 calculates the difference between the noisy image 1001 and the clean image 1002. This yields a difference image map in which only the noise contained in the noisy image is extracted. This map is designated as the noise map 1004.

[0048] In S903, the image processing unit 801 applies a given image processing to the clean image to create a clean image 1006. In this embodiment, motion blur is added as an example of image processing. Motion blur is the blurring (blurring) of a subject caused by movement in an image. In this embodiment, by simulating motion blur that occurs in a moving subject, the reproducibility of the moving subject as a moving image (multiple frame images) is improved.

[0049] Figure 11 illustrates the motion blur addition process. A kernel 1101 that generates motion blur (in this case, lateral blur addition) is prepared in advance, and a convolution operation 1102 is performed between image 1100 and kernel 1101 to generate a motion blurred image 1103 (blurred image).

[0050] Furthermore, the image processing applied to clean images is not limited to motion blur; various image processing techniques can be used. In particular, image processing techniques that are difficult to apply to noisy images can be used. For example, if motion blur is applied to a noisy image, the noise will also be blurred, and the noise will be reduced by averaging the pixel values ​​in the spatial direction, making it unsuitable as a noisy image for training. On the other hand, by applying motion blur to a clean image and adding a noise map (described later), it is possible to artificially generate a noisy image with motion blur. In addition, other image processing techniques such as optical blurring and aberration correction should not be applied to noisy images because they change the characteristics of the noise, such as its shape and color.

[0051] In S904, the image processing unit 801 generates a noisy image 1008 by adding a noise map 1004 to the clean image 1006. This process makes it possible to obtain a noisy image 1008 with image processing (in this case, motion blur) applied while maintaining the characteristics of the noise.

[0052] In S905, the labeling unit 221 assigns labels to the image pair of noisy image 1008 and clean image 1006, and saves the image pair in the database 202.

[0053] As described above, the third embodiment generates image pairs consisting of a noisy image to which image processing has been applied and a clean image. By using a dataset containing such image pairs for training, it becomes possible to obtain a model that improves image quality.

[0054] (Fourth Embodiment) In the fourth embodiment, a configuration in which a noise reduction model is further trained in a video surveillance system that incorporates a noise reduction model will be described.

[0055] <System Configuration> Figure 12 shows the functional configuration of the system in the fourth embodiment. The system comprises an information processing device 1200, an imaging device 201, and a database 202. The information processing device 1200 further comprises a noise reduction unit 1201 and a learning unit 1202. The noise reduction unit 1201 removes noise from video images (multiple frame images) acquired from the imaging device 201 using a trained noise reduction model. The learning unit 1202 performs additional training on the noise reduction model used by the noise reduction unit 1201. The information processing device 1200 may also include a display unit 1203 and an operation unit 1204. The display unit 1203 displays video images acquired from the imaging device 201 and displays a user interface (UI) that accepts operations from the user. The operation unit 1204 accepts operations from the user on the UI displayed on the display unit 1203. The other devices and functional units have the same functions as those described in the first embodiment, so their description is omitted.

[0056] Furthermore, a machine learning-based noise reduction model can be used that is based on a convolutional neural network (CNN), as described in reference A. A CNN consists of a large number of convolutional layers and activation functions. In particular, a U-shaped network called a U-Net is used as a neural network to achieve high-resolution image processing such as noise reduction and super-resolution. The network in reference A also uses a U-Net for noise reduction. In this embodiment, a structure based on the U-Net used in reference A is also used. Document A: Matias Tassano et al., "DVDNet: A Fast Network for Deep Video Denoising", arXiv:1906.11890v1, 2019

[0057] <Device Operation> Figure 13 is a flowchart of the additional learning process in the fourth embodiment.

[0058] In S1301, the operation unit 1204 receives instructions for additional learning from the user. For example, the user presses the button displayed on the display unit 1203 to start the additional learning process, which initiates the subsequent processing.

[0059] Furthermore, the system may be configured to automatically start the additional learning process without accepting user input. For example, it may be configured to start the additional learning process at a specified time.

[0060] In S1302, the information processing device 1200 generates an image pair consisting of a noisy image and a clean image. This is the same process as in the first embodiment (Figure 3). Alternatively, the image pairs may be stored in the database 202 beforehand and retrieved from there.

[0061] In S1303, the learning unit 1202 starts training the neural network. The training process repeatedly updates multiple parameters of the network, such as weights and biases.

[0062] In S1304, the learning unit 1202 retrieves multiple image pairs (noisy image and clean image) from the database 202. Here, it retrieves the number of image pairs necessary for one inference run of the denoising model. The image pairs retrieved at this time are based on the labels assigned when they were stored in the database 202. In the first embodiment, labels were assigned to the image pairs in the order of the time the images were taken. Therefore, the number of image pairs necessary for training are retrieved in the order of the time the images were taken.

[0063] Figure 14 illustrates the operation of a neural network (NN) for noise reduction. The NN takes multiple time-series consecutive images as input and outputs a denoised image of the middle time point from among the multiple input images from which noise has been removed. Here, input 1400 is used, which consists of three images (noisy images 1401, 1402, and 1403) taken at time points t=0, 1, and 2, concatenated in the channel direction. The NN is configured to output the denoised image at time t=1.

[0064] At this time, the noisy images acquired by the learning unit 1202 in S1304 are noisy images 1401, 1402, and 1403. Also, the clean image acquired by the learning unit 1202 in S1304 is the clean image (GT) corresponding to the noisy image at t=1.

[0065] In S1305, the noise reduction unit 1201 inputs the noisy image from the noisy image and clean image acquired in S1304 into the noise reduction model to obtain a noise-reduced image 1404. The obtained noise-reduced image 1404 is output to the learning unit 1202.

[0066] The inference mechanism of the neural network shown in Figure 14 will be explained. Here, we assume that U-Net is used as the neural network. U-Net consists of an encoder that generates features while compressing the image, and a decoder that reconstructs the image from the compressed features.

[0067] First, the encoder generates feature vectors with different resolutions and channel counts from the input 1400. The network applies a convolution operation and the ReLU function 1411 multiple times to the input image 1400 to generate feature vectors 1412. The resolution of the generated feature vectors 1412 is reduced by a pooling operation 1413. Then, the convolution operation and ReLU function are repeated again to obtain feature vectors with an increased channel count. The feature vectors 1412 generated at this time are used in the image reconstruction process described later, and are skip-combined 1414 with other feature vectors generated while upsampling.

[0068] By repeating a series of processes, the compressed features are deconvolved (1415), reducing the number of channels and increasing the resolution while restoring the features to the image. At this time, the upsampled features from the deconvolution process are skip-combined with the features generated by the encoder, and multiple convolution operations, the application of the ReLU function, and the deconvolution process are repeatedly executed. Finally, a denoised image (1404) with the desired resolution and number of channels is output.

[0069] As described above, the denoised image 1404 is an image from which noise has been removed from the noisy image 1402. In this embodiment, the network configured as described above is used, but the structure of the network is not limited as long as it can achieve noise removal from an image.

[0070] In S1306, the learning unit 1202 calculates the error using the clean image (GT) acquired in S1304 and the denoised image acquired in S1305. Here, the L1 error, expressed by the following formula (1), is used as the error. In formula (1), I^t is the denoised image and It is the clean image (GT).

[0071]

number

[0072] In S1307, the learning unit 1202 updates the weights of the denoising model using the error calculated in S1306 via backpropagation. The above processes S1304 to S1307 are repeated to perform additional training on the neural network that performs denoising.

[0073] As described above, according to the fourth embodiment, in the video surveillance system, the noise reduction model can be further trained using video footage from the actual operating environment. Since the model is trained using data from the actual operating environment, the noise reduction performance can be efficiently improved.

[0074] The disclosures herein include the following information processing devices, control methods, learning devices, and programs. (Item 1) An acquisition means for acquiring an image captured by an imaging device capable of controlling the field of view, A generation means that generates an image pair including a noisy image containing noise and a clean image with reduced noise, based on multiple captured images taken at the same angle of view, A control means for controlling the generation means to generate multiple image pairs corresponding to multiple different field angles, A means for assigning a label to each of the plurality of image pairs that can uniquely identify each image pair, An information processing device characterized by comprising: (Item 2) The acquisition means acquires multiple captured images for each of the multiple angles of view, The generation means generates the plurality of image pairs corresponding to each of the plurality of angles of view. The information processing device described in item 1, characterized by the features described herein. (Item 3) The imaging device further comprises angle-of-view control means for controlling the angle of view of the imaging device. An information processing device according to item 1 or 2, characterized by the above. (Item 4) The labeling means assigns to each of the plurality of image pairs information as the time when the plurality of captured images used to generate the image pair were taken, as the label. An information processing device according to any one of items 1 to 3, characterized by the above. (Item 5) The labeling means assigns to each of the plurality of image pairs information regarding the field of view of the plurality of captured images used to generate the image pair as the label. An information processing device according to any one of items 1 to 3, characterized by the above. (Item 6) The generation means generates the clean image by performing an averaging process on the plurality of captured images. An information processing device according to any one of items 1 to 5, characterized by the above. (Item 7) The system further includes a mask creation means for creating a mask that indicates areas where the position of the subject does not coincide in multiple captured images taken at the same angle of view. An information processing device according to any one of items 1 to 6, characterized by the above. (Item 8) The mask creation means creates the mask based on the distribution of pixel values ​​in the plurality of captured images at each pixel position of the plurality of captured images. The information processing device described in item 7, characterized by the features described herein. (Item 9) The labeling means further assigns the same label to each mask corresponding to each of the plurality of image pairs as to the corresponding image pair. An information processing device according to item 7 or 8, characterized by the features described therein. (Item 10) The system further comprises a second generation means for generating a second image pair, which includes a second noisy image and a second clean image, based on the noisy image and clean image included in the aforementioned image pair. An information processing device according to any one of items 1 to 9, characterized by the above. (Item 11) The second generation means generates a difference image based on the difference between the noisy image and the clean image contained in the image pair, applies a given image processing to the clean image to generate a second clean image, and adds the difference image and the second clean image to generate a second noisy image. The information processing device according to item 10, characterized in that it is a processing device. (Item 12) The aforementioned image processing is a motion blur addition process. The information processing device described in item 11, characterized by the features described herein. (Item 13) A learning device for training a noise reduction model that removes noise contained in an image, The system includes a learning means for training the noise reduction model using a plurality of image pairs generated by an information processing device described in any one of items 1 to 12. A learning device characterized by the following features. (Item 14) A method for controlling an information processing device that generates multiple image pairs used for training a denoising model to remove noise contained in images, An acquisition step in which an image is obtained using an imaging device capable of controlling the field of view, A generation process that generates image pairs including a noisy image containing noise and a clean image with reduced noise, based on multiple captured images taken at the same angle of view, A control step that controls the generation of multiple image pairs corresponding to multiple different field angles, A labeling step of assigning a label to each of the aforementioned plurality of image pairs that can uniquely identify each image pair, A control method characterized by including (Item 15) A program to cause a computer to execute the control method described in item 14.

[0075] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0076] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0077] 100 Information processing device; 201 Imaging device; 210 Imaging unit; 211 Drive unit; 220 Image generation unit; 221 Labeling unit; 202 Database

Claims

1. An acquisition means for acquiring an image captured by an imaging device capable of controlling the field of view, A generation means that generates an image pair including a noisy image containing noise and a clean image with reduced noise, based on multiple captured images taken at the same angle of view, A control means for controlling the generation means to generate multiple image pairs corresponding to multiple different field angles, A means for assigning a label to each of the plurality of image pairs that can uniquely identify each image pair, An information processing device characterized by comprising:

2. The acquisition means acquires multiple captured images for each of the multiple angles of view, The generation means generates the plurality of image pairs corresponding to each of the plurality of angles of view. The information processing apparatus according to feature 1.

3. The imaging device further comprises angle-of-view control means for controlling the angle of view of the imaging device. The information processing apparatus according to feature 1.

4. The labeling means assigns to each of the plurality of image pairs information as the time when the plurality of captured images used to generate the image pair were taken, as the label. The information processing apparatus according to feature 1.

5. The labeling means assigns to each of the plurality of image pairs information regarding the field of view of the plurality of captured images used to generate the image pair as the label. The information processing apparatus according to feature 1.

6. The generation means generates the clean image by performing an averaging process on the plurality of captured images. The information processing apparatus according to feature 1.

7. The system further includes a mask creation means for creating a mask that indicates areas where the position of the subject does not coincide in multiple captured images taken at the same angle of view. The information processing apparatus according to feature 1.

8. The mask creation means creates the mask based on the distribution of pixel values ​​in the plurality of captured images at each pixel position of the plurality of captured images. The information processing apparatus according to feature 7.

9. The labeling means further assigns the same label to each mask corresponding to each of the plurality of image pairs as to the corresponding image pair. The information processing apparatus according to feature 7.

10. The system further comprises a second generation means for generating a second image pair, which includes a second noisy image and a second clean image, based on the noisy image and clean image included in the aforementioned image pair. The information processing apparatus according to feature 1.

11. The second generation means generates a difference image based on the difference between the noisy image and the clean image contained in the image pair, applies a given image processing to the clean image to generate a second clean image, and adds the difference image and the second clean image to generate a second noisy image. The information processing apparatus according to feature 10.

12. The aforementioned image processing is a motion blur addition process. The information processing apparatus according to feature 11.

13. A learning device for training a noise reduction model that removes noise contained in an image, The system comprises a learning means for training the noise reduction model using a plurality of image pairs generated by an information processing device according to any one of claims 1 to 12. A learning device characterized by the following features.

14. A method for controlling an information processing device that generates multiple image pairs used for training a denoising model to remove noise contained in images, An acquisition step in which an image is obtained using an imaging device capable of controlling the field of view, A generation process that generates image pairs including a noisy image containing noise and a clean image with reduced noise, based on multiple captured images taken at the same angle of view, A control step that controls the generation of multiple image pairs corresponding to multiple different field angles, A labeling step of assigning a label to each of the aforementioned plurality of image pairs that can uniquely identify each image pair, A control method characterized by including

15. A program for causing a computer to execute the control method described in claim 14.

Images