Learning method for learning device, program and noise reduction device
The learning device and method generate training data by combining and superimposing noise on images from different positions and settings to effectively infer high-quality videos from low-quality videos, overcoming the challenge of data acquisition and enhancing video processing efficiency.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- MAXELL LTD
- Filing Date
- 2022-08-31
- Publication Date
- 2026-07-09
AI Technical Summary
Generating large amounts of training data for inferring high-quality videos from low-quality videos is difficult due to the challenge of obtaining high-quality and low-quality videos of the same subject with different settings, especially in the case of video processing, where conventional methods face significant data requirements and complexity.
A learning device and method that generates training data by acquiring and combining images from different positions of high-quality and low-quality images with varying settings, and superimposing noise to create high-quality videos, incorporating affine transformations and trajectory vectors, to infer high-quality videos from low-quality videos, using a learning model trained on these data sets, and generating high-quality videos from low-quality videos.
Enables the generation of teacher data for inferring high-quality videos from low-quality videos, addressing the difficulty in obtaining such data and improving the efficiency of video processing.
Smart Images
Figure 0007887320000001 
Figure 0007887320000002 
Figure 0007887320000003
Abstract
Description
Technical Field
[0001] The present invention relates to a learning device, a program, and a learning method for a noise reduction device.
Background Art
[0002] Conventionally, there has been a technique for image processing of a low-quality image into a high-quality image using machine learning. In such a technical field, a learning model is learned using a combination of a noise image with noise superimposed thereon and a high-quality image as teacher data. The creation of teacher data is performed by obtaining a high-quality image and a noise image by imaging the same object with different exposure settings using an imaging device. In general, it is known that a large amount of teacher data is required for machine learning, and there has been a problem that creating teacher data by imaging using a camera is laborious. Therefore, a technique for creating teacher data by adding random noise to a high-quality image is known (see, for example, Patent Document 1). Using such a conventional technique, it is known to create teacher data for inferring a high-quality image from a low-quality image by adding random noise to the high-quality image.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] Here, it is known that, just like with still images, a large amount of training data for machine learning is required when processing low-quality video to create high-quality video. However, in the case of video, it is extremely difficult to easily obtain high-quality and low-quality videos of the same subject by shooting the same object with different settings. Therefore, it is conceivable to generate low-quality video by superimposing noise onto each frame of a pre-shot high-quality video using the conventional techniques described above, but this is extremely difficult due to problems such as the enormous amount of data that would be required.
[0005] Therefore, the present invention aims to provide a technology capable of generating training data for inferring high-quality videos from low-quality videos. [Means for solving the problem]
[0006] (1) One aspect of the present invention is a first image information comprising at least one image, and a subject identical to the subject captured in the image included in the first image information. , with the same field of view and imaging angle as when the image included in the first image information was captured, but with at least one of the ISO sensitivity or exposure time differed. An image acquisition unit acquires second image information which includes at least one image of lower quality than the image included in the first image information that has been captured; a unit extracts multiple images from different positions that are part of the acquired first image information, combines the extracted multiple images to generate first video information, and extracts multiple images from different positions that are part of the acquired second image information. Each of the multiple images extracted from the second image information corresponds to each of the multiple images extracted from the first image information, and the positions from which the corresponding images are extracted are approximately the same. The learning device comprises a video information generation unit that generates second video information by combining multiple extracted images, and a learning unit that trains the device to infer high-resolution video from low-resolution video based on training data that includes the first video information and the second video information generated by the video information generation unit.
[0007] (2) In one aspect of the present invention, in the learning device described in (1) above, the second image information includes a plurality of images in which the same subject as the subject captured in the image included in the first image information is captured, and each of these images has different noises superimposed on it, and the video information generation unit generates the second video information by cutting out different parts from each of the plurality of images included in the second image information.
[0008] (3) In one aspect of the present invention, in the learning device described in (1) or (2) above, the plurality of images included in the second image information are images taken at different times in close proximity.
[0009] (4) In one aspect of the present invention, in a learning device described in any of (1) to (3) above, the video information generation unit generates the first video information by cutting out different parts from one image included in the first image information.
[0010] (5) In one aspect of the present invention, in the learning device described in any of (1) to (4) above, the video information generation unit extracts multiple images at different positions by shifting the extracted multiple images in a predetermined direction.
[0011] (6) In one aspect of the present invention, in the learning device described in any of (1) to (5) above, the video information generation unit extracts a plurality of images at positions moved by a predetermined number of bits in a predetermined direction.
[0012] (7) In one aspect of the present invention, in the learning device described in (6) above, the predetermined direction in which the video information generation unit extracts an image is calculated by an affine transformation.
[0013] (8) In one aspect of the present invention, the learning device described in (6) above further comprises a trajectory vector acquisition unit that acquires a trajectory vector, wherein the predetermined direction in which the video information generation unit extracts an image is calculated based on the acquired trajectory vector.
[0014] (9) One aspect of the present invention includes an image acquisition unit that acquires image information including at least one image, and an extraction unit that extracts a plurality of images from different positions which are part of the acquired image information, By the aforementioned cut portion Cut So A first video information generation unit generates first video information by combining multiple images, and multiple images extracted by the extraction unit and a plurality of images used to generate the first video information The learning device comprises a noise superposition unit that superimposes noise on each of the images, a second video information generation unit that generates second video information by combining a plurality of images on which noise has been superimposed by the noise superposition unit, and a learning unit that trains the device to infer high-quality video from low-quality video based on training data that includes the first video information generated by the first video information generation unit and the second video information generated by the second video information generation unit.
[0015] (10) One aspect of the present invention provides a computer with first image information including at least one image, and a subject identical to the subject captured in the image included in the first image information. The image included in the first image information is captured at the same angle of view and imaging angle as when the image was captured, but with at least one of the ISO sensitivity or exposure time differed. Image acquisition step of acquiring second image information which includes at least one image of lower quality than the image included in the first image information that has been captured; extracting multiple images from different positions which are part of the acquired first image information, combining the multiple extracted images to generate first video information, and extracting multiple images from different positions which are part of the acquired second image information, Each of the multiple images extracted from the second image information corresponds to each of the multiple images extracted from the first image information, and the positions from which the corresponding images are extracted are approximately the same. This program performs a video information generation step of generating second video information by combining multiple extracted images, and a learning step of training the program to infer high-resolution video from low-resolution video based on training data that includes the first video information and the second video information generated in the video information generation step.
[0016] (11) One aspect of the present invention includes first image information comprising at least one image, and an image of the same subject as the subject captured in the image included in the first image information, The image included in the first image information is captured at the same angle of view and imaging angle as when the image was captured, but with at least one of the ISO sensitivity or exposure time differed.An image acquisition step of acquiring second image information including at least one image with lower image quality than the image included in the first image information, and cutting out a plurality of images at different positions from a part of the acquired first image information, and combining the cut-out plurality of images to generate first video information, and cutting out a plurality of images at different positions from a part of the acquired second image information, Each of the multiple images extracted from the second image information corresponds to each of the multiple images extracted from the first image information, and the positions from which the corresponding images are extracted are approximately the same. A video information generation step of combining the cut-out plurality of images to generate second video information, and a learning step of learning to infer a high-quality video from a low-quality video based on the teacher data including the first video information and the second video information generated by the video information generation step. A learning method for a noise reduction device having the above steps.
Effect of the Invention
[0017] According to the present invention, teacher data for inferring a high-quality video from a low-quality video can be generated.
Brief Description of the Drawings
[0018] [Figure 1] It is a diagram for explaining the outline of a learning system according to a first embodiment. [Figure 2] It is a diagram showing an example of the functional configuration of a learning device according to a first embodiment. [Figure 3] It is a diagram for explaining an example of the position of an image cut out by the learning device according to the first embodiment from a high-quality image. [Figure 4] It is a diagram for explaining an example of the position of an image cut out by the learning device according to the first embodiment from a low-quality image. [Figure 5] It is a diagram for explaining an example of the direction cut out by the learning device according to the first embodiment. [Figure 6] It is a diagram showing an example of the functional configuration of the learning device when the learning device according to the first embodiment generates a video based on a trajectory vector. [Figure 7]It is a diagram for explaining an example of the position of an image cut out from a still image when the learning device according to the modification of the first embodiment generates a video based on a trajectory vector. [Figure 8] It is a flowchart showing an example of a series of operations of a learning method of a noise reduction device according to a modification of the first embodiment. [Figure 9] It is a diagram for explaining an overview of a learning system according to the second embodiment. [Figure 10] It is a diagram showing an example of a functional configuration of a video information generation unit according to the second embodiment.
Embodiments for Carrying Out the Invention
[0019] Hereinafter, a learning device, a program, and a learning method of a noise reduction device according to an aspect of the present invention will be described in detail with reference to the accompanying drawings by listing preferred embodiments. Note that the aspects of the present invention are not limited to these embodiments, and also include those with various changes or improvements. That is, the constituent elements described below include those that can be easily assumed by those skilled in the art and substantially identical ones, and the constituent elements described below can be combined as appropriate. Also, various omissions, substitutions, or changes of the constituent elements can be made without departing from the gist of the present invention. Also, in the following drawings, in order to make each configuration easy to understand, the scale, number, etc. in each structure may be different from those in the actual structure.
[0020] First, the prerequisites for this embodiment will be described. The learning method for the learning device, program, and noise reduction device according to this embodiment trains a learning model to infer high-quality video from low-quality video information with superimposed noise, taking it as input. Low-quality video includes low-resolution video, and high-quality video includes high-resolution video. The training data used for learning by the learning device, program, and noise reduction device learning method according to this embodiment is generated from still images of a subject. A still image of a subject may be a single high-quality image, or it may be a combination of multiple images of the same subject (one or more high-quality images and one or more low-quality images). Multiple images of the same subject may be taken under different imaging conditions. Also, an image of a subject may be any other image, including at least one image. As an example, a high-quality image can be an image with high image quality taken with low ISO sensitivity and long exposure. In the following description, a high-quality image may be referred to as GT (Ground Truth). Low-quality images can be exemplified by images with poor image quality captured using high ISO sensitivity and short exposure times.
[0021] In the following description, image quality degradation due to noise will be explained as an example of a low-quality image, but this embodiment can be broadly applied to factors other than noise that degrade image quality. Examples of factors that degrade image quality include a decrease in resolution or color shift due to optical aberrations, a decrease in resolution due to camera shake or subject blur, uneven black levels due to dark current or circuit-induced causes, ghosting or flare due to high-brightness subjects, and abnormal signal levels.
[0022] Note that pre-prepared images may be used to generate training data. In the following explanation, low-quality images may be referred to as low-resolution images or noisy images. Similarly, high-quality images may be referred to as high-resolution images or GT. Likewise, low-quality videos may be referred to as low-resolution videos or noisy videos. Similarly, high-quality videos may be referred to as high-resolution videos or GT.
[0023] The images targeted by the learning device according to this embodiment may be still images or frames included in a video. Furthermore, the data format may be a format that does not undergo compression encoding, such as the Raw format, or a format that undergoes compression encoding, such as the JPEG or MPEG format. Unless otherwise specified, the following description will focus on the case where the image is a still image in Raw format.
[0024] Furthermore, the image targeted by the learning device according to this embodiment may be an image captured by a CCD camera using a CCD (Charge Coupled Devices) image sensor. Alternatively, the image targeted by the learning device according to this embodiment may be an image captured by a CMOS camera using a CMOS (Complementary Metal Oxide Semiconductor) image sensor. Also, the image targeted by the learning device according to this embodiment may be a color image or a monochrome image. Furthermore, the image targeted by the learning device according to this embodiment may be an image captured by acquiring the invisible light component, such as by an infrared camera using an infrared sensor.
[0025] [First Embodiment] First, the first embodiment will be described with reference to Figures 1 to 8. Figure 1 is a diagram illustrating the overview of the learning system according to the first embodiment. The overview of the learning system 1 will be explained with reference to this figure. The learning system 1 shown in this figure is an example of the configuration during the learning stage of machine learning. The learning system 1 trains the learning model 40 using training data TD generated based on images captured by the imaging device 20.
[0026] The learning system 1 captures high-resolution images 31 and low-resolution images 32 by including an imaging device 20. The high-resolution image 31 and low-resolution image 32 are images of the same subject. For example, the high-resolution image 31 and low-resolution image 32 are captured with the same field of view and imaging angle, but with different settings such as ISO sensitivity and exposure time. It is preferable that there be one high-resolution image 31, but there may be multiple. Similarly, it is preferable that there be multiple low-resolution images 32, but there may be one. It is preferable that the multiple low-resolution images 32 are different images captured with different settings such as ISO sensitivity and exposure time. The imaging device 20 may be, for example, a smartphone or tablet terminal with communication means. Alternatively, the imaging device 20 may be a surveillance camera with communication means.
[0027] The learning system 1 generates a high-resolution video 33 from a high-resolution image 31 and a low-resolution video 34 from a low-resolution image 32. Preferably, the high-resolution video 33 is generated from a single high-resolution image 31, and preferably, the low-resolution video 34 is generated from multiple low-resolution images 32. The high-resolution video 33 and low-resolution video 34 generated from the high-resolution image 31 and low-resolution image 32, which capture the same subject, are associated with each other. The corresponding high-resolution video 33 and low-resolution video 34 are input to the learning model 40 as training data TD for learning.
[0028] Furthermore, the corresponding high-resolution video 33 and low-resolution video 34 may be temporarily stored in a predetermined storage device for subsequent learning. That is, the learning system 1 may generate multiple training data TDs in advance before subsequent learning. Also, the high-resolution image 31 and low-resolution image 32 captured by the imaging device 20 may be temporarily stored in a predetermined storage device. In this case, the learning system 1 may store multiple combinations of corresponding high-resolution image 31 and low-resolution image 32 and generate training data TDs during learning.
[0029] The learning model 40 is trained using the training data TD generated by the learning system 1. Specifically, the learning model 40 is trained to infer high-quality videos from low-quality videos. In other words, the trained learning model 40 takes low-quality videos as input, infers high-quality videos, and outputs the inferred results. That is, the trained learning model 40 may be used as a noise reduction device to remove noise from low-quality videos.
[0030] The high-resolution images 31 and low-resolution images 32 captured by the imaging device 20 are stored in a predetermined storage device for temporarily storing information. This predetermined storage device may be provided in the imaging device 20 or in a cloud server, etc. In other words, the learning system 1 may be configured on an edge device or may include both an edge device and a cloud server. Furthermore, the learning of the learning model 40 may also utilize a GPU or the like provided on the server.
[0031] Figure 2 is a diagram showing an example of the functional configuration of a learning device according to the first embodiment. The functional configuration of the learning device 10 will be described with reference to this figure. The learning device 10 is used to realize the learning system 1 described above. The learning device 10 generates high-resolution video 33 and low-resolution video 34 based on high-resolution image 31 and low-resolution image 32 captured by the imaging device 20. The learning device 10 trains the learning model 40 using the generated high-resolution video 33 and low-resolution video 34 as training data TD. The learning device 10 comprises an image acquisition unit 11, a video information generation unit 12, and a learning unit 13. The learning device 10 includes a CPU (Central Processing Unit), ROM (Read-only memory), or RAM (Random access memory), etc., connected by a bus. The learning device 10 functions as a device comprising the image acquisition unit 11, the video information generation unit 12, and the learning unit 13 by executing a learning program.
[0032] Furthermore, all or part of the functions of the learning device 10 may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field-Programmable Gate Array). The learning program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The learning program may also be transmitted via a telecommunications line.
[0033] The image acquisition unit 11 acquires image information I from the imaging device 20. Image information I includes first image information I1 and second image information I2. First image information I1 includes at least one high-resolution image 31. Second image information I2 includes at least one low-resolution image 32. The low-resolution image 32 included in second image information I2 captures the same subject as the high-resolution image 31 included in first image information I1. The image included in second image information I2 is of lower quality than the image included in first image information I1. The image acquisition unit 11 outputs the acquired image information I to the video information generation unit 12.
[0034] The video information generation unit 12 extracts multiple parts of an image contained in the image information I, and generates video information M by connecting the extracted images as frame images at a predetermined time interval (or frame rate). The frame rate may be, for example, 60 [FPS (frames per second)]. The position of the images extracted by the video information generation unit 12 may differ from frame to frame. For example, the size of the extracted image may be fixed, and the video information generation unit 12 may extract multiple images at positions shifted by a predetermined number of pixels (bits) in a predetermined direction. Specifically, the size of the extracted image may be fixed at 256 pixels × 256 pixels. Alternatively, the video information generation unit 12 may extract images at positions shifted by 10 pixels each frame. If the amount of shift is too large, the amount of change in the image from frame to frame will become too large, resulting in an unnatural video, so it is preferable to set a limit (upper limit) so as not to shift beyond a predetermined amount. The amount of shift and the limitations thereof are preferably determined based on the shooting angle of view, shooting resolution, focal length of the optical system, distance to the subject, shooting frame rate, etc. Furthermore, in the case of a falling subject, since the velocity increases exponentially, the amount of shift may be increased for frames that are temporally further away from the target image.
[0035] The video information generation unit 12 generates first video information M1 from images included in first image information I1 and second video information M2 from images included in second image information I2. That is, the video information generation unit 12 extracts multiple images from different positions that are part of the first image information I1 and combines the extracted images to generate first video information M1. The video information generation unit 12 also extracts multiple images from different positions that are part of the second image information I2 and combines the extracted images to generate second video information M2. Generating a video by combining multiple images may also mean converting the multiple images into a file format that displays them at predetermined time intervals according to the frame rate. The video information generation unit 12 outputs the information containing the generated first video information M1 and second video information M2 as video information M to the learning unit 13.
[0036] Here, the size and position of the multiple images extracted by the video information generation unit 12 can be arbitrarily determined. However, it is preferable that the position from which the image included in the first image information I1 is extracted and the position from which the image included in the second image information I2 is extracted be approximately the same. This is because the first video information M1, which is a high-quality video, and the second video information M2, which is a low-quality video, should be of the same subject.
[0037] The learning unit 13 acquires video information M from the video information generation unit 12. The learning unit 13 trains the learning model 40 by inputting the acquired video information M as training data TD into the learning model 40. The learning model 40 is trained to infer high-resolution video from low-resolution video. That is, the learning unit 13 trains to infer high-resolution video from low-resolution video based on training data TD which includes the first video information M1 and the second video information M2 generated by the video information generation unit 12. It can also be said that the learning model 40 is trained to infer by removing noise from the input video.
[0038] Next, with reference to Figures 3 to 5, we will describe the images that the learning device 10 extracts from the images captured by the imaging device 20. In the following description, the method for generating high-quality video from high-quality images (the method described with reference to Figure 3) and the method for generating low-quality video from low-quality images (the method described with reference to Figure 4) are described as being different from each other, but this embodiment is not limited to this example. Instead of the following description, high-quality video may be generated from high-quality images and low-quality video from low-quality images by similar methods. That is, low-quality video may be generated by the method described with reference to Figure 3, and high-quality video may be generated by the method described with reference to Figure 4.
[0039] Figure 3 is a diagram illustrating an example of the position of an image extracted from a high-quality image by the learning device according to the first embodiment. An example of the position of an image extracted from a high-quality image by the learning device 10 will be explained with reference to this figure. Figure 3(A) shows image I-11, which is an example of an image included in the first image information I1. Figure 3(B) shows image I-12, which is an example of a case where multiple images are extracted from image I-11 shown in Figure 3(A).
[0040] As shown in Figure 3(A), image I-11 captures the subject, ball B. The video information generation unit 12 extracts multiple images from image I-11 and connects these extracted images in time to generate a video from the still image I-11.
[0041] Image I-12 in Figure 3(B) shows multiple extracted images CI, which are images extracted by the video information generation unit 12. Specifically, extracted images CI-11 to CI-15 are shown as examples of images extracted by the video information generation unit 12. When not distinguishing between extracted images CI-11 to CI-15, they may simply be referred to as extracted images CI.
[0042] The cropped images CI-11 to CI-15 are each shifted by a predetermined number of pixels in the vertical and horizontal directions. According to the first video information M1 generated by the video information generation unit 12, image C-11 is displayed at a certain time t1, image C-12 is displayed at a certain time t2, image C-13 is displayed at a certain time t3, image C-14 is displayed at a certain time t4, and image C-15 is displayed at a certain time t5. In this way, by connecting different cropped images CI in time, it is possible to generate a video in which the subject ball B in the still image appears to be moving. When the video information generation unit 12 generates a video with a frame rate of 60 [fps], the interval between each time point may be 1 / 60th of a second.
[0043] The shift direction and amount of the image extracted by the video information generation unit 12 are preferably determined based on shooting conditions such as the shooting angle of view, shooting resolution, focal length of the optical system, distance to the subject, and shooting frame rate. Furthermore, in cases such as simulating a falling object, the speed increases exponentially, so it is preferable to gradually change (increase) the amount of shift.
[0044] Here, the high-quality video (first video information M1) generated by the learning device 10 is a high-quality video without superimposed noise. Therefore, it is ideal that the still images used to generate the video are free of superimposed noise. Furthermore, it is ideal that each frame of the high-quality video generated from noise-free images is also free of superimposed noise. Therefore, it is preferable for the video information generation unit 12 to generate a video from a single noise-free image. In other words, it is preferable for the video information generation unit 12 to generate the first video information M1 by extracting different parts from a single high-quality image included in the first image information I1.
[0045] Figure 4 is a diagram illustrating an example of the position of images extracted from low-quality images by the learning device according to the first embodiment. Referring to this figure, an example of the position of images extracted from low-quality images by the learning device 10 will be explained. The learning device 10 extracts images of different frames from multiple low-quality images. Figures 4(A) to 4(E) show images I-21 to I-25, which are different images. The learning device 10 extracts images of different frames from images I-21 to I-25.
[0046] The composition of the low-quality images I-21 to I-25 is the same as that of image I-11 shown in Figure 3(A). That is, ball B is captured in the same position in images I-21 to I-25. Images I-21 to I-25 differ from image I-11 in that different noises are superimposed on each other. Images I-21 to I-25 may have different noises superimposed on each other, for example, by using different imaging conditions during acquisition.
[0047] The video information generation unit 12 extracts image CI-21 from image I-21, image CI-22 from image I-22, image CI-23 from image I-23, image CI-24 from image I-24, and image CI-25 from image I-25. Each of the extracted images CI-21 to CI-25 is shifted by a predetermined number of pixels in the vertical and horizontal directions. According to the second video information M2 generated by the video information generation unit 12, image C-21 is displayed at a certain time t1, image C-22 is displayed at a certain time t2, image C-23 is displayed at a certain time t3, image C-24 is displayed at a certain time t4, and image C-25 is displayed at a certain time t5. Since each of the extracted images CI-21 through CI-25 has different noise superimposed on it, the resulting video will also have different noise superimposed on it at different points in time.
[0048] Here, the low-quality video (second video information M2) generated by the learning device 10 is a low-resolution video with superimposed noise. If multiple different positions are cut out from a single image with superimposed noise to create a video, the same noise is included at every moment (in other words, the noise does not change over time), so it may not be suitable as a low-resolution video. Therefore, in this embodiment, a low-resolution video is generated by cutting out from multiple different low-resolution images. Each of the multiple different low-resolution images captures the same subject as the subject captured in the high-resolution image. That is, the second image information M2 includes multiple images in which the same subject as the subject captured in the image included in the first image information I1 is captured, and each of these images has different noise superimposed on it. The multiple images included in the second image information I2 may be images captured at different times in close proximity. The video information generation unit 12 generates the second video information M2 by cutting out different parts from each of the multiple images included in the second image information. Furthermore, it is not necessary to prepare a separate low-resolution image for each frame; you can extract multiple images in a non-consecutive order. The order in which you extract images from multiple images can also be random.
[0049] Figure 5 is a diagram illustrating an example of the direction in which the learning device according to the first embodiment extracts an image. In the example described with reference to Figures 3 and 4, an example was described in which a position moved by a predetermined number of pixels in both the vertical and horizontal directions is extracted. However, the video information generation unit 12 may also extract a position moved in other directions. Referring to Figures 5(A) to 5(C), another example of the direction in which the video information generation unit 12 extracts the extracted image CI will be described.
[0050] Figure 5(A) shows image I-31. Figure 5(A) is an example of a case where a position is extracted only when moving in the horizontal direction. In this case, the video information generation unit 12 extracts multiple extracted images CI at different positions by fixing the y-coordinate in the vertical direction and changing only the x-coordinate in the horizontal direction. By extracting in this way, it is possible to generate a video in which the subject moves horizontally. Similarly, the video information generation unit 12 may also extract extracted images CI at positions where the subject has moved only vertically. By extracting in this way, it is possible to generate a video in which the subject moves vertically. Furthermore, as shown in Figures 3 and 4, the video information generation unit 12 may extract cropped image CIs at positions moved in both the vertical and horizontal directions. In this case, the amount of movement in the vertical direction and the amount of movement in the horizontal direction may be different from each other.
[0051] Figure 5(B) shows image I-32. Figure 5(B) is an example of a case where a position moved in the rotational direction is extracted. In this case, the video information generation unit 12 extracts multiple extracted images CI at different positions by moving the extraction position in an arc shape with a rotation center 0 and radius r. In the example shown in the figure, the video information generation unit 12 extracts a position rotated counterclockwise. By extracting in this way, it is possible to generate a video in which the subject moves in the rotational direction. The position of the rotation center O and the size of the radius r may differ from frame to frame.
[0052] Figure 5(C) shows image I-33. Figure 5(C) is an example of enlarging and reducing the cropping area. In this embodiment, it is preferable that the size of the cropped image CI remains constant. Therefore, the video information generation unit 12 enlarges or reduces image I while maintaining the size of the cropped image CI to crop it. If the size of the cropped image CI is fixed at 256 pixels × 256 pixels, the video information generation unit 12 enlarges and reduces image I so that it fits within the size of the cropped image CI. By cropping in this way, it is possible to generate a video that appears as if the subject is zoomed in or zoomed out.
[0053] The cropping positions described with reference to Figures 5(A) to 5(C) are just examples of this embodiment, and the video information generation unit 12 may generate video information by cropping and joining other different positions. The video information generation unit 12 may also crop cropped image CIs by combining, for example, the cropping methods described with reference to Figures 5(A) to 5(C). In this case, it is possible to generate videos that, for example, rotate after horizontal or vertical movement, or enlarge or reduce after movement.
[0054] Furthermore, the movement of the cropping position as described above may be calculated by affine transformation. In other words, the predetermined direction in which the video information generation unit 12 crops the image can also be described as being calculated by affine transformation.
[0055] In addition, the video information generation unit 12 may generate a video by cutting out a part of an image and then moving it, instead of using the example of changing the cutting position as described above. In this case, the video information generation unit 12 cuts out a 256-pixel x 256-pixel image and generates multiple pixels by moving the cut-out image in a predetermined direction. The video information generation unit 12 generates a video by connecting the cut-out images. That is, the video information generation unit 12 may cut out multiple images at different positions by shifting the multiple cut-out images in a predetermined direction. However, moving the image after cropping can result in areas around the image where no data exists. By pre-defining the surrounding area as a margin, this can be excluded from the image range used for training, preventing problems in later training stages.
[0056] In the above description, an example was given in which the video information generation unit 12 generates a video by extracting images moved in a direction calculated by some method such as affine transformation. However, in actual videos, subjects often do not move in these calculated directions, and more often than not, they move randomly. Therefore, the learning device 10 can generate a video by extracting images moved in a direction based on the actual trajectory of the object's movement, thereby generating training data that is more effective for machine learning. An example of such a case will be described as a modification of the first embodiment with reference to Figures 6 and 7.
[0057] In bright scenes, such as on a sunny day, it is common practice to increase the shutter speed to maintain exposure. As a result, the smoothness of moving subjects is lost, and the resulting video is known to be jerky. Similarly, when creating a video from a high-resolution still image, the resulting video may be jerky and unnatural with little smoothness. For this reason, the video information generation unit 12 may perform a correction to add pseudo-subject blur to the still image from which the video is to be created before generating the video. For example, subject blur may be added by performing a predetermined averaging process in the shift direction or by performing a process that reduces the resolution.
[0058] Figure 6 is a diagram showing an example of the functional configuration of a learning device when a modified learning device according to the first embodiment generates a video based on a trajectory vector. Referring to this figure, an example of the functional configuration of a learning device 10A according to a modified learning device of the first embodiment will be described. The learning system 1A according to a modified learning device of the first embodiment differs from the learning system 1 in that it further comprises a trajectory vector generation device 50. The learning device 10A differs from the learning device 10 in that it further comprises a trajectory vector acquisition unit 14. Furthermore, the learning device 10A differs from the learning device 10 in that it comprises a video information generation unit 12A instead of a video information generation unit 12. In the description of the learning device 10A, components similar to those of the learning device 10 may be denoted by the same reference numerals, and their description may be omitted.
[0059] The trajectory vector generator 50 acquires information about the trajectory of an object captured in a video. The trajectory vector generator 50 receives video information as input and analyzes the trajectory of the object captured in the input video information. The trajectory vector generator 50 outputs the analysis result as a trajectory vector TV. The trajectory vector TV shows the trajectory of the object captured in the video information. The trajectory vector generator 50 acquires the trajectory vector TV from the video information using conventional techniques such as optical flow. Furthermore, the trajectory vector TV may include, in addition to or instead of, vector information, coordinate information showing the trajectory of the object's movement.
[0060] The trajectory vector acquisition unit 14 acquires the trajectory vector TV from the trajectory vector generation device 50. The trajectory vector acquisition unit 14 outputs the acquired trajectory vector TV to the video information generation unit 12A. The video from which the trajectory vector TV was acquired by the trajectory vector generation device 50 and the image acquired by the image acquisition unit 11 may have a predetermined relationship. In this case, for example, the image acquisition unit 11 may acquire one frame of the video from which the trajectory vector TV was acquired by the trajectory vector generation device 50 as an image. However, this embodiment is not limited to this example, and the video obtained by the trajectory vector generation device 50 and the video obtained by the image acquisition unit 11 may not have a predetermined relationship.
[0061] The video information generation unit 12A acquires image information I from the image acquisition unit 11 and the trajectory vector TV from the trajectory vector acquisition unit 14. Based on the acquired image information I and trajectory vector TV, the video information generation unit 12A generates video information. Based on the trajectory shown in the trajectory vector TV, the video information generation unit 12A determines the cropping direction of the cropped image CI and the amount of shift per frame. In other words, the predetermined direction in which the video information generation unit 12A crops the image is calculated based on the acquired trajectory vector TV.
[0062] Figure 7 illustrates an example of the position of an image extracted from a still image when a learning device according to a modification of the first embodiment generates a video based on a trajectory vector. Referring to this figure, an example of the position coordinates of an extracted image CI when generating a video based on a trajectory vector TV will be explained. Figure 7(A) shows image I-41, which is an example of an image included in the first image information I1. Figure 7(B) shows an example of multiple extracted images CI extracted from image I-41.
[0063] As shown in Figure 7(A), image I-41 shows the trajectory vector TV, which is the trajectory of the subject, ball B. The trajectory vector TV represents a vector that shows ball B falling from the upper right to the lower center of the figure, bouncing in the lower center, and then moving towards the upper left of the figure. The video information generation unit 12A extracts a position coordinate cutout image CI based on the trajectory vector TV shown in image I-41, and generates a video from the still image I-41 by temporally stitching together the extracted images.
[0064] Figure 7(B) shows an example of a cropped image CI, which is an image extracted by the video information generation unit 12. Specifically, cropped images CI-41 to CI-49 are shown as examples of images extracted by the video information generation unit 12. Cropped images CI-41 to CI-49 are located in coordinates based on the trajectory vector TV. That is, cropped image CI-41 is located in the upper right direction in the figure, and the cropping position moves towards the lower center in the figure from cropped image CI-45 to cropped image CI-49. Furthermore, the cropping position moves towards the upper left direction in the figure from cropped image CI-45 to cropped image CI-49.
[0065] Figure 8 is a flowchart showing an example of a series of operations for the learning method of the noise reduction device according to the first embodiment. Referring to this figure, an example of a series of operations for the learning method of the noise reduction device using the learning device 10 will be explained.
[0066] (Step S110) First, the image acquisition unit 11 acquires an image. The image acquisition unit 11 acquires first image information I1 which contains a high-quality image and second image information I2 which contains a low-quality image. The step of acquiring an image by the image acquisition unit 11 may be described as the image acquisition step or image acquisition process.
[0067] (Step S130) Next, the video information generation unit 12 extracts a portion of the acquired image. The video information generation unit 12 extracts multiple extracted image CIs from the acquired image. The video information generation unit 12 extracts multiple extracted image CIs from the high-quality image included in the first image information I1 and from the low-quality image included in the second image information I2. It is preferable that the position coordinates extracted from the high-quality image included in the first image information I1 and the low-quality image included in the second image information I2 are the same. However, if there is a time difference between the timing of acquiring the high-quality image included in the first image information I1 and the timing of acquiring the low-quality image included in the second image information I2, a shift due to the time difference may occur in the subject included in the extracted image. In such cases, it is preferable that the position coordinates extracted from the high-quality image included in the first image information I1 and the low-quality image included in the second image information I2 be determined taking into account the shift due to the time difference. More specifically, it is preferable to change the position coordinates extracted from the high-quality image included in the first image information I1 or the low-quality image included in the second image information I2 in a way that reduces the amount of shift caused by the time difference.
[0068] (Step S150) Next, the video information generation unit 12 generates a video by stitching together the extracted images. The video information generation unit 12 generates a high-quality video by stitching together multiple images extracted from high-quality images, and generates a low-quality video by stitching together multiple images extracted from low-quality images. The steps of generating video information in steps S130 and S150 may be described as the video information generation step or the video information generation process.
[0069] (Step S170) Finally, the learning unit 13 uses the combination of the generated high-quality and low-quality videos as training data TD to learn to infer high-quality videos from low-quality videos. This step may be referred to as the learning step or learning process.
[0070] [Summary of the first embodiment] According to the embodiment described above, the learning device 10 includes an image acquisition unit 11 to acquire first image information I1 and second image information I2. The first image information I1 includes at least one image, and the second image information I2 includes at least one image of the same subject as the image captured in the first image information I1, but of lower quality than the image included in the first image information I1. The learning device 10 also includes a video information generation unit 12 to extract multiple images from different positions that are part of the first image information I1, and combine the extracted images to generate first video information M1. Similarly, the learning device 10 includes a video information generation unit 12 to extract multiple images from different positions that are part of the second image information I2, and combine the extracted images to generate second video information M2. The learning device 10 also includes a learning unit 13 to train the learning unit 13 to infer high-quality video from low-quality video based on training data TD which includes the first video information M1 and second video information M2 generated by the video information generation unit 12. In other words, according to this embodiment, the learning device 10 does not need to acquire training data, which includes low-quality and high-quality videos, by shooting videos, as was previously required, but can generate it from still images. Therefore, according to this embodiment, training data for inferring high-quality videos from low-quality videos can be easily generated.
[0071] Furthermore, according to this embodiment, the learning device 10 can generate multiple different videos from the same still image. Therefore, according to this embodiment, it is not necessary to prepare a large number of still images to generate a large amount of training data TD, and many videos can be generated from a small number of still images. Thus, according to this embodiment, the time required to capture images for use in learning can be shortened.
[0072] Furthermore, according to the embodiment described above, the second image information I2 includes a plurality of images in which the same subject as the subject captured in the image included in the first image information I1 is captured, and each image has different noise superimposed on it. The video information generation unit 12 generates the second video information M2 by cutting out different parts from each of the plurality of images included in the second image information I2. In other words, according to this embodiment, a low-quality video with superimposed noise is generated based on a plurality of different low-quality images with superimposed noise. Therefore, the second video information M2 generated by this embodiment has different noise superimposed on each frame, and can reproduce and generate a low-quality video with superimposed noise with greater accuracy.
[0073] Furthermore, according to the embodiment described above, the multiple images included in the second image information I2 are images captured at different times in close proximity. That is, the low-quality images for generating low-quality video are captured at close proximity. Close proximity may be, for example, 1 / 60th of a second. Here, unlike still images, video may be superimposed with video-specific noise that has a temporal component. Images captured at different times in close proximity contain this video-specific noise. Therefore, according to this embodiment, since the learning device 10 generates video based on images captured at different times in close proximity, it can reproduce and generate video-specific noise that has a temporal component.
[0074] Furthermore, according to the embodiment described above, the video information generation unit 12 generates the first video information M1 by extracting a different portion from a single image included in the first image information I1. In other words, according to this embodiment, a high-quality video is generated based on a single image. Therefore, according to this embodiment, it is not necessary to capture many high-quality images, and a high-quality video can be easily generated.
[0075] Furthermore, according to the embodiment described above, the video information generation unit 12 extracts multiple images at different positions by shifting each of the extracted images by different amounts in a predetermined direction. In other words, according to this embodiment, the learning device 10 shifts the images in a predetermined direction after they have been extracted. To put it another way, after the learning device 10 has extracted the images, it does not need to process the large images, but processes the small extracted images. Therefore, according to this embodiment, the learning device 10 can reduce the processing load.
[0076] Furthermore, according to the embodiment described above, the video information generation unit 12 extracts multiple images at positions moved by a predetermined number of bits in a predetermined direction. The video information generation unit 12 generates a video by stitching together the extracted images. That is, the subject captured in the video generated by the video information generation unit 12 appears to move in a predetermined direction within the video. Therefore, according to this embodiment, a video can be easily generated from a still image.
[0077] Furthermore, according to the embodiment described above, the predetermined direction in which the video information generation unit 12 extracts an image is calculated by affine transformation. In other words, the predetermined direction in which the video information generation unit 12 extracts an image is the direction in which the subject moves within the video. Therefore, according to this embodiment, the learning device 10 can generate videos in which the subject moves in various directions.
[0078] Furthermore, according to the embodiment described above, the learning device 10 further includes a trajectory vector acquisition unit 14 to acquire the trajectory vector TV. The predetermined direction in which the video information generation unit 12 extracts an image is calculated based on the acquired trajectory vector TV. The trajectory vector TV is information about a vector that shows the trajectory of the subject's actual movement in the captured video. Therefore, according to this embodiment, it is possible to generate a video based on the trajectory of the subject's actual movement.
[0079] [Second Embodiment] Next, a second embodiment will be described with reference to Figures 9 and 10. In the first embodiment, high-quality and low-quality images were required to create the training data TD, whereas the second embodiment differs from the first embodiment in that only high-quality images are required.
[0080] Figure 9 is a diagram illustrating the overview of the learning system according to the second embodiment. The overview of the learning system 1B according to the second embodiment will be described with reference to this figure. In the description of this figure, components similar to those in the first embodiment may be denoted by the same reference numerals and their description may be omitted. In the second embodiment, the imaging device 20 captures a high-resolution image 31. A low-resolution image 32 is generated by the learning device 10B according to the second embodiment based on the high-resolution image 31. The low-resolution image 32 is generated by, for example, superimposing noise on the high-resolution image 31 through image processing. That is, according to this embodiment, the imaging device 20 captures only the high-resolution image 31 and does not need to capture the low-resolution image 32.
[0081] Figure 10 shows an example of the functional configuration of the video information generation unit according to the second embodiment. The video information generation unit 12B provided in the learning device 10B will be described with reference to this figure. The learning device 10B according to the second embodiment differs from the learning device 10 in that it includes a video information generation unit 12B instead of a video information generation unit 12. The video information generation unit 12B includes a cutting unit 121, a noise superposition unit 123, a first video information generation unit 125, and a second video information generation unit 127.
[0082] The cropping unit 121 acquires an image from the image acquisition unit 11. In this embodiment, since the learning device 10B acquires a high-quality image from the imaging device 20, the cropping unit 121 acquires a high-quality image from the image acquisition unit 11. The cropping unit 121 extracts multiple cropped images CI from a portion of the acquired high-quality image, each with different position coordinates. The cropping unit 121 outputs the extracted cropped images CI to the first video information generation unit 125 and the noise superposition unit 123.
[0083] The noise superposition unit 123 acquires the cropped image CI extracted by the cropping unit 121. The noise superposition unit 123 superimposes noise onto the acquired cropped image CI. The noise superposition unit 123 acquires multiple cropped image CIs extracted from multiple position coordinates and superimposes noise onto each of the acquired multiple cropped image CIs. The noise superimposed by the noise superposition unit 123 may be pre-modeled. Examples of modeled noise include shot noise due to fluctuations in the number of photons, noise generated when light incident on the image sensor is converted into electrons, noise generated when the converted electrons are converted into analog voltage values, and noise generated when the converted analog voltage values are converted into digital signals. The intensity of the superimposed noise may be adjusted by a predetermined method. It is preferable for the noise superposition unit 123 to superimpose different noises onto each of the multiple cropped image CIs. The noise superposition unit 123 outputs the image after noise superimposition as a noise image NI to the second video information generation unit 127.
[0084] The first video information generation unit 125 acquires multiple extracted images CI from the extraction unit 121. The first video information generation unit 125 combines the extracted images to generate the first video information M1. The first video information generation unit 125 outputs the generated first video information M1 to the learning unit 13.
[0085] The second video information generation unit 127 acquires multiple noise images NI from the noise superposition unit 123. The second video information generation unit 127 combines the multiple noise images NI with superimposed noise to generate second video information M2. The second video information generation unit 127 outputs the generated second video information M2 to the learning unit 13.
[0086] The learning unit 13 obtains the first video information M1 from the first video information generation unit 125 and the second video information M2 from the second video information generation unit 127. Based on the first video information M1 and the second video information M2 generated by the video information generation unit 12B, the learning unit 13 trains the learning model 40.
[0087] [Summary of the second embodiment] According to the embodiment described above, the learning device 10B includes an image acquisition unit 11 to acquire image information I including at least one high-quality image. The learning device 10B also includes a video information generation unit 12B to generate both high-quality and low-quality videos from the high-quality image. The video information generation unit 12B includes a cropping unit 121 to crop multiple images that are part of the acquired image information I but at different positions. The video information generation unit 12B also includes a noise superposition unit 123 to superimpose noise onto each of the multiple images cropped by the cropping unit 121. The video information generation unit 12B includes a first video information generation unit 125 to generate first video information M1, which is a high-quality video, by combining the multiple images cropped by the cropping unit 121, and a second video information generation unit 127 to generate second video information M2, which is generated by combining multiple images on which noise has been superimposed by the noise superposition unit 123. Furthermore, the learning device 10B, equipped with a learning unit 13, is trained to infer high-quality videos from low-quality videos based on training data TD which includes first video information M1 generated by the first video information generation unit 125 and second video information M2 generated by the second video information generation unit 127. In other words, the learning device 10B generates high-quality videos and low-quality videos based on a single high-quality image and trains a learning model 40 that infers high-quality videos from low-quality videos. Inferring high-quality videos from low-quality videos is, in other words, noise reduction. Therefore, according to this embodiment, it is possible to easily train a noise reduction model without requiring time to acquire training data TD.
[0088] In the second embodiment, a high-quality video was generated from a high-quality image, a low-quality image was generated by superimposing noise onto the high-quality image, and a low-quality video was generated based on the generated low-quality image. However, this embodiment is not limited to this example. For example, as a modification of this embodiment, the learning device 10 may create training data TD based only on low-quality images. That is, a low-quality video may be generated from a low-quality image, a high-quality image may be generated by removing noise from the low-quality video, and a high-quality video may be generated based on the generated high-quality image. The images used to generate the video may be one image or multiple images.
[0089] In the first embodiment, the learning devices 10 and 10A, and in the second embodiment, the learning device 10B, were described as examples of being used to train a learning model 40 that infers high-resolution video from low-resolution video, but the model is not limited to these examples. For example, the learning model 40 may be configured to include a function to detect specific subjects such as people in the high-resolution video after inferring high-resolution video from low-resolution video, or it may be configured to include a function to perform character recognition of signs and billboards in the high-resolution video. In other words, the high-resolution video inferred by the learning model 40 is not limited to an example of a video for viewing, but may also be used for purposes such as object detection.
[0090] Traditionally, to improve the generalization performance of a learning model, it has been preferable to include as many anticipated scenes as possible in the training data. In other words, a video that includes as many anticipated movements of the subject as possible can be considered ideal training data. On the other hand, obtaining such training data through actual shooting is difficult and requires enormous costs and time. By using this embodiment to train a learning model, the costs and time required to collect training data can be significantly reduced, and by using this embodiment to train a learning model, it is possible to improve the generalization performance of the learning model.
[0091] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of symbols]
[0092] 1...Learning system, 10...Learning device, 11...Image acquisition unit, 12...Video information generation unit, 13...Learning unit, 14...Trajectory vector acquisition unit, 20...Imaging device, 31...High-resolution image, 32...Low-resolution image, 33...High-resolution video, 34...Low-resolution video, 40...Learning model, 50...Trajectory vector generation device, TD...Training data, I...Image information, I1...First image information, I2...Second image information, M...Video information, M1...First video information, M2...Second video information, TV...Trajectory vector, 121...Extraction unit, 123...Noise superposition unit, First video information generation unit 125, Second video information generation unit 127
Claims
1. An image acquisition unit acquires first image information including at least one image, and second image information including at least one image of the same subject as the image captured in the first image information, captured at the same angle of view and imaging angle as when the image included in the first image information was captured, but with at least one of the ISO sensitivity or exposure time differing, and having a lower image quality than the image included in the first image information. A video information generation unit generates first video information by combining multiple images from different positions within a portion of the acquired first image information, each of the multiple images extracted from the second image information corresponds to each of the multiple images extracted from the first image information, the positions from which the corresponding images are extracted are approximately the same, and the multiple images extracted are combined to generate second video information. A learning unit that learns to infer high-resolution video from low-resolution video based on training data including the first video information and the second video information generated by the video information generation unit. A learning device equipped with the following features.
2. The second image information includes a plurality of images in which the same subject as the subject captured in the image included in the first image information is captured, and each of these images has different noises superimposed on it. The video information generation unit generates the second video information by extracting different parts from each of the multiple images included in the second image information. The learning device according to claim 1.
3. The multiple images included in the second image information are images taken at different times in close proximity. The learning device according to claim 2.
4. The video information generation unit generates the first video information by extracting different parts from one image included in the first image information. A learning device according to claim 1 or claim 2.
5. The aforementioned video information generation unit extracts multiple images at different positions by shifting the extracted images in a predetermined direction. A learning device according to claim 1 or claim 2.
6. The video information generation unit extracts multiple images at positions shifted by a predetermined number of bits in a predetermined direction. A learning device according to claim 1 or claim 2.
7. The predetermined direction in which the video information generation unit extracts the image is calculated by affine transformation. The learning device according to claim 5.
8. It further includes a trajectory vector acquisition unit that acquires the trajectory vector, The predetermined direction in which the video information generation unit extracts the image is calculated based on the acquired trajectory vector. The learning device according to claim 5.
9. An image acquisition unit that acquires image information including at least one image, A cropping unit that extracts multiple images from different locations, which are part of the acquired image information, A first video information generation unit generates first video information by combining a plurality of images extracted by the aforementioned extraction unit, A noise superposition unit superimposes noise onto each of the multiple images extracted by the extraction unit, which were used to generate the first video information. A second video information generation unit generates second video information by combining multiple images on which noise has been superimposed by the noise superimposition unit, A learning unit that learns to infer high-resolution video from low-resolution video based on training data which includes the first video information generated by the first video information generation unit and the second video information generated by the second video information generation unit. A learning device equipped with the following features.
10. On the computer, Image acquisition step: Acquire first image information including at least one image, and second image information including at least one image of the same subject as the image captured in the first image information, captured at the same angle of view and shooting angle as when the image included in the first image information was captured, but with at least one of the ISO sensitivity or exposure time differing, and having a lower image quality than the image included in the first image information. A video information generation step which involves extracting multiple images from different positions within the acquired first image information, combining the extracted images to generate first video information, extracting multiple images from different positions within the acquired second image information, each of the multiple images extracted from the second image information corresponding to each of the multiple images extracted from the first image information, the positions from which the corresponding images are extracted being approximately the same, and combining the extracted images to generate second video information, A learning step in which training is performed to infer high-resolution video from low-resolution video based on training data that includes the first video information and the second video information generated in the video information generation step, A program that executes the command.
11. Image acquisition steps include acquiring first image information including at least one image, and second image information including at least one image of the same subject as the image captured in the first image information, with the same field of view and imaging angle as when the image included in the first image information was captured, but with at least one of the ISO sensitivity or exposure time differing, resulting in lower quality than the image included in the first image information. A video information generation step which involves extracting multiple images from different positions within the acquired first image information, combining the extracted multiple images to generate first video information, extracting multiple images from different positions within the acquired second image information, each of the multiple images extracted from the second image information corresponding to each of the multiple images extracted from the first image information, the positions from which the corresponding images are extracted being approximately the same, and combining the extracted multiple images to generate second video information, A learning process is performed to train the system to infer high-resolution video from low-resolution video based on training data that includes the first video information and the second video information generated by the video information generation process. A learning method for a noise reduction device having the following characteristics.