Input information generating device, image processing device, input information generating method, learning device, program, and learning method of noise reduction device

Abstract

The input information generating device comprises: an image acquiring unit, which acquires a plurality of frames constituting a dynamic image, including at least a plurality of object frames as an object of generating input information, as input images; an input transforming unit, which transforms the pixel values ​​of the acquired plurality of input images into a plurality of 2-bit input data; a synthesizing unit, which synthesizes the transformed plurality of input data into one synthesized data; and an output unit, which outputs the synthesized data after synthesis.

Description

Technical Field

[0001] The present invention relates to an input information generating device, an image processing device, an input information generating method, a learning device, a program, and a learning method for a noise reduction device.

[0002] This application claims priority based on Japanese Patent Application No. 2022-137834 filed in Japan on August 31, 2022, and Japanese Patent Application No. 2022-137843 filed in Japan on August 31, 2022, and cites all the contents described in those applications. Background Art

[0003] When an image is captured by a camera, if the ambient light is insufficient or the camera settings such as shutter speed, aperture, or ISO sensitivity are inappropriate, the image may be of low quality. There is a technology for converting a captured low-quality image into a high-quality image by image processing. For example, there is a technology for processing a low-quality image into a high-quality image using machine learning (for example, see Patent Document 1). In such a technical field, making the image high-quality is the top priority.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: U.S. Patent No. 10623756 Summary of the invention

[0007] Problem that the invention aims to solve

[0008] Here, it is considered to apply the above-mentioned prior art to dynamic images, so that the image quality of each frame constituting the dynamic image is improved, thereby converting the low-quality dynamic image into a high-quality dynamic image. In the case of making the dynamic image captured by the camera device high-quality in real time, if the image processing takes time, the problem of sacrificing the frame rate of the dynamic image sometimes arises. That is, in the case of converting a low-quality dynamic image into a high-quality dynamic image, it is not possible to prioritize only the high-quality of the frame image, and it is also required to maintain the frame rate through lightweight image processing. Furthermore, the high-quality processing of dynamic images is sometimes also performed on edge devices. If the processing power of edge devices is considered, there is a strong demand for lightweight image processing.

[0009] Therefore, an object of the present invention is to provide a technology capable of converting a low-quality dynamic image into a high-quality dynamic image through light calculation.

[0010] Solutions for solving problems

[0011] (1) One aspect of the present invention is an input information generating device, wherein the input information generating device comprises: an image acquisition unit, which acquires a plurality of frames constituting a dynamic image, including at least an object frame as an object of generating input information, as input images; an input conversion unit, which converts pixel values ​​of the acquired input images of the plurality of frames into input data having a bit number less than the bit number of the pixel values ​​representing the input images; a synthesis unit, which synthesizes the plurality of transformed input data into one synthesized data; and an output unit, which outputs the synthesized data after synthesis.

[0012] (2) In one embodiment of the present invention, based on the input information generating device described in (1) above, the dynamic image is a color dynamic image, the image acquisition unit acquires pixel values ​​of each color from one frame as a plurality of different images, and the input conversion unit converts the plurality of images acquired from one frame into the input data respectively.

[0013] (3) According to one aspect of the present invention, in the input information generating device described in (1) or (2), the image acquisition unit acquires images of a plurality of frames that are continuously adjacent to and before the target frame, among the frames constituting the dynamic image.

[0014] (4) According to one embodiment of the present invention, based on the input information generating device described in any one of (1) to (3) above, after the output unit outputs the synthesized data about the object frame, the image acquisition unit takes a frame adjacent to the object frame as the object frame, and acquires a plurality of frames including at least the object frame as the input image.

[0015] (5) One aspect of the present invention is based on the input information generating device described in any one of (1) to (4) above, and further includes an integration unit, which integrates multiple adjacent frames into one integrated frame by performing operations based on pixel values ​​of adjacent frames that are frames other than the object frame among the multiple frames acquired by the image acquisition unit, the input transformation unit transforms the pixel values ​​of the object frame into the input data, and further transforms the pixel values ​​of the integrated frame into the input data, and the synthesis unit synthesizes the multiple input data transformed based on the object frame and the multiple input data transformed based on the integrated frame into one synthesized data.

[0016] (6) According to one aspect of the present invention, in the input information generating device described in (5) above, the integration unit uses an average value of pixel values ​​of the plurality of adjacent frames as a pixel value of the integration frame.

[0017] (7) According to one aspect of the present invention, in the input information generating device described in (6) above, the integrating unit calculates the pixel value of the integrated frame by calculating a weighted average according to a temporal distance from the target frame among the plurality of adjacent frames.

[0018] (8) One aspect of the present invention is the input information generating device described in (6) above, wherein the integration unit excludes frames having large brightness changes from among frames constituting the moving image from a calculation target of the average value.

[0019] (9) One embodiment of the present invention is based on the input information generating device described in (5) above, and further includes an average value temporary storage unit for storing the average value of pixel values ​​of predetermined frames among the frames constituting the dynamic image, and the integration unit calculates the pixel value of the integrated frame by performing operations based on the value stored in the average value temporary storage unit and the object frame.

[0020] (10) One aspect of the present invention is based on the input information generating device described in (9) above, and further comprises: a camera condition acquisition unit for acquiring the camera conditions of the dynamic image; and an adjustment unit for adjusting the average value stored in the average value temporary storage unit according to the acquired camera conditions.

[0021] (11) One aspect of the present invention is based on the input information generating device described in (9) above, and further includes a comparison unit for comparing the value stored in the temporary storage unit for the average value with the pixel value of the object frame; when the result of the comparison by the comparison unit is that the difference is less than a predetermined value, the integration unit calculates the pixel value of the integration frame by calculating the moving average based on the value stored in the temporary storage unit for the average value and the object frame; when the difference is not less than the predetermined value, the pixel value of the object frame is used as the pixel value of the integration frame.

[0022] (12) One aspect of the present invention is the input information generating device described in (5) above, wherein the integration unit uses a randomly determined frame among the adjacent frames acquired by the image acquisition unit as the integration frame.

[0023] (13) One embodiment of the present invention is an image processing device comprising: an input information generating device described in any one of (1) to (12) above; and a convolutional neural network that uses the synthesized data output by the input information generating device as input information.

[0024] (14) One aspect of the present invention is an input information generating method, wherein the input information generating method comprises: an image acquisition step of acquiring a plurality of frames constituting a dynamic image, including at least an object frame as an object of generating input information, as input images; an input transformation step of transforming pixel values ​​of the acquired input images of the plurality of frames into input data having a bit number less than the bit number of the pixel values ​​representing the input images; a synthesis step of synthesizing the plurality of transformed input data into one synthesized data; and an output step of outputting the synthesized data after synthesis.

[0025] (A1) One aspect of the present invention is a learning device, wherein the learning device comprises: an image acquisition unit, which acquires first image information and second image information, the first image information including at least one image, the second image information including at least one image having a lower quality than the image included in the first image information, the image capturing the same subject as the subject captured in the image included in the first image information; a dynamic image information generation unit, which cuts out a plurality of images which are parts of the acquired first image information and at different positions, combines the cut out plurality of images to generate the first dynamic image information, cuts out a plurality of images which are parts of the acquired second image information and at different positions, combines the cut out plurality of images to generate the second dynamic image information; and a learning unit, which performs learning in a manner of inferring a high-quality dynamic image from a low-quality dynamic image based on training data including the first dynamic image information and the second dynamic image information generated by the dynamic image information generation unit.

[0026] (A2) One method of the present invention is based on the learning device described in (A1) above, wherein the second image information includes a plurality of images, which are a plurality of images obtained by capturing the same subject as the subject captured in the image included in the first image information, and each of which is superimposed with mutually different noises, and the dynamic image information generating unit generates the second dynamic image information by cutting out different parts from each of the plurality of images included in the second image information.

[0027] (A3) One aspect of the present invention is the learning device described in (A1) or (A2) above, wherein the plurality of images included in the second image information are images captured at close but different times.

[0028] (A4) According to one aspect of the present invention, based on the learning device described in any one of (A1) to (A3) above, the dynamic image information generating unit generates the first dynamic image information by cutting out different parts from an image included in the first image information.

[0029] (A5) One aspect of the present invention is the learning device according to any one of (A1) to (A4) above, wherein the dynamic image information generating unit cuts out a plurality of images at different positions by shifting the plurality of cut-out images in a predetermined direction.

[0030] (A6) One aspect of the present invention is the learning device according to any one of (A1) to (A5) above, wherein the dynamic image information generation unit cuts out a plurality of images at positions shifted by a predetermined number of bits in a predetermined direction.

[0031] (A7) One aspect of the present invention is the learning device described in (A6) above, wherein the predetermined direction in which the moving image information generating unit cuts out the image is calculated by affine transformation.

[0032] (A8) One aspect of the present invention is a learning device according to (A6) further comprising a trajectory vector acquisition unit for acquiring a trajectory vector, wherein the predetermined direction in which the dynamic image information generation unit cuts out an image is calculated based on the acquired trajectory vector.

[0033] (A9) One aspect of the present invention is a learning device, wherein the learning device comprises: an image acquisition unit for acquiring image information including at least one image; a cutting unit for cutting out a plurality of images which are parts of the acquired image information and are at different positions; a first dynamic image information generating unit for combining the plurality of cut-out images to generate the first dynamic image information; a noise superimposing unit for superimposing noise on the plurality of images cut out by the cutting unit respectively; a second dynamic image information generating unit for combining the plurality of images superimposed with noise by the noise superimposing unit to generate the second dynamic image information; and a learning unit for learning in a manner of inferring a high-quality dynamic image from a low-quality dynamic image based on training data including the first dynamic image information generated by the first dynamic image information generating unit and the second dynamic image information generated by the second dynamic image information generating unit.

[0034] (A10) One aspect of the present invention is a program, wherein the program causes a computer to execute the following steps: an image acquisition step of acquiring first image information and second image information, wherein the first image information includes at least one image, and the second image information includes at least one image having a lower quality than the image included in the first image information, and the image captures the same subject as the subject captured in the image included in the first image information; a dynamic image information generation step of cutting out a plurality of images which are parts of the acquired first image information and at different positions, combining the cut-out plurality of images to generate the first dynamic image information, cutting out a plurality of images which are parts of the acquired second image information and at different positions, and combining the cut-out plurality of images to generate the second dynamic image information; and a learning step of learning in a manner of inferring a high-quality dynamic image from a low-quality dynamic image based on training data including the first dynamic image information and the second dynamic image information generated by the dynamic image information generation step.

[0035] (A11) One aspect of the present invention is a learning method for a noise reduction device, wherein the learning method for the noise reduction device comprises: an image acquisition step of acquiring first image information and second image information, wherein the first image information includes at least one image, and the second image information includes at least one image with lower image quality than the image included in the first image information, and the image captures the same subject as the subject captured in the image included in the first image information; a dynamic image information generation step of cutting out a plurality of images that are parts of the acquired first image information and at different positions, combining the cut-out plurality of images to generate the first dynamic image information, cutting out a plurality of images that are parts of the acquired second image information and at different positions, and combining the cut-out plurality of images to generate the second dynamic image information; and a learning step of learning in a manner of inferring a high-quality dynamic image from a low-quality dynamic image based on training data including the first dynamic image information and the second dynamic image information generated by the dynamic image information generation step.

[0036] Effects of the Invention

[0037] According to the present invention, a low-quality moving image can be converted into a high-quality moving image through light calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a block diagram showing an example of the functional configuration of the high-quality moving image generation system according to the first embodiment.

[0039] Figure 2 This is a diagram showing an example of a convolutional neural network according to the first embodiment.

[0040] Figure 3It is a diagram showing frames constituting a moving image according to the first embodiment.

[0041] Figure 4 This is a diagram for explaining the outline of the input information generating method according to the first embodiment.

[0042] Figure 5 This is a block diagram showing an example of a functional configuration for generating input information according to the first embodiment.

[0043] Figure 6 This is a block diagram showing an example of the functional configuration of the input conversion unit according to the first embodiment.

[0044] Figure 7 This is a diagram for explaining the outline of the input information generating method according to the second embodiment.

[0045] Figure 8 This is a block diagram showing an example of a functional configuration for generating input information according to the second embodiment.

[0046] Fig. 9 This is a block diagram showing an example of a functional configuration for generating input information according to the third embodiment.

[0047] Fig.10 This is a block diagram showing an example of a functional configuration for generating input information according to the fourth embodiment.

[0048] Fig.11 This is a diagram for explaining the outline of the learning system according to the fifth embodiment.

[0049] Fig.12 This is a diagram showing an example of the functional configuration of a learning device according to the fifth embodiment.

[0050] Fig.13 This is a diagram for explaining an example of the position of an image cut out from a high-quality image by the learning device according to the fifth embodiment.

[0051] Fig.14 This is a diagram for explaining an example of the position of an image cut out from a low-quality image by the learning device according to the fifth embodiment.

[0052] Fig.15 This is a diagram for explaining an example of the direction of cutting out by the learning device according to the fifth embodiment.

[0053] Fig.16 This is a diagram showing an example of the functional configuration of the learning device in the case where the learning device according to the fifth embodiment generates a moving image based on trajectory vectors.

[0054] Fig.17This is a diagram for explaining an example of the position of an image cut out from a still image when the learning device according to the modification example of the fifth embodiment generates a dynamic image based on a trajectory vector.

[0055] Fig.18 This is a flowchart showing an example of a series of operations of a learning method of a noise reduction device according to a modification of the fifth embodiment.

[0056] Fig.19 This is a diagram for explaining the outline of the learning system according to the sixth embodiment.

[0057] Fig. 20 This is a diagram showing an example of the functional structure of the dynamic image information generating unit according to the sixth embodiment. DETAILED DESCRIPTION

[0058] Hereinafter, with respect to the input information generating device, image processing device and input information generating method of the present invention, preferred embodiments are disclosed and described in detail with reference to the attached drawings. It should be noted that the present invention is not limited to these embodiments, but also includes a method in which a variety of changes or improvements are applied. That is, the constituent elements described below include elements that can be easily thought of by a person skilled in the art, substantially the same elements, and the constituent elements described below can be appropriately combined. In addition, various omissions, substitutions or changes of constituent elements can be made within the scope of the gist of the present invention. In addition, in the following drawings, in order to facilitate the understanding of each structure, the scale and quantity in each structure are sometimes different from the scale and quantity in the actual structure.

[0059] First, the matters that are the premise of the present embodiment are described. The input information generating device, image processing device, and input information generating method of the present embodiment take low-quality dynamic image information with noise superimposed as input, and generate high-quality dynamic image information with noise removed. Low-quality dynamic images include low-quality dynamic images, and high-quality dynamic images include high-quality dynamic images. As an example, a high-quality dynamic image can be exemplified by a dynamic image with high image quality captured by low ISO sensitivity and long exposure. As an example, a low-quality dynamic image can be exemplified by a dynamic image with low image quality captured by high ISO sensitivity and short exposure.

[0060] In the following description, as an example of low-quality dynamic images, the image quality degradation caused by noise is described, but in addition to noise, the present embodiment can also be widely applied to matters that cause the quality degradation of dynamic images. As matters that degrade the quality of dynamic images, it is possible to exemplify the reduction in resolution or color deviation caused by optical aberration, the reduction in resolution caused by hand shaking or subject shaking, the unevenness of black level caused by dark current and circuit reasons, ghosting, flare, signal level abnormality caused by high-brightness subjects, etc. In addition, in addition to the random noise generated by each pixel, the noise also includes striped noise generated in the horizontal or vertical direction of the image, noise generated in a fixed pattern in the image, etc. In addition, it can also include noise unique to dynamic images such as flickering noise that changes between consecutive frames. The input information generation device, image processing device, and input information generation method of this embodiment improve the quality of the frame by performing image processing on each frame contained in the dynamic image, thereby improving the quality of the frame as a dynamic image.

[0061] It should be noted that the input dynamic image to be improved in quality may be a dynamic image captured by a camera or a dynamic image prepared in advance. In the following description, a low-quality dynamic image is sometimes described as a low-quality dynamic image or a noisy dynamic image. In addition, in the following description, a high-quality dynamic image is sometimes described as a high-quality dynamic image.

[0062] In addition, the dynamic image that is the object of the input information generating device, image processing device, and input information generating method of this embodiment may also be a dynamic image captured by a CCD camera using a CCD (Charge Coupled Devices) image sensor. In addition, the dynamic image that is the object of the input information generating device, image processing device, and input information generating method of this embodiment may also be an image captured by a CMOS (Complementary Metal Oxide Semiconductor) camera using a CMOS image sensor. In addition, the dynamic image that is the object of the input information generating device, image processing device, and input information generating method of this embodiment may be a color dynamic image or a monochrome dynamic image. In addition, the dynamic image that is the object of the input information generating device, image processing device, and input information generating method of this embodiment may also be a dynamic image captured by acquiring invisible light components using an infrared camera or the like using an infrared sensor.

[0063] [First embodiment]

[0064] First, refer to Figures 1 to 6The first embodiment will be described.

[0065] Figure 1 It is a block diagram showing an example of the functional structure of the high-quality dynamic image generation system of the first embodiment. With reference to this figure, an example of the functional structure of the high-quality dynamic image generation system 1 is described. The high-quality dynamic image generation system 1 includes a camera device 100, an input information generation device 10, and a convolutional neural network 200 (hereinafter referred to as "CNN200") as its functions. The input information generation device 10 and CNN200 perform image processing on each frame constituting the dynamic image captured by the camera device 100. The input information generation device 10 and CNN200 include a learned model that has been learned in advance. In the following description, the structure including the input information generation device 10 and CNN200 is sometimes described as an image processing device 2. It should be noted that the high-quality dynamic image generation system 1 can also be configured to include a coding unit that compresses and encodes the output of the image processing device 2 and a predetermined memory that stores the result of the compression encoding by the coding unit.

[0066] The camera device 100 captures a dynamic image. The dynamic image captured by the camera device 100 is a low-quality dynamic image that is to be improved in quality. The camera device 100 may be, for example, a surveillance camera installed in a darker place (where the amount of light is low). The camera device 100 captures a low-quality dynamic image caused by insufficient light, for example. The camera device 100 outputs the captured dynamic image to the input information generating device 10. The dynamic image captured by the camera device 100 becomes an input to the image processing device 2. Therefore, the dynamic image output from the camera device 100 to the input information generating device 10 is sometimes recorded as dynamic image information IM.

[0067] It should be noted that the camera device 100 and the image processing device 2 may also be present in the housing of a smartphone, a tablet terminal, etc. That is, the high-quality dynamic image generation system 1 may also exist as an element constituting an edge device. In addition, the camera device 100 may also be connected to the image processing device 2 via a predetermined communication network. That is, the high-quality dynamic image generation system 1 may also exist by connecting the constituent elements to each other via a predetermined communication network.

[0068] In addition, the camera device 100 may also be a structure having a plurality of lenses and a plurality of image sensors corresponding to the plurality of lenses, respectively. As a specific example of such a structure, a structure in which the camera device 100 has a plurality of lenses and image sensors to obtain images of different perspectives can be exemplified. According to the camera device 100 configured in this way, the images obtained from each image sensor can be spatially adjacent to each other. The high-quality dynamic image generation system 1 can be applied not only to a plurality of images that are temporally adjacent to each other, such as dynamic images, but also to a plurality of images that are spatially adjacent to each other.

[0069] The input information generating device 10 acquires the dynamic image information IM from the camera device 100. The input information generating device 10 generates input information IN based on the acquired dynamic image information IM. The input information IN is generated for each frame constituting the dynamic image information IM. The input information IN may also be generated based on the target frame and other frames determined based on the frame. The other frames determined based on the frame may also be frames that are temporally adjacent to the target frame.

[0070] CNN200 is a convolutional neural network that uses the data output by the input information generating device 10 as input information IN. Figure 2 An example of CNN200 is described.

[0071] Figure 2 : is a diagram showing an example of CNN200 of the first embodiment. With reference to this figure, the details of CNN200 are described in detail. CNN200 is a neural network with a multi-layer structure. CNN200 is a multi-layer network including an input layer 210 to which input information IN is input, a convolution layer 220 for performing convolution operations, a pooling layer 230 for performing pooling, and an output layer 240. In at least a part of CNN200, the convolution layer 220 and the pooling layer 230 are alternately connected. CNN200 is a model widely used for image recognition and dynamic image recognition. CNN200 may also have layers (layers) with other functions such as fully connected layers. It should be noted that the pooling layer 230 may also include a quantization layer for performing a quantization operation for low-bit quantization of the operation result of the convolution layer 220. Specifically, when the result of the convolution operation in the convolution layer 220 is 16 bits, the quantization layer performs an operation in the quantization layer to reduce the number of bits of the result of the convolution operation to less than 8 bits.

[0072] It should be noted that CNN200 may also adopt a structure in which the outputs of the multiple convolutional layers 220 and pooling layers 230 included in CNN200 are used as intermediate outputs and as inputs to other layers. In addition, as another embodiment, CNN200 may also use the outputs of the multiple convolutional layers 220 and pooling layers 230 included in CNN200 as intermediate outputs and as inputs to other layers, thereby forming a U-net. In this case, CNN200 has: an encoder unit that extracts feature quantities through convolution operations; and a decoder unit that performs inverse convolution operations based on the extracted feature quantities.

[0073] Input information IN is input to the input layer 210. The input information IN is generated based on an input image. The input image is a frame image constituting a dynamic image. The input information generating device 10 of the present embodiment generates input information IN from the input image. In the present embodiment, the elements of the input information IN may be, for example, 2-bit unsigned integers (0, 1, 2, 3). In addition, the elements of the input data may be, for example, 4-bit or 8-bit integers.

[0074] The convolution layer 220 performs a convolution operation on the input information IN input to the input layer 210. The convolution layer 220 performs a convolution operation on the low-bit input information IN. As a result of the convolution operation performed by the convolution layer 220, predetermined output data is output to the pooling layer 230.

[0075] The pooling layer extracts a representative value of a certain area based on the result of the convolution operation performed by the convolution layer 220. Specifically, the pooling layer 230 performs operations such as average pooling and MAX pooling on the output data of the convolution operation output by the convolution layer 220 to compress the output data of the convolution layer 220.

[0076] The output layer 240 is a layer that outputs the result of CNN 200. The output layer 240 may output the result of CNN 200 through, for example, an identity function, a Softmax function, etc. The layer provided at the front stage of the output layer 240 may be a convolution layer 220, a pooling layer 230, or other layers.

[0077] Figure 3 1 is a diagram showing frames constituting a dynamic image according to the first embodiment. Referring to this diagram, the frames used by the input information generating device 10 to generate input information IN are described. This diagram shows a plurality of continuous frames constituting a dynamic image. Frames F1 to F7 shown in this diagram are examples of a plurality of continuous frames constituting a dynamic image.

[0078] It should be noted that each frame is a RAW image that has not been compressed and encoded, and each pixel is represented by 12 bits or 14 bits. In addition, the number of pixels of each frame has the number of pixels required to meet the predetermined dynamic image format such as 1920×1080 or 4096×2160. It should be noted that in this embodiment, the processing object in CNN200 is described as a RAW image, but it is not limited to this. When the image of the processing object fully contains the signal component, the image that has been processed by compression encoding can also be used as the object.

[0079] The input information generating device 10 generates input information IN based on the target frame TF as the target frame and the adjacent frame AF as the frame adjacent to the target frame TF. The adjacent frame AF is, for example, a frame that is continuously adjacent before or after the target frame TF. In the example shown in the figure, the two frames before and after the target frame TF are set as the adjacent frames AF. That is, when the target frame TF is set as frame F4, frames F2, F3, F5, and F6 become adjacent frames AF.

[0080] It should be noted that the number of adjacent frames AF is not limited to this example, and may be one frame or three frames before and after the object frame TF. In addition, the adjacent frames AF are not limited to the example of the case where they are adjacent to the object frame TF, and may be, for example, only frames that are adjacent to either the front or the back of the object frame TF. In addition, the adjacent frames AF do not need to be continuous with the object frame TF. For example, when the frame F4 is used as the object frame TF, the adjacent frames AF may be frames F2 and F6 that are not continuous with the frame F4.

[0081] Figure 4 This is a diagram for explaining an overview of the input information generating method of the first embodiment. With reference to this figure, the method for generating input information IN performed by the input information generating device 10 is explained. In this figure, a frame at time t-2, a frame at time t-1, a frame at time t, a frame at time t+1, and a frame at time t+2 are shown. The frame at time t is equivalent to the above-mentioned object frame TF, and the frames at time t-2, time t-1, time t+1, and time t+2 are equivalent to the adjacent frame AF. It should be noted that since each frame contains a plurality of pixels, the circuit for processing all frames at the same time will be large-scale. Therefore, when processing in CNN200, it is preferable to divide each frame into a predetermined size. In this embodiment, as an example, a case where a size of 256×256 is divided into a plurality of patches as one patch is illustrated.

[0082] Each frame is composed of image data of 4 channels, namely R (red), G (green) × 2 channels, and B (blue). The input information generating device 10 performs quantization and vectorization on each channel. The input information generating device 10 generates, for example, 9-channel vectorized data from the image data of each channel. That is, the input information generating device 10 generates 4 channels × 9 channels = 36 channels of data from 1 frame. The 9-channel data may also be data obtained by quantizing pixel values ​​using different thresholds. The input information generating device 10 synthesizes (concat) 5 frames × 36 channels = 180 channels of data generated from the object frame TF and the adjacent frame AF (a total of 5 frames in the illustrated example). The input information generating device 10 outputs the synthesized 180-channel data as input information IN to the input layer of CNN200.

[0083] It should be noted that, in the example shown in the figure, the case of generating input information IN using image data of 4 channels constituting 1 frame is described, but the method of the present embodiment is not limited to this example. The input information generating device 10 may also generate input information IN based on data of 3 channels including RGB, for example. In addition, in the example shown in the figure, the case of generating 9 channels of data by quantizing and vectorizing based on image data is described, but the method of the present embodiment is not limited to this example. The number of data generated is preferably a number that can be efficiently calculated after synthesis. The input information generating device 10 may also generate N channel × M channel data by generating M channel (M is a natural number greater than 1) data for N channel (N is a natural number greater than 1) image data constituting 1 frame, for example. It should be noted that the value of N×M is preferably a value close to a multiple of 32 (or 64).

[0084] Figure 5 1 is a block diagram showing an example of a functional structure of input information generation according to the first embodiment. With reference to this figure, an example of a functional structure of an input information generation device 10 is described. The input information generation device 10 includes an image acquisition unit 11, an input conversion unit 12, a synthesis unit 13, and an output unit 14. The input information generation device 10 includes a storage device such as a CPU (Central Processing Unit) not shown in the figure, a ROM (Read only memory) or a RAM (Random access memory), etc. connected via a bus. The input information generation device 10 functions as a device including the image acquisition unit 11, the input conversion unit 12, the synthesis unit 13, and the output unit 14 by executing an input information generation program.

[0085] It should be noted that all or part of the functions of the input information generating device 10 can also be implemented using hardware such as ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device) or FPGA (Field-Programmable Gate Array). The input information generating program can also be recorded in a computer-readable recording medium. Computer-readable recording media include removable media such as floppy disks, optical magnetic disks, ROMs, CD-ROMs, and storage devices such as hard disks built into computer systems. The input information generating program can also be sent via an electrical communication line.

[0086] In the example shown in the figure, a memory storing moving image information IM of a moving image captured by the camera device 100 is described as a first memory M1, and a memory storing input information IN generated by the input information generating device 10 is described as a second memory M2. The first memory M1 and the second memory M2 are storage devices such as ROM or RAM.

[0087] The image acquisition unit 11 acquires image information IMD including an input image to be processed from the dynamic image information IM stored in the first memory M1. Specifically, the image acquisition unit 11 acquires a plurality of frames including at least a target frame TF as a target for generating the input information IN, from a plurality of frames constituting the dynamic image, as input images. As an example, the image acquisition unit 11 acquires an adjacent frame AF as an input image in addition to the target frame TF. The adjacent frame AF may also be a plurality of frames that are continuously adjacent to each other before and after the target frame TF.

[0088] It should be noted that, when the dynamic image captured by the camera 100 is a color dynamic image, the image acquisition unit 11 acquires the pixel values ​​of each color from one frame as a plurality of different images. For example, when the camera 100 is a device using an image sensor adopting a Bayer arrangement, the image acquisition unit 11 acquires image information of four channels composed of RGGB from one frame. The pixel value of the image acquired by the image acquisition unit 11 includes a multi-bit element.

[0089] The input conversion unit 12 obtains the image information IMD from the image acquisition unit 11. The input conversion unit 12 converts the pixel values ​​of the input images of the multiple frames included in the image information IMD into low-bit input data IND based on comparison with multiple threshold values. The input image is a RAW image, and its pixel value includes elements of multiple bits (e.g., 12 bits or 14 bits, etc.). Therefore, the image acquisition unit 11 converts the pixel values ​​into input data IND with a number of bits (e.g., 2 bits or 1 bit) less than the number of bits (e.g., 8 bits) representing the pixel values ​​of the input image based on multiple threshold values. The input conversion unit 12 outputs the converted input data IND to the synthesis unit 13.

[0090] It should be noted that, when the image information IMD includes image information of each color of RGB, the input conversion unit 12 performs conversions respectively. That is, the input conversion unit 12 converts each of the plurality of images acquired from one frame into the input data IND.

[0091] Figure 6 1 is a block diagram showing an example of the functional structure of the input conversion unit of the first embodiment. With reference to this figure, the details of the function of the input conversion unit 12 are described. As shown in the figure, the input conversion unit 12 includes a plurality of conversion units 121 and a threshold storage unit 122. In the example shown in the figure, the input conversion unit 12 includes a conversion unit 121-1, a conversion unit 121-2, ..., a conversion unit 121-n (n is a natural number greater than 1) as an example of a plurality of conversion units 121. The number of conversion units 121 included in the input conversion unit 12 may also be the number of input data IND generated by the input conversion unit 121 from a 1-channel input image. That is, when the input conversion unit 12 converts a 1-channel input image into 9-channel input data IND, the input conversion unit 12 includes 9 conversion units 121, namely, conversion units 121-1 to 121-9.

[0092] In the example shown in the figure, the image data of the input image has a matrix data structure with multiple values ​​as pixel data, and the multiple values ​​have more bits than 8 bits as each element in the x-axis direction and the y-axis direction. When the image data of the input image is transformed by the input transformation unit 12, each element is quantized to become low-bit (for example, 2 bits or 1 bit below 8 bits) input data.

[0093] The transformation unit 121 compares each element of the input image with a predetermined threshold value. The transformation unit 121 quantizes each element of the input image based on the comparison result. The transformation unit 121 quantizes a 12-bit input image into a 2-bit or 1-bit value, for example. The transformation unit 121 may also perform quantization by comparing with a threshold value corresponding to the number of bits after the transformation. For example, if it is a transformation to 1 bit, 1 threshold value may be sufficient, and if it is a transformation to 2 bits, 3 threshold values ​​may be used. In other words, regarding the quantization performed in the transformation unit 121, 1 threshold value may be used in the case of 1-bit quantization, and 3 threshold values ​​may be used in the case of 2-bit quantization. It should be noted that in the case where multiple threshold values ​​are required, such as 8 bits, quantization may be performed using functions or tables instead of threshold values.

[0094] Each conversion unit 121 quantizes the same element using an independent threshold. That is, the input conversion unit 12 outputs a vector containing elements corresponding to the number of conversion units 121 as a calculation result (input data IND) for the input of one channel. It should be noted that the bit precision of the conversion result output by the conversion unit 121 can also be appropriately changed based on the bit precision of the input image, etc.

[0095] The threshold storage unit 122 stores a plurality of thresholds used in the calculation performed by the conversion unit 121. The thresholds stored in the threshold storage unit 122 are predetermined values, and are set corresponding to the plurality of conversion units 121. It should be noted that each threshold may be a parameter of the learning object, or may be determined and updated in the learning step.

[0096] It should be noted that, in one example shown in the figure, an example of inputting the same elements of the input image to multiple conversion units 121 is shown, but the method of inputting the conversion unit 12 is not limited to this. For example, in the case where the input image is image data containing elements of more than 3 channels including color components, the conversion unit 121 can also be divided into corresponding multiple groups, and the corresponding elements can be input to each group and transformed. In addition, even for elements other than color components, some kind of transformation processing can be applied in advance to the elements input to the predetermined conversion unit 121, and it can also be switched to which conversion unit 12 is input according to whether there is prior processing. In addition, it is also possible not to perform transformation processing on all elements of the input image, for example, only specific elements in the input image, that is, elements corresponding to specific colors, can be transformed.

[0097] It should be noted that the number of transformation units 121 may not be fixed, and may be appropriately determined in accordance with the structure of the neural network or hardware information. It should be noted that, in the case where it is necessary to compensate for the reduction in calculation accuracy caused by the quantization of the transformation unit 121, the number of transformation units 121 is preferably set to be greater than the bit accuracy of each element of the input image. More generally, it is preferred to set the number of transformation units 121 to be greater than the difference in bit accuracy of the input image before and after quantization. Specifically, in the case of quantizing an input image whose pixel values ​​are represented by 8 bits to 1 bit, the number of transformation units 121 is preferably set to be greater than 7 (for example, 16, 32) which is equivalent to 7 bits as the difference.

[0098] return Figure 5 The synthesizing unit 13 synthesizes (concats) the converted plurality of input data IND into one data. The data synthesized from the plurality of input data is also recorded as synthesized data CD. The synthesizing process performed by the synthesizing unit 13 may also be a process of arranging (or concatenating) the plurality of input data IND into one data.

[0099] The output unit 14 outputs the synthesized data CD synthesized by the synthesizing unit 13. The synthesized data CD may be temporarily stored in the second memory M2. The synthesized data CD is the input information IN input to the input layer 210 of the CNN 200.

[0100] After generating input information IN for the target frame TF, the input information generating device 10 generates input information IN for the next frame of the target frame TF. The next frame may be a frame that is temporally continuous with the target frame TF. That is, after the output unit 14 outputs the synthesized data CD about the target frame TF, the image acquiring unit 11 shifts the target frame TF by one frame and acquires a frame adjacent to the target frame TF as the target frame TF. In this way, the input information generating device 10 acquires a plurality of frames including at least the target frame TF as input images and generates the synthesized data CD.

[0101] The input information generating device 10 generates input information IN for all frames included in the dynamic image information IM. It should be noted that, in the above example, an example of a case where the input information generating device 10 generates input information IN for all frames included in the dynamic image information IM is described, but the method of the present embodiment is not limited to this example. The input information generating device 10 may also generate input information IN, for example, every predetermined frame. In addition, the high-quality dynamic image generating system 1 transforms the dynamic image information IM into a high-quality dynamic image, but the output format is not limited to the dynamic image format. For example, the high-quality dynamic image generating system 1 may also generate a static image from a dynamic image. That is, the high-quality dynamic image generating system 1 can apply the present embodiment even when a static image is generated by extracting a frame included in the dynamic image information IM by setting the frame extracted from the dynamic image as the object frame TF.

[0102] [Summary of the First Embodiment]

[0103] According to the above-described embodiment, the input information generating device 10 acquires a plurality of frames including at least the target frame TF as the object of generating the input information IND among the frames constituting the dynamic image by means of the image acquiring unit 11. In addition, the input information generating device 10 is provided with an input conversion unit 12, and based on comparison with a plurality of threshold values, the pixel values ​​of the acquired plurality of frames of the input image are converted into input data IND having a number of bits (e.g., 2 bits or 1 bit below 8 bits) less than the number of bits representing the pixel values ​​of the input image (e.g., 12 bits). In addition, the input information generating device 10 is provided with a synthesis unit 13, and synthesizes the converted plurality of input data IND into one synthesized data CD, and is provided with an output unit 14, and outputs the synthesized synthesized data CD. That is, according to the present embodiment, the input information generating device 10 synthesizes a plurality of input data IND obtained from a plurality of images including at least the target frame TF to generate the input information IN. The generated input information IN is input to the input layer 210 of the CNN 200.

[0104] Here, input information IN refers to information with lower bits than the input image in each element. CNN200 can process low-bit information by processing input information IN instead of the input image. Therefore, according to this embodiment, the processing of CNN200 can be made lightweight. In addition, input information IN refers to information generated based on multiple images. Therefore, when the dynamic image is improved in quality based on the input information IN, it is possible to perform processing that also takes into account multiple frames adjacent to the object frame TF, so that noise can be removed with high precision. Therefore, according to this embodiment, a low-quality dynamic image can be converted into a high-quality dynamic image through lightweight calculations. It should be noted that the processing of CNN200 is not limited to noise removal.

[0105] In addition, according to the above-mentioned embodiment, the dynamic image to be processed is a color dynamic image, so the dynamic image information IM includes, for example, pixel values ​​of each color of RGB. The image acquisition unit 11 acquires the pixel values ​​of each color from one frame as a plurality of different images, and the input conversion unit 12 converts the plurality of images acquired from one frame into different input data IND. Therefore, according to this embodiment, it is possible to perform image processing with higher accuracy. Therefore, according to this embodiment, it is possible to remove noise with higher accuracy.

[0106] In addition, according to the above-mentioned embodiment, the image acquisition unit 11 acquires images of a plurality of frames that are continuously adjacent to the front and back of the target frame TF, respectively, among the plurality of frames constituting the dynamic image. According to the present embodiment, input information IN is generated based on information of frames that are adjacent to the front and back of the target frame TF, so that image processing with higher accuracy can be performed. Therefore, according to the present embodiment, noise can be removed with high accuracy.

[0107] Furthermore, according to the above-described embodiment, after the synthesized data CD about the target frame TF is outputted by the output unit 14, the image acquisition unit 11 takes the frame adjacent to the target frame TF as the target frame TF and acquires a plurality of frames including at least the target frame TF as the input image. That is, the input information generation device 10 generates input information IN for each of the plurality of frames included in the dynamic image by sequentially shifting the target frame TF. Therefore, according to the present embodiment, a low-quality dynamic image can be converted into a high-quality dynamic image.

[0108] [Second embodiment]

[0109] Next, refer to Figure 7 and Figure 8 The second embodiment is described. First, the problem to be solved in the second embodiment is described. The input information generating device 10 of the first embodiment reads out a plurality of frames including the object frame TF and the adjacent frame AF, and quantizes each of the plurality of frames. Therefore, according to the input information generating device 10 of the first embodiment, the number of frames to be quantized is large, and the calculation load of the first layer becomes large. In the second embodiment, this problem is solved and the calculation load is further reduced.

[0110] Figure 7 This is a diagram for explaining the outline of the input information generating method of the second embodiment. With reference to this diagram, the generating method of the input information IN performed by the input information generating device 10A of the second embodiment is explained. In this diagram, a frame at time t is shown as the object frame TF. In addition, in this diagram, a frame at time t-2, a frame at time t-1, a frame at time t+1, and a frame at time t+2 are shown as adjacent frames AF.

[0111] Each frame is composed of 4-channel image data including RGGB. The input information generating device 10A quantizes and vectorizes each channel of the object frame TF. In addition, the input information generating device 10A quantizes and vectorizes each channel of the average image of the adjacent frames AF. That is, in the second embodiment, the difference from the first embodiment is that the adjacent frames AF are not quantized and vectorized separately, but the average image of the adjacent frames AF is quantized and vectorized.

[0112] In the example shown in the figure, the image data of 4 channels of RGGB are respectively converted into data of 16 channels. Therefore, 4 channels × 16 channels = 64 channels of data are generated from 1 frame. The input information generating device 10A converts the target frame TF and the average image of the adjacent frame AF into data of 64 channels, respectively, thereby generating data of 128 channels in total.

[0113] The averaging process may also be performed by simply averaging the pixel values. In addition, the input information generating device 10A may generate an average image for each color by averaging each color. That is, the input information generating device 10A may generate an average image based on the R images of the frames in the adjacent frames AF from time t-2 to time t+2, generate an average image based on the G images of the frames in the adjacent frames AF from time t-2 to time t+2, and generate an average image based on the B images of the frames in the adjacent frames AF from time t-2 to time t+2.

[0114] In the example shown in the figure, quantization and vectorization are performed after averaging adjacent frame AFs, but the present embodiment is not limited to this example. The input information generating device 10A may be configured to average after quantization and vectorization, for example.

[0115] After quantizing and vectorizing the object frame TF, and quantizing and vectorizing the average image of the adjacent frames AF, these are synthesized to generate input information IN. In the example shown in the figure, 128 channels of data are generated. The synthesized input information IN has less information than the first embodiment. On the other hand, the data representing the object frame TF is more than the first embodiment.

[0116] Figure 8 1 is a block diagram showing an example of a functional structure of input information generation according to the second embodiment. With reference to this figure, an example of a functional structure of an input information generation device 10A is described. The input information generation device 10A is different from the input information generation device 10 in that it further includes an integration unit 15, an input conversion unit 12A instead of the input conversion unit 12, and a synthesis unit 13A instead of the synthesis unit 13. In the description of the input information generation device 10A, the same structure as the input information generation device 10 is sometimes omitted by attaching the same reference numerals to the same structure.

[0117] The integration unit 15 acquires information related to the adjacent frame AF from the image acquisition unit 11. The adjacent frame AF refers to a frame other than the target frame TF among the multiple frames acquired by the image acquisition unit 11. The integration unit 15 integrates the multiple adjacent frames AF into one integrated frame IF by performing calculations based on the pixel values ​​of the adjacent frames AF. The integration unit 15 outputs the integrated frame IF obtained as a result of the calculation to the input conversion unit 12A.

[0118] The integration unit 15 may perform the integration process by, for example, obtaining a simple average of the adjacent frames AF. In this case, the integration unit 15 uses, for example, the average value of the pixel values ​​of the plurality of adjacent frames AF as the pixel value of the integration frame IF.

[0119] It should be noted that the integration processing method of the integration unit 15 of the present embodiment is not limited to the example of the case of obtaining a simple average. The integration unit 15 may also perform the integration processing by obtaining a weighted average, for example. The weighted average is obtained according to the temporal distance from the object frame TF. The integration unit 15 may also reduce the weight by multiplying the pixel values ​​of the frames at time t-2 and time t+2 which are temporally far from time t by 0.7 which is less than 1, and increase the weight by multiplying the pixel values ​​of the frames at time t-1 and time t+1 which are temporally close to time t by 1.3 which is greater than 1. That is, the integration unit 15 may also calculate the pixel value of the integration frame IF by calculating the weighted average corresponding to the temporal distance from the object frame TF in a plurality of adjacent frames AF. By integrating using the weighted average, the contribution of the adjacent frames AF to the object frame TF can be reflected in the integration frame IF. It should be noted that the size of the multiplied weight may be the same with respect to the temporal distance from the object frame TF, or may be different depending on whether it is before or after the object frame TF. For example, the weight may be increased when the adjacent frame AF is before the target frame TF, and the weight may be decreased when the adjacent frame AF is after the target frame TF.

[0120] The input conversion unit 12A converts the pixel value of the target frame TF obtained from the image acquisition unit 11 into the input data IND based on the comparison with the plurality of threshold values. Furthermore, the input conversion unit 12A converts the pixel value of the integration frame IF obtained from the integration unit 15 into the input data based on the comparison with the plurality of threshold values. The input conversion unit 12A outputs the converted input data IND to the synthesis unit 13A.

[0121] The combining unit 13A combines a plurality of input data IND converted based on the target frame TF and a plurality of input data IND converted based on the integration frame IF into one combined data CD. The amount of combined input information IN is smaller than that of the first embodiment.

[0122] [Summary of the Second Embodiment]

[0123] According to the above-described embodiment, the input information generating device 10A further includes an integration unit 15, which performs operations based on the pixel values ​​of the adjacent frames AF and integrates the plurality of adjacent frames AF into one integrated frame IF. In addition, the input conversion unit 12A converts the pixel values ​​of the object frame TF into input data IND based on comparison with a plurality of threshold values, and further converts the pixel values ​​of the integrated frame IF into input data IND based on comparison with a plurality of threshold values. In addition, the synthesis unit 13A synthesizes the plurality of input data IND converted based on the object frame TF and the plurality of input data IND converted based on the integrated frame IF into one synthesized data CD. That is, the input information generating device 10A does not quantize and vectorize the adjacent frames AF like the object frame TF, but obtains the integrated frame IF based on the plurality of adjacent frames AF, and performs quantization and vectorization on the integrated frame IF. Therefore, the input information IN synthesized by the synthesis unit 13A has less information than that of the first embodiment, and can reduce the computational load of the first layer.

[0124] In addition, according to the above-mentioned embodiment, the integration unit 15 uses the average value of the pixel values ​​of the plurality of adjacent frames AF as the pixel value of the integration frame IF. That is, the integration unit 15 uses the simple average of the adjacent frames AF as the integration frame IF. Therefore, according to this embodiment, it is possible to generate input information IN having less information than the input information IN of the first embodiment through easy calculation, and it is possible to reduce the calculation load of the first layer.

[0125] In addition, according to the above-mentioned embodiment, the integration unit 15 calculates the pixel value of the integration frame IF by calculating the weighted average corresponding to the time distance from the object frame TF in the plurality of adjacent frames AF. Therefore, according to this embodiment, the integration frame IF can be generated in consideration of the contribution of the adjacent frames AF to the object frame TF. Therefore, if the input information IN generated by the input information generating device 10A is used, the CNN 200 can perform image processing with higher accuracy.

[0126] [Third Embodiment]

[0127] Next, refer to Fig. 9 The third embodiment is described. First, the problem to be solved in the third embodiment is described. The input information generating device 10A of the second embodiment calculates the average value of adjacent frames AF by including the integration unit 15. When the object frame TF is shifted by one frame at a time, the calculation of the average value in each frame will cause duplication of processing, which is not efficient. Therefore, in the third embodiment, this problem is solved and the calculation load is further reduced.

[0128] Fig. 91 is a block diagram showing an example of a functional structure of input information generation according to the third embodiment. With reference to this figure, an example of a functional structure of an input information generation device 10B will be described. The input information generation device 10B is different from the input information generation device 10A in that it further includes an average value temporary storage unit 16, an imaging condition acquisition unit 17, and an adjustment unit 18, and includes an integration unit 15B instead of the integration unit 15. In the description of the input information generation device 10B, the same structure as the input information generation device 10A may be omitted by attaching the same reference numerals.

[0129] The average value temporary storage unit 16 stores the average value of the pixel values ​​of a predetermined frame among the frames constituting the dynamic image. The value stored in the average value temporary storage unit 16 is calculated by the integration unit 15B. The integration unit 15B obtains information about the target frame TF from the image acquisition unit 11 and obtains the storage value SV from the average value temporary storage unit 16. The integration unit 15B calculates the pixel value of the integration frame IF by calculation based on the target frame TF and the value stored in the average value temporary storage unit 16, that is, the storage value SV. The integration unit 15B stores the calculated value as the calculation value CV in the average value temporary storage unit 16. That is, every time the integration unit 15B performs calculation of a new target frame TF, the value stored in the average value temporary storage unit 16 is updated. By repeating such calculation, the input information generating device 10B calculates the moving average based on the target frame TF. It should be noted that, regarding the calculation of the first frame, since the storage value SV does not yet exist in the average value temporary storage unit 16, the integration unit 15B can also perform calculation based only on the target frame TF.

[0130] The imaging condition acquisition unit 17 acquires the imaging condition of the moving image from the imaging device 100. The imaging condition of the moving image acquired by the imaging condition acquisition unit 17 may be, for example, the settings of the imaging device such as shutter speed, aperture, or ISO sensitivity. In addition, the imaging condition of the moving image acquired by the imaging condition acquisition unit 17 may also include other information related to the operation and driving of the imaging device 100.

[0131] The adjustment unit 18 adjusts the value (average value) stored in the average value temporary storage unit 16 according to the imaging condition acquired by the imaging condition acquisition unit 17. Here, when the imaging condition of the imaging device 100 is changed, the relationship between the pixel value of the object frame TF and the past moving average changes. For example, when the ISO sensitivity is changed to 2 times by changing the setting of the imaging device 100, the pixel value of the object frame TF is brighter than the past moving average, so the moving average suddenly becomes darker. Therefore, when the ISO sensitivity is changed to 2 times by changing the setting of the imaging device 100, by setting the value stored in the average value temporary storage unit 16 to 2 times, the integration unit 15B can continue to use the value stored in the average value temporary storage unit 16 and calculate the moving average.

[0132] In addition, when the shooting scene of the dynamic image is switched, if the moving average is calculated between the past frames, it is sometimes impossible to properly perform image processing related to the object frame TF by CNN200. Therefore, when the shooting scene of the dynamic image is switched, it is preferable not to calculate the moving average with the past frames. The adjustment unit 18 can also be configured to reset the value stored in the average temporary storage unit 16 when the shooting scene of the dynamic image is switched, and start the integration unit 15B to recalculate the moving average. It is also possible to determine whether the shooting scene of the dynamic image has changed based on the on / off of the power button of the camera device 100, the on / off of the shooting button or the stop button, etc.

[0133] It should be noted that sometimes frames with large brightness changes are inserted into multiple frames included in a dynamic image. As the main reason for inserting frames with large brightness changes, the situation where a light source is captured due to a change in the shooting angle, the situation where the headlights of a car are reflected, etc. are considered. In such a case, the integration unit 15B may also exclude frames with large brightness changes from the calculation object of the average value among the multiple frames. By excluding frames with large brightness changes from the calculation object of the average value, it is possible to suppress the moving average from being dragged by frames with large brightness changes. As an example of determining whether the brightness change is large, it is also possible to determine whether the difference is below a threshold by comparing the pixel value of the immediately preceding object frame TF with the pixel value of the object frame TF as the calculation object.

[0134] [Summary of the third embodiment]

[0135] According to the embodiment described above, the input information generating device 10B further includes an average value temporary storage unit 16, which stores the average value of the pixel values ​​of predetermined frames among the frames constituting the dynamic image. In addition, according to the input information generating device 10B, the integration unit 15B calculates the pixel value of the integration frame IF by performing an operation based on the value stored in the average value temporary storage unit 16 and the object frame TF. That is, the input information generating device 10B calculates the pixel value of the integration frame IF based on the object frame TF and the stored moving average. Therefore, the calculation load of the input information generating device 10B is lighter than that of the input information generating device 10A. Therefore, according to this embodiment, the calculation load can be further reduced.

[0136] In addition, according to the above-mentioned embodiment, the integration unit 15B excludes the frame with a large brightness change from the calculation object of the average value among the multiple frames constituting the dynamic image. Therefore, according to this embodiment, when the brightness change suddenly increases, by excluding the frame from the calculation of the moving average, it is possible to suppress the pixel value of the integrated frame IF from being dragged by the pixel value of the frame with a sudden brightness change.

[0137] In addition, according to the above-mentioned embodiment, the input information generating device 10B further includes the imaging condition acquiring unit 17 to acquire the imaging condition of the dynamic image, and further includes the adjusting unit 18 to adjust the average value stored in the average value temporary storage unit 16 according to the acquired imaging condition. Therefore, according to this embodiment, the average value can be adjusted according to the change of the imaging condition. Therefore, according to this embodiment, even if the imaging condition changes, the moving average can be continuously calculated.

[0138] [Fourth embodiment]

[0139] Next, refer to Fig.10 The fourth embodiment is described. First, the problem to be solved in the fourth embodiment is described. The input information generating device 10B of the third embodiment calculates the moving average by having an average value temporary storage unit 16. According to the third embodiment, the integration unit 15B calculates the moving average of the entire image. The input information generating device 10B generates input data IN based on the object frame TF and the moving average, and the CNN 200 performs noise removal on the dynamic image based on the generated input data IN. Since the integration unit 15B calculates the moving average of the entire image, when a moving subject is captured in a part of the dynamic image, the result of the noise removal sometimes produces a problem of an afterimage in a part of the captured moving subject. This problem is to be solved in the fourth embodiment.

[0140] Fig.10 It is a block diagram showing an example of the functional structure of the input information generation of the fourth embodiment. With reference to this figure, an example of the functional structure of the input information generating device 10C of the fourth embodiment is described. The difference between the input information generating device 10C and the input information generating device 10B is that it also includes a comparison unit 19, and an integration unit 15C is provided instead of the integration unit 15B. The input information generating device 10C may also include a camera condition acquisition unit 17 and an adjustment unit 18 like the input information generating device 10B, but it may not include them. In the example shown in the figure, an example of a case where the input information generating device 10C does not include the camera condition acquisition unit 17 and the adjustment unit 18 is described. In the description of the input information generating device 10C, the description of the same structure as the input information generating device 10B is sometimes omitted by marking the same figure mark.

[0141] The comparison unit 19 acquires the storage value SV from the average value temporary storage unit 16 and acquires the object frame TF from the integration unit 15C. The comparison unit 19 compares the storage value SV acquired and stored in the average value temporary storage unit 16 with the pixel value of the object frame TF. The comparison unit 19 may compare the entire image, each pixel, or each patch composed of a plurality of pixels. The comparison unit 19 outputs the comparison result as a comparison result CR to the integration unit 15C. The comparison result CR may include the difference in pixel values ​​or information on the result of comparing the difference with a predetermined threshold value.

[0142] The integration unit 15C obtains the comparison result CR from the comparison unit 19 and obtains the storage value SV from the average value temporary storage unit 16. The integration unit 15C calculates a moving average based on the storage value SV stored in the average value temporary storage unit 16 and the average value of the object frame TF, based on the comparison result CR, which is the result of the comparison by the comparison unit 19, when the difference is less than a predetermined value. The integration unit 15C uses the calculated value as the pixel value of the integration frame IF. In addition, the integration unit 15C uses the pixel value of the object frame TF as the pixel value of the integration frame IF, based on the comparison result CR, which is the result of the comparison by the comparison unit 19, when the difference is not less than a predetermined value.

[0143] The integration process of the integration unit 15C can be performed on the entire image, on each pixel, or on each patch composed of multiple pixels. When the integration process is performed on each pixel or each patch, the pixel value of the integrated frame IF becomes a moving average at the part where the difference is less than the predetermined value (i.e., the part with small movement), and becomes the pixel value of the object frame TF at the part where the difference is not less than the predetermined value (i.e., the part with large movement). The integration unit 15C stores the result of the calculation as the calculation value CV in the average value temporary storage unit 16. According to the input information generating device 10C, the subject with large movement is not taken into the average image, and the background with small movement is taken into the average image. The calculation of the input information generating device 10C can also be called selective averaging.

[0144] It should be noted that, as a modified example of the processing performed by the integration unit 15C, a coefficient may be multiplied instead of the choice of taking in or not taking in the average image. For example, the integration unit 15C may calculate, based on the comparison result CR as the result of the comparison by the comparison unit 19, a moving average of a value obtained by multiplying the storage value SV stored in the average temporary storage unit 16 by a predetermined coefficient (for example, 0.9) and the average value of the object frame TF as the pixel value of the integration frame IF when the difference is less than a predetermined value. In addition, the integration unit 15C may calculate, based on the comparison result CR as the result of the comparison by the comparison unit 19, a moving average of a value obtained by multiplying the storage value SV stored in the average temporary storage unit 16 by a predetermined coefficient (for example, 0.1) and the average value of the object frame TF as the pixel value of the integration frame IF when the difference is not less than a predetermined value.

[0145] [Summary of the Fourth Embodiment]

[0146] According to the embodiment described above, the input information generating device 10C further includes a comparison unit 19, which compares the storage value SV stored in the average value temporary storage unit 16 with the pixel value of the object frame TF. When the result of the comparison by the comparison unit 19 is that the difference is less than the predetermined value, the integration unit 15C calculates the pixel value of the integration frame IF by calculating the moving average based on the storage value SV stored in the average value temporary storage unit 16 and the object frame TF. In addition, when the result of the comparison by the comparison unit 19 is that the difference is not less than the predetermined value, the integration unit 15C uses the pixel value of the object frame TF as the pixel value of the integration frame IF. That is, according to the input information generating device 10C, the pixel value of the integration frame IF is determined by selectively performing the averaging process by distinguishing between the subject with large motion and the background with small motion. The input information generating device 10C generates input data IN based on the object frame TF and the integration frame IF. Therefore, according to this embodiment, for the part with large motion, the pixel value of the adjacent frame AF is not reflected in the pixel value of the integration frame IF, so that the problem of afterimage can be suppressed.

[0147] It should be noted that, in the above-mentioned embodiments 2 to 4, a predetermined operation is required to determine the pixel value of the integration frame IF, and the processing may become heavy. In order to further reduce the processing weight, any one of the frames adjacent to the object frame TF (for example, the two frames before and after) may be determined by a predetermined algorithm as the integration frame IF. The predetermined algorithm may also be an algorithm that randomly determines any one of the frames adjacent to the object frame TF. In this case, the integration unit 15 uses the randomly determined frame among the adjacent frames AF acquired by the image acquisition unit 11 as the integration frame IF.

[0148] (Other embodiments)

[0149] It should be noted that in the above-mentioned embodiment, an example is shown in which the synthesis unit 13 and the integration unit 15 perform a process of synthesizing and integrating the acquired multiple frames. However, the present embodiment is not limited to this example, and the object of synthesis or integration is not limited to the case of multiple frames themselves. As other embodiments, the intermediate output in at least a part of CNN200 can also be used as an object for synthesis processing and integration processing. In more detail, the intermediate output generated when CNN200 processes the object frame TF and the adjacent frame AF can also be used as the object of synthesis or integration. In addition, the intermediate output generated when CNN200 processes the adjacent frame AF and the result of vectorizing the object frame TF can also be used as the object of synthesis or integration. It should be noted that the present embodiment is not limited to the above-mentioned example, and the object of synthesis or integration widely includes all information obtained as a result of frame-based processing.

[0150] It should be noted that the method of this embodiment is not limited to any of the methods in the first to fourth embodiments described above, and any of the methods in the first to fourth embodiments may be selectively used based on predetermined conditions. The predetermined conditions may also be shooting conditions, shooting modes, exposure conditions, types of subjects, etc. of dynamic images.

[0151] It should be noted that, so far, an example of performing calculations using adjacent frames AF in order to improve the quality of the object frame TF has been shown, but in the case of learning a learned model included in CNN200, it is also preferable to use not only the object frame TF but also the adjacent frames AF for learning, as in any of the first to fourth embodiments. The calculations related to learning do not necessarily need to be performed in the image processing device 2, and the results of parameters and the like learned in advance in a dedicated learning device may be included in CNN200 as a learned model.

[0152] As mentioned above, although the form for implementing this invention was demonstrated using embodiment, this invention is not limited to such embodiment at all, Various deformation|transformation and substitution can be added in the range which does not deviate from the summary of this invention.

[0153] Next, refer to Figures 11 to 20, regarding the learning device, program and learning method of the noise reduction device of the method of the present invention, a preferred embodiment is disclosed, and a detailed description is given with reference to the attached drawings. It should be noted that the method of the present invention is not limited to these embodiments, and also includes methods in which various changes or improvements are applied. That is, the constituent elements described below include elements that can be easily thought of by a person skilled in the art and substantially the same elements, and the constituent elements described below can be appropriately combined. In addition, various omissions, substitutions or changes of constituent elements can be made without departing from the gist of the present invention. In addition, in the following drawings, in order to facilitate the understanding of each structure, the scale and quantity in each structure are sometimes different from the scale and quantity in the actual structure.

[0154] First, the background technology of the present invention and the problems to be solved by the present invention are described.

[0155] In the past, there was a technology that used machine learning to process low-quality images into high-quality images. In such a technical field, a combination of a noisy image superimposed with noise and a high-quality image is used as training data to enable the learning model to learn. The production of training data is performed by obtaining high-quality images and noisy images by photographing the same object with different exposure settings by a camera. Generally, it is known that a large amount of training data is required for machine learning, and there is a problem that it is troublesome to use a camera to produce training data by shooting. Therefore, it is known that a technology for producing training data by adding random noise to high-quality images (for example, refer to Japanese Patent Gazette No. 2021-071936). It is known that using such prior art, training data for inferring high-quality images from low-quality images is produced by adding random noise to high-quality images.

[0156] Here, it is known that in the case of processing a low-quality moving image into a high-quality moving image, a large amount of training data for machine learning is required, as in the case of the above-mentioned static image. However, in the case of a moving image, it is very difficult to easily obtain a high-quality moving image and a low-quality moving image of the same subject by shooting the same object with different settings. Therefore, it is also considered to apply the above-mentioned prior art to superimpose noise on each frame of a high-quality moving image shot in advance to generate a low-quality moving image, but there is a problem of huge capacity, etc., which is very difficult.

[0157] Therefore, an object of the present invention is to provide a technology capable of generating training data for inferring high-quality dynamic images from low-quality dynamic images.

[0158] Next, the matters that are the premise of this embodiment are described. The learning device, program, and learning method of the noise reduction device of this embodiment take low-quality dynamic image information superimposed with noise as input, and make the learning model learn to infer high-quality dynamic images with noise removed. Low-quality dynamic images include low-quality dynamic images, and high-quality dynamic images include high-quality dynamic images. In the learning device, program, and learning method of the noise reduction device of this embodiment, training data used for learning is generated based on static images obtained by shooting a subject. The static image of the subject can be either a high-quality image or a plurality of images of the same subject (a combination of one or more high-quality images and one or more low-quality images). The multiple images obtained by shooting the same subject can also be shot under different shooting conditions. In addition, the image obtained by shooting the subject can also be another image containing at least one image. As an example, a high-quality image can be exemplified by an image with high image quality shot with low ISO sensitivity and long exposure. In the following description, a high-quality image is sometimes recorded as GT (Ground Truth). As an example, the low-quality image may be an image with low image quality captured with a high ISO sensitivity and a short exposure time.

[0159] In the following description, the image quality degradation caused by noise is described as an example of a low-quality image, but even if it is not noise, the present embodiment can be widely applied to matters that reduce the quality of an image. Matters that reduce the quality of an image include, for example, a reduction in resolution or color deviation caused by optical aberration, a reduction in resolution caused by hand shaking or subject shaking, uneven black level caused by dark current or circuit reasons, ghosting caused by a high-brightness subject, flare, abnormal signal level, etc.

[0160] It should be noted that in the generation of training data, pre-prepared images can also be used. In the following description, low-quality images are sometimes recorded as low-quality images or noisy images. In addition, in the following description, high-quality images are sometimes recorded as high-quality images or GT. Similarly, low-quality dynamic images are sometimes recorded as low-quality dynamic images or noisy dynamic images. In addition, in the following description, high-quality dynamic images are sometimes recorded as high-quality dynamic images or GT.

[0161] The image used as the object of the learning device of this embodiment may be a still image or a frame included in a moving image. In addition, as a data format, it may be a format that is not subjected to compression coding, such as the Raw format, or a format that is subjected to compression coding, such as the JPEG format or the MPEG format. In the following, unless otherwise specified, the case where the image is a still image based on the Raw format is described.

[0162] In addition, the image of the object of the learning device of this embodiment may be an image captured by a CCD camera using a CCD (Charge Coupled Devices) image sensor. In addition, the image of the object of the learning device of this embodiment may be an image captured by a CMOS (Complementary Metal Oxide Semiconductor) camera using a CMOS image sensor. Furthermore, the image of the object of the learning device of this embodiment may be either a color image or a monochrome image. Furthermore, the image of the object of the learning device of this embodiment may be an image captured by acquiring invisible light components using an infrared camera or the like using an infrared sensor.

[0163] [Fifth embodiment]

[0164] First, refer to Figures 11 to 18 A fifth embodiment will be described.

[0165] Fig.11 1001 is a diagram for explaining the outline of the learning system of the fifth embodiment. Referring to this figure, the outline of the learning system 1001 is explained. The learning system 1001 shown in this figure is an example of a structure in the learning stage of machine learning. The learning system 1001 uses the training data TD generated based on the image captured by the camera device 1020 to make the learning model 1040 learn.

[0166] The learning system 1001 is provided with a camera 1020 to capture a high-quality image 1031 and a low-quality image 1032. The high-quality image 1031 and the low-quality image 1032 are images of the same subject. For example, the high-quality image 1031 and the low-quality image 1032 are captured with the same viewing angle and shooting angle and with different settings for ISO sensitivity, exposure time, etc. In addition, the high-quality image 1031 is preferably one image, but may be multiple images. In addition, the low-quality image 1032 is preferably multiple images, but may be one image. The multiple low-quality images 1032 are preferably different images captured with different settings for ISO sensitivity, exposure time, etc. The camera 1020 may be, for example, a smartphone or a tablet terminal having a communication component. In addition, the camera 1020 may be a surveillance camera having a communication component.

[0167] The learning system 1001 generates a high-quality moving image 1033 from a high-quality image 1031, and generates a low-quality moving image 1034 from a low-quality image 1032. The high-quality moving image 1033 is preferably generated from one high-quality image 1031, and the low-quality moving image 1034 is preferably generated from a plurality of low-quality images 1032. The high-quality moving image 1033 and the low-quality moving image 1034 generated from the high-quality image 1031 and the low-quality image 1032 obtained by photographing the same subject are associated with each other. The high-quality moving image 1033 and the low-quality moving image 1034 that correspond to each other are input as training data TD to the learning model 1040 for learning.

[0168] It should be noted that the high-quality dynamic images 1033 and the low-quality dynamic images 1034 corresponding to each other may also be temporarily stored in a predetermined storage device for subsequent learning. That is, the learning system 1001 may also generate a plurality of training data TD in advance before the subsequent learning. In addition, the high-quality images 1031 and the low-quality images 1032 captured by the camera device 1020 may also be temporarily stored in a predetermined storage device. In this case, the learning system 1001 may also store a plurality of combinations of high-quality images 1031 and low-quality images 1032 corresponding to each other, and generate training data TD during learning.

[0169] The learning model 1040 is learned using the training data TD generated by the learning system 1001. Specifically, the learning model 1040 is learned in a manner of inferring a high-quality dynamic image from a low-quality dynamic image. In other words, the learned learning model 1040 uses the low-quality dynamic image as input to infer a high-quality dynamic image and outputs the inferred result. That is, the learned learning model 1040 can also be used in a noise reduction device that removes noise from a low-quality dynamic image.

[0170] It should be noted that the high-quality image 1031 and the low-quality image 1032 captured by the camera 1020 are stored in a predetermined storage device for temporarily storing information. The predetermined storage device can be set in the camera 1020 or in a cloud server. In other words, the learning system 1001 can be configured as an edge device or can be configured to include an edge device and a cloud server. In addition, in the learning of the learning model 1040, a GPU set on a server can also be used.

[0171] Fig.121 is a diagram showing an example of the functional structure of the learning device of the fifth embodiment. The functional structure of the learning device 1010 will be described with reference to this figure. The learning device 1010 is used to implement the above-mentioned learning system 1001. The learning device 1010 generates a high-quality dynamic image 1033 and a low-quality dynamic image 1034 based on the high-quality image 1031 and the low-quality image 1032 captured by the camera 1020. The learning device 1010 uses the generated high-quality dynamic image 1033 and the low-quality dynamic image 1034 as training data TD to make the learning model 1040 learn. The learning device 1010 includes an image acquisition unit 1011, a dynamic image information generation unit 1012, and a learning unit 1013. The learning device 1010 includes a CPU (Central Processing Unit) not shown in the figure, a storage device such as a ROM (Read only memory) or a RAM (Random access memory), etc. connected by a bus. The learning device 1010 functions as a device including an image acquisition unit 1011 , a moving image information generation unit 1012 , and a learning unit 1013 by executing a learning program.

[0172] It should be noted that all or part of the functions of the learning device 1010 may also be implemented using hardware such as ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device) or FPGA (Field-Programmable Gate Array). The learning program may also be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, removable media such as floppy disks, optical magnetic disks, ROMs, CD-ROMs, and storage devices such as hard disks built into computer systems. The learning program may also be sent via an electrical communication line.

[0173] The image acquisition unit 1011 acquires image information I from the camera device 1020. The image information I includes first image information I1 and second image information I2. The first image information I1 includes at least one high-quality image 1031. The second image information I2 includes at least one low-quality image 1032. The low-quality image 1032 included in the second image information I2 captures the same subject as the subject captured in the high-quality image 1031 included in the first image information I1. The image quality of the image included in the second image information I2 is lower than the image quality of the image included in the first image information I1. The image acquisition unit 1011 outputs the acquired image information I to the dynamic image information generation unit 1012.

[0174] The dynamic image information generating unit 1012 cuts out a part of the image included in the image information I into a plurality of images, and connects the cut images as frame images at a predetermined time interval (or can also be called a frame rate), thereby generating dynamic image information M. The frame rate may be, for example, 60 [FPS (frames per second)]. The position of the image cut out by the dynamic image information generating unit 1012 may also be different for each frame. For example, the size of the cut out image may be fixed, and the dynamic image information generating unit 1012 cuts out a plurality of images at positions that are moved by a predetermined number of pixels (bits) in a predetermined direction each time. Specifically, the size of the cut out image may also be fixed to 256 pixels × 256 pixels. In addition, the dynamic image information generating unit 1012 may also cut out images at positions that are shifted by 10 pixels per frame with respect to the size. If the amount of shift is too large, the amount of change in the image per frame becomes too large, resulting in an unnatural dynamic image, so it is preferable to set a limit (upper limit value) to avoid shifting beyond the predetermined amount. The amount of offset and its limitation are preferably determined based on the shooting angle of view, shooting resolution, focal length of the optical system, distance to the subject, shooting frame rate, etc. In addition, in the case of a falling subject, since the speed increases at an accelerated rate, the amount of offset may be increased as the frame is temporally distant from the target image.

[0175] The dynamic image information generating unit 1012 generates the first dynamic image information M1 based on the image included in the first image information I1, and generates the second dynamic image information M2 based on the image included in the second image information I2. That is, the dynamic image information generating unit 1012 cuts out a plurality of images that are parts of the first image information I1 and at different positions, and combines the cut-out plurality of images to generate the first dynamic image information M1. In addition, the dynamic image information generating unit 1012 cuts out a plurality of images that are parts of the second image information I2 and at different positions, and combines the cut-out plurality of images to generate the second dynamic image information M2. Generating a dynamic image by combining a plurality of images may also be converting a plurality of images into a file format that is displayed at a predetermined time interval corresponding to a frame rate. The dynamic image information generating unit 1012 outputs information including the generated first dynamic image information M1 and second dynamic image information M2 as dynamic image information M to the learning unit 1013.

[0176] Here, the sizes and positions of the multiple images cut out by the dynamic image information generating unit 1012 can also be determined arbitrarily. However, the positions of the images cut out from the first image information I1 and the positions of the images cut out from the second image information I2 are preferably substantially the same positions. This is because the first dynamic image information M1 as a high-quality dynamic image and the second dynamic image information M2 as a low-quality dynamic image should capture the same subject.

[0177] The learning unit 1013 acquires the dynamic image information M from the dynamic image information generating unit 1012. The learning unit 1013 inputs the acquired dynamic image information M as training data TD to the learning model 1040, thereby causing the learning model 1040 to learn. The learning model 1040 learns in a manner of inferring a high-quality dynamic image from a low-quality dynamic image. That is, the learning unit 1013 learns in a manner of inferring a high-quality dynamic image from a low-quality dynamic image based on the training data TD included in the first dynamic image information M1 and the second dynamic image information M2 generated by the dynamic image information generating unit 1012. It can also be said that the learning model 1040 learns in a manner of inferring a high-quality dynamic image from a low-quality dynamic image.

[0178] Next, refer to Figures 13 to 15 , the image cut out by the learning device 1010 from the image captured by the camera device 1020 is described. It should be noted that in the following description, a method for generating a high-quality dynamic image from a high-quality image is described (refer to Fig.13 Method described in the description) and method for generating low-quality dynamic images from low-quality images (refer to Fig.14 The present embodiment is not limited to this example. Instead of the following description, the same method can be used to generate a high-quality dynamic image from a high-quality image and a low-quality dynamic image from a low-quality image. Fig.13 The method described generates low-quality dynamic images, which can also be obtained by referring to Fig.14 The described method generates high-quality dynamic images.

[0179] Fig.13 1010 is a diagram for explaining an example of the position of an image cut out from a high-quality image by the learning device of the fifth embodiment. Referring to this diagram, an example of the position of an image cut out from a high-quality image by the learning device 1010 is explained. Fig.13 (A) shows an image I-11 as an example of an image included in the first image information I1. Fig.13 In (B), the Fig.13 An example of a case where a plurality of images are cut out from the image I-11 shown in (A) is shown as image I-12.

[0180] like Fig.13 As shown in (A), the sphere B as the subject is captured in the image I-11. The dynamic image information generating unit 1012 cuts out a plurality of images from the image I-11 and temporally connects the cut out images, thereby generating a dynamic image from the image I-11 which is a static image.

[0181] exist Fig.13The image I-12 shown in (B) shows a plurality of images cut out by the dynamic image information generating unit 1012, namely, cut out images CI. Specifically, cut out images CI-11 to CI-15 are shown as examples of images cut out by the dynamic image information generating unit 1012. When the cut out images CI-11 to CI-15 are not distinguished, they may be simply described as cut out images CI.

[0182] Cut-out images CI-11 to CI-15 are shifted by a predetermined number of pixels in the vertical and horizontal directions, respectively. According to the first dynamic image information M1 generated by the dynamic image information generation unit 1012, image C-11 is projected at a certain time t1, image C-12 is projected at a certain time t2, image C-13 is projected at a certain time t3, image C-14 is projected at a certain time t4, and image C-15 is projected at a certain time t5. In this way, by connecting different cut-out images CI in time, a dynamic image can be generated in which the subject, that is, the ball B in the static image, seems to be moving. In the case where the dynamic image information generation unit 1012 generates a dynamic image with a frame rate of 60 [fps], the interval between each time may also be 1 / 60 seconds.

[0183] The shift direction and shift amount of the image cut out by the dynamic image information generating unit 1012 are preferably determined based on shooting conditions such as shooting angle of view, shooting resolution, focal length of the optical system, distance to the subject, shooting frame rate, etc. In addition, in the case of a subject simulating a falling object, since the speed increases at an accelerated rate, it is preferable to gradually change (increase) the shift amount.

[0184] Here, the high-quality dynamic image (first dynamic image information M1) generated by the learning device 1010 is a high-quality dynamic image without superimposed noise. Therefore, it is ideal that noise is not superimposed in the image that is the static image used to generate the dynamic image. In addition, it is ideal that noise is not superimposed in each frame of the high-quality dynamic image generated from the image without superimposed noise. Therefore, the dynamic image information generation unit 1012 preferably generates the dynamic image from one image without superimposed noise. That is, the dynamic image information generation unit 1012 preferably generates the first dynamic image information M1 by cutting out different parts from one high-quality image included in the first image information I1.

[0185] Fig.14 1 is a diagram for explaining an example of the position of an image cut out from a low-quality image by the learning device of the fifth embodiment. Referring to this diagram, an example of the position of an image cut out from a low-quality image by the learning device 1010 is explained. The learning device 1010 cuts out images of different frames from a plurality of low-quality images. Fig.14 (A) to Fig.14In (E), images I-21 to I-25 are shown as different images. The learning device 1010 cuts out images of different frames from images I-21 to I-25.

[0186] The compositions of images I-21 to I-25 as low-quality images are similar to Fig.13 The image I-11 shown in (A) is the same. That is, in the images I-21 to I-25, the ball B is photographed at the same position. The images I-21 to I-25 are different from the image I-11 in that different noises are superimposed. The images I-21 to I-25 may be superimposed with different noises, for example, by using different shooting conditions during shooting.

[0187] The dynamic image information generating unit 1012 cuts out a cut image CI-21 from the image I-21, cuts out a cut image CI-22 from the image I-22, cuts out a cut image CI-23 from the image I-23, cuts out a cut image CI-24 from the image I-24, and cuts out a cut image CI-25 from the image I-25. The cut images CI-21 to CI-25 are shifted by a predetermined number of pixels in the longitudinal and lateral directions, respectively. According to the second dynamic image information M2 generated by the dynamic image information generating unit 1012, the image C-21 is projected at a certain time t1, the image C-22 is projected at a certain time t2, the image C-23 is projected at a certain time t3, the image C-24 is projected at a certain time t4, and the image C-25 is projected at a certain time t5. Since different noises are superimposed on the cut images CI-21 to CI-25, different noises are superimposed on the generated dynamic image according to time.

[0188] Here, the low-quality dynamic image (second dynamic image information M2) generated by the learning device 1010 is a low-quality dynamic image superimposed with noise. When multiple different positions are cut out from one image superimposed with noise as a dynamic image, since the same noise is contained at any moment (in other words, since the noise does not change with time), it is sometimes not suitable as a low-quality dynamic image. Therefore, in this embodiment, a low-quality dynamic image is generated by cutting out from different multiple low-quality images. In each of the different multiple low-quality images, the same subject as the subject captured in the high-quality image is captured. That is, the second image information M2 contains multiple images, which are images obtained by capturing the same subject as the subject captured in the image included in the first image information I1, and each of which is superimposed with different noises. The multiple images included in the second image information I2 may also be images captured at different times close to each other. The dynamic image information generation unit 1012 generates the second dynamic image information M2 by cutting out different parts from each of the multiple images included in the second image information.

[0189] It should be noted that, for example, it is not necessary to prepare low-quality images corresponding to the number of frames, and multiple cuts may be made from multiple images discontinuously. The order of cutting out from multiple images may also be random.

[0190] Fig.15 FIG. 5 is a diagram for explaining an example of the direction in which the learning device of the fifth embodiment cuts out. Fig.13 and Fig.14 In the example described above, an example of cutting out a position moved by a predetermined number of pixels in both the vertical and horizontal directions is described. However, the dynamic image information generating unit 1012 may also cut out a position moved in other directions. Fig.15 (A) to Fig.15 (C) will now be described another example of the direction in which the dynamic image information generating unit 1012 cuts out the cut-out image CI.

[0191] exist Fig.15 (A) shows image I-31. Fig.15 (A) is an example of a case where a position that moves only in the lateral direction (horizontal direction) is cut out. In this case, the dynamic image information generation unit 1012 fixes the longitudinal y coordinate and changes only the lateral x coordinate, thereby cutting out a plurality of cut-out images CI at different positions. By cutting out in this way, a dynamic image in which the subject moves in the lateral direction (horizontal direction) can be generated. Similarly, the dynamic image information generation unit 1012 can also cut out the cut-out images CI at the position that moves only in the longitudinal direction (vertical direction). By cutting out in this way, a dynamic image in which the subject moves in the longitudinal direction (vertical direction) can be generated.

[0192] In addition, if Fig.13 and Fig.14 As shown in FIG. 1 , the dynamic image information generating unit 1012 may cut out the cutout image CI at a position moved in both the vertical and horizontal directions. In this case, the vertical movement amount and the horizontal movement amount may be different from each other.

[0193] exist Fig.15 (B) shows image I-32. Fig.15 (B) is an example of a case where a position moving in the rotation direction is cut out. In this case, the dynamic image information generation unit 1012 cuts out a plurality of cut-out images CI at different positions by moving the cut-out position in an arc shape having a rotation center O and a radius r. In the example shown in the figure, the dynamic image information generation unit 1012 cuts out a position rotating counterclockwise. By cutting out in this way, a dynamic image of a subject moving in the rotation direction can be generated. The position of the rotation center O and the size of the radius r may also be different for each frame.

[0194] exist Fig.15 (C) shows image I-33. Fig.15 (C) is an example of a case where the cut-out position is enlarged or reduced. In the present embodiment, the size of the cut-out image CI is preferably constant. Therefore, the dynamic image information generation unit 1012 cuts out the image I by enlarging or reducing the image I while maintaining the size of the cut-out image CI. When the size of the cut-out image CI is fixed to 256 pixels × 256 pixels, the dynamic image information generation unit 1012 enlarges or reduces the image I in a manner that converges to the size of the cut-out image CI. By cutting out in this way, a dynamic image in which the subject is enlarged or reduced can be generated.

[0195] It should be noted that, refer to Fig.15 (A) to Fig.15 The cut-out position described in (C) is an example of the present embodiment, and the dynamic image information generating unit 1012 may also generate dynamic image information by cutting out other different positions and connecting them. The dynamic image information generating unit 1012 may also generate dynamic image information by combining reference Fig.15 (A) to Fig.15 The cutout image CI is cutout by the cutout method described in (C). In this case, for example, a moving image can be generated which is rotated after horizontal or vertical movement, or which is enlarged or reduced after movement.

[0196] It should be noted that the above-mentioned movement of the cutout position can also be calculated by affine transformation. In other words, the predetermined direction in which the moving image information generating unit 1012 cuts out the image can also be described as being calculated by affine transformation.

[0197] It should be noted that the dynamic image information generation unit 1012 may generate a dynamic image by cutting out a part of an image and moving it instead of the above-mentioned example of changing the cut-out position. In this case, the dynamic image information generation unit 1012 cuts out a 256-pixel × 256-pixel image and generates a plurality of pixels by moving the cut-out image in a predetermined direction. The dynamic image information generation unit 1012 generates a dynamic image by connecting the cut-out images. That is, the dynamic image information generation unit 1012 may cut out a plurality of images at different positions by staggering the cut-out images in a predetermined direction.

[0198] It should be noted that by moving the image after cutting it out, an area without data will be generated around the image. However, by pre-defining the surrounding part of the image as a pasting area, it can be excluded from the range of the image to be learned, thereby avoiding problems in the subsequent learning stage.

[0199] In the above description, an example of a case where the dynamic image information generation unit 1012 generates a dynamic image by cutting out an image moving in a direction calculated by a certain method such as affine transformation. However, in actual dynamic images, the subject often does not move in these calculated directions, but often moves randomly. Therefore, the learning device 1010 generates a dynamic image by cutting out an image moving in a direction based on the actual movement trajectory of the object, and can generate training data that is more effective for machine learning. For an example of such a case, refer to Fig.16 and Fig.17 This will be described as a modified example of the fifth embodiment.

[0200] Here, in bright scenes such as on sunny days, the shutter speed is usually increased to continue to maintain exposure. Therefore, the smoothness of the known moving subject disappears, resulting in a harsh image. Similarly, when generating a dynamic image from a static image with a high sense of resolution, it sometimes becomes a harsh and unnatural dynamic image with less smoothness. Therefore, the dynamic image information generation unit 1012 may also generate a dynamic image after performing additional simulated subject jitter correction on the static image used to produce the dynamic image. As an example, subject jitter may also be added by performing a predetermined averaging process on the shift direction or performing a process of reducing the resolution.

[0201] Fig.16This is a diagram showing an example of the functional structure of a learning device in a case where a learning device of a modification of the fifth embodiment generates a dynamic image based on a trajectory vector. With reference to this diagram, an example of the functional structure of a learning device 1010A of a modification of the fifth embodiment is described. A learning system 1001A of a modification of the fifth embodiment is different from the learning system 1001 in that it further includes a trajectory vector generating device 1050. The learning device 1010A is different from the learning device 1010 in that it further includes a trajectory vector acquiring unit 1014. In addition, the learning device 1010A is different from the learning device 1010 in that it includes a dynamic image information generating unit 1012A instead of the dynamic image information generating unit 1012. In the description of the learning device 1010A, the description of the same structure as that of the learning device 1010 is sometimes omitted by attaching the same reference numerals.

[0202] The trajectory vector generating device 1050 acquires information related to the trajectory of the object captured in the dynamic image. The dynamic image information is input to the trajectory vector generating device 1050, and the trajectory vector generating device 1050 analyzes the trajectory of the object captured in the input dynamic image information. The trajectory vector generating device 1050 outputs the analysis result as a trajectory vector TV. The trajectory vector TV represents the trajectory of the object captured in the dynamic image information. The trajectory vector generating device 1050 acquires the trajectory vector TV from the dynamic image information using existing technologies such as optical flow.

[0203] It should be noted that the trajectory vector TV may include coordinate information indicating the trajectory of the object's movement in addition to or instead of the vector information.

[0204] The trajectory vector acquisition unit 1014 acquires the trajectory vector TV from the trajectory vector generation device 1050. The trajectory vector acquisition unit 1014 outputs the acquired trajectory vector TV to the dynamic image information generation unit 1012A. It should be noted that the dynamic image from which the trajectory vector TV is acquired by the trajectory vector generation device 1050 and the image acquired by the image acquisition unit 1011 may also have a predetermined correlation. In this case, for example, the image acquisition unit 1011 may also acquire one frame of the dynamic image from which the trajectory vector TV is acquired by the trajectory vector generation device 1050 as an image.

[0205] However, the present embodiment is not limited to this example, and the dynamic image for which the trajectory vector TV is acquired by the trajectory vector generating device 1050 and the dynamic image acquired by the image acquiring unit 1011 may not have a predetermined correlation.

[0206] The dynamic image information generating unit 1012A acquires the image information I from the image acquiring unit 1011 and acquires the trajectory vector TV from the trajectory vector acquiring unit 1014. The dynamic image information generating unit 1012A generates dynamic image information based on the acquired image information I and the trajectory vector TV. The dynamic image information generating unit 1012A determines the cutting direction of the cut image CI and the amount of shift per frame based on the trajectory indicated by the trajectory vector TV. That is, the predetermined direction of the cut image CI by the dynamic image information generating unit 1012A is calculated based on the acquired trajectory vector TV.

[0207] Fig.17 This is a diagram for explaining an example of the position of an image cut out from a static image when a learning device in a modified example of the fifth embodiment generates a dynamic image based on a trajectory vector. With reference to this diagram, an example of the position coordinates of a cut-out image CI when a dynamic image is generated based on a trajectory vector TV is explained. Fig.17 (A) shows an image I-41 as an example of an image included in the first image information I1. Fig.17 (B) shows an example of a plurality of cut-out images CI cut out from the image I-41.

[0208] like Fig.17 As shown in (A), the trajectory of the ball B as the subject, i.e., the trajectory vector TV, is shown in the image I-41. The trajectory vector TV represents the vector that the ball B falls from the upper right direction in the figure to the lower center direction, bounces up from the lower center, and then moves toward the upper left direction in the figure. The dynamic image information generation unit 1012A cuts out a cutout image CI based on the position coordinates of the trajectory vector TV shown in the image I-41, and connects the cutout images in time, thereby generating a dynamic image from the image I-41 as a static image.

[0209] exist Fig.17 (B) shows an example of an image cut out by the dynamic image information generating unit 1012, namely, a cut out image CI. Specifically, as an example of an image cut out by the dynamic image information generating unit 1012, cut out images CI-41 to CI-49 are shown. Cut out images CI-41 to CI-49 are located at coordinates based on the trajectory vector TV. That is, cut out image CI-41 is located in the upper right direction of the figure, and until cut out image CI-45, the cut out position moves toward the middle and lower direction in the figure. In addition, from cut out image CI-45 to cut out image CI-49, the cut out position moves toward the upper left direction in the figure.

[0210] Fig.18 1010 is a flowchart showing an example of a series of operations of the learning method of the noise reduction device according to the fifth embodiment. With reference to this figure, an example of a series of operations of the learning method of the noise reduction device using the learning device 1010 will be described.

[0211] (Step S110) First, the image acquisition unit 1011 acquires an image. The image acquisition unit 1011 acquires first image information I1 including a high-quality image and second image information I2 including a low-quality image. It should be noted that the step of acquiring an image by the image acquisition unit 1011 is sometimes described as an image acquisition step or an image acquisition process.

[0212] (Step S130) Next, the dynamic image information generation unit 1012 cuts out a part of the acquired image. The dynamic image information generation unit 1012 cuts out a plurality of cut images CI from the acquired image. The dynamic image information generation unit 1012 cuts out a plurality of cut images CI from the high-quality image included in the first image information I1 and the low-quality image included in the second image information I2. It should be noted that the position coordinates cut out from the high-quality image included in the first image information I1 and the low-quality image included in the second image information I2 are preferably the same as each other. However, when there is a time difference between the timing of acquiring the high-quality image included in the first image information I1 and the timing of acquiring the low-quality image included in the second image information I2, sometimes the subject included in the cut image has a deviation caused by the time difference. In such a case, the position coordinates cut out from the high-quality image included in the first image information I1 and the low-quality image included in the second image information I2 are preferably determined in consideration of the deviation caused by the time difference. More specifically, it is preferable to change the position coordinates cut out from the high-quality image included in the first image information I1 or the low-quality image included in the second image information I2 in a direction that reduces the amount of deviation caused by the time difference.

[0213] (Step S150) Next, the dynamic image information generating unit 1012 connects the cut images to generate a dynamic image. The dynamic image information generating unit 1012 generates a high-quality dynamic image by connecting a plurality of images cut from the high-quality image, and generates a low-quality dynamic image by connecting a plurality of images cut from the low-quality image. The steps of generating dynamic image information in steps S130 and S150 are sometimes described as a dynamic image information generating step or a dynamic image information generating process.

[0214] (Step S170) Finally, the learning unit 1013 uses the generated combination of high-quality and low-quality moving images as training data TD to perform learning in a manner of inferring high-quality moving images from low-quality moving images. This step may be described as a learning step or a learning process.

[0215] [Summary of the Fifth Embodiment]

[0216] According to the above-described embodiment, the learning device 1010 acquires the first image information I1 and the second image information I2 by having an image acquisition unit 1011. The first image information I1 includes at least one image, and the second image information I2 includes at least one image of lower quality than the image included in the first image information I1, and the image captures the same subject as the subject captured in the image included in the first image information I1. In addition, the learning device 1010 cuts out a plurality of images that are parts of the first image information I1 and at different positions, and combines the cut-out plurality of images to generate the first dynamic image information M1. Similarly, the learning device 1010 cuts out a plurality of images that are parts of the second image information I2 and at different positions, and combines the cut-out plurality of images to generate the second dynamic image information M2 by having a dynamic image information generation unit 1012. In addition, the learning device 1010 includes a learning unit 1013, which performs learning in a manner of inferring a high-quality dynamic image from a low-quality dynamic image based on training data TD including the first dynamic image information M1 and the second dynamic image information M2 generated by the dynamic image information generation unit 1012. That is, according to this embodiment, the learning device 1010 does not need to obtain training data including low-quality dynamic images and high-quality dynamic images by shooting dynamic images, which was previously required, but can generate them from static images. Therefore, according to this embodiment, training data for inferring a high-quality dynamic image from a low-quality dynamic image can be easily generated.

[0217] In addition, according to this embodiment, the learning device 1010 can generate a plurality of different dynamic images from the same static image. Therefore, according to this embodiment, it is not necessary to prepare a large number of static images in order to generate a large number of training data TD, and a large number of dynamic images can be generated from a small number of static images. Therefore, according to this embodiment, the time required to capture images for learning can be shortened.

[0218] In addition, according to the above-described embodiment, the second image information I2 includes a plurality of images obtained by photographing the same subject as the subject photographed in the image included in the first image information I1, and each of the images is superimposed with mutually different noises. The dynamic image information generation unit 1012 generates the second dynamic image information M2 by cutting out different parts from the plurality of images included in the second image information I2. That is, according to the present embodiment, a low-quality dynamic image superimposed with noise is generated based on a plurality of different low-quality images superimposed with noise. Therefore, the second dynamic image information M2 generated by the present embodiment can reproduce and generate a low-quality dynamic image in which different noises are superimposed on each frame and the noise is superimposed with greater accuracy.

[0219] In addition, according to the embodiment described above, the multiple images included in the second image information I2 are images captured at close different times. That is, the low-quality images used to generate the low-quality dynamic images are captured at close times. The close time may be, for example, 1 / 60 seconds. Here, in the case of dynamic images, unlike static images, noise specific to dynamic images with a time component is sometimes superimposed. The images captured at close different times contain noise specific to the dynamic images. Therefore, according to the present embodiment, the learning device 1010 generates dynamic images based on images captured at close different times, and is therefore able to reproduce and generate noise specific to dynamic images with a time component.

[0220] In addition, according to the above-described embodiment, the dynamic image information generating unit 1012 generates the first dynamic image information M1 by cutting out different parts from one image included in the first image information I1. That is, according to this embodiment, a high-quality dynamic image is generated based on one image. Therefore, according to this embodiment, it is not necessary to shoot a large number of high-quality images, and a high-quality dynamic image can be easily generated.

[0221] In addition, according to the above-described embodiment, the dynamic image information generation unit 1012 cuts out multiple images at different positions by shifting the multiple images cut out by different amounts in a predetermined direction. That is, according to this embodiment, the learning device 1010 shifts the images in a predetermined direction after cutting out the images. In other words, after cutting out the images, the learning device 1010 does not need to process based on the large image, but processes based on the cut out small image. Therefore, according to this embodiment, the learning device 1010 can reduce the processing.

[0222] In addition, according to the above-described embodiment, the dynamic image information generating unit 1012 cuts out a plurality of images at positions that are moved by a predetermined number of bits in a predetermined direction. The dynamic image information generating unit 1012 generates a dynamic image by connecting the cut out images. That is, the subject captured in the dynamic image generated by the dynamic image information generating unit 1012 appears to move in a predetermined direction in the dynamic image. Therefore, according to this embodiment, a dynamic image can be easily generated from a static image.

[0223] In addition, according to the above-described embodiment, the predetermined direction in which the dynamic image information generating unit 1012 cuts out the image is calculated by affine transformation. In other words, the predetermined direction in which the dynamic image information generating unit 1012 cuts out the image refers to the direction in which the subject moves in the dynamic image. Therefore, according to this embodiment, the learning device 1010 can generate dynamic images in which the subject moves in various directions.

[0224] In addition, according to the above-described embodiment, the learning device 1010 further includes a trajectory vector acquisition unit 1014, which acquires a trajectory vector TV. In addition, the dynamic image information generation unit 1012 calculates the predetermined direction of the cut-out image based on the acquired trajectory vector TV. The trajectory vector TV refers to information related to a vector representing the trajectory of the subject actually moving in the dynamic image actually captured. Therefore, according to this embodiment, a dynamic image based on the trajectory of the subject actually moving can be generated.

[0225] [Sixth embodiment]

[0226] Next, refer to Fig.19 and Fig. 20 The sixth embodiment will be described. In the first embodiment, both high-quality images and low-quality images are required to create training data TD. However, in the sixth embodiment, only high-quality images are required, which is different from the fifth embodiment.

[0227] Fig.19 This is a diagram for describing the overview of the learning system of the sixth embodiment. With reference to this figure, the overview of the learning system 1001B of the sixth embodiment is described. In the description of this figure, for the same structure as the fifth embodiment, the description is sometimes omitted by marking the same figure mark. In the sixth embodiment, the camera device 1020 captures a high-quality image 1031. The low-quality image 1032 is generated by the learning device 1010B of the sixth embodiment based on the high-quality image 1031. For example, the low-quality image 1032 is generated by performing image processing on the high-quality image 1031 and superimposing noise. That is, according to this embodiment, the camera device 1020 only captures the high-quality image 1031, and there is no need to capture the low-quality image 1032.

[0228] Fig. 20 This is a diagram showing an example of the functional structure of the dynamic image information generating unit of the sixth embodiment. Referring to this diagram, the dynamic image information generating unit 1012B included in the learning device 1010B is described. The learning device 1010B of the sixth embodiment is different from the learning device 1010 in that it includes a dynamic image information generating unit 1012B instead of the dynamic image information generating unit 1012. The dynamic image information generating unit 1012B includes a cutting unit 1121, a noise superimposing unit 1123, a first dynamic image information generating unit 1125, and a second dynamic image information generating unit 1127.

[0229] The cutting unit 1121 obtains an image from the image acquisition unit 1011. In the present embodiment, the learning device 1010B obtains a high-quality image from the camera 1020, so the cutting unit 1121 obtains the high-quality image from the image acquisition unit 1011. The cutting unit 1121 cuts out a plurality of cut images CI at different position coordinates, and the cut images CI are a part of the obtained high-quality image. The cutting unit 1121 outputs the cut images CI to the first dynamic image information generation unit 1125 and the noise superposition unit 1123.

[0230] The noise superimposing unit 1123 obtains the cut-out image CI cut out by the cut-out unit 1121. The noise superimposing unit 1123 superimposes noise on the obtained cut-out image CI. The noise superimposing unit 1123 obtains a plurality of cut-out images CI cut out at a plurality of position coordinates, and superimposes noise on each of the obtained plurality of cut-out images CI. The noise superimposed by the noise superimposing unit 1123 may also be modeled in advance. As the modeled noise, shot noise caused by fluctuations in the number of photons, noise generated when light incident on the imaging element is converted into electrons, noise generated when the converted electrons are converted into analog voltage values, noise generated when the converted analog voltage values ​​are converted into digital signals, etc. may be exemplified. The intensity of the superimposed noise may also be adjusted by a predetermined method. The noise superimposing unit 1123 preferably superimposes different noises on each of the plurality of cut-out images CI. The noise superimposing unit 1123 outputs the image superimposed with noise as a noise image NI to the second dynamic image information generating unit 1127.

[0231] The first moving image information generating unit 1125 acquires the plurality of cut-out images CI from the cutting unit 1121 . The first moving image information generating unit 1125 combines the plurality of cut-out images to generate the first moving image information M1 . The first moving image information generating unit 1125 outputs the generated first moving image information M1 to the learning unit 1013 .

[0232] The second moving image information generating unit 1127 acquires a plurality of noise images NI from the noise superimposing unit 1123 . The second moving image information generating unit 1127 combines the plurality of noise images NI superimposed with noise to generate second moving image information M2 . The second moving image information generating unit 1127 outputs the generated second moving image information M2 to the learning unit 1013 .

[0233] The learning unit 1013 obtains the first moving image information M1 from the first moving image information generating unit 1125 and obtains the second moving image information M2 from the second moving image information generating unit 1127. The learning unit 1013 causes the learning model 1040 to perform learning based on the first moving image information M1 and the second moving image information M2 generated by the moving image information generating unit 1012B.

[0234] [Summary of the Sixth Embodiment]

[0235] According to the above-described embodiment, the learning device 1010B acquires image information I including at least one high-quality image by including an image acquisition unit 1011. In addition, the learning device 1010B is provided with a dynamic image information generation unit 1012B, thereby generating both a high-quality dynamic image and a low-quality dynamic image from a high-quality image. The dynamic image information generation unit 1012B is provided with a cutting unit 1121, and cuts out a plurality of images at different positions as a part of the acquired image information I. In addition, the dynamic image information generation unit 1012B is provided with a noise superimposition unit 1123, and superimposes noise on each of the plurality of images cut out by the cutting unit 1121. The dynamic image information generation unit 1012B is provided with a first dynamic image information generation unit 1125, and combines the plurality of images cut out by the cutting unit 1121 to generate the first dynamic image information M1 as a high-quality dynamic image, and is provided with a second dynamic image information generation unit 1127, and combines the plurality of images superimposed with noise by the noise superimposition unit 1123 to generate the second dynamic image information M2. In addition, the learning device 1010B is provided with a learning unit 1013, and performs learning in a manner of inferring a high-quality dynamic image from a low-quality dynamic image based on training data TD including the first dynamic image information M1 generated by the first dynamic image information generating unit 1125 and the second dynamic image information M2 generated by the second dynamic image information generating unit 1127. That is, according to the learning device 1010B, a high-quality dynamic image and a low-quality dynamic image are generated based on one high-quality image, and the learning model 1040 is learned in a manner of inferring the high-quality dynamic image from the low-quality dynamic image. In other words, the inference from the low-quality dynamic image to the high-quality dynamic image is the removal of noise. Therefore, according to this embodiment, the time for acquiring the training data TD is not required, and the noise removal model can be easily learned.

[0236] It should be noted that in the sixth embodiment, a high-quality dynamic image is generated from a high-quality image, and then a low-quality image is generated by superimposing noise on the high-quality image, and a low-quality dynamic image is generated based on the generated low-quality image. However, the present embodiment is not limited to this example. For example, as a variation of the present embodiment, the learning device 1010 may also produce training data TD based only on low-quality images. That is, a low-quality dynamic image may be generated from a low-quality image, and then a high-quality image is generated by removing noise from the low-quality dynamic image, and a high-quality dynamic image is generated based on the generated high-quality image. The image used to generate the dynamic image may be one or more images.

[0237] It should be noted that the learning device 1010 and the learning device 1010A described in the fifth embodiment and the learning device 1010B described in the sixth embodiment are used to learn the learning model 1040 for inferring a high-quality dynamic image from a low-quality dynamic image, but the invention is not limited thereto. For example, the learning model 1040 may be configured to have a function of detecting a specific subject such as a person in a high-quality dynamic image after the high-quality dynamic image is inferred from the low-quality dynamic image, or may be configured to have a function of recognizing text such as a logo or a sign in a high-quality dynamic image. That is, the high-quality dynamic image inferred by the learning model 1040 is not limited to an example of a dynamic image for appreciation, but may also be used for purposes such as object detection.

[0238] In the past, in order to improve the general performance of the learning model, it is preferred to include as many of the conceivable scenes as possible in the training data. In other words, it can be said that dynamic images that include as many of the conceivable movements of the subject as possible are ideal training data. On the other hand, it is difficult to obtain such training data through actual shooting, which requires huge costs and time. By using this embodiment for learning the learning model, the cost and time required for collecting training data can be greatly reduced. In addition, by using this embodiment for learning the learning model, the general performance of the learning model can be improved.

[0239] Although the mode for carrying out the present invention has been described above using the embodiment, the present invention is not limited to such embodiment at all, and various modifications and substitutions can be added without departing from the gist of the present invention.

[0240] Description of Reference Numerals

[0241] 1…High-quality dynamic image generation system, 2…Image processing device, 10…Input information generation device, 11…Image acquisition unit, 12…Input conversion unit, 13…Synthesis unit, 14…Output unit, 15…Integration unit, 16…Average value temporary storage unit, 17…Camera condition acquisition unit, 18…Adjustment unit, 19…Comparison unit, 100…Camera device, 200…CNN, 210…Input layer, 220…Convolution layer, 230…Pooling layer, 240…Output layer, IM…Dynamic image information, IN…Input information, TF…Object frame, AF…Adjacent frame, IF…Integration frame, M1…First memory, M2…Second memory, IMD…Image information, IND…Input data, CD…Synthesis data, SV…Storage value, CV…Calculation value, CR…Comparison result, 1001…Learning system, 1010…learning device, 1011…image acquisition unit, 1012…dynamic image information generation unit, 1013…learning unit, 1014…trajectory vector acquisition unit, 1020…camera, 1031…high-quality image, 1032…low-quality image, 1033…high-quality dynamic image, 1034…low-quality dynamic image, 1040…learning model, 1050…trajectory vector generation device, TD…training data, I…image information, I1…first image information, I2…second image information, M…dynamic image information, M1…first dynamic image information, M2…second dynamic image information, TV…trajectory vector, 1121…cutout unit, 1123…noise superposition unit, 1125…first dynamic image information generation unit, 1127…second dynamic image information generation unit

Claims

1. An input information generating device, wherein: The input information generating device comprises: An image acquisition unit that acquires, as input images, a plurality of frames including at least a target frame that is a target of generation of input information, among frames constituting a dynamic image; An input conversion unit that converts the pixel values ​​of the input images of the acquired plurality of frames into input data having a bit number less than the bit number representing the pixel values ​​of the input images; A synthesis unit synthesizes the transformed multiple input data into one synthesized data; as well as The output unit outputs the synthesized data after synthesis.

2. The input information generating device according to claim 1, wherein: The dynamic image is a color dynamic image, The image acquisition unit acquires pixel values ​​of each color from one frame as a plurality of different images. The input conversion unit converts each of a plurality of images acquired from one frame into the input data.

3. The input information generating device according to claim 1, wherein: The image acquisition unit acquires images of a plurality of frames that are continuously adjacent to and before the target frame, among frames constituting the moving image.

4. The input information generating device according to claim 1, wherein: After the synthesized data on the target frame is output by the output unit, the image acquisition unit sets a frame adjacent to the target frame as the target frame and acquires a plurality of frames including at least the target frame as the input image.

5. The input information generating device according to claim 1, wherein: The input information generating device further includes an integration unit that integrates a plurality of adjacent frames into one integrated frame by performing a calculation based on pixel values ​​of adjacent frames other than the target frame among the plurality of frames acquired by the image acquisition unit. The input conversion unit converts the pixel value of the target frame into the input data, and further converts the pixel value of the integration frame into the input data. The synthesis unit synthesizes the plurality of input data transformed by the target frame and the plurality of input data transformed by the integration frame into one piece of synthesized data.

6. The input information generating device according to claim 5, wherein: The integration unit uses an average value of pixel values ​​of the plurality of adjacent frames as a pixel value of the integration frame.

7. The input information generating device according to claim 6, wherein: The integration unit calculates the pixel value of the integration frame by calculating a weighted average according to a temporal distance from the target frame among the plurality of adjacent frames.

8. The input information generating device according to claim 6, wherein: The integration unit excludes frames having large brightness changes from among frames constituting the moving image from calculation targets of the average value.

9. The input information generating device according to claim 5, wherein: The input information generating device further includes an average value temporary storage unit for storing an average value of pixel values ​​of predetermined frames among the frames constituting the dynamic image. The integration unit calculates a pixel value of the integration frame through an operation based on the value stored in the average value temporary storage unit and the target frame.

10. The input information generating device according to claim 9, wherein: The input information generating device further comprises: a photographing condition acquisition unit, which acquires photographing conditions of the dynamic image; and The adjustment unit adjusts the average value stored in the average value temporary storage unit according to the acquired imaging condition.

11. The input information generating device according to claim 9, wherein: further comprising a comparing unit for comparing the value stored in the average value temporary storage unit with the pixel value of the target frame, When the result of comparison by the comparison unit is that the difference is less than a predetermined value, the integration unit calculates the pixel value of the integration frame by calculating the moving average based on the value stored in the average temporary storage unit and the object frame; when the difference is not less than the predetermined value, the pixel value of the object frame is used as the pixel value of the integration frame.

12. The input information generating device according to claim 5, wherein: The integration section uses a randomly determined frame among the adjacent frames acquired by the image acquisition section as the integration frame.

13. An image processing device, wherein: The image processing device comprises: The input information generating device according to any one of claims 1 to 12; and A convolutional neural network uses the synthetic data output by the input information generating device as input information.

14. A method for generating input information, wherein: The input information generation method has the following features: An image acquisition step of acquiring, as input images, a plurality of frames including at least a target frame to be generated as input information, among frames constituting the dynamic image; An input conversion step of converting the pixel values ​​of the input images of the acquired plurality of frames into input data having a bit number less than the bit number representing the pixel values ​​of the input images; A synthesis step of synthesizing the transformed multiple input data into one synthesized data; and The output step is to output the synthesized data after synthesis.

15. A learning device, wherein: The learning device has: An image acquisition unit acquires first image information and second image information, wherein the first image information includes at least one image, and the second image information includes at least one image whose image quality is lower than that of the image included in the first image information, and the image captures the same subject as the subject captured in the image included in the first image information; A dynamic image information generating unit, which cuts out a plurality of images which are parts of the acquired first image information and are located at different positions, combines the cut-out plurality of images to generate the first dynamic image information, cuts out a plurality of images which are parts of the acquired second image information and are located at different positions, combines the cut-out plurality of images to generate the second dynamic image information; as well as The learning unit performs learning in such a manner as to infer a high-quality moving image from a low-quality moving image based on training data including the first moving image information and the second moving image information generated by the moving image information generating unit.

16. The learning device according to claim 15, wherein: The second image information includes a plurality of images, the plurality of images being images of the same subject as the subject captured in the image included in the first image information, and each of the images having mutually different noises superimposed thereon. The moving image information generating unit generates the second moving image information by cutting out different parts from each of a plurality of images included in the second image information.

17. The learning device according to claim 16, wherein: The plurality of images included in the second image information are images captured at close but different times.

18. The learning device according to claim 15 or 16, wherein: The moving image information generating unit generates the first moving image information by cutting out different parts from one image included in the first image information.

19. The learning device according to claim 15 or 16, wherein: The moving image information generating unit cuts out a plurality of images at different positions by shifting the plurality of cut-out images in a predetermined direction.

20. The learning device according to claim 15 or 16, wherein: The moving image information generating unit cuts out a plurality of images at positions shifted by a predetermined number of bits in a predetermined direction.

21. The learning device according to claim 19, wherein: The predetermined direction in which the moving image information generating unit cuts out the image is calculated by affine transformation.

22. The learning device according to claim 19, wherein: The learning device further includes a trajectory vector acquisition unit for acquiring a trajectory vector. The predetermined direction in which the moving image information generating unit cuts out an image is calculated based on the acquired trajectory vector.

23. A learning device, wherein: The learning device has: An image acquisition unit, which acquires image information including at least one image; A cutting unit, cutting out a plurality of images at different positions which are a part of the acquired image information; a first dynamic image information generating unit that generates first dynamic image information by combining the plurality of cut images; a noise superimposing unit for superimposing noise on each of the plurality of images cut out by the cutting unit; a second dynamic image information generating unit that generates second dynamic image information by combining the plurality of images on which noise is superimposed by the noise superimposing unit; as well as The learning unit performs learning in a manner of inferring a high-quality dynamic image from a low-quality dynamic image based on training data including the first dynamic image information generated by the first dynamic image information generating unit and the second dynamic image information generated by the second dynamic image information generating unit.

24. A program, wherein: The program causes the computer to perform the following steps: An image acquisition step of acquiring first image information and second image information, wherein the first image information includes at least one image, and the second image information includes at least one image having a lower image quality than the image included in the first image information, and the image captures the same subject as the subject captured in the image included in the first image information; A step of generating dynamic image information, cutting out a plurality of images which are parts of the first image information obtained and are at different positions, combining the plurality of cut images to generate the first dynamic image information, cutting out a plurality of images which are parts of the second image information obtained and are at different positions, and combining the plurality of cut images to generate the second dynamic image information; as well as The learning step performs learning in such a manner as to infer a high-quality moving image from a low-quality moving image based on training data including the first moving image information and the second moving image information generated by the moving image information generating step.

25. A learning method for a noise reduction device, wherein: The learning method of the noise reduction device has the following features: An image acquisition step of acquiring first image information and second image information, wherein the first image information includes at least one image, and the second image information includes at least one image having a lower image quality than the image included in the first image information, and the image captures the same subject as the subject captured in the image included in the first image information; A step of generating dynamic image information, cutting out a plurality of images which are parts of the first image information obtained and which are at different positions, combining the plurality of cut images to generate the first dynamic image information, cutting out a plurality of images which are parts of the second image information obtained and which are at different positions, and combining the plurality of cut images to generate the second dynamic image information; as well as The learning step performs learning in such a manner as to infer a high-quality moving image from a low-quality moving image based on training data including the first moving image information and the second moving image information generated by the moving image information generating step.

Images