Apparatus for removing noise in time-lapse image, method for removing noise in time-lapse image, and computer-readable recording medium having recorded thereon program for executing same in computer

Abstract

The apparatus for removing noise in a time-lapse image comprises a central frame processing convolutional neural network, a peripheral frame processing convolutional neural network, and an information combining convolutional neural network. The central frame processing convolutional neural network receives the current frame of an input image and outputs a central processing value. The peripheral frame processing convolutional neural network receives a plurality of past frames adjacent to the current frame and a plurality of future frames adjacent to the current frame, and outputs a peripheral processing value. The information combining convolutional neural network calculates the central processing value and the peripheral processing value and outputs an output image. The central frame processing convolutional neural network does not refer to the central pixel of the current frame.

Description

A device for removing noise from a time-lapse video, a method for removing noise from a time-lapse video, and a computer-readable recording medium having recorded thereon a program for executing the same on a computer.

[0001] The present invention relates to a device for removing noise from a time-lapse image, a method for removing noise from a time-lapse image, and a computer-readable recording medium having recorded thereon a program for executing the same on a computer, and more particularly, to a device for removing noise from a time-lapse image capable of removing noise with high performance without separate learning data, a method for removing noise from a time-lapse image, and a computer-readable recording medium having recorded thereon a program for executing the same on a computer.

[0002] To remove noise from existing time-lapse images, several algorithms have been developed based on Noise2Noise, a methodology for removing noise by training a convolutional neural network using two images with the same signal but different noise as input and output.

[0003] Noise2Noise is a methodology for training a neural network to remove noise. Instead of training a neural network with a noisy image as input and a clean image as a target image, it trains the neural network using an image with different noise but matching the signal other than the noise. This methodology has the advantage of being able to train a noise removal neural network without a clean image, but has the disadvantage of requiring pairs of images with different noise but identical signal as training data.

[0004] Further development of these algorithms has led to numerous recent papers published in Nature, a sister journal of the journal Nature. These algorithms exploit the inherent applicability of the Noise2Noise framework, treating adjacent frames within time-lapse video data as images with identical signal but different noise. They demonstrate superior noise removal performance compared to existing algorithms.

[0005] Because these algorithms fundamentally rely on similarities between adjacent frames within time-lapse video data, they cannot handle abrupt changes within the video. Even if a specific frame actually exhibits abrupt changes, the algorithms will filter out these changes, distorting the information contained within the video.

[0006] In cases where there are rapid changes in an image, it is known that an algorithm that removes noise through low-dimensional approximation of image data based on singular value decomposition (SVD) actually performs better than a machine learning-based algorithm.

[0007] An object of the present invention to solve the above problems is to provide a device for removing noise in a time-lapse image, which can remove noise in a time-lapse image and restore time series information contained in the original image data.

[0008] Another object of the present invention is to provide a method for removing noise in a time-lapse image, which can remove noise from the time-lapse image and restore time series information contained in the original image data.

[0009] Another object of the present invention is to provide a computer-readable recording medium having recorded thereon a program for executing a method for removing noise in a time-lapse image on a computer.

[0010] According to one embodiment of the present invention, a device for removing noise in a time-lapse image to achieve the above-described object includes a central frame processing convolutional neural network, a peripheral frame processing convolutional neural network, and an information-binding convolutional neural network. The central frame processing convolutional neural network receives a current frame of an input image and outputs a central processing value. The peripheral frame processing convolutional neural network receives a plurality of past frames neighboring the current frame and a plurality of future frames neighboring the current frame and outputs a peripheral processing value. The information-binding convolutional neural network calculates the central processing value and the peripheral processing value and outputs an output image. The central frame processing convolutional neural network does not refer to a central pixel of the current frame.

[0011] In one embodiment of the present invention, the central frame processing convolutional neural network can further receive the peripheral processing values.

[0012] In one embodiment of the present invention, the peripheral frame processing convolutional neural network can refer to the central pixel of the past frame and the central pixel of the future frame.

[0013] In one embodiment of the present invention, the impulse response of the central frame processing convolutional neural network may not have a central impulse value corresponding to the central pixel of the current frame, but may have an impulse value for an impulse area of ​​a predetermined size surrounding the central pixel of the current frame.

[0014] In one embodiment of the present invention, the central frame processing convolutional neural network may include a convolution operation, a plurality of extended convolution operations that receive the results of the convolution operation and are sequentially arranged. The size of the kernel of the extended convolution operation may be larger than the size of the kernel of the convolution operation.

[0015] In one embodiment of the present invention, the kernel of the convolution operation may have a first value in the first row and the first column, a second value in the first row and the second column, a third value in the first row and the third column, a fourth value in the first row and the second column, a fifth value in the third row and the second column, a sixth value in the first row and the third column, a seventh value in the second row and the third column, an eighth value in the third row and the third column, and may not have a value in the second row and the second column corresponding to the central pixel of the current frame.

[0016] In one embodiment of the present invention, the impulse response of the convolution operation may have a first impulse value in the first row and first column, a second impulse value in the first row and second column, a third impulse value in the first row and third column, a fourth impulse value in the first row and second column, a fifth impulse value in the second row and third column, a sixth impulse value in the first row and third column, a seventh impulse value in the second row and third column, and an eighth impulse value in the third row and third column, and may not have an impulse value in the second row and second column corresponding to the central pixel of the current frame.

[0017] In one embodiment of the present invention, the kernel of the first extended convolution operation may have a first value in the first row and the first column, a second value in the first row and the third column, a third value in the first row and the fifth column, a fourth value in the first row and the third column, a fifth value in the fifth row and the fifth column, a sixth value in the first row and the fifth column, a seventh value in the third row and the fifth column, an eighth value in the fifth row and the fifth column, and may not have a value in the third row and the third column corresponding to the central pixel of the current frame, and may not have a value in the second row, the fourth row, the second column, and the fourth column.

[0018] In one embodiment of the present invention, the impulse response of the first extended convolution operation may not have an impulse value in a 4-row, 4-column matrix corresponding to the central pixel of the current frame, and the remaining areas may have respective impulse values.

[0019] In one embodiment of the present invention, the kernel of the second extended convolution operation may have a first value in the first row and the first column, a second value in the first row and the fifth column, a third value in the first row and the ninth column, a fourth value in the first row and the fifth column, a fifth value in the fifth row and the ninth column, a sixth value in the first row and the ninth column, a seventh value in the fifth row and the ninth column, an eighth value in the ninth row and the ninth column, and may not have a value in the fifth row and the 5th column corresponding to the central pixel of the current frame, and may not have values ​​in the second to fourth rows, the sixth to the eighth rows, the second to the fourth columns, and the sixth to the eighth columns.

[0020] In one embodiment of the present invention, the central frame processing convolutional neural network may not refer to a central pixel group of the current frame, which includes the central pixel of the current frame and a plurality of surrounding pixels adjacent to the central pixel.

[0021] In one embodiment of the present invention, the central frame processing convolutional neural network may include a convolution operation, a plurality of extended convolution operations that receive the results of the convolution operation and are sequentially arranged. The size of the kernel of the extended convolution operation may be larger than the size of the kernel of the convolution operation.

[0022] In one embodiment of the present invention, the kernel of the convolution operation may have a first value in the first row and first column, a second value in the first row and second column, a third value in the first row and third column, a fourth value in the first row and fourth column, a fifth value in the first row and fifth column, a sixth value in the second row and first column, a seventh value in the second row and fifth column, an eighth value in the third row and first column, a ninth value in the third row and second column, a tenth value in the third row and third column, an eleventh value in the third row and fourth column, and a twelfth value in the third row and fifth column, and may not have values ​​in the second row and second column, the second row and third column, and the second row and fourth column corresponding to the central pixel group of the current frame.

[0023] In one embodiment of the present invention, the impulse response of the convolution operation may have a first impulse value in the first row and first column, a second impulse value in the first row and second column, a third impulse value in the first row and third column, a fourth impulse value in the first row and fourth column, a fifth impulse value in the first row and fifth column, a sixth impulse value in the second row and first column, a seventh impulse value in the second row and fifth column, an eighth impulse value in the third row and first column, a ninth impulse value in the third row and second column, a tenth impulse value in the third row and third column, an eleventh impulse value in the third row and fourth column, and a twelfth impulse value in the third row and fifth column, and may not have impulse values ​​in the second row and second column, the second row and third column, and the second row and fourth column corresponding to the central pixel group of the current frame.

[0024] In one embodiment of the present invention, the kernel of the first extended convolution operation may have a first value in the first row and the first column, a second value in the first row and the fourth column, a third value in the first row and the seventh column, a fourth value in the first row and the third column, a fifth value in the seventh row and the third column, a sixth value in the first row and the fifth column, a seventh value in the fourth row and the fifth column, and an eighth value in the seventh row and the fifth column, and may not have values ​​in the third row and the third column, the third row and the fourth column, and the third row and the fifth column corresponding to the central pixel group of the current frame, and may not have values ​​in the second row, the fourth row, the second to the third columns, and the fifth to the sixth columns.

[0025] In one embodiment of the present invention, the impulse response of the first extended convolution operation may not have impulse values ​​in the 4th row, 5th column, 4th row, 6th column, and 4th row, 7th column corresponding to the central pixel group of the current frame in a matrix of 7 rows and 11 columns, and the remaining areas may have respective impulse values.

[0026] In one embodiment of the present invention, the central frame processing convolutional neural network may include a first sequential path that sequentially performs a convolution operation by receiving the peripheral processing values ​​as input, and a second sequential path that sequentially performs a convolution operation by receiving the current frame as input.

[0027] In one embodiment of the present invention, the number of convolution operation layers of the first sequential path may be less than the number of convolution operation layers of the second sequential path.

[0028] In one embodiment of the present invention, the number of convolution operation layers of the first sequential path may be three, and the number of convolution operation layers of the second sequential path may be five.

[0029] In one embodiment of the present invention, the size of the kernel of the convolution operation of the first sequential path may be larger than the size of the kernel of the convolution operation of the second sequential path.

[0030] In one embodiment of the present invention, the size of the kernel of the convolution operation of the first sequential path may be 5×5, and the size of the kernel of the convolution operation of the second sequential path may be 3×3.

[0031] In one embodiment of the present invention, the central frame processing convolutional neural network may include a first convolution operation layer that receives the peripheral processing value as input and outputs a first convolution result, a second convolution operation layer that receives the first convolution result as input and outputs a second convolution result, a third convolution operation layer that receives the second convolution result as input and outputs a third convolution result, a fourth convolution operation layer that receives the current frame as input and outputs a fourth convolution result, a fifth convolution operation layer that receives the fourth convolution result as input and outputs a fifth convolution result, a sixth convolution operation layer that receives the fifth convolution result as input and outputs a sixth convolution result, a seventh convolution operation layer that receives the sixth convolution result as input and outputs a seventh convolution result, and an eighth convolution operation layer that receives the seventh convolution result as input and outputs an eighth convolution result.

[0032] In one embodiment of the present invention, the central frame processing convolutional neural network may further include a ninth convolution operation layer that performs a 1×1 convolution operation and a ReLU operation on the concatenated first convolution result to the eighth convolution result.

[0033] In one embodiment of the present invention, the first convolution result and the input of the central frame processing convolutional neural network that performed the 1×1 convolution operation may be input to the second convolution operation layer. The second convolution result and the input of the central frame processing convolutional neural network that performed the 1×1 convolution operation may be input to the third convolution operation layer. The fourth convolution result and the input of the central frame processing convolutional neural network that performed the 1×1 convolution operation may be input to the fifth convolution operation layer. The fifth convolution result and the input of the central frame processing convolutional neural network that performed the 1×1 convolution operation may be input to the sixth convolution operation layer. The sixth convolution result and the input of the central frame processing convolutional neural network that performed the 1×1 convolution operation may be input to the seventh convolution operation layer. The above-mentioned 8th convolution operation layer may receive the above-mentioned 7th convolution result and the above-mentioned input of the central frame processing convolutional neural network that performed the above-mentioned 1×1 convolution operation.

[0034] In one embodiment of the present invention, the information combining convolutional neural network can perform a 1×1 convolution operation on the central processing value and the peripheral processing value.

[0035] In one embodiment of the present invention, i is a time index, k is a spatial index, L is a loss function, go is a neural network parameterized by , is the central pixel, are the neighboring pixels of the central pixel excluding the central pixel, When the correct data of the central pixel is the statistical prediction model used in the device for removing noise in the time-lapse image, It can be implemented by solving the optimization problem.

[0036] According to one embodiment of the present invention, a device for removing noise in a time-lapse image to achieve the above-described object includes a central frame processing convolutional neural network, a peripheral frame processing convolutional neural network, and an information-binding convolutional neural network. The central frame processing convolutional neural network receives a current frame of an input image and outputs a central processing value. The peripheral frame processing convolutional neural network receives a plurality of past frames neighboring the current frame and a plurality of future frames neighboring the current frame and outputs a peripheral processing value. The information-binding convolutional neural network calculates the central processing value and the peripheral processing value and outputs an output image. The median value of the kernel of the central frame processing convolutional neural network corresponds to the central pixel of the current frame. The median value of the kernel is set to 0.

[0037] According to one embodiment of the present invention, a method for removing noise in a time-lapse image includes the steps of receiving a current frame of an input image and generating a central processing value, receiving a plurality of past frames neighboring the current frame and a plurality of future frames neighboring the current frame and generating a peripheral processing value, and calculating the central processing value and the peripheral processing value to output an output image. The central frame processing convolutional neural network for generating the central processing value may not refer to the central pixel of the current frame.

[0038] In one embodiment of the present invention, the central frame processing convolutional neural network can further receive the peripheral processing values.

[0039] In one embodiment of the present invention, the peripheral frame processing convolutional neural network that generates the peripheral processing value may refer to the central pixel of the past frame and the central pixel of the future frame.

[0040] In one embodiment of the present invention, a program for executing a method for removing noise in a time-lapse image on a computer can be recorded on a computer-readable recording medium.

[0041] According to the device and method for removing noise in a time-lapse image according to the embodiment of the present invention as described above, noise in a time-lapse image can be effectively removed and time series information contained in the original image data can be restored.

[0042] Additionally, noise removal from images is possible without separate learning data, without assuming that adjacent frames within a time-lapse image have the same signal components.

[0043] Additionally, by utilizing information contained in the current frame as well as past and future frames, it is possible to effectively remove noise without distorting or removing the signal even when there are rapid changes in the image data or when the camera shakes.

[0044] Fig. 1 is a drawing showing an example of noise removal according to the present embodiment.

[0045] FIG. 2a is a block diagram showing an example of a device for removing noise in a time-lapse image according to the present embodiment.

[0046] FIG. 2b is a block diagram showing an example of a device for removing noise in a time-lapse image according to the present embodiment.

[0047] Figures 3 and 4 are diagrams showing dynamic indicators according to time and space.

[0048] FIG. 5 is a drawing showing an example of a device for removing noise in a time-lapse image according to the present embodiment.

[0049] Figure 6 is a block diagram illustrating the central frame processing convolutional neural network of Figures 2a and 2b.

[0050] Figure 7 is a diagram showing the kernel of the convolution operation of Figure 6.

[0051] Fig. 8 is a diagram showing the kernel of the first extended convolution operation of Fig. 6.

[0052] Fig. 9 is a diagram showing the kernel of the second extended convolution operation of Fig. 6.

[0053] FIG. 10 is a drawing showing an example of a device for removing noise in a time-lapse image according to the present embodiment.

[0054] FIG. 11 is a block diagram showing a central frame processing convolutional neural network of a device for removing noise in a time-lapse image of FIG. 10.

[0055] Figure 12 is a diagram showing the kernel of the convolution operation of Figure 11.

[0056] Fig. 13 is a diagram showing the kernel of the first extended convolution operation of Fig. 11.

[0057] FIG. 14 is a detailed block diagram showing an example of a device for removing noise in the time-lapse image of FIG. 2b.

[0058] Figure 15 is a detailed block diagram showing an example of a peripheral frame processing convolutional neural network of Figure 14.

[0059] Figure 16 is a detailed block diagram showing an example of the central frame processing convolutional neural network of Figure 14.

[0060] Figure 17 is a diagram showing the impulse response of the central frame processing convolutional neural network of Figure 14.

[0061] Fig. 18 is a diagram showing the noise removal result according to the present embodiment.

[0062] Fig. 19 is a diagram showing the noise removal result according to the present embodiment.

[0063] Fig. 20 is a diagram showing the noise removal result according to the present embodiment.

[0064] Fig. 21 is a diagram showing the noise removal result according to the present embodiment.

[0065] With respect to the embodiments of the present invention disclosed in the text, specific structural and functional descriptions are merely illustrative for the purpose of explaining the embodiments of the present invention, and the embodiments of the present invention may be implemented in various forms and should not be construed as being limited to the embodiments described in the text.

[0066] The present invention is susceptible to various modifications and takes various forms. Specific embodiments are illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to a specific disclosed form, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention. Similar reference numerals have been used to designate components throughout the description of each drawing.

[0067] While terms like "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, a first component could be referred to as a "second component," and similarly, a second component could also be referred to as a "first component," without departing from the scope of the present invention.

[0068] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

[0069] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprises" or "has" indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0070] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0071] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. Identical components in the drawings are designated by the same reference numerals, and redundant descriptions of identical components are omitted.

[0072] Fig. 1 is a drawing showing an example of noise removal according to the present embodiment.

[0073] Time-lapse video is data obtained by continuously taking pictures of the same location or the same object at regular time intervals, and is taken for the purpose of observing changes in the subject over time.

[0074] If the environment in which you are shooting time-lapse videos has low light conditions, you need to shoot very quickly, or the noise performance of the camera you are using is not good, the video will be very noisy.

[0075] Figure 1 illustrates input and output images of a device and method for removing noise from a time-lapse image according to the present embodiment. By passing through the device for removing noise from a time-lapse image, an output image with noise removed can be obtained from an input image containing noise.

[0076] FIG. 2a is a block diagram showing an example of a device for removing noise in a time-lapse image according to the present embodiment.

[0077] Referring to FIG. 2a, the present invention comprises (1) a convolutional neural network configured to receive pixel values ​​of a surrounding area of ​​a 3-dimensional (or n-dimensional) tensor centered on each pixel as input for given image data (tensor), but not to receive the center pixel value as input, (2) updating the convolutional parameters of the neural network through backpropagation, and (3) an inference method for each pixel value in the image data.

[0078] A device for removing noise in a time-lapse image includes a central frame processing convolutional neural network, a peripheral frame processing convolutional neural network, and an information-binding convolutional neural network. The central frame processing convolutional neural network receives a current frame of an input image as input and outputs a central processing value. The peripheral frame processing convolutional neural network receives a plurality of past frames adjacent to the current frame and a plurality of future frames adjacent to the current frame as input and outputs a peripheral processing value. The information-binding convolutional neural network calculates the central processing value and the peripheral processing value and outputs an output image. The central frame processing convolutional neural network does not refer to a central pixel of the current frame.

[0079] For example, the median value of the kernel of the central frame processing convolutional neural network corresponds to the central pixel of the current frame, and the median value of the kernel is set to 0.

[0080] On the other hand, the peripheral frame processing convolutional neural network can refer to the central pixel of the past frame and the central pixel of the future frame.

[0081] The above information combining convolutional neural network can perform a 1×1 convolution operation on the central processing value and the peripheral processing value.

[0082] where i is the time index, k is the spatial index, and L is the loss function. go is a neural network parameterized by , is the central pixel, are the neighboring pixels of the central pixel excluding the central pixel, When the correct data of the central pixel is the statistical prediction model used in the device for removing noise in the time-lapse image, It can be implemented by solving the optimization problem.

[0083] The present invention enables noise removal from an image without separate learning data, without assuming that signal components are the same between adjacent frames within a time-lapse image.

[0084] In conventional methodologies, to train a neural network to remove noise from time-lapse video data, past and future frames are used as input to infer the current frame with the noise removed. This is not a problem when changes in the video data are slow compared to the shooting speed, but when there are rapid changes within a specific frame, both noise and signal are removed. In contrast, the present invention utilizes information contained in the current frame as well as past and future frames, thereby effectively removing noise without distorting or removing the signal even when there are rapid changes in the video data.

[0085] FIG. 2b is a block diagram showing an example of a device for removing noise in a time-lapse image according to the present embodiment.

[0086] Referring to FIG. 2b, a device for removing noise in a time-lapse image includes a central frame processing convolutional neural network, a peripheral frame processing convolutional neural network, and an information-binding convolutional neural network. The central frame processing convolutional neural network receives a current frame and peripheral processing values ​​of an input image and outputs a central processing value. The peripheral frame processing convolutional neural network receives a plurality of past frames adjacent to the current frame and a plurality of future frames adjacent to the current frame and outputs the peripheral processing values. The information-binding convolutional neural network calculates the central processing value and the peripheral processing values ​​and outputs an output image. The central frame processing convolutional neural network does not refer to a central pixel of the current frame.

[0087] A method for removing noise in a time-lapse image includes the steps of receiving a current frame of an input image and generating a central processing value, receiving a plurality of past frames adjacent to the current frame and a plurality of future frames adjacent to the current frame and generating peripheral processing values, and calculating the central processing value and the peripheral processing value to output an output image. The central frame processing convolutional neural network that generates the central processing value may not refer to the central pixel of the current frame.

[0088] For example, the median value of the kernel of the central frame processing convolutional neural network corresponds to the central pixel of the current frame, and the median value of the kernel is set to 0.

[0089] Unlike the embodiment of FIG. 2A, the embodiment of FIG. 2B allows the central frame processing convolutional neural network to further receive the peripheral processing values. When the central frame processing convolutional neural network receives the current frame and the peripheral processing values ​​as inputs, the peripheral processing values ​​are referenced when generating the central processing value corresponding to the current frame, and thus the noise removal performance can be improved.

[0090] Figures 3 and 4 are diagrams showing dynamics indicators according to time and space.

[0091] Figure 3 illustrates a dynamics indicator in a self-supervised learning system of the present invention and a conventional methodology when the change in an image over time is slow, and Figure 4 illustrates a dynamics indicator in a self-supervised learning system of the present invention and a conventional methodology when the change in an image over time is fast. Here, the conventional methodology is a method that utilizes frames temporally adjacent to the current frame to remove noise from the current frame of image data.

[0092] In FIGS. 3 and 4, the portion indicated on a specific time plane represents a prediction target area, and the portion extending behind the specific time plane represents a receptive field.

[0093] As shown in Fig. 4, when there is a significant change in only one frame within the image data, if an existing noise removal methodology is utilized, the information of this frame is lost and cannot be recovered, but if the present invention is utilized, noise can be removed without losing information by utilizing spatial surrounding pixel information of the pixel from which noise is to be removed.

[0094] FIG. 5 is a drawing showing an example of a device for removing noise in a time-lapse image according to the present embodiment.

[0095] Referring to FIG. 5, in order to actually implement a device capable of removing noise without losing information by utilizing spatial surrounding pixel information of a pixel from which noise is to be removed, the present invention proposes and utilizes a convolutional neural network structure that does not receive as input a center value (meaning the middle value of the tensor, not the median) within a 3-dimensional or n-dimensional tensor.

[0096] The first frame of the input image represents a past frame, the second frame represents a current frame, and the third frame represents a future frame. For convenience of explanation, Figure 5 illustrates one past frame and one future frame, but the present invention is not limited thereto. Multiple past frames and multiple future frames can be used for noise removal.

[0097] As shown in Fig. 5, the central frame processing convolutional neural network may not refer to the central pixel of the current frame. In other words, the median value of the kernel of the central frame processing convolutional neural network may be set to 0.

[0098] On the other hand, the peripheral frame processing convolutional neural network may refer to the central pixel of the past frame and the central pixel of the future frame. In other words, the median value of the kernel of the peripheral frame processing convolutional neural network may have a non-zero value.

[0099] Fig. 6 is a block diagram illustrating the central frame processing convolutional neural network of Figs. 2a and 2b. Fig. 7 is a diagram illustrating the kernel of the convolution operation of Fig. 6. Fig. 8 is a diagram illustrating the kernel of the first extended convolution operation of Fig. 6. Fig. 9 is a diagram illustrating the kernel of the second extended convolution operation of Fig. 6.

[0100] Referring to FIGS. 1 to 9, the central frame (current frame) of the time-lapse image data does not refer to pixel values ​​at the same location within the input frame (current frame) when inferring each pixel value.

[0101] To achieve this, a central frame processing neural network can be configured as shown in Figure 6. Input data passes through a convolution layer and a repeated dilated convolution layer. Both the convolution and dilated convolution kernels initialize the center value to 0 and do not update it. This ensures that each pixel in the input data affects only pixels in the output data, excluding those in the same location.

[0102] As shown in the rightmost figure of Fig. 6, the impulse response of the central frame processing convolutional neural network may not have a central impulse value corresponding to the central pixel of the current frame, but may have an impulse value for an impulse area of ​​a predetermined size surrounding the central pixel of the current frame. The impulse area may be the entire pixel area of ​​the input image. Alternatively, the impulse area may be smaller or larger than the entire pixel area of ​​the input image.

[0103] The central frame processing convolutional neural network may include a convolution operation, a result of the convolution operation, and a plurality of extended convolution operations arranged sequentially. The kernel size of the extended convolution operation may be larger than the kernel size of the convolution operation.

[0104] As shown in FIG. 7, the kernel of the convolution operation may have a first value in the first row and first column, a second value in the first row and second column, a third value in the first row and third column, a fourth value in the first row and second column, a fifth value in the third row and third column, a sixth value in the first row and third column, a seventh value in the second row and third column, an eighth value in the third row and third column, and may not have a value in the second row and second column corresponding to the central pixel of the current frame.

[0105] As shown in the lower figure of FIG. 6, the impulse response of the convolution operation generated through the kernel of the convolution operation has a first impulse value in the first row and first column, a second impulse value in the first row and second column, a third impulse value in the first row and third column, a fourth impulse value in the first row and second column, a fifth impulse value in the third row and third column, a sixth impulse value in the first row and third column, a seventh impulse value in the second row and third column, an eighth impulse value in the third row and third column, and may not have an impulse value in the second row and second column corresponding to the central pixel of the current frame.

[0106] As shown in FIG. 8, the kernel of the first extended convolution operation has a first value in the first row and first column, a second value in the first row and third column, a third value in the first row and fifth column, a fourth value in the first row and third column, a fifth value in the fifth row and fifth column, a sixth value in the first row and fifth column, a seventh value in the third row and fifth column, an eighth value in the fifth row and fifth column, and may not have a value in the third row and third column corresponding to the central pixel of the current frame, and may not have values ​​in the second row, the fourth row, the second column, and the fourth column.

[0107] As shown in the lower figure of Fig. 6, the impulse response of the first extended convolution operation generated through the kernel of the first extended convolution operation may not have an impulse value in the 4 rows and 4 columns corresponding to the central pixel of the current frame in the matrix of 7 rows and 7 columns, and the remaining areas may have respective impulse values.

[0108] By applying the kernel of the first extended convolution operation of FIG. 7 to the impulse response of the above convolution operation, it is possible to generate an impulse response of the first extended convolution operation that does not have an impulse value in the 4 rows and 4 columns corresponding to the center pixel.

[0109] In addition, as shown in FIG. 9, the kernel of the second extended convolution operation may have a first value in the first row and first column, a second value in the first row and fifth column, a third value in the first row and ninth column, a fourth value in the first row and ninth column, a fifth value in the fifth row and ninth column, a sixth value in the first row and ninth column, a seventh value in the fifth row and ninth column, an eighth value in the ninth row and ninth column, and may not have a value in the fifth row and ninth column corresponding to the central pixel of the current frame, and may not have values ​​in the second to fourth rows, the sixth to eighth rows, the second to fourth columns, and the sixth to eighth columns.

[0110] By applying the kernel of the second extended convolution operation of FIG. 9 to the impulse response of the first extended convolution operation, an impulse response in which the impulse region is extended can be generated while still not having an impulse value for the central pixel.

[0111] In this way, as the above dilated convolution operation is repeated, the size of the kernel becomes larger and larger, and accordingly, an impulse response can be generated in which the impulse region gradually expands without having an impulse value for the central pixel.

[0112] FIG. 10 is a drawing showing an example of a device for removing noise in a time-lapse image according to the present embodiment.

[0113] The device for removing noise in a time-lapse image of FIG. 10 is the same as the device for removing noise in a time-lapse image of FIG. 5, except that the central frame processing convolutional neural network does not refer to a central pixel group including a central pixel and a plurality of surrounding pixels adjacent to the central pixel, rather than a central pixel.

[0114] As shown in Fig. 10, the central frame processing convolutional neural network may not refer to the central pixel group of the current frame, which includes the central pixel of the current frame and a plurality of surrounding pixels adjacent to the central pixel. In other words, the values ​​corresponding to the central pixel group among the kernels of the central frame processing convolutional neural network may be set to 0.

[0115] In FIG. 10, the central pixel group is exemplified as including 9 pixels in 3 rows and 3 columns, but the present invention is not limited thereto.

[0116] Fig. 11 is a block diagram illustrating a central frame processing convolutional neural network of a device for removing noise from a time-lapse image of Fig. 10. Fig. 12 is a diagram illustrating a kernel of the convolution operation of Fig. 11. Fig. 13 is a diagram illustrating a kernel of the first extended convolution operation of Fig. 11.

[0117] The central frame processing convolutional neural network of FIGS. 11 to 13 is identical to the central frame processing convolutional neural network of FIGS. 6 to 9, except that the central frame processing convolutional neural network does not refer to a central pixel group including a central pixel and a plurality of surrounding pixels adjacent to the central pixel, rather than a single central pixel.

[0118] In FIGS. 11 to 13, the central pixel group is exemplified as including three pixels in one row and three columns.

[0119] Referring to FIGS. 1 to 4 and 10 to 13, the central frame processing neural network can be configured as shown in FIG. 11. The input data passes through a convolution layer and a repeated dilated convolution layer. At this time, the kernels of both the convolution and dilated convolution initialize the center values ​​of the first row and third column to 0 and do not update them.

[0120] As shown in the rightmost figure of Fig. 11, the impulse response of the central frame processing convolutional neural network may not have an impulse value corresponding to the central pixel group including the central pixel and the left and right pixels of the current frame, but may have an impulse value for an impulse area of ​​a predetermined size surrounding the central pixel group of the current frame. The impulse area may be the entire pixel area of ​​the input image. Alternatively, the impulse area may be smaller or larger than the entire pixel area of ​​the input image.

[0121] The central frame processing convolutional neural network may include a convolution operation, a result of the convolution operation, and a plurality of extended convolution operations arranged sequentially. The kernel size of the extended convolution operation may be larger than the kernel size of the convolution operation.

[0122] As shown in FIG. 12, the kernel of the convolution operation has a first value in the first row and first column, a second value in the first row and second column, a third value in the first row and third column, a fourth value in the first row and fourth column, a fifth value in the first row and fifth column, a sixth value in the second row and first column, a seventh value in the second row and fifth column, an eighth value in the third row and first column, a ninth value in the third row and second column, a tenth value in the third row and third column, an eleventh value in the third row and fourth column, and a twelfth value in the third row and fifth column, and may not have values ​​in the second row and second column, the second row and third column, and the second row and fourth column corresponding to the central pixel group of the current frame.

[0123] As shown in the lower figure of FIG. 11, the impulse response of the convolution operation generated through the kernel of the convolution operation has a first impulse value in the first row and first column, a second impulse value in the first row and second column, a third impulse value in the first row and third column, a fourth impulse value in the first row and fourth column, a fifth impulse value in the first row and fifth column, a sixth impulse value in the second row and first column, a seventh impulse value in the second row and fifth column, an eighth impulse value in the third row and first column, a ninth impulse value in the third row and second column, a tenth impulse value in the third row and third column, an eleventh impulse value in the third row and fourth column, and a twelfth impulse value in the third row and fifth column, and the second row and second column, the second row and third column, and the second row corresponding to the central pixel group of the current frame The 4th column may not have an impulse value.

[0124] As shown in FIG. 13, the kernel of the first extended convolution operation has a first value in the first row and first column, a second value in the first row and fourth column, a third value in the first row and seventh column, a fourth value in the first row and third column, a fifth value in the seventh row and third column, a sixth value in the first row and fifth column, a seventh value in the fourth row and fifth column, an eighth value in the seventh row and fifth column, and may not have values ​​in the third row and third column, the third row and fourth column, and the third row and fifth column corresponding to the central pixel group of the current frame, and may not have values ​​in the second row, the fourth row, the second to third columns, and the fifth to sixth columns.

[0125] As shown in the lower figure of Fig. 11, the impulse response of the first extended convolution operation generated through the kernel of the first extended convolution operation may not have impulse values ​​in the 4th row, 5th column, 4th row, 6th column, and 4th row, 7th column corresponding to the central pixel group of the current frame in the matrix of 7 rows and 11 columns, and the remaining areas may have respective impulse values.

[0126] By applying the kernel of the first extended convolution operation of FIG. 13 to the impulse response of the above convolution operation, it is possible to generate an impulse response of the first extended convolution operation that does not have an impulse value in the 4th row and 5th column, the 4th row and 6th column, and the 4th row and 7th column corresponding to the central pixel group.

[0127] In this way, as the dilated convolution operation is repeated, the kernel size becomes larger and larger, and accordingly, an impulse response can be generated in which the impulse region gradually expands without having an impulse value for the central pixel group.

[0128] Fig. 14 is a detailed block diagram illustrating an example of a device for removing noise in the time-lapse image of Fig. 2b. Fig. 15 is a detailed block diagram illustrating an example of a peripheral frame processing convolutional neural network of Fig. 14. Fig. 16 is a detailed block diagram illustrating an example of a central frame processing convolutional neural network of Fig. 14. Fig. 17 is a diagram illustrating an impulse response of the central frame processing convolutional neural network of Fig. 14.

[0129] Referring to Figures 1 to 17, the peripheral frame processing neural network can utilize a general convolutional neural network structure (e.g., U-Net). Furthermore, the output of the peripheral frame processing neural network can be concatenated with the central frame and provided together as input to the central frame processing neural network.

[0130] The outputs of the central frame processing neural network and the outputs of the peripheral frame processing neural networks are concatenated and then input to an information association neural network with a 1×1 sized convolutional kernel.

[0131] The final output of the neural network is compared with the current frame to compute a loss function. This loss function utilizes a measure of the distance between the two tensors, such as the L1 distance or L2 distance.

[0132] The neural network is trained by repeating the process of updating the parameters within the neural network to minimize the above loss function.

[0133] Once the neural network has completed training, a tensor concatenating past and future frames based on the target frame for noise removal is input to the neural network and the results are obtained. This process is repeated for multiple frames to remove noise from the time-lapse video.

[0134] Figure 14 shows an example of the overall convolutional neural network structure. Figure 15 shows an example of a peripheral frame processing convolutional neural network, and Figure 16 shows an example of a central frame processing convolutional neural network.

[0135] In Fig. 14, Blind spot Network means a network that does not refer to the central pixel of the current frame, and in Fig. 16, Blind spot Conv means a convolution operation that does not refer to the central pixel of the current frame.

[0136] Referring to FIG. 16, the central frame processing convolutional neural network may include a first sequential path that sequentially performs a convolution operation by inputting the peripheral processing values, and a second sequential path that sequentially performs a convolution operation by inputting the current frame.

[0137] The number of convolution operation layers of the first sequential path may be less than the number of convolution operation layers of the second sequential path. For example, the number of convolution operation layers of the first sequential path may be three, and the number of convolution operation layers of the second sequential path may be five.

[0138] The kernel size of the convolution operation of the first sequential path may be larger than the kernel size of the convolution operation of the second sequential path. For example, the kernel size of the convolution operation of the first sequential path may be 5×5, and the kernel size of the convolution operation of the second sequential path may be 3×3.

[0139] The first sequential path may include a first convolution operation layer that receives the peripheral processing value as input and outputs a first convolution result, a second convolution operation layer that receives the first convolution result as input and outputs a second convolution result, and a third convolution operation layer that receives the second convolution result as input and outputs a third convolution result.

[0140] The second sequential path may include a fourth convolution operation layer that receives the current frame as input and outputs a fourth convolution result, a fifth convolution operation layer that receives the fourth convolution result as input and outputs a fifth convolution result, a sixth convolution operation layer that receives the fifth convolution result as input and outputs a sixth convolution result, a seventh convolution operation layer that receives the sixth convolution result as input and outputs a seventh convolution result, and an eighth convolution operation layer that receives the seventh convolution result as input and outputs an eighth convolution result.

[0141] The central frame processing convolutional neural network may further include a ninth convolution operation layer that performs a 1×1 convolution operation and a ReLU operation on the concatenated first convolution result to the eighth convolution result.

[0142] In addition, the central frame processing convolutional neural network may include a shortcut connection. That is, the first convolution result and the input of the central frame processing convolutional neural network that performed the 1×1 convolution operation may be input to the second convolution operation layer. The second convolution result and the input of the central frame processing convolutional neural network that performed the 1×1 convolution operation may be input to the third convolution operation layer. The fourth convolution result and the input of the central frame processing convolutional neural network that performed the 1×1 convolution operation may be input to the fifth convolution operation layer. The fifth convolution result and the input of the central frame processing convolutional neural network that performed the 1×1 convolution operation may be input to the sixth convolution operation layer. The sixth convolution result and the input of the central frame processing convolutional neural network that performed the 1×1 convolution operation may be input to the seventh convolution operation layer. The above-mentioned 8th convolution operation layer may receive the above-mentioned 7th convolution result and the above-mentioned input of the central frame processing convolutional neural network that performed the above-mentioned 1×1 convolution operation.

[0143] When a central frame processing neural network is designed as in Fig. 16, it will have an impulse response as in Fig. 17. The right figure of Fig. 17 is an enlarged view of the central portion of the left figure of Fig. 17.

[0144] As shown in Figure 17, since the center value of the impulse response is 0, each pixel in the input image does not affect the pixels in the same location in the output image.

[0145] Fig. 18 is a diagram showing the noise removal result according to the present embodiment.

[0146] Figure 18a shows a larval zebrafish (small fish) expressing the GCaMP7a calcium indicator under the control of the huc promoter. The upper part of Figure 18b shows raw data containing noise, and the lower part of Figure 18b shows the noise removal result according to the present invention. The upper part of Figure 18c shows raw data containing noise, and the lower part of Figure 18c shows the noise removal result according to the present invention. The left part of Figure 18d shows raw data containing noise, and the right part of Figure 18d shows the noise removal result according to the present invention.

[0147] Referring to Fig. 18, it was confirmed that noise in a time-lapse image of a zebrafish (small fish) brain can be removed very effectively through the present invention.

[0148] Fig. 19 is a diagram showing the noise removal result according to the present embodiment.

[0149] Figure 19 shows voltage signals along an axonal branch. Figure 19(a) shows raw data, and Figure 19(b) shows the noise removal result (SUPPORT denoised) according to the present invention. Figure 19(c) shows voltage traces of the corresponding electrophysiological recording.

[0150] Referring to Figure 19, it was confirmed that the noise in voltage imaging images can be very effectively removed through the present invention. Since the brightness of each cell in voltage imaging data changes rapidly in milliseconds, utilizing existing methods has the problem of removing the signal along with the noise. However, when noise is removed through the present invention, the signal is preserved.

[0151] Fig. 20 is a diagram showing the noise removal result according to the present embodiment.

[0152] In Fig. 20, Noisy means raw data containing noise, SUPPORT Denoised means the noise removal result according to the present invention, and DeepCAD-RT Denoised means the noise removal result according to a conventional methodology that performs noise removal by referring only to past and future frames without referring to the current frame.

[0153] Referring to Fig. 20, it was confirmed that noise in a time-lapse image of a moving C. elegans (a small annelid) can be very effectively removed using the present invention. In the case of C. elegans, since it moves quickly, the signal changes greatly between adjacent frames. Therefore, when noise is removed using a conventional method, numerous artifacts occur in the image, whereas when noise is removed using the present invention, noise is removed without signal distortion.

[0154] Fig. 21 is a diagram showing the noise removal result according to this embodiment.

[0155] In Fig. 21, Low SNR means an image with a low signal-to-noise ratio of the axial slice of Penicillium, and SUPPORT denoised means the result of noise removal of the axial slice of Penicillium according to the present invention.

[0156] Referring to Figure 21, it was confirmed that noise was well removed even when a 3D microscope image obtained through multiple tomographic scans was regarded as a time-lapse image and noise was removed.

[0157] According to this embodiment, noise in a time-lapse image can be effectively removed and time-series information contained in the original image data can be restored.

[0158] Additionally, noise removal from images is possible without separate learning data, without assuming that adjacent frames within a time-lapse image have the same signal components.

[0159] Additionally, by utilizing information contained in the current frame as well as past and future frames, it is possible to effectively remove noise without distorting or removing the signal even when there are rapid changes in the image data or when the camera shakes.

[0160] According to one embodiment of the present invention, the method for removing noise in the time-lapse image can be executed by a computing device.

[0161] According to one embodiment of the present invention, a computer-readable recording medium having recorded thereon a program for executing a method for removing noise in a time-lapse image according to the above embodiments on a computer may be provided. The above-described method can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that executes the program using the computer-readable medium. In addition, the structure of data used in the above-described method can be recorded on a computer-readable medium through various means. The computer-readable medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the medium may be those specially designed and configured for the present invention, or may be known and usable by those skilled in the art in the field of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program instructions include not only machine language codes, such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention.

[0162] Additionally, the method for removing noise in a time-lapse image described above can also be implemented in the form of a computer program or application executed by a computer and stored in a recording medium.

[0163] According to the present invention, noise in time-lapse images can be effectively removed and time-series information contained in the original image data can be restored. Furthermore, noise removal from images can be achieved without additional training data, without assuming that adjacent frames within a time-lapse image have the same signal components. Furthermore, by utilizing information contained in the current frame as well as past and future frames, noise can be effectively removed without distorting or removing the signal, even when there are rapid changes in the image data or when the camera shakes.

[0164] As described above, although the present invention has been described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various modifications and changes may be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

Claims

1. A central frame processing convolutional neural network that receives the current frame of the input image and outputs the central processing value; A peripheral frame processing convolutional neural network that inputs a plurality of past frames adjacent to the current frame and a plurality of future frames adjacent to the current frame and outputs peripheral processing values; and It includes an information combining convolutional neural network that calculates the central processing value and the peripheral processing value and outputs an output image, A device for removing noise in a time-lapse image, characterized in that the central frame processing convolutional neural network does not refer to the central pixel of the current frame.

2. A device for removing noise in a time-lapse image, characterized in that in the first paragraph, the central frame processing convolutional neural network further receives the peripheral processing value as input.

3. In the first paragraph, the peripheral frame processing convolutional neural network A device for removing noise in a time-lapse image, characterized in that it refers to the central pixel of the past frame and the central pixel of the future frame.

4. A device for removing noise in a time-lapse image, characterized in that in the first paragraph, the impulse response of the central frame processing convolutional neural network does not have a central impulse value corresponding to the central pixel of the current frame, but has an impulse value for an impulse area of ​​a predetermined size surrounding the central pixel of the current frame.

5. In the first paragraph, the central frame processing convolutional neural network A convolution operation, comprising a plurality of extended convolution operations that are sequentially arranged and receive the result of the convolution operation, A device for removing noise in a time-lapse image, characterized in that the size of the kernel of the above extended convolution operation is larger than the size of the kernel of the above convolution operation.

6. In the fifth paragraph, the kernel of the convolution operation is A device for removing noise in a time-lapse image, characterized in that it has a first value in the first row and the first column, a second value in the first row and the second column, a third value in the first row and the third column, a fourth value in the first row and the second column, a fifth value in the third row and the second column, a sixth value in the first row and the third column, a seventh value in the second row and the third column, an eighth value in the third row and the third column, and no value in the second row and the second column corresponding to the central pixel of the current frame.

7. In the 6th paragraph, the impulse response of the convolution operation is A device for removing noise in a time-lapse image, characterized in that the device has a first impulse value in the first row and first column, a second impulse value in the first row and second column, a third impulse value in the first row and third column, a fourth impulse value in the first row and second column, a fifth impulse value in the third row and third column, a sixth impulse value in the first row and third column, a seventh impulse value in the second row and third column, an eighth impulse value in the third row and third column, and no impulse value in the second row and second column corresponding to the central pixel of the current frame.

8. In the 6th paragraph, the kernel of the first extended convolution operation is A device for removing noise in a time-lapse image, characterized in that it has a first value in the first row and the first column, a second value in the first row and the third column, a third value in the first row and the fifth column, a fourth value in the first row and the third column, a fifth value in the fifth row and the fifth column, a sixth value in the first row and the fifth column, a seventh value in the third row and the fifth column, an eighth value in the fifth row and the fifth column, no value in the third row and the third column corresponding to the central pixel of the current frame, and no values ​​in the second row, the fourth row, the second column, and the fourth column.

9. In the 8th paragraph, the impulse response of the first extended convolution operation is A device for removing noise in a time-lapse image, characterized in that in a matrix of 7 rows and 7 columns, 4 rows and 4 columns corresponding to the central pixel of the current frame do not have an impulse value, and the remaining areas have respective impulse values.

10. In the 8th paragraph, the kernel of the second extended convolution operation is A device for removing noise in a time-lapse image, characterized in that it has a first value in the first row and first column, a second value in the first row and fifth column, a third value in the first row and ninth column, a fourth value in the first row and ninth column, a fifth value in the fifth row and ninth column, a sixth value in the first row and ninth column, a seventh value in the fifth row and ninth column, an eighth value in the ninth row and ninth column, no value in the fifth row and ninth column corresponding to the central pixel of the current frame, and no values ​​in the second to fourth rows, the sixth to eighth rows, the second to fourth columns, and the sixth to eighth columns.

11. A device for removing noise in a time-lapse image, characterized in that in the first paragraph, the central frame processing convolutional neural network does not refer to the central pixel group of the current frame including the central pixel of the current frame and a plurality of surrounding pixels adjacent to the central pixel.

12. In the 11th paragraph, the central frame processing convolutional neural network A convolution operation, comprising a plurality of extended convolution operations that are sequentially arranged and receive the result of the convolution operation, A device for removing noise in a time-lapse image, characterized in that the size of the kernel of the above extended convolution operation is larger than the size of the kernel of the above convolution operation.

13. In the 12th paragraph, the kernel of the convolution operation is A device for removing noise in a time-lapse image, characterized in that it has a first value in row 1, column 1, a second value in row 1, column 2, a third value in row 1, column 3, a fourth value in row 1, column 4, a fifth value in row 1, column 5, a sixth value in row 2, column 1, a seventh value in row 2, column 5, an eighth value in row 3, column 1, a ninth value in row 3, column 2, a tenth value in row 3, column 3, an eleventh value in row 3, column 4, and a twelfth value in row 3, column 5, and has no values ​​in row 2, column 2, row 2, column 3, and row 2, column 4 corresponding to the central pixel group of the current frame.

14. In the 13th paragraph, the impulse response of the convolution operation is A device for removing noise in a time-lapse image, characterized in that it has a first impulse value in the first row and first column, a second impulse value in the first row and second column, a third impulse value in the first row and third column, a fourth impulse value in the first row and fourth column, a fifth impulse value in the first row and fifth column, a sixth impulse value in the first row and second column, a seventh impulse value in the second row and second column, an eighth impulse value in the first row and third column, a ninth impulse value in the second row and third column, a tenth impulse value in the third row and third column, an eleventh impulse value in the fourth row and third column, and a twelfth impulse value in the third row and fifth column, and has no impulse values ​​in the second row and second column, the second row and third column, and the second row and fourth column corresponding to the central pixel group of the current frame.

15. In the 13th paragraph, the kernel of the first extended convolution operation is A device for removing noise in a time-lapse image, characterized in that it has a first value in the first row and column, a second value in the first row and column, a third value in the first row and column, a fourth value in the first row and column, a fifth value in the third row and column, a sixth value in the first row and column, a seventh value in the fourth row and column, an eighth value in the fifth row and column, a value not in the third row and column, the third row and column, and the third row and column, and the fifth and sixth columns corresponding to the central pixel group of the current frame, and a value not in the second row, the fourth row, the second to third columns, and the fifth to sixth columns.

16. In the 15th paragraph, the impulse response of the first extended convolution operation is A device for removing noise in a time-lapse image, characterized in that in a matrix of 7 rows and 11 columns, 4 rows and 5 columns, 4 rows and 6 columns, and 4 rows and 7 columns corresponding to the central pixel group of the current frame do not have impulse values, and the remaining areas have respective impulse values.

17. In the first paragraph, the central frame processing convolutional neural network A first sequential path that sequentially performs a convolution operation by receiving the above peripheral processing values; and A device for removing noise in a time-lapse image, characterized in that it includes a second sequential path that sequentially performs a convolution operation by receiving the current frame as input.

18. A device for removing noise in a time-lapse image, characterized in that the number of convolution operation layers of the first sequential path is less than the number of convolution operation layers of the second sequential path in the 17th paragraph.

19. A device for removing noise in a time-lapse image, characterized in that in paragraph 18, the number of convolution operation layers of the first sequential path is 3 and the number of convolution operation layers of the second sequential path is 5.

20. A device for removing noise in a time-lapse image, characterized in that, in the 17th paragraph, the kernel size of the convolution operation of the first sequential path is larger than the kernel size of the convolution operation of the second sequential path.

21. A device for removing noise in a time-lapse image, characterized in that in the 20th paragraph, the size of the kernel of the convolution operation of the first sequential path is 5×5, and the size of the kernel of the convolution operation of the second sequential path is 3×3.

22. In the 17th paragraph, the central frame processing convolutional neural network A first convolution operation layer that receives the above peripheral processing values ​​as input and outputs a first convolution result; A second convolution operation layer that receives the first convolution result as input and outputs the second convolution result; A third convolution operation layer that receives the second convolution result as input and outputs a third convolution result; A fourth convolution operation layer that receives the current frame as input and outputs the fourth convolution result; A fifth convolution operation layer that receives the fourth convolution result as input and outputs the fifth convolution result; A sixth convolution operation layer that receives the fifth convolution result as input and outputs the sixth convolution result; A seventh convolution operation layer that receives the sixth convolution result as input and outputs the seventh convolution result; and A device for removing noise in a time-lapse image, characterized in that it includes an eighth convolution operation layer that receives the seventh convolution result as input and outputs an eighth convolution result.

23. In the 22nd paragraph, the central frame processing convolutional neural network A device for removing noise in a time-lapse image, characterized in that it further includes a ninth convolution operation layer that performs a 1×1 convolution operation and a ReLU operation on the concatenated first convolution result to the eighth convolution result.

24. In the 22nd paragraph, the second convolution operation layer receives the first convolution result and the input of the central frame processing convolution neural network that performed the 1×1 convolution operation, The third convolution operation layer receives the second convolution result and the input of the central frame processing convolution neural network that performed the 1×1 convolution operation. The fifth convolution operation layer receives the fourth convolution result and the input of the central frame processing convolution neural network that performed the 1×1 convolution operation. The sixth convolution operation layer receives the fifth convolution result and the input of the central frame processing convolution neural network that performed the 1×1 convolution operation. The 7th convolution operation layer receives the 6th convolution result and the input of the central frame processing convolution neural network that performed the 1×1 convolution operation. A device for removing noise in a time-lapse image, characterized in that the 8th convolution operation layer receives the 7th convolution result and the input of the central frame processing convolution neural network that performed the 1×1 convolution operation.

25. A device for removing noise in a time-lapse image, characterized in that, in the first paragraph, the information-combined convolutional neural network performs a 1×1 convolution operation on the central processing value and the peripheral processing value.

26. In paragraph 1, where i is the time index, k is the spatial index, and L is the loss function. go is a neural network parameterized by , is the central pixel, are the neighboring pixels of the central pixel excluding the central pixel, When the correct data of the central pixel above is The statistical prediction model used in the device for removing noise in the above time-lapse video is A device for removing noise in a time-lapse image, characterized in that it is implemented by solving an optimization problem.

27. A central frame processing convolutional neural network that receives the current frame of the input image and outputs the central processing value; A peripheral frame processing convolutional neural network that inputs a plurality of past frames adjacent to the current frame and a plurality of future frames adjacent to the current frame and outputs peripheral processing values; and It includes an information combining convolutional neural network that calculates the central processing value and the peripheral processing value and outputs an output image, The median value of the kernel of the central frame processing convolutional neural network corresponds to the central pixel of the current frame, A device for removing noise in a time-lapse image, characterized in that the median value of the above kernel is set to 0.

28. A step of receiving the current frame of the input image and generating a central processing value; A step of generating a peripheral processing value by receiving a plurality of past frames adjacent to the current frame and a plurality of future frames adjacent to the current frame; and It includes a step of calculating the central processing value and the peripheral processing value and outputting an output image, A method for removing noise in a time-lapse image, characterized in that the central frame processing convolutional neural network that generates the central processing value does not refer to the central pixel of the current frame.

29. A method for removing noise in a time-lapse image, characterized in that in paragraph 28, the central frame processing convolutional neural network further receives the peripheral processing value as input.

30. In the 28th paragraph, the peripheral frame processing convolutional neural network that generates the peripheral processing value is A method for removing noise in a time-lapse image, characterized by referencing the central pixel of the past frame and the central pixel of the future frame.

31. A computer-readable recording medium having recorded thereon a program for executing the method of any one of claims 28 to 30 on a computer.

Images