Image processing method, image processing apparatus, image processing system, and program

Abstract

To provide an image processing method capable of performing image processing of a moving image using a diffusion model while reducing an increase in a used memory amount and an adverse effect in moving image generation.SOLUTION: A step of dividing an input moving image into a plurality of input frame groups, a step of acquiring a plurality of output frame groups by converting each of the plurality of input frame groups using a diffusion model, and a step of acquiring an output moving image by combining the plurality of output frame groups, wherein the plurality of input frame groups include a first input frame group including a first input frame and a second input frame group including a second input frame; The second input frame is temporally adjacent to the first input frame, and in the step of acquiring the plurality of output frame groups, the second input frame group is converted to acquire the second output frame group by updating the noise data based on the first output frame group acquired by converting the first input frame group and the second input frame group.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing method, an image processing device, an image processing system, and a program. [Background technology]

[0002] Patent Document 1 discloses a method for generating moving images using a diffusion model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2024 / 0087179 [Non-patent literature]

[0004] [Non-Patent Document 1] arXiv:2006.11239(Denoising Diffusion Probabilistic Models) Summary of the Invention [Problem to be solved by the invention]

[0005] Image processing of moving images using a diffusion model involves randomness (moving images are generated based on random numbers). For this reason, when processing moving images using a diffusion model, it is generally necessary to generate the moving images in a batch by processing all frames that make up the moving images in a batch in order to ensure temporal consistency of the moving images. However, generating moving images in a batch increases memory usage as the resolution and number of frames of the moving images increase. Patent Document 1 does not disclose a solution to such drawbacks in generating moving images.

[0006] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing method that can perform image processing of moving images using a diffusion model while reducing the increase in memory usage and the adverse effects that occur when generating moving images. [Means for solving the problem]

[0007] An image processing method as one aspect of the present invention includes the steps of: dividing an input video into a plurality of input frame groups; obtaining a plurality of output frame groups by transforming each of the plurality of input frame groups using a diffusion model; and obtaining an output video by combining the plurality of output frame groups, wherein the plurality of input frame groups include a first input frame group including a first input frame and a second input frame group including a second input frame, the second input frame being temporally adjacent to the first input frame; and in the step of obtaining the plurality of output frame groups, converting the first input frame group to obtain a second output frame group by updating noise data based on the first output frame group and the second input frame group.

[0008] Other objects and features of the present invention are illustrated in the following examples. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an image processing method that can perform image processing of moving images using a diffusion model while reducing an increase in memory usage and adverse effects on moving image generation. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram showing a flow of learning of a neural network in the first embodiment. [Figure 2] 1 is a block diagram of an image processing system according to a first embodiment. [Figure 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4]4 is a flowchart relating to learning of a neural network in the first embodiment. [Figure 5] 10 is a flowchart relating to generation of an output moving image using a neural network in the first embodiment. [Figure 6] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 7] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 8] FIG. 10 is a block diagram of an image processing system according to a third embodiment. [Figure 9] 11 is a flowchart showing the generation of an output moving image using a neural network in the third embodiment. [Figure 10] FIG. 1 is an explanatory diagram of the gist of each embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted. (Explanation of the diffusion model) First, before providing a detailed description of each embodiment, a diffusion model and an outline of each embodiment will be described. In each embodiment, a diffusion model is used to perform frame group segmentation processing without causing any problems in image processing of video using the diffusion model. A diffusion model is a type of generative AI (artificial intelligence) primarily used to generate images and videos. By sampling from a distribution of images (videos) conditioned by inputs such as images, videos, and text, a high-quality image (video) is generated based on the input image, video, or text. For example, Stable Diffusion uses a diffusion model to generate realistic images that appear to have been drawn by a human in response to text entered by a user. Note that in each embodiment, segmentation processing based on frame groups means dividing an input video into multiple shorter videos (frame groups) and sequentially processing the frame groups. A frame group is a set of multiple frames of a video.

[0012] The diffusion model is disclosed in Non-Patent Document 1 and consists of two processes: a forward diffusion process and a reverse diffusion process. In the diffusion process, Gaussian noise is gradually added to a target image (target video) to remove information from the target image (target video), resulting in a pure noise image. In the reverse diffusion process, a pure noise image (noise video) is started, and the target image (target video) is generated (estimated) by iteratively removing noise through conditioning with image input, video input, and text input. The reverse diffusion process can be trained using a neural network (machine learning model) that removes noise by inputting a noise-added image (video) and a conditioned image (or signal such as video or text). In each embodiment, a video is generated by conditioning with video input, so the input video is essentially converted into an output video. Details will be described later. Although Non-Patent Document 1 does not generate a video, video image processing can be performed by performing similar diffusion and reverse diffusion processes on data (video) that combines multiple consecutive images.

[0013] The diffusion process is given by the following equation (1):

[0014]

number

[0015] where y t-1 and y t represents the step (iteration number) t-1 of the diffusion process q given by the Markov chain and the intermediate video of the next step t (t = 1 to T), respectively. N is a standard normal distribution (Gaussian distribution), α t (0<α t <1) is step t and y t-1 respectively represent the variance of the Gaussian noise to be added to By transforming (marginalizing) equation (1), the intermediate video y given the target video y0 is t The distribution of is given by the following equation (2):

[0016]

number

[0017] where γ t Add it to the target video y0 and t In other words, equation (2) expresses the variance of the Gaussian noise to obtain the intermediate video y t-1 Add Gaussian noise to the intermediate video y t At each step, we repeat the process of obtaining the intermediate video y t This indicates that it is possible to obtain

[0018] Further transforming equations (1) and (2), we obtain the target video y0 and the intermediate video y t Given an intermediate video y t-1 The posterior distribution of is given by the following equation (3):

[0019]

number

[0020] where μ and σ 2 is the target video y0 and the intermediate video y t Given and the intermediate video y t-1 In other words, equation (3) expresses the mean and variance of the intermediate video y t and the mean μ obtained from the target video y0, with the variance σ 2 By adding Gaussian noise with t-1 In other words, the diffusion process can be reversed. In the de-diffusion process described next, the relationship in equation (3) is used.

[0021] The despreading process is given by the following equation (4).

[0022]

number

[0023] Here, the inverse diffusion process p(t=T~1) is defined as an inverse Markov chain that proceeds in the reverse direction of the diffusion process q. Note that, following the diffusion process q, the pure noise video y T We start with (Gaussian noise sampled from a standard normal distribution, the second equation of equation (4)).

[0024] The mean μθ of the Gaussian distribution at each step of the de-diffusion process is calculated from the posterior distribution of Equation (3), but the target motion image y0 is required for this calculation. However, the pure noise motion image y T In the despreading process starting from , the target video y0 is generally unknown.

[0025] Therefore, in the diffusion model, a neural network (machine learning model) is trained to remove noise from a noise video in which Gaussian noise with various variances is added to the target video y0, and then the intermediate video y t The target video y0^ is estimated by removing noise from the

[0026] The target motion image y0^ estimated by removing noise in the de-diffusion process is given by the following equation (5).

[0027]

number

[0028] Here, f is a noise removal neural network with θ as a parameter, and x is a condition used for noise removal. For example, in Stable Diffusion, a vector with embedded text is used as a condition for noise removal. Also, the intermediate image (video) y t By concatenating the conditioned image (video) and inputting it into a neural network, noise is removed by conditioning with the conditioned image (video).

[0029] In addition, the noise reduction neural network uses a noise level (noise strength) γ t is introduced and noise is removed according to the noise level. Note that in each embodiment, the variance of the Gaussian noise added in the diffusion process is used as the noise level, but this is not limiting. A method for inputting the noise level into the noise removal neural network will be described later.

[0030] By substituting the estimated target video y0^ into equation (3), the mean of the posterior distribution is given parametrically by the following equation (6).

[0031]

number

[0032] The variance of the Gaussian noise is given by the variance αt of the Gaussian noise added in the diffusion process. Using this, each step of the de-diffusion process is given by the following equation (7).

[0033]

number

[0034] By repeating this procedure from t=T to 1, we obtain a pure noise video y T We can generate the output video y0' (i.e., sample from the distribution of target video y0 conditioned on x) from the estimated target video y0^ or the mean μ of the posterior distribution. θ is calculated at each iterative step, and the target video y0^ or the mean μθ of the posterior distribution calculated at step t=1 is the output video y0'. The output video y0' is not imparted with the noise represented by the second term of the first equation in equation (7). This can also be seen from the fact that the second term of the second equation and the third equation in equation (3) are 0. The diffusion model is also known as the denoised diffusion probabilistic model (DDPM).

[0035] In the prior art, the intermediate video y t The estimated target video y0^ is then converted into the intermediate video y t-1 However, in the de-diffusion process, Gaussian noise (standard normal random number) ε t Therefore, when a video is divided into frame groups using a conventional diffusion model, a random texture is generated for each frame group, and there is no continuity in the texture between the frame groups. As a result, when the output frame group y0' is combined, the video changes discontinuously at the frame transitions, resulting in a video with no temporal consistency.

[0036] Therefore, in each embodiment, instead of equation (7), the intermediate frame group y t The estimated target frame set y0^ is then converted into the intermediate frame set y t-1 is calculated. Note that equation (8) assumes that the original video I is divided into multiple frame groups x, which are then used as conditioned frame groups x and sequentially processed using the diffusion model (division processing is performed for each frame group). Furthermore, equation (8) assumes that when dividing the original video I into frame groups, there are some overlapping frames between temporally adjacent frame groups (corresponding frames exist). Hereinafter, adjacent frame groups refer to frame groups that are temporally adjacent.

[0037]

number

[0038] Here, z is the output frame set y0' obtained by processing the conditioned frame set x and the adjacent frame sets using the diffusion model, and m1 and m2 are the adjacent frame set z and the intermediate frame set y t and the corresponding frames y t , the mask extracted from z, ∇ is the processed adjacent frame set z and intermediate frame set y tThis is an operator that calculates the gradient by taking the difference between the corresponding parts of the adjacent frames z and the intermediate frames yt. In other words, in addition to approaching the distribution of the target frame group conditioned by the frame group x through each step of the de-diffusion process given by equation (7), a constraint is added to each step of the de-diffusion process so that the corresponding parts of the processed adjacent frame group z and the intermediate frame group yt approach each other.

[0039] Alternatively, the gradient may be calculated from the processed adjacent frame set z and the estimated target frame set y0^, and the mean μ of the posterior distribution given by equation (6) is used. θ Instead of using equation (8), the intermediate frame group y t-1 After calculating the adjacent frame group z and the intermediate frame group y t-1 The intermediate frame set y t-1 The calculated gradient may be weighted. In addition, depending on the position and order of extracting the frame group x from the original video I, there may be cases where the processed adjacent frame group z does not exist. In such cases, formula (7) can be used as usual. If there are multiple processed adjacent frame groups z, the intermediate frame group y t Instead of using equation (8), we can extract all the parts corresponding to the intermediate frame group y t-1 After calculating the intermediate frame group y t-1 The corresponding frames of z may be replaced based on the processed adjacent frames z. When replacing, noise based on the de-spreading step may be added to the processed adjacent frames z.

[0040] The adjacent frame group z is a processed frame group, and it is preferable that the intermediate frame group (adjacent intermediate frame group) used when generating the adjacent frame group z is not used in the de-diffusion process. When using adjacent intermediate frame groups to obtain a time-consistent output video, the adjacent intermediate frame groups must be stored in order to generate each output frame group. Since the adjacent intermediate frame groups are generated at each step, storing them would require an extremely large amount of data. By not using adjacent intermediate frame groups, the data volume, which is one of the issues in the diffusion model, can be reduced, and the video generation process of each embodiment can be performed with a small amount of memory.

[0041] Although it is preferable for adjacent frames to overlap, they do not necessarily have to. If they do not overlap, the intermediate frame group y can be created by extrapolating multiple frames at the ends of the processed adjacent frame groups using linear interpolation or the like. t Then, we generate a frame corresponding to the frame z.

[0042] It is preferable that the number of overlapping frames with an adjacent frame group be two or more. If the number of overlapping frames is one, the end frame of the adjacent frame group and the end frame of the conditioning frame group can be brought closer together, but the motion of the adjacent frame group and the conditioning frame group cannot be brought closer together. This is because motion is determined by the positional shift between multiple frames. Therefore, in a video in which multiple frame groups are combined, the number of overlapping frames needs to be multiple in order to maintain temporal consistency in both the texture and motion within the frames.

[0043] In each embodiment, the intermediate frame set y tand corresponding frames of adjacent frame group z processed by the diffusion model gradually approach each other. Therefore, in the final output frame group y0' obtained by repeating the de-diffusion process (t=T→1), the adjacent frame group processed by the diffusion model and the corresponding frames ideally match. As a result, even when the output frame groups resulting from processing in each embodiment are combined, temporal consistency is maintained between the adjacent frame groups that have already been processed, and division processing for each frame group can be performed using the diffusion model without any problems.

[0044] Next, an example of upscaling (increasing resolution) a video by dividing it into frames using a diffusion model will be described with reference to Fig. 10. Here, upscaling refers to image processing that generates a high-resolution video by estimating and enlarging high-frequency components that cannot be expressed in a low-resolution video. Fig. 10 is an explanatory diagram of the gist of each embodiment.

[0045] In each embodiment, a low-resolution input video is first acquired, and input frame groups (corresponding to x in the formula) are extracted from the bicubic-interpolated video (corresponding to I in the formula), and sequential processing (division processing for each frame group) is performed using a diffusion model. When extracting input frame groups from the bicubic-interpolated video, each frame group is acquired so that it partially overlaps with adjacent frame groups (having overlapping frames (common frames)). In Figure 10, the bicubic-interpolated input video is divided into input frame groups of four frames each, so that each frame overlaps with an adjacent frame group by two frames. Therefore, for example, the second input frame group from the beginning of the input video has frames that correspond (overlap) with the first and third input frame groups. Also, Figure 10 shows an example of sequential processing, starting with the input frame group closest to the beginning of the video and processing the input frame groups in chronological order (the first frame group, the second frame group, the third frame group). The black arrows in the figure indicate the order of sequential processing.

[0046] Next, the input frame group is processed according to equation (8). Note that Fig. 10 shows an example of processing the second input frame group. Therefore, it is assumed that the first frame group has already been processed by the diffusion model (a corresponding processed output frame group exists). On the other hand, it is assumed that the other input frame groups (the third and subsequent ones) have not yet been processed by the diffusion model (a corresponding processed output frame group does not exist).

[0047] First, the current intermediate frame set (y t In the first step, the intermediate frame group (corresponding to y T The number of frames and the horizontal and vertical pixel count of the noise video are the same as those of the interpolated video.

[0048] Next, the input frame group, along with the current intermediate frame group, is input to the neural network for noise removal. Information regarding the variance of the noise contained in the intermediate frame group (i.e., the noise level, which corresponds to γT in the formula) is also input to the neural network. The input frame group is also linked to the current intermediate frame group in the channel direction and input to the neural network, where it is used as a condition for noise removal. The information regarding the noise variance uses a value determined in advance for each iteration of the de-diffusion process. For example, the maximum and minimum values ​​of the variance of the Gaussian noise contained in the intermediate frame group can be determined, and a value determined linearly from the number of iterations can be used. The neural network is also trained in advance so that it can perform the noise removal given by equation (5).

[0049] Next, the mean of the posterior distribution (μ in the formula) given by formula (5) is calculated from the noise-removed frame group (corresponding to y0^ in the formula). θ (equivalent to) is calculated.

[0050] Next, the gradient given by Equation (8) is calculated from the denoised frame group and the processed adjacent frame group (corresponding to z in the formula). The adjacent frame group is the output frame group resulting from processing the first input frame group already processed by the diffusion model. In FIG. 10, a mask (not shown, corresponding to m1 and m2 in the formula) is used to extract corresponding frames between the output frame group obtained by processing the first adjacent input frame group and the second input frame group currently being processed, and then the gradient is calculated. Note that when processing the third frame group, the second frame group becomes the adjacent frame group. However, there are cases where the second frame group is combined with the first output frame group immediately after processing. In this case, the frame corresponding to the second output frame group can be used from the frame group obtained by combining the first output frame group and the second output frame group. Furthermore, as described with reference to Equation (8), the gradient may be calculated using the current intermediate frame group or the average of the posterior distribution, rather than the denoised frame group.

[0051] Next, the intermediate frame set (y t-1 The procedure according to equation (8) is repeated a predetermined number of times for each input frame group to obtain an output frame group (corresponding to y0' above).

[0052] The above process of obtaining an output frame group is performed sequentially on multiple input frame groups extracted from the bicubic interpolated video. The resulting multiple upscaled output frame groups are then combined. In other words, by combining the output frame groups, a high-resolution video (output video) is obtained in which each frame of the low-resolution video (input video) is upscaled. In this case, for overlapping frames between frame groups, one can be removed and the other used. Alternatively, instead of removing the overlapping frames, the overlapping frames can be replaced with a single frame obtained by weighted averaging. As described above, according to each embodiment, an upscaled image that has been subjected to division processing for each frame group using a diffusion model can be generated without any drawbacks associated with the frame group division processing.

[0053] In conventional diffusion models, including Patent Document 1, when split processing is performed for each frame group, frame discontinuity (temporal inconsistency) occurs between adjacent processed frame groups when combining the frame groups for the reasons described above. Therefore, in order to process video with high resolution and a large number of frames without any problems using conventional diffusion models, it is necessary to process the video in a batch, even if the video is high resolution and has a large number of frames. However, batch processing of high resolution and a large number of frames requires expensive graphics processing units (GPUs) with large memory, which increases the financial burden on users. In contrast, each embodiment can split and process video with high resolution and a large number of frames using a diffusion model without any problems, so that even an inexpensive GPU can process the video, reducing the financial burden on users. Even if data is output from GPU memory to CPU memory or a recording medium once, the capacity of the CPU memory or recording medium is required, and the processing time increases due to the communication speed between the GPU memory and the CPU memory or recording medium.

[0054] Here, the input video is divided into frame groups for processing. However, it can also be divided into spatial directions (horizontal and vertical directions of the image) for processing. This further reduces the memory capacity required for processing. In this case, each frame group is further divided into subregions in the spatial direction, and multiple subregion frame groups are generated from the same frame group of the input video. Just as overlapping frames are created between multiple frame groups, overlapping pixel regions between multiple subregion frame groups in the spatial division can also be used to generate video with spatial consistency. In other words, in the dediffusion process, it is effective to use adjacent frame groups in the spatial direction, just as frame groups adjacent in the temporal direction are used. For example, for the second term of equation (8), gradients can be calculated and added from corresponding pixel regions of subregion frame groups adjacent in the spatial direction.

[0055] In each embodiment, an input video is converted into an output video. While the method of each embodiment can generate an output video that maintains temporal consistency at the seams between frames, this presupposes that the input video has temporal consistency. In other words, by using the temporal consistency of the input video as a base and providing temporal consistency at the seams between frames using the method of each embodiment, it is possible to provide temporal consistency to the output video after conversion. Therefore, it is difficult to generate video that is temporally consistent at all times using video generation processes that do not use images for each time point, such as video generation from text or video generation from images.

[0056] The image processing method described above is an example, and the embodiments are not limited to this. Details of other image processing methods will be described in the following embodiments.

[0057] [Example 1] First, an image processing system according to a first embodiment of the present invention will be described. In this embodiment, an image processing system is trained and executed to divide a low-resolution input video into frame groups, and generate a high-resolution frame group by upscaling the obtained low-resolution video using a diffusion model.

[0058] Fig. 2 is a block diagram of an image processing system 100 in this embodiment. Fig. 3 is an external view of the image processing system 100. The image processing system 100 includes a learning device 101, an imaging device 102, an image estimation device (image processing device) 103, a display device 104, a recording medium 105, an input device 106, an output device 107, and a network 108.

[0059] The learning device 101 includes a storage unit 101a, an acquisition unit 101b, a generation unit 101c, and a learning unit 101d.

[0060] The imaging device 102 has an optical system 102a and an imaging element 102b. The optical system 102a collects light incident on the imaging device 102 from the subject space. The imaging element 102b receives an optical image of the subject formed via the optical system 102a to acquire a captured video (input video). The imaging element 102b is a charge coupled device (CCD) sensor, a complementary metal-oxide semiconductor (CMOS) sensor, or the like. Information regarding the shooting conditions of the captured image (such as the pixel pitch of the imaging element 102b, the type of optical low-pass filter, and ISO sensitivity) can be acquired along with the image. Development conditions for the captured image (such as noise reduction strength, sharpness strength, and image compression rate) can also be acquired along with the image. This information acquired along with the image can also be transmitted, along with the video, to an input acquisition unit 103b of the image estimation device 103 (described later). A storage unit for storing the acquired video, a display unit for displaying it, a transmission unit for transmitting it to an external device, an output unit for storing it in an external storage medium, and the like are not shown. A control unit that controls each unit of the imaging device 102 is also not shown.

[0061] The image estimation device 103 includes a storage unit 103a, an input acquisition unit 103b, an initialization unit 103c, an output acquisition unit 103d, a noise reduction unit 103e, and an update unit 103f. The input acquisition unit 103b acquires a low-resolution frame group (input frame group) from a captured video (more precisely, a bicubic interpolated video of the captured video). The initialization unit 103c then initializes the current intermediate frame group with Gaussian noise. The noise reduction unit 103e estimates a noise-removed frame group from the current intermediate frame group using a neural network conditioned on the low-resolution frame group. The output acquisition unit 103d acquires a frame group (adjacent frame group) extracted from frames temporally adjacent to the low-resolution frame group in the captured video and already processed using a diffusion model. The update unit 103f updates the current intermediate frame group using the noise-removed frame group and the adjacent frame group.

[0062] This procedure is repeated to perform image processing to generate a final upscaled high-resolution frame group (output frame group). Details of the method for initializing the current intermediate frame group in the initialization unit, the method for removing noise from the current intermediate frame group using a neural network in the noise removal unit, and the method for updating the current intermediate frame group from the noise-removed frame group and adjacent frame groups in the update unit will be described later.

[0063] The low-resolution frame group may be a frame group extracted from captured images taken by the imaging device 102, or may be extracted from images stored in the recording medium 105. Alternatively, a frame group that has already been divided into frame groups from a moving image and stored in the recording medium 105 may be used. Some frames from the moving image may be temporally trimmed and used.

[0064] The weight information of the neural network used for noise removal is read from the storage unit 103a. The weights are learned by the learning device 101, and the image estimation device 103 reads the weight information from the storage unit 101a via the network 108 in advance and stores it in the storage unit 103a. The stored weight information may be the weight's numerical value itself or may be in an encoded format. Details of the neural network weight learning method and the image processing method using a diffusion model that uses a neural network will be described later.

[0065] The upscaled high-resolution frame group is output to at least one of a display device 104, a recording medium 105, and an output device 107 after combining the frame groups. The display device 104 is, for example, an LCD display or a projector. A user can check the frame group being processed via the display device 104 and perform video editing work via the input device 106. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, etc. The input device 106 is, for example, a keyboard or a mouse, etc. The output device 107 is, for example, a printer, etc.

[0066] Next, a weight learning method (a method for manufacturing a trained model) executed by the learning device 101 in this embodiment will be described with reference to Fig. 1 and Fig. 4. Fig. 1 is a diagram showing the flow of learning weights of a neural network (machine learning model). Fig. 4 is a flowchart related to weight learning. Each step in Fig. 4 is mainly executed by the acquisition unit 101b, the generation unit 101c, or the learning unit 101d.

[0067] First, in step S101, the acquisition unit 101b acquires a low-resolution frame group (degraded frame group) 201 and a corresponding high-resolution frame group (correct frame group) 200, as shown in FIG. 1 . In this embodiment, the frame group is a divided video having a predetermined number of frames. For example, the low-resolution frame group may be 128 x 128 x 3 pixels x 10 frames (corresponding to the number of pixels and frames in the vertical x horizontal x depth channel directions), and the corresponding high-resolution frame group may be 256 x 256 x 3 pixels x 10 frames. In this case, since the vertical and horizontal sizes are doubled, the upscaling factor is 2 (4 times the number of pixels). Note that the upscaling factor is not limited to 2 and may be any factor as long as the low-resolution frame group and the corresponding high-resolution frame group can be acquired.

[0068] Alternatively, a low-resolution frame group and a corresponding high-resolution frame group may be obtained by capturing images of the same subject using optical systems with different focal lengths and cutting out corresponding portions of the two resulting moving images. Alternatively, a corresponding low-resolution frame group may be generated by downsampling a high-resolution frame group. Alternatively, a corresponding high-resolution frame group may be generated by numerical calculation, with less influence (aberration, diffraction) of an equivalent low-resolution frame group captured by the imaging device 102 and the optical system 102a. In this embodiment, a low-resolution frame group and a corresponding high-resolution frame group are generated by numerical calculation, but the present invention is not limited to this.

[0069] Next, in step S102, the generation unit 101c generates an interpolated frame group 202 by interpolating and enlarging the low-resolution frame group 201 to have the same number of pixels as the corresponding high-resolution frame group 200. In this embodiment, bicubic interpolation is used as the interpolation method, but this embodiment is not limited to this, and nearest neighbor interpolation or bilinear interpolation may also be used.

[0070] Next, in step S103, the generating unit 101c generates a noise frame group 203 by adding noise to the high resolution frame group 200.

[0071] Next, in step S104, the learning unit 101d uses a neural network to generate a noise-removed frame group 204 by removing noise from the noise frame group 203. The noise-removed frame group 204 is generated based on the interpolated frame group 202, the noise frame group 203, and information about the strength of noise (noise level) added to the noise frame group 203. Note that the noise-removed frame group 204 is an estimate of the high-resolution frame group 200, and ideally, the two would match. In this embodiment, the variance of the added Gaussian noise is used as the noise level, but this is not limited thereto. Furthermore, noise removal is performed according to the strength of the noise added to the noise frame group 203, but a method for inputting the noise level (Gaussian noise variance) to the neural network will be described later. In this embodiment, the interpolated frame group 202 and the noise frame group 203 are concatenated in the channel direction and input to the neural network, but this is not limited thereto.

[0072] Next, in step S105, the learning unit 101d updates the weights of the neural network based on the error between the high-resolution frame group 200 and the noise-removed frame group 204, which is an estimate thereof (performs neural network learning). Here, the weights include the filter components and biases of each layer. In this embodiment, back propagation is used to update the weights, but this is not limitative. In mini-batch learning, the error between the high-resolution frame group 200 and the corresponding noise-removed frame group 204 is calculated, and the weights are updated. For example, the L2 norm or the L1 norm may be used as the loss function. The weight update method (learning method) is not limited to mini-batch learning, and batch learning or online learning may also be used.

[0073] Next, in step S106, the learning unit 101d determines whether learning (weight update) is complete. Completion can be determined by, for example, whether the number of iterations of learning (weight update) has reached a specified value, or whether the amount of change in weight during update is smaller than a specified value. If it is determined that learning is incomplete, the process returns to step S101, and multiple new low-resolution frame groups 201 and corresponding high-resolution frame groups 200 are acquired. On the other hand, if it is determined that learning is complete, the learning device 101 ends learning and saves weight information in the storage unit 101a.

[0074] In this embodiment, a method in image space (dealing with pixel values ​​of images and videos) has been described as a method for learning neural network weights, but the present invention is not limited to this. Learning may also be performed using a method in latent space (dealing with feature maps) from the low-resolution frame group 201 and the interpolated frame group 202. In this case, learning can be performed using a feature map obtained using a separately prepared neural network and a noise frame group 203 obtained by adding noise to a feature map obtained from the high-resolution frame group 200 using a separately prepared neural network. However, it is more preferable to perform learning in image space. In other words, it is preferable to process the tensor of pixel values ​​directly using a diffusion model. By being able to directly compare pixel values ​​of overlapping frames, it is possible to more accurately reduce adverse effects and obtain output videos with temporal consistency. Furthermore, the intermediate frame group y output by equation (8) during estimation can be used to t Since -1 is the pixel value itself, the intermediate frame set y t-1 It is also possible to replace the pixel values ​​of the output frame group y0' and the adjacent frame group so that they are equal to or less than a specified value. It is also possible to replace the pixel values ​​of the output frame group y0' and the adjacent frame group so that they are equal to or less than a specified value.

[0075] In this embodiment, the neural network configuration shown in Fig. 1 is used, but is not limited to this. For example, a U-net or a Transformer may also be used.

[0076] CN in Figure 1 represents a convolutional layer. CN calculates the sum of the input, the convolution of the filter, and the bias, and performs nonlinear transformation on the result using an activation function. The initial values ​​of each filter component and the bias are arbitrary, and in this embodiment they are determined by random numbers. The activation function can be, for example, a ReLU (Rectified Linear Unit) or a sigmoid function. The multidimensional array output from each layer except the final layer is a feature map. Skip connection 205 combines feature maps output from discontinuous layers. Feature maps can be combined by taking the sum of each element, or by concatenating in the channel direction. In this embodiment, the sum of each element is used.

[0077] The elements (blocks or modules) within the dotted line frame in Figure 1 represent residual blocks. A network in which residual blocks are layered is called a residual network and is widely used in video processing using machine learning models. However, this embodiment is not limited to this, and other elements may be layered to form a network.

[0078] The vertical and horizontal sizes of the feature map are also enlarged or reduced within the network. Here, pooling, stride convolution, etc. can be used to reduce the feature map. Furthermore, deconvolution (or transposed convolution), pixel shuffle, interpolation, etc. can be used to expand the feature map. Although not shown in the figure in this embodiment, stride convolution is used to reduce the feature map, and nearest neighbor interpolation is used to expand it.

[0079] Additionally, a noise level (γ in the figure) is input to the residual block, and noise removal is performed from the noise frame group according to the noise level. In this embodiment, a separately prepared multilayer perceptron (MLP in the figure) is used to generate a vector with embedded information based on the noise level. Then, based on this, the feature map obtained between the two convolution layers of the residual block is scaled (an affine transformation using a gain to be multiplied and a bias to be added to the feature map). However, this embodiment is not limited to this. A multilayer perceptron is a type of neural network. Furthermore, since the noise level of the noisy video differs with each iteration in the dediffusion process, a different noise level γt is input to the residual block for each iteration.

[0080] Next, with reference to Fig. 5, the generation (conversion) of an upscaled video (output video) using a diffusion model executed by the image estimation device 103 in this embodiment will be described. Since the output video is generated based on the input video, the generation process in this embodiment can be regarded as a process of converting the input video into an output video using the diffusion model. Fig. 5 is a flowchart related to the generation of an upscaled video using the diffusion model. Each step in Fig. 5 is mainly executed by the input acquisition unit 103b, initialization unit 103c, output acquisition unit 103d, noise removal unit 103e, or update unit 103f of the image estimation device 103.

[0081] First, in step S201, the input acquisition unit 103b acquires a group of input frames to the diffusion model from a bicubic interpolated video of the captured video. The captured video is a low-resolution video, as in learning. In this embodiment, the captured video is transmitted from the imaging device 102, but this is not limiting. Video information may also be acquired along with the captured video and used in steps described later. Also, as in learning, an interpolated video is generated by bicubic interpolating the captured video using an interpolation unit (not shown), and then an input frame group is acquired; however, the interpolation may also be performed by the input acquisition unit 103b. Also, the captured video may be bicubic interpolated based on the video size acquired along with the captured video.

[0082] In this embodiment, multiple input frame groups are acquired from a captured video (input video), and sequential processing (division processing for each frame group) is performed using a diffusion model in the steps described below. When dividing the frame groups, frames (corresponding frames) that overlap with adjacent frame groups (between the first input frame group and the second input frame group) are provided. That is, each input frame group has overlapping frames (common frames, corresponding frames) that overlap with (are common to) other input frame groups. In addition, information about the location where the input frame group is extracted from the captured video (location information, frame number) and frame group size, which are acquired along with the captured video, may be used to acquire the input frame groups.

[0083] As described above, in this embodiment, it is preferable that the first input frame group and the second input frame group partially include overlapping frames. More preferably, the first input frame group and the second input frame group include a plurality of overlapping frames. In other words, it is preferable that the last frame in time of the first input frame group is the first input frame, the nth frame in time of the second input frame group is the first input frame, and the (n+1)th frame in time of the second input frame group is the second input frame.

[0084] For example, if there is one overlapping frame, this corresponds to n = 1. In this case, the last frame of the first input frame group (the frame at time t) is the first input frame. Also, the first frame of the second input frame group (the frame at time t) is the first input frame, and the second frame of the second input frame group (the frame at time t+1) is the second input frame.

[0085] For example, if there are two overlapping frames, then n=2. In this case, the last frame of the first input frame group (the frame at time t) is the first input frame. The second frame of the second input frame group (the frame at time t) is the first input frame, and the third frame of the second input frame group (the frame at time t+1) is the second input frame. The first frame of the second input frame group (the frame at time t-1) is a common frame (overlapping frame) with the second-to-last frame of the first input frame group (the frame at time t-1).

[0086] However, this embodiment is not limited to this. The multiple input frame groups may include a first input frame group including a first input frame and a second input frame group including a second input frame, and the second input frame may be adjacent to the first input frame in terms of time. For example, there may be no common frames (overlapping frames) between the first input frame group and the second input frame group. In this case, the last frame in terms of time of the first input frame group is the first input frame (the frame at time t), and the first frame in terms of time of the second input frame group (the frame at time t+1) is the second input frame.

[0087] In this embodiment, acquisition of an input frame group starts from the beginning of the bicubic interpolated video of the captured video. Then, subsequent input frame groups are acquired by shifting in the time direction. By repeating this process up to the final frame, multiple input frame groups are acquired from the entire captured video. However, this embodiment is not limited to this. Also, depending on the location where the input frame group is acquired, there may be an insufficient number of frames in the captured video that overflow. In such cases, the shift amount or the amount of overlap with adjacent frame groups may be adjusted. The number of frames may also be adjusted by padding frames in the time direction. Furthermore, the input frame group corresponds to the conditioned frame group x of the formula, and the captured video (or more accurately, the bicubic interpolated video) corresponds to the original video I of the formula.

[0088] Next, in step S202, the initialization unit 103c generates an array (noise frame group, noise data) of the same size as the input frame group from Gaussian noise as the first (t=T) intermediate frame group (intermediate solution) (initializes the intermediate frame group). The noise frame group may be generated using pseudo-random numbers with a fixed random number seed. In this case, if a recommended random number seed is included in the metadata of the video information acquired together with the captured video and the input frame group, this may be used. In addition, the noise frame group here is generated by the pure noise video y T is equivalent to

[0089] Next, in step S203, the output acquisition unit 103d acquires a frame group (adjacent frame group) that is a frame group that is temporally adjacent to the input frame group in the captured video and has already been processed using a diffusion model. Note that the adjacent frame group may be acquired using information on the location (position information, frame number) for extracting the input frame group from the captured video, which information was acquired together with the captured video and the input frame group. The output acquisition unit 103d also simultaneously acquires a mask for extracting corresponding frames between the adjacent frame group and the input frame group. The adjacent frame group corresponds to z in the formula, and the mask corresponds to m1 and m2 in the formula, respectively.

[0090] Next, in step S204, the noise removal unit 103e concatenates the input frame group and the intermediate frame group in the channel direction and inputs them into a neural network to generate a noise-removed frame group. The noise-removed frame group is generated using a neural network with the same configuration as that shown in FIG. 1. The weight information is pre-trained information transmitted from the learning device 101 and stored in the memory unit 103a. In this embodiment, a predetermined value corresponding to the number of iterations is used as the noise level used to generate the noise-removed video, but this is not limited to this. For example, a schedule of noise levels determined for each iteration may be acquired and used along with the captured video. The noise-removed frame group corresponds to the estimated target frame group y^ in the formula.

[0091] Next, in step S205, the update unit 103f generates an updated intermediate frame group using the noise-removed frame group, the adjacent frame group, and the mask based on equation (8). In this embodiment, the Gaussian noise used during the update may be generated using pseudo-random numbers with a fixed random number seed. In this case, the value of the random number seed may be a value corresponding to the number of iterations of the process or a value corresponding to the number of iterations from a separately prepared sequence. Also, if a recommended random number seed is included in the metadata of the video information acquired together with the captured video or the input frame group, this may be used. The updated intermediate frame group is generated by using the intermediate frame group y t-1 is equivalent to

[0092] Next, in step S206, the update unit 103f determines whether the processing is complete. If the processing is complete, the updated intermediate frame group is output as an output frame group to the storage unit 103a, the recording medium 105, or the like. If the processing is not complete, the intermediate frame group is used as a new intermediate frame group and the processing is repeated from step S204. In this embodiment, the number of processing iterations is used as a criterion for determining whether the processing is complete, but this is not limited to this. Note that information (position information, frame number) about the location where the input frame group was extracted from the captured video may also be output together with the output frame group and used in combining the output frame group, as described below. Furthermore, the upper limit of the number of processing iterations may be, for example, a predetermined value acquired together with the captured video or the input frame group.

[0093] Next, in step S207, the update unit 103f determines whether processing has been completed for all input frame groups acquired from the bicubic interpolated moving image of the captured moving image. If processing has been completed, the processed output frame groups are combined by a combining unit (not shown) and output as an upscaled moving image (output moving image) to at least one of the display device 104, the recording medium 105, and the output device 107. If processing has not been completed, a new input frame group is acquired from the bicubic interpolated moving image of the captured moving image, and the processing is repeated from step S201.

[0094] In this way, the updating unit 103f iteratively updates the intermediate frame group by using a diffusion model based on the input frame group, the intermediate frame group, and the adjacent frame group. Preferably, the updating unit 103f iteratively updates the intermediate frame group based on an updated denoised frame group obtained by using a diffusion model based on the input frame group and the intermediate frame group, and corresponding frames between the intermediate frame group and the adjacent frame group. More preferably, the updating unit 103f iteratively updates the intermediate frame group based on gradients of corresponding frames. Note that in this embodiment, if there are no corresponding frames, the intermediate frame group may be updated based on gradients obtained from two frames (adjacent frames) of the adjacent frame group.

[0095] The input acquisition unit 103b may acquire multiple input frame groups divided from a moving image. The multiple input frame groups include a first input frame group and a second input frame group, and the second input frame group has an area common to part of the first input frame group. A combining unit (not shown) provided in the image estimation device 103 performs a combining process based on multiple output frame groups including a first output frame group corresponding to the first input frame group and a second output frame group corresponding to the second input frame group.

[0096] Note that the combining unit (not shown) may combine the processed output frame group and the adjacent processed output frame group by cutting (trimming) them so that there are no overlapping frames between them. Alternatively, the combining unit (not shown) may combine the processed output frame group and the adjacent processed output frame group by taking a weighted average of the overlapping frames between them. In this embodiment, the frame groups are combined by trimming, but this is not limited to this. Furthermore, the output frame groups are combined using information (position information, frame number) of the location where the input frame group was extracted from the captured video, but this is not limited to this. Note that the update unit 103f may also function as the combining unit.

[0097] In this embodiment, a method in image space has been described as a method for generating upscaled video using a diffusion model, but this is not limiting. The method may also be performed in latent space using a feature map obtained using a separately prepared neural network for a group of input frames extracted from a bicubic interpolated video of a captured video. In this case, the initialization unit 103c generates a group of noise frames (a group of intermediate frames) of the same size as the feature map obtained from the group of input frames, and the output acquisition unit 103d generates a feature map for a group of adjacent frames using a separately prepared neural network. The noise removal unit 103e uses a neural network trained in latent space to remove noise in latent space from the feature map obtained from the group of input frames, the feature map obtained from the group of adjacent frames, and the group of intermediate frames in feature space.

[0098] Furthermore, in this embodiment, a method of estimating the upscaled video itself has been described as a method of generating an upscaled video using a diffusion model, but the present invention is not limited to this. For example, it is also possible to estimate only the residual component between the upscaled video and a bicubic interpolated video of a captured video. Furthermore, instead of the bicubic interpolated video, an interpolated video enlarged using a neural network or the like different from the diffusion model may be obtained, and only the residual component between the interpolated video and the bicubic interpolated video may be estimated.

[0099] In addition, in this embodiment, the description has been given on the assumption that the captured video to be used for generating an upscaled video using a diffusion model is implicitly a color video, but this is not limited to this. For example, an upscaled luminance video may be generated using a monochrome luminance video extracted from a captured color video.

[0100] In this embodiment, the input frame group may be a frame group extracted from a video and not processed by the diffusion model. In this embodiment, the input frame group may include, as additional information, at least one of the number of pixels in the input frame group, the number of frames in the input frame group, and metadata of the input video. The input frame group may also include, as additional information, at least one of information regarding frame positions (frame numbers) when extracting the frame group from the video and information regarding random numbers used when processing the frame group extracted from the video. The input frame group may also include, as additional information, at least one of information regarding the noise level of the frame group extracted from the input video.

[0101] Furthermore, the additional information may be different for each frame group (i.e., the additional information may be different for at least two of the multiple input frame groups). For example, if the input video's metadata includes the shooting conditions of the input video (optical system zoom, F-number, focal length, ISO sensitivity, etc.), the additional information may be different for each frame group, allowing video generation processing to be performed based on the shooting conditions of the input video.

[0102] In this embodiment, the adjacent frame group is a frame group that overlaps with a part of the input frame group, is extracted from a moving image, and is processed by a diffusion model.

[0103] As described above, in this embodiment, the division means (input acquisition unit) 103b divides the input video into multiple input frame groups. The acquisition means (output acquisition unit 103d, noise removal unit 103e, update unit 103f) acquires multiple output frame groups by transforming each of the multiple input frame groups using a diffusion model, and acquires the output video by combining the multiple output frame groups. The multiple input frame groups include a first input frame group including a first input frame and a second input frame group including a second input frame. The second input frame is temporally adjacent to the first input frame. The acquisition means converts the first input frame group to acquire the first output frame group and updates noise data based on the second input frame group, thereby converting the second input frame group to acquire the second output frame group.

[0104] Preferably, the obtaining means outputs the intermediate frame group based on the second input frame group by using a diffusion model, and updates the intermediate frame group based on the first output frame group. More preferably, the obtaining means updates the intermediate frame group based on gradients obtained from corresponding frames of the intermediate frame group and the first output frame group.

[0105] Preferably, each of the plurality of output frame groups has no noise added. Also preferably, the acquisition means acquires the first intermediate frame group by updating noise data based on the first input frame group using a diffusion model, and acquires the first output frame group by updating the first intermediate frame group based on the first input frame group using the diffusion model. Then, the acquisition means acquires the second output frame group by using the first output frame group without using the first intermediate frame group.

[0106] Preferably, the input image may be divided not only in the time direction but also in the spatial direction. That is, the first input frame group includes a first input partial region frame group obtained by spatially extracting partial regions from frames of the input video. The second input frame group includes a second input partial region frame group that is spatially located at the same positions as the first input partial region frame group from frames of the input video. The acquisition means acquires a third output frame group by converting a third input partial region frame group that includes the same frames as the second input partial region frame group and is spatially adjacent to the second input partial region frame group. The acquisition means then acquires the second output frame group by using a diffusion model and further updating noise data based on the third output frame group.

[0107] With the above configuration, according to this embodiment, division processing can be performed for each frame group without any problems in upscale video processing using a diffusion model.

[0108] [Example 2] Next, a moving image processing system according to a second embodiment of the present invention will be described. In this embodiment, a blurred captured moving image is divided into frame groups, and the resulting blurred frame groups are deblurred using a diffusion model. The moving image processing system of this embodiment differs from the first embodiment in that an imaging device acquires a captured moving image (blurred moving image) and performs deblurred moving image processing.

[0109] Fig. 6 is a block diagram of a video processing system 300 in this embodiment. Fig. 7 is an external view of the video processing system 300. The video processing system 300 includes a learning device 301 and an imaging device 302 connected via a network 303. Note that the learning device 301 and the imaging device 302 do not need to be constantly connected via the network 303.

[0110] The learning device 301 includes a storage unit 311, an acquisition unit 312, a generation unit 313, and a learning unit 314. These units are used to learn a neural network (update weights) for video processing that generates a deblurred frame group from a blurred frame group using a diffusion model.

[0111] The imaging device 302 captures an image of a subject space, acquires a captured video (blurred video), and generates a blurred video from the captured video. Details of the video processing executed by the imaging device 302 will be described later. The imaging device 302 has an optical system 321 and an image sensor 322. The video estimation unit (image processing device) 323 has an input acquisition unit 323a, an initialization unit 323b, an output acquisition unit 323c, a noise removal unit 323d, and an update unit 323e.

[0112] Regarding the neural network learning (weight update) executed by the learning device 301, only the differences from the first embodiment will be described with reference to Fig. 4. Note that step S102 in this embodiment is different from step S102 in Fig. 4 (first embodiment), but will be described as step S102 of this embodiment inserted in place of step S102 in the first embodiment.

[0113] First, in step S101, the acquisition unit 312 acquires a group of sharp frames with little blur (a group of correct frames, a group of high-resolution frames) corresponding to a group of blurred frames (a group of degraded frames, a group of low-resolution frames).

[0114] Next, in step S102, the generation unit 313 generates an adjacent frame group by extracting some frames from the sharp frame group. Here, the adjacent frame group corresponds to z in the formula (a processed adjacent frame group obtained by processing a frame group extracted from a location temporally adjacent to the conditioned frame group x using a diffusion model). That is, this embodiment is characterized in that the adjacent frame group is taken into consideration from the time of learning the weights of the neural network and is used to remove noise from the noisy frame group generated in step S103.

[0115] Next, in step S103, the generation unit 313 generates a noise frame group by adding noise to the sharp frame group.

[0116] Next, in step S104, the learning unit 314 uses a neural network to generate a frame group in which noise has been removed from the noise frame group. Here, the frame group in which noise has been removed from the noise frame group is generated using the blur frame group, the noise frame group, the adjacent frame group, and information on the strength of noise (noise level) added to the noise frame group.

[0117] Note that the denoised frame group is an estimate of the sharp frame group, and ideally, the two would match. Furthermore, by removing noise from the noise frame group while taking into account adjacent frame groups, the corresponding portions of the adjacent frame group and the denoised frame group ideally match. This ensures that when the deblurred video processed frame group and the adjacent frame group are combined using a diffusion model (described below), no sudden changes occur between the frames, maintaining temporal consistency. In other words, the frame group can be divided and processed without any problems. Furthermore, in this embodiment, the blurred frame group, the noise frame group, and the adjacent frame group are concatenated in the channel direction and input to the neural network, but this is not a limitation.

[0118] The subsequent steps S105 and S106 are the same as those in the first embodiment, and therefore their explanation will be omitted.

[0119] Next, details of the video processing executed by the imaging device 302 will be described. Neural network weight information is learned in advance by the learning device 301 and stored in the storage unit 311. The imaging device 302 reads the weight information from the storage unit 311 via the network 303 and stores it in the storage unit 324. The video estimation unit 323 generates a deblurred frame group from a blurred frame group obtained by dividing a blurred captured video into frame groups by the video estimation unit 323. The deblurred frame group can be generated using the trained neural network weight information stored in the storage unit 324 and the blurred captured video and its video information acquired by the input acquisition unit 323a. The generated deblurred frame group is combined into a deblurred video, and then stored on the recording medium 325a.

[0120] When a user issues an instruction regarding display of a deblurred video via the input unit 326, the saved video is read out and displayed on the display unit 325b. Note that the captured video and its video information saved in the recording medium 325a may be read out, and the video estimation unit 323 may generate a deblurred video. The above series of controls are performed by the system controller 327. Since a deblurred video (output video) is generated based on a blurred captured video (input video), the generation process of this embodiment can be regarded as a process of converting the input video into an output video using a diffusion model.

[0121] Next, generation of a deblurred moving image using a diffusion model executed by the moving image estimation unit 323 in this embodiment will be described. The procedure of moving image processing in this embodiment is substantially the same as that shown in FIG. 5 of the first embodiment, and therefore the moving image processing in this embodiment will be described with reference to FIG. 5. Each step of the moving image processing is mainly executed by the input acquisition unit 103b, initialization unit 103c, output acquisition unit 103d, noise reduction unit 103e, or update unit 103f of the first embodiment, which corresponds to the moving image estimation unit 323. The corresponding parts are the input acquisition unit 323a, initialization unit 323b, output acquisition unit 323c, noise reduction unit 323d, or update unit 323e.

[0122] First, in step S201, the input acquisition unit 323a acquires a captured video (blurred video). The captured video is a blurred video, as in the learning process. In this embodiment, the captured video is acquired by the imaging device 302 and stored in the storage unit 324, but this is not limited to this. Note that video information may also be acquired along with the captured video and used in the steps described below. The input acquisition unit 323a also acquires multiple input frame groups from the captured video and performs sequential processing (division processing for each frame group) using a diffusion model in the steps described below. When dividing the frame groups, frames that partially overlap (corresponding frames) are provided between adjacent frame groups. Furthermore, information on the location where the input frame groups are extracted from the captured video (position information, frame number) and the number of frames in the frame group, which are acquired along with the captured video, may be used to acquire the input frame groups. In this embodiment, multiple input frame groups are acquired randomly so as to cover the entire captured video, but this is not limited to this. The input frame groups correspond to the conditioning frame group x of the formula, and the captured video corresponds to the original video I of the formula.

[0123] The subsequent steps S202 and S203 are the same as those in the first embodiment, and therefore their explanation will be omitted.

[0124] Next, in step S204, the noise removal unit 323d concatenates the input frame group, the intermediate frame group, and the adjacent frame group in the channel direction and inputs them to the neural network to generate a noise-removed frame group. The difference from Figure 1 is the number of moving images input to the neural network; whereas in Figure 1 there are two (202 and 203), in this embodiment, there are three, including the adjacent frame group. Furthermore, the weight information is learned information transmitted from the learning device 301 and stored in the memory unit 324. The adjacent frame group to be concatenated may consist of only corresponding frames.

[0125] The noise-removed frame group is generated using a neural network having substantially the same configuration as that shown in Fig. 1. The noise level used when generating the noise-removed frame group is also the same as in the first embodiment.

[0126] Next, in step S205, the update unit 323e generates an updated intermediate frame group from the noise-removed frame group based on equation (7). Furthermore, since equation (7) represents the de-diffusion process in a conventional diffusion model, the adjacent frame group z does not exist in the input of the neural network fθ, but in this embodiment, the input includes the adjacent frame group z. The neural network learning and the method of inputting the adjacent frame group z are as described above. Furthermore, the updated intermediate frame group is generated by adding the intermediate frame group y t-1 is equivalent to

[0127] The subsequent steps S206 and S207 are substantially the same as those in the first embodiment.

[0128] In step S206, the output frame group is output to the recording medium 325a instead of the storage unit 103a or the recording medium 105 or the like.

[0129] In step S207, the deblurred video is output as the output video instead of the upscaled video to at least one of the recording medium 325a or the display unit 325b instead of at least one of the display device 104, the recording medium 105, and the output device 107. The frame groups are joined at joints (not shown) in the same manner as in the first embodiment.

[0130] As described above, in this embodiment, the update unit 323e iteratively updates the intermediate frame group based on corresponding frames among the input frame group, the intermediate frame group, and the adjacent frame group. Preferably, the update unit 323e iteratively updates the intermediate frame group by concatenating the input frame group, the intermediate frame group, and the adjacent frame group (corresponding frames therein) and inputting the combined results into the diffusion model.

[0131] In this embodiment, the acquisition unit (the output acquisition unit 323c, the noise removal unit 323d, and the update unit 323e) outputs the intermediate frame group based on the second input frame group and at least a portion of the first output frame group using a diffusion model. Preferably, the acquisition unit inputs at least a portion of the first output frame group into the diffusion model.

[0132] With the above configuration, according to this embodiment, it is possible to perform division processing for each frame group without any drawbacks in the deblurred moving image processing using the diffusion model, and to generate a deblurred moving image with temporal consistency.

[0133] [Example 3] Next, an image processing system according to a third embodiment of the present invention will be described. The image processing system of this embodiment transmits a captured video (Bayer video) to be processed to an image estimation device. The image processing system of this embodiment also includes a processing device (computer) that processes Bayer frame groups, which have been divided into frame groups, using a diffusion model and receives a video (deBayer video) obtained by combining the resulting deBayer frame groups from the image estimation device. This embodiment differs from the first and second embodiments in these respects. Note that Bayer video is raw video data acquired by an image sensor with a Bayer array in which the color filter has a checkerboard color pattern. In Bayer video, each pixel has information on only one of the RGB colors. Therefore, to obtain a full-color video with information on all three RGB colors in each pixel, interpolation processing from adjacent pixels is required. This video processing is called deBayering (or demosaicing).

[0134] 8 is a block diagram of an image processing system 400 in this embodiment. The image processing system 400 includes a learning device 401, an imaging device 402, an image estimation device (image processing device) 403, and a computer (control device) 404. The learning device 401 and the image estimation device 403 are, for example, servers. The computer 404 is, for example, a user terminal (a personal computer or a smartphone). The computer 404 is connected to the image estimation device 403 via a network 405. The image estimation device 403 is connected to the learning device 401 via a network 406. That is, the computer 404 and the image estimation device 403 are configured to be able to communicate with each other, and the image estimation device 403 and the learning device 401 are configured to be able to communicate with each other.

[0135] The configuration of the learning device 401 is similar to that of the learning device 101 of the first embodiment, and therefore a description thereof will be omitted. The learning of the neural network by the learning device 401 is substantially similar to that of the first embodiment, except that a Bayer frame group is used instead of a low-resolution frame group as a degraded frame group, and a DeBayer frame group is used instead of a high-resolution frame group as a corresponding correct frame group. Therefore, a description thereof will be omitted. The configuration of the imaging device 402 is similar to that of the imaging device 102 of the first embodiment, and therefore a description thereof will be omitted.

[0136] The image estimation device 403 includes a storage unit 403a, an input acquisition unit 403b, an initialization unit 403c, an output acquisition unit 403d, a noise reduction unit 403e, an update unit 403f, and a communication unit 403g. The storage unit 403a, the input acquisition unit 403b, the initialization unit 403c, the output acquisition unit 403d, the noise reduction unit 403e, and the update unit 403f function as an image processing unit of the image estimation device 403. The components of the image processing unit are similar to the storage unit 103a, the input acquisition unit 103b, the initialization unit 103c, the output acquisition unit 103d, the noise reduction unit 103e, and the update unit 103f of the image estimation device 103 of the first embodiment. The communication unit 403g functions as a receiving unit that receives a request transmitted from the computer 404 and as a transmitting unit that transmits an output video (deBayered video) generated by the image estimation device 403 to the computer 404.

[0137] The computer 404 has a communication unit 404a, a display unit 404b, an input unit 404c, a processing unit 404d, and a memory recording unit 404e. The communication unit 404a functions as a transmitter that transmits a request to the image estimation device 403 to cause the image estimation device 403 to execute processing on a captured video (Bayer video). The communication unit 404a also functions as a receiver that receives an output video (deBayer video) processed by the image estimation device 403. The display unit 404b has a function to display various information. The information displayed by the display unit 404b includes, for example, the captured video (Bayer video) to be transmitted to the image estimation device 403 and the output video (deBayer video) received from the image estimation device 403. The input unit 404c receives instructions from a user to start video processing, etc. The processing unit 404d has a function to perform video processing, including white balance adjustment and gamma correction, on the output video (deBayer video) received from the image estimation device 403. The recording unit 404e stores the captured moving image acquired from the imaging device 402, the output moving image received from the image estimation device 403, and the like.

[0138] Next, moving image processing in this embodiment will be described with reference to Fig. 9. The moving image processing in this embodiment is equivalent to the moving image processing described in embodiment 1 with reference to Fig. 5. Fig. 9 is a flowchart related to generation of an output moving image using a neural network in this embodiment. The moving image processing shown in Fig. 9 is started when a command to start moving image processing is given by a user via the computer 404. First, the operation of the computer 404 will be described. Note that, since a de-Bayer moving image (output moving image) is generated based on a Bayer moving image (input moving image), the generation processing in this embodiment can be considered as processing to convert the input moving image into an output moving image using a diffusion model.

[0139] First, in step S501, the computer 404 transmits a request for processing a captured video (Bayer video, input video) (processing of input frame groups related to the captured video) to the image estimation device 403. Note that the method for transmitting the captured video to be processed to the image estimation device 403 does not matter. For example, the captured video may be uploaded to the image estimation device 403 simultaneously with step S501, or may be uploaded to the image estimation device 403 before step S501. Furthermore, the captured video may be a video stored on a server different from the image estimation device 403. Furthermore, in step S501, the computer 404 may transmit an ID for authenticating a user, video information, etc. together with the request for processing the captured video.

[0140] Next, in step S 502 , the computer 404 receives the output video (deBayer video) generated in the image estimation device 403 .

[0141] Next, a description will be given of the operation of the image estimation device 403. First, in step S601, the image estimation device 403 receives a request for processing a captured moving image (Bayer moving image) transmitted from the computer 404. The image estimation device 403 determines that processing for the captured moving image has been instructed, and executes the processing from step S602 onwards.

[0142] Next, in step S602, the input acquisition unit 403b acquires a captured video (Bayer video, input video). Similar to the upscaling in the first embodiment, an interpolated video is generated by an interpolation unit (not shown) that simply de-Bayered the captured video using bilinear interpolation. However, the interpolation may also be performed by the input acquisition unit 403b. In this embodiment, the captured video is transmitted from the imaging device 402, but this is not limiting. Multiple input frame groups are acquired from the bilinearly interpolated captured video, and sequential processing (division processing for each frame group) is performed using a diffusion model in a step described later. When dividing the frame groups, frames that overlap (=corresponding frames) are provided between adjacent frame groups. The input frame groups may be acquired using information about the location (position information, frame number) where the input frame groups are extracted from the captured video, or the number of frames in the frame groups, which information was acquired together with the captured video.

[0143] In this embodiment, acquisition of an input frame group starts from the beginning of the bilinearly interpolated video of the captured video. The acquisition of subsequent input frame groups is repeated to acquire multiple input frame groups from the entire captured video. However, this embodiment is not limited to this. Furthermore, depending on the location where the input frame group is acquired, it may extend beyond the captured video. In such cases, the shift amount or the amount of overlap with adjacent frame groups may be adjusted. Furthermore, the input frame group corresponds to the conditioned frame group x of the formula, and the captured video (or more precisely, the bilinearly interpolated video) corresponds to the original video I of the formula.

[0144] Next, in step S603, the initialization unit 403c initializes the input frame group by generating an intermediate frame group having the same size as the input frame group from Gaussian noise.

[0145] Next, in step S604, the output acquisition unit 403d acquires a group of frames (adjacent frame group) that have been acquired from a location adjacent to the input frame group in the bilinear interpolated moving image of the captured moving image and have already been processed using the diffusion model. Note that the adjacent frame group may be acquired using information (position information, frame number) about the location where the input frame group is to be extracted from the captured moving image, which information was acquired together with the captured moving image and the input frame group. A mask for extracting corresponding frames from the adjacent frame group and the input frame group is also acquired at the same time.

[0146] Next, in step S605, the noise removal unit 403e concatenates the input frame group and the intermediate frame group in the channel direction and inputs them into a neural network to generate a noise-removed frame group. The noise-removed frame group is generated using a neural network with the same configuration as that shown in FIG. 1. The weight information is pre-trained information transmitted from the learning device 401 and stored in the storage unit 403a. In this embodiment, a predetermined value corresponding to the number of iterations is used as the noise level used when generating a noise-removed video, but this is not limiting.

[0147] Next, in step S606, the update unit 403f generates an updated de-Bayered frame group (updated intermediate frame group) from the noise-removed frame group, the adjacent frame group, and the mask based on equation (8). The updated de-Bayered frame group is generated by the intermediate frame group y t-1 is equivalent to

[0148] Next, in step S607, the update unit 403f determines whether the processing is complete. If the processing is complete, the updated de-Bayered frame group is output to the storage unit 403a as an output frame group. If the processing is not complete, the updated intermediate frame group is used as a new intermediate frame group and the processing is repeated from step S605. In this embodiment, the number of processing iterations is used as a criterion for determining whether the processing is complete, but this is not limiting. Note that information (position information, frame number) about the location where the input frame group was extracted from the captured video may also be output together with the output frame group and used in combining the output frame group, as described below.

[0149] Next, in step S608, the update unit 403f determines whether processing has been completed for all input frame groups acquired from the bilinear interpolated moving image of the captured moving image. If processing has been completed, the processed output frame groups are combined in a combining unit (not shown) to generate a de-Bayered moving image (output moving image). If processing has not been completed, a new input frame group is acquired from the bilinear interpolated moving image of the captured moving image, and the processing is repeated from step S602. Note that the frame group combination in the combining unit (not shown) is the same as in the first embodiment.

[0150] Next, in step S609, the image estimation device 403 sends the combined de-Bayered video to the computer 404.

[0151] With the above configuration, according to this embodiment, division processing for each frame group can be performed without any problems in de-Bayer video processing using a diffusion model.

[0152] [Other Examples] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0153] In each embodiment, the frame group division process can be appropriately performed using a diffusion model. Therefore, according to each embodiment, it is possible to provide an image processing method, an image processing device, an image processing system, and a program that can perform image processing of moving images using a diffusion model while reducing an increase in memory usage and adverse effects during moving image generation. The image processing device may be any device that has the image processing function of each embodiment, and can be realized in the form of an imaging device, a personal computer, or the like.

[0154] The disclosure of each embodiment includes the following methods and compositions: (Method 1) Dividing an input video into a plurality of input frame groups; transforming each of the plurality of input frames using a diffusion model to obtain a plurality of output frames; and combining the plurality of output frame groups to obtain an output moving image; the plurality of input frame groups include a first input frame group including a first input frame and a second input frame group including a second input frame; the second input frame is temporally adjacent to the first input frame; In the step of obtaining a plurality of output frame groups, an image processing method, characterized in that a second output frame group is obtained by converting the second input frame group by updating noise data based on a first output frame group obtained by converting the first input frame group and the second input frame group. (Method 2) The step of obtaining a plurality of output frames includes: outputting a set of intermediate frames based on the second set of input frames by using the diffusion model; and updating said intermediate set of frames based on said first output set of frames. (Method 3) The image processing method described in Method 1, characterized in that in the step of updating the intermediate frame group based on the first output frame group, the intermediate frame group is updated based on gradients obtained from corresponding frames of the intermediate frame group and the first output frame group. (Method 4) 4. The image processing method according to Method 2 or 3, wherein the step of obtaining the plurality of output frame groups comprises the step of outputting the intermediate frame group based on the second input frame group and at least a portion of the first output frame group using the diffusion model. (Method 5) outputting the intermediate group of frames based on the second input group of frames and at least a portion of the first output group of frames using the diffusion model, 5. The image processing method of method 4, further comprising inputting at least a portion of the first set of output frames into the diffusion model. (Method 6) 6. The image processing method according to any one of methods 1 to 5, wherein each of the plurality of input frame groups is a frame group that has not been processed by the diffusion model. (Method 7) each of the plurality of input frame groups includes additional information; An image processing method described in any of methods 1 to 6, characterized in that the additional information includes at least one of the number of pixels in the input frame group, the number of frames in the input frame group, metadata of the input video, information regarding frame positions when extracting the input frame group from the input video, information regarding random numbers used when processing the input frame group, and information regarding the noise level of the input frame group. (Method 8) 8. The image processing method according to Method 7, wherein the additional information is different for at least two of the plurality of input frame groups. (Method 9) 9. The image processing method according to any one of Methods 1 to 8, wherein the first input frame group and the second input frame group partially include overlapping frames. (Method 10) 10. The image processing method according to any one of Methods 1 to 9, wherein the first set of input frames and the second set of input frames include a plurality of overlapping frames. (Method 11) the first input frame is the last frame in time in the first input frame group; the n-th frame in time in the second input frame group is the first input frame; 11. The image processing method according to any one of Methods 1 to 10, wherein the n+1th frame in time in the second input frame group is the second input frame. (Method 12) the first input frame is the last frame in time in the first input frame group; 9. The image processing method according to any one of Methods 1 to 8, wherein the first frame in time in the second input frame group is the second input frame. (Method 13) 13. The image processing method according to any one of Methods 1 to 12, wherein each of the plurality of output frame groups has no noise added. (Method 14) The step of obtaining a plurality of output frames includes: updating the noise data based on the first set of input frames using the diffusion model to obtain a first set of intermediate frames; updating the first set of intermediate frames based on the first set of input frames using the diffusion model to obtain the first set of output frames; and obtaining the second output frame group by using the first output frame group without using the first intermediate frame group. (Method 15) 15. The image processing method according to any one of Methods 1 to 14, wherein the diffusion model transforms each of the plurality of input frames in image space. (Method 16) the first input frame group includes a first input partial region frame group obtained by spatially extracting partial regions from frames of the input video; the second input frame group includes a second input partial region frame group at a spatially identical position to the first input partial region frame group from frames of the input video; obtaining a third output group of frames by transforming a third input group of partial region frames that includes the same frames as the second input group of partial region frames and that are spatially adjacent to the second input group of partial region frames; 16. The image processing method according to any one of Methods 1 to 15, wherein the second output frame group is obtained by using the diffusion model and further updating the noise data based on the third output frame group. (Configuration 1) a division means for dividing an input video into a plurality of input frame groups; an acquisition means for acquiring a plurality of output frame groups by converting each of the plurality of input frame groups using a diffusion model, and acquiring an output moving image by combining the plurality of output frame groups; the plurality of input frame groups include a first input frame group including a first input frame and a second input frame group including a second input frame; the second input frame is temporally adjacent to the first input frame; The image processing device is characterized in that the acquisition means converts the second input frame group to acquire a second output frame group by updating noise data based on a first output frame group acquired by converting the first input frame group and the second input frame group. (Configuration 2) An image processing system including the image processing device according to configuration 1 and a control device capable of communicating with the image processing device, The control device a transmission unit that transmits a request to cause the image processing device to execute processing on the input moving image; the image processing device has a receiving unit and an image processing unit, the receiving unit receives the request transmitted by the transmitting unit; The image processing system is characterized in that the image processing unit executes the processing in response to the request. (Configuration 3) A program causing a computer to execute the image processing method according to any one of Methods 1 to 16.

[0155] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0156] 103 Image estimation device (image processing device) 103b Input acquisition unit (division means) 103d Output acquisition unit (acquisition means) 103f Update section (obtaining means) 103f

Claims

1. Dividing an input video into a plurality of input frame groups; transforming each of the plurality of input frames using a diffusion model to obtain a plurality of output frames; and combining the plurality of output frame groups to obtain an output moving image; the plurality of input frame groups include a first input frame group including a first input frame and a second input frame group including a second input frame; the second input frame is temporally adjacent to the first input frame; In the step of obtaining a plurality of output frame groups, An image processing method characterized by converting the second input frame group to obtain a second output frame group by updating noise data based on a first output frame group obtained by converting the first input frame group and the second input frame group.

2. The step of obtaining a plurality of output frames includes: outputting a set of intermediate frames based on the second set of input frames by using the diffusion model; 2. The image processing method of claim 1, further comprising updating the intermediate set of frames based on the first output set of frames.

3. 2. The image processing method according to claim 1, wherein in the step of updating the intermediate frame group based on the first output frame group, the intermediate frame group is updated based on gradients obtained from corresponding frames of the intermediate frame group and the first output frame group.

4. 3. The image processing method according to claim 2, wherein the step of obtaining the plurality of output frame groups includes a step of outputting the intermediate frame group based on the second input frame group and at least a portion of the first output frame group using the diffusion model.

5. outputting the intermediate group of frames based on the second input group of frames and at least a portion of the first output group of frames using the diffusion model, 5. The image processing method according to claim 4, wherein at least a portion of the first output frame group is input to the diffusion model.

6. 2. The image processing method according to claim 1, wherein each of the plurality of input frames is a frame group that has not been processed by the diffusion model.

7. each of the plurality of input frame groups includes additional information; 2. The image processing method according to claim 1, wherein the additional information includes at least one of the number of pixels in the input frame group, the number of frames in the input frame group, metadata of the input video, information regarding frame positions when extracting the input frame group from the input video, information regarding random numbers used when processing the input frame group, and information regarding noise levels of the input frame group.

8. 8. The image processing method according to claim 7, wherein the additional information is different for at least two of the plurality of input frame groups.

9. 2. The image processing method according to claim 1, wherein the first input frame group and the second input frame group partially include overlapping frames.

10. 2. The image processing method of claim 1, wherein the first input frame group and the second input frame group include a plurality of overlapping frames.

11. the first input frame is the last frame in time of the first input frame group; the n-th frame in time of the second input frame group is the first input frame; 2. The image processing method according to claim 1, wherein the (n+1)th frame in terms of time of the second input frame group is the second input frame.

12. the first input frame is the last frame in time of the first input frame group; 2. The image processing method according to claim 1, wherein the first frame in time of the second input frame group is the second input frame.

13. 2. The image processing method according to claim 1, wherein each of the plurality of output frame groups has no noise added thereto.

14. The step of obtaining a plurality of output frames includes: updating the noise data based on the first set of input frames using the diffusion model to obtain a first set of intermediate frames; updating the first set of intermediate frames based on the first set of input frames using the diffusion model to obtain the first set of output frames; 14. The image processing method according to claim 1, further comprising: obtaining the second output frame group by using the first output frame group without using the first intermediate frame group.

15. 14. The image processing method of claim 1, wherein the diffusion model transforms each of the plurality of input frames in image space.

16. the first input frame group includes a first input partial region frame group obtained by spatially extracting partial regions from frames of the input video; the second input frame group includes a second input partial region frame group at a spatially identical position to the first input partial region frame group from a frame of the input video; obtaining a third output group of frames by transforming a third input group of partial region frames that includes the same frames as the second input group of partial region frames and that are spatially adjacent to the second input group of partial region frames; 14. The image processing method according to claim 1, wherein the second output frame group is obtained by using the diffusion model and further updating the noise data based on the third output frame group.

17. a division means for dividing an input video into a plurality of input frame groups; an acquisition means for acquiring a plurality of output frame groups by converting each of the plurality of input frame groups using a diffusion model, and acquiring an output moving image by combining the plurality of output frame groups; the plurality of input frame groups include a first input frame group including a first input frame and a second input frame group including a second input frame; the second input frame is temporally adjacent to the first input frame; The image processing device is characterized in that the acquisition means converts the second input frame group to acquire a second output frame group by updating noise data based on a first output frame group acquired by converting the first input frame group and the second input frame group.

18. An image processing system including the image processing device according to claim 17 and a control device capable of communicating with the image processing device, The control device a transmission unit that transmits a request to cause the image processing device to execute processing on the input moving image; the image processing device has a receiving unit and an image processing unit, the receiving unit receives the request transmitted by the transmitting unit; The image processing system is characterized in that the image processing unit executes the processing in response to the request.

19. A program causing a computer to execute the image processing method according to any one of claims 1 to 13.

Images