Image processing device, control method for the same, and program

Abstract

To provide an image processing device that can more strongly suppress afterimages in multiple-frame composition.SOLUTION: An image processing device 100 for training a model of multiple image composition includes an image acquisition unit 204, an image processing unit 205, an error calculation unit 206, and a learning unit 207. The image acquisition unit 204 acquires a group of images consisting of images of a plurality of frames including an image of a reference frame and including a predetermined subject in a plurality of patterns. The image processing unit 205 uses the image of the reference frame as a teacher image and performs predetermined image processing on the image group of the plurality of patterns to generate a student image group of the plurality of patterns. The error calculating unit 206 calculates reconstruction errors between the composite output images of the plurality of patterns of the model corresponding to the inputs of the student image group of the plurality of patterns respectively and the teacher image, and errors between the outputs of the composite output images of the plurality of patterns. The learning unit 207 trains the model on the basis of the reconstruction errors and the inter-output errors. The image acquisition unit 204 differentiates the position of the subject between the plurality of patterns in the images of one or more frames other than the image of the reference frame.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing device, a control method for an image processing device, and a program. [Background technology]

[0002] In recent years, deep learning, which uses a multi-layer neural network for learning, has achieved high performance in various fields such as images, voice, and language. Non-Patent Document 1 discloses a technology that uses a neural network for noise reduction processing of images. The technology described in Non-Patent Document 1 aims to improve image quality, such as the sense of resolution after noise reduction processing, by inputting and synthesizing a burst image composed of multiple images (multiple frames) compared to the case of inputting a single image. At this time, the technology described in Non-Patent Document 1 inputs and learns burst image patches composed of multiple frames with motion between frames in order to suppress residual images in the multiple-image synthesis output of moving parts. Specifically, the technology described in Non-Patent Document 1 shifts the patch cut-out position from a still image dataset image between frames, thereby learning the movement of the entire image patch shifting (for example, see FIG. 4(a) described later) in bursts.

[0003] On the other hand, the technology described in Patent Document 1 generates multiple patterns of student images by adding multiple different noise patterns to one teacher image in the learning of a model (neural network) that performs noise reduction processing on still images. Furthermore, the technology described in Patent Document 1 suppresses artifacts in the output image caused by the noise patterns of the input images by learning the model so that multiple output images obtained by inputting multiple patterns of student images into the model approach each other. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2021-163319 A [Non-patent literature]

[0005] [Non-Patent Document 1] Zhihao Xia, Federico Perazzi, Michael Gharbi, Kalyan Sunkavalli, Ayan Chakrabarti, “Basis Prediction Networks for Effective Burst Denoising With Large Kernels”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, p. 11844-11853 Summary of the Invention [Problem to be solved by the invention]

[0006] Incidentally, the technology described in Non-Patent Document 1 needs to learn a large amount of motion in order to suppress afterimages with large movements in multiple-image synthesis. However, since the resolution of the multiple-image synthesis output decreases as the amount of motion learned increases, it is desirable to learn the minimum amount of motion required. Therefore, for the technology described in Non-Patent Document 1, it is desirable to learn a model that can more strongly suppress afterimages when the same amount of motion is given. In addition, the technology described in Patent Document 1 is related to learning a model that performs noise reduction processing on still images, and does not mention learning of motion in multiple-image synthesis.

[0007] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an image processing device, a control method for the image processing device, and a program that can more effectively suppress afterimages when compositing multiple images. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, the image processing device of the present invention is an image processing device that learns a model of image processing in which multiple images are synthesized, and is equipped with: an acquisition means that acquires a group of images including a specified subject in multiple patterns, which is composed of images of multiple frames including an image of a reference frame; a generation means that uses the image of the reference frame as a teacher image including the subject and performs a specified image processing on the multiple patterns of the image group to generate the multiple patterns of student image groups; a calculation means that calculates a first error between the multiple patterns of synthesized output images of the model and the teacher image corresponding to the input of the multiple patterns of the student image group, respectively, and a second error between the multiple patterns of synthesized output images; and a learning means that learns the model based on the first error and the second error, and is characterized in that the acquisition means makes the position of the subject different between the multiple patterns in images of one or more frames other than the image of the reference frame. Effect of the Invention

[0009] According to the present invention, afterimages in multiple-image synthesis can be more effectively suppressed. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a block diagram showing an example of a hardware configuration of the image processing device. [Diagram 2] FIG. 2 is a block diagram showing an example of a functional configuration of the image processing device. [Diagram 3] 4 is a flowchart showing a flow of processing by the image processing device. [Figure 4] FIG. 13 is a diagram for explaining an example of a burst image patch. [Diagram 5] FIG. 13 is a diagram for explaining an example of a burst image patch having a plurality of patterns. [Figure 6] 13A and 13B are diagrams for explaining examples of changes to burst image patches of multiple patterns. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in the embodiments. For example, each part constituting the present invention can be replaced with any configuration that can exert the same function. In addition, any configuration may be added. In addition, any two or more configurations (features) of the present embodiments can be combined.

[0012] Hereinafter, the present embodiment will be described with reference to Figs. 1 to 5. Fig. 1 is a diagram showing an example of a hardware configuration of an image processing device 100 according to the present embodiment. In the present embodiment, the image processing device 100 learns a model of image processing in which a plurality of images are synthesized (that is, multiple images are synthesized). The image processing device 100 has a CPU 101, a RAM 102, a ROM 103, a secondary storage device 104, an input interface 105, and an output interface 106. The components of the image processing device 100 are connected to each other by a system bus 107. In addition, the image processing device 100 is connected to an external storage device 108 and an operation unit 109 via the input interface 105. Furthermore, the image processing device 100 is connected to an external storage device 108 and a display device 110 via the output interface 106.

[0013] The CPU 101 executes a program stored in the ROM 103 using the RAM 102 as a work memory, and controls each component of the image processing device 100 via a system bus 107. This executes various processes described below. The secondary storage device 104 is a storage device that stores various data handled by the image processing device 100, and in this embodiment, a HDD is used. The CPU 101 writes data to the secondary storage device 104 and reads data stored in the secondary storage device 104 via the system bus 107. Note that the secondary storage device 104 can be various storage devices such as an optical disk drive and a flash memory in addition to a HDD.

[0014] The input interface 105 is, for example, a serial bus interface such as USB or IEEE1394. The image processing device 100 acquires data, commands, and the like from an external device via the input interface 105. In this embodiment, the image processing device 100 acquires data from an external storage device 108 (for example, a storage medium such as a hard disk, a memory card, a CF card, an SD card, or a USB memory) via the input interface 105. The image processing device 100 also acquires user instructions input to an operation unit 109 via the input interface 105. The operation unit 109 is an input device such as a mouse or a keyboard, and inputs user instructions.

[0015] The output interface 106 is, for example, a serial bus interface such as USB or IEEE1394. This is similar to the input interface 105. The output interface 106 may be, for example, a video output terminal such as DVI or HDMI (registered trademark). The image processing device 100 outputs data and the like to an external device via the output interface 106. In this embodiment, the image processing device 100 outputs data (for example, image data) processed by the CPU 101 to a display device 110 (for example, various image display devices such as a liquid crystal display) via the output interface 106. The image processing device 100 has other components in addition to those mentioned above, but since this is not the main focus of the present disclosure, a description thereof will be omitted.

[0016] FIG. 2 is a block diagram showing an example of a functional configuration of the image processing device 100. The image processing device 100 has, as its functional units, a storage unit 201, a parameter acquisition unit 202, a parameter processing unit 203, an image acquisition unit 204, an image processing unit 205, an error calculation unit 206, and a learning unit 207. A detailed explanation of each functional unit will be given in the explanation of FIG. 3. FIG. 3 is a flowchart showing a processing flow of the image processing device 100. The processing flow of the image processing device 100 (a control method of the image processing device) shown in the flowchart of FIG. 3 is realized by the CPU 101 (computer) expanding a program stored in the ROM 103 into the RAM 102 and executing it. Note that some or all of the functions of the steps in the flowchart may be realized by hardware such as an ASIC or an electronic circuit.

[0017] FIG. 3 will be described with reference to FIG. 4 and FIG. 5 in addition to FIG. 2. FIG. 4 is a diagram for explaining an example of a burst image patch. In the example of a burst image patch shown in FIG. 4(a), the background constituting the subject in a group of multiple images is shifted significantly. In the example of a burst image patch shown in FIG. 4(b), the subject is composed of a local moving object (hereinafter, referred to as a "local moving object") and the background in a group of multiple images, and the local moving object among the subject is shifted significantly. In the example of the burst image patch shown in FIG. 4(b), the number of local moving objects is two. FIG. 5 is a diagram for explaining an example of a burst image patch of multiple patterns.

[0018] The image processing device 100 performs learning of a model simulating a local moving object by, for example, changing the superimposition position of another partial image on a group of multiple images between frames as shown in FIG. 4(b) by executing the process shown in the flowchart of FIG. 3. However, the number of local moving objects in the present embodiment (FIG. 5) described below is 1, which is different from the number of local moving objects in FIG. 4(b), which is 2. Also, as shown in FIG. 5, the image processing device 100 generates a group of multiple patterns of student images (burst image patches) with different movements of the subject (in FIG. 5, the local moving object) for one teacher image. Furthermore, the image processing device 100 performs learning of the model so that the multiple patterns of synthesized output images obtained by inputting the multiple patterns of student images into the model respectively approach each other, thereby more strongly suppressing afterimages in the multiple-image synthesis.

[0019] FIG. 3 will be described below. In step S301, the learning unit 207 sets parameters of an image processing model in which a plurality of images are synthesized. The model parameters are mainly weight parameters of a neural network, and may also include a learning rate in model learning and settings of an optimizer (optimization algorithm). The weight parameters are typically initialized with random numbers. When additional learning is performed, an existing model held in the storage unit 201 is loaded. The model may be a CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), Transformer, or any other structure.

[0020] In step S302, the parameter acquisition unit 202 acquires parameters related to model motion learning. The parameters related to motion learning include the number of motion patterns of the subject with respect to one teacher image, the shift probability of the entire image patch, the maximum horizontal and vertical shift amount of the entire image patch, the occurrence probability of a local moving object, the number of local moving objects, and the maximum horizontal and vertical shift amount between frames. These parameters are specified by the user, for example, via the operation unit 109 of the image processing device 100. In this embodiment, since there are multiple motion patterns of the subject with respect to one teacher image, they may be expressed as "multiple motion patterns". Furthermore, "multiple motion patterns" may be abbreviated to "multiple patterns". Steps S303 to S317 are processes during model learning that update the weight parameters of the neural network, and are therefore performed in a learning loop (for example, a loop in which model learning is performed for a predetermined number of epochs with a predetermined data set).

[0021] In step S303, the image acquisition unit 204 selects a background image from the still image data set held in the storage unit 201, for example, by random selection. In step S304, the parameter processing unit 203 determines a local moving object flag of a burst image patch to be generated in the current learning loop based on the occurrence probability of the local moving object acquired in step S302. For example, when the occurrence probability of the local moving object is 0.3 (30%), the parameter processing unit 203 generates a random number between 0 and 1, and if the value of the random number is 0.3 or less, determines the local moving object flag to be True. Also, if the value of the random number is greater than 0.3, the parameter processing unit 203 determines the local moving object flag to be False.

[0022] In step S305, parameter processing unit 203 determines whether the local moving object flag determined in step S304 is True or False. If parameter processing unit 203 determines that the local moving object flag determined in step S304 is True, the process proceeds to step S306. On the other hand, if parameter processing unit 203 determines that the local moving object flag determined in step S304 is False, the process proceeds to step S308. In step S306, image acquisition unit 204 selects a foreground image to be a source (material) of the local moving object from the still image data set held in storage unit 201, for example, by random selection. At this time, it is preferable that the number of foreground images selected is the same as the number of local moving objects acquired by parameter acquisition unit 202 in step S302.

[0023] In step S307, the parameter processing unit 203 determines inter-frame horizontal and vertical shift amounts corresponding to each of a plurality of motion patterns in the burst image patch of the current learning loop for each local moving object. For example, when the maximum inter-frame horizontal shift amount acquired in step S302 is 16px (pixels), the parameter processing unit 203 randomly selects and determines the inter-frame horizontal shift amount corresponding to each motion pattern from integers in the range of ±16px. Similarly, when the maximum inter-frame vertical shift amount is 12px, the parameter processing unit 203 randomly selects and determines the inter-frame vertical shift amount corresponding to each motion pattern from integers in the range of ±12px.

[0024] In addition, when the image format is a Bayer array RAW image, the parameter processing unit 203 limits the inter-frame horizontal and vertical shift amounts to even numbers. In this way, when the inter-frame horizontal and vertical shift amounts corresponding to multiple motion patterns are determined by random selection, the motion of the local moving object is generally made different between the multiple motion patterns. This is because the model is trained in a direction that does not use the reference frame of the burst image, that is, in a direction that more strongly suppresses the afterimage, since the part with large movement is likely to cause an afterimage when multiple images are combined.

[0025] Steps S308 to S313 are performed in a motion pattern loop and a frame loop since they are processes related to each frame image constituting a burst image of a plurality of motion patterns. In this embodiment, the number of motion patterns is two. However, this embodiment can be easily extended to a case where the number of motion patterns is three or more. In this embodiment, the number of frames of the burst image is four, the first frame is the base frame, and the remaining second and subsequent frames are the reference frames. The learning model inputs a burst image consisting of four frames, and aims to obtain a multiple-frame synthesis output corresponding to the base frame with improved image quality compared to the case of a single frame input. The base frame may not be the first frame, but may be any frame.

[0026] In step S308, the parameter processing unit 203 determines the horizontal and vertical shift amounts of the background image patch in each frame relative to the reference frame. In this embodiment, it is assumed that alignment (shake correction) is performed as pre-processing during inference using the trained model. This pre-processing is a process of shifting the entire image of the reference frame, which is an image other than the image of the reference frame, to match the image of the reference frame so that the position of the main subject is aligned between frames. It is assumed that the movement of the entire screen, such as camera shake between multiple input images, is largely absorbed before being input to the model. Therefore, in model learning, it is desirable to minimize the amount of shift of the entire image patch, which is considered to be effective for the movement of the entire screen but is likely to cause a loss of resolution in the multiple-image synthesis output.

[0027] In this regard, if the alignment (camera stabilization) is ideal, the entire image patch is not shifted, that is, the maximum horizontal and vertical shift amount of the entire image patch may be 0px. However, in general, camera stabilization residuals occur in the alignment. Therefore, it is preferable to set the maximum horizontal and vertical shift amount of the entire image patch acquired in step S302 to, for example, 2px, depending on the accuracy of the alignment combined as preprocessing (expected camera stabilization residual amount). However, if the shift probability of the entire image patch is set to 100%, even in burst images in which camera stabilization does not occur due to tripod shooting, etc., there may be a problem that the lines become blurred by unnecessarily moving thin lines of each frame image and then compositing multiple images. Therefore, it is preferable to set the shift probability of the entire image patch to about 50%, because this can deal with camera stabilization residuals and also suppresses problems when camera stabilization does not occur.

[0028] Based on the above, in step S308, the horizontal and vertical shift amounts of the background image patch are determined. First, for the first frame (reference frame), since it is the reference, the horizontal and vertical shift amounts of the background image patch are determined to be 0px. Next, for the second and subsequent frames (reference frames), as described above, an example will be described in which the shift probability of the entire image patch acquired in step S302 is 0.5 (50%) and the maximum horizontal and vertical shift amount is 2px. In this case, the parameter processing unit 203 generates a random number between 0 and 1, and if the value of the random number is 0.5 or less, the horizontal and vertical shift amounts of the background image patch are randomly selected from integers in the range of ±2px and determined.

[0029] On the other hand, if the value of the random number is greater than 0.5, the parameter processing unit 203 determines the horizontal and vertical shift amounts of the background image patch to be 0px. Note that while the movement of the local moving object is made different among the multiple patterns in step S307, the movement of the background (the horizontal and vertical shift amounts of the background image patch in each frame) determined in step S308 is made the same among the multiple patterns. This is because, in a portion with small movement where only camera shake correction remains, afterimages are unlikely to occur when multiple images are combined, so that the model is trained in the direction of using the reference frame of the burst image, that is, in the direction of improving the sense of resolution by combining multiple images rather than suppressing afterimages.

[0030] In step S309, the image acquisition unit 204 shifts the patch cut-out position from the background image selected in step S303 based on the horizontal and vertical shift amounts of the background image patches in each frame of the multiple patterns determined in step S308. As a result, the image acquisition unit 204 acquires background image patches 501-504 of a first pattern and background image patches 511-514 of a second pattern, as shown in FIG. 5. In this regard, in step S308, as described above, the horizontal and vertical shift amounts of the background image patches in each frame are made the same among the multiple patterns. Therefore, the background image patches 501-504 and the background image patches 511-514 become the same image. Note that in FIG. 5, the background image patches 501-504 and the background image patches 511-514 are shown as schematic white images. For ease of understanding, let us use an example in which the background image patches in each frame of multiple patterns have a horizontal and vertical shift range greater than ±2px. As shown in Figure 4(a), the entire image patch becomes a burst of shifting motion.

[0031] Returning to the description of FIG. 3, in step S310, parameter processing unit 203 determines whether the local moving object flag determined in step S304 is True or False. If parameter processing unit 203 determines that the local moving object flag determined in step S304 is True, the process proceeds to step S311. On the other hand, if parameter processing unit 203 determines that the local moving object flag determined in step S304 is False, the process proceeds to step S314. Steps S311 to S313 are performed in a local moving object loop, since they are processes for each local moving object whose number was acquired in step S302. In the case of FIG. 5, the number of local moving objects acquired in step S302 is 1.

[0032] In step S311, image acquisition unit 204 acquires partial images that are to become local moving objects from the foreground image corresponding to each local moving object acquired in step S306. Specifically, image acquisition unit 204 randomly determines the size of the partial image, for example, to a size equal to or smaller than the image patch size, randomly determines the position from which the partial image is to be cut out from the foreground image, and cuts out a circular partial image from the foreground image based on these determinations (see FIG. 4(b)). Here, the partial image is circular because a circle has boundaries in various directions and is more suitable than a rectangle or the like. However, the partial image may have any shape.

[0033] In step S312, parameter processing unit 203 determines the superimposition position of the partial image acquired in step S311 onto the background image patch of the multiple patterns acquired in step S309. First, for the first frame (reference frame), the horizontal and vertical superimposition positions are each randomly determined within the background image patch, but since this is a reference, the superimposition positions are the same among the multiple patterns. For the second and subsequent frames (reference frames), the superimposition positions are determined to be positions shifted from the superimposition position of the previous frame by the inter-frame horizontal and vertical shift amounts corresponding to each motion pattern determined in step S307.

[0034] In step S313, the image acquisition unit 204 (acquisition means) superimposes partial images on the background image patches of the multiple patterns based on the superimposition positions of the multiple patterns determined in step S312 (acquisition step). The superimposition of partial images will be specifically described below. FIG. 5 shows two different patterns of movement for one local moving object. For example, it is assumed that the inter-frame horizontal shift amount of the first pattern determined in step S307 is 12px and the vertical shift amount is 8px. In this case, when the partial image corresponding to the local moving object is superimposed on the background image patches 501-504 of the first pattern, it becomes local moving objects 521-524 that flow to the upper right.

[0035] Similarly, the inter-frame horizontal shift amount of the second pattern determined in step S307 is 12px and the vertical shift amount is -8px. In this case, when the partial image corresponding to the local moving object is superimposed on the background image patches 511-514 of the second pattern, it becomes the local moving objects 531-534 that move to the lower right. In this way, the movement of each of the number of local moving objects specified in step S302 differs among the multiple patterns. Note that, when superimposing the partial image, the parameter processing unit 203 may blur the boundary to blend with the background, or may alpha-blend the background instead of completely blocking it. Also, when the partial image protrudes from the background image patch during superimposition, the parameter processing unit 203 may selectively superimpose only the area of ​​the partial image that is within the background image patch.

[0036] Return to the description of Fig. 3. In step S314, the image processing unit 205 (generation means) determines an image patch of a reference frame that is the same among the multiple patterns of burst image patches generated in the processing up to step S313 as a teacher image (generation process). In Fig. 5, the first frame is the reference frame, and 540 is the teacher image. Return to the description of Fig. 3. In step S315, the image processing unit 205 (generation means) generates multiple patterns of student images by performing a predetermined image processing on the multiple patterns of burst image patches generated in the processing up to step S313 (generation process).

[0037] Here, the predetermined image processing is a process of adding noise when noise reduction by multiple image synthesis is the purpose. In this case, noise of an intensity according to the sensitivity targeted by the model is added in particular. In addition, the predetermined image processing is a process of narrowing the dynamic range when HDR (High Dynamic Range) by multiple image synthesis is the purpose. In particular, the reference frame may be processed so as to be overexposed or underexposed. In addition, the predetermined image processing is a process of lowering the resolution when super-resolution by multiple image synthesis is the purpose. That is, the present invention using multiple motion patterns can be applied to high image quality by multiple image synthesis in general. Note that the burst image patch consisting of four frames shown in FIG. 4 is one to which noise has been added as a predetermined image processing. In addition, in FIG. 5, when noise reduction by multiple image synthesis is the purpose, noise is added to two patterns of burst image patches consisting of four frames. At this time, the noise patterns are made different between frames and between motion patterns. Note that the image processing unit 205 may perform multiple predetermined image processing.

[0038] Returning to the explanation of Fig. 3, in step S316, error calculation unit 206 (calculation means) calculates the error to be minimized in the next step S317 (calculation step). This calculation is performed based on a plurality of patterns of composite output images obtained by inputting the plurality of patterns of student images generated in step S315 into the model, and the teacher image determined in step S314. In this embodiment, as shown in Fig. 5, a first pattern of composite output image 541 and a second pattern of composite output image 542 are obtained.

[0039] In general, in learning a model that performs multiple image synthesis, an error (for example, mean absolute error) between a (first pattern) synthesis output image 541 and a teacher image 540, which corresponds to a bidirectional arrow 551 in FIG. 5, is calculated as a (first pattern) reconstruction error. However, in this embodiment, as described above, a second pattern synthesis output image 542 is also obtained. Therefore, an error between a second pattern synthesis output image 542 and a teacher image 540, which corresponds to a bidirectional arrow 552 in FIG. 5, is also calculated as a reconstruction error of the second pattern. Then, the reconstruction error of the first pattern and the reconstruction error of the second pattern are calculated as a reconstruction error (first error) by, for example, taking an average and combining them. In this embodiment, further, an error between the synthesis output image 541 and the synthesis output image 542, which corresponds to a bidirectional arrow 553 in FIG. 5, is calculated as an inter-output error (second error), and a combined error expressed by Equation (1) is calculated by taking the sum of the reconstruction error and the weighted inter-output error. Combining error = reconstruction error + α × error between outputs … (1)

[0040] Here, α is the weight of the inter-output error with respect to the reconstruction error. For example, if the reconstruction error and the inter-output error are both mean absolute errors, α may be set to 1. There is little qualitative difference between the composite output image 541 and the composite output image 542, as there is between the composite output image and the teacher image. Therefore, the inter-output error occurs predominantly in the residual image portion 561 and the residual image portion 562, which are located at different positions in the composite output image 541 and the composite output image 542. Therefore, by minimizing the combined error obtained by adding the inter-output error to the reconstruction error, the residual image can be suppressed more strongly than when only the reconstruction error is minimized.

[0041] In step S317, the learning unit 207 (learning means) updates the model parameters, typically by backpropagation, so as to minimize the combined error calculated in step S316 (learning step). In step S318, the learning unit 207 stores the model that has left the learning loop and completed learning in the storage unit 201. This trained model can be used for inference. As described above, a model in which the weight parameters of a neural network are trained using multiple patterns of burst image patches exhibits a strong afterimage suppression effect even for the input of a single burst image in inference.

[0042] As described above, the image processing device 100 of this embodiment can more effectively suppress afterimages when compositing multiple images by learning a model so that multiple composite output images corresponding to multiple patterns of student images with different subject movements approach each other.

[0043] Although the preferred embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and various modifications and changes are possible within the scope of the gist of the present invention. Below, a modified example will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining a modified example of a burst image patch having a plurality of patterns.

[0044] For example, in step S307 of this embodiment, the inter-frame horizontal and vertical shift amounts corresponding to the multiple motion patterns of the local moving object are randomly selected. However, the inter-frame horizontal and vertical shift amounts (i.e., the direction in which the position of the subject changes) corresponding to the multiple motion patterns of the local moving object may be determined to be in the maximum different directions. Specifically, for example, when the inter-frame horizontal shift amount of the first pattern is 12px and the vertical shift amount is 8px, the signs are inverted and the inter-frame horizontal shift amount of the second pattern is determined to be -12px and -8px. Furthermore, when the partial image corresponding to the local moving object is superimposed on the background image patches 511-514 of the second pattern by the image acquisition unit 204 in step S313, the local moving objects 631-634 flow to the lower left as shown in FIG. 6. In this embodiment, since there is a small overlap between the residual image portion 561 of the composite output image 541 and the residual image portion 562 of the composite output image 542 in FIG. 5, the overlapping portion is less likely to cause an inter-output error, and the effect of suppressing the residual image may be slightly reduced. On the other hand, in this modified example, there is no overlap between afterimage portion 561 of composite output image 541 in FIG. 6 and afterimage portion 662 of composite output image 642, and it is believed that by making the multiple movement patterns as different in direction as possible, the afterimage can be suppressed more effectively than in the case of FIG. 5.

[0045] In addition, in step S308 of this embodiment, in order to make the model learn in a direction that improves the sense of resolution by combining multiple images rather than suppressing residual images, the movement of the background image patch with a small horizontal and vertical shift amount is made the same among multiple patterns for a part with small movement. This idea can also be applied to the movement of a locally moving object. Specifically, for a locally moving object, if the magnitude of the motion vector having the inter-frame horizontal and vertical shift amounts of the first pattern as elements is smaller than the threshold value (predetermined value) of 6px, the inter-frame horizontal and vertical shift amounts of the second pattern are made the same as those of the first pattern. For example, if the inter-frame horizontal shift amount of the first pattern is 4px and the vertical shift amount is 4px, the magnitude of the motion vector having these as elements (the amount of change in the position of the subject) is about 5.7px, so the inter-frame horizontal and vertical shift amounts of the second pattern are also made the same.

[0046] Moreover, by generalizing the above, the directions of multiple motion patterns may be made different so that the model learns in a direction that suppresses afterimages more strongly in areas with greater motion. For example, for a locally moving object, the larger the magnitude of the motion vector of the first pattern (the amount of change in the position of the object), the larger the angle formed with the motion vector is determined by the horizontal and vertical shift amounts between frames of the second pattern (i.e., the direction in which the position of the object changes).

[0047] In addition to the above, the error calculation unit 206 may control the magnitude of the inter-output error term by controlling, for example, the weight α of the inter-output error in formula (1). That is, the error calculation unit 206 can strengthen the effect of suppressing afterimages by increasing the inter-output error term. Conversely, the error calculation unit 206 can weaken the effect of suppressing afterimages and strengthen the effect of improving the sense of resolution by the reconstruction error term by decreasing the inter-output error term. For example, when the magnitude (amount of change in the position of the subject) of the first pattern of the locally moving body is smaller than a threshold value, the error calculation unit 206 may set the weight α of the inter-output error to 0. Furthermore, the error calculation unit 206 may increase the weight α of the inter-output error for the locally moving body as the magnitude (amount of change in the position of the subject) of the first pattern increases. Note that the error calculation unit 206 may, for example, weight the reconstruction error in formula (1) to control the weight of the reconstruction error, or may control the weight of the reconstruction error together with the weight α of the inter-output error described above.

[0048] In this embodiment, among the background and the local moving object constituting the subject, the movement of the local moving object is made different among the multiple patterns as shown in FIG. 5. However, the present invention is also applicable to a case where only the movement of the background constituting the subject is learned without using the local moving object as shown in FIG. 4(a). In this case, for example, the maximum horizontal and vertical shift amounts of the entire image patch acquired in step S302 may be designated as large values ​​like the local moving object. Furthermore, in step S308, the parameter processing unit 203 may determine the horizontal and vertical shift amounts of the background image patch in each frame to be made different among the multiple patterns like the local moving object.

[0049] In addition, the present invention, which uses multiple motion patterns, can be applied not only to an image processing model in which multiple burst images are input and multiple images are synthesized and output, but also to a video processing model in which multiple video frames are input sequentially and multiple synthesized video frames are sequentially output.

[0050] The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiment to a system or device via a network or a storage medium, and having one or more processors of a computer in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0051] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An image processing device that learns a model of image processing in which multiple images are combined, an acquisition means for acquiring an image group including a predetermined subject in a plurality of patterns, the image group being composed of a plurality of frame images including an image of a reference frame; a generating means for generating a group of student images of the plurality of patterns by performing a predetermined image processing on the group of images of the plurality of patterns, while using the image of the reference frame as a teacher image including the subject; a calculation means for calculating a first error between the teacher image and the composite output images of the plurality of patterns of the model corresponding to the inputs of the group of the plurality of student images, respectively, and a second error between the composite output images of the plurality of patterns; a learning means for learning the model based on the first error and the second error, The image processing device according to claim 1, wherein the acquisition means changes a position of the subject among the plurality of patterns in an image of one or more frames other than the image of the reference frame. (Configuration 2) The image processing device according to configuration 1, wherein the model is a neural network. (Configuration 3) The image processing device according to configuration 1 or 2, wherein the predetermined image processing is at least one of a process of adding noise, a process of narrowing a dynamic range, and a process of lowering resolution. (Configuration 4) The image processing device according to any one of configurations 1 to 3, wherein the acquisition means maximizes the difference between the multiple patterns in the direction in which the position of the subject changes between the multiple frame images. (Configuration 5) The image processing device described in any one of configurations 1 to 3, characterized in that the acquisition means makes the position of the subject the same across the multiple patterns when the amount of change in position of the subject between the images of the multiple frames in at least any of the multiple patterns is smaller than a predetermined value. (Configuration 6) An image processing device as described in any one of configurations 1 to 3, characterized in that the acquisition means makes the direction in which the position of the subject changes between the images of the multiple frames more different between the multiple patterns when the amount of change in the position of the subject between the images of the multiple frames in at least any of the multiple patterns is greater. (Configuration 7) The image processing device described in any one of configurations 1 to 6, characterized in that the calculation means controls at least one of the first error weight or the second error weight depending on the amount of change in the position of the subject between the images of the multiple frames in at least one of the multiple patterns. (Method 1) A control method for an image processing device that learns a model of image processing in which multiple images are combined, comprising the steps of: an acquisition step of acquiring an image group including a plurality of frames of images including a reference frame, the image group including a predetermined subject, in a plurality of patterns; a generating step of generating a group of student images of the plurality of patterns by performing a predetermined image processing on the group of images of the plurality of patterns while using the image of the reference frame as a teacher image including the subject; A calculation step of calculating a first error between the teacher image and the composite output images of the plurality of patterns of the model corresponding to the inputs of the group of the plurality of student images, respectively, and a second error between the composite output images of the plurality of patterns; A learning step of learning the model based on the first error and the second error, A control method for an image processing apparatus, wherein the obtaining step varies a position of the subject among the plurality of patterns in an image of one or more frames other than the image of the reference frame. (Program 1) A program for causing a computer to execute each of the means of the image processing device according to any one of configurations 1 to 7. [Explanation of symbols]

[0052] 100 Image processing device 204 Image acquisition unit (acquisition means) 205 Image processing unit (generation means) 206 Error calculation unit (calculation means) 207 Learning section (learning means)

Claims

1. An image processing device that learns a model of image processing in which multiple images are combined, an acquisition means for acquiring an image group including a predetermined subject in a plurality of patterns, the image group being composed of a plurality of frame images including an image of a reference frame; a generating means for generating a group of student images of the plurality of patterns by performing a predetermined image processing on the group of images of the plurality of patterns, while using the image of the reference frame as a teacher image including the subject; a calculation means for calculating a first error between the teacher image and the plurality of patterns of composite output images of the model corresponding to the input of the plurality of patterns of the student image group, and a second error between the plurality of patterns of composite output images; a learning means for learning the model based on the first error and the second error, The image processing device according to claim 1, wherein the acquisition means changes a position of the subject among the plurality of patterns in an image of one or more frames other than the image of the reference frame.

2. 2. The image processing apparatus according to claim 1, wherein the model is a neural network.

3. 2. The image processing apparatus according to claim 1, wherein the predetermined image processing is at least one of a process of adding noise, a process of narrowing a dynamic range, and a process of lowering a resolution.

4. 2 . The image processing apparatus according to claim 1 , wherein the acquisition means maximizes the difference between the plurality of patterns in a direction in which the position of the subject changes between the images of the plurality of frames.

5. The image processing device according to claim 1, characterized in that the acquisition means makes the position of the subject the same across the multiple patterns when an amount of change in the position of the subject between the images of the multiple frames in at least any of the multiple patterns is smaller than a predetermined value.

6. The image processing device according to claim 1, characterized in that the acquisition means makes the direction in which the position of the subject changes between the images of the multiple frames more different between the multiple patterns when the amount of change in the position of the subject between the images of the multiple frames in at least any of the multiple patterns is greater.

7. 2. The image processing device according to claim 1, wherein the calculation means controls at least one of the first error weight and the second error weight in accordance with an amount of change in the position of the subject between the images of the multiple frames in at least one of the multiple patterns.

8. A control method for an image processing device that learns a model of image processing in which multiple images are combined, comprising: an acquisition step of acquiring an image group including a predetermined subject in a plurality of patterns, the image group being composed of a plurality of frame images including an image of a reference frame; a generating step of generating a group of student images of the plurality of patterns by performing a predetermined image processing on the group of images of the plurality of patterns while using the image of the reference frame as a teacher image including the subject; a calculation step of calculating a first error between the teacher image and the composite output images of the plurality of patterns of the model corresponding to the inputs of the group of the plurality of student images, respectively, and a second error between the composite output images of the plurality of patterns; A learning step of learning the model based on the first error and the second error, A control method for an image processing apparatus, wherein the acquiring step varies a position of the subject among the plurality of patterns in an image of one or more frames other than the image of the reference frame.

9. 2. A program for causing a computer to execute each of the means of the image processing apparatus according to claim 1.

Images