Image processing apparatus, image processing method, imaging apparatus, storage medium, and computer program product

Abstract

The invention provides an image processing apparatus, an image processing method, an image capturing apparatus, a storage medium, and a computer program product. The image processing apparatus obtains a plurality of frames of first images, wherein viewpoints of the first images are different from each other. The apparatus generates a frame of second image having a higher resolution than the first image by combining the plurality of frames of first images, and generates a third image having the same resolution as the second image from a frame of first image among the plurality of frames of first images. The apparatus detects a changed region that has changed between the plurality of frames of first images, and also generates a fourth image as a high-resolution image based on the plurality of frames of first images from the second image and a region in the third image corresponding to a region in the second image that corresponds to the changed region.

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method, and an image pickup device, and particularly to a technique for increasing the resolution of an image (the number of pixels constituting an image) using image synthesis. Background Art

[0002] A technology is known that allows a digital camera including an image sensor whose position can be changed to generate a still image having a resolution exceeding the number of pixels in the image sensor by combining a plurality of still images having a position difference smaller than a pixel pitch (Japanese Patent Laid-Open No. 2012-226489).

[0003] In the case of combining multiple frames of still images with positional differences, areas showing changes in position and brightness between frames give an unnatural impression in the composite image. This imposes significant restrictions on the subjects and scenes for which a desired composite image can be obtained. Summary of the invention

[0004] In one embodiment of the present invention, the present invention provides an image processing apparatus capable of improving the above-mentioned problem of the technique of increasing the resolution of an image using image synthesis.

[0005] According to one aspect of the present invention, there is provided an image processing device, comprising: an obtaining component for obtaining a plurality of frames of first images, wherein the viewpoints of the first images are different from each other; a generating component for generating a frame of second image having a higher resolution than the first image by combining the plurality of frames of first images, and generating a third image having the same resolution as the second image from a frame of first image among the plurality of frames of first images; and a detecting component for detecting a changed area that has changed between the plurality of frames of first images, wherein the generating component also generates a fourth image as a high-resolution image based on the plurality of frames of first images from the second image and an area in the third image corresponding to an area in the second image corresponding to the changed area.

[0006] According to another aspect of the present invention, there is provided an image processing method, which is performed by an image processing device, and the image processing method includes: obtaining a plurality of frames of first images, wherein the viewpoints of the first images are different from each other; generating a frame of second image having a higher resolution than the first image by combining the plurality of frames of first images; generating a third image having the same resolution as the second image from a frame of first image among the plurality of frames of first images; detecting a changed area that has changed between the plurality of frames of first images; and generating a fourth image as a high-resolution image based on the plurality of frames of first images from an area in the second image and in the third image corresponding to an area in the second image corresponding to the changed area.

[0007] According to still another aspect of the present invention, there is provided an imaging device including: an image sensor; and an image processing device according to the present invention.

[0008] Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings). BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram showing a typical functional configuration of a digital camera as one example of the image processing apparatus according to the embodiment.

[0010] Figure 2 is a flowchart related to a process for increasing resolution according to an embodiment.

[0011] Figure 3 is a schematic diagram showing an example of a plurality of frames of low-resolution images including a changed region.

[0012] Figure 4 is a schematic diagram showing a high-resolution image obtained by combining a plurality of frames of low-resolution images.

[0013] Figure 5 It is a schematic diagram of obtaining a high-resolution image from a low-resolution image.

[0014] Figure 6 is a schematic diagram showing an example of a change region map.

[0015] Figure 7 is a schematic diagram showing an example of a high-resolution image finally obtained. DETAILED DESCRIPTION

[0016] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but the invention requiring all such features is not limited, and a plurality of such features may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are assigned to the same or similar structures, and redundant descriptions thereof are omitted.

[0017] Note that the following embodiments will be described in relation to the case where the present invention is implemented on a camera device (digital camera). However, the camera function is not essential in the present invention. The present invention can be implemented on any electronic device capable of handling image data. Such electronic devices include not only camera devices, but also computer devices (personal computers, tablet computers, media players or PDAs, etc.), smart phones, game devices, robots, drones, and driving recorders. These are examples, and the present invention can also be implemented on other electronic devices.

[0018] (First embodiment)

[0019] Figure 1 1 is a block diagram showing an example of a basic functional configuration of a digital camera 100 as an example of an image processing device according to the first embodiment of the present invention. Except for components that can obviously be realized only by hardware (for example, a lens and an image sensor included in the optical system 103, etc.), each functional block of the digital camera 100 can be realized by software or by a combination of software and hardware. For example, a functional block can be realized by dedicated hardware such as an ASIC. In addition, a functional block can be realized by a processor such as a CPU executing a program stored in a memory. Note that a plurality of functional blocks can be realized by the same constituent element (for example, one ASIC). In addition, hardware for realizing a part of the functions of a specific functional block can be included in hardware for realizing other functional blocks.

[0020] The control unit 101 includes one or more processors (CPU, MPU, etc.) that can execute programs, reads programs stored in, for example, the ROM 107 into the RAM 108, and executes the programs using the processor(s). The control unit 101 controls the operations of the functional blocks of the digital camera 100 by executing the programs and implements the functions of the digital camera 100. In addition, the control unit 101 performs automatic focus detection (AF), automatic exposure control (AE), and control of the operations of the optical system 103 and the imaging unit 105 according to the results of AF and AE.

[0021] The ROM 107 is a rewritable nonvolatile memory, and stores programs executed by the control unit 101, various types of setting values ​​of the digital camera 100, image data (GUI data) for menu screens and on-screen displays (OSD), and the like.

[0022] The RAM 108 is used as a main memory of the control unit 101, a buffer for temporarily storing captured image data, a work memory of the image processing unit 109, etc. Furthermore, a part of the RAM 108 can be used as a video memory for storing image data for display.

[0023] The optical system 103 includes a plurality of lenses including a movable lens, a shutter, an aperture, and the like. The movable lens includes a focus lens for adjusting the focusing distance of the optical system and a zoom lens for adjusting the focal length (angle of view) of the optical system 103, and the like. The movable member included in the optical system 103 is driven by a motor and an actuator included in the drive unit 102. The control unit 101 controls the operation of the drive unit 102. The control unit 101 can obtain information related to the state of the optical system 103 (for example, the position and F value of the focus lens, etc.) via the communication unit 104.

[0024] The optical system 103 forms an optical image of a subject on an imaging surface of the imaging unit 105. The imaging unit 105 may be, for example, a known CCD or CMOS color image sensor including a color filter based on a primary color Bayer arrangement.

[0025] The imaging unit 105 includes a pixel array in which a plurality of pixels are arranged two-dimensionally and a peripheral circuit for reading out a signal from each pixel. Each pixel accumulates a charge corresponding to the amount of incident light by means of photoelectric conversion. A signal having a voltage corresponding to the amount of charge accumulated during the exposure period is read out from each signal; as a result, a pixel signal group (analog image signal) representing the subject image formed on the imaging plane is obtained. The analog image signal is supplied to the image processing unit 109. The control unit 101 controls the operation of the imaging unit 105.

[0026] The image pickup unit 105 is also configured to be able to move a predetermined amount in each of the horizontal direction and the vertical direction and the clockwise direction and the counterclockwise direction around the optical axis on a plane perpendicular to the optical axis of the optical system 103. The control unit 101 can control the position of the image pickup unit 105 by controlling a driving member such as a motor included in the driving unit 102. The position of the image pickup unit 105 is controlled to realize a function for correcting optical image blur and a function for generating a high-resolution image.

[0027] The communication unit 104 is, for example, a communication interface between the control unit 101 and the optical system 103 .

[0028] The shake detection unit 106 outputs a signal corresponding to the motion of the digital camera 100. The shake detection unit 106 may include a gyroscope for detecting angular velocities about respective axes of a three-dimensional Cartesian coordinate system including an axis in the gravity direction and an axis in the optical axis direction, and an acceleration sensor for detecting accelerations in the respective axis directions.

[0029] The image processing unit 109 applies image processing to the analog image signal output from the camera unit 105 or the image data that has been read from the recording medium 111 into the RAM 108, thereby generating a signal and image data that match the desired use and obtaining and / or generating various types of information. The image processing unit 109 may be, for example, a dedicated hardware circuit such as an application-specific integrated circuit (ASIC) that has been designed to realize a specific function. The image processing unit 109 outputs the obtained or generated information and data to the control unit 101 and the RAM 108 according to the desired use.

[0030] The image processing applied by the image processing unit 109 may include, for example, preprocessing, color interpolation processing, correction processing, detection processing, data editing processing, evaluation value calculation processing, special effect processing, and the like.

[0031] Pre-processing may include A / D conversion, noise reduction, signal amplification, reference level adjustment, and defective pixel correction.

[0032] The color interpolation process is a process that is performed when the image pickup unit 105 includes a color filter and interpolates the values ​​of color components not included in the respective pixel data constituting the image data. The color interpolation process is also called a demosaic process.

[0033] The correction processing may include processing such as white balance adjustment, color tone correction, correction of image degradation caused by optical aberration of the optical system 103 (image restoration), correction of vignetting effects of the optical system 103, and color correction. The color interpolation processing and the correction processing may be collectively referred to as development processing.

[0034] The detection process may include detection of a feature region (a face region or a human body region, etc.) and motion in the region, and processing for recognizing a person, etc.

[0035] The data editing process may include processes such as cutting out (cropping), synthesis, scaling, encoding and decoding of an area, and generation of header information (generation of a data file). The data editing process also includes generation of display image data and recording image data. The data editing process also includes generation of a high-resolution image to be described later.

[0036] The evaluation value calculation process may include processes such as generation of a signal and an evaluation value used in automatic focus detection (AF) and generation of an evaluation value used in automatic exposure control (AE).

[0037] Special effect processing may include processing such as the addition of blur effects, changes in color shading, and relighting.

[0038] Note that these are examples of processing that the image processing unit 109 can apply, and are not intended to limit the processing applied by the image processing unit 109. A portion of the processing listed above may be performed as a result of the control unit 101 executing a program.

[0039] In the present embodiment, the image processing unit 109 performs a first resolution increase by combining a plurality of frames of low-resolution images having a position difference (i.e., different viewpoints or observation points) with each other, thereby generating a high-resolution image. The plurality of frames of low-resolution images include a reference frame and a frame having a position difference of a certain amount in a certain direction relative to the viewpoint (observation point) of the reference frame. In addition, the image processing unit 109 performs a second resolution increase, thereby converting a frame of low-resolution image into a high-resolution image. Any known method may be used to perform each of the first resolution increase and the second resolution increase.

[0040] For example, the method described in Japanese Patent Laid-Open No. 2012-226489, or the method of increasing the pixel density by inserting pixels at positions corresponding to the position difference, etc. can be used as the first resolution increase. On the other hand, a method (based on ML) using a trained machine learning (ML) model or a method (non-based on ML) not using a machine learning model can be used as the second resolution increase. Non-ML-based methods include, but are not limited to, the nearest neighbor method, the bilinear method, and the bicubic method, etc.

[0041] Because the first resolution increase can achieve similar benefits to those achieved by reducing the pixel pitch of the image sensor, the first resolution increase enables the generation of a high-resolution image with higher quality than the second resolution increase, as long as the optical resolution limit is not exceeded.

[0042] In the case where an ML-based method is used as the second resolution increase, a trained machine learning model that has been configured to increase the resolution of an input image of a certain size at a predetermined ratio and output the resulting image is provided to the image processing unit 109. The machine learning model can be implemented by using, for example, a two-dimensional convolutional neural network (CNN). Note that such a machine learning model can be implemented by training a CNN using training data including a high-resolution image as ground truth data and a low-resolution image generated from the high-resolution image as input data. Note that the machine learning model can be implemented based on any other known configuration and method.

[0043] The display unit 110 is provided, for example, on the surface of the housing of the digital camera 100, and displays images that have been captured or reproduced, information of the digital camera 100, and a graphical user interface, etc. The display unit 110 may be a liquid crystal or organic EL display. Performing the capturing of moving images and the display of the captured moving images in parallel and continuously allows the display unit 110 to function as an electronic viewfinder (EVF). The operation of causing the display unit 110 to function as an EVF is referred to as live view display, and the moving image displayed as a live view is referred to as a live view image.

[0044] The recording medium 111 serves as a destination for recording data files for storing data such as images captured by the imaging unit 105 or images processed by the image processing unit 109. The recording medium 111 may be a memory card attachable to and detachable from the digital camera 100, or a nonvolatile memory built into the digital camera 100, or the like.

[0045] The operation unit 112 is a general term for input devices (buttons, switches, dials, etc.) provided for a user to input various types of instructions to the digital camera 100. In addition, the operation unit 112 may include, for example, a zoom ring and a focus ring mounted on a lens barrel accommodating the optical system 103.

[0046] The input devices constituting the operation unit 112 have names corresponding to the functions assigned thereto. For example, the operation unit 112 includes a release switch, a moving image recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, a direction key, and an OK key. The release switch is a switch for recording a still image; the control unit 101 recognizes a state in which the release switch is half-pressed as a shooting preparation instruction, and recognizes a state in which the release switch is fully pressed as a shooting start instruction. In addition, the control unit 101 recognizes pressing the moving image recording switch in the shooting standby state as an instruction to start recording a moving image, and recognizes pressing the moving image recording switch during recording a moving image as an instruction to stop recording. Note that the function assigned to the same input device may be variable. In addition, the input device may be a software button or key using a touch display. In addition, the operation unit 112 may include an input device that supports non-contact input methods such as voice input and line of sight input.

[0047] Next, the processing for generating a high-resolution image according to the present embodiment will be described. As described above, the image processing unit 109 performs processing for generating a high-resolution image. Note that the generation of a high-resolution image can be performed in a case where, for example, a shooting mode (super-resolution mode) for generating a high-resolution image has been set. In a case where the super-resolution mode has been set, the control unit 101 performs the shooting of multiple frames of low-resolution images to be used in the first resolution increase. The image processing unit 109 generates one frame of high-resolution image using the multiple frames of low-resolution images that have been shot. Note that the generation of a high-resolution image can be performed immediately after shooting the multiple frames of low-resolution images, or can be performed for multiple frames of low-resolution images that have been pre-recorded in the recording medium 111.

[0048] Figure 21 is a flowchart related to the operations performed by the digital camera 100 when a shooting start instruction for a still image has been detected in a state where the super-resolution mode is set. The shooting start instruction for a still image may be detection of a full press of a release switch, or may be expiration of the standby period of the Selfie. Note that, assuming that when a shooting preparation instruction is detected before the shooting start instruction is detected, the control unit 101 has already performed processing for determining shooting conditions (F value, shutter speed, and shooting sensitivity) (AE) and processing for automatic focus detection (AF) of the optical system 103. Note that the shooting conditions may be values ​​that have been set by the user. In addition, the focusing distance of the optical system 103 may also be manually set by the user.

[0049] In step S201, the control unit 101 determines the number of captured frames and the pixel shift amount of the low-resolution image (first image) used as the basis of the high-resolution image (second image). Note that the low resolution (first resolution) refers to an image with a smaller number of pixels than the resolution (second resolution) of the composite image to be generated in the end. In practice, it can be a still image with the highest resolution that the imaging unit 105 can capture.

[0050] The number of photographing frames may be a predetermined fixed value, or may be selectable by the user from a plurality of options, such as 3, 5, and 9. As an example, it is assumed here that nine frames are photographed.

[0051] Various methods can be used to determine the pixel offset intended to achieve different viewpoints (observation points). The pixel offset can be a fixed value, or it can be a value corresponding to the number of captured frames. In the case of increasing the pixel density by synthesis, the pixel offset has a value representing a scale smaller than the pixel pitch. The pixel pitch is the distance between the centers of adjacent pixels; it is assumed here that the pixel pitch is the same in the horizontal and vertical directions. As an example, it is also assumed that the pixel offset is pixel pitch / 2 in both the horizontal and vertical directions.

[0052] The method of determining at least one of the number of photographing frames and the pixel shift amount may be changed depending on which method is used to combine a plurality of frames of low-resolution images and generate one frame of high-resolution image.

[0053] Note that here, the imaging unit 105 is moved on the premise that the digital camera 100 is fixed to a tripod or the like so that the low-resolution image has different viewpoints. However, it is possible to take advantage of the movement of the digital camera 100 such as camera shake to shoot a plurality of frames of low-resolution images with different viewpoints. In addition, although it is assumed here that a function of continuously shooting still images is used to shoot a plurality of frames of low-resolution images, it is permitted to shoot a moving image and use one frame of the moving image as a low-resolution image. In the case of shooting a moving image, the frame rate may be determined in consideration of the period required to move the imaging unit 105.

[0054] From step S202 to step S205 , the control unit 101 performs processing for capturing a low-resolution image corresponding to the determined frame number.

[0055] In step S202, the control unit 101 moves the imaging unit 105 (image sensor) to achieve a pixel shift amount corresponding to a frame by controlling the driving unit 102. It is assumed that the moving direction of the imaging unit 105 has been determined in advance for each frame according to the number of captured frames. Note that the reference frame (e.g., the first frame) is captured without moving the imaging unit 105.

[0056] Here, in order to capture nine frames, the control unit 101 moves the imaging unit 105 as follows when capturing each frame, for example. Thus, nine frames of low-resolution images having different viewpoints are obtained.

[0057] First frame: no movement

[0058] Second frame: pixel spacing to the right / 2

[0059] Third frame: pixel spacing to the left / 2

[0060] Fourth frame: pixel spacing in upward direction / 2

[0061] Fifth frame: pixel spacing in downward direction / 2

[0062] Sixth frame: pixel spacing in the right direction / 2, pixel spacing in the upward direction / 2

[0063] Seventh frame: pixel spacing in the right direction / 2, pixel spacing in the downward direction / 2

[0064] Eighth frame: pixel spacing to the left / 2, pixel spacing to the top / 2

[0065] Ninth frame: pixel spacing to the left / 2, pixel spacing to the bottom / 2

[0066] When the movement of the imaging unit 105 has been completed, the control unit 101 executes step S203 .

[0067] In step S203 , the control unit 101 performs capturing of one frame of a still image by controlling the optical system 103 and the imaging unit 105 .

[0068] In step S204, the control unit 101 reads out an analog image signal from the imaging unit 105. The image processing unit 109 applies A / D conversion to the analog image signal and stores the resulting signal into the RAM 108. At this time, the image processing unit 109 does not perform processing for increasing the resolution.

[0069] In step S205, the control unit 101 determines whether the shooting of the number of frames determined in step S201 is completed; if it is determined that the shooting is completed, step S206 is executed, and if it is not determined that the shooting is completed, step S202 is executed.

[0070] In step S206, the control unit 101 instructs the image processing unit 109 to perform generation of a high-resolution image. In response to the instruction from the control unit 101, the image processing unit 109 performs a first resolution increase on the low-resolution image data corresponding to the plurality of frames stored in the RAM 108.

[0071] Figure 3 An example of low-resolution image data corresponding to nine frames stored in the RAM 108 is shown in the form of frame-by-frame images. 301 represents an image of the first frame. In images 302 to 309 of the second and subsequent frames, the position of the human subject 310 has changed from the first frame image.

[0072] Using the data of the low-resolution image (first image) corresponding to nine frames, the image processing unit 109 performs a first resolution increase using pixel interpolation described in, for example, Japanese Patent Laid-Open No. 2012-226489, thereby generating data of a first high-resolution image (second image) of one frame. Note that the method for generating data of a high-resolution image of one frame by combining low-resolution image data corresponding to a plurality of frames may be other known methods.

[0073] Figure 4 Schematically showing the combination of Figure 3 The first high-resolution image 401 is generated from the low-resolution images 301 to 309 shown in FIG. Figure 4 As shown, a human subject 310 whose position changes in the multiple frame images to be combined is a blurred image or an overlapping image in the high-resolution image as a composite image. A similar problem occurs for a subject whose shape changes in the multiple frame images to be combined.

[0074] The image processing unit 109 stores the data of the first high-resolution image generated by the first resolution increase in the RAM 108. Thereafter, the image processing unit 109 executes step S207.

[0075] In step S207, the image processing unit 109 performs a second resolution increase on the data of one frame of low-resolution images among the nine frames of low-resolution images (first images), thereby generating data of one frame of second high-resolution image (third image). It is assumed here that the second resolution increase uses a trained machine learning model for increasing the resolution of an input image and outputting the resulting image. The second high-resolution image has the same resolution as the first high-resolution image. Note that the use of a machine learning model requires a large amount of memory and computing resources. For this reason, a frame of low-resolution image can be divided into multiple blocks, and the resolution can be increased in units of blocks. In this case, the second high-resolution image can be obtained by splicing (joining) the high-resolution images in each block together.

[0076] It is assumed that the image to which the second resolution increase is applied is, for example, the image of the first frame. This is because the image of the first frame is considered to be the image desired by the user. However, the image of the second frame or subsequent frames may be used. Note that the second resolution increase may be performed based on a method that does not use a machine learning model.

[0077] Figure 5 Schematically showing the Figure 3 The second high-resolution image 501 is generated from the low-resolution image 301 shown. Figure 5 As shown, since the resolution of one frame of image has been increased, the human subject 310 is not a blurred image or an overlapping image. The image processing unit 109 stores the data of the second high resolution image generated by the second resolution increase into the RAM 108. Thereafter, the image processing unit 109 executes step S208.

[0078] In step S208, the image processing unit 109 detects an area (change area) showing a change between multiple frames of low-resolution images. Although various methods can be used to detect the change area, the following describes a method using a first high-resolution image and a second high-resolution image. The first high-resolution image is a composite image generated using nine frames of low-resolution images. On the other hand, the second high-resolution image is an image generated using one frame of low-resolution image. Therefore, in the difference image between the first high-resolution image and the second high-resolution image, an area where the absolute value of the difference is equal to or greater than a specific value is regarded as a change area. The difference can be obtained in pixels, or can be obtained in blocks. In addition, the difference image can be obtained for each color plane of the color filter. In addition, the difference between elements other than brightness values ​​can be obtained.

[0079] Figure 6 An example of a change region map 601 obtained by binarizing a difference image by applying a threshold is shown, wherein the change region map 601 is obtained by deriving the pixel value in units of pixels. Figure 4The first high resolution image 401 is shown with Figure 5 The difference image is obtained by measuring the absolute value of the difference in brightness between the image 501 and the second high resolution image 501. Note that the threshold value may be obtained in advance through experiments and stored in the ROM 107 or the image processing unit 109, for example.

[0080] In the change region map 601, the black portion is an area where the difference is less than the threshold and no change occurs. On the other hand, the white portion is a change region where the difference is equal to or greater than the threshold. It can be said that the change region map 601 is an image for indicating the location where the change region exists in the first high-resolution image.

[0081] The image processing unit 109 stores the generated change region map 601 into the RAM 108. Thereafter, the image processing unit 109 executes step S209.

[0082] In step S209, the image processing unit 109 generates a third high-resolution image (fourth image) from the first high-resolution image (second image) and the second high-resolution image (third image) based on the change region detected in step S208. Specifically, the image processing unit 109 generates the third high-resolution image by replacing the region corresponding to the detected change region in the first high-resolution image with the second high-resolution image. The third high-resolution image is the high-resolution image finally generated. The image processing unit 109 cuts out the region corresponding to the change region from the second high-resolution image, for example, using the change region map 601 as a mask. Then, the image processing unit 109 replaces the region corresponding to the change region in the first high-resolution image with the image that has been cut out.

[0083] Alternatively, the third high-resolution image may also be generated by selecting the first high-resolution image for the black area in the change area map 601 and selecting the second high-resolution image for the white area therein.

[0084] Figure 7 An example of a third high resolution image 701 is shown which has been generated from the first high resolution image 401 and the second high resolution image 501 using the change region map 601. An appropriate high resolution image has been obtained without blurring or overlapping of the human subject 310.

[0085] As described above, the first high-resolution image has higher quality than the second high-resolution image. Therefore, only the portion of the first high-resolution image corresponding to the changed area is replaced with the second high-resolution image, which results in a third high-resolution image having higher quality than the second high-resolution image and not including defects in the area showing the changes between the multiple frames of low-resolution images.

[0086] Note that the smaller the area replaced with the second high-resolution image, the more advantageous it is in terms of image quality; however, the processing load associated with the generation of the third high-resolution image can be reduced by simplifying the changed area. For example, a rectangular area inscribed with the changed area can be the area to be replaced with the second high-resolution image.

[0087] The image processing unit 109 stores the generated data of the third high resolution image in the RAM 108. Thereafter, the image processing unit 109 executes step S210.

[0088] In step S210, the image processing unit 109 generates a data file storing data of the third high resolution image, and records the data file into the recording medium 111. Note that in the case where the digital camera 100 can communicate with an external device, the data file may be output to the external device.

[0089] Although it is assumed here that only the data of the third high-resolution image is recorded, at least one of the data of the first high-resolution image and the data of the second high-resolution image may also be recorded. In addition, although the foregoing description relates to the case where the high-resolution image is recorded as data in a raw (RAW) format, it may be recorded after the image processing unit 109 has performed image processing such as development processing and encoding processing thereon.

[0090] (Variation Example)

[0091] A description is now given of a modified example of the process of detecting a changed region in step S208. A changed region may be detected using a low-resolution image instead of the first high-resolution image and the second high-resolution image. Specifically, the image processing unit 109 may detect a changed region based on an image of a difference between a plurality of frames of low-resolution images obtained by the process of steps S202 to S205.

[0092] For example, the image processing unit 109 stores the image of the first frame as the reference image in the RAM 108, and generates images that are different from the image of the first frame for each of the images of the second frame and the subsequent frames. The image processing unit 109 can generate the reference image by applying a threshold to a summed difference image obtained by summing the difference images of eight frames, or by applying a threshold to a difference image obtained by dividing the summed difference image by the number of frames (8). Figure 6The change region map 601 shown. Note that in the case where the change region map 601 is generated from the low-resolution image, the image processing unit 109 uses the change region map 601 after enlarging the change region map 601 to the same resolution as the high-resolution image in step S209. Alternatively, the difference image may be generated after enlarging the low-resolution image to the same resolution as the high-resolution image. The second resolution increase may be used for these types of enlargement. Since the change region map 601 is a binary image, a non-ML-based method with simple processing is used in the case of enlarging the change region map 601.

[0093] Note that in the case of detecting a changed region from a low resolution image, a difference image may be generated while performing the processing of steps S202 to S205. In this case, a changed region may be detected by performing step S208 before generating a first high resolution image in step S206.

[0094] According to the present embodiment, in a first high-resolution image obtained by combining a plurality of low-resolution images, a region showing a change between the plurality of low-resolution images is replaced with a high-resolution image obtained from the first low-resolution image. Thus, a high-resolution image having higher quality in an unchanged region and not including defects in a changed region can be obtained.

[0095] Note that even if the region showing the change between the multiple frames of low-resolution images is not completely replaced with the high-resolution image obtained from the one frame of low-resolution image, the image quality can be improved by using the high-resolution image obtained from the one frame of low-resolution image. For example, in the region showing the change between the multiple frames of low-resolution images, a frame of the first high-resolution image obtained by combining the multiple frames of low-resolution images can be combined with the high-resolution image obtained from the one frame of low-resolution image. It is sufficient to set the proportion in the combination according to the degree of change in the region showing the change between the low-resolution images; it is sufficient that the proportion of the high-resolution image obtained from the one frame of low-resolution image in the combination is larger when the degree of change is large than when the degree of change is small. In this way, by using an image based on the high-resolution image obtained from the one frame of low-resolution image in the region showing the change between the multiple frames of low-resolution images, the image quality can be improved compared to the case where the high-resolution image obtained from the one frame of low-resolution image is not used.

[0096] Other embodiments

[0097] The above-mentioned embodiment has been described in relation to a configuration in which a digital camera performs the capture of a low-resolution image and the generation of a high-resolution image as a processing sequence. However, a high-resolution image can also be generated by using a low-resolution image that has already been recorded. In addition, in the case of using a low-resolution image that has already been recorded, a camera function is not required. Therefore, an electronic device that does not have a camera function can also perform the above-mentioned embodiment. In order to allow the above-mentioned processing for generating a high-resolution image to be performed using a low-resolution image that has already been recorded, it is sufficient to record information indicating the amount and direction of pixel shift (information related to the viewpoint) in association with each of the multiple frames of low-resolution images that have already been captured. In this way, the first resolution increase can be appropriately performed. The second resolution increase can be performed for, for example, a low-resolution image with the earliest recording time or a low-resolution image with a pixel shift amount of 0.

[0098] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0099] Although the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An image processing device, comprising: An obtaining component, used for obtaining a plurality of frames of first images, wherein the viewpoints of the first images are different from each other; A generating means for generating a second image having a higher resolution than the first image by combining the plurality of first image frames, and generating a third image having the same resolution as the second image from a first image frame among the plurality of first image frames; as well as a detection component, configured to detect a changed region between the plurality of frames of first images, The generating component further generates a fourth image as a high-resolution image based on the multiple frames of first images from the second image and an area in the third image corresponding to an area in the second image corresponding to the changed area.

2. The image processing apparatus according to claim 1, wherein: The detection section detects the changed region based on a difference between the second image and the third image.

3. The image processing apparatus according to claim 1, wherein: The detection section detects the change region based on a difference between the plurality of frames of first images.

4. The image processing apparatus according to claim 3, wherein: The detecting component enlarges each of the plurality of first image frames to a resolution same as that of the second image, and uses the enlarged plurality of first image frames when detecting the changed region.

5. The image processing apparatus according to any one of claims 1 to 4, wherein: The generating unit generates the second image by combining the plurality of frames of the first image to increase pixel density.

6. The image processing apparatus according to any one of claims 1 to 4, wherein: The generating component generates the third image from a first frame of the multiple first images using a machine learning model that has been trained to increase the resolution of an input image and output the resulting image.

7. The image processing apparatus according to any one of claims 1 to 4, wherein: The generating means generates the second image by combining the plurality of frames of first images using the information related to the viewpoint which has been recorded in association with the plurality of frames of first images, respectively.

8. The image processing apparatus according to any one of claims 1 to 4, wherein: The detecting means generates a change region map for the plurality of frames of first images, the change region map having different values ​​between the change region and the unchanged region, and The generating unit generates the fourth image from the second image and the third image based on the change region map.

9. The image processing apparatus according to any one of claims 1 to 4, wherein: The generating means generates the fourth image by replacing the region corresponding to the changed region in the second image with the region corresponding to the changed region in the third image.

10. An image processing method, which is performed by an image processing device, the image processing method comprising: Obtaining a plurality of frames of first images, wherein viewpoints of the first images are different from each other; Generating a second image frame having a higher resolution than the first image frame by combining the plurality of first image frames; generating a third image having the same resolution as the second image from a first image frame among the plurality of first image frames; Detecting a changed region that has changed between the plurality of frames of first images; A fourth image is generated as a high-resolution image based on the plurality of frames of first images from the second image and an area in the third image corresponding to an area in the second image corresponding to the changed area.

11. A camera device, comprising: Image sensor; as well as An image processing apparatus according to any one of claims 1 to 9. 12 . The imaging device according to claim 11 , further comprising a control unit configured to control a position of the image sensor so that the position of the image sensor differs between respective shooting times of the plurality of frames of first images.

13. The imaging device according to claim 11, wherein: The imaging device records the fourth image generated by the image processing device. 14 . A computer-readable storage medium storing a program for causing a computer to execute the image processing method according to claim 10 . 15 . A computer program product comprising a program for causing a computer to execute the image processing method according to claim 10 .

Images