Image capturing apparatus for generating wide-angle image using multiple images, control method thereof, and storage medium

Abstract

The invention discloses an image capturing apparatus that generates a wide-angle image using a plurality of images, a method of controlling the same, and a storage medium. The image capturing apparatus includes: an image sensor configured to capture a plurality of RAW images for synthesizing a panoramic image; a setting unit configured to set an image capturing condition for capturing the plurality of RAW images; a determination unit configured to determine a region within each of a plurality of RAW images for generating the composite panoramic image; and a synthesizing unit configured to synthesize a plurality of the regions to generate the synthesized panoramic image, in which the determining unit determines a width of the regions based on the image capturing condition, and in which the setting unit changes the image capturing condition in a case where the width does not satisfy a predetermined condition, and in a case where the width does not satisfy the predetermined condition, the synthesizing unit generates the synthesized panoramic image. Or the determination unit changes the width.

Description

Technical Field

[0001] This disclosure relates to an image capturing apparatus, and more particularly to an image capturing apparatus that extracts a portion of each of a plurality of images and synthesizes the extracted portion of the image. Background Technology

[0002] A method is already in use in which still images are captured continuously while the image capturing device is panning, the central portion of the continuously captured images is extracted in strips, and a portion of the extracted images is synthesized to generate an image with a wide field of view (hereinafter referred to as a panoramic image).

[0003] Japanese Patent Application Publication No. 2021-185689 describes a method for generating panoramic images by mapping captured images onto a virtual cylinder and then combining them. This process is used to reduce differences at coupled portions of the images when combining multiple images captured in a panning shot.

[0004] When mapping captured images onto a virtual cylinder, image quality degradation may occur depending on the image format. For example, it is well known that in the case of Bayer arrays, which are commonly found in RAW format image data read from the imaging device, resolution degradation and increased false colors may occur due to geometric distortion. Therefore, when using RAW format image data for panoramic composition with geometric distortion, it is necessary to suppress image quality degradation. Summary of the Invention

[0005] This disclosure relates to an image capturing apparatus and control method that suppresses image quality degradation when capturing RAW images in panoramic synthesis, while also taking into account the photographer's usability.

[0006] According to one aspect of this disclosure, an image capturing apparatus includes: an image sensor configured to capture a plurality of RAW images; at least one processor; and at least one memory connected to the at least one processor, the memory storing instructions that, when executed by the at least one processor, cause the at least one processor to function as: a setting unit configured to set image capturing conditions for capturing the plurality of RAW images; a determining unit configured to determine a region within each of the plurality of RAW images for generating a composite RAW image; and a combining unit configured to combine the plurality of said regions to generate the composite RAW image, wherein the determining unit determines the width of the region based on the image capturing conditions, and wherein, if the width does not meet predetermined conditions, the setting unit changes the image capturing conditions, or the determining unit changes the width.

[0007] The features of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. The following description of the embodiments is by way of example. Attached Figure Description

[0008] Figure 1 This is a block diagram illustrating the hardware configuration of a digital camera according to an embodiment of the present disclosure.

[0009] Figure 2 This is a flowchart illustrating panoramic image capture according to an embodiment of the present disclosure.

[0010] Figure 3 This is a flowchart illustrating a method for determining various conditions during panoramic image capture according to an embodiment of the present disclosure.

[0011] Figure 4 This is a graph illustrating the synthesis ratio in panoramic synthesis according to embodiments of the present disclosure. Detailed Implementation

[0012] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings. In the drawings, the same parts or elements are indicated by the same reference numerals, and repeated descriptions are omitted or simplified. Furthermore, the following description illustrates an example using a digital (still) camera as an image capturing device. However, image capturing devices may include electronic devices such as movie cameras, smartphones with cameras, tablets with cameras, vehicle cameras, and webcams.

[0013] (First Embodiment)

[0014] Figure 1 This is a block diagram illustrating an example configuration of a digital camera used as an image capturing device according to the first embodiment. The digital camera can continuously capture still images and store data about the captured images. In this embodiment, the description is given based on the premise that the continuously captured still images are in RAW format.

[0015] Control unit 101 is a processor such as a central processing unit (CPU) or microprocessor unit (MPU), and controls each unit of the digital camera by reading and executing various programs stored in read-only memory (ROM) 105, as described below. For example, control unit 101 issues image capture start command and image capture end command to image capturing unit 103, as described below. Alternatively, control unit 101 issues image processing commands to image processing unit 107, as described below, based on programs stored in ROM 105. Commands from the user are input to the digital camera via operation unit 110, as described below, and reach each unit of the digital camera via control unit 101.

[0016] The optical system 102 includes lenses, apertures, drive units that drive these optical components, etc., and forms an image of light from the subject on the image capturing unit 103. The image capturing unit 103 includes an image capturing element such as a charge-coupled device (CCD) image sensor or a complementary metal-oxide-semiconductor (CMOS) image sensor. The image capturing element performs photoelectric conversion on the optical image formed by the optical system 102 to form an analog image signal, thereby generating a captured image. The analog image signal is converted from analog to digital (A / D) data by the A / D conversion unit of the image capturing unit 103. The converted digital image data is stored in the random access memory (RAM) 106 by the image processing unit 107.

[0017] The detection unit 104 includes a gyroscope sensor and an accelerometer sensor, and acquires information such as angular velocity, acceleration, and attitude of the digital camera. The detection unit 104 can also detect vibrations such as camera shake or sway.

[0018] ROM 105 is a read-only non-volatile memory used as a recording medium, and stores the operating programs of the units included in the digital camera, as well as the parameters required for unit operation.

[0019] RAM 106 is a rewritable volatile memory and serves as a temporary storage area for data output during the operation of units included in a digital camera. RAM 106 has a storage capacity sufficient to store a predetermined number of still images, panoramic images (wide-angle images), and other images. RAM 106 can also be used as a working area for programs, etc., read from ROM 105 by the control unit 101.

[0020] The image processing unit 107 can perform various image processing or compositing processes on image data stored in RAM 106 or recorded in storage medium 109, such as white balance adjustment, color interpolation, reduction / enlargement, or filtering. Furthermore, the image processing unit 107 performs compression and development processing on image data captured by the image capturing unit 103 according to standards such as Joint Image Experts Group (JPEG).

[0021] Image processing unit 107 can perform the panoramic compositing process described below to synthesize multiple images to generate a panoramic image. Furthermore, image processing unit 107 includes a circuit that performs a specific image compositing process on the result of the panoramic compositing process. Instead of including image processing unit 107, the panoramic compositing function can be implemented through software processing of control unit 101. For example, one or more or all of the functions equivalent to those of image processing unit 107 can be executed based on a program read from ROM 105 by control unit 101.

[0022] The display unit 108 includes a display device such as a liquid crystal display (LCD) and performs tasks such as through-image display of captured images obtained by the image capturing unit 103. Furthermore, the display unit 108 displays image data temporarily stored in RAM 106, image data stored in storage medium 109, and a so-called graphical user interface (such as a menu).

[0023] The operation unit 110 includes, for example, buttons, switches, keys, and mode dials for a digital camera, as well as a touch panel that also serves as a display unit 108. Commands from the user are transmitted to the control unit 101 via the operation unit 110.

[0024] According to this embodiment, a RAW image is digital image data of an image captured by the image capturing unit 103. This digital image data is stored as an array without development processing or alteration of the color components of the digital image data. Therefore, RAW images cannot be directly displayed on the display unit 108. However, because RAW images are not compressed, the user can select optional processing and develop the RAW images later.

[0025] Figure 2 This is a flowchart illustrating panoramic image capture according to this embodiment. When the user sets the digital camera to panoramic image capture mode, the process begins... Figure 2 The processing is as shown. For example, the mode can be set by operating the mode setting dial corresponding to the operation unit 110, or by operating the menu displayed on the touch panel corresponding to the operation unit 110, which also serves as the display unit 108. In addition, a mode setting button corresponding to the operation unit 110 can be provided on the digital camera, and the panoramic image shooting mode can be set in response to the operation of the mode setting button.

[0026] As described above, this embodiment is based on the premise that RAW format Bayer array images (RAW images) are acquired as input for panoramic synthesis in panoramic image shooting, and a RAW format panoramic image (synthesized RAW image) is generated from multiple RAW images. The data format of each image to be recorded can be selected by the user in the mode settings, and the user can choose, for example, to record the data of RAW images together with the data of JPEG format images.

[0027] In step S201, the control unit 101 determines the conditions for panoramic image capture. The details of the method for determining the conditions are described below.

[0028] In step S202, the digital camera performs panoramic image capture. For example, image capture begins when the user issues an image capture command to the digital camera via the operation unit 110. The method of instructing the start of image capture is not limited to this example. When the user pans the digital camera, the control unit 101 causes the image capture unit 103 to perform image capture based on the image capture conditions determined in step S201. The captured image is temporarily recorded in RAM 106 and used later to generate a panoramic image.

[0029] In step S203, the control unit 101 determines whether to continue or end the panoramic image capture. For example, it can use a method that determines whether there is an image capture end command by the operation unit 110, or a method that determines whether the size of the data to be recorded exceeds the upper limit of the data capacity that can be recorded in the RAM 106, but the method is not limited to these methods.

[0030] If the panoramic image capture ends as determined by the control unit 101 (No in step S203), the process proceeds to step S204. If the panoramic image capture continues as determined by the control unit 101 (Yes in step S203), the process proceeds to step S202.

[0031] In step S204, the image processing unit 107 extracts strip-shaped RAW images for panoramic synthesis from the individual RAW images acquired in step S202. The width determined in step S201 as a condition for panoramic image capture is used as the width of the strip-shaped RAW image to be extracted.

[0032] In step S205, the image processing unit 107 performs processing for aligning the striped RAW images extracted in step S204. This aligning process is performed by calculating the positional deviation between the target striped RAW images and performing a transformation.

[0033] The following describes an example of a method for calculating the positional deviation. First, the image processing unit 107 sets up multiple blocks in a reference striped RAW image. Each block is of the same size. Next, in the striped RAW image to be aligned, the image processing unit 107 sets a search range wider than the size of each block in the reference striped RAW image, at the same positions as each block in the reference striped RAW image. Finally, the image processing unit 107 calculates the corresponding point within the search range of the striped RAW image to be aligned where the sum of the absolute differences (SAD) of the brightness of the corresponding blocks in the reference striped RAW image is minimized. Based on the center of each block in the reference striped RAW image and the aforementioned corresponding point, the image processing unit 107 calculates the positional deviation as a vector. In the calculation of the aforementioned corresponding point, in addition to SAD, the sum of squared differences, normalized cross-correlation, etc., can also be used.

[0034] Next, the image processing unit 107 calculates transformation coefficients based on the positional deviation between the reference strip RAW image and the strip RAW image to be aligned. The image processing unit 107 uses, for example, projection transformation coefficients as transformation coefficients. However, the transformation coefficients are not limited to projection transformation coefficients; affine transformation coefficients or simplified transformation coefficients for horizontal / vertical displacement can also be used.

[0035] The image processing unit 107 performs a geometric transformation on the target stripe RAW image by using the calculated transformation coefficients. For example, the transformation can be performed using expression (1).

[0036]

[0037] In expression (1), (x', y') represent the transformed coordinates, and (x, y) represent the original coordinates. Matrix A represents the transformation coefficients.

[0038] In step S206, image processing unit 107 performs panoramic compositing processing of the striped RAW images. To make the seams of the striped RAW images in the alignment process of step S205 inconspicuous, image processing unit 107 uses a method such as... Figure 4 The composite ratio curves shown are composited using a weighted average. The composite method is not limited to this one; different methods, such as additive averaging, can be used. The processing to panoramic composite processing has been described above.

[0039] Subsequently, referring to Figure 3 The flowchart shown describes the details of the method for determining the panoramic image shooting conditions in step S201.

[0040] In step S301, the control unit 101 temporarily sets the image shooting conditions for panoramic image capture. These temporary image shooting conditions include the target synthesis viewpoint in panoramic synthesis and the image shooting frame rate during panoramic image capture. The temporary image shooting conditions also depend on the buffer capacity of RAM 106, the processing performance of the image processing unit 107 used for panoramic synthesis, and the exposure conditions of the digital camera during image capture. The temporary image shooting conditions are preset within each digital camera.

[0041] Under temporarily set image capture conditions, for example, an optional value can be used as the target synthesis viewpoint. Preferably, the target synthesis viewpoint can be set to have a predetermined size or larger. In other words, the target synthesis viewpoint is set to give the panoramic image a predetermined value or larger viewpoint and image size. The processing performance of the image processing unit 107 for panoramic synthesis is determined by the panoramic synthesis algorithm and the engine performance of the control unit 101 and the image processing unit 107. Furthermore, the image capture frame rate can also be predetermined as a value corresponding to exposure conditions (such as shutter speed).

[0042] In step S302, the control unit 101 determines the number of images that can be captured (the number of images that can be captured) before the buffer of RAM 106, prepared for panoramic image capture, becomes full, based on the image capture conditions temporarily set in step S301. The number of images that can be captured can be determined based on the buffer capacity in RAM 106 prepared for panoramic image capture, the capacity required to record one RAW image acquired during panoramic image capture, the image capture frame rate, and the processing performance of the image processing unit 107. More specifically, the buffer capacity is represented by Cbuffer [bytes], the capacity required to record one RAW image is represented by Coneshot [bytes / image], the image capture frame rate is represented by Sshot [images / second], and the processing performance of the image processing unit 107 (the number of images processed per unit time) is represented by Sproc [images / second]. The number of images that can be captured, Nfull [images], can then be determined as follows:

[0043] Nfull=Cbuffer*Sshot / ((Sshot-Sproc)*Coneshot). … (2)

[0044] During panoramic image capture, the file size required to record a single RAW image actually varies. Therefore, for example, the file size required to record a single RAW image can be determined by referring to the file size of the data corresponding to the RAW image acquired by the image capture unit 103 immediately prior to the panoramic image capture start command. Alternatively, the file size required to record a single RAW image can be a preset value, since a rough calculation shows that the number of images that can be captured is sufficient.

[0045] In step S303, the control unit 101 determines the width of the strip extracted from each RAW image acquired during panoramic image capture. The width of the strip to be extracted can be determined based on, for example, the number of images that can be captured and the composite viewpoint. More specifically, when the composite viewpoint is represented by Acompsite[deg], the width (viewpoint) of a strip, Astrip[deg / piece], can be determined as follows:

[0046] Astrip=Acomposite / Nfull. … (3)

[0047] The width (viewpoint) of a strip determined by expression (3) is assumed to correspond to the number of images used for synthesis as the number of images that can be captured, but panoramic image capture may end before the number of images reaches the upper limit of the number of images that can be captured. This value is a rough calculation of the width (viewpoint) of a strip in a temporary setting. In and after step S304, it is determined whether to use this value as is in subsequent panoramic image captures.

[0048] In step S304, the control unit 101 determines the ratio of the width (view of view) of a strip calculated in step S303 to the entire view of a captured RAW image. When the entire view of a RAW image is represented by Afull[deg], the ratio Rstrip[%] of the width (view of view) of a strip to the entire view of a RAW image used to synthesize a panoramic RAW image is determined as follows:

[0049] Rstrip = Astrip / Afull … (4)

[0050] The full angle of view (Afull) of a RAW image is a value that depends on the focal length of the lens of the optical system 102 and the size of the sensor of the image capturing unit 103.

[0051] The full angle of view Afull [deg] of a RAW image can also depend on the settings of the recorded image size, image processing, and the size of the sensor's readout area, rather than the size of the sensor in the image capturing unit 103. For example, the angle of view of a captured RAW image can be adjusted by the image processing unit 107. In this case, the image processing unit 107 extracts the RAW image read from the image capturing element to have a predetermined size, and the angle of view of the extracted RAW image can be defined as Afull [deg].

[0052] In step S305, the control unit 101 determines whether the value of the strip width ratio determined in step S304 is acceptable. More specifically, a threshold for the strip width ratio can be set, and if the strip width ratio is within the threshold, the control unit 101 can determine that the current strip width ratio is acceptable. It is well known that as the strip width ratio increases, geometric transformations of the RAW image in panoramic compositing can lead to image quality degradation. Therefore, an upper limit is set on the strip width ratio used for compositing based on the viewpoint of a RAW image.

[0053] Furthermore, a lower limit can be set on the ratio of the strip width used for synthesis based on the viewpoint of a RAW image. Theoretically, a lower strip width ratio suppresses image quality degradation caused by geometric transformations of the RAW image during panoramic synthesis. However, the relationship between the synthesized viewpoint (viewpoint of the panoramic RAW image) and the width of a strip (viewpoint) is as described in expression (3). A lower strip width ratio increases the number of RAW images required for synthesis. In other words, the number of RAW images is limited by the buffer capacity for temporarily storing the RAW images required for synthesis. Furthermore, as the number of images required for synthesis increases, performing the synthesis process takes more time, leading to reduced usability. Therefore, a lower limit can be set on the strip width ratio.

[0054] Thresholds (upper and lower limits) for the strip width ratio can be determined by, for example, experimentally examining the relationship between the ratio of strip width and the degree of image quality degradation.

[0055] If the control unit 101 determines that the strip width ratio is acceptable ("Yes" in step S305), the process proceeds to step S307. Otherwise ("No" in step S305), the process proceeds to step S306.

[0056] In this embodiment, the width (viewpoint) of a strip is determined based on the ratio of the width of a strip to the entire viewpoint of a RAW image, but it can also be determined based on the width of a strip. As described above, as the ratio of the strip width to the entire viewpoint of the RAW image increases, image quality deteriorates due to geometric transformations of the RAW image during panoramic synthesis. This means that when the strip has a width greater than or equal to a predetermined size, image quality deteriorates due to geometric transformations of the RAW image. Therefore, in step S305, it can be determined whether the width of the strip is acceptable based on the width of the strip determined in step S303.

[0057] In step S306, the control unit 101 resets the image capturing conditions. For example, in order to further reduce and stabilize the strip width ratio within an allowable range, it is necessary to reduce the width (viewing angle) of one strip (Astrip [deg]). More specifically, the strip width ratio can be reduced by increasing the buffer capacity Cbuffer [bytes] of RAM 106 or by reducing the difference between the image capturing frame rate and the processing speed of the image processing unit 107. After the control unit 101 resets the image capturing conditions in this manner, the process proceeds to step S302.

[0058] In step S307, the control unit 101 calculates the swing speed during panoramic image capture. The swing speed Sswing [deg / second] can be determined as follows:

[0059] Sswing=Acomposite / (Nfull / Sshot). … (5)

[0060] This value corresponds to the panning speed recommended to the user during panoramic image capture.

[0061] In step S308, the control unit 101 determines whether the swing speed calculated in step S307 is within a predetermined range. In panoramic image capture, the user typically performs image capture while panning the digital camera. Therefore, panoramic image capture becomes difficult when the swing speed calculated in step S307 (i.e., the speed required for panning) is too high or too low. Therefore, upper and lower limits are set for the swing speed required during actual panoramic image capture. For example, the upper and lower limits of the swing speed can be determined experimentally and can be set based on the user's experience during actual panoramic image capture.

[0062] The methods for setting upper and lower limits are not limited to this one.

[0063] If the control unit 101 determines that the swing speed calculated in step S307 is within a predetermined range ("Yes" in step S308), the image capture conditions for panoramic image capture are fixed, and the process ends. Conversely, if the control unit 101 determines that the swing speed calculated in step S307 is not within a predetermined range ("No" in step S308), the process proceeds to step S309.

[0064] In step S309, the control unit 101 resets the image capturing conditions. To increase the swing speed Sswing [deg / sec], it is necessary to increase the composite viewing angle Acomposite [deg] or the image capturing frame rate Sshot [frames / sec]. To decrease the swing speed Sswing [deg / sec], it is necessary to decrease the image capturing frame rate Sshot [frames / sec] or increase the buffer capacity Cbuffer [bytes]. After resetting the conditions in this way, the process proceeds to step S302.

[0065] When resetting the image capturing conditions in step S309, the image capturing frame rate Sshot [frames / second] can be increased by limiting the readout area of ​​the image capturing element in the image capturing unit 103. By limiting the readout area of ​​the image capturing element, image data can be generated only from signals in a portion of the region of the image capturing element in the image capturing unit 103. This allows the readout area of ​​the image capturing element in the image capturing unit 103 to approach the size of the strip width required for synthesis. Performing readout from a region smaller than the entire area of ​​the image capturing element (image sensor) in the above manner increases the readout speed and the image capturing frame rate Sshot [frames / second]. Note that this adjustment of the image capturing conditions can be performed in steps S301 and S306 as needed.

[0066] As mentioned above, when shooting multiple RAW images and generating a panoramic RAW image in panoramic imaging, suitable image shooting conditions can be presented. This allows for the suppression of image quality degradation when synthesizing the acquired RAW images in panoramic imaging, and ensures that panoramic imaging is performed with the photographer's availability in mind.

[0067] (Other embodiments)

[0068] This disclosure can be implemented by providing a program that implements one or more functions of the above embodiments to a system or device via a network or recording medium, and causing one or more processors in the computer of the system or device to read and execute the program. Alternatively, this disclosure can be implemented by circuitry (e.g., an application-specific integrated circuit (ASIC)) that implements one or more functions. Furthermore, functions can be categorized into functions executed by the processor reading the program and functions executed by the circuitry, and these functions can be combined.

[0069] This disclosure is not limited to the above embodiments, and various modifications and changes can be made within the scope of this disclosure.

[0070] While this disclosure has been described with reference to embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. The scope of the following claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An image capturing device, comprising: An image sensor is configured to capture multiple RAW images; The setting unit is configured to set image shooting conditions for capturing the plurality of RAW images; The determining unit is configured to determine regions within each RAW image used to generate the composite RAW image; as well as The compositing unit is configured to synthesize multiple said regions to generate the synthesized RAW image. The determining unit determines the width of the region based on the image capture conditions, and If the width does not meet the predetermined conditions, the setting unit changes the image capturing conditions, or the determining unit changes the width.

2. The image capturing device according to claim 1, wherein, The predetermined condition is that the width is less than a first threshold.

3. The image capturing device according to claim 1, wherein, The predetermined condition is that the ratio of the width to the viewpoint corresponding to one of the plurality of RAW images is less than a first threshold.

4. The image capturing device according to claim 1, wherein, The setting unit sets the image capturing conditions based on the processing speed of the synthesis unit.

5. The image capturing device according to claim 1, wherein, The synthesis unit synthesizes multiple regions by performing geometric transformations on the multiple regions.

6. The image capturing device according to claim 1, wherein, The multiple RAW images captured by the image sensor are images of the Bayer array.

7. The image capturing device according to claim 1, wherein, The synthesized RAW image has a wider field of view than each of the plurality of RAW images.

8. A method for controlling an image capturing device, the method comprising: Set the image shooting conditions for capturing multiple RAW images; Capture the multiple RAW images; Identify the region within each of the multiple RAW images used to generate the composite RAW image; as well as Multiple regions are synthesized to generate the synthesized RAW image. The width of the region is determined based on the image capture conditions, and If the width does not meet the predetermined conditions, the image shooting conditions or the width are changed.

9. A non-transitory computer-readable storage medium storing a program, wherein executing the program on a computer causes the computer to perform a method for controlling an image capturing device, the method comprising: Set the image shooting conditions for capturing multiple RAW images; Capture the multiple RAW images; Identify the region within each of the multiple RAW images used to generate the composite RAW image; as well as Multiple regions are synthesized to generate the synthesized RAW image. The width of the region is determined based on the image capture conditions, and If the width does not meet the predetermined conditions, the image shooting conditions or the width are changed.

Images