Imaging device and control method thereof

Abstract

To improve an image blur correction function in generating an image with an angle of view smaller than that of an omnidirectional image, in an imaging apparatus capable of generating an omnidirectional image by using a plurality of imaging lenses, and a control method of the imaging apparatus.SOLUTION: The imaging apparatus is capable of generating an omnidirectional image by using a plurality of imaging lenses. In an operation mode of generating an image with an angle of view smaller than that of the omnidirectional image, the imaging apparatus identifies an imaging range required to generate the image, on the basis of the difference between a reference posture and a present posture. Then, the imaging apparatus validates the imaging lenses required to image the imaging range and invalidates the imaging lenses not required to image the imaging range, among the plurality of imaging lenses.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an imaging device and a control method thereof, and particularly to an imaging device capable of capturing an omnidirectional image and a control method thereof.

Background Art

[0002] An imaging device capable of capturing an omnidirectional image with an angle of view of 360 degrees is known (Patent Document 1). Such an imaging device is also called a 360-degree camera or an omnidirectional camera.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, some 360-degree cameras equipped with two circular fisheye lenses with an angle of view of 180 degrees have a function of cutting out a rectangular area from an image captured using only one lens to generate an image as if it were captured with a wide-angle lens with an angle of view of less than 180 degrees.

[0005] In this case, for the rotation of the camera around the optical axis (movement in the roll direction), electronic image blur correction is possible regardless of the rotation angle. However, for the rotation of the camera around the axis orthogonal to the optical axis (movement in the pitch and yaw directions), the rotation angle for which image blur correction is possible is limited.

[0006] In one embodiment of the present invention, in an imaging device capable of generating an omnidirectional image using a plurality of lenses and a control method thereof, the image blur correction function when generating an image with an angle of view smaller than the omnidirectional image is improved.

Means for Solving the Problems

[0007] The above objective is an imaging device capable of generating omnidirectional images using multiple imaging lenses, comprising detection means for detecting the orientation of the imaging device, and, when the imaging device is in an operating mode that generates an image with a smaller field of view than an omnidirectional image, based on the difference between the reference orientation and the current orientation of the imaging device. ,before A means for identifying the shooting range necessary for image generation, which is not affected by changes in the orientation of the imaging device; and a control means for activating the shooting lens necessary for capturing the shooting range and disabling the shooting lens unnecessary for capturing the shooting range when the imaging device is in operation mode. The system includes a generation means that generates an image by cropping a region of the shooting range from an image obtained using an activated shooting lens among multiple shooting lenses, and a synthesis means that, if there are multiple activated shooting lenses, stitches together the images taken using each of the activated shooting lenses to generate a composite image. When there are multiple activated shooting lenses, the generation means crops a region of the shooting range from the composite image and generates an image by superimposing a predetermined guide display, the guide display includes a display that shows how to change the orientation of the imaging device to reduce the number of activated lenses. This is achieved by an imaging device characterized by the following features. [Effects of the Invention]

[0008] According to the present invention, in an imaging device capable of generating an omnidirectional image using multiple lenses and a control method thereof, the image blur correction function when generating an image with a smaller field of view than the omnidirectional image can be improved. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram relating to a digital camera as an example of an imaging device according to the embodiment. [Figure 2] A diagram relating to a smartphone as an example of an electronic device according to the embodiment. [Figure 3] Schematic diagram illustrating the operation control in the embodiment. [Figure 4] Flowchart of the operation of the digital camera in the embodiment [Figure 5] Flowchart of the operation of the digital camera in the embodiment [Figure 6] A diagram showing an example of a guide display in an embodiment. [Figure 7] Flowchart of the operation of the digital camera in the embodiment [Modes for carrying out the invention]

[0010] The present invention will be described in detail below with reference to the attached drawings, based on exemplary embodiments thereof. Note that the following embodiments do not limit the invention to the claims. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the attached drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0011] In the following embodiments, the present invention will be described in relation to cases where it is implemented using an imaging device such as a digital camera. However, the present invention can be implemented with any electronic device having an imaging function. Such electronic devices include video cameras, computer equipment (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and dashcams. These are examples, and the present invention can be implemented with other electronic devices as well.

[0012] Figures 1(a) and 1(b) are perspective views showing an example of the external appearance of a digital camera 100 as an example of an imaging device according to an embodiment of the present invention. Here, the side on which the shutter button 61 is provided is considered the front of the digital camera 100. Therefore, Figure 1(a) shows an example of the front side configuration, and Figure 1(b) shows an example of the rear side configuration. Figure 1(c) is a block diagram showing an example of the functional configuration of the digital camera 100.

[0013] Hereinafter, an exemplary configuration of the digital camera 100 will be described using Figures 1(a) to 1(c). The digital camera 100 has photographic lenses 103a and 103b, which are circular fisheye lenses with a field of view of 180 degrees, provided on the front and back, respectively. The photographic lenses 103a and 103b are, for example, on the same optical axis and are designed so that their shooting directions differ by 180 degrees. In this specification, the shooting direction of the photographic lens 103a provided on the front of the digital camera 100 is forward, and the photographic lens 103 provided on the back of the digital camera 100 is forward. b The direction in which the image is taken is called the rear.

[0014] The digital camera 100 is an omnidirectional camera or 360-degree camera that can generate an omnidirectional image with a horizontal viewing angle of 360 degrees by synthesizing or stitching the images formed by the photographing lenses 103a and 103b. Note that the omnidirectional image may be a full-spherical image, or the vertical viewing angle may be less than 360 degrees like a half-spherical image. In this embodiment, it is assumed that the two photographing lenses 103a and 103b are each circular fisheye lenses with a horizontal viewing angle of 180 degrees or more, and the digital camera 100 can generate a full-spherical image.

[0015] Here, the two photographing lenses 103a and 103b cover a horizontal viewing angle of 360 degrees or more, but a configuration in which three or more photographing lenses with different photographing directions cover a horizontal viewing angle of 360 degrees or more may also be used.

[0016] The digital camera 100 has a plurality of imaging units including a front camera 110 that photographs the front and a rear camera 120 that photographs the rear. The front camera 110 has a photographing lens 103a and a barrier 102a (protective member) provided on the front surface of the photographing lens 103a. The barrier 102a is, for example, a transparent hemispherical member. The photographing lens 103a has a focus lens that is movable in the optical axis direction. The shutter 101a also serves as an aperture. The photographing lens 103a forms an optical image on the imaging surface of the imaging element 22a. <​​​​​In addition, the front camera 110 has a mechanism for moving the focus lens, a mechanism for detecting the position of the focus lens, and a mechanism for driving the shutter 101a. Note that the front camera 110 may have a mechanism for moving the imaging element 22a and / or a mechanism for moving the shift lens included in the photographing lens 103a to realize an optical blur correction function.

[0019] Since the rear camera 120 has the same configuration as the front camera 110, descriptions of individual elements are omitted. In the following description, the imaging elements 22a and 22b are collectively referred to as the imaging element 22. The same applies to other components shared by both the front camera 110 and the rear camera 120. In the present embodiment, since one imaging unit uses one photographing lens, selectively using the imaging unit is synonymous with selectively using the photographing lens. Therefore, in the following description, descriptions regarding activation and deactivation of the imaging unit (camera) can be read as activation and deactivation of the photographing lens. However, the present invention can also be implemented in a configuration where one imaging unit uses a plurality of photographing lenses simultaneously or by switching.

[0020] The image data output by the A / D converter 23 is written into the memory 32 via both the image processing unit 24 and the memory control unit 15, or via only the memory control unit 15.

[0021] The memory 32 is used as a buffer for image data, a working memory for the image processing unit 24, a video memory for the display unit 28, and the like.

[0022] The image processing unit 24 applies predetermined image processing to image data output by the A / D converter 23 or the memory control unit 15, or stored in the memory 32, to acquire and / or generate signals, image data, and various types of information according to the application. The image processing unit 24 may be a dedicated hardware circuit, such as an ASIC (Application Specific Integrated Circuit) designed to implement a specific function. Alternatively, the image processing unit 24 may be configured such that a processor, such as a DSP (Digital Signal Processor) or GPU (Graphics Processing Unit), executes software to implement a specific function.

[0023] Image processing that the image processing unit 24 can apply to image data includes, for example, preprocessing, color interpolation, correction, detection, data processing, evaluation value calculation, and special effects processing. Preprocessing may include signal amplification, reference level adjustment, and defective pixel correction. Color interpolation is performed when a color filter is provided on the image sensor 22, and it is a process that interpolates the values ​​of color components that are not included in the individual pixel data that make up the image data. Color interpolation is also called demosaicing. Correction processing may include white balance adjustment, tone correction, correction of image degradation caused by optical aberrations of the shooting lens 103 (image recovery), and color correction. Correction processing also includes correcting peripheral distortion, which is noticeable with ultra-wide-angle lenses such as circular fisheye lenses. Detection processes may include detecting feature regions (such as face regions or human body regions) and their movements, as well as recognizing people. Data processing may include processes such as region extraction (trimming), merging, scaling, encoding and decoding, and header information generation (data file generation). The generation of display image data and recording image data is also included in data processing. The evaluation value calculation process may include processes such as generating signals and evaluation values ​​used for autofocus detection (AF), and generating evaluation values ​​used for automatic exposure control (AE). Special effects processing may include adding blur effects, changing color tones, and relighting. These are merely examples of processing that the image processing unit 24 can apply, and do not limit the processing that the image processing unit 24 can apply. Furthermore, the image processing that the image processing unit 24 can apply may also be performed by the system control unit 50.

[0024] In this embodiment, the image processing unit 24 stitches together two circular fisheye images captured by the front camera 110 and the rear camera 120 to generate a 360-degree spherical image. The generated 360-degree spherical image is, for example, an image using equirectangular projection, and the position of each pixel can be associated with the coordinates of the 360-degree spherical surface.

[0025] Furthermore, the image processing unit 24 applies region cropping, distortion correction, and other processes to the circular fisheye image data to generate rectangular live view display image data suitable for the display unit 28 or an external display device.

[0026] The system control unit 50 is, for example, a program-executable processor (CPU, MPU, microprocessor, etc.). The system control unit 50 controls the operation of each functional block of the digital camera 100 by reading programs stored in the non-volatile memory 56 into the system memory 52 and executing them, thereby realizing the functions of the digital camera 100.

[0027] The non-volatile memory 56 is electrically rewritable and stores programs executed by the system control unit 50, various settings for the digital camera 100, GUI data, etc. The system memory 52 is the main memory used by the system control unit 50 when executing programs. Note that memory 32 and system memory 52 may be separate areas within a contiguous memory space.

[0028] The shutter button 61 has a switch SW1 62 that turns on when half-pressed and a switch SW2 64 that turns on when fully pressed. The system control unit 50 recognizes the ON state of SW1 62 as an instruction to prepare for still image shooting, and the ON state of switch SW2 64 as an instruction to start still image shooting. When the system control unit 50 detects that switch SW1 62 is ON, it uses signals and evaluation values ​​generated by the image processing unit 24 to perform autofocus detection (AF) and automatic exposure control (AE) of the front camera 110 and / or rear camera 120. Also, when the system control unit 50 detects that switch SW2 64 is ON, it controls the shutter 101 according to the exposure conditions determined by the AE processing and performs still image shooting and recording processing. The still image data for recording generated by the image processing unit 24 is first stored in the memory 32, and then recorded to the recording medium 90 via the I / F 18 by the system control unit 50.

[0029] The shutter button 61 may have only one switch. In this case, when the system control unit 50 detects that the switch is turned on, it will perform the shooting preparation operation and the shooting process in succession.

[0030] The user can switch the operating mode of the digital camera 100 by operating the mode switch 60. The operating modes may include, for example, a mode for capturing a 360-degree spherical image, a mode for capturing a general wide-angle image with a horizontal field of view of less than 180 degrees (crop mode), a playback mode, and a mode for operation in cooperation with external devices. The mode for capturing wide-angle images may have multiple modes depending on the horizontal field of view (e.g., 90-degree mode, 150-degree mode).

[0031] Furthermore, the operating mode can also be selected by combining the operation of the mode selector switch 60 with the operation of other operating components. For example, the operating mode can be selected by selecting a broad category of operating mode with the mode selector switch 60 and then selecting a more detailed category displayed on the display unit 28.

[0032] The power switch 72 is a switch used to turn the digital camera 100 on and off. It controls the operation of the power control unit 80.

[0033] The power control unit 80 includes a battery detection circuit, a DC-DC converter, a switch circuit for switching the function block that supplies power, and controls the power supply from the power supply unit 30, which may be a battery or an AC adapter, to the components of the digital camera 100. The power control unit 80 detects the type of power supply unit 30. In addition, if the power supply unit 30 is a battery, the power control unit 80 detects the type and remaining charge. The power control unit 80 can change the components to which power is supplied and the power supplied, according to the state of the power switch 72 and the control of the system control unit 50.

[0034] The microphone 20 is directed outwards from the digital camera 100 and outputs audio signals to the system control unit 50. When recording video, image data and audio data are recorded.

[0035] The control unit 70 is a general term for all input devices (buttons, switches, dials, etc.) other than the mode switching switch 60, shutter button 61, and power switch 72. This includes the video recording switch, menu button, directional keys, and select key. If the display unit 28 is a touch display, touch-operable software buttons or keys also constitute the control unit 70.

[0036] The display unit 28 is, for example, a liquid crystal display (LCD). The display unit 28 may be a touch display. The display unit 28 displays text and images. By immediately displaying continuously recorded video on the display unit 28, the display unit 28 can function as an electronic viewfinder (EVF). The video displayed to enable the display device to function as an EVF is called a live view image. Furthermore, by outputting the live view image to an external device connected to the communication unit 54, the display device of the external device can also function as an EVF.

[0037] The light-emitting unit 21 is a light-emitting diode (LED) or the like, and notifies the user of the status of the digital camera 100 through the pattern and color of the light emission.

[0038] The fixing part 40 (Figure 1(b)) provided on the bottom of the digital camera 100 is, for example, a screw hole for attaching a tripod.

[0039] The system timer 53 outputs the time from the built-in clock or measures time in response to a request from the system control unit 50.

[0040] The communication unit 54 is an interface for wired and / or wireless communication with electronic devices and external devices such as external display devices, which will be described later. The communication unit 54 conforms to one or more wired and / or wireless communication standards and has connectors, transceivers, etc. according to the standards. Typical standards that the communication unit 54 may conform to include, but is not limited to, USB, HDMI®, Bluetooth®, and wireless LAN.

[0041] The attitude detection unit 55 includes, for example, a gyro sensor and an acceleration sensor, and outputs signals representing the attitude and movement of the digital camera 100 to the system control unit 50. The attitude of the digital camera 100 is represented by the rotation angles (roll, pitch, yaw) around the angular axes, with respect to the x-axis parallel to the optical axis, the y-axis extending horizontally, and the z-axis extending vertically. The attitude of the digital camera 100 at the time of shooting may be recorded in association with the image data. Furthermore, the detected attitude and movement of the digital camera 100 can also be used for image blur correction and tilt correction.

[0042] I / F18 is an interface for writing data to a recording medium 90, such as a memory card or hard disk, and reading data recorded on the recording medium 90. The recording medium 90 may or may not be detachable from the digital camera 100.

[0043] Figure 2 shows a smartphone 200 as an example of an external device that can interact with the digital camera 100 via communication through the communication unit 54. Figure 2(a) is a perspective view showing an example of the external appearance, and Figure 2(b) is a block diagram showing an example of the functional configuration. Note that the external device is not limited to a smartphone; any electronic device that can communicate with the digital camera 100 and has a processor capable of executing applications for interacting with the digital camera 100 is acceptable.

[0044] This section describes the configuration of the Smartphone 200. The internal bus 250 connects each block to enable bidirectional data exchange.

[0045] CPU 201 is a program-executable processor that loads programs stored in non-volatile memory 203 into memory 202 and executes them, thereby controlling the operation of each functional block of the smartphone 200 and realizing the functions of the smartphone 200.

[0046] Memory 202 is the main memory used by the CPU 201 when executing programs. A portion of memory 202 is used as video memory for the display 205. The non-volatile memory 203 is electrically rewritable and stores programs executed by the CPU 201 (OS and applications), smartphone 200 settings, GUI data, user data, and the like.

[0047] Display 205 is, for example, an LCD, and the OS and applications display images and various information on it. Display 205 is a touch display having a touch panel 206a, and is capable of detecting touch operations on the display surface of display 205. Display 205 may also be an external device.

[0048] The image processing unit 24 applies image processing to image data stored in the non-volatile memory 203 and recording medium 208, image data acquired via the external I / F 209, and image data acquired via the communication I / F 210, based on the control of the CPU 201.

[0049] The image processing that the image processing unit 24 can apply to the image data may be the same as that of the image processing unit 24 of the digital camera 100. If the smartphone 200 does not have a camera, it is not necessary to generate evaluation values ​​used for AF and AE. In addition, the image processing that the image processing unit 24 can apply may be performed by the CPU 201.

[0050] The image processing unit 24, under the control of the CPU 201, can generate image data for VR (Virtual Reality) display that corresponds to the movement of the smartphone 200 from ultra-wide-angle images such as omnidirectional images (for example, images with a horizontal field of view exceeding 180 degrees). VR display is achieved by displaying image data generated by extracting a region of the shooting range that follows the change in posture of the smartphone 200 from the ultra-wide-angle image on the display 205. With VR display, for example, if the display 205 of the smartphone 200 is fixed in front of the user's eyes using goggles, the displayed image changes in accordance with the movement of the user's head. As a result, the user can experience the sensation of being in a virtual space represented by the ultra-wide-angle image.

[0051] Conversely, by controlling the image processing unit 24 to extract the same shooting range from the ultra-wide-angle image regardless of the orientation of the smartphone 200, electronic image blur correction can be achieved.

[0052] The operation unit 206 is a collective term for a group of input devices that a user can use to give instructions to the smartphone 200. Generally, the input devices of a smartphone 200 include buttons, switches, and touch panels, but are not limited to these. Keyboards and mice that are connected to the smartphone 200 in a communicative manner can also constitute the operation unit 206. Note that although the touch panel 206a is shown separately from the display 205 in Figure 2(b), it is actually built into or attached to the display screen of the display 205.

[0053] The power button 206b, volume buttons 206c and 206d, and home button 206e are examples of input devices that make up the control unit 206. The power button 206b switches the smartphone 200 on and off. The volume buttons 206c and 206d are buttons that increase and decrease the volume output from the audio output unit 212. The home button 206e is a button that displays a specific screen provided by the OS on the display 205.

[0054] The media interface 207 is an interface for accessing the recording medium 208. If the recording medium 208 is a removable medium such as a memory card, the media interface 207 has a slot into which the recording medium 208 can be inserted. The CPU 201 can write data to the recording medium 208 and read data from the recording medium 208 through the media interface 207.

[0055] External I / F209 is an interface for wired and / or wireless communication with external devices such as the digital camera 100 and external display devices. External I / F209 conforms to one or more wired and / or wireless communication standards and has connectors, transceivers, etc. according to the standards. Typical standards that the communication unit 54 may conform to include, but are not limited to, USB, HDMI®, Bluetooth®, and wireless LAN.

[0056] Communication I / F 210 is an interface for communication via the mobile phone network 211. Communication I / F 210 may be a communication interface compliant with mobile communication standards established by 3GPP, such as a 3G, 4G, or 5G modem.

[0057] The audio output unit 212 outputs audio (sounds based on video or music data, operation sounds, ringtones, various notification sounds, etc.). The audio output unit 212 includes an audio output terminal 212a for connecting earphones, etc., and a speaker 212b, but audio may also be output to an external device via an external interface.

[0058] The attitude detection unit 213 includes, for example, a gyroscope and an accelerometer, and outputs signals representing the attitude and movement of the smartphone 200 to the CPU 201. The attitude of the smartphone 200 is represented by rotation angles (roll, pitch, yaw) around the angular axes, with respect to the x-axis perpendicular to the display screen of the display 205, the y-axis extending horizontally, and the z-axis extending vertically. The attitude detected by the attitude detection unit 213 can be used, for example, in the VR display described above.

[0059] Figure 3 schematically illustrates the operation control of the front camera 110 and rear camera 120 according to the orientation when the digital camera 100 is operating in a mode (crop mode) that captures still images with a horizontal field of view of less than 180 degrees.

[0060] Figure 3(a) shows the orientation of the digital camera 100 at the start of shooting (reference orientation), and Figure 3(b) shows an example of the orientation of the digital camera 100 during shooting. Figure 3(a) shows the state where roll, pitch, and yaw are all 0 degrees (upright position). Figure 3(b) shows the state in Figure 3(a) where only the pitch (rotation angle around the y-axis) has changed so that the rear camera 120 is facing downwards.

[0061] In crop mode, an image for recording or display is generated by cropping a portion of an omnidirectional or ultra-wide-angle image. If the horizontal field of view of the image generated in crop mode can be covered by a certain shooting lens, there is no need to shoot with other shooting lenses. Therefore, only the shooting lenses and associated circuits necessary for shooting can be used.

[0062] For example, if the horizontal field of view of the image generated by the digital camera 100 of this embodiment in crop mode is less than 180 degrees, power consumption can be reduced by enabling only one of the front camera 110 and the rear camera 120. Here, shooting is started using the rear camera 120 so that the display unit 28 can be viewed at the start of shooting. Also, an image with a horizontal field of view of 150 degrees and a vertical field of view of 90 degrees is generated.

[0063] Furthermore, during shooting, electronic image blur correction is performed by changing the image cropping position so that the same shooting range as at the start of shooting is maintained even if the posture of the digital camera 100 changes from the reference posture (for example, the posture at the start of shooting). In the following, for convenience, the range cropped in crop mode from the shooting range will be referred to as the recording range. Note that the use of image data generated in crop mode is not limited to recording.

[0064] At the start of shooting, the orientation (reference orientation) of the digital camera 100 is upright. In this case, arrow 306 indicates the shooting direction (the optical axis direction of the shooting lens 103). Therefore, of the shooting range 305 (right semicircle) of the rear camera 120, the gray-colored 307 represents the vertical recording range. Since the front camera 110 is disabled, the shooting range 303 (left semicircle) of the front camera 110 is not captured. The direction of the boundary line 300 between the shooting range 303 of the front camera 110 and the shooting range 305 of the rear camera 120 coincides with the direction of gravity 308.

[0065] When the digital camera 100 changes position from the orientation shown in Figure 3(a) to the orientation shown in Figure 3(b), in order to achieve image blur correction, it is necessary to crop the same range as the recording range 307 at the start of shooting. However, the hatched portion of the recording range 307 in Figure 3(b) is not included in the shooting range 305 of the rear camera 120, but is included in the shooting range 303 of the front camera 110. Therefore, image blur correction cannot be achieved using only the image data obtained from the rear camera 120.

[0066] Thus, if image blur correction cannot be achieved with the shooting range of a single camera, the system control unit 50 enables another camera (in this case, the front camera 110) whose shooting range includes the range necessary for image blur correction, and performs shooting with multiple cameras. As a result, even if the digital camera 100 changes its posture to that shown in Figure 3(b), it becomes possible to extract the recording range 307, thereby achieving image blur correction.

[0067] On the other hand, if image stabilization can be achieved within the shooting range of one camera, power consumption can be reduced by disabling other cameras. Furthermore, when the recording range approaches the boundary of the current shooting range, cameras capable of shooting the range required if the recording range exceeds the shooting range may be enabled. This avoids the situation where image stabilization becomes unavailable during the period between the activation of the additionally enabled camera and the acquisition of an image.

[0068] While Figure 3 illustrates control in response to changes in attitude in the pitch direction, the same principle applies to changes in attitude in the yaw direction.

[0069] The operation of the system control unit 50 in crop mode will be further explained using the flowchart in Figure 4. This operation is performed when the digital camera 100 is operating in crop mode, image stabilization is enabled, and video recording has started. Note that video recording may be for recording purposes or for live view display purposes, but we will explain recording purposes. In the case of recording purposes, S400 may be executed by detecting the operation of the video recording switch in standby mode, or in the case of live view display purposes, by detecting the operation of enabling image stabilization in standby mode, but it is not limited to these.

[0070] If remote operation by an external device is enabled, S400 may be executed in response to a recording start instruction received from the external device via the communication unit 54.

[0071] In S400, the system control unit 50 acquires the orientation (reference orientation) of the digital camera 100 from the orientation detection unit 55. Based on the acquired orientation, the system control unit 50 determines the optical axis direction of the front camera 110 or the rear camera 120 as the shooting direction. The shooting direction is a constant direction that is not affected by subsequent changes in the orientation of the digital camera 100. The system control unit 50 stores the acquired orientation and the determined shooting direction in, for example, the system memory 52.

[0072] In S401, the system control unit 50 determines the recording range based on the shooting direction determined in S400 and the settings of the digital camera 100. Here, in crop mode, the horizontal field of view is selectable, and the vertical field of view is assumed to be constant or determined according to the horizontal field of view. Here, a horizontal field of view of 150 degrees is selected, and the vertical field of view is set to 90 degrees.

[0073] The system control unit 50 stores in the system memory 52, in association with the orientation of the digital camera 100 at the start of recording, a range determined by the orientation and shooting direction of the digital camera 100 and the shooting angle of view, as the recording range. For example, the system control unit 50 can define the recording range as a rectangular area centered on the shooting direction and having a size based on the horizontal and vertical angles of view, which is mapped onto the omnidirectional image according to the orientation of the digital camera 100. Therefore, the recording range corresponds to a specific area in the omnidirectional image that the digital camera 100 can generate.

[0074] In S402, the system control unit 50 starts the video recording operation and continuously executes the processing from S403 onwards, frame by frame.

[0075] In S403, the system control unit 50 acquires the attitude of the digital camera 100 from the attitude detection unit 55. In S404, the system control unit 50 determines which camera to use based on the recording range and the change in posture since the start of shooting. Details will be described later.

[0076] In S405, the system control unit 50 performs one frame of image capture using the active camera. Since only the rear camera 120 is active at the start of shooting, the system control unit 50 performs one frame of image capture using only the rear camera 120. The image processing unit 24 generates recording image data and display image data for one frame read from the active camera and stores them in the memory 32. Note that the determination of shooting conditions and the operation of the focus lens and image sensor during shooting are well known and therefore will not be explained.

[0077] In S406, the system control unit 50 determines whether there are multiple active cameras. This determination corresponds to determining whether image synthesis is necessary. Since the digital camera 100 in this embodiment has a front camera 110 and a rear camera 120, the system control unit 50 executes S407 if both cameras are active, or skips S407 and executes S408 if only one camera is active.

[0078] In S407, the system control unit 50 instructs the image processing unit 24 to stitch together the image data captured by the active camera. In response to the instruction, the image processing unit 24 stitches together the image data of the current frame stored in the memory 32 to generate data for a composite image with a continuous shooting range. It is assumed that the positional relationship in which the images obtained from each camera are stitched together is known to the image processing unit 24. The image processing unit 24 then generates the data for the composite image. memory It is stored in 32. When the composite image data is generated, the system control unit 50 executes S408.

[0079] In S408, the system control unit 50 instructs the image processing unit 24 to crop the image data within the recording range. The system control unit 50 reads, for example, the orientation at the start of shooting acquired in S400, the orientation acquired in S403, and the recording range determined in S401 from the system memory 52 and provides them to the image processing unit 24 along with the cropping instruction. Alternatively, instead of providing orientation information, the image processing unit 24 may be provided with the position of the recording range corresponding to the shooting direction at the start of shooting, which was determined when determining the camera to be used in S404.

[0080] The image processing unit 24 calculates the position of the recording range in the image obtained at the current orientation based on the change in the orientation of the digital camera 100 from the start of shooting to the present. Then, the image processing unit 24 extracts the image data of the recording range, thereby obtaining image data of the same shooting range as the recording range at the start of shooting, and stores it in the memory 32. Note that known techniques can be used for distortion correction and other necessary steps to convert the area extracted from the circular fisheye image into a normal rectangular image.

[0081] The image processing unit 24 performs the synthesis process in S407 and the cropping process in S408 for both the image data for recording and the image data for display. The image processing unit 24 also performs distortion correction and other necessary steps to convert the region cropped from the circular fisheye image into a normal rectangular image.

[0082] The system control unit 50 records the recording image data from the image data within the recording range extracted by the image processing unit 24 onto the recording medium 90 as data for a wide-angle image generated in crop mode. The system control unit 50 also processes the display image data in S40. 9 The system then performs guide display processing and outputs the result to the display unit 28 or an external device. Note that recording to the recording medium 90 may be performed for multiple frames at once, depending on the encoding method.

[0083] In S409, the system control unit 50 performs guide display processing. Details will be described later. In S410, the system control unit 50 determines whether a recording termination operation has been performed. If it determines that a recording termination operation has been performed, it terminates the operations related to video recording. If it does not determine that a recording termination operation has been performed, it executes the processing of the next frame from S403. The recording termination operation may be the operation of the video recording switch or a recording termination instruction received from an external device via the communication unit 54.

[0084] The operation of the system control unit 50 in S404 will be explained using the flowchart shown in Figure 5. In S500, the system control unit 50 calculates the difference between the reference orientation of the digital camera 100 (in this case, the orientation at the start of shooting) and the current orientation of the digital camera 100 acquired in S403. Then, based on the difference in orientation, the system control unit 50 calculates the position of the recording range in the image obtained with the current orientation.

[0085] Furthermore, the system control unit 50 determines whether the recording range corresponding to the current orientation exceeds the shooting range of the currently active camera. The system control unit 50 detects when the recording range 307 exceeds the shooting range 305 of the active rear camera 120 (i.e., it has an area that is not included in the shooting range 305), as shown in the example in Figure 3(b). If the system control unit 50 determines that the recording range corresponding to the current orientation exceeds the shooting range of the currently active camera, it executes S501; otherwise, it executes S503.

[0086] In S501, the system control unit 50 activates the cameras that are currently disabled and whose shooting range includes the recording range corresponding to the current orientation of the digital camera 100, and then terminates the process in S404. For example, in the example shown in Figure 3(b), the front camera 110 is activated.

[0087] S503 to S508 are processes that the system control unit 50 executes for each currently active camera in the digital camera 100. In S504, the system control unit 50 acquires the range of shooting possible. In S505, the system control unit 50 determines whether the shooting range acquired in S504 includes at least a portion of the recording range corresponding to the current orientation of the digital camera 100, which was calculated in S500. If the system control unit 50 determines that the shooting range includes at least a portion of the recording range, it executes S506; otherwise, it executes S507.

[0088] In S506, the system control unit 50 maintains the target camera active and terminates the process in S404. In S507, the system control unit 50 disables the target camera and terminates the process in S404. Disabling means, for example, a state that consumes less power than the enabled state, and power supply to at least some parts may continue. For example, the image sensor and A / D converter may be put into a power-saving state. This can also be described as transitioning the circuit related to the disabled photographic lens to a state that consumes less power than when the lens is enabled.

[0089] In this way, the cameras necessary to capture the recording range corresponding to the current orientation (i.e., the shooting range required for image generation) are enabled, and the cameras not necessary to capture the recording range corresponding to the current orientation are disabled. Therefore, only the necessary cameras are dynamically enabled in response to changes in the orientation of the digital camera 100, making it possible to reduce power consumption.

[0090] Next, using Figures 6 and 7, S40 9The details of the process will now be explained. For convenience, it is assumed that the digital camera 100 is operating in a linked mode with the smartphone 200, and that image data for live view display is being transmitted to the smartphone 200 via the communication unit 54. It is also assumed that the smartphone 200 is running a linked application for the digital camera 100, and that the display 205 is showing the live view display image data received from the digital camera 100. Furthermore, it is assumed that the user is able to hold the digital camera 100 while looking at the display 205 of the smartphone 200. If it is possible to view the live view on the display unit 28 of the digital camera 100, the digital camera 100 may perform the guide display operation described below on its own.

[0091] Figure 6 shows S40 9 The following shows a specific example of the guide display implemented. The live view image 600 is displayed on the display 205. The guide display consists of predetermined images such as indicators and icons that are superimposed on the live view image when multiple cameras are enabled. Here, the boundary line 601, the warning for the number of cameras in use 602, and the recommended direction display 603 are included in the guide display.

[0092] The dotted boundary line 601 indicates the boundary of the camera's shooting range and corresponds to boundary line 300 in Figure 3. Boundary line 601 is displayed when multiple cameras are required to capture the recording range, as shown in Figure 3(b). Boundary line 601 is not displayed when the entire recording range can be captured by a single camera, as shown in Figure 3(a).

[0093] The camera usage warning 602 is an icon indicating that multiple cameras are being used to capture the recording area. Both the boundary line 601 and the camera usage warning 602 are displayed. Note that alternative representations such as text or lines may be used instead of icons.

[0094] The recommended direction indicator 603 is an icon that shows how to change the orientation of the digital camera 100 to reduce the number of cameras needed to capture the recording area. In the example in Figure 3, to change the orientation from Figure 3(b) to Figure 3(a), the digital camera 100 should be positioned in a near-upright orientation. In this case, the recommended direction indicator 603 is the upright direction (the direction in which the digital camera 100 is held upright). ) This can be represented by an icon. More specifically, the current posture and the recommended posture may be displayed alternately, or an animation may be used to display them.

[0095] The digital camera 100 (system control unit 50) generates a live view display image with the guide display superimposed using the image processing unit 24 when the requirements for guide display are met.

[0096] The operation of the system control unit 50 in S409 will be explained using the flowchart shown in Figure 7. In S700, the system control unit 50 determines whether multiple cameras are enabled or not. If it determines that multiple cameras are enabled, it executes S701; otherwise, it executes S704.

[0097] In S701, the system control unit 50 decides to hide the camera usage warning 602. In S702, the system control unit 50 decides to hide the boundary line 601. In S703, the system control unit 50 decides to hide the recommended direction indicator 603. The system control unit 50 notifies the image processing unit 24 of the decisions made in S701 to S703. Alternatively, you may decide to hide the entire guide display (boundary line 601, camera usage warning 602, recommended direction display 603) all at once.

[0098] In S704, the system control unit 50 decides to display the camera usage warning 602. In S705, the system control unit 50 calculates the difference in angle between the optical axis direction and the shooting direction for each of the valid cameras, and determines the camera with the smallest angle difference to be the primary camera.

[0099] In S706, the system control unit 50 determines the display of the boundary line 601 of the main camera's shooting range.

[0100] In S707, the system control unit 50 calculates the direction in which the digital camera 100 should be pointed in order to minimize the difference in angle between the optical axis direction of the main camera and the shooting direction. In S708, the system control unit 50 determines the display of the recommended direction indicator 603. The system control unit 50 notifies the image processing unit 24 of the determinations made in S704, S706, and S708, and the direction calculated in S707.

[0101] In S709, the image processing unit 24 generates image data for live view display based on a notification from the system control unit 50. If steps S701 to S703 are executed, the image processing unit 24 generates image data for normal live view display without superimposing the guide display.

[0102] On the other hand, if S704 to S708 are executed, the image processing unit 24 generates image data for live view display with guide displays superimposed. The superimposition positions of the camera count warning 602 and the recommended direction display 603 among the guide displays, as well as the image of the recommended direction display 603 corresponding to the notified direction, are predetermined. The image processing unit 24 also superimposes boundary lines 601 on the image data for live view display at positions corresponding to the boundaries of the multiple images used to generate the composite image in S407.

[0103] The image processing unit 24 then stores the generated image data for live view display in the memory 32. The system control unit 50 transmits the image data for live view display to the smartphone 200 via the communication unit 54. The system control unit 50 can also output the image data for live view display to the display unit 28.

[0104] As described above, according to this embodiment, in an imaging device capable of generating an omnidirectional image using multiple imaging units, when generating an image with a smaller field of view than the omnidirectional image, only the imaging units necessary for image generation are enabled. Furthermore, if image blur correction cannot be achieved within the shooting range of the enabled imaging units, the necessary imaging units are enabled. Therefore, power consumption can be reduced while improving the range in which image blur correction is possible compared to conventional devices.

[0105] (Other embodiments) The above-described embodiment described a configuration in which one imaging unit uses one photographic lens. However, a similar embodiment can also be realized in a configuration in which the number of photographic lenses used by one imaging unit can be changed. In this case, when using multiple lenses, shooting can be performed sequentially by switching between lenses. Also, the operation related to enabling and disabling the camera in the above-described embodiment can be read as enabling and disabling the lenses. In the case of a configuration in which the number of lenses used is increased or decreased according to the change in the attitude of the digital camera, the effect of reducing power consumption is lower than in a configuration in which the number of imaging units used is increased or decreased, but a similar effect can be obtained in terms of improving the range in which image blur correction is possible.

[0106] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0107] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0108] 100…Digital camera, 110…Front camera, 120…Rear camera, 120, 22a, 22b…Image sensor, 103a, 103b…Shooting lens, 50…System control unit

Claims

1. An imaging device capable of generating omnidirectional images using multiple imaging lenses, A detection means for detecting the orientation of the imaging device, When the imaging device is in an operating mode that generates an image with a smaller field of view than an omnidirectional image, the imaging device includes a means for identifying the shooting range necessary for generating the image, which is unaffected by changes in the imaging device's posture, based on the difference between the imaging device's reference posture and its current posture. When the imaging device is in the operating mode, a control means enables the photographic lens among the plurality of photographic lenses that is necessary to photograph the shooting range, and disables the photographic lens that is not necessary to photograph the shooting range. A generation means for generating an image by cropping out the area of ​​the shooting range from an image obtained using the activated shooting lens among the plurality of shooting lenses, If there are multiple activated photographic lenses, the system includes a synthesis means that stitches together images captured using each of the activated photographic lenses to generate a composite image. If there are multiple activated photographic lenses, the generation means extracts the area of ​​the shooting range from the composite image, superimposes a predetermined guide display onto it, and generates the image. The guide display includes a display showing how changing the orientation of the imaging device can reduce the number of lenses that are activated. An imaging device characterized by the following features.

2. The imaging apparatus according to claim 1, characterized in that the guide display includes a display indicating the boundaries of a plurality of images used to generate the composite image or a display indicating that a plurality of lenses are activated.

3. The imaging apparatus according to claim 1 or 2, characterized in that the image generated by the generation means is output to a display device of the imaging apparatus or an external device having a display device.

4. The imaging device according to any one of claims 1 to 3, characterized in that the identification means performs the identification by converting the position of the imaging range when the imaging device is in the reference posture to a position corresponding to the current posture of the imaging device.

5. The imaging apparatus according to any one of claims 1 to 4, characterized in that the control means transitions the circuit relating to the disabled photographic lens to a state that consumes less power than when the photographic lens is enabled.

6. The imaging apparatus according to claim 5, characterized in that the circuit includes an image sensor.

7. The imaging apparatus according to any one of claims 1 to 6, characterized in that the identifying means and the control means continuously perform the identifying and enabling and disabling of the imaging lens.

8. The imaging device according to any one of claims 1 to 7, characterized in that the plurality of photographic lenses are circular fisheye lenses.

9. A control method for an imaging device capable of generating omnidirectional images using multiple imaging lenses, To detect the orientation of the imaging device, When the imaging device is in an operating mode that generates an image with a smaller field of view than an omnidirectional image, the imaging range necessary for generating the image, which is unaffected by changes in the imaging device's orientation, is determined based on the difference between the imaging device's reference orientation and its current orientation. When the imaging device is in the operating mode, the imaging lens necessary for capturing the shooting range among the plurality of imaging lenses is enabled, and the imaging lens not necessary for capturing the shooting range is disabled. From the image obtained using the activated photographic lens among the plurality of photographic lenses, the region of the photographic range is extracted to generate the image. If there are multiple activated photographic lenses, the system will stitch together the images captured using each of the activated photographic lenses to generate a composite image. If there are multiple activated photographic lenses, the generation process involves extracting the area of ​​the shooting range from the composite image, superimposing a predetermined guide display, and generating the image. The guide display includes a display showing how changing the orientation of the imaging device can reduce the number of lenses that are activated. A control method for an imaging device, characterized by the following:

10. A program for causing the computer of an imaging device to function as each of the means of the imaging device described in any one of claims 1 to 8.

Images