Image capture apparatus, control method therefor and computer-readable medium

Abstract

An imaging device that generates an omnidirectional image using a shooting lens is disclosed. The device detects the orientation of the imaging device. When the imaging device is in an operation mode that generates a first image with a smaller field of view than the omnidirectional image, the device specifies the shooting range required for generating the image based on the difference between the reference orientation and the current orientation of the imaging device. When the imaging device is in an operation mode, the device enables shooting using a shooting lens necessary to capture the shooting range among a plurality of shooting lenses, and disables shooting using a shooting lens unnecessary to capture the shooting range among a plurality of shooting lenses.

Description

Technology Field

[0001] The present invention relates to an imaging device, a control method thereof, and a computer-readable medium, and in particular to an imaging device capable of capturing omnidirectional images and a control method thereof. Background Technology

[0002] An imaging device capable of capturing omnidirectional images with a 360-degree field of view is known (see Japanese Patent Publication No. 2019-205161). Such an imaging device is called a 360-degree camera or an omnidirectional camera.

[0003] For example, a 360-degree camera equipped with two circular fisheye lenses with a field of view of 180 degrees has the function of cropping a rectangular area from an image taken using only one lens to produce an image that looks like it was taken with a wide-angle lens with a field of view of less than 180 degrees.

[0004] In this case, electronic image stabilization can be performed on camera rotation around the optical axis (movement in the roll direction) regardless of the rotation angle. However, for camera rotation around an axis orthogonal to the optical axis (movement in the pitch and yaw directions), there is a limitation on the rotation angle at which image stabilization can be performed. The problem to be solved

[0005] According to the present invention, an imaging device capable of generating an omnidirectional image using a plurality of lenses and a control method thereof are provided, which can improve hand shake correction when generating an image with a smaller field of view than an omnidirectional image. means of solving the problem

[0006] According to one aspect of the present invention, an imaging device is provided that generates an omnidirectional image using a plurality of imaging lenses, the imaging device comprising: a detection means for detecting the posture of the imaging device; a specification means for specifying a shooting range required for generating an image based on the difference between a reference posture and the current posture of the imaging device detected by the detection means when the imaging device is in an operation mode that generates a first image having a smaller angle of view than the omnidirectional image; and a control means for enabling shooting using a shooting lens necessary to capture the shooting range among the plurality of imaging lenses and deactivating shooting using a shooting lens unnecessary to capture the shooting range among the plurality of imaging lenses when the imaging device is in the operation mode.

[0007] According to one aspect of the present invention, a control method for an imaging device capable of generating an omnidirectional image using a plurality of imaging lenses is provided, comprising the steps of: detecting the posture of the imaging device; determining a shooting range required for generating an image based on the difference between a reference posture and the current posture of the imaging device detected in the detection step when the imaging device is in an operation mode that generates a first image having a smaller angle of view than an omnidirectional image; and enabling shooting using a shooting lens among the plurality of imaging lenses that is necessary to capture the shooting range and deactivating shooting using a shooting lens among the plurality of imaging lenses that is unnecessary to capture the shooting range when the imaging device is in the operation mode.

[0008] According to one aspect of the present invention, a computer-readable medium is provided that stores a program for executing a method according to the present invention in a computer equipped with an imaging device capable of generating omnidirectional images using a plurality of imaging lenses.

[0009] Another feature of the present invention will become apparent from the description of the following embodiments with reference to the accompanying drawings. Brief explanation of the drawing

[0010] FIGS. 1a to 1c are drawings relating to a digital camera showing an example of an imaging device according to an embodiment. FIGS. 2 and FIGS. 2b are drawings relating to a smartphone showing an example of an electronic device according to an embodiment. FIGS. 3a and FIGS. 3b are schematic diagrams for explaining operation control in an embodiment. FIG. 4 is a flowchart regarding the operation of a digital camera according to an embodiment. FIG. 5 is a flowchart regarding the operation of a digital camera according to an embodiment. FIG. 6 is a drawing showing an example of a guide mark according to an embodiment. FIG. 7 is a flowchart regarding the operation of a digital camera according to an embodiment. Specific details for implementing the invention

[0011] Hereinafter, embodiments are described in detail with reference to the attached drawings. At this time, the following embodiments do not limit the invention with respect to the claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined at will. Furthermore, in the attached drawings, the same reference number is assigned to identical or similar components, and redundant descriptions are omitted.

[0012] In the following embodiments, the present invention is implemented using an imaging device such as a digital camera. However, the present invention can be implemented in any electronic device having a shooting function. Examples of such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are examples, and the present invention can be implemented in other electronic devices.

[0013] FIGS. 1A and FIGS. 1B are perspective views showing an example of the exterior of a digital camera (100) representing an example of an imaging device according to an embodiment of the present invention. Here, the side on which the shutter button (61) is installed corresponds to the front of the digital camera (100). Accordingly, FIG. 1A shows the front side of the configuration example, and FIG. 1B shows the rear side. FIG. 1C is a block diagram showing an example of the functional configuration of the digital camera (100).

[0014] Hereinafter, an exemplary configuration of a digital camera (100) will be described with reference to FIGS. 1a to 1c. The digital camera (100) is provided with shooting lenses 103a and 103b, which are circular fisheye lenses with a field of view of 180 degrees, on the front and rear sides, respectively. The shooting lenses 103a and 103b are configured, for example, to have the same optical axis and different shooting directions of 180 degrees. In this specification, the shooting direction of the shooting lens 103a installed on the front side of the digital camera (100) is referred to as the front, and the shooting direction of the shooting lens 103b installed on the rear side of the digital camera (100) is referred to as the rear.

[0015] The digital camera (100) is an omnidirectional camera or a 360-degree camera capable of generating an omnidirectional image with a horizontal angle of view of 360 degrees by synthesizing or stitching images formed by shooting lenses 103a and 103b. At this time, the omnidirectional image may be a 360-degree image, or may have a vertical angle of view of less than 360 degrees, such as a 180-degree image. In this embodiment, the two shooting lenses 103a and 103b are circular fisheye lenses with a horizontal angle of view of 180 degrees or more, and the digital camera (100) is capable of generating a 360-degree image.

[0016] In this embodiment, two shooting lenses 103a and 103b cover a horizontal field of view of 360 degrees or more, but a configuration may be used in which three or more shooting lenses with different shooting directions cover a horizontal field of view of 360 degrees or more.

[0017] A digital camera (100) has a plurality of imaging units including a front camera (110) that captures the front and a rear camera (120) that captures the rear. The front camera (110) has a shooting lens 103a and a barrier 102a (protective member) installed on the front surface of the shooting lens 103a. The barrier 102a is, for example, a transparent hemispherical member. The shooting lens 103a has a focus lens that is movable in the direction of the optical axis. The shutter 101a serves as an aperture. The shooting lens 103a forms an optical image on the imaging surface of an imaging element 22a.

[0018] The imaging element 22a may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The imaging element 22a has a pixel array in which a plurality of pixels are arranged in a two-dimensional array, and a peripheral circuit for reading a signal from each pixel. Each pixel accumulates a charge according to the amount of incident light by photoelectric conversion. By reading a signal having a voltage according to the amount of charge accumulated during the exposure period from each pixel, a pixel signal group (analog image signal) representing an optical image formed on an imaging surface is obtained. An A / D converter 23a converts the analog image signal read from the imaging element 22a into a digital image signal (image data).

[0019] Additionally, the front camera (110) has a mechanism for moving a focus lens, a mechanism for detecting the position of the focus lens, and a mechanism for driving a shutter 101a. At this time, the front camera (110) may have a moving mechanism of an image element 22a for realizing an optical image stabilization function and / or a moving mechanism of a shift lens included in a shooting lens 103a.

[0020] Since the rear camera (120) has the same configuration as the front camera (110), the description of each element is omitted. Hereinafter, the imaging elements 22a and 22b are collectively referred to as imaging element 22. The same applies to other components provided by the front camera (110) and the rear camera (120). In this embodiment, since a configuration in which one imaging unit uses one shooting lens is used, selectively using the imaging unit and selectively using the shooting lens have the same meaning. Therefore, the description regarding the activation and deactivation of the imaging unit (camera) can be read as the activation and deactivation of shooting through the shooting lens. However, the present invention is also applicable to a configuration in which one imaging unit uses multiple shooting lenses simultaneously, or switches between using multiple imaging lenses.

[0021] Image data output by the A / D converter (23) is recorded in memory (32) by passing through both the image processing unit (24) and the memory control unit (15), or by passing only through the memory control unit (15).

[0022] The memory (32) is used as a buffer for image data, a working memory of the image processing unit (24), a video memory of the display unit (28), etc.

[0023] The image processing unit (24) applies a predetermined image processing to image data output by the A / D converter (23) or memory control unit (15), or stored in memory (32), to acquire or generate signals, image data, and various information according to the intended use. The image processing unit (24) may be a dedicated hardware circuit, such as an Application Specific Integrated Circuit (ASIC), configured to realize a specific function, for example. Alternatively, the image processing unit (24) may have a configuration in which a processor, such as a Digital Signal Processor (DSP) or a Graphics Processing Unit (GPU), executes software to realize a specific function.

[0024] Image processing that can be applied to image data by the image processing unit (24) may include, for example, preprocessing, color interpolation, correction, detection, data processing, evaluation value calculation, special effect processing, etc.

[0025] Preprocessing may include signal amplification, reference level adjustment, and defective pixel correction.

[0026] Color interpolation processing is a process performed when a color filter is installed in the image element (22) to interpolate the values ​​of color components that are not included in each pixel data constituting the image data. Color interpolation processing is also called demosaicing processing.

[0027] The correction processing may include various processes such as white balance adjustment, gradation correction, correction of image degradation caused by optical aberrations of the shooting lens (103a, 130b) (image recovery), and color correction. The correction processing further includes a process for correcting significant distortion in the periphery of an ultra-wide angle lens, such as a circular fisheye lens.

[0028] Detection processing may include detection processing of feature regions (e.g., face regions or human body regions) or their movements, and person recognition processing.

[0029] Data processing may include processing such as trimming, synthesizing, scaling, encoding and decoding, and generating header information (creating data files). Data processing further includes the generation of image data for display and image data for recording.

[0030] The evaluation value calculation process may include processing such as generating signals or evaluation values ​​used for autofocus detection (AF), or generating evaluation values ​​used for auto exposure control (AE).

[0031] Special effects processing may include the addition of blurring effects, changes in color tone, relighting, etc.

[0032] At this time, these are examples of processing that can be applied by the image processing unit (24), and do not limit the processing that the image processing unit (24) applies. Additionally, the image processing that can be applied by the image processing unit (24) may be executed by the system control unit (50).

[0033] In this embodiment, the image processing unit (24) stitches two circular fisheye images captured by the front camera (110) and the rear camera (120) to generate a 360-degree image. The generated 360-degree image is converted into an image using an equidistant cylinder transformation, and the position of each pixel can be associated with the coordinates of the 360-degree surface.

[0034] Additionally, the image processing unit (24) cuts out an area from the circular fisheye image data and applies distortion correction, etc., to generate a rectangular image data suitable for a display unit (28) or an external display device.

[0035] The system control unit (50) is, for example, a processor capable of executing a program (CPU, MPU, microprocessor, etc.). The system control unit (50) loads a program stored in non-volatile memory (56) into system memory (52) and executes the program, thereby controlling the operation of each function block of the digital camera (100) and realizing the function of the digital camera (100).

[0036] The non-volatile memory (56) is electrically rewriteable and stores programs executed by the system control unit (50), various setting values ​​of the digital camera (100), GUI data, etc. The system memory (52) is used as the main memory when the system control unit (50) executes a program. At this time, the memory (32) and the system memory (52) may be different areas within a contiguous memory space.

[0037] The shutter button (61) has a switch SW1 (62) that turns ON with a half-press and a switch SW2 (64) that turns ON with a full press. The system control unit (50) recognizes the ON of SW1 (62) as a signal to prepare for shooting a still image and the ON of switch SW2 (64) as a signal to start shooting a still image. When the system control unit (50) detects the ON of switch SW1 (62), it uses signals, evaluation values, etc. generated by the image processing unit (24) to perform autofocus detection (AF) and auto exposure control (AE) of the front camera (110) and / or rear camera (120). Additionally, when the system control unit (50) detects the ON of switch SW2 (64), it controls the shutter (101) according to the exposure conditions determined in the AE processing to shoot a still image and perform recording processing. The still image data for recording generated by the image processing unit (24) is temporarily stored in memory (32) before being recorded on the recording medium (90) via I / F (18) by the system control unit (50).

[0038] At this time, the shutter button (61) may have only one switch. In this case, when the ON of the switch is detected, the system control unit (50) continuously executes a shooting preparation operation and a shooting process.

[0039] The user can change the operating mode of the digital camera (100) by operating the mode switching switch (60). The operating mode includes, for example, a mode for shooting 360-degree images, a mode for shooting general wide-angle images with a horizontal angle of view of less than 180 degrees (called a crop mode), a playback mode, and a mode for operating in conjunction with an external device. At this time, the mode for shooting wide-angle images may have multiple modes according to the horizontal angle of view (90-degree mode, 150-degree mode, etc.).

[0040] At this time, an operation mode may be selected by a combination of the operation of the mode switching switch (60) and the operation of other operating members. For example, after selecting a major category of operation mode using the mode switching switch (60), an operation mode may be selected from a more detailed category displayed on the display unit (28).

[0041] The power switch (72) is a switch for indicating the ON or OFF of the power of the digital camera (100). The power control unit (80) controls these operations.

[0042] The power control unit (80) has a battery detection circuit, a DC-DC converter, a switch circuit for switching a function block that supplies power, etc., and controls the power supply to the components of the digital camera (100) from a power supply unit (30), which may be a battery or an AC adapter. The power control unit (80) detects the type of the power supply unit (30). In addition, if the power supply unit (30) is a battery, the power control unit (80) detects the type and the remaining battery amount. The power control unit (80) can change the components that supply power and the supplied power according to the state of the power switch (72) and the control of the system control unit (50).

[0043] The microphone (20) is directed outside the digital camera (100) and outputs a voice signal to the system control unit (50). When recording a video, image data and voice data are recorded.

[0044] The control unit (70) is a collective term for a group of input devices (buttons, switches, dials, etc.) other than the mode switching switch (60), shutter button (61), and power switch (72). This includes a video recording switch, a menu button, a directional key, a confirmation key, etc. If the display unit (28) is a touch display, the control unit (70) may be composed of touch-operable software buttons or keys.

[0045] The display unit (28) is, for example, a liquid crystal display (LCD). The display unit (28) may also be a touch display. The display unit (28) displays text and images. By continuously displaying a captured video image on the display unit (28) immediately, the display unit (28) can function as an electronic viewfinder (EVF). The video image displayed by functioning the display device as an EVF is called a live view image. At this time, by outputting the live view image to an external device connected to the communication unit (54), the display device of the external device can also function as an EVF.

[0046] The light-emitting part (21) is a light-emitting diode (LED) that notifies the user of the status of the digital camera (100), etc., by the pattern or color of the light emission.

[0047] The fixing part (40) (Fig. 1b) installed on the bottom surface of the digital camera (100) is, for example, a screw hole to which a tripod is attached.

[0048] The system timer (53) outputs the time of the built-in clock and measures time according to the request from the system control unit (50).

[0049] The communication unit (54) is an interface for performing wired or wireless communication with external devices, such as electronic devices and external display devices, as described later. The communication unit (54) is compatible with one or more wired or wireless communication standards and has a connector, transceiver, etc. suitable for these standards. Representative standards supported by the communication unit (54) include USB, HDMI (registered trademark), Bluetooth (registered trademark), wireless LAN, etc., but are not limited to these.

[0050] The attitude detection unit (55) has, for example, a gyroscope sensor and an accelerometer sensor and outputs a signal indicating the attitude and movement of the digital camera (100) to the system control unit (50). The attitude of the digital camera (100) is indicated by the rotation angle (roll, pitch, and yaw) around each axis with respect to the x-axis parallel to the optical axis, the y-axis extending in the horizontal direction, and the z-axis extending in the vertical direction. The attitude of the digital camera (100) at the time of shooting may be recorded by corresponding it to image data. In addition, the detected attitude and movement of the digital camera (100) may be used for shake correction and tilt correction.

[0051] The I / F (18) is an interface for recording data on a recording medium (90), such as a memory card or a hard disk, and for reading the data recorded on the recording medium (90). The recording medium (90) does not need to be detachable from the digital camera (100).

[0052] FIGS. 2A and FIGS. 2B are drawings relating to a smartphone (200), which is an example of an external device capable of being linked with a digital camera (100) by communication using a communication unit (54). FIG. 2A is a perspective view showing an example of an external appearance, and FIG. 2B is a block diagram showing an example of a functional configuration. At this time, the external device is not limited to a smartphone, and any electronic device capable of communicating with the digital camera (100) and having a processor capable of executing an application for linking with the digital camera (100) is sufficient.

[0053] The configuration of the smartphone (200) is described.

[0054] The internal bus (250) connects each block so that data, etc. can be exchanged in both directions.

[0055] The CPU (201) is a processor capable of executing a program by loading a program stored in non-volatile memory (203) into memory (202) and executing the program, and controls the operation of each function block of the smartphone (200) to realize the functions of the smartphone (200).

[0056] Memory (202) is used as main memory when the CPU (201) executes a program. A portion of memory (202) is used as video memory for the display (205).

[0057] The non-volatile memory (203) is electrically rewriteable and stores programs (OS and applications) executed by the CPU (201), settings of the smartphone (200), GUI data, user data, etc.

[0058] The display (205) is, for example, an LCD, and displays images and various information for the OS and applications. The display (205) is a touch display having a touch panel (206a) and is capable of detecting touch operations on the display surface of the display (205). The display (205) may be an external device.

[0059] The image processing unit (24), based on the control of the CPU (201), applies image processing to image data stored in the non-volatile memory (203) and the recording medium (208), image data acquired via an external I / F (209), image data acquired via a communication I / F (210), etc.

[0060] The image processing that the image processing unit (24) can apply to the image data may be similar to the image processing unit (24) of the digital camera (100). If the smartphone (200) does not have a camera, the generation of evaluation values ​​used for AF and AE, etc., may not be performed. In addition, the image processing that the image processing unit (24) can execute may be executed by the CPU (201).

[0061] The image processing unit (24) can generate image data for displaying Virtual Reality (VR) according to the movement of the smartphone (200) from an ultra-wide angle image (e.g., an image with a horizontal angle of view exceeding 180 degrees) under the control of the CPU (201). VR display is realized by cutting out an area of ​​the shooting range corresponding to the change in posture of the smartphone (200) from the ultra-wide angle image and displaying the generated cut image data on the display (205). Through VR display, for example, by fixing the display (205) of the smartphone (200) in front of the user's eyes using goggles, the displayed image changes to follow the movement of the user's head. Because of this, the user can experience the sensation of being in a virtual space displayed as an ultra-wide angle image.

[0062] In contrast, electronic shake correction can be realized by controlling the image processing unit (24) to cut out the same shooting range area from the ultra-wide angle image regardless of the position of the smartphone (200).

[0063] The control unit (206) is a general term for a group of input devices that a user can use to give instructions to the smartphone (200). Generally, input devices that the smartphone (200) has include buttons, switches, and touch panels, but are not limited to these. Additionally, a keyboard and a mouse connected to the smartphone (200) for communication can also constitute the control unit (206). In this case, in FIG. 2b, the touch panel (206a) is shown separately from the display (205). However, in reality, the touch panel (206a) is embedded in or attached to the display screen of the display (205).

[0064] The power button (206b), volume buttons 206c, 206d, and home button (206e) are examples of input devices that constitute the control unit (206). The power button (206b) switches the power of the smartphone (200) ON and OFF. Volume buttons 206c and 206d are buttons that increase or decrease the volume output from the voice output unit (212). The home button (206e) is a button for displaying a specific screen provided by the OS on the display (205).

[0065] The media I / F (207) is an interface for accessing the recording medium (208). If the recording medium (208) is a removable medium, such as a memory card, for example, the media I / F (207) has a slot into which the recording medium (208) is inserted or removed. The CPU (201) can write data to the recording medium (208) via the media I / F (207) and read data from the recording medium (208).

[0066] The external I / F (209) is an interface for wired or wireless communication with an external device, such as a digital camera (100) or an external display device. The external I / F (209) is compatible with one or more wired or wireless communication standards and has a connector, transceiver, etc. suitable for these standards. Representative standards supported by the communication unit (54) include USB, HDMI (registered trademark), Bluetooth (registered trademark), wireless LAN, etc., but are not limited to these.

[0067] The communication I / F (210) is an interface that performs communication through a mobile phone network (211). The communication I / F (210) may be a communication I / F that conforms to mobile communication standards set by 3GPP, such as, for example, a 3G, 4G, or 5G modem.

[0068] The voice output unit (212) outputs voice (voice based on video or music data, operation sound, ringtone, various notification sounds, etc.). The voice output unit (212) includes a voice output terminal (212a) for connecting earphones, etc. and a speaker (212b), but may output voice to an external device via an external I / F.

[0069] The attitude detection unit (213) has, for example, a gyroscope sensor and an accelerometer sensor and outputs a signal indicating the attitude and movement of the smartphone (200) to the CPU (201). The attitude of the smartphone (200) is indicated by the rotation angle (roll, pitch, and yaw) around each axis with respect to the x-axis orthogonal to the display screen of the display (205), the y-axis extending in the horizontal direction, and the z-axis extending in the vertical direction. The attitude detected by the attitude detection unit (213) can be used in the aforementioned VR display.

[0070] FIGS. 3a and 3b schematically illustrate the control of operation of the front camera (110) and rear camera (120) according to the position when the digital camera (100) is operating in an operation mode (crop mode) for capturing a still image with a horizontal angle of view of less than 180 degrees.

[0071] FIG. 3a shows an example of the posture (reference posture) of a digital camera (100) at the start of shooting, and FIG. 3b shows an example of the posture of a digital camera (100) during shooting. In the state shown in FIG. 3a (upright state), roll, pitch, and yaw are all 0 degrees. FIG. 3b also shows a state in which, from the state of FIG. 3a, the rear camera (120) is facing downward and only the pitch (rotation angle around the y-axis) has changed.

[0072] In crop mode, an image for recording or display is generated by cropping a portion of an area from an omnidirectional image or an ultra-wide-angle image. If the horizontal field of view of the image generated in crop mode can be covered by one of the shooting lenses, there is no need to use another shooting lens for shooting. For this reason, only the shooting lens and related circuitry required for shooting can be used.

[0073] For example, when the horizontal viewing angle of an image generated in crop mode by the digital camera (100) according to the present embodiment is less than 180 degrees, power consumption can be reduced by making only one of the front camera (110) or the rear camera (120) active. In the present embodiment, the rear camera (120) is used when shooting is started so that the display unit (28) can be seen when shooting is started. Additionally, an image with a horizontal viewing angle of 150 degrees and a vertical viewing angle of 90 degrees is generated.

[0074] Furthermore, during shooting, electronic image stabilization is performed by changing the cropping position of the image so that the same shooting range as at the start of shooting is maintained even if the posture of the digital camera (100) changes from the reference posture (e.g., the posture at the start of shooting). Hereinafter, the range that is cropped from the shooting range in crop mode is referred to as the recording range for convenience. At this time, the use of the image data generated in crop mode is not limited to recording.

[0075] At the start of shooting, the position (reference position) of the digital camera (100) is in an upright state. In this case, arrow 306 indicates the shooting direction (the direction of the optical axis of the shooting lenses (103a, 103b)). Accordingly, among the shooting range (305) (the semicircle on the right) of the rear camera (120), the range 307 shown in gray corresponds to the recording range in the vertical direction. Since the front camera (110) is disabled, the shooting range (303) (the semicircle on the left) of the front camera (110) is not shot. The direction of the boundary line (300) between the shooting range (303) of the front camera (110) and the shooting range (305) of the rear camera (120) coincides with the direction of gravity (308).

[0076] When the digital camera (100) changes from the position shown in FIG. 3a to the position shown in FIG. 3b, in order to realize shake correction, it is necessary to cut out a range such as the recording range (307) at the start of shooting. However, among the recording range (307), the hatched portion in FIG. 3b is not included in the shooting range (305) of the rear camera (120) but is included in the shooting range (303) of the front camera (110). Therefore, shake correction cannot be realized with only the image data obtained by the rear camera (120).

[0077] In this way, when hand shake correction cannot be realized within the shooting range of a single camera, the system control unit (50) enables another camera (in this example, the front camera (110)) having a shooting range that includes the range required for hand shake correction, and performs shooting with multiple cameras. Accordingly, even if the digital camera (100) changes to the position of FIG. 3b, the recording range (307) can be cut and hand shake correction can be realized.

[0078] Meanwhile, if image stabilization can be achieved within the shooting range of one camera, power consumption can be reduced by disabling another camera. In this case, when the recording range approaches the boundary of the current shooting range, the camera capable of shooting the range required when the recording range exceeds the shooting range may be enabled. Accordingly, it is possible to avoid the inability to perform image stabilization while the additionally enabled camera is in operation and an image is being obtained.

[0079] In this case, in the aforementioned example using FIGS. 3a and 3b, control is performed according to the change in attitude in the pitch direction. However, the same applies to the change in attitude in the yaw direction.

[0080] The operation of the system control unit (50) in crop mode is further explained using the flowchart of FIG. 4. This operation is executed when the digital camera (100) is operating in crop mode, image stabilization is set to effective, and video recording is started. At this time, video recording may be for recording or for live view display, but in the example described later, video recording is performed for recording. In the case of recording, S400 may be executed by the operation of the video recording switch in standby state, or in the case of live view display, by the detection of the operation to enable image stabilization in standby state, but is not limited to this.

[0081] At this time, if remote operation by an external device is valid, S400 may be executed by a recording start instruction received from the external device via the communication unit (54).

[0082] In step S400, the system control unit (50) obtains the posture (reference posture) of the digital camera (100) from the posture detection unit (55). Based on the obtained posture, the system control unit (50) determines the optical axis direction of the rear camera (120) or the front camera (110) as the shooting direction. The shooting direction is a constant direction that is not affected by subsequent changes in the posture of the digital camera (100). The system control unit (50) stores the obtained posture and the determined shooting direction in, for example, the system memory (52).

[0083] In step S401, the system control unit (50) determines the recording range based on the shooting direction determined in step S400 and the settings of the digital camera (100). In this example, in crop mode, the horizontal angle of view is selectable, and the vertical angle of view is constant or determined according to the horizontal angle of view. In this example, the horizontal angle of view is selected as 150 degrees, and the vertical angle of view is determined as 90 degrees.

[0084] The system control unit (50) stores the range determined as a recording range from the position, shooting direction, and shooting angle of the digital camera (100) at the start of recording in relation to the position of the digital camera (100) at the start of recording in the system memory (52). The system control unit (50) may, for example, set the area obtained by mapping a rectangular area having a size based on the horizontal angle of view and the vertical angle of view centered on the shooting direction to the omnidirectional image according to the position of the digital camera (100) as the recording range. Accordingly, the recording range corresponds to a specific area in the omnidirectional image that the digital camera (100) can generate.

[0085] In step S402, the system control unit (50) initiates a video recording operation and continues to execute the processing after step S403 frame by frame.

[0086] In step S403, the system control unit (50) obtains the posture of the digital camera (100) from the posture detection unit (55).

[0087] In step S404, the system control unit (50) determines the camera to be used based on the recording range and the change in posture from the start of shooting. Details will be described later.

[0088] In step S405, the system control unit (50) executes one frame of shooting using an effective camera. Since only the rear camera (120) is effective at the start of shooting, the system control unit (50) executes one frame of shooting using only the rear camera (120). The image processing unit (24) generates image data for recording and image data for display for one frame read from the effective camera and stores this in the memory (32). At this time, the operation regarding the determination of shooting conditions and the driving of the focus lens and image element during shooting is known, so the explanation is omitted.

[0089] In step S406, the system control unit (50) determines whether multiple cameras are valid. This determination corresponds to the determination of whether image synthesis is required. The digital camera (100) according to the present embodiment has a front camera (110) and a rear camera (120). If both cameras are valid, the system control unit (50) executes step S407, and if only one camera is valid, the system control unit (50) skips step S407 and executes step S408.

[0090] In step S407, the system control unit (50) instructs the image processing unit (24) to combine (stitch) image data captured by a valid camera. In response to the instruction, the image processing unit (24) combines (stitches) the image data of the current frame stored in the memory (32) to generate composite image data having a continuous shooting range. At this time, the image processing unit (24) already knows the positional relationship in which the images obtained by each camera will be stitched. The image processing unit (24) stores the composite image data in the memory (32). When the system control unit (50) generates the composite image data, step S408 is executed.

[0091] In step S408, the system control unit (50) instructs the image processing unit (24) to cut the image data of the recording range. The system control unit (50) reads, for example, the shooting start position acquired in step S400, the position acquired in step S403, and the recording range determined in step S401 from the system memory (52), and transmits this and the cutting instruction to the image processing unit (24). At this time, instead of the position information, the position corresponding to the shooting direction at the start of shooting, which was obtained when determining the camera to be effective in step S404, may be transmitted to the image processing unit (24).

[0092] The image processing unit (24) calculates the position of the recording range in the image obtained at the current position based on the change in the position of the digital camera (100) from the start of shooting to the present. Then, the image processing unit (24) obtains image data of the same shooting range as the recording range at the start of shooting by cutting out the recording range from the image data, and stores this in the memory (32). At this time, known techniques can be used for distortion correction, etc., necessary to make the area cut out from the circular fisheye image into a normal rectangular image.

[0093] At this time, the image processing unit (24) performs the composite processing of step S407 and the cropping processing of step S408 on both the image data for recording and the image data for display. Additionally, the image processing unit (24) may perform distortion correction, etc., necessary to make the area cropped from the circular fisheye image into a normal rectangular image.

[0094] Among the image data of the recording range obtained by cutting by the image processing unit (24), the recording image data is recorded on the recording medium (90) as wide-angle image data generated in crop mode by the system control unit (50). Additionally, regarding the display image data, the system control unit (50) performs guide display processing in step S409 before outputting it to the display unit (28) or an external device. At this time, multiple frames may be combined and recorded on the recording medium (90) according to the encoding method, etc.

[0095] In step S409, the system control unit (50) performs guide display processing. Details will be described later.

[0096] In step S410, the system control unit (50) determines whether or not a shooting end operation has been performed. If it is determined that a shooting end operation has been performed, the operation regarding video recording is terminated, and if it is not determined that, the processing of the next frame is executed from step S403. The shooting end operation may be the operation of a video recording switch or a recording end instruction received from an external device via the communication unit (54).

[0097] Next, the operation of the system control unit (50) in step S404 will be explained using the flowchart shown in FIG. 5.

[0098] In step S500, the system control unit (50) calculates the difference between the reference position of the digital camera (100) (in this example, the position at the start of shooting) and the current position of the digital camera (100) acquired in step S403. Then, based on the difference in position, the system control unit (50) calculates the position of the recording range in the image obtained in the current position.

[0099] Furthermore, the system control unit (50) determines whether the recording range corresponding to the current posture is larger than the shooting range of the currently valid camera. The system control unit (50) detects cases where the recording range (307) exceeds the shooting range (305) of the valid rear camera (120) as shown in the example of FIG. 3b (having a range not included in the shooting range (305)). If the system control unit (50) determines that the recording range corresponding to the current posture exceeds the shooting range of the currently valid camera, step S501 is performed, and if this is not determined, step S503 is executed.

[0100] In step S501, the system control unit (50) enables a camera among the currently disabled cameras that includes a recording range corresponding to the current digital camera (100) in the shooting range, and terminates the processing of step S404. For example, in the example shown in FIG. 3b, the front camera (110) is enabled.

[0101] Steps S503 to S508 correspond to processing performed by the system control unit (50) on the camera of the currently valid digital camera (100).

[0102] In step S504, the system control unit (50) obtains the shooting range.

[0103] In step S505, the system control unit (50) determines whether the shooting range acquired in step S504 includes at least a portion of the recording range corresponding to the current position of the digital camera (100) calculated in step S500. If the system control unit (50) determines that the shooting range includes at least a portion of the recording range, step S506 is executed, and if it is not determined that way, step S507 is executed.

[0104] In step S506, the system control unit (50) keeps the target camera active and terminates the processing of step S404.

[0105] In step S507, the system control unit (50) disables the target camera and terminates the processing of step S404. Disabling is a state in which power consumption is lower than when in an active state, and power supply to at least some parts may continue. For example, the image element and the A / D converter may be put into a power-saving state. This can be said to be a transition of the circuit regarding the disabled shooting lens to a state in which power consumption is lower than when the lens is enabled.

[0106] In this way, cameras necessary to capture the recording range corresponding to the current posture (i.e., the shooting range required for image generation) are enabled, and cameras unnecessary to capture the recording range corresponding to the current posture are disabled. Accordingly, only the cameras necessary for the change in the posture of the digital camera (100) are dynamically enabled, thereby reducing power consumption.

[0107] Next, the processing of step S409 is described in detail using FIGS. 6 and FIGS. 7. In this example, for convenience, the digital camera (100) operates in a linkage mode with the smartphone (200), and image data for live view display is transmitted to the smartphone (200) via the communication unit (54). Additionally, on the smartphone (200), an application for linkage with the digital camera (100) is executed, and a display based on the image data for live view display received from the digital camera (100) is performed on the display (205). Additionally, the user is in a state where they can hold the digital camera (100) and view the display (205) of the smartphone (200). At this time, if it is possible to view the live view display through the display unit (28) of the digital camera (100), the guide display operation described below may be performed by the digital camera (100) alone.

[0108] FIG. 6 is a diagram showing a specific example of a guide display performed in step S409. A live view image (600) is displayed on a display (205). The guide display is a predetermined image, such as an indicator or an icon, that is superimposed on the live view image when multiple cameras are enabled. In this example, the guide display includes a boundary line (601), a warning of the number of cameras in use (602), and a recommended direction indicator (603).

[0109] The boundary line (601) indicated by a dotted line represents the boundary of the camera's shooting range and corresponds to the boundary line (300) in FIGS. 3a and FIG. 3b. In order for the boundary line (601) to be displayed, multiple cameras may be required to capture the recording range, as shown in FIG. 3b. If the entire recording range can be captured by a single camera, as shown in FIG. 3a, the boundary line (601) is not displayed.

[0110] The camera usage warning (602) is an icon indicating that multiple cameras are being used to capture the recording range. Both the boundary line (601) and the camera usage warning (602) are displayed. At this time, other methods of expression, such as text strings or line displays, may be used instead of icons.

[0111] The recommended direction indicator (603) is an icon indicating how to change the posture of the digital camera (100) to reduce the number of cameras required to capture the recording range. Looking at the examples shown in FIGS. 3a and 3b, the digital camera (100) can be changed to a posture close to upright to change the posture of FIG. 3b to the posture of FIG. 3a. In this case, the recommended direction indicator (603) may be an icon indicating the upright direction (the direction in which the digital camera (100) is set up). More specifically, the current posture and the recommended posture may be displayed alternately, or an animation may be used when displaying them.

[0112] When the requirements for a guide display are met, the digital camera (100) (system control unit (50)) uses an image processing unit (24) to generate a live view display image with the guide display superimposed.

[0113] Next, the operation of the system control unit (50) in step S409 will be explained using the flowchart shown in FIG. 7.

[0114] In step S700, the system control unit (50) determines whether the plurality of cameras are valid or not. If it is determined that the plurality of cameras are valid, step S704 is executed, and if it is not determined that they are valid, step S701 is executed.

[0115] In step S701, the system control unit (50) is configured not to display the camera usage count warning (602).

[0116] In step S702, the system control unit (50) is set so that the boundary line (601) is not displayed.

[0117] In step S703, the system control unit (50) sets the recommended direction indicator (603) not to be displayed. The system control unit (50) notifies the image processing unit (24) of the settings in steps S701 to S703.

[0118] At this time, you may set it so that the entire guide display (boundary line (601), warning of number of cameras in use (602), and recommended direction display (603)) is not displayed together.

[0119] In step S704, the system control unit (50) sets the display of the number of cameras in use warning (602).

[0120] In step S705, the system control unit (50) calculates the difference in angle between the direction of the optical axis and the shooting direction for each valid camera, and sets the camera with the smallest difference in angle as the main camera.

[0121] In step S706, the system control unit (50) sets the indication of the boundary line (601) of the shooting range of the main camera.

[0122] In step S707, the system control unit (50) calculates the direction in which the digital camera (100) should be directed to reduce the difference in angle between the optical axis direction of the main camera and the shooting direction.

[0123] In step S708, the system control unit (50) sets the display of the recommended direction indicator (603). The system control unit (50) notifies the image processing unit (24) of the settings in steps S704, S706, and S708 and the direction calculated in step S707.

[0124] In step S709, the image processing unit (24) generates image data for live view display based on a notification from the system control unit (50).

[0125] When steps S701 to S703 are executed, the image processing unit (24) generates image data for a normal live view display that does not overlap guide displays.

[0126] Meanwhile, when steps S704 to S708 are executed, the image processing unit (24) generates image data for a live view display with superimposed guide markings. At this time, for the guide markings, the superimposed position of the camera count warning (602) and the recommended direction marking (603), and the image of the recommended direction marking (603) corresponding to the notified direction are predetermined. Additionally, regarding the image data for the live view display, the image processing unit (24) superimposes a boundary line (601) at a position corresponding to the boundary of the plurality of images used to generate the composite image in step S407.

[0127] Then, the image processing unit (24) stores the generated image data for live view display in the memory (32). The system control unit (50) transmits the image data for live view display to the smartphone (200) via the communication unit (54). Additionally, the system control unit (50) can also output the image data for live view display to the display unit (28).

[0128] As described above, according to the present embodiment, in an imaging device capable of generating omnidirectional images using a plurality of imaging units, when generating an image with a smaller angle of view than the omnidirectional image, only the imaging units necessary for generating the image are made active. In addition, when image stabilization cannot be realized within the shooting range of the effective imaging units, the necessary imaging units are made active. Accordingly, power consumption can be reduced, and the range in which image stabilization can be performed can be increased.

[0129] Other embodiments

[0130] In the configuration of the above-described embodiment, one imaging unit uses one shooting lens. However, in the configuration of other similar embodiments, the number of shooting lenses used by one imaging unit may be changed. In this case, shooting using multiple lenses may be performed sequentially while switching lenses. Furthermore, in the above-described embodiment, the operation regarding the activation and deactivation of the camera has the same meaning as the activation and deactivation of the lens. A configuration that increases the number of lenses used according to a change in the position of the digital camera has a lower effect on reducing power consumption compared to a configuration that increases or decreases the number of imaging units used, but a similar effect is obtained regarding the increase in the range in which image stabilization can be performed.

[0131] Embodiments of the present invention may be implemented by a method performed by a computer of a system or device comprising one or more circuits (e.g., application-specific integrated circuits (ASICs)) that perform one or more functions of the aforementioned embodiment(s) of the present invention, by reading and executing computer-executable instructions (e.g., one or more programs) recorded in a storage medium (which may be referred to more specifically as a 'non-transient computer-readable storage medium') to perform one or more functions of the aforementioned embodiment(s), or by reading and executing computer-executable instructions from a storage medium to perform one or more functions of the aforementioned embodiment(s). The computer may have one or more central processing units (CPUs), microprocessors (MPUs), or other circuits, and may have a network of separate computers or separate computer processors. Computer-executable instructions may be given to the computer, for example, from a network of storage media. The storage medium may be, for example, one or more hard disks, random access memory (RAM), read-only memory (ROM), or a distributed computing system. Storage, optical disc (Compact Disc (CD), Digital Multifunction Disc (DVD), or Blu-ray Disc (BD)TM You may also provide flash memory devices, memory cards, etc.), etc.

[0132] The present invention can be executed in a process in which a program realizing one or more functions of the above-described embodiment is supplied to a system or device via a network or storage medium, and one or more processors in a computer of the system or device read and execute the program. In addition, it can also be executed by a circuit (e.g., an ASIC) realizing one or more functions.

[0133] Although the present invention has been described with reference to exemplary embodiments, it is obvious that the invention is not limited to these embodiments. The scope of protection of the following claims shall be interpreted as broadly as possible to encompass all variations, equivalent structures, and functions.

Claims

Claim 1 An imaging device that generates an omnidirectional image using a plurality of imaging units each comprising a plurality of shooting lenses having different shooting directions, the device comprising: a detection means for detecting the orientation of the imaging device; a specification means for specifying a reference orientation, which is a constant direction not affected by subsequent changes in the orientation of the imaging device, when the imaging device is in an operation mode that generates a first image having a smaller angle of view than the omnidirectional image, and specifying a shooting range required for generating the first image based on the difference between the reference orientation and the current orientation of the imaging device detected by the detection means; a control means for transitioning the first imaging unit among the plurality of imaging units to a first state capable of image processing when the imaging device is in the operation mode if the shooting range of the first imaging unit includes at least a part of the first range, and transitioning the second imaging unit among the plurality of imaging units to a power-saving state with less power consumption than the first state when the shooting range of the second imaging unit does not include any part of the first range; and the number of shooting lenses to be used for shooting. An imaging device having a generating means for generating a guide mark including a mark indicating how to change the posture of the imaging device to reduce. Claim 2 In claim 1, the generating means is an imaging device that generates the first image by cutting out an area of ​​the shooting range from an image obtained using a shooting lens among the plurality of shooting lenses, the shooting of which has been enabled by the control means. Claim 3 In claim 2, when shooting using each of the plurality of shooting lenses is enabled by the control means, the imaging device further comprises a compositing means for stitching images shot using the plurality of shooting lenses to generate a composite image, and the generating means for cutting out an area of ​​the shooting range from the composite image to generate the first image. Claim 4 In claim 3, when shooting using each of the plurality of shooting lenses is enabled by the control means, the generating means is an imaging device that generates the first image having a predetermined guide mark superimposed. Claim 5 An imaging device according to claim 4, wherein the guide mark further comprises a mark indicating the boundary between a plurality of images used to generate the composite image or a mark indicating that shooting by the plurality of shooting lenses has been enabled. Claim 6 In any one of claims 2 to 5, the first image generated by the generating means is an imaging device that outputs to a display device provided in the imaging device or to an external device having a display device. Claim 7 An imaging device according to any one of claims 1 to 5, wherein the specifying means specifies the shooting range by converting the position of the shooting range when the imaging device is in the reference position to a position corresponding to the current position of the imaging device. Claim 8 In any one of claims 1 to 5, the control means is an imaging device that transitions a circuit related to a shooting lens to be used when shooting is invalidated to a state with less power consumption than when shooting using the shooting lens is enabled. Claim 9 In claim 8, the above circuit comprises an imaging device including an imaging element. Claim 10 An imaging device according to any one of claims 1 to 5, wherein the specification performed by the specification means and the activation and deactivation performed by the control means are repeatedly executed. Claim 11 An imaging device according to any one of claims 1 to 5, wherein the plurality of shooting lenses are circular fisheye lenses. Claim 12 A control method for an imaging device capable of generating omnidirectional images using a plurality of imaging units each comprising a plurality of imaging lenses having different shooting directions, comprising: a step of detecting the attitude of the imaging device; a step of, when the imaging device is in an operation mode that generates a first image having a smaller angle of view than an omnidirectional image, specifying a reference attitude which is a constant direction not affected by subsequent changes in the attitude of the imaging device, and specifying a shooting range required for generating the first image based on the difference between the reference attitude and the current attitude of the imaging device detected in the detection step; a step of, when the imaging device is in the operation mode, if the shooting range of the first imaging unit includes at least a part of the first range, transitioning the first imaging unit among the plurality of imaging units to a first state capable of image processing, and if the shooting range of the second imaging unit does not include any part of the first range, transitioning the second imaging unit among the plurality of imaging units to a power-saving state that consumes less power than the first state; and the number of imaging lenses to be used for shooting A method for controlling an imaging device comprising the step of generating a guide mark including a mark indicating how to change the posture of the imaging device to reduce it. Claim 13 A computer-readable medium storing a program for executing the method described in claim 12 in a computer equipped with an imaging device capable of generating omnidirectional images using multiple shooting lenses.

Images