Imaging apparatus, control method thereof, and computer readable medium

Abstract

This invention provides a camera device, a control method thereof, and a computer-readable medium. The camera device is disclosed that uses a camera lens to generate an omnidirectional image. The device detects the pose of the camera device. When the camera device is in an operating mode for generating a first image with a smaller field of view compared to an omnidirectional image, the device identifies the range to be captured required to generate the first image based on the difference between a reference pose and the current pose of the camera device. While the camera device is in this operating mode, the device enables imaging using the camera lens required to capture the range to be captured, and disables imaging using the camera lens not required to capture the range to be captured.

Description

Technical Field

[0001] This invention relates to camera equipment and its control method, as well as computer-readable media, and more particularly to camera equipment and its control method capable of capturing omnidirectional images. Background Technology

[0002] Known camera devices can capture 360-degree panoramic images (see Japanese Patent Application Publication No. 2019-205161). Such camera devices are called 360-degree cameras or panoramic cameras.

[0003] For example, a 360-degree camera equipped with two circular fisheye lenses with a 180-degree field of view includes the ability to crop a rectangular area from an image taken using only one lens and generate an image similar to one taken using a wide-angle lens with a field of view of less than 180 degrees.

[0004] In this case, electronic image stabilization can be performed independently of the rotation angle for camera rotation about the optical axis (movement in the tilt direction). However, for camera rotation about an axis orthogonal to the optical axis (movement in the pitch and yaw directions), there are limitations on the rotation angle at which image stabilization can be performed. Summary of the Invention

[0005] According to embodiments of the present invention, a camera device and a control method thereof are provided that can use multiple lenses to generate omnidirectional images, the camera device and the control method thereof can enhance image stabilization when generating images with a smaller field of view compared to omnidirectional images.

[0006] According to an aspect of the present invention, a camera device is provided for generating an omnidirectional image using a plurality of camera lenses, the camera device comprising: a detection unit for detecting the pose of the camera device; an identification unit for identifying, when the camera device is in an operating mode for generating a first image having a smaller field of view compared to an omnidirectional image, a range to be captured based on the difference between a reference pose of the camera device and a current pose detected by the detection unit; and a control unit for, when the camera device is in the operating mode, enabling the use of camera lenses among the plurality of camera lenses required to capture the range to be captured, and disabling the use of camera lenses among the plurality of camera lenses not required to capture the range to be captured.

[0007] According to an aspect of the present invention, a control method for a camera device is provided, the camera device being capable of generating an omnidirectional image using a plurality of camera lenses, the control method comprising: detecting the pose of the camera device; when the camera device is in an operating mode for generating a first image having a smaller field of view compared to an omnidirectional image, identifying a shooting range required to generate the first image based on a difference between a reference pose of the camera device and a current pose detected in the detection; and when the camera device is in the operating mode, enabling imaging using the camera lens among the plurality of camera lenses required to capture the shooting range, and disabling imaging using the camera lens among the plurality of camera lenses not required to capture the shooting range.

[0008] According to an aspect of the invention, a computer-readable medium storing a program is provided for causing a computer included in a camera device capable of generating omnidirectional images using multiple camera lenses to perform the method according to the invention.

[0009] Further features of the invention will become apparent from the following description of exemplary embodiments (with reference to the accompanying drawings). Attached Figure Description

[0010] Figures 1A to 1C This is a diagram related to an example digital camera representing a camera device, according to an embodiment.

[0011] Figure 2A and Figure 2B This is a diagram related to an example smartphone representing an electronic device, according to an embodiment.

[0012] Figure 3A and Figure 3B This is a schematic diagram used to describe the operation control in the embodiments.

[0013] Figure 4 This is a flowchart related to the operation of a digital camera according to an embodiment.

[0014] Figure 5 This is a flowchart related to the operation of a digital camera according to an embodiment.

[0015] Figure 6 This is a diagram illustrating an example of a wizard displayed according to an embodiment.

[0016] Figure 7 This is a flowchart related to the operation of a digital camera according to an embodiment. Detailed Implementation

[0017] In the following, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. Several features are described in the embodiments, but the invention is not limited to requiring all such features, and multiple such features can be appropriately combined. Furthermore, in the drawings, the same reference numerals are given the same or similar configuration, and redundant descriptions thereof are omitted.

[0018] Note that in the embodiments described below, the invention is embodied as a camera device such as a digital camera. However, the invention can be implemented using any electronic device with camera functionality. Examples of such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, and PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and dashcams. These electronic devices are examples, and the invention can be implemented using other electronic devices.

[0019] Figure 1A and Figure 1B This is a perspective view illustrating the appearance of a digital camera 100, representing an example of an imaging device according to an embodiment of the present invention. In this document, the side where the shutter button 61 is located corresponds to the front of the digital camera 100. Therefore, in Figure 1A The example configuration is shown on the front side, and... Figure 1B The back side is shown in the image. Furthermore, Figure 1C This is a block diagram illustrating an example of the functional configuration of a digital camera 100.

[0020] In the following text, we will use Figures 1A to 1C An exemplary configuration of a digital camera 100 is described. The digital camera 100 includes camera lenses 103a and 103b, respectively, on its front and rear sides, which are circular fisheye lenses with a 180-degree field of view. Camera lenses 103a and 103b, for example, include the same optical axis and are configured to have shooting directions that differ from each other by 180 degrees. In this specification, the shooting direction of camera lens 103a, located on the front side of the digital camera 100, is referred to as the front, and the shooting direction of camera lens 103b, located on the rear side of the digital camera 100, is referred to as the rear.

[0021] The digital camera 100 is an omnidirectional camera or a 360-degree camera that can generate an omnidirectional image with a 360-degree horizontal field of view by combining or stitching together the images formed by the camera lenses 103a and 103b. Note that the omnidirectional image can be a 360-degree image, and can also have a vertical field of view of less than 360 degrees, such as a 180-degree image. In this embodiment, the two camera lenses 103a and 103b are circular fisheye lenses with a horizontal field of view of more than 180 degrees, and the digital camera 100 can generate a 360-degree image.

[0022] Note that in this example, two camera lenses 103a and 103b cover a horizontal field of view of more than 360 degrees, but a configuration with three or more camera lenses having different shooting directions can be used to cover a horizontal field of view of more than 360 degrees.

[0023] The digital camera 100 includes multiple camera units having a front camera 110 for capturing images of the front and a rear camera 120 for capturing images of the rear. The front camera 110 includes a camera lens 103a and a baffle 102a (protective member) disposed on the front of the camera lens 103a. For example, the baffle 102a is a transparent member having a hemispherical shape. The camera lens 103a includes a focusing lens that can move in the optical axis direction. The shutter 101a also serves as an aperture. The camera lens 103a forms an optical image on the imaging surface of the image sensor 22a.

[0024] For example, image sensor 22a can be a known CCD or CMOS color image sensor including a primary color Bayer array color filter. Image sensor 22a includes a pixel array having multiple pixels in a two-dimensional array and peripheral circuitry for reading signals from the pixels. Pixels accumulate charge corresponding to incident light via photoelectric conversion. By reading signals from each pixel, including voltages corresponding to the amount of charge accumulated during the exposure period, a group of pixel signals (analog image signals) representing the optical image formed on the imaging surface is obtained. A / D converter 23a converts the analog image signals read from image sensor 22a into digital image signals (image data).

[0025] Furthermore, the front camera 110 includes a mechanism for moving the focusing lens, a mechanism for detecting the position of the focusing lens, and a mechanism for driving the shutter 101a. Note that the front camera 110 may include a moving mechanism for the image sensor 22a to implement optical blur correction and / or a moving mechanism for the shift lens included in the camera lens 103a.

[0026] The rear camera 120 has the same configuration as the front camera 110, therefore the description of its components is omitted. Note that in the following text, image sensors 22a and 22b are collectively referred to as image sensor 22. This also applies to other components included in the front camera 110 and the rear camera 120. In this embodiment, since a configuration of one camera unit using one camera lens is used, selectively using a camera unit and selectively using a camera lens have the same meaning. Therefore, descriptions related to enabling and disabling the camera unit (camera) can be read as enabling and disabling imaging via the camera lens. However, the invention can be applied to configurations where one camera unit uses multiple camera lenses simultaneously or switches between using multiple camera lenses.

[0027] Image data output from A / D converter 23 is written to memory 32 via either the image processing unit 24 or the memory control unit 15, or only via the memory control unit 15.

[0028] The memory 32 is used as a buffer for image data, a working memory for the image processing unit 24, and a video memory for the display unit 28.

[0029] The image processing unit 24 applies predetermined image processing to image data output by the A / D converter 23 or the memory control unit 15, or stored in the memory 32, and acquires or generates signals, image data, and various types of information corresponding to the intended use. The image processing unit 24 may be, for example, dedicated hardware such as an application-specific integrated circuit (ASIC) configured to perform a specific function. Alternatively, the image processing unit 24 may be configured to execute software via a processor such as a digital signal processor (DSP) or a graphics processing unit (GPU) to perform a specific function.

[0030] Image processing applied to image data by the image processing unit 24 includes, for example, preprocessing, color interpolation processing, correction processing, detection processing, data modification processing, evaluation value calculation processing, and special effects processing.

[0031] Preprocessing may include signal amplification, reference level adjustment, and defective pixel correction.

[0032] Color interpolation is a process performed when the image sensor 22 is equipped with a color filter, used to interpolate the values ​​of color components that are not included in the pixel data that forms the image data. Color interpolation is also known as demosaicing.

[0033] The correction process may include various processes such as white balance adjustment, tone correction, correction (image restoration) of image degradation caused by optical aberrations of camera lenses 103a and 103b, and color correction. The correction process also includes processes for correcting significant peripheral distortions in ultra-wide-angle lenses such as circular fisheye lenses.

[0034] The detection process may include processing for detecting feature regions (e.g., facial regions or human body regions) or their movement, as well as processing for identifying people.

[0035] Data modification processing may include processes such as region cropping, compositing, scaling, encoding and decoding, and header information generation (data file generation). Data modification processing also includes generating image data for display and image data for recording.

[0036] The evaluation value calculation process may include the following processes, wherein the process includes: generating a signal or evaluation value used in autofocus detection (AF); generating an evaluation value used in automatic exposure control (AE); and so on.

[0037] Special effects processing can include adding blur effects, changing color tones, and relighting.

[0038] Note that these are examples of processing that can be applied by the image processing unit 24, and are not intended to limit the processing that can be applied by the image processing unit 24. Furthermore, image processing that can be applied by the image processing unit 24 can be performed by the system control unit 50.

[0039] In this embodiment, the image processing unit 24 stitches together two circular fisheye images captured by the front camera 110 and the rear camera 120 to generate a 360-degree image. The generated 360-degree image is then transformed into an image using an equidistant cylindrical transformation, and the position of the pixels can be correlated with the coordinates of the 360-degree surface.

[0040] In addition, the image processing unit 24 crops a region from the data of the circular fisheye image, applies distortion correction, etc., and generates image data for real-time viewing display in a rectangular shape suitable for the display unit 28 or an external display device.

[0041] The system control unit 50 is, for example, a processor (CPU, MPU, or microprocessor, etc.) capable of executing programs. The system control unit 50 controls the operation of the function blocks of the digital camera 100 and implements the functions of the digital camera 100 by loading the program stored in the non-volatile memory 56 into the system memory 52 and executing the program.

[0042] The non-volatile memory 56 is electrically rewritable and stores programs executed by the system control unit 50, various settings of the digital camera 100, and GUI data. When the system control unit 50 executes a program, the system memory 52 is used as main memory. Note that memory 32 and system memory 52 can be different areas within a contiguous storage space.

[0043] The shutter button 61 includes a switch SW1 62 that is activated in a half-pressed state and a switch SW2 64 that is activated in a fully pressed state. The system control unit 50 recognizes the activation (ON) of switch SW1 62 as a still image capture preparation indication and the activation of switch SW2 64 as a still image capture start indication. When the activation of switch SW1 62 is detected, the system control unit 50 uses signals or evaluation values ​​generated by the image processing unit 24 and performs autofocus detection (AF) and auto exposure control (AE) for the front camera 110 and / or the rear camera 120. Furthermore, when the activation of switch SW2 64 is detected, the system control unit 50 controls the shutter 101 according to the exposure conditions determined by the AE processing, captures a still image, and performs recording processing. The still image data generated by the image processing unit 24 is temporarily stored in the memory 32 before being recorded by the system control unit 50 onto the recording medium 90 via I / F 18.

[0044] Note that the shutter button 61 may consist of only a switch. In this case, when the switch is detected to be on, the system control unit 50 continuously performs image preparation and image processing.

[0045] The operating mode of the digital camera 100 can be changed via the user operation mode change switch 60. Operating modes include, for example, a mode for capturing 360-degree images, a mode for capturing general wide-angle images with a horizontal field of view of less than 180 degrees (referred to as a cropping mode), a playback mode, and a mode for collaborative operation with external devices. Note that the mode for capturing wide-angle images can include multiple modes corresponding to the horizontal field of view (such as 90-degree mode and 150-degree mode).

[0046] Note that the operating mode can be selected through a combination of operating the mode change switch 60 and operating other operating components. For example, the mode change switch 60 can be used to select a broad category of operating modes, and then the operating mode can be selected from a more specific category displayed on the display unit 28.

[0047] The power switch 72 is used to indicate whether the power supply to the digital camera 100 is turned on or off. The power control unit 80 controls these operations.

[0048] The power control unit 80 includes a battery detection circuit, a DC-DC converter, and a switching circuit for switching between functional blocks supplying power. It controls the power supply from the power unit 30, which can be a battery or an AC adapter, to the components of the digital camera 100. The power control unit 80 detects the type of the power unit 30. Furthermore, when the power unit 30 is a battery, the power control unit 80 detects the type and remaining charge. The power control unit 80 can change the components used for power supply and the supplied power based on the state of the power switch 72 and the control of the system control unit 50.

[0049] The microphone 20 is pointed outside the digital camera 100 and outputs audio signals to the system control unit 50. When recording moving images, it records image data and audio data.

[0050] The operation unit 70 is a general term for all input device components (buttons, switches, and dials, etc.) excluding the mode change switch 60, shutter button 61, and power switch 72. The operation unit 70 includes a motion image recording switch, menu button, directional keys, and enter key, etc. When the display unit 28 is a touch screen, the operation unit 70 can also be composed of touch-operable software buttons or keys.

[0051] For example, display unit 28 is a liquid crystal display (LCD). Display unit 28 may not be a touch display. Display unit 28 displays characters and images. By immediately displaying continuously captured moving images on display unit 28, display unit 28 can be used as an electronic viewfinder (EVF). The moving images displayed using a display device used as an EVF are called live view images. Note that by outputting live view images to an external device connected to communication unit 54, the display device included in the external device can be used as an EVF.

[0052] The light-emitting unit 21 is a light-emitting diode (LED) used to notify the user of the status of the digital camera 100, etc., by means of the pattern or color of the emitted light.

[0053] The fixing part 40 is set on the bottom surface of the digital camera 100. Figure 1B For example, a threaded hole for attaching a tripod.

[0054] The system timer 53 responds to a request from the system control unit 50 by outputting the time of the internal clock and measuring the time.

[0055] Communication unit 54 is an interface for wired or wireless communication with external devices such as electronic devices or external display devices described below. Communication unit 54 is compatible with one or more wired or wireless communication standards and includes connectors and transceivers suitable for those standards. Representative standards that communication unit 54 may support include, but are not limited to, USB, HDMI (registered trademark), Bluetooth (registered trademark), and wireless LAN.

[0056] The posture detection unit 55 includes, for example, a gyroscope sensor and an accelerometer sensor, and outputs signals representing the posture and movement of the digital camera 100 to the system control unit 50. The posture of the digital camera 100 is represented by rotation angles (tilt, pitch, and yaw) about angular axes, where the x-axis is parallel to the optical axis, the y-axis extends horizontally, and the z-axis extends vertically. The posture of the digital camera 100 during image capture can be correlated with and recorded with image data. Furthermore, the detected posture and movement of the digital camera 100 can be used for image stabilization and tilt correction.

[0057] I / F 18 is an interface for writing data to a recording medium 90, such as a memory card or hard drive, and for reading data recorded on the recording medium 90. The recording medium 90 may or may not be removable from the digital camera 100.

[0058] Figure 2A and Figure 2B This is a diagram of a smartphone 200, which is an example of an external device that can cooperate with a digital camera 100 via communication using a communication unit 54. Figure 2A It is a perspective view showing an example of the appearance, and Figure 2B This is a block diagram illustrating an example of the functional configuration. Note that the external device is not limited to a smartphone, and it is sufficient for the electronic device to communicate with the digital camera 100 and include a processor capable of executing applications for working with the digital camera 100.

[0059] The configuration of the Smart Phone 200 will now be described.

[0060] The internal bus 250 connects the various blocks to enable data exchange in both directions.

[0061] CPU 201 is a processor capable of executing programs. It controls the operation of function blocks of smartphone 200 and implements the functions of smartphone 200 by loading the program stored in non-volatile memory 203 onto memory 202 and executing the program.

[0062] When CPU 201 executes a program, memory 202 is used as main memory. A portion of memory 202 is used as video memory for display 205.

[0063] The non-volatile memory 203 is electrically rewritable and stores programs (OS and applications) executed by the CPU 201, settings of the smartphone 200, GUI data, and user data.

[0064] Display 205 is, for example, an LCD, and is where the OS and applications display images and various types of information. Display 205 is a touch display that includes a touch panel 206a, and can detect touch operations on the display surface of display 205. Display 205 can be an external device.

[0065] The image processing unit 24, under the control of the CPU 201, applies image processing to image data stored in the non-volatile memory 203 and the recording medium 208, image data obtained via the external I / F 209, and image data obtained via the communication I / F 210.

[0066] Image processing applied to image data by image processing unit 24 can be similar to that of image processing unit 24 in digital camera 100. When smartphone 200 does not include a camera, the generation of evaluation values ​​used in AF and AE may not be performed. Furthermore, image processing that image processing unit 24 can perform can be executed by CPU 201.

[0067] The image processing unit 24, under the control of the CPU 201, generates image data for virtual reality (VR) display corresponding to the movement of the smartphone 200 from an ultra-wide-angle image (e.g., an image with a horizontal field of view greater than 180 degrees) such as a 360-degree view. VR display is achieved by cropping a region from the ultra-wide-angle image corresponding to changes in the smartphone 200's posture and displaying the generated cropped image data on the display 205. Using VR display, for example, by using goggles or the like to fix the smartphone 200's display 205 in front of the user's eyes, the displayed image changes, thereby tracking the user's head movement. In this way, the user can experience the feeling of being within a virtual space represented by the ultra-wide-angle image.

[0068] Alternatively, electronic image stabilization can be achieved by controlling the image processing unit 24 to crop the same area to be captured from the ultra-wide-angle image, independent of the posture of the smartphone 200.

[0069] The operation unit 206 is a general term for the input device assembly that a user can use to give instructions to the smartphone 200. Input devices typically included in the smartphone 200 include, but are not limited to, buttons, switches, and touch panels. Furthermore, the operation unit 206 may consist of a keypad and a mouse communicatively connected to the smartphone 200. Note that in Figure 2B In the image, touch panel 206a is shown as separate from display 205. However, touch panel 206a is actually a display screen that is either built into or attached to display 205.

[0070] Power button 206b, volume buttons 206c and 206d, and home button 206e are examples of input devices constituting operation unit 206. Power button 206b toggles the power on and off of smartphone 200. Volume buttons 206c and 206d are used to increase or decrease the volume output from audio output unit 212. Home button 206e is used to display a specific screen provided by the OS on display 205.

[0071] Media I / F 207 is an interface for accessing recording medium 208. For example, when recording medium 208 is a removable medium such as a memory card, media I / F 207 includes a slot for inserting / removing recording medium 208. CPU 201 can write data to and read data from recording medium 208 via media I / F 207.

[0072] External I / F 209 is an interface for wired or wireless communication with external devices such as digital camera 100 or external display devices. External I / F 209 is compatible with one or more wired or wireless communication standards and includes connectors and transceivers suitable for those standards. Representative standards that communication unit 54 may support include, but are not limited to, USB, HDMI (registered trademark), Bluetooth (registered trademark), and wireless LAN.

[0073] Communication I / F 210 is an interface for communication via cellular network 211. For example, communication I / F 210 may be a communication I / F conforming to mobile communication standards established by 3GPP, such as a 3G, 4G, or 5G modem.

[0074] The audio output unit 212 outputs audio (audio based on motion graphics or music data, operation sounds, ringtones, and various types of notification sounds, etc.). The audio output unit 212 includes an audio output terminal 212a that can be connected to headphones or the like, and a speaker 212b, but can also output audio to external devices via an external I / F.

[0075] The posture detection unit 213 includes, for example, a gyroscope sensor and an accelerometer, and outputs signals to the CPU 201 indicating the posture and movement of the smartphone 200. The posture of the smartphone 200 is represented by rotation angles (tilt, pitch, and yaw) about angular axes, where the x-axis is perpendicular to the display screen of the display 205, the y-axis extends horizontally, and the z-axis extends vertically. The posture detected by the posture detection unit 213 can be used for the aforementioned VR display.

[0076] Figure 3A and Figure 3B The operation control of the front camera 110 and rear camera 120 corresponding to the posture is schematically shown when the digital camera 100 is operating in an operation mode (crop mode) for capturing still images with a horizontal field of view of less than 180 degrees.

[0077] Figure 3A An example of the pose (reference pose) of the digital camera 100 at the start of recording is shown, and Figure 3B An example of the posture of the digital camera 100 during video recording is shown. Figure 3A In the position shown (upright position), tilt, pitch, and yaw are all at 0 degrees. Furthermore, Figure 3B Showing from Figure 3A The state is such that, with the rear camera facing downwards at 120 degrees, only the pitch (rotation angle around the y-axis) is changed.

[0078] In cropping mode, an image for recording or display is generated by cropping a portion of the image from a full-angle or ultra-wide-angle image. When the horizontal field of view of the image generated in cropping mode might be covered by one of the camera lenses, it is not necessary to use other lenses for recording. This allows for the use of only the necessary camera lenses and associated circuitry.

[0079] For example, if the horizontal field of view of the image generated by the digital camera 100 in cropping mode according to this embodiment is less than 180 degrees, power consumption can be reduced by making only one of the front camera 110 and the rear camera 120 active. In this example, the rear camera 120 is used when recording begins, so that the display unit 28 can be viewed when recording begins. Furthermore, an image with a horizontal field of view of 150 degrees and a vertical field of view of 90 degrees is generated.

[0080] Furthermore, electronic image stabilization is performed by changing the cropping position of the image, ensuring that the shooting range remains the same as at the start of recording, even if the pose of the digital camera 100 changes from a reference pose (e.g., the pose at the start of recording). In the following text, for convenience, the area cropped from the shooting range in cropping mode is referred to as the recording range. Note that the use of the image data generated in cropping mode is not limited to recording.

[0081] The digital camera 100 is in an upright position when recording begins (reference position). In this case, arrow 306 indicates the shooting direction (optical axis direction of camera lenses 103a and 103b). Therefore, within the shooting range 305 (right semicircle) of the rear camera 120, the gray area 307 corresponds to the vertical recording range. Since the front camera 110 is disabled, its shooting range 303 (left semicircle) is not captured. The direction of the boundary line 300 between the shooting range 303 of the front camera 110 and the shooting range 305 of the rear camera 120 is aligned with the direction of gravity 308.

[0082] When digital camera 100 from Figure 3A The posture shown has changed to Figure 3B In the pose shown, to achieve image stabilization, it is necessary to crop the same area as the recording range 307 at the start of recording. However, within the recording range 307, Figure 3B The shadow areas are not included in the field of view 305 of the rear camera 120, but are included in the field of view 303 of the front camera 110. Therefore, image stabilization cannot be achieved using only the image data obtained by the rear camera 120.

[0083] In this way, when image stabilization cannot be achieved using the shooting range of a single camera, the system control unit 50 activates another camera (in this example, the front camera 110) with a shooting range including the range required for image stabilization, and performs video recording using multiple cameras. Therefore, even when the digital camera 100 is changed to... Figure 3B When posing the image, the recording range can also be cropped by 307 to enable image stabilization.

[0084] On the other hand, when image stabilization can be achieved using the shooting range of one camera, power consumption can be reduced by disabling other cameras. Note that when the recording range is close to the boundary of the current shooting range, the camera capable of capturing the required range when the recording range exceeds the shooting range can be enabled. This helps avoid the inability to perform image stabilization while activating and acquiring images from additional enabled cameras.

[0085] Note that when using Figure 3A and Figure 3B In the example above, control is based on changes in attitude in the pitch direction. However, this also applies to changes in attitude in the yaw direction.

[0086] Will use Figure 4 The flowchart further describes the operation of the system control unit 50 in crop mode. These operations are performed when the digital camera 100 is operating in crop mode, image stabilization is enabled, and motion picture capture begins. Note that motion picture capture can be used for recording or for live view display, but in the example described below, motion picture capture is used for recording. Step S400 can be performed during recording by operating the motion picture switch in standby mode, or during live view display shooting by detecting the activation of image stabilization operation in standby mode, but is not limited to these methods.

[0087] Note that when enabling remote operation of an external device, step S400 can be performed by receiving a start recording instruction from the external device via communication unit 54.

[0088] In step S400, the system control unit 50 obtains the pose (reference pose) of the digital camera 100 from the pose detection unit 55. Based on the obtained pose, the system control unit 50 determines the optical axis direction of the rear camera 120 or the front camera 110 as the shooting direction. The shooting direction is a unidirectional direction that is not affected by subsequent changes in the pose of the digital camera 100. For example, the system control unit 50 stores the obtained pose and the determined shooting direction in the system memory 52.

[0089] In step S401, the system control unit 50 determines the recording range based on the shooting direction determined in step S400 and the settings of the digital camera 100. In this example, in cropping mode, the horizontal field of view is selectable, and the vertical field of view is constant or determined based on the horizontal field of view. In this example, a horizontal field of view of 150 degrees is selected, and a vertical field of view of 90 degrees is determined.

[0090] The system control unit 50 associates the range determined as the recording range based on the pose and shooting direction of the digital camera 100 at the start of recording with the pose of the digital camera 100 at the start of recording, and stores these ranges and poses in the system memory 52. ​​For example, the system control unit 50 may set the recording range as the area obtained by mapping a rectangular region centered on the shooting direction and having a size based on the horizontal and vertical fields of view onto the omnidirectional image according to the pose of the digital camera 100. Therefore, the recording range corresponds to a specific region in the omnidirectional image that the digital camera 100 can generate.

[0091] In step S402, the system control unit 50 starts the motion picture recording operation and continuously executes the processing after step S403 for each frame.

[0092] In step S403, the system control unit 50 obtains the posture of the digital camera 100 from the posture detection unit 55.

[0093] In step S404, the system control unit 50 determines which camera is valid based on the recording range and the changes in posture since the start of recording. This will be described in detail below.

[0094] In step S405, the system control unit 50 uses the active camera to perform a single frame of video recording. Since only the rear camera 120 is active when recording begins, the system control unit 50 uses only the rear camera 120 to perform a single frame of video recording. The image processing unit 24 generates recording image data and display image data for the single frame read from the active camera and stores this data in the memory 32. Note that the operations related to determining the recording conditions and driving the focusing lens and image sensor when capturing images are known and therefore will not be described.

[0095] In step S406, the system control unit 50 determines whether to enable multiple cameras. This determination corresponds to the determination of whether to synthesize an image. The digital camera 100 according to this embodiment includes a front camera 110 and a rear camera 120. If both cameras are enabled, the system control unit 50 executes step S407. If only one camera is enabled, the system control unit 50 skips step S407 and executes step S408.

[0096] In step S407, the system control unit 50 instructs the image processing unit 24 to composite (stitch together) the effective image data captured by the camera. In response to this instruction, the image processing unit 24 composites (stitches together) the image data of the current frame stored in the memory 32, and generates composite image data with a continuous range to be captured. Note that the image processing unit 24 already knows the positional relationships that will stitch the images obtained by the camera together. The image processing unit 24 stores the composite image data in the memory 32. When the system control unit 50 generates the composite image data, step S408 is executed.

[0097] In step S408, the system control unit 50 instructs the image processing unit 24 to crop the image data of the recording area. For example, the system control unit 50 reads the posture obtained in step S400 at the start of recording, the posture obtained in step S403, and the recording area determined in step S401 from the system memory 52, and sends these postures, recording areas, and cropping instructions to the image processing unit 24. Note that instead of posture information, the position corresponding to the shooting direction at the start of recording, obtained in step S404 when determining which camera to enable, can be sent to the image processing unit 24.

[0098] The image processing unit 24 calculates the position of the recording area in the image acquired in the current posture based on the pose changes of the digital camera 100 from the start of recording to the present. Then, by cropping the recording area from the image data, the image processing unit 24 obtains image data with the same recording area as at the start of recording, and stores this image data in the memory 32. Note that known techniques can be used for desired distortion correction, etc., to make the area cropped from the circular fisheye image a regular rectangular image.

[0099] Note that the image processing unit 24 performs the compositing process of step S407 and the cropping process of step S408 on both the image data for recording and the image data for display. Furthermore, the image processing unit 24 can perform necessary distortion corrections, etc., to make the area cropped from the circular fisheye image a regular rectangular image.

[0100] In the recording range image data obtained by the image processing unit 24 through cropping, the system control unit 50 stores the recording image data as wide-angle image data generated in the cropping mode on the recording medium 90. Furthermore, before outputting the display image data to the display unit 28 or an external device, the system control unit 50 performs wizard display processing on the display image data in step S409. Note that, depending on the encoding method, data from multiple frames can be recorded uniformly on the recording medium 90.

[0101] In step S409, the system control unit 50 performs a wizard display process. This will be described in detail below.

[0102] In step S410, the system control unit 50 determines whether a recording end operation has been performed. If it is determined that a recording end operation has been performed, the operation related to motion picture recording ends; otherwise, processing of the next frame, starting from step S403, is executed. The recording end operation can be the operation of a motion picture recording switch or a recording end instruction received from an external device via the communication unit 54.

[0103] Next, we will use Figure 5 The flowchart shown describes the operation of the system control unit 50 in step S404.

[0104] In step S500, the system control unit 50 obtains the difference between the reference pose of the digital camera 100 (in this example, the pose at the start of recording) and the current pose of the digital camera 100 obtained in step S403. Then, the system control unit 50 calculates the position of the recording range in the image obtained with the current pose based on the pose difference.

[0105] In addition, the system control unit 50 determines whether the recording range corresponding to the current posture is greater than the currently effective shooting range of the camera. For example... Figure 3B As shown in the example, the system control unit 50 detects when the recording range 307 exceeds the effective shooting range 305 of the rear camera 120 (including ranges not included in the shooting range 305). When the system control unit 50 determines that the recording range corresponding to the current posture is greater than the currently effective shooting range of the camera, it executes step S501, and when it does not determine that the recording range corresponding to the current posture is greater than the currently effective shooting range of the camera, it executes step S503.

[0106] In step S501, the system control unit 50 enables the camera among the currently invalid cameras that has a recording range corresponding to the current pose of the digital camera 100, which is included in the shooting range, and the processing in step S404 ends. For example, in Figure 3B In the example shown, the front camera 110 is enabled.

[0107] Steps S503 to S508 correspond to the processing performed by the system control unit 50 for the currently active camera of the digital camera 100.

[0108] In step S504, the system control unit 50 obtains the shooting range.

[0109] In step S505, the system control unit 50 determines whether the field of view obtained in step S504 includes at least a portion of the recording range calculated in step S500 corresponding to the current pose of the digital camera 100. When the system control unit 50 determines that the field of view includes at least a portion of the recording range, it executes step S506; and when it does not determine that the field of view includes at least a portion of the recording range, it executes step S507.

[0110] In step S506, the system control unit 50 keeps the object camera active, and the processing in step S404 ends.

[0111] In step S507, the system control unit 50 disables the target camera, and the process in step S404 ends. Disabling it consumes less power compared to when it is active, and power supply to at least one component can continue. For example, the image sensor and A / D converter can be placed in a power-saving state. This can also be described as switching the circuitry associated with the disabled camera lens to a state that consumes less power compared to when the lens is active.

[0112] In this way, the camera required to capture the recording range corresponding to the current posture (in other words, the range to be captured for image generation) is enabled, while cameras not required to capture the recording range corresponding to the current posture are disabled. Therefore, by dynamically enabling only the necessary cameras in response to changes in the posture of the digital camera 100, power consumption can be reduced.

[0113] Next, we will use Figure 6 and Figure 7 The processing of step S409 is described in detail. For convenience, in this example, the digital camera 100 operates in a cooperative mode with the smartphone 200, and transmits live view image data to the smartphone 200 via the communication unit 54. Furthermore, using the smartphone 200, an application for cooperation with the digital camera 100 is executed, and a display based on the live view image data received from the digital camera 100 is shown on the display 205. Additionally, the user keeps the digital camera 100 in a state where they can see the display 205 of the smartphone 200. Note that if the live view can be seen via the display unit 28 of the digital camera 100, the guided display operation described below can be performed using only the digital camera 100.

[0114] Figure 6 This diagram illustrates a specific example of the wizard display performed in step S409. A live view image 600 is displayed on the monitor 205. The wizard display is a predetermined image, such as an indicator or icon, superimposed on the live view image when multiple cameras are active. In this example, the wizard display includes a boundary line 601, a warning 602 indicating the number of cameras in use, and a recommended orientation display 603.

[0115] The boundary line 601, represented by the dashed line, indicates the boundary of the camera's field of view, and is consistent with... Figure 3A and Figure 3B The boundary line in the middle corresponds to 300. For example... Figure 3B As shown, multiple cameras may be needed to capture the recording range for the boundary line 601 to be displayed. When one camera can capture, for example... Figure 3A When the entire record range is in the range, the boundary line 601 is not displayed.

[0116] Warning 602, indicating the number of cameras in use, is an icon indicating that multiple cameras are being used to capture the recording area. Both boundary line 601 and warning 602 are displayed. Note that other representations, such as using strings or lines instead of icons, can be used.

[0117] The recommended orientation display 603 is an icon indicating how to change the orientation of the digital camera 100 to reduce the number of cameras required for capturing a recording range. See also Figure 3A and Figure 3B The example shown is for the purpose of Figure 3B The posture changed to Figure 3A The digital camera 100 can be changed to a nearly upright position depending on the camera's orientation. In this case, the recommended orientation display 603 can be an icon indicating the upright orientation (the orientation used to make the digital camera 100 upright). More specifically, the current orientation and the recommended orientation can be displayed alternately, or animation can be used in the display.

[0118] When the wizard display requirements are met, the digital camera 100 (system control unit 50) uses the image processing unit 24 to generate a live view display image overlaid with the wizard display.

[0119] Next, we will use Figure 7 The flowchart shown describes the operation of the system control unit 50 in step S409.

[0120] In step S700, the system control unit 50 determines whether to enable multiple cameras. If it is determined that multiple cameras are enabled, step S704 is executed; otherwise, step S701 is executed.

[0121] In step S701, the system control unit 50 sets the warning 602 for the number of cameras in use to not be displayed.

[0122] In step S702, the system control unit 50 sets the boundary line 601 to not be displayed.

[0123] In step S703, the system control unit 50 sets the recommended orientation display 603 to be off. The system control unit 50 then notifies the image processing unit 24 of the settings from steps S701 to S703.

[0124] Note that all wizard displays (boundary line 601, number of cameras in use warning 602, and recommended direction display 603) can be uniformly set to not be displayed.

[0125] In step S704, the system control unit 50 sets the warning 602 for the number of cameras in use to be displayed.

[0126] In step S705, the system control unit 50 calculates the angle difference between the optical axis direction and the shooting direction for each valid camera, and sets the camera with the smallest angle difference as the main camera.

[0127] In step S706, the system control unit 50 sets the boundary line 601 of the main camera's shooting range to be displayed.

[0128] In step S707, the system control unit 50 calculates the direction for orienting the digital camera 100 to reduce the angular difference between the optical axis direction and the shooting direction of the main camera.

[0129] In step S708, the system control unit 50 sets the recommended direction display 603 to be displayed. The system control unit 50 notifies the image processing unit 24 of the settings in steps S704, S706, and S708, as well as the direction calculated in step S707.

[0130] In step S709, the image processing unit 24 generates image data for real-time view display based on a notification from the system control unit 50.

[0131] When steps S701 to S703 are executed, the image processing unit 24 generates image data for normal live view display without the need for overlaying the wizard display.

[0132] On the other hand, when steps S704 to S708 are executed, the image processing unit 24 generates image data for a live view display overlaid with a guide display. Note that for the guide display, the overlay positions of the warning 602 for the number of cameras in use and the recommended direction display 603, as well as the image of the recommended direction display 603 corresponding to the notified direction, are predetermined. Furthermore, regarding the image data for the live view display, the image processing unit 24 overlays the boundary line 601 at a position corresponding to the boundary of the plurality of images used to generate the composite image in step S407.

[0133] Then, the image processing unit 24 stores the generated live view display image data in the memory 32. The system control unit 50 transmits the live view display image data to the smartphone 200 via the communication unit 54. The system control unit 50 can also output the live view display image data to the display unit 28.

[0134] As described above, according to this embodiment, when using a camera device that can generate omnidirectional images using multiple camera units, only the camera units required for image generation are activated when generating an image with a smaller field of view compared to an omnidirectional image. Furthermore, when image stabilization cannot be achieved with the shooting range of the activated camera units, the required camera units are activated. Therefore, power consumption can be reduced, and the range capable of image stabilization can be increased.

[0135] Other embodiments

[0136] In the configuration of the above embodiments, a single camera unit uses a single camera lens. However, in other similar embodiments, the number of camera lenses used by a single camera unit can be varied. In this case, multiple lenses can be used sequentially while switching lenses. Furthermore, in the above embodiments, the operations related to enabling and disabling the camera are synonymous with enabling and disabling the lenses. Compared to a configuration that increases the number of camera units used, a configuration that increases the number of lenses used in response to changes in the pose of the digital camera achieves less power consumption reduction, but achieves a similar effect to the increase in the range of image stabilization that can be performed.

[0137] Other embodiments

[0138] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0139] Although the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be accorded the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An image pickup apparatus for generating an omnidirectional image using a plurality of image pickup lenses having different photographing directions, wherein, The plurality of camera lenses are respectively included in a plurality of camera units, and the camera device includes: A detection component for detecting the posture of the camera device; The identification component is used to identify a reference pose in a unidirectional direction that is unaffected by subsequent changes in the pose of the camera device when the camera device is in an operating mode for generating a first image with a smaller field of view compared to the omnidirectional image, and to identify the range to be captured required to generate the first image based on the difference between the reference pose of the camera device and the current pose detected by the detection component. A control unit is configured to, when the camera device is in the operating mode, place the first camera unit from the plurality of camera units in a first state capable of performing image processing if the field of view of the first camera unit includes at least a portion of the first region, and place the second camera unit from the plurality of camera units in a power-saving state that consumes less power compared to being in the first state if the field of view of the second camera unit does not include any portion of the first region; and A generation component is used to generate a wizard display, which includes instructions on how to change the orientation of the camera equipment to reduce the number of camera units to be used for recording.

2. The camera device according to claim 1, wherein, The generating component generates the first image by cropping the area to be photographed from an image obtained using the first camera lens among the plurality of camera lenses, wherein the controlling component enables the recording using the first camera lens.

3. The camera device according to claim 2, further comprising: A compositing component is used to generate a composite image by stitching together images captured using the plurality of camera lenses when the control component enables imaging using each of the plurality of camera lenses. The generating component generates the first image by cropping the area to be photographed from the synthesized image.

4. The camera device according to claim 3, wherein, When the control unit enables video recording using each of the plurality of camera lenses, the generation unit generates the first image superimposed on the wizard display.

5. The camera device according to claim 4, wherein, The wizard display also includes one or more of the following displays: a display indicating the boundaries between the multiple images used to generate the composite image; and a display indicating that the multiple camera lenses are valid for recording.

6. The camera device according to any one of claims 2 to 5, wherein, The first image generated by the generating component is output to a display device included in the camera device or to an external device including a display device.

7. The camera device according to any one of claims 1 to 5, wherein, The recognition component identifies the shooting range by converting the position of the shooting range when the camera device is in the reference posture to a position corresponding to the current posture of the camera device.

8. The camera device according to any one of claims 1 to 5, wherein, The control unit causes the circuitry associated with the camera lens used for invalid recordings to switch to a state that consumes less power compared to when the recordings using the same lens are valid.

9. The camera device according to claim 8, wherein, The circuit includes an image sensor.

10. The camera device according to any one of claims 1 to 5, wherein, The identification performed by the identification component and the activation and deactivation performed by the control component are repeated.

11. The camera device according to any one of claims 1 to 5, wherein, The multiple camera lenses are circular fisheye lenses.

12. A control method of an image pickup apparatus capable of generating an omnidirectional image using a plurality of image pickup lenses having different photographing directions, wherein The plurality of camera lenses are respectively included in a plurality of camera units, and the control method includes: Detect the posture of the camera device; When the camera device is in an operating mode for generating a first image with a smaller field of view compared to the omnidirectional image, a reference pose in a unidirectional direction is identified as unaffected by subsequent changes in the pose of the camera device, and the range to be captured for generating the first image is identified based on the difference between the reference pose of the camera device and the current pose detected in the detection. When the camera device is in the operating mode, if the field of view of the first camera unit includes at least a portion of the first region, the first camera unit is placed in a first state capable of performing image processing from the plurality of camera units; and if the field of view of the second camera unit does not include any portion of the first region, the second camera unit is placed in a power-saving state that consumes less power compared to being in the first state; and The wizard display includes instructions on how to change the orientation of the camera equipment to reduce the number of camera units to be used for recording.

13. A computer-readable medium storing a program that, when executed by a computer included in a camera device capable of generating omnidirectional images using a plurality of camera lenses with different shooting directions, implements the steps of the method according to claim 12.

14. A computer program product comprising a program that, when executed by a computer included in a camera device capable of generating omnidirectional images using a plurality of camera lenses with different shooting directions, implements the steps of the method according to claim 12.

Images