Imaging apparatus and control method

Abstract

To secure visibility at the time of framing by properly informing a user of the position of a main subject, even when the user loses sight of the main subject and preventing unnecessary information from being displayed, when the main subject is in a display screen.SOLUTION: In an imaging apparatus including first imaging means capable of imaging an imaging area, display means for displaying image data acquired by the first imaging means as the display screen, acquisition means for acquiring a sensing result from sensing means capable of sensing a sensing area wider than the imaging area, and main subject information acquisition means for acquiring main subject information which is information relating to the main subject from the image data, the display means displays framing assist information to assist the framing of the main subject on the basis of the sensing result by the sensing means, in a state where the main subject does not exist in the display screen or in a state where it is predicted that the main subject disappears from the display screen.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus and a control method. [Background technology]

[0002] Conventionally, imaging devices that display information obtained from a plurality of cameras with different angles of view by superimposing the information are known. For example, a digital camera disclosed in Patent Document 1 superimposes image data captured by a camera with a wide angle of view while displaying image data captured by a camera with a narrow angle of view. In this way, even if the main subject disappears from the screen while being captured by the camera with the narrow angle of view, the main subject is visible in the camera with the wide angle of view, and the main subject is not completely lost. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-148330 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the digital camera disclosed in Patent Document 1, when framing in a situation where the main subject is identified, simply superimposing and displaying a wide-angle image may result in unnecessary display being constantly displayed in a part of the display screen, and therefore, in the case of the digital camera, framing is hindered when the main subject comes into an area where the unnecessary display is superimposed.

[0005] In view of such problems, the present invention aims to ensure visibility during framing by properly informing the user of the position of a main subject even if the user loses sight of the main subject, and by not displaying unnecessary information when the main subject is within the display screen. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the imaging device of the present invention is an imaging device comprising a first imaging means capable of imaging an imaging area, a display means for displaying image data acquired by the first imaging means as a display screen, an acquisition means for acquiring sensing results from a sensing means capable of sensing a sensing area larger than the imaging area, and a main subject information acquisition means for acquiring main subject information, which is information about a main subject, from the image data, wherein the display means displays framing assist information that assists in framing the main subject based on the sensing result by the sensing means when the main subject is not present on the display screen or when it is predicted that the main subject will no longer be present on the display screen. Effect of the Invention

[0007] According to the present invention, even if a user loses sight of the main subject, the position of the main subject is appropriately notified, and when the main subject is within the display screen, unnecessary information is not displayed, thereby ensuring visibility during framing. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of an imaging device having a display device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing a pixel array according to the first embodiment. [Diagram 3] FIG. 2 is a diagram showing a pixel structure according to the first embodiment. [Figure 4] 4 is a diagram for explaining the correspondence between pixels of the image sensor and pupil intensity distribution in the first embodiment. FIG. [Diagram 5] FIG. 4 is a diagram for explaining main subject detection in the first embodiment. [Figure 6] FIG. 1 is a diagram for explaining a configuration of an imaging device according to a first embodiment. [Figure 7] 1A to 1C are diagrams illustrating examples in which the framing assist display of the first embodiment is required. [Figure 8]11 is a diagram illustrating a user interface of framing assist information according to the first embodiment. FIG. [Figure 9] 11 is a flowchart of framing assist information display in the first embodiment. [Figure 10] 13 is a flowchart of framing assist information display according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The imaging device according to the embodiment includes a first imaging means, a display means, a sensing means, an acquisition means, and a main subject information acquisition means. The first imaging means is, for example, a telephoto camera described later, and is capable of capturing an image of an imaging area. The display means displays image data acquired by the first imaging means as a display screen. The sensing means is, for example, a wide-angle camera described later, and is capable of sensing a sensing area wider than the imaging area. The acquisition means acquires a sensing result from the sensing means. The main subject information acquisition means acquires main subject information, which is information about the main subject, from the image data. In addition, the display means displays framing assist information that assists in framing the main subject based on the sensing result by the sensing means when the main subject does not exist on the display screen or when it is predicted that the main subject will no longer exist on the display screen.

[0010] Furthermore, the display means does not need to display the framing assist information when the main subject is present on the display screen or when it is not predicted that the main subject will no longer be present on the display screen.

[0011] The sensing result may also include a detection result of the main subject by the sensing means.

[0012] The sensing means may be provided in a first sensing device that is detachable from the imaging device.

[0013] Furthermore, when the first sensing device is connected, the imaging device may receive at least one of sensing specification-related information related to the sensing specification of the sensing means and parallax-related information related to the parallax between the first imaging means and the sensing means. In this case, the display means displays the framing assist information based on at least one of the sensing specification-related information and the parallax-related information.

[0014] The display means may change the timing for switching between displaying and not displaying the framing assist information depending on the magnitude of the shake of the main subject or the magnitude of the hand shake of the user who uses the imaging device.

[0015] The imaging device may further include a camera shake detection unit that detects the magnitude of camera shake of a user using the imaging device. In this case, the display unit may display framing assist information when it is determined that there is a discrepancy between the direction of camera shake of the main subject and the direction of camera shake detected by the camera shake detection unit.

[0016] When the display means determines that there is a discrepancy between the direction of shaking of the main subject and the direction of camera shake detected by the camera shake detection means, it may display framing assist information indicating that the main subject is predicted to disappear from the display screen.

[0017] In addition, when the display means determines that the main subject will no longer be present in a specified area of ​​the display screen or that part of the main subject will no longer be present on the display screen, the display means may display framing assist information indicating that the main subject is predicted to no longer be present on the display screen.

[0018] In addition, when the display means detects that the main subject has changed position on the display screen by more than a predetermined distance, it may display framing assist information indicating that the main subject is predicted to disappear from the display screen.

[0019] Furthermore, when the main subject is detected by the first imaging means, the sensing means may start at least one of focus detection control and exposure control based on main subject information.

[0020] Furthermore, when the main subject is detected by the sensing means, the first imaging means may start at least one of focus detection control and exposure control based on the main subject information.

[0021] The sensing means may be capable of acquiring main subject information. In this case, when the main subject is not present on the display screen and a process of acquiring the main subject information by the sensing means is obstructed by the user, the notifying means notifies the user of the error by at least one of a display, a sound, and a vibration.

[0022] Furthermore, when the sensing means is unable to sense the main subject in a situation where the main subject is not present on the display screen, the display means does not need to display the framing assist information.

[0023] The display means may display the sensing result of a sensing area, which is an area where the sensing means is capable of sensing, as framing assist information by superimposing it on the display screen.

[0024] The display means may display at least one of information indicating a direction of the main subject and information indicating a distance from the main subject to the imaging area as framing assist information superimposed on the display screen based on the sensing result.

[0025] The sensing means may be at least one of a second imaging means separate from the first imaging means, a heat detection means, a reflected light receiving means, and a reflected sound receiving means.

[0026] The second imaging means may be capable of outputting an imaging signal and a signal from which phase difference information or distance information can be obtained.

[0027] The imaging device may further include a first focus detection means corresponding to the first imaging means and a second focus detection means corresponding to the sensing means. The first focus detection means performs focus detection using a result of focus detection performed by the second focus detection means. The second focus detection means performs focus detection using a result of focus detection performed by the first focus detection means.

[0028] The imaging device may further include a first focus detection means corresponding to the first imaging means and a second focus detection means corresponding to the sensing means. The first exposure control means performs exposure control using a result of the exposure control performed by the second exposure control means. The second exposure control means performs exposure control using a result of the exposure control performed by the first exposure control means.

[0029] The display means may display the framing assist information in a direction opposite to the main subject with respect to the direction in which the main subject comes into the frame, the up-down direction, or the left-right direction.

[0030] A control method according to an embodiment is a control method for an imaging device including a first imaging means capable of imaging an imaging area, and a display means for displaying image data acquired by the first imaging means as a display screen. This control method includes a sensing result acquisition step, a main subject information acquisition step, and a display step. The sensing result acquisition step is a step of acquiring a sensing result from a sensing means capable of sensing a sensing area larger than the imaging area. The main subject information acquisition step is a step of acquiring main subject information, which is information related to the main subject, from the image data. The display step is a step of displaying framing assist information that assists in framing the main subject based on the sensing result by the sensing means when the main subject is not present on the display screen or when it is predicted that the main subject will no longer be present on the display screen.

[0031] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to refer to the same components, and duplicated descriptions will be omitted.

[0032] In this embodiment, the present invention will be described as being applied to an imaging device such as a digital camera, but the present invention can be widely applied to devices other than imaging devices, such as display devices, distance detection devices, information processing devices, and electronic devices.

[0033] <Example 1> [Overall configuration] 1 is a block diagram showing the configuration of an imaging device having a display device according to the first embodiment. The imaging device 100 is a digital camera system equipped with a camera body and an interchangeable lens (imaging optical system or imaging optical system) that is detachable from the camera body. However, the present invention is not limited to this, and can also be applied to an imaging device in which the camera body and the lens are integrally configured.

[0034] The imaging optical system (imaging optical system) generates a subject image (optical image) of a subject on a predetermined imaging plane. The first lens group 101 is disposed at the front (subject side) of the multiple lens groups constituting the imaging optical system, and is held by a lens barrel so as to be movable forward and backward along the optical axis OA. The aperture / shutter (aperture) 102 adjusts the amount of light during imaging by adjusting its opening diameter, and functions as an exposure time adjustment shutter during still image imaging. The second lens group 103 advances and retreats along the optical axis OA together with the aperture / shutter 102, and has a zoom function of performing a variable magnification operation in conjunction with the advance and retreat operation of the first lens group 101. The third lens group 105 is a focus lens group that performs focus adjustment (focus operation) by advancing and retreating along the optical axis OA. The optical low-pass filter 106 is an optical element for reducing false colors and moire in the captured image. Each lens group in this embodiment corresponds to a lens unit, and in particular, the third lens group 105 corresponds to a lens unit with adjustable focus.

[0035] The image sensor 107 is composed of, for example, a CMOS (Complementary Metal-Oxide-Semiconductor) sensor or a CCD (Charge-Coupled Device) sensor and its peripheral circuits, and performs photoelectric conversion of the subject image. For example, a two-dimensional single-plate color sensor is used as the image sensor 107, in which a primary color mosaic filter in a Bayer array is formed on-chip on a light receiving pixel having m pixels in the horizontal direction and n pixels in the vertical direction. Note that the imaging optical system and the image sensor 107 constitute one imaging unit, but are not limited to a single-plate system as shown in this embodiment, and may be, for example, a three-plate system. Also, a configuration having a plurality of imaging units may be used. In other words, the present invention is applicable as long as the imaging device has a corresponding imaging optical system.

[0036] The zoom actuator 111 moves the first lens group 101 and the second lens group 103 along the optical axis OA by rotating a cam barrel (not shown) when performing a magnification change operation. The aperture / shutter actuator 112 adjusts the opening diameter of the aperture / shutter 102 when adjusting the amount of light (amount of image capture light). The focus actuator 114 moves the third lens group 105 along the optical axis OA when performing focus adjustment. Note that it is not necessary for the aperture and the shutter to be combined, and they may be provided separately.

[0037] The CPU (Central Processing Unit) 121 is a control device that controls various aspects of the imaging device 100. The CPU 121 includes a calculation unit, a ROM (Read Only Memory), a RAM (Random Access Memory), an A / D converter (Analog-to-digital converter), a D / A converter (Digital-to-analog converter), a communication interface circuit, and the like. The CPU 121 reads out and executes a predetermined program stored in the ROM or RAM to drive various circuits of the imaging device 100 and control a series of operations such as focus detection (AF: Auto Focus), imaging, image processing, and recording. Some of the functions of the CPU 121 may be implemented as a hardware circuit. A reconfigurable circuit such as an FPGA (Field-Programmable Gate Array) may be used for at least a part of the CPU 121 or a circuit related thereto. For example, a dedicated hardware circuit may be used for a part of the calculation for focus detection, which will be described later, to reduce the calculation time.

[0038] Moreover, the CPU 121 includes a pixel signal acquiring means 121a, a signal generating means 121b, a focus detecting means 121c, a lens information acquiring means 121d, and a main subject information acquiring means 121e. The communication interface circuit included in the CPU 121 may be connected to an external device via a wired cable such as a Universal Serial Bus (USB) cable or a Local Area Network (LAN) cable. Alternatively, the communication interface circuit included in the CPU 121 may be connected to an external device via wireless communication such as a wireless LAN or a mobile communication line. The connection with the communication partner is not limited to a connection method of directly connecting to a personal computer or a smartphone, and may be a method of connecting to a nearby or remote device via an access point or a network.

[0039] The imaging element drive circuit 124 controls the imaging operation of the imaging element 107, and performs A / D conversion on the acquired image signal and transmits it to the CPU 121. The image processing circuit 125 performs processes such as gamma conversion, color interpolation, and JPEG (Joint Photographic Experts Group) compression on the image data output from the imaging element 107.

[0040] The focus driving circuit 126 drives the focus actuator 114 based on the focus detection result of the focus detection means 121c and moves the third lens group 105 along the optical axis OA to adjust the focus. The aperture / shutter driving circuit 128 drives the aperture / shutter actuator 112 to control the aperture diameter of the aperture / shutter 102 and to control the exposure time when capturing a still image. The zoom driving circuit 129 drives the zoom actuator 111 in response to the zoom operation of the photographer to perform a magnification change operation by moving the first lens group 101 and the second lens group 103 along the optical axis OA.

[0041] The lens communication circuit 130 communicates with the interchangeable lens attached to the camera body, acquires lens information of the interchangeable lens, and sets various focus detection correction values. The acquired lens information is output to a lens information acquisition means 121d of the CPU 121. Also, imaging information detected by the camera body may be transmitted to the interchangeable lens. The interchangeable lens and the camera body are bayonet-coupled via a mount, and multiple terminals are in contact with each other when coupled.

[0042] The imaging optical system 140 includes a first lens group 101, which is an imaging optical system, an aperture / shutter (aperture) 102, a second lens group 103, a third lens group 105, and an optical low-pass filter 106. The imaging optical system 140 also includes a zoom actuator 111, an aperture / shutter actuator 112, and a focus actuator 114. The imaging optical system 140 also includes a focus drive circuit 126, an aperture / shutter drive circuit 128, a zoom drive circuit 129, a lens communication circuit 130, an image sensor 107, which is an imaging system, an image sensor drive circuit 124, and an image processing circuit 125.

[0043] The display unit 131 is configured to include, for example, a liquid crystal display (LCD). The display unit 131 displays information about the imaging mode of the imaging device 100, a preview image before imaging, a confirmation image after imaging, a focus state display image at the time of focus detection, and the like. The operation unit 132 is configured to include a power switch, a release switch, a zoom operation switch, an imaging mode selection switch, and the like. The release switch has two states, a half-pressed state and a full-pressed state. The half-pressed state is a state in which the release switch SW1 is on. The full-pressed state is a state in which the release switch SW2 is on. The recording medium 133 is, for example, a flash memory detachable from the imaging device 100, and records the captured image (image data). The storage unit 134 stores the captured image and the like in a predetermined format.

[0044] The sub-camera connection unit 135 is a connection unit for separately connecting from the outside an external imaging device having the same function as the above-mentioned imaging optical system 140. The sub-camera connection unit 135 is also a communication connection unit for transmitting and receiving signals for driving and controlling various actuators and imaging elements in the external imaging device from the CPU 121 and receiving image signals from the external imaging element.

[0045] Note that a part of the functions of operation unit 132 may be provided in the display unit 131 in the form of a touch panel or the like. By operating the touch panel while a preview image is being displayed on display unit 131, it becomes possible to perform focus detection for any position in the image.

[0046] A contrast AF section (TVAF section) (not shown) may be provided to perform focus detection processing using a contrast detection method based on the generated contrast AF evaluation value (contrast information of image data). During focus detection processing using a contrast detection method, the focus lens group 105 is moved and the lens position at which the evaluation value (focus evaluation value) reaches a peak is detected as the in-focus position.

[0047] In this way, it is possible to perform a combination of image plane phase difference AF and contrast AF, and these can be used selectively or in combination depending on the situation. Each block functions as a control unit that controls the position of the focus lens group 105 using the respective focus detection results.

[0048] Next, the pixel array and pixel structure of the image sensor 107 in this embodiment will be described with reference to Figs. 2 and 3. Fig. 2 is a diagram showing a pixel array of the first embodiment. Specifically, Fig. 2 shows the pixel array of the image sensor 107 in an area of ​​4 rows x 4 columns. Fig. 3 is a diagram showing the pixel structure of the first embodiment. Specifically, Fig. 3 shows the pixel structure of the image sensor 107. Fig. 3(a) is a plan view of the pixel 200G of the image sensor 107 when viewed from the +z direction. Fig. 3(b) is a cross-sectional view taken along line aa when the pixel 200G of the image sensor 107 is viewed from the +z direction.

[0049] In this embodiment, the pixel group 200 of 2 columns x 2 rows has pixels 200R, 200G, and 200B arranged in a Bayer array. That is, among the pixel group 200, the pixel 200R having the spectral sensitivity of R (red) is arranged at the upper left, the pixel 200G having the spectral sensitivity of G (green) is arranged at the upper right and lower left, and the pixel 200B having the spectral sensitivity of B (blue) is arranged at the lower right. The pixel 200R, the pixel 200G, and the pixel 200B are configured by focus detection pixels (first focus detection pixel) 201 and focus detection pixels (second focus detection pixel) 202 arranged in 2 columns x 1 row. For this reason, the array of the focus detection pixels is shown in a range of 8 columns x 4 rows in FIG. 2. Note that the pixel in this embodiment is configured by two focus detection pixels arranged in the x direction, but this is not limited to this, and may be arranged in the y direction. Also, it may be configured by two or more focus detection pixels, or may be configured by combining several configurations.

[0050] As shown in FIG. 2, the image sensor 107 is configured by arranging a large number of pixels (focus detection pixels of 8 columns x 4 rows) in 4 columns x 4 rows on a surface, and outputs an image signal (focus detection signal). In the image sensor 107 of this embodiment, the pixel period P is 6 μm, and the number of pixels N is 6000 columns x 4000 rows = 24 million pixels. In addition, the image sensor 107 has a period PSUB in the column direction of the focus detection pixels of 3 μm, and the number of focus detection pixels NSUB is 12000 columns x 4000 rows = 48 million pixels. When a video or the like in 4K format is acquired using the image sensor 107, it is desirable for the image sensor 107 to have pixels in 4000 columns or more. In addition, when an image is acquired in a format of a size larger than this, it is desirable for the image sensor 107 to have a number of pixels that matches the format.

[0051] As shown in FIG. 3B, the pixel 200G of this embodiment is provided with a microlens 305 for collecting incident light on the light receiving surface side, which is the interface of a semiconductor such as silicon on which the photodiode of the pixel is formed. A plurality of microlenses 305 are arranged two-dimensionally, and are arranged at a position a predetermined distance away from the light receiving surface in the z-axis direction (direction of the optical axis OA). In addition, the pixel 200G is formed with a photoelectric conversion unit 301 (a generating unit for a first focus detection signal) and a photoelectric conversion unit 302 (a generating unit for a second focus detection signal) having a division number NLF=Nx×Ny (division number 2) divided into Nx divisions (2 divisions) in the x direction and Ny divisions (1 division) in the y direction. The photoelectric conversion unit 301 and the photoelectric conversion unit 302 correspond to the focus detection pixel 201 and the focus detection pixel 202, respectively.

[0052] The photoelectric conversion unit 301 and the photoelectric conversion unit 302 are formed on a semiconductor substrate such as silicon, and are configured as pn junction photodiodes each consisting of a p-type layer and an n-type layer. The photoelectric conversion unit 301 and the photoelectric conversion unit 302 may be configured as a pin structure photodiode with an intrinsic layer sandwiched between a p-type layer and an n-type layer as necessary. The pixel 200G (each pixel) is provided with a color filter 306 between the microlens 305 and the photoelectric conversion unit 301 and the photoelectric conversion unit 302. The pixel 200G can change the spectral transmittance of the color filter 306 for each pixel or each photoelectric conversion unit as necessary. The pixel 200G may be provided without a color filter.

[0053] Light incident on pixel 200G is collected by microlens 305, dispersed by color filter 306, and then received by photoelectric conversion unit 301 and photoelectric conversion unit 302. In photoelectric conversion unit 301 and photoelectric conversion unit 302, pairs of electrons and holes are generated according to the amount of received light, and after they are separated by a depletion layer, negatively charged electrons are accumulated in the n-type layer. Meanwhile, holes are discharged to the outside of image sensor 107 through a p-type layer connected to a constant voltage source (not shown). The electrons accumulated in the n-type layers of photoelectric conversion unit 301 and photoelectric conversion unit 302 are transferred to a capacitance unit (FD) via a transfer gate and converted into a voltage signal.

[0054] In this embodiment, the microlens 305 corresponds to the optical system in the image sensor 107. The optical system may include a plurality of microlenses, or may be configured using materials with different refractive indices, such as a waveguide. The image sensor 107 may be a back-illuminated image sensor having a circuit or the like on the surface opposite to the surface having the microlens 305, or may be a stacked image sensor further having some circuits, such as the image sensor drive circuit 124 and the image processing circuit 125. A material other than silicon may be used as the semiconductor substrate, and for example, an organic material may be used as the photoelectric conversion material.

[0055] Fig. 4 is a diagram for explaining the correspondence between the pixels of the image sensor of the first embodiment and the pupil intensity distribution. Specifically, Fig. 4 shows a cross-sectional view of the aa cross section of the pixel 200G arranged in the image sensor 107 of the present embodiment shown in Fig. 3(a) viewed from the +y side, and a pupil surface at a position away from the image sensor 600 of the image sensor 107 by a distance Z in the z-axis direction (direction of the optical axis OA). Note that in Fig. 4, the x-axis and y-axis of the cross-sectional view are inverted with respect to Fig. 3 in order to correspond to the coordinate axes of the exit pupil surface. The image sensor 600 of the image sensor 107 is disposed on the image surface of the image-forming optical system.

[0056] The pupil intensity distribution (first pupil intensity distribution) 501 is in a substantially conjugate relationship with the light receiving surface of the photoelectric conversion unit 301, whose center of gravity is decentered in the -x direction, via the microlens 305. Therefore, the first pupil intensity distribution 501 corresponds to a pupil region that can receive light at the focus detection pixel 201. The center of gravity of the first pupil intensity distribution 501 is decentered on the +xp side on the pupil surface. Similarly, the pupil intensity distribution (second pupil intensity distribution) 502 is in a substantially conjugate relationship with the light receiving surface of the photoelectric conversion unit 302, whose center of gravity is decentered in the +x direction, via the microlens 305. Therefore, the second pupil intensity distribution 502 corresponds to a pupil region that can receive light at the focus detection pixel 202. The center of gravity of the second pupil intensity distribution 502 is decentered on the -xp side on the pupil surface. Furthermore, pupil intensity distribution 500 is a pupil region that can receive light in the entire pixel 200G when all of the photoelectric conversion units 301 and 302 (focus detection pixels 201 and 202) are combined. That is, the first pupil intensity distribution 501 is decentered on the +xp side on the pupil plane with respect to the center of the pupil intensity distribution 500. Furthermore, the second pupil intensity distribution 502 is decentered on the -xp side on the pupil plane with respect to the center of the pupil intensity distribution 500.

[0057] By using the imaging element described above, it is possible to obtain phase difference information by performing correlation calculation between the output signal of the focus detection pixel 201 and the output signal of the focus detection pixel 202. Then, by using a defocus amount conversion coefficient that converts the calculated phase difference information and the phase difference information calculated by a known technology into a defocus amount (amount of focus deviation), it is possible to calculate the defocus amount, and focus detection is possible. By using an imaging element capable of obtaining phase difference information, it is possible to perform focus detection while constantly calculating the defocus amount even for a subject that moves vigorously. Therefore, by the user using the framing assist display described later, it is possible to immediately focus and shoot when the main subject moves from outside the display screen into the display screen.

[0058] In this embodiment, a simple example of the system configuration will be described by using an image sensor that also has a focus detection function, but since the effect of this case is not impaired by the focus detection method, other focus detection methods may be adopted, such as the TOF (Time of Flight) method, which is a focus detection method that measures distance by shining laser light or the like on the subject and detecting reflected light, or a contrast method that performs focus detection by looking at the contrast of the subject.

[0059] When adopting the TOF method, the system configuration may be simplified by incorporating it into the image sensor, but if a TOF sensor integrated with the image sensor is used, the TOF pixels will be arranged discretely among the image pixels. This will result in a decrease in the spatial resolution of the object sensing by the TOF sensor. Therefore, a TOF sensor for focus detection may be provided separately from the image sensor depending on the specifications.

[0060] When the contrast method is adopted, if focus detection is performed using the well-known hill-climbing method, it is expected that it will take time to detect the in-focus point when the main subject moves from outside the display screen to inside the screen, and it will take time to take a picture. For this reason, it is preferable to use the well-known wobbling method, which detects the focus while constantly slightly driving the focus lens to detect the direction of focus deviation.

[0061] Also, in this embodiment, an image sensor with 2×1 pupil division as shown in Figures 3 and 4 has been described, but the effect of this invention is not impaired by the number of pupil divisions or the pupil division direction. For this reason, in addition to the case where the pupil is divided only in the x direction as shown in Figure 3, the pupil division direction may be only in the y direction, or a 2×2 pupil division with pupil division in both the x and y directions may be used.

[0062] In addition, in this embodiment, an example of a pixel structure capable of acquiring phase difference information with one pixel as shown in Fig. 3 and Fig. 4 will be described. However, a pixel structure in which a half-opening light-shielding layer is provided between the microlens 305 and the light receiving surface in Fig. 3 to separate a pixel capable of acquiring pupil intensity distribution 501 from a pixel capable of acquiring pupil intensity distribution 502, and phase difference information is acquired using a corresponding pair of pixels may be used.

[0063] [Main subject detection and main subject out of frame] For example, a method of detecting a main subject includes a method of selecting an object close to the center of the screen from among main subject candidates in the screen. Alternatively, a method of detecting a main subject includes a method of performing focus detection in the screen and selecting a subject on the closest side. Alternatively, in recent years, a method of detecting a main subject includes a method of extracting and selecting a specific subject from the screen based on a preset criterion using deep learning. FIG. 5 is a diagram for explaining the detection of a main subject in the first embodiment. For example, in the example shown in FIG. 5, a composition is shown in which a car is on the close side, a bird is in the distance, and a person is between the car and the bird. In this case, if the main subject detection is focus detection with close priority, the car becomes the main subject, if the main subject detection is a subject in the center of the screen with priority, the person becomes the main subject, and if the main subject detection is a bird with priority, the bird becomes the main subject.

[0064] In this embodiment, the method of selecting the main subject may be any method; the main subject may be selected using a method that the user has previously set in the imaging display, or the main subject may be designated by the user directly specifying a subject within the screen.

[0065] In recent years, a method of detecting a main subject and then performing focus detection and exposure control while tracking the main subject moving around within the screen has become known as a publicly-known technology and has been commercialized. Therefore, after an imaging device detects a main subject, a user can easily adjust the composition (framing) so that the main subject is in a desired position before taking a picture.

[0066] However, when a subject whose movements tend to be erratic, such as a bird or a child, is photographed with a telephoto camera, the user is unable to frame the subject in time, and the main subject is likely to disappear from the screen (out of frame). There are three reasons for this. The first reason is that when photographing a subject whose movements tend to be erratic, it is difficult for the user to predict the subject's movement, and framing takes time. The second reason is that when photographing with a telephoto camera, the shooting angle of view is narrow, the movement of the main subject within the screen angle of view is large, and it becomes difficult to frame the main subject at a desired position. The third reason is similar to the second reason, but when photographing with a telephoto camera, the narrow shooting angle of view increases the influence of the user's hand shake, making the main subject in the screen more likely to blur, and making it difficult to frame the main subject at a desired position.

[0067] For these three reasons, even if focus detection and exposure control are performed while detecting and tracking the main subject, when a subject with irregular movements is photographed with a telephoto camera, the main subject may go out of frame, making it difficult for the user to perform the desired photographing. Once the user loses sight of the main subject that moves irregularly from the display screen, he or she will not know how to move the imaging device to bring the main subject into the screen. For this reason, when a user photographs an irregular subject with a telephoto camera, it is not sufficient to simply track the main subject and perform focus detection and exposure control in order to perform the photographing desired by the user. Therefore, when the main subject goes out of frame or is expected to go out of frame, it is necessary to display framing assist information, which is information that assists framing, to make it easier to return the main subject to the screen. Then, when it is determined that the main subject has returned to the screen and that the main subject can be stably framed, it is desirable not to display the framing assist information. This is because if the framing assist information is continuously displayed, visibility is hindered in the area where the framing assist information and the photographing display screen overlap.

[0068] [Framing assist display using imaging and sensing means] FIG. 6 is a diagram for explaining the configuration of the imaging device of the first embodiment. The imaging optical system 140a is an imaging means and can be used as a telephoto camera. The imaging optical system 140b is a sensing means and can be used as a wide-angle camera. Since the telephoto camera and the wide-angle camera are systems that perform imaging through different optical systems, parallax occurs between the telephoto camera and the wide-angle camera. In this embodiment, the display of the framing assist information is turned on / off by linking the results of detecting the main subject using images from two different imaging / optical systems. For this reason, it is possible to perform display and prediction calculation with higher accuracy by displaying the position of the imaging area in the sensing area and performing prediction calculation of the main subject entering or leaving the frame while taking into account the parallax between the telephoto camera and the wide-angle camera.

[0069] In this embodiment, the main subject information acquisition means in CPU 121 has a ROM area (not shown), in which parallax information between the telephoto camera and the wide-angle camera, information on the angle of view of the telephoto camera, and information on the angle of view of the wide-angle camera are stored. The information stored in the ROM area is used to set display parameters when acquiring main subject information and when displaying frame assist information on display 131.

[0070] In this embodiment, the sensing means is connected to the imaging device in advance as shown in FIG. 6, but as shown in FIG. 1, the sensing means may be connected by the sub-camera connection unit 135 and may be connected from an external device. In this case, any one or more of the sensing specification related information, the parallax related information, and the main subject information may be transmitted to the CPU 121 by communication from the sub-camera connection unit and used to calculate the display parameters of the frame assist information. The sensing specification related information is information related to the sensing specification of the sensing means. The parallax related information is information related to the parallax between the imaging means and the sensing means. The main subject information is information related to the main subject, and is transmitted to the CPU 121 when the main subject position can be specified by the sensing means, and is used to calculate the display parameters of the frame assist information. Note that wireless communication has also developed in recent years, so the sensing specification related information, the parallax related information, and the main subject information may be transmitted to the CPU 121 without going through the sub-camera connection unit 135.

[0071] In addition, in this embodiment, the sensing means is described using an imaging means having a wider angle than the imaging means, but the sensing means is not limited to the imaging means. As long as the sensing means has a wider angle of view than the imaging means and can sense a sensing area wider than the imaging area, it is not limited to a sensing method, and other means than the imaging means may be used depending on the application. For example, when taking a picture in the dark where there is no light, a heat detection means such as an infrared sensor that detects heat and detects the subject is effective as the other means. Alternatively, a reflected light receiving means such as LIDAR (Light Detection and Ranging) that projects active light onto the subject and detects the reflected light to detect the subject is effective as the other means. Also, for example, in the sea, a reflected sound receiving means that detects the reflected sound of a sonar or the like to detect the subject position may be used as the other means.

[0072] [Framing Assist display switching timing] A case where the framing assist display is required will be described with reference to FIG. 7. FIG. 7 is a diagram for explaining a case where the framing assist display is required in the first embodiment. FIG. 7 shows an imaging area 701, a sensing area 702, a bird 703, and a bird 704. The imaging area 701 is the angle of view of a telephoto camera. The sensing area 702 is the angle of view of a wide-angle camera. The bird 703 is a bird that exists inside the imaging area, which is a display screen. The bird 704 is a bird that does not exist outside the imaging area, which is a display screen, and exists inside the sensing area.

[0073] First, as described above, a case where the framing assist display is expected to be necessary is when the main subject, a bird, goes out of frame as the user is framing the image and changes from state 703 to state 704. Another expected case is when sensing area 702 is displayed as the display screen, and while capturing an image of bird 704 with a wide-angle camera, which is the sensing means, the camera display is switched and the display screen becomes imaging area 701.

[0074] Therefore, when a main subject is detected and goes out of frame in the above example, it is desirable to promptly display framing assist information so that the user can grasp the position of the main subject. This enables the user to quickly capture the main subject within the display screen based on the framing assist information and capture the image with the desired framing.

[0075] In addition to the above-mentioned cases, there are cases where the main subject is expected to go out of frame in the near future, taking into account human reaction speed, because the main subject moves very fast, starts to move suddenly, changes direction, etc. Examples of such cases include a bird that has been hovering and suddenly starts to move, a butterfly that has been sucking nectar from a flower and suddenly flies away, and a child that has been running around in a park and suddenly changes direction.

[0076] In these cases, it is difficult for a human to react quickly enough, so it is desirable to detect the amount of change in position of the main subject and determine whether or not the amount of change in position exceeds a predetermined threshold. If it is determined that the amount of change in position exceeds the predetermined threshold, it is desirable to determine that there is a high possibility that the main subject will go out of frame in the near future and display framing assist information.

[0077] At this time, the user's camera shake information is also detected, and if the amount of camera shake is large, and the direction of the camera shake and the direction of the change in position of the main subject are compared and there is a large discrepancy, it is predicted that the user has not been able to frame the main subject well. Therefore, in such cases, an adjustment may be made to make it easier to display the framing assist information.

[0078] In addition to the above, framing assist information may be displayed according to user preference even when a part of the main subject goes out of the frame before the main subject is completely out of the display screen or when it is expected that the main subject will go out of the frame. Alternatively, framing assist information may be displayed according to the amount of positional change of the main subject when the amount of positional change is large and it is expected that even a part of the main subject will go out of the frame. On the other hand, adjustment may be made according to the amount of positional change of the main subject to display framing assist information when the amount of positional change is small and it is expected that the entire main subject will go out of the frame.

[0079] The above is an explanation of the timing for displaying the framing assist information. As for the timing for hiding the framing assist information, it is preferable that the timing be when the above-mentioned condition that the main subject has gone out of frame or the condition that the main subject is expected to go out of frame is released. This makes it possible to switch between displaying and hiding the framing assist information without setting up a complicated programming flow chart.

[0080] [Framing Assist display example] 8 is a diagram illustrating a user interface of framing assist information in the first embodiment. The imaging area 801 is an area of ​​the display screen where an image from a telephoto camera is displayed. The sensing area 802 is an area of ​​the display screen where an image from a wide-angle camera is displayed, and is superimposed on the imaging area 801. The imaging area 803 is an area of ​​the display screen for assist display where an image from a telephoto camera corresponding to the sensing area 802 is displayed. The main subject frame 805 is a frame indicating that a bird 804 has been detected as the main subject.

[0081] Sensing area 806 is an area shown superimposed on imaging area 801, which is the display screen, for assist display. Imaging area 807 is an imaging area for the angle of view of sensing area 806, which is for assist display. Deformed display 808 is a deformed display of bird 804, which is the main subject. Sensing area 809 is not actually displayed on the display screen, but is an area of ​​the display screen where an image from a wide-angle camera is displayed. Bird 810 is the main subject captured by the wide-angle camera. Arrow 811 indicates the direction and distance of bird 810, which is the main subject.

[0082] A first example of displaying framing assist information will be described. As shown in Fig. 8(A), a sensing area 802 for assist display and an imaging area 803 for assist display are displayed superimposed on an imaging area 801 displayed on the display screen. This makes it possible to visually convey to the user in which direction to move the camera in order to bring a bird 804, which is the main subject, into the imaging area 803 for assist display. In this case, the imaging area 803 for assist display may be a simple display such as a frame display, or an image from a telephoto camera may be displayed.

[0083] Also, since the superimposed display screen is reduced in size relative to the display screen and difficult to see when the wide-angle image is simply displayed, it is advisable to highlight the position of the main subject. In the example shown in Fig. 8(A), the bird 804 is located at the upper left of the imaging area 803 for assist display, so it is expected that the main subject will come into the frame from the upper left of the display screen as a result of the user framing the image based on the framing assist information. Therefore, if the framing assist information is displayed at the lower right, which is opposite to the position where the main subject will come into the frame, the framing assist information will not impede the visibility of the main subject when the main subject comes into the frame.

[0084] As a modification of Fig. 8(A), even if a reduced wide-angle image is superimposed on the display screen, it is assumed that the screen is too small and some people find it difficult to see. For this reason, as shown in Fig. 8(B), a simplified display may be used in which only an assist display imaging area 807, also displayed in a frame, and a main subject 808 deformed by a dot display or the like are displayed within an assist display sensing area 806, also displayed in a frame. This can be said to be an effort to achieve both visibility with the display screen.

[0085] A second example of displaying framing assist information will be described. As shown in Fig. 8(C), if a bird 810, which is the main subject, can be detected in a sensing area 809, it is possible to identify the position of the main subject even after the main subject goes out of frame. For this reason, a method can be considered that visually informs the user in which direction and how far they should move the camera.

[0086] A representative method of indicating the distance from the imaging region 801 to the position of the main subject is to change the size or length of the arrow 811. For example, the method of changing the size of the arrow 811 is to make the arrow 811 larger as the distance to the main subject bird 810 increases, and to make the arrow 811 smaller as the distance to the main subject bird 810 decreases. Alternatively, the method of changing the length of the arrow 811 is to make the arrow 811 longer as the distance to the main subject bird 810 increases, and to make the arrow 811 shorter as the distance to the main subject bird 810 decreases.

[0087] Alternatively, a representative method of indicating the distance from the imaging area 801 to the position of the main subject is to change the transparency of the arrow 811. For example, this method is a method in which the longer the distance to the bird 810 that is the main subject, the greater the transparency of the arrow 811, and the shorter the distance to the bird 810 that is the main subject, the smaller the transparency of the arrow 811.

[0088] Alternatively, a representative method of indicating the distance from the imaging area 801 to the position of the main subject is to change the color of the arrow 811. For example, this method makes the arrow 811 a warmer color as the distance to the bird 810, which is the main subject, increases, and makes the arrow 811 a cooler color as the distance to the bird 810, which is the main subject, decreases.

[0089] Alternatively, a representative method of indicating the distance from the imaging area 801 to the position of the main subject is to change the color density of the arrow 811. For example, this method is a method in which the color of the arrow 811 becomes darker as the distance to the bird 810, which is the main subject, increases, and the color of the arrow 811 becomes lighter as the distance to the bird 810, which is the main subject, decreases.

[0090] Alternatively, a representative method of indicating the distance from the imaging area 801 to the position of the main subject is to change the blinking interval of the arrow 811. For example, this method is a method in which the longer the distance to the bird 810, which is the main subject, the longer the blinking interval of the arrow 811, and the shorter the distance to the bird 810, which is the main subject, the shorter the blinking interval of the arrow 811.

[0091] Although the arrows have been used to simultaneously indicate the direction of the bird 810 which is the main subject and the distance to the bird 810 which is the main subject, methods other than those described in this embodiment may be used as long as the direction of the main subject and the distance to the main subject are known. Therefore, methods other than those described above are also effective as a display form for the framing assist information, and the effect of this embodiment is not lost in any method.

[0092] [Framing Assist display flowchart] Fig. 9 is a flowchart of framing assist information display in embodiment 1. The process shown in Fig. 9 is a process for displaying framing assist information in the examples shown in Figs. 8(A) and 8(B).

[0093] In step S901, CPU 121 detects a main subject from image data acquired by at least one of imaging / optical system 1 (telephoto camera) 140a and imaging / optical system 2 (wide-angle camera) 140b, and calculates position information of the main subject, which is main subject information. The process of step S901 is executed based on the specifications previously set in the camera described above with reference to FIG. 5. In this embodiment, the telephoto camera and the wide-angle camera are linked to detect the main subject, so in step S901, it is also confirmed whether the same subject is detected as the main subject by the telephoto camera and the wide-angle camera using color, shape, detection position, etc. Note that this embodiment will be described as a case in which the camera automatically detects the main subject, but the main subject may also be detected by the user touching the display screen, etc.

[0094] In step S902, CPU 121 determines whether or not the main subject was detected in at least one of the imaging region and the sensing region in step S901. That is, in step S902, CPU 121 determines whether or not the main subject was detected in at least one of the telephoto camera and the wide-angle camera in step S901. If CPU 121 determines that the main subject was detected in at least one of the imaging region and the sensing region, that is, that the main subject was detected in at least one of the telephoto camera and the wide-angle camera, CPU 121 advances the process to step S906. On the other hand, if CPU 121 determines that the main subject was not detected in at least one of the imaging region and the sensing region, that is, that the main subject was not detected in at least one of the telephoto camera and the wide-angle camera, CPU 121 advances the process to step S903.

[0095] In step S903, the CPU 121 turns off the display setting of the framing assist information.

[0096] In step S904, the CPU 121 performs error detection to determine whether the user has unintentionally performed an action that prevents the imaging of the imaging / optical system 1 (telephoto camera) 140a or the imaging / optical system 2 (wide-angle camera) 140b. For example, such an action is blocking the imaging / optical system 1 (telephoto camera) 140a or the imaging / optical system 2 (wide-angle camera) 140b with a hand, finger, or an obstructing object. The CPU 121 can determine whether the user has performed such an action by comparing the image of the imaging / optical system 1 (telephoto camera) 140a with the image of the imaging / optical system 2 (wide-angle camera) 140b. In addition, in the case of a device with many small cameras such as a smartphone, the CPU 121 may detect whether the camera is blocked by a finger or the like using a pressure sensor or the like because the cameras are small. If the CPU 121 determines that the user has performed such an action, the process proceeds to step S905. On the other hand, if the CPU 121 determines that the user has not performed such an action, the process ends.

[0097] In step S905, the CPU 121 notifies the user by means of a display, sound, vibration, or the like that the user has unintentionally performed an action that interferes with imaging by the imaging / optical system 1 (telephoto camera) 140a or the imaging / optical system 2 (wide-angle camera) 140b.

[0098] In step S906, CPU 121 checks whether the image on the display screen is a telephoto camera setting. If CPU 121 checks that the image on the display screen is a telephoto camera setting, CPU 121 advances the process to step S907. On the other hand, if CPU 121 checks that the image on the display screen is not a telephoto camera setting, CPU 121 advances the process to step S903.

[0099] In step S907, CPU 121 checks whether or not the main subject is present in the telephoto camera image that is the display screen, i.e., in the imaging area. If CPU 121 checks that the main subject is present in the telephoto camera image that is the display screen, CPU 121 advances the process to step S908. On the other hand, if CPU 121 checks that the main subject is not present in the telephoto camera image that is the display screen, CPU 121 advances the process to step S911.

[0100] In step S908, CPU 121 calculates the amount of change in the position of the main subject from the previous frame using position information of the main subject, which is one piece of the main subject information detected and acquired in step S901.

[0101] In step S909, the CPU 121 calculates camera shake information from the acquired image data. Note that, although the present embodiment describes a case in which camera shake information is calculated from image data, camera shake information may be detected and acquired using a gyro sensor or the like.

[0102] In step S910, CPU 121 predicts to which position the main subject will move in the display screen in the next frame, using the amount of change in position of the main subject per frame calculated in step S909 and the camera shake information calculated in step S910. If CPU 121 predicts that the main subject will go out of the field of view of the telephoto camera, i.e., the imaging area, in the next frame, CPU 121 advances the process to step S911. Alternatively, if CPU 121 predicts that the main subject will not go out of the field of view of the telephoto camera, i.e., the imaging area, in the next frame, but will move to an area at the edge of the screen where there is a concern of going out of the frame, CPU 121 advances the process to step S903.

[0103] In step S911, CPU 121 sets the display of framing assist information to on, and displays the contents of Fig. 8(A) or Fig. 8(B) as the framing assist information on display 131. Specifically, when CPU 121 displays the contents of Fig. 8(A), it displays a wide-angle image which is the sensing area, a telephoto image or frame which is the imaging area, and a highlighted display which indicates the position of the main subject on display 131 in the configuration shown in Fig. 8(A). When CPU 121 displays the contents of Fig. 8(B), it displays a frame of the sensing area, a frame of the imaging area, and a deformed display of the position of the main subject on display 131 in the configuration shown in Fig. 8(B).

[0104] The above is the flowchart of this embodiment. In the flowchart of this embodiment, the flow of the method for switching framing assist information on / off has been omitted to make it easier to understand. However, if framing assist information is displayed after the main subject goes out of frame according to the flowchart of this embodiment, and the main subject comes into frame, there is a possibility that focus deviation or exposure deviation will occur. This is because while the main subject is out of frame in the telephoto camera, which is the display camera, the telephoto camera cannot obtain main subject information.

[0105] The user wants to be able to quickly frame the subject with the correct focus and exposure when the main subject comes back into the frame due to the framing assist information. Therefore, while the main subject is being detected only by the wide-angle camera, the CPU 121 performs focus adjustment and exposure control on the telephoto camera side using the focus detection results of the wide-angle camera, i.e., the focus deviation detection results and the exposure detection results. This allows the CPU 121 to provide the user with a more comfortable framing operation.

[0106] As described above, in this embodiment, the display of the frame assist information is switched on and off depending on whether the main subject is out of frame or whether the main subject is expected to be out of frame. In this way, by switching the display of the frame assist information on and off, the user can confirm where the main subject is located when the main subject is out of frame. Also, when the main subject is in frame or present within the display screen, unnecessary information is not displayed, improving the visibility of the display screen and allowing comfortable shooting.

[0107] <Example 2> Next, a second embodiment of the present invention will be described. In the second embodiment, the embodiment is different from the embodiment in the flowchart of the framing assist information in FIG. 9 shown in the first embodiment. In the first embodiment, the process of displaying the framing assist information shown in FIG. 8(A) or FIG. 8(B) is described with reference to FIG. 9. In the second embodiment, the process of displaying the framing assist information shown in FIG. 8(C) is described with reference to FIG. 10. FIG. 10 is a flowchart of the framing assist information display in the second embodiment. The process shown in FIG. 10 is a process of displaying the framing assist in the example shown in FIG. 9(C). Note that the description of the same contents as those in FIG. 9 will be omitted, and the description of the contents of FIG. 10 that are different from those in FIG. 9 will be mainly given.

[0108] Steps S901 to S911 shown in FIG. 10 are the same as steps S901 to S911 shown in FIG.

[0109] In step S912, if it is determined in step S907 that the main subject is out of frame, CPU 121 calculates the positional relationship of the main subject with respect to the imaging area using the position of the main subject determined in step S901. Then, CPU 121 calculates the direction and distance in which the main subject exists with respect to the imaging area. Alternatively, in step S912, if it is determined in step S910 that the main subject is predicted to go out of frame, CPU 121 calculates the positional relationship of the main subject with respect to the imaging area determined in step S910. Then, CPU 121 calculates the direction and distance in which the main subject exists with respect to the imaging area.

[0110] In step S913, CPU 121 sets the display of the framing assist information to on, and in this embodiment, displays the arrow shown in Fig. 8C as the framing assist information on display 131. At this time, CPU 121 transmits to display 131 a display instruction to change the angle of the arrow depending on the direction in which the main subject is present relative to the imaging area calculated in step S911. At this time, CPU 121 also transmits to display 131 a display instruction to change the size, transparency, color, blinking speed, etc. of the arrow depending on the distance in which the main subject is present relative to the imaging area calculated in step S911.

[0111] The present invention includes the following inventions that appropriately combine the above-mentioned features.

[0112] (Configuration 1) An imaging device comprising a first imaging means capable of imaging an imaging area, a display means for displaying image data acquired by the first imaging means as a display screen, an acquisition means for acquiring sensing results from a sensing means capable of sensing a sensing area larger than the imaging area, and a main subject information acquisition means for acquiring main subject information, which is information relating to a main subject, from the image data, characterized in that when the main subject is not present on the display screen or when it is predicted that the main subject will no longer be present on the display screen, the display means displays framing assist information to assist in framing the main subject based on the sensing results by the sensing means.

[0113] (Configuration 2) The imaging device described in configuration 1, characterized in that the display means does not display the framing assist information when the main subject is present on the display screen or when it is not predicted that the main subject will no longer be present on the display screen.

[0114] (Configuration 3) 3. The imaging device according to configuration 1 or 2, wherein the sensing result includes a detection result of the main subject by the sensing means.

[0115] (Configuration 4) The imaging device according to any one of configurations 1 to 3, wherein the sensing means is provided in a first sensing device that is detachable from the imaging device.

[0116] (Configuration 5) The imaging device described in configuration 4, characterized in that when the first sensing device is connected, the imaging device receives at least one of sensing specification related information regarding the sensing specifications of the sensing means and disparity related information related to the disparity between the first imaging means and the sensing means, and the display means displays the framing assist information based on at least one of the sensing specification related information and the disparity related information.

[0117] (Configuration 6) The imaging device according to any one of configurations 1 to 5, characterized in that the display means changes the timing for switching between displaying and not displaying the framing assist information depending on the magnitude of shaking of the main subject or the magnitude of hand shaking of a user using the imaging device.

[0118] (Configuration 7) The imaging device according to any one of configurations 1 to 6, further comprising a camera shake detection means for detecting a magnitude of camera shake of a user using the imaging device, wherein the display means displays the framing assist information when it is determined that there is a discrepancy between a direction of camera shake of the main subject and a direction of camera shake detected by the camera shake detection means.

[0119] (Configuration 8) The imaging device according to any one of configurations 1 to 7, characterized in that when the display means determines that there is a discrepancy between the direction of shake of the main subject and the direction of camera shake detected by the camera shake detection means, it displays the framing assist information as it is predicted that the main subject will no longer be present on the display screen.

[0120] (Configuration 9) The imaging device according to any one of configurations 1 to 8, characterized in that when the display means determines that the main subject will no longer be present in a specified area of ​​the display screen or when it determines that a part of the main subject will no longer be present on the display screen, it displays the framing assist information as it is predicted that the main subject will no longer be present on the display screen.

[0121] (Configuration 10) The imaging device described in any one of configurations 1 to 9, characterized in that when the display means detects that the main subject has changed its position on the display screen by more than a predetermined distance, it displays the framing assist information as it is predicted that the main subject will no longer be present on the display screen.

[0122] (Configuration 11) The imaging device described in any one of configurations 1 to 10, characterized in that when the main subject is detected by the first imaging means, the sensing means starts at least one of focus detection control and exposure control based on the main subject information.

[0123] (Configuration 12) The imaging device according to any one of configurations 1 to 11, characterized in that, when the main subject is detected by the sensing means, the first imaging means starts at least one of focus detection control and exposure control based on the main subject information.

[0124] (Configuration 13) The imaging device according to any one of configurations 1 to 12, characterized in that the sensing means is capable of acquiring the main subject information, and further comprises a notification means for notifying the user of an error by at least one of display, sound, and vibration when a process of the sensing means acquiring the main subject information is obstructed by a user in a state in which the main subject is not present on the display screen.

[0125] (Configuration 14) The imaging device according to any one of configurations 1 to 13, characterized in that the display means does not display the framing assist information when the sensing means is unable to sense the main subject in a situation where the main subject is not present on the display screen.

[0126] (Configuration 15) The imaging device described in any one of configurations 1 to 14, characterized in that the display means displays the sensing result of a sensing area, which is an area capable of being sensed by the sensing means, as the framing assist information by superimposing it on the display screen.

[0127] (Configuration 16) The imaging device according to any one of configurations 1 to 15, characterized in that the display means displays at least one of information indicating a direction of the main subject and information indicating a distance from the main subject to the imaging area as the framing assist information by superimposing it on the display screen based on the sensing result.

[0128] (Configuration 17) The imaging device according to any one of configurations 1 to 16, wherein the sensing means is at least one of a second imaging means separate from the first imaging means, a heat detection means, a reflected light receiving means, and a reflected sound receiving means.

[0129] (Configuration 18) 18. The imaging device according to configuration 17, wherein the second imaging means is capable of outputting an imaging signal and a signal from which phase difference information or distance information can be obtained.

[0130] (Configuration 19) The imaging device described in any one of configurations 1 to 18, further comprising a first focus detection means corresponding to the first imaging means and a second focus detection means corresponding to the sensing means, wherein the first focus detection means performs focus detection using a result of focus detection performed by the second focus detection means, and the second focus detection means performs focus detection using a result of focus detection performed by the first focus detection means.

[0131] (Configuration 20) The imaging device described in any one of configurations 1 to 19, further comprising a first exposure control means corresponding to the first imaging means and a second exposure control means corresponding to the sensing means, wherein the first exposure control means performs exposure control using a result of the exposure control performed by the second exposure control means, and the second exposure control means performs exposure control using a result of the exposure control performed by the first exposure control means.

[0132] (Configuration 21) The imaging device described in any one of configurations 1 to 20, characterized in that the display means displays the framing assist information in a direction opposite to the main subject, in terms of the direction in which the main subject comes into the frame, the up-down direction, or the left-right direction.

[0133] (Configuration 22) A control method for an imaging device equipped with a first imaging means capable of imaging an imaging area and a display means for displaying image data acquired by the first imaging means as a display screen, the control method for an imaging device comprising: a sensing result acquisition step for acquiring sensing results from a sensing means capable of sensing a sensing area larger than the imaging area; a main subject information acquisition step for acquiring main subject information, which is information relating to a main subject, from the image data; and a display step for displaying framing assist information, which assists in framing the main subject, based on the sensing result by the sensing means when the main subject is not present on the display screen or when it is predicted that the main subject will no longer be present on the display screen.

[0134] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a recording medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit for implementing one or more of the functions, such as an ASIC (Application Specific Integrated Circuit).

[0135] The above describes preferred embodiments of the present invention. However, the present invention is not limited to the above-described embodiments. In other words, the present invention includes embodiments in which various modifications have been made based on the gist of the present invention, and these embodiments are not excluded from the scope of the present invention. [Explanation of symbols]

[0136] 101 First lens group 102 Aperture / shutter 103 Second lens group 105 Third lens group 106 Optical Low-pass Filter 107 Image sensor 121 CPU 126 Focus drive circuit 128 Aperture shutter drive circuit 129 Zoom driver circuit

Claims

1. A first imaging means capable of imaging an imaging area; a display means for displaying image data acquired by the first imaging means on a display screen; an acquisition means for acquiring a sensing result from a sensing means capable of sensing a sensing area wider than the imaging area; a main subject information acquiring means for acquiring main subject information, which is information regarding a main subject, from the image data; In an imaging device comprising: the display means displays framing assist information to assist in framing the main subject based on the sensing result by the sensing means when the main subject is not present on the display screen or when it is predicted that the main subject will no longer be present on the display screen.

1. An imaging device comprising:

2. the display means does not display the framing assist information in a state where the main subject is present on the display screen or in a state where it is not predicted that the main subject will no longer be present on the display screen.

2. The imaging device according to claim 1 .

3. the sensing result includes a detection result of the main subject by the sensing means.

2. The imaging device according to claim 1 .

4. The sensing means is provided in a first sensing device that is detachable from the imaging device.

2. The imaging device according to claim 1 .

5. When the first sensing device is connected, the imaging device receives at least one of sensing specification related information related to a sensing specification of the sensing means and parallax related information related to a parallax between the first imaging means and the sensing means; the display means displays the framing assist information based on at least one of the sensing specification related information and the parallax related information.

5. The imaging device according to claim 4.

6. the display means changes a timing for switching between displaying and not displaying the framing assist information depending on a magnitude of shaking of the main subject or a magnitude of hand shaking of a user who uses the imaging device.

6. The imaging device according to claim 1, wherein the imaging device further comprises:

7. a camera shake detection unit for detecting a magnitude of camera shake caused by a user of the imaging device; the display means displays the framing assist information when it is determined that there is a deviation between the direction of the shake of the main subject and the direction of the shake detected by the shake detection means.

6. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

8. when it is determined that there is a deviation between the direction of the shake of the main subject and the direction of the camera shake detected by the camera shake detection means, the display means displays the framing assist information as it is predicted that the main subject will no longer be present on the display screen.

6. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

9. when it is determined that the main subject will no longer be present in a predetermined area of ​​the display screen or when it is determined that a part of the main subject will no longer be present on the display screen, the display means displays the framing assist information as it is predicted that the main subject will no longer be present on the display screen.

6. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

10. when detecting that the main subject has changed position on the display screen by a distance equal to or greater than a predetermined distance, the display means displays the framing assist information as it is predicted that the main subject will no longer be present on the display screen.

6. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

11. the sensing means starts at least one of focus detection control and exposure control based on information about the main subject when the main subject is detected by the first imaging means; 6. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

12. when the main subject is detected by the sensing means, the first imaging means starts at least one of focus detection control and exposure control based on the main subject information; 6. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

13. the sensing means is capable of acquiring the main subject information; a notification unit that notifies the user of an error by at least one of a display, a sound, and a vibration when a process of acquiring the main subject information by the sensing unit is obstructed by a user in a state in which the main subject is not present on the display screen.

6. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

14. the display means does not display the framing assist information when the sensing means is unable to sense the main subject in a situation where the main subject is not present on the display screen.

6. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

15. the display means displays a sensing result of a sensing area, which is an area where the sensing means can perform sensing, as the framing assist information by superimposing it on the display screen.

6. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

16. the display means displays at least one of information indicating a direction of the main subject and information indicating a distance from the main subject to the imaging region as the framing assist information by superimposing the information on the display screen based on the sensing result.

6. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

17. The sensing means is at least one of a second imaging means separate from the first imaging means, a heat detection means, a reflected light receiving means, and a reflected sound receiving means.

6. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

18. The second imaging means is capable of outputting an imaging signal and a signal from which phase difference information or distance information can be obtained.

20. The imaging device according to claim 17,

19. a first focus detection means corresponding to the first imaging means; a second focus detection means corresponding to the sensing means; Further comprising: the first focus detection means performs focus detection using a result of the focus detection performed by the second focus detection means; the second focus detection means performs focus detection using a result of the focus detection performed by the first focus detection means; 6. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

20. a first exposure control means corresponding to the first imaging means; a second exposure control means corresponding to the sensing means; Further comprising: the first exposure control means performs exposure control using a result of the exposure control performed by the second exposure control means; the second exposure control means performs exposure control using a result of the exposure control performed by the first exposure control means.

6. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

21. the display means displays the framing assist information in a direction opposite to the main subject with respect to a direction in which the main subject comes into the frame, a vertical direction, or a horizontal direction.

6. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

22. A first imaging means capable of imaging an imaging area; a display unit that displays image data acquired by the first imaging unit as a display screen, a sensing result acquisition step of acquiring a sensing result from a sensing means capable of sensing a sensing area wider than the imaging area; a main subject information acquisition step of acquiring main subject information, which is information regarding a main subject, from the image data; a display step of displaying framing assist information for assisting in framing the main subject based on the sensing result by the sensing means when the main subject is not present on the display screen or when it is predicted that the main subject will not be present on the display screen; 13. A method for controlling an imaging apparatus comprising:

Images