Imaging device and information processing method

Abstract

To provide an imaging device capable of performing suitable processing for imaging.SOLUTION: An imaging device includes: an imaging unit; a distance detection unit that detects a first distance to a first position in an imaging range and a second distance to a second position in the imaging range; a calculation unit that calculates a third distance between the first position and the second position, a fourth distance between the first position and the second position on an imaging plane of the imaging unit, and a first angle of a line connecting the first position and the second position with respect to a plane parallel to the imaging plane; an acquisition unit that acquires first information on an angle of view of an optical system, second information on a size of a virtual subject, and third information on a distance to a position at which the virtual subject is to be placed; a first generation unit that generates fourth information indicating a size of an image of when the virtual subject is imaged by the imaging unit, and generates information indicating a rotation angle of the virtual subject on the basis of the first angle; and a second generation unit that generates image data for rotating an image of the fourth information by only an angle indicating the rotation angle of the virtual subject and displaying the image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging device and an information processing method.

Background Art

[0002] There is known a camera that arranges and synthesizes a virtual subject image cut out from a captured image of a real subject in a live view screen (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] According to a first aspect, an imaging device includes an imaging unit that images an image formed by an optical system, a distance detection unit that detects a distance to a position near a virtual subject, a first distance to a first position in an imaging range of the imaging unit, and a second distance to a second position in the imaging range, a calculation unit that calculates a third distance between the first position and the second position from the first distance and the second distance, a fourth distance between the first position and the second position on an imaging surface of the imaging unit, and a first angle of a line connecting the first position and the second position with respect to a plane parallel to the imaging surface, an acquisition unit that acquires first information regarding an angle of view of the optical system, second information regarding a size of the virtual subject, and third information regarding a distance to a position where the virtual subject is arranged, a first generation unit that generates fourth information indicating a size of an image when the virtual subject is imaged by the imaging unit based on the first information, the second information, and the third information, and generates information indicating a rotation angle of the virtual subject based on the first angle, and a second generation unit that generates image data for displaying the image by rotating the image by an angle indicating the rotation angle of the virtual subject. The virtual subject based on The imaging device further includes a second generation unit that generates image data for displaying the image by rotating the image by an angle indicating the rotation angle of the virtual subject. According to the second embodiment, the information processing method includes: capturing an image formed by an optical system with an imaging unit; detecting the distance to a position near the virtual subject, which is a first distance to a first position within the imaging range of the imaging unit and a second distance to a second position within the imaging range; calculating the third distance between the first position and the second position from the first distance and the second distance, the fourth distance between the first position and the second position on the imaging surface of the imaging unit, and the first angle of the line connecting the first position and the second position with respect to a plane parallel to the imaging surface; acquiring first information relating to the field of view of the optical system, second information relating to the size of the virtual subject, and third information relating to the distance to a position where the virtual subject is placed; generating fourth information indicating the size of the image when the virtual subject is imaged based on the first information, the second information, and the third information, and generating information indicating the rotation angle of the virtual subject based on the first angle; and the fourth information The virtual subject based on This includes generating image data for displaying an image rotated by an angle indicating the rotation angle of the virtual subject. [Brief explanation of the drawing]

[0005] [Figure 1] This figure shows an example of the configuration of an imaging device according to an embodiment. [Figure 2] This flowchart shows an example of the operation of the imaging device according to the embodiment. [Figure 3] This is a diagram illustrating an example of processing performed by an imaging device according to an embodiment. [Figure 4] This is a diagram illustrating another example of processing by the imaging device according to the embodiment. [Figure 5] This diagram illustrates an example of processing performed by an imaging device in a modified form. [Modes for carrying out the invention]

[0006] (Embodiment) Figure 1 shows an example of the configuration of a camera 1, which is an example of an imaging device according to an embodiment. The camera 1 has a camera body 2 and a lens unit 3, which is an accessory that can be attached to the camera body 2. The lens unit 3 is an interchangeable lens. Note that the camera 1 may be a camera in which the camera body 2 and the lens unit 3 are integrally configured.

[0007] The lens unit (interchangeable lens) 3 is detachably attached to the camera body 2 by a mount unit (not shown). When the interchangeable lens 3 is attached to the camera body 2, multiple terminals provided on the body-side connection unit 202 and multiple terminals provided on the lens-side connection unit 302 are electrically connected. This enables power supply from the camera body 2 to the interchangeable lens 3 and communication between the camera body 2 and the interchangeable lens 3.

[0008] The camera 1 according to this embodiment generates information about a virtual subject that can be used to determine the composition (hereinafter referred to as virtual subject information). The virtual subject may be an avatar of a subject to be photographed (candidate), an object representing the size of the subject's area, or other objects that can substitute for a subject. In camera 1, the camera body 2 and the interchangeable lens 3 communicate to generate the virtual subject information. Camera 1 can overlay an image of the virtual subject based on the virtual subject information onto the image obtained by shooting.

[0009] The interchangeable lens 3 comprises an optical system 31, a lens control unit 32, and a lens memory 33. The optical system 31 is an imaging optical system (imaging optical system) 31 having a plurality of lenses including a focus lens (focus adjustment lens) and an aperture (aperture diaphragm), and forms an image of the subject on the imaging unit 21 of the camera body 2.

[0010] The lens control unit 32 has a processor and memory, and controls each part of the interchangeable lens 3 based on a program. The lens control unit 32 has devices such as a CPU, FPGA, and ASIC, and memory such as ROM and RAM. The lens control unit 32 can also be called an information processing unit that performs information processing based on a program. When the lens control unit 32 receives a signal from the body control unit 25 of the camera body 2 regarding the direction and amount of movement of the focus lens, it moves the focus lens forward and backward in the direction of the optical axis L based on that signal to adjust the focal position of the photographic optical system 31. The lens control unit 32 also controls the aperture diameter based on a signal output from the body control unit 25.

[0011] The lens control unit 32 detects the position of the focus lens in the optical axis L direction. The lens control unit 32 detects the position of the focus lens based on a signal output from an encoder (not shown), for example. The lens control unit 32 also detects the aperture diameter. The lens control unit 32 transmits information regarding the position of the focus lens, information regarding the aperture value (F number), etc., to the body control unit 25.

[0012] The lens memory 33 is composed of a non-volatile storage medium or the like. Information related to the interchangeable lens 3 is stored (recorded) in the lens memory 33. The lens memory 33 stores data on the infinity and close positions of the focus lens, data on the shortest and longest focal lengths and angle of view of the interchangeable lens 3, data on the aperture value, etc. The lens control unit 32 writes data to the lens memory 33 and reads data from the lens memory 33.

[0013] Next, an example of the configuration of the camera body 2 will be described. The camera body 2 comprises an imaging unit 21, a memory 22, a display unit 23, an operation unit 24, and a body control unit 25. The imaging unit 21 is an image sensor such as a CMOS image sensor or a CCD image sensor. The imaging unit (image sensor) 21 receives a light beam that has passed through the imaging optical system 31 and captures an image of the subject formed by the imaging optical system 31. The image sensor 21 has a plurality of pixels arranged in two dimensions, each having a photoelectric conversion unit. The photoelectric conversion unit is composed of a photodiode and converts incident light into electric charge. The image sensor 21 generates a signal by photoelectric conversion of the received light and outputs the generated signal to the body control unit 25.

[0014] The image sensor 21 has pixels that output signals used for image generation (imaging pixels) and pixels that output signals used for focus detection (AF pixels). The imaging pixels are arranged according to a Bayer array. The AF pixels are arranged by replacing some of the imaging pixels and are distributed over almost the entire surface of the image plane (imaging surface) of the image sensor 21.

[0015] Memory 22 is composed of a non-volatile storage medium or the like. Memory 22 stores image data, programs and data used to control each part of camera 1, etc. Memory 22 (or the internal memory of the body control unit 25) also stores data related to virtual subjects, data related to the size of the image sensor 21, etc. The body control unit 25 writes data to memory 22 and reads data from memory 22.

[0016] The display unit 23 is a liquid crystal display, an organic EL display, etc. The display unit 23 displays a live view image of the subject, an image based on image data stored in memory 22, an image indicating the focus detection area (AF area) such as an AF frame, shooting-related information such as shutter speed and aperture value, and a menu screen, etc. The display unit 23 may include a touch panel and may also function as an input / output unit. The display unit (input / output unit) 23 may generate signals based on user operation and output them to the body control unit 25. The display unit 23 is located, for example, on the back of the camera body 2 or inside a viewfinder (not shown).

[0017] The control unit 24 includes components such as a release button, a power button (switch), operation buttons, and switches for switching between various modes, and receives operations on the camera 1. The control unit 24 detects operations by the user and outputs a signal based on the operation to the body control unit 25. The control unit 24 may include the touch panel of the display unit 23.

[0018] The body control unit 25 has a processor and memory, and controls each part of the camera 1 based on a program. The body control unit 25 has devices such as a CPU, FPGA, and ASIC, and memory such as ROM and RAM. The body control unit 25 can also be called an information processing unit that performs information processing based on a program. The body control unit 25 supplies signals to the image sensor 21 to control the image sensor 21 and controls the operation of the image sensor 21. When taking still images, taking video, or displaying a through image on the display unit 23, the body control unit 25 causes the image sensor 21 to capture an image of the subject and outputs pixel signals.

[0019] The body control unit 25 includes a focus detection unit 26, a distance detection unit 27, an acquisition unit 28, and a generation unit 29. The focus detection unit 26 performs focus detection processing necessary for autofocus (AF) of the imaging optical system 31. The focus detection unit 26 calculates the defocus amount by the phase difference detection method using the signals of the AF pixels output from the imaging device 21. The focus detection unit 26 calculates the movement amount of the focus lens to the in-focus position based on the defocus amount. The focus lens is moved according to the movement amount, and focus adjustment is performed. Thus, the focus detection unit 26 controls the position of the focus lens so that the image of the subject by the imaging optical system 31 is in focus (imaged) on the imaging device 21.

[0020] The focus detection unit 26 can perform focus detection processing by the contrast detection method instead of or in addition to the focus detection processing by the phase difference detection method. The focus detection unit 26 sequentially calculates the focus evaluation value (contrast evaluation value) of the subject image based on the signals of the pixels output from the imaging device 21 while moving the focus lens in the optical axis direction. The focus detection unit 26 associates the position of the focus lens with the focus evaluation value, and calculates the position of the focus lens where the focus evaluation value shows a peak, that is, a maximum value, as the in-focus position. Based on the calculation result, the focus lens is moved to the in-focus position, and focus adjustment is performed.

[0021] The distance detection unit 27 detects the distance to the target and generates information (distance information) regarding the distance between the camera 1 and a specific position. In the present embodiment, the distance detection unit 27 detects the distance from the camera 1 to the position where the virtual subject is to be arranged, and generates distance information regarding the distance between the camera 1 and the position where the virtual subject is to be arranged. The distance information generated by the distance detection unit 27 can also be said to be information representing the distance between the position in the real space corresponding to the position where the virtual subject is arranged in the image and the camera 1. The position where the virtual subject is to be arranged is determined by the user or the camera 1. For example, the user determines the position where the virtual subject is to be arranged within the imaging range by designating a position within the image displayed on the display unit 23 by operating the operation unit 24, operating the touch panel, or the like. The user can select the position where the subject to be photographed is assumed to be located as the arrangement position of the virtual subject. For example, distance detection may be performed using an encoder (not shown). As an example, if the virtual subject is a person performing a track and field event, the position where the virtual subject is to be arranged is, for example, the ground where the feet of the person to be arranged as the virtual subject should touch the ground. Also, when the virtual subject is a vehicle running on a track (rail), the position where the virtual subject is to be arranged is, for example, the rail portion where the wheels of the vehicle to be arranged as the virtual subject contact the rail.

[0022] The distance detection unit 27 obtains the distance between the position where the selected virtual subject is to be arranged and the camera 1 and generates distance information. The distance detection unit 27, for example, uses the position information of the focus lens when the focus lens is at the in-focus position to calculate the distance between the camera 1 and the position of the virtual subject and generates distance information. The distance detection unit 27 may generate distance information by using the focus information of the imaging optical system 31 such as the defocus amount, the focus evaluation value, or the like. Also, a device such as a ToF sensor or a Lidar may be provided in the camera 1, and distance information may be generated based on the signal output from the device. Further, the user may operate the operation unit 24 or the display unit (input / output unit) 23 to directly input the distance information.

[0023] The acquisition unit 28 acquires information regarding the focal length of the interchangeable lens 3 and distance information regarding the distance to the position where the virtual subject is placed. The acquisition unit 28 acquires information regarding the focal length from the interchangeable lens 3 and distance information regarding the virtual subject from the distance detection unit 27. In addition, the acquisition unit 28 calculates the angle of view based on the focal length and the size of the image sensor 21 and acquires information regarding the angle of view (angle of view information). The acquisition unit 28 may also acquire the angle of view information from the lens control unit 32 and lens memory 33 of the interchangeable lens 3.

[0024] The acquisition unit 28 acquires information regarding the size of the expected subject (for example, the actual size of the person to be photographed) as information regarding the size of the virtual subject (hereinafter referred to as "actual size information"). The actual size information represents the size of the virtual subject in real space and is set by the user or camera 1. The size of the subject in real space is input by the user through operations on the operation unit 24, etc., and the actual size information of the virtual subject is generated. If an avatar is used for the virtual subject, the actual size of the subject may be determined from the avatar information and the actual size information of the virtual subject may be acquired. The acquisition unit 28 also serves as an input unit into which distance information, etc., is input. The actual size information is, for example, the height if the virtual subject is a person, and the total length or total height if it is a vehicle. Alternatively, the avatar size may be changed by directly inputting the avatar size.

[0025] The generation unit 29 has a first generation unit 29a and a second generation unit 29b. The first generation unit 29a generates virtual subject information relating to a virtual subject that can be used for composition determination. The first generation unit 29a generates virtual subject information based on distance information and actual size information of the virtual subject acquired by the acquisition unit 28, field of view information, and information relating to the size of the image sensor 21. The virtual subject information is information relating to the size of the image when the virtual subject is captured, and can also be said to be information representing the size of the image area of ​​the virtual subject. The virtual subject information includes information relating to the image of the virtual subject (e.g., an avatar image), information relating to the position of the virtual subject, etc., and is used for image display. The virtual subject information may also include information relating to an object (e.g., an image of a rectangular or elliptical frame) corresponding to the size of the image area of ​​the avatar.

[0026] The first generation unit 29a calculates the size of the image of the virtual subject when the virtual subject is captured, based on the distance from the camera 1 to the position of the virtual subject, the actual size of the virtual subject, the field of view, and the size of the image sensor 21, using the following equation (1). The size of the virtual subject image = size of the image sensor × actual size of the virtual subject / (distance to the virtual subject × tanθ1) …(1)

[0027] In equation (1) above, angle θ1 = field of view / 2. The size of the image sensor 21 is, for example, the diagonal length of the imaging surface of the image sensor 21 (or half the diagonal length). The size of the virtual subject image obtained by the first generation unit 29a is the size of the area on the imaging surface of the image sensor 21 when an image of the virtual subject is formed, and corresponds to the size of the area of ​​the virtual subject in the image based on the signal of each pixel of the image sensor 21. The size of the virtual subject image may also be calculated by converting it to the number of pixels. The first generation unit 29a generates virtual subject information that includes information indicating the size of the calculated virtual subject image. The information in the above calculation equation (1) is stored in the memory 22 (or the internal memory of the body control unit 25).

[0028] The second generation unit 29b performs various image processing on the signals of the imaging pixels output from the image sensor 21 to generate image data including the signals of each pixel. The second generation unit 29b may also use the signals of the AF pixels of the image sensor 21 to generate image data. The second generation unit 29b is an image processing unit 29b, which performs image processing such as color interpolation and gradation conversion.

[0029] The second generation unit 29b can generate image data relating to an image (imported image) based on the signals of each pixel of the image sensor 21 obtained by imaging, and an image of a virtual subject based on the virtual subject information described above. The second generation unit 29b generates image data for displaying an image of a virtual subject (for example, an avatar image) of a size corresponding to the size indicated by the virtual subject information.

[0030] The display unit 23 displays the captured image obtained by imaging and the image of the virtual subject superimposed on each other, based on the image data generated by the second generation unit 29b. The display unit 23 displays an image of the virtual subject, whose size is determined according to the actual size of the virtual subject and the distance to the virtual subject, superimposed on the captured image at a position specified by the user.

[0031] As described above, the camera 1 according to this embodiment can calculate the size of a virtual subject when it is captured and display an image of the virtual subject corresponding to the calculated size. The user can check the composition and appearance of the image in advance by looking at the image displayed on the display unit 23, without having to place an actual subject. For example, even if there is only one person when setting up the camera 1 for remote shooting, it is possible to check the composition.

[0032] Figure 2 is a flowchart illustrating an example of the operation of the imaging device according to the embodiment. Figure 3 is a diagram illustrating an example of processing by the imaging device according to the embodiment. Below, an example of the operation of camera 1 will be described using the case of checking the composition before photographing a track and field event.

[0033] In step S100 shown in Figure 2, the body control unit 25 of camera 1 determines the image to be used as the image of the virtual subject based on the user's operation of the operation unit 24, and also acquires the actual size information of the virtual subject. The body control unit 25 sets the image and actual size of the virtual subject to be, for example, the image and actual size of the avatar of athlete 10a who is predicted to be in the lead during the competition.

[0034] In step S110, the body control unit 25 determines the position selected by the user within the through image displayed on the display unit 23 as the position to place the virtual subject. In the example shown in Figure 3(a), the user predicts the shooting scene (shooting situation) and selects position A1, where the leading player 10a is expected to be located, as the position to place the virtual subject. For example, the position at the feet of the leading player 10a may be selected as the position to place the virtual subject. Once the position to place the virtual subject is set, distance measurement is performed with the virtual subject position A1 as the distance measurement point, and the distance between the virtual subject position A1 and the camera 1 is measured. The body control unit 25 acquires distance information indicating the distance between the virtual subject position A1 and the camera 1.

[0035] In step S120, the body control unit 25 calculates the size of the virtual subject image based on the distance information and actual size information of the virtual subject, the angle of view information, the size information of the image sensor 21, and the above-described equation (1). The body control unit 25 generates virtual subject information that includes information indicating the calculated size of the virtual subject image.

[0036] In step S130, the body control unit 25 displays on the display unit 23 an image of the virtual subject with a size corresponding to the size calculated based on the virtual subject information, and a through image, which is the captured image obtained by imaging. The virtual subject information represents the size of the virtual subject image relative to the through image and is reflected in the image displayed on the display unit 23. In the example schematically shown in Figure 3(b), the display unit 23 overlays the image of the athlete 10a's avatar onto the through image obtained by photographing the track of the stadium.

[0037] When multiple virtual subjects are placed, camera 1 repeats the series of operations from step S110 to step S130. The body control unit 25 sets, for example, the image and actual size of the avatar of the player of interest 10b. As shown in the example in Figure 3(a), the position B1 where the player of interest 10b is predicted to be located is specified, and the distance between the virtual subject placement position B1 and camera 1 is measured. The body control unit 25 calculates the size of the virtual subject image based on distance information indicating the distance between the virtual subject placement position B1 and camera 1, actual size information, field of view information, information on the size of the image sensor 21, and the above-described equation (1), and generates virtual subject information for player 10b. Alternatively, distance information indicating the distance between the virtual subject placement position B1 and camera 1 may be obtained based on the focus information when the optical system of camera 1 is focused on the placement position A1 of virtual subject 10a, the distance between camera 1 and the placement position A1 of subject 10a, and the focus information of the virtual subject placement position B1 at that time.

[0038] As shown in Figure 3(b), the body control unit 25 displays an image based on the virtual subject information of player 10b on the display unit 23. In the example shown in Figure 3(b), the display unit 23 overlays the avatar image of the leading player 10a and the avatar image of the featured player 10b onto the through image. Alternatively, as shown in Figure 3(c), the display unit 23 may display an object (for example, a rectangular frame image) corresponding to the size of the avatar's image area. The user can check the composition and appearance in advance by looking at the image displayed on the display unit 23.

[0039] Normally, when setting up a camera in advance for remote filming of track and field events, it is possible that the composition cannot be checked if the athletes are not present and there are no collaborators (for example, other photographers). On the other hand, as described above, camera 1 according to this embodiment can superimpose and display an image of a virtual subject of a size determined according to the distance to the virtual subject onto the captured image. Therefore, it is possible to check the composition even when there is only one person present, without having other photographers or others stand in the positions where the athletes are likely to pass.

[0040] In this embodiment, it becomes possible to check in advance what size the image will be when the athletes actually run the predetermined course, allowing for confirmation and adjustment of the composition. Furthermore, the composition can be determined by considering the relative positions of multiple athletes and the expected shooting results. This avoids the need to repeatedly have a collaborator stand in the area where the athletes will pass near the finish line to adjust the composition.

[0041] Next, another example of operation of the imaging device according to the embodiment will be described using Figure 4. Figure 4 is a diagram illustrating an example of processing by camera 1, schematically showing an example of checking the composition before shooting an equestrian competition. The body control unit 25 calculates the size of the virtual subject image based on the distance information and actual size information of the virtual subject (the avatar of the horse 15 in Figure 4), the field of view information, and the size information of the image sensor 21, similar to the example described using Figures 2 and 3.

[0042] Camera 1 selects multiple positions within the imaging range of Camera 1 as distance measurement points. The user selects, for example, the position of a virtual subject and positions in the vicinity (around) of the virtual subject as distance measurement points in the image displayed on the display unit 23. In the example shown in Figure 4(a), the position A2 of one of two obstacles lined up one behind the other and the position B2 of the other obstacle are selected as distance measurement points. The distance detection unit 27 detects the distance from Camera 1 to position A2 and generates distance information regarding the distance between Camera 1 and position A2. The distance detection unit 27 also detects the distance from Camera 1 to position B2 and generates distance information regarding the distance between Camera 1 and position B2.

[0043] Figure 4(c) schematically shows the relationship between the distance d1 between position A2 and position B2, the distance d2 between position A2 and position B2 on the imaging surface of the image sensor 21, and the angle θ2 of the line connecting position A2 and position B2 with respect to a plane parallel to the imaging surface of the image sensor 21. The distance d1 between position A2 and position B2 can be expressed by the following equation (2). The distance d1 between position A2 and position B2 is approximately equal to (distance from camera 1 to position A2) - (distance from camera 1 to position B2) ... (2) Furthermore, the angle θ2 of the line connecting position A2 and position B2 with respect to the plane parallel to the imaging plane of the image sensor 21 can be expressed by the following equation (3). cosθ2 = (d2 distance between position A2 and position B2 on the imaging surface of the image sensor 21) / distance d1 …(3)

[0044] The first generation unit 29a of the body control unit 25 calculates the angle θ2 using the above-described equation (3) and determines the rotation angle of the virtual subject based on the calculated angle θ2. The first generation unit 29a can also be said to determine the angle (tilt) of the virtual subject with respect to the optical axis L of the camera 1. The first generation unit 29a generates virtual subject information that includes information indicating the rotation angle of the determined virtual subject image and information indicating the size of the virtual subject image. The second generation unit 29b generates image data for displaying an image of a virtual subject (an avatar image of the horse 15 in Figure 4) of a size corresponding to the size indicated by the virtual subject information, rotated by the angle indicated by the virtual subject information.

[0045] The display unit 23 can superimpose and display the captured image obtained by imaging and the image of the virtual subject based on the image data generated by the second generation unit 29b. In the example shown in Figure 4(b), the display unit 23 superimposes an image of the horse 15 avatar, rotated according to the angle calculated as described above, onto a through image obtained by photographing the stadium. Camera 1 adjusts the angle (tilt) of the virtual subject, allowing the image of the horse 15 avatar jumping over obstacles to be positioned at an appropriate angle and orientation. In this embodiment, it is possible to position virtual subjects such as the horse jumping over obstacles and vehicles running on rails, taking depth into consideration. It becomes possible to deform the image to give it depth (for example, to taper towards the rear) according to the focal length information of the lens.

[0046] According to the above-described embodiment, the following effects and advantages can be obtained. (1) The imaging device (camera 1) includes an imaging unit (image sensor 21) that captures an image formed by an optical system, an acquisition unit (acquisition unit 28) that acquires first information regarding the field of view of the optical system, second information regarding the size of a virtual subject, and third information regarding the distance to the position where the virtual subject is placed, and a generation unit (generation unit 29) that generates fourth information regarding the size of the image when the virtual subject is captured by the imaging unit, based on the first information, second information, and third information. In this embodiment, the generation unit 29 generates virtual subject information representing the size of the virtual subject image based on the distance information and actual size information of the virtual subject, the field of view information, and the size information of the image sensor 21. The display unit 23 displays an image of the virtual subject based on the virtual subject information. Therefore, the user can check the composition and how well the image is captured by the image displayed on the display unit 23.

[0047] (2) The generation unit 29 according to this embodiment can generate virtual subject information including information indicating the rotation angle of the virtual subject image. The display unit 23 displays an image of the virtual subject rotated according to the rotation angle indicated by the virtual subject information. Therefore, the captured image and the image of the virtual subject can be displayed in an appropriate superimposed manner.

[0048] The following modifications are also within the scope of the present invention, and it is possible to combine one or more of these modifications with the embodiments described above.

[0049] (Variation 1) The camera body control unit 25 may adjust the amount of blur applied to the image of the virtual subject in accordance with information regarding the aperture value and the focus information of the optical system 31. The generation unit 29 of the body control unit 25 calculates the amount of blur of the image of the virtual subject based on, for example, the aperture value and the amount of defocus when the virtual subject is in focus, and changes the amount of blur of the image of the virtual subject. In this case, the generation unit 29 may apply the PSF (Point spread function) determined according to the aperture value and the amount of defocus to the image of the virtual subject and display it on the display unit 23. The user can check the composition and image quality using the image of the virtual subject with the blur effect applied and the captured image.

[0050] The camera 1 in this modified version controls and displays the amount of blur on the avatar image according to the aperture value, making it possible to simulate in advance the aperture value (aperture amount) required to obtain the desired depth of field. For avatar images placed in out-of-focus positions, PSF is applied based on the amount of defocus and aperture value, and blur is added. This makes it possible to predict the state of the captured image of subjects that are expected to be located outside the area to be in focus. It becomes possible to check in advance how much to stop down the aperture of the imaging optical system 31 in order to fit multiple subjects within the depth of field, without actually placing the subjects.

[0051] (Modification 2) The body control unit 25 of camera 1 may deform the shape of the image of the virtual subject based on the elevation angle of camera 1. The body control unit 25 detects the elevation angle of camera 1 based on a signal output from, for example, an angle sensor of a spirit level (not shown). The generation unit 29 of the body control unit 25 calculates the distance from camera 1 to the ground based on the distance from camera 1 to the position of the virtual subject and the elevation angle θ3 of camera 1 using the following equation (4). Distance between camera 1 and the ground = (distance to virtual subject) × sinθ³ …(4)

[0052] The generation unit 29 calculates the distance between camera 1 and the ground using the equation (4) described above. The generation unit 29 determines the viewing direction (downward, upward, or straight ahead) of the virtual subject based on the distance between camera 1 and the ground and the actual size of the virtual subject, and generates virtual subject information that includes information about the shape of the virtual subject. The virtual subject information can also be described as information indicating the amount of deformation of the image of the virtual subject. The generation unit 29 generates image data for displaying the image of the virtual subject that has been deformed according to the virtual subject information.

[0053] The display unit 23 overlays the distorted image of the virtual subject onto the captured image based on the image data. The body control unit 25 may distort the image of the virtual subject displayed on the display unit 23, for example, the image of an avatar, into a trapezoidal shape according to the elevation angle of the camera 1. When remote shooting is performed with the camera installed at a low altitude near the ground or at a high altitude such as the ceiling, the image of the virtual subject can be distorted into a trapezoidal shape according to the elevation angle of the camera body. In this modified example, the shape of the virtual subject is changed according to the elevation angle of the camera 1, making it possible to display the image of the virtual subject more accurately.

[0054] The body control unit 25 may adjust the shape of the image of the virtual subject according to the distance between the camera 1 and the position of the virtual subject. Figure 5 is a diagram illustrating an example of processing by the camera 1 according to a modified example. In the example shown in Figure 5, avatars of people in the first to fifth rows (people 20a in the first row to people 20e in the fifth row) are placed as virtual subjects, and an example of checking the composition of a group photograph is schematically shown.

[0055] The generation unit 29 determines the amount of deformation of the virtual subject's image based on the elevation angle of the camera 1 and the distance from the camera 1 to the ground position where the virtual subject is standing, and generates virtual subject information relating to the deformed image of the virtual subject. The generation unit 29 generates virtual subject information relating to the shape of the avatar of person 20a determined according to the distance to position C1, and virtual subject information relating to the shape of the avatar of person 20b determined according to the distance to position C2.

[0056] Furthermore, the generation unit 29 generates virtual subject information relating to the shape of the avatar of person 20c determined according to the distance to position C3, virtual subject information relating to the shape of the avatar of person 20d determined according to the distance to position C4, and virtual subject information relating to the shape of the avatar of person 20e determined according to the distance to position C5. Based on the virtual subject information generated for each person, the generation unit 29 displays the images of each avatar of person 20a to person 20e on the display unit 23. The shape of the images of each avatar in the first to fifth rows is adjusted according to the placement position of the virtual subjects, making it possible to display the images of the virtual subjects more accurately.

[0057] (Variation 3) The user may be able to select whether or not to display the captured image and the image of the virtual subject superimposed on each other by operating the control unit 24, operating the touch panel, etc. The display unit 23 may not display the captured image and the image of the virtual subject superimposed on each other, but may display only one of them by operating the control unit 24, etc.

[0058] (Modification 4) The user may adjust the position of the virtual subject, the shape of the virtual subject, the orientation of the virtual subject, etc., by operating the control unit 24, operating the touch panel, etc.

[0059] (Variation 5) The functions of the imaging device may be realized by having a computer (processor) execute a program that performs the processing described in the above-described embodiments and modifications. The program can be supplied as a computer program product in various forms, such as through a storage medium or a communication line.

[0060] Although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention. [Explanation of symbols]

[0061] 1...Camera, 2...Camera body, 3...Interchangeable lens, 21...Image sensor, 23...Display unit, 25...Body control unit, 27...Distance detection unit, 28...Acquisition unit, 29...Generation unit, 31...Photography optical system, 32...Lens control unit

Claims

1. An imaging unit that captures an image formed by an optical system, A distance detection unit that detects the distance to a position near a virtual subject, which includes a first distance to a first position within the imaging range of the imaging unit and a second distance to a second position within the imaging range. A calculation unit that calculates a third distance between the first position and the second position from the first distance and the second distance, a fourth distance between the first position and the second position on the imaging surface of the imaging unit, and a first angle of the line connecting the first position and the second position with respect to a plane parallel to the imaging surface, An acquisition unit that acquires first information relating to the angle of view of the optical system, second information relating to the size of the virtual subject, and third information relating to the distance to the position where the virtual subject is placed. A first generation unit generates fourth information indicating the size of the image when the virtual subject is captured by the imaging unit, based on the first information, second information, and third information, and generates information indicating the rotation angle of the virtual subject based on the first angle. A second generation unit generates image data for displaying the image of the virtual subject based on the fourth information, rotated by an angle indicating the rotation angle of the virtual subject; An imaging device equipped with the following features.

2. In the imaging apparatus according to claim 1, It has a display unit that displays images, The display unit is an imaging device that displays an image of the virtual subject based on information indicating the rotation angle of the virtual subject and the fourth information.

3. In the imaging device according to claim 2, The display unit is an imaging device that displays an image based on a signal output from the imaging unit and an image of the virtual subject based on the fourth information superimposed on it.

4. In the imaging apparatus according to claim 2 or claim 3, The display unit is an imaging device that displays an image of an avatar or an object corresponding to the size of the avatar's area as an image of the virtual subject.

5. In the imaging apparatus according to any one of claims 1 to 4, A generating unit comprising the first generating unit and the second generating unit, The acquisition unit acquires fifth information relating to the aperture value of the optical system, The generation unit is an imaging device that calculates the amount of blur to be applied to the image of the virtual subject based on the fifth information and the focus information of the optical system.

6. The process involves capturing an image formed by an optical system using an imaging unit, The distance to a position near the virtual subject is to detect a first distance to a first position within the imaging range of the imaging unit and a second distance to a second position within the imaging range. A calculation unit that calculates a third distance between the first position and the second position from the first distance and the second distance, a fourth distance between the first position and the second position on the imaging surface of the imaging unit, and a first angle of the line connecting the first position and the second position with respect to a plane parallel to the imaging surface, The first information relating to the angle of view of the optical system, and the second information relating to the size of the virtual subject, To obtain third information regarding the distance to the position where the virtual subject is placed, Based on the first information, the second information, and the third information, a fourth piece of information is generated indicating the size of the image when the virtual subject is captured, and information indicating the rotation angle of the virtual subject is generated based on the first angle. To generate image data for displaying the image of the virtual subject based on the fourth information, rotated by an angle indicating the rotation angle of the virtual subject, Information processing methods including

Images