Imaging device and program

Abstract

To make it possible to easily perform focus adjustment operation in an imaging device capable of expanding a depth of field.SOLUTION: An imaging device 1 includes: a depth-of-field expansion processing unit 15 that generates one depth-of-field expanded image in which a depth-of-field is expanded using a set of sub-frames captured at first speed; a focus adjustment image selection unit 16 that selects one sub-frame from a set of sub-frames; and a frame rate conversion unit 17 that generates a frame rate conversion video in which a frame rate of the selected sub-frame is converted to second speed slower than the first speed and displays the generated video on a display unit 18.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging device and a program.

Background Art

[0002] Conventionally, means for displaying camera images on a viewfinder and allowing a camera operator to perform a focus adjustment operation has been carried out. In addition, a technique for rapidly acquiring images at a plurality of focal positions and synthesizing these images to expand the depth of field is known (see, for example, Patent Document 1 and Non-Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when the camera operator performs a focus adjustment operation while viewing the camera image through the viewfinder, it becomes difficult to perform the adjustment operation to the optimal focus position in an image with an expanded depth of field. In particular, the resolution of a small viewfinder is generally lower than that of the lens, and in such a case, the focus may appear to be in focus on the viewfinder but is actually out of focus. Since an image with a shallow focus is easier for the camera operator to perform a focus adjustment operation on, it is desirable that a location with correct focus and a location with a significant out-of-focus before and after it coexist.

[0006] In view of such circumstances, an object of the present invention made is to provide an imaging device and a program that have a function capable of expanding the depth of field and that enable an easy focus adjustment operation.

Means for Solving the Problems

[0007] The gist of the present invention for solving the above problems is as follows.

[0008] (1) An imaging device that photographs a subject using a lens and an image sensor, comprising: a depth-of-field expansion processing unit that generates one depth-of-field expanded image with an expanded depth of field using a set of sub-frames photographed at a first speed; a focus adjustment image selection unit that selects one sub-frame from the set of sub-frames; and a frame rate conversion unit that generates a frame rate conversion video obtained by converting the frame rate of the selected sub-frame to a second speed slower than the first speed and causes the display unit to display the video.

[0009] (2) The imaging device according to (1), further comprising a minute focus position control unit that calculates the depth of field at the position of the subject at the time of photographing the sub-frame and moves the focus of the lens so that the depths of field do not overlap for each sub-frame.

[0010] (3) The imaging device according to (2), further comprising a synchronization unit that synchronizes the timing of photographing the sub-frame and the timing of moving the focus of the lens.

[0011] (4) The imaging device according to (1) or (2), further comprising an autofocus unit that determines the focus position of the sub-frame having the largest amount of high-frequency components among the set of sub-frames, and a focus control unit that moves the focus position of the lens to the focus position determined by the autofocus unit before photographing the next set of sub-frames.

[0012] (5) A program for causing a computer to function as the imaging device according to any one of (1) to (4).

Advantages of the Invention

[0013] According to the present invention, it has a function of expanding the depth of field, and it is possible to easily perform a focus adjustment operation while viewing the viewfinder image.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0016] Figure 1 is a block diagram showing a configuration example of an imaging device 1 according to an embodiment of the present invention. The imaging device 1 shown in Figure 1 includes a condensing unit 11, an imaging element 12, an imaging processing unit 13, a storage unit 14, a depth of field expansion processing unit 15, a focus adjustment image selection unit 16, a frame rate conversion unit 17, a viewfinder (display unit) 18, a micro focus position control unit 19, a synchronization unit 20, an autofocus unit 21, and a focus control unit 22.

[0017] The condensing unit 11 has one or more lenses 111 and a micro focus position adjustment mechanism 112. The condensing unit 11 outputs lens information such as the focal length, focal position, and aperture value of the lens 111 to the micro focus position control unit 19.

[0018] The lens 111 condenses the light emitted from the subject and forms an image on the imaging element 12.

[0019] The imaging element 12 outputs the image data that has been photoelectrically converted to the imaging processing unit 13. At this time, the shooting speed is made higher (for example, 300 fps, shooting time per frame is 1 / 300 second) than the normal shooting speed (for example, 60 fps, shooting time per frame is 1 / 60 second). The frames shot at a speed (first speed) higher than this normal shooting speed (second speed) are referred to as "sub-frames" in this specification. Also, a sub-frame whose number is the value obtained by dividing the first speed by the second speed is referred to as a "set of sub-frames".

[0020] The micro focus position adjustment mechanism 112 finely adjusts the focal position of the lens 111 every time a sub-frame is shot.

[0021] Figure 2 shows a structural example of the micro focus position adjustment mechanism 112. In the example shown in Figure 2(a), the micro focus position adjustment mechanism 112 is a piezo element that can be finely moved at high speed and is fixed to the lens 111. The condensing unit 11 can adjust the focal position by moving the lens 111 in the optical axis direction within the range of the length a of the lens barrel 110, and can finely adjust the focal position by finely moving the piezo element in the optical axis direction within the range of the length b.

[0022] In the example shown in FIG. 2(b), the micro focus position adjustment mechanism 112 is a piezo element that can be finely moved at high speed and a lens (micro focus position changing lens) to which the piezo element is fixed. The condenser unit 11 can adjust the focus position by moving the lens 111 in the range of the length a' from one end of the lens barrel 110 to the micro focus position adjustment mechanism 112 in the optical axis direction, and can finely adjust the focus position by finely moving the piezo element in the range of the length b in the optical axis direction. Note that the micro focus position adjustment mechanism 112 may be on the image side or the object side of the lens 111.

[0023] In the example shown in FIG. 2(c), the micro focus position adjustment mechanism 112 is a lens 111 with a variable focal length that can be adjusted at high speed using ultrasonic waves or the like. The condenser unit 11 can adjust the focus position by moving the lens 111 (that is, the micro focus position adjustment mechanism 112) in the range of the length a of the lens barrel 110 in the optical axis direction, and can finely adjust the focus position by finely moving in the range of the length b in the optical axis direction.

[0024] In the example shown in FIG. 2(d), the micro focus position adjustment mechanism 112 is a lens (micro focus position changing lens) provided separately from the lens 111 and having a variable focal length at high speed. The condenser unit 11 can adjust the focus position by moving the lens 111 in the range of the length a' from one end of the lens barrel 110 to the micro focus position adjustment mechanism 112 in the optical axis direction, and can finely adjust the focus position by finely moving the micro focus position adjustment mechanism 112 in the range of the length b in the optical axis direction. Note that the micro focus position adjustment mechanism 112 may be on the image side or the object side of the lens 111. In the examples shown in FIGS. 2(b) and 2(d), an image is formed at the combined focus position of the lens 111 and the micro focus position changing lens.

[0025] The micro focus position adjustment mechanism 112 may be provided outside the condenser unit 11. In the example shown in Fig. 2(e), the micro focus position adjustment mechanism 112 is a piezo element that can be finely moved at high speed and is fixed to the imaging element 12. The condenser unit 11 can adjust the focus position by moving the lens 111 in the optical axis direction within the range of the length a of the lens barrel 110, and the micro focus position adjustment mechanism 112 can finely adjust the focus position by finely moving the piezo element in the optical axis direction within the range of the length b.

[0026] Alternatively, as a means for realizing the micro focus position adjustment function, the subject or the imaging device 1 may be finely moved at high speed in the optical axis direction using a piezo element or the like. Since the mechanism for moving the lens 111 within the ranges of lengths a and a' is known, such as a helicoid or a cam, the description thereof is omitted.

[0027] Referring to Fig. 1 again. The imaging processing unit 13 converts the image data input from the imaging element 12 into a two-dimensional image, performs necessary processing on the camera signal such as level adjustment and noise processing, and outputs it to the storage unit 14 as a sub-frame.

[0028] The storage unit 14 sequentially stores the sub-frames and outputs a set of sub-frames simultaneously within one frame time. Fig. 1 shows an example in the case of n sub-frames of a set of sub-frames sf(1), sf(2), sf(3), sf(4), ···, sf(n). The n sub-frames output from the storage unit 14 are input in parallel to the depth of field expansion processing unit 15, the focus adjustment image selection unit 16, and the autofocus unit 21.

[0029] The depth of field expansion processing unit 15 generates one depth-of-field expanded image with an expanded depth of field using existing techniques such as extracting and synthesizing the high-frequency signals of a set of sub-frames, and outputs it as a camera image outside the imaging device 1. As a method for synthesizing sub-frames, the synthesis method described in Non-Patent Document 1 or Patent Document 1 may be used.

[0030] The image selection unit 16 for focus adjustment selects one sub-frame from a set of sub-frames and outputs it to the frame rate conversion unit 17. For example, the image selection unit 16 for focus adjustment selects the sub-frame sf((n + 1) / 2) with the middle sub-frame number.

[0031] The frame rate conversion unit 17 generates a frame rate conversion video by converting the frame rate of the sub-frame selected by the image selection unit 16 for focus adjustment to a second speed (60 fps in this embodiment) slower than the first speed (300 fps in this embodiment), and outputs it to the viewfinder 18.

[0032] The viewfinder (display unit) 18 displays the frame rate conversion video so that the user (camera person) U can view it. The user U watches the video of the viewfinder 18, manually adjusts the focal position of the lens 111 as necessary, and outputs a manual focal position adjustment instruction to the focus control unit 22.

[0033] The micro focus position control unit 19 calculates the depth of field at the subject position based on the lens information input from the condenser 11 when shooting a sub-frame. Then, the micro focus position control unit 19 outputs a control signal for slightly moving the focal position to the condenser 11 in order to move the lens 111 so that the depths of field do not exactly overlap for each sub-frame. The condenser 11 slightly moves the focal position of the lens 111 by an amount corresponding to the depth of field each time a sub-frame is shot. For example, when the focal length of the lens 111 is long, the amount of slightly moving the focal position is decreased, and when the focal length of the lens 111 is short, the amount of slightly moving the focal position is increased.

[0034] The synchronization unit 20 synchronizes the timing of shooting each sub-frame by the imaging device 12 and the timing of the minute focus movement of the lens 111. Also, the synchronization unit 20 synchronizes the output of the camera video by the depth of field expansion processing unit 15 and the output to the viewfinder 18 by the frame rate conversion unit 17.

[0035] The autofocus unit 21 determines the focal position of the sub-frame with the largest amount of high-frequency components among a set of sub-frames, and outputs focus adjustment information indicating the focal position to the focus control unit 22.

[0036] Based on the focus adjustment information input from the autofocus unit 21, the focus control unit 22 outputs a lens position adjustment signal for adjusting the lens position to the condenser unit 11. Before shooting the next set of sub-frames, the focus control unit 22 moves the focus of the lens 111 to the focal position determined by the autofocus unit 21.

[0037] Figure 3 is a conceptual diagram for explaining the operations of the micro focal position control unit 19 and the autofocus unit 21. Here, the case where the frame time is 1 / 60 second and the sub-frame time is 1 / 300 second, that is, the case where five (a set) of sub-frames sf(1) to sf(5) are shot in the time of one frame, is shown.

[0038] Assuming that the allowable confusion circle diameter is δ, the F value of the lens 111 is F, the focal length is f, and the subject distance (object distance) is l, the front depth of field Df at that time is expressed by Equation (1), and the rear depth of field Dr is expressed by Equation (2). Since the depth of field D is the sum thereof, it is expressed by Equation (3). The × mark in the figure indicates the position of the focus (object focus). Df = δ×F×l 2 / (f 2 + δ×F×l) (1) Dr = δ×F×l 2 / (f 2 - δ×F×l) (2) D = Df + Dr (3)

[0039] Assuming that the subject distance at the time of shooting the sub-frame sf(1) is sl(1), the front depth of field sDf(1) and the rear depth of field sDr(1) at that time are the values obtained by substituting l = sl(1) into equations (1) and (2) respectively, and from equation (3), the depth of field sD(1) = sDf(1) + sDr(1). Similarly, hereinafter, assuming that the subject distance at the time of shooting the sub-frame sf(n) is sl(n), the depth of field sD(n) = sDf(n) + sDr(n) at that time. The micro focus position control unit 19 calculates the depth of field sD(n) from the lens information in this way.

[0040] Next, the micro focus position control unit 19 slightly moves the focus of the lens 111 so that the depths of field do not exactly overlap for each sub-frame. For example, the lens 111 is moved so that the subject distance satisfies equation (4). By synchronizing the timing of shooting each sub-frame and the minute focus movement by the synchronization unit 20, each sub-frame becomes a continuous image with the focus position shifted slightly within the range of the depth of field. sl(2) = sl(1) + sD(1) sl(3) = sl(2) + sD(2) … sl(n) = sl(n - 1) + sD(n - 1) (4)

[0041] The micro focus position control unit 19 may make the focus position determined by the autofocus unit 21 coincide with the focus position of the middle sub-frame sf(3). As a result, the sub-frames captured by the image sensor 12 can capture the images of the focus positions before and after the image of the optimal focus position at high speed with the image of the optimal focus position as the center.

[0042] The autofocus unit 21 calculates the amount of high-frequency components in the target area (the screen area that is the target of focus adjustment) or the entire image for each sub-frame. In the example shown in Fig. 3, since the amount of high-frequency components in the sub-frame sf(4) is the largest, the autofocus unit 21 determines that the focus of the sub-frame sf(4) is the in-focus position. Before capturing the next set of sub-frames, the focus control unit 22 moves the focus of the lens 111 to the focus position of the sub-frame sf(4). In this way, the focus of the lens 111 in the next frame is determined, enabling the realization of an autofocus function that determines the focus position within the frame time. Note that the amount of high-frequency components may be obtained using a high-pass filter.

[0043] Next, with reference to Fig. 4, an overview of other operations of the imaging device 1 will be described. Similar to Fig. 3, the case where the frame time is 1 / 60 second and the sub-frame time is 1 / 300 second is shown. Also shown is a state where in frame f(1), the autofocus unit 21 determines that the focus of the sub-frame sf(4) is the in-focus position, and this position is set as the focus position of the central sub-frame in frame f(2).

[0044] The synchronization unit 20 synchronizes the movement of the fine focus position of the lens 111 and the time when the imaging element 12 rapidly captures sub-frames at 1 / 300 second intervals.

[0045] The depth-of-field expansion processing unit 15 generates one frame (depth-of-field expanded image) with an expanded depth of field by synthesizing the depths of field of the five obtained sub-frames sf(1) to sf(5). The depth of field L of the depth-of-field expanded image is the sum of the depths of field sD(1) to sD(5) of the five sub-frames sf(1) to sf(5). The depth-of-field expansion processing unit 15 outputs the depth-of-field expanded image every 1 / 60 second as a camera video. This results in a camera output video with less blur.

[0046] The focus adjustment image selection unit 16 selects the middle sub-frame sf(3) from the five obtained sub-frames sf(1) to sf(5).

[0047] The frame rate conversion unit 17 converts the frame frequency of the sub-frame sf(3) to a speed that can be displayed on a normal monitor and displays it on the viewfinder 18.

[0048] The viewfinder 18 displays only the video of the sub-frame sf(3) at 60p. As a result, a video with a shallow depth of field is displayed on the viewfinder 18. By viewing this video, the user U can easily perform manual focusing. In particular, by selecting an image near the sub-frame with the middle sub-frame number as the image for focus adjustment, it becomes possible to adjust at the focal position in the middle of the depth-of-field-expanded image, the focal position adjustment becomes more accurate, and even when the manually adjusted focal position is slightly deviated from the central position, a camera video with less blurring can be obtained. That is, while obtaining a video with an expanded depth of field and less blurring as the camera output video, the user U can perform more precise manual focusing using a video with a shallow depth of field.

[0049] As described above, the present invention acquires sub-frames at a plurality of focal positions at high speed, synthesizes the depth of focus of these sub-frames to output a camera video with an expanded depth of field, and at the same time, by frame rate converting a part of the sub-frames captured at high speed and displaying it on the viewfinder 18, it becomes possible to realize a viewfinder function that facilitates the focus adjustment operation when observed by the user U. Furthermore, it becomes possible to realize an autofocus function that determines an optimal focal position at high speed within the frame time from sub-frames at a plurality of focal positions.

[0050] <Modification example> In the above-described embodiment, among the multi-frame images obtained by slightly moving the focus, the focus image at the center of the moving range was selected and displayed on the viewfinder 18. However, the image to be displayed does not necessarily have to be the one at the center of the moving range, and it may be selected by changing the position before and after the moving range according to the purpose. For example, as a photographer, while focusing on the front side of the subject, following the movement of the subject, as a result, a synthesized video with a deep focus in the rear direction of the subject can be obtained.

[0051] Also, in the above-described embodiment, the focus depth synthesis is performed in real time. However, at a site such as an outdoor location where it is desired to make the equipment as small and light as possible, instead of performing the focus depth synthesis in real time, all the sub-frames taken at high speed may be recorded and the focus depth synthesis may be performed in post-processing. Needless to say, the synchronization information is recorded at the same time in that case. Thus, it is obvious that some functions such as image processing can be separated as long as the viewfinder video seen by the user U, the minute focus movement, and the high-speed shooting are synchronized.

[0052] Also, as shown in FIG. 5, the imaging device 2 may include a switching unit 23 that switches between a camera video with an expanded depth of field by the depth-of-field expansion processing unit 15 and a frame rate conversion video generated by the frame rate conversion unit 17, and displays them on an external display (display unit) 3 according to the instruction of the user U. In this case, the user U can switch between a video with a shallow focus and a deep video to confirm the effect in advance. The present invention is also useful in a medical field. For example, when performing an endoscopic surgery, it becomes possible for a doctor to easily adjust the focus.

[0053] <Program> In order to function as the video conversion unit 13, depth of field expansion processing unit 15, focus adjustment image selection unit 16, frame rate conversion unit 17, micro focus position control unit 19, synchronization unit 20, autofocus unit 21, and focus control unit 22 of the imaging devices 1 and 2 described above, it is also possible to use a computer capable of executing program instructions. Here, the computer may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), or the like. The program instructions may be program codes, code segments, etc. for executing necessary tasks.

[0054] The computer includes a processor, a storage unit, an input unit, an output unit, and a communication interface. The processor may be a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), SoC (System on a Chip), etc., and may be composed of a plurality of processors of the same type or different types. The processor reads and executes a program from the storage unit to perform control of the above-described respective components and various arithmetic processes. Note that at least a part of these processing contents may be realized by hardware. The input unit is an input interface that receives a user's input operation and acquires information based on the user's operation, such as a pointing device, a keyboard, a microphone, etc. The output unit is an output interface that outputs information, such as a display, a speaker, etc. The communication interface is an interface for communicating with an external device.

[0055] The program may be recorded on a computer-readable recording medium. By using such a recording medium, it is possible to install the program on a computer. Here, the recording medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, a USB (Universal Serial Bus) memory, or the like. Further, the program may be in a form downloaded from an external device via a network.

[0056] For example, the program causes the computer to execute steps of generating a single depth-of-field enlarged image with an expanded depth of field using a set of sub-frames captured at a first speed, selecting one sub-frame from the set of sub-frames, generating a frame rate converted video in which the frame rate of the selected sub-frame is converted to a second speed slower than the first speed, and displaying the frame rate converted video.

[0057] Although the above-described embodiments have been described as representative examples, it is obvious to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications or changes are possible without departing from the scope of the claims. For example, it is possible to integrate a plurality of constituent blocks described in the configuration diagrams of the embodiments or to divide one constituent block.

Description of Reference Numerals

[0058] 1, 2 Imaging device 3 Display (display unit) 11 Condensing unit 12 Image sensor 13 Video conversion processing unit 14 Storage unit 15 Depth-of-field enlargement processing unit 16 Focus adjustment image selection unit 17 Frame Rate Conversion Unit 18 Viewfinder (Display Unit) 19 Micro Focus Position Control Unit 20 Synchronization Unit 21 Auto Focus Unit 22 Focus Control Unit 23 Switching Unit 110 Lens Barrel 111 Lens 112 Micro Focus Position Adjustment Mechanism

Claims

1. An imaging device that captures a subject using a lens and an imaging element, comprising: a depth-of-field expansion processing unit that generates one depth-of-field expanded image with an expanded depth of field using a set of sub-frames captured at a first speed; a focus adjustment image selection unit that selects one sub-frame from the set of sub-frames; a frame rate conversion unit that generates a frame rate converted video by converting the frame rate of the selected sub-frame to a second speed slower than the first speed and causes the video to be displayed on a display unit; An imaging device comprising the above components.

2. The imaging device according to claim 1, further comprising a micro focus position control unit that calculates the depth of field at the position of the subject when the sub-frame is captured and moves the focus of the lens so that the depths of field do not overlap for each sub-frame.

3. The imaging device according to claim 2, further comprising a synchronization unit that synchronizes the timing of capturing the sub-frame and the timing of moving the focus of the lens.

4. An autofocus unit that determines the focus position of the sub-frame having the largest amount of high-frequency components among the set of sub-frames; a focus control unit that moves the focus of the lens to the focus position determined by the autofocus unit before capturing the next set of sub-frames; The imaging device according to claim 1 or 2, comprising the above components.

5. A program for causing a computer to function as the imaging device according to claim 1.

Images