Regeneration device

Abstract

To clarify the relationship between the focus position and playback position of each image in a series of refocused images. [Solution] The playback device includes a playback unit that plays a video composed of a composite image whose focus position can be changed by combining multiple images on a display screen, and a display control unit that controls the display screen to display information indicating the relationship between the focus position of the composite image and the playback position on the display screen, and displays information indicating the playback position of the composite image in relation to the focus position of the composite image.

Description

Technical Field

[0004] , , , [Figure 1] , ,

[0005] , , [Figure 5] , , [Figure 4] , , [Figure 3] , [Figure 2]

[0001] The present invention relates to a playback device.

Background Art

[0002] Conventionally, an imaging device that generates a refocused image with a changed focus position of an image has been disclosed (see, for example, Patent Document 1 below). In this conventional technology, the relationship between the focus position of each of a series of refocused images and the playback position where the series of refocused images are played back is unknown.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] A playback device that is one aspect of the invention disclosed in the present application includes a playback unit that plays back on a display screen a moving image composed of a composite image whose focus position can be changed by synthesizing a plurality of images, and a display control unit that controls the display screen and displays on the display screen information indicating the relationship between the focus position and the playback position of the composite image. Further, the display control unit displays information indicating the playback position of the composite image in association with the focus position of the composite image.

Brief Description of the Drawings

[0005] [Figure 1] FIG. 1 is an explanatory diagram showing an example of a display screen of a refocused moving image. [Figure 2] FIG. 2 is an explanatory diagram showing an example of an operation on a display screen of a refocused moving image. [Figure 3] FIG. 3 is an explanatory diagram showing an example of changing the line width of a line graph. <​​​Figure 5 is an explanatory diagram showing the relationship between the microlens array and the image sensor. [Figure 6] Figure 6 is an explanatory diagram showing an example of a disparity image. [Figure 7] Figure 7 is an explanatory diagram showing an example of refocus image generation. [Figure 8] Figure 8 is a flowchart showing an example of the recording process for refocused video images by a playback device. [Figure 9] Figure 9 is flowchart 1, which shows an example of the procedure for downsampling refocused video images. [Figure 10] Figure 10 is flowchart 2, which shows an example of the procedure for downsampling refocused video images. [Figure 11] Figure 11 is an explanatory diagram showing an example 1 of how the display control unit modifies the line graph displayed in the second display area. [Figure 12] Figure 12 is an explanatory diagram showing example 2 of the modification of the line graph displayed in the second display area by the display control unit. [Figure 13] Figure 13 is an explanatory diagram showing example 3 of how the display control unit modifies the line graph displayed in the second display area. [Figure 14] Figure 14 is an explanatory diagram showing example 4 of the modification of the line graph displayed in the second display area by the display control unit. [Figure 15] Figure 15 is an explanatory diagram showing example 5 of the modification of the line graph displayed in the second display area by the display control unit. [Figure 16] Figure 16 is an explanatory diagram showing example 6 of the modification of the line graph displayed in the second display area by the display control unit. [Figure 17] Figure 17 is a flowchart showing an example of the re-imagining process for refocused images. [Figure 18] Figure 18 is an explanatory diagram showing an example of recombining a refocused image after updating the depth of field. [Figure 19] Figure 19 is an explanatory diagram showing an example of how it looks on other devices. [Figure 20] Figure 20 is an explanatory diagram showing an example of a two-screen display. [Figure 21]FIG. 21 is an explanatory diagram showing a display example of one-frame extraction. [Figure 22] FIG. 22 is a flowchart showing an example of a one-frame extraction processing procedure. [Figure 23] FIG. 23 is a flowchart showing an example of a generation processing procedure of a refocused moving image from a drawn line graph. [Figure 24] FIG. 24 is an explanatory diagram showing an example of a thumbnail display in a video file. [Figure 25] FIG. 25 is a flowchart showing an example of a thumbnail image display processing procedure.

MODE FOR CARRYING OUT THE INVENTION

[0006] [[ID=二十]]<Relationship between the focus position and the playback position of a series of frames in the refocused moving image>[[ID=二十一]] [[ID=二十二]]The relationship between the focus position and the playback position of a series of frames in the refocused moving image will be described with reference to FIGS. 1 to 3. [[ID=二十三]] [[ID=二十四]]

[0007] [[ID=二十五]] [[ID=二十六]]FIG. 1 is an explanatory diagram showing an example of a display screen of a refocused moving image. The refocused moving image is a series of refocused images in time series. The refocused image is a composite image obtained by synthesizing a plurality of images taken of a certain subject. When using a parallax image, the refocused image is a composite image obtained by synthesizing a plurality of images with different parallaxes taken of a certain subject. The refocused image is a composite image that can change the focus position after shooting and resynthesize. [[ID=二十七]] [[ID=二十八]]

[0008] [[ID=二十九]] [[ID=三十]]In FIG. 1, the display screen 100 is displayed, for example, on the display of a digital camera, a digital video camera, a smartphone, a tablet, a portable game machine, or a personal computer. The display is, for example, a touch panel type. [[ID=三十一]] [[ID=三十二]]<000****>[[ID=三十三]] The display screen 100 has a first display area 101 and a second display area 102. The first display area 101 is an area for playing and displaying a series of refocused images. In the first display area 101, an image including objects O1 to O3 is displayed as a composite image (refocused image RF4) at a certain playback position. Here, among objects O1 to O3, object O1 is the closest to the shooting source, and object O3 is the farthest from the shooting source.

[0010] The second display area 102 is an area for displaying information indicating the relationship between the focus position and the playback position of a series of refocused images. The information indicating the relationship between the focus position and the playback position of a series of refocused images is displayed in a coordinate system 120 composed of, for example, a time axis 121 (t-axis) indicating the playback position of a series of refocused images and a distance axis 122 (z-axis) indicating the focus position of a series of refocused images. The distance axis 122 indicates the focus position, for example, the focal length between the imaging element and the imaging lens.

[0011] In the coordinate system 120, a line graph WF indicating the change in the focus position according to the playback position of a series of refocused images is displayed as an example of information indicating the relationship between the focus position and the playback position of a series of refocused images. That is, the line graph WF indicates the focus position of the refocused image at a certain playback position. The operator can change the shape of the line graph WF with a finger, for example, through a user interface. Points P1 to P4 are points on the line graph WF (hereinafter collectively referred to as point P), and are generated, for example, each time a predetermined playback position interval or the focus position is changed, or according to the user's designation.

[0012] Also, in the second display area 102, a bar B indicating the playback position of the refocused moving image is displayed. The bar B moves from left to right as the refocused moving image is played back. The bar B in FIG. 1 indicates the playback position t4. The refocused image RF4 displayed in the first display area 101 is the refocused image at the playback position t4. The focus position at this time is d2, and it is assumed that the object O2 is in focus.

[0013] Furthermore, the first display area 101 displays related information 111 indicating the time and location (coordinates) at which the displayed image was plotted. For example, related information 111(P1, time t1) corresponds to point P1 on the line graph WF, related information 111(P4, time t4) corresponds to point P4 on the line graph WF, and related information 111(P2, time t2) corresponds to point P2 on the line graph WF. Note that for related information 111 corresponding to point P, the related information 111 may display a mark of the same color or shape as the mark of point P in the second display area 102. For example, if the mark for point P1 in the second display area 102 is a white circle, the related information 111 for object O1 corresponding to point P1 will also be a white circle of the same color and shape. In this case, the string (P1, time t1) indicating related information 111 may be deleted. Also, point P2 may have a different color or shape than point P1. For example, if the mark for point P2 in the second display area 102 is a white triangle, then the associated information 111 of object O2 corresponding to point P2 will also be a white triangle of the same color and shape. In this case, the string (P4, time t4) indicating the associated information 111 may be deleted. The same applies to point P2.

[0014] In this way, the relationship between the focus position of each refocused image in a series of images and the playback position becomes clear, allowing the user to intuitively see which subject in the refocused image at which playback position is in focus at which focus position.

[0015] Specifically, for example, the user can generate point P on the line graph WF and adjust the depth position of pointer point P using touch focus. By tapping or clicking on the subject, point P on the line graph WF is updated to the in-focus position. Therefore, it becomes unnecessary to repeatedly fine-tune the line graph WF for the subject to be in focus, improving convenience.

[0016] For example, let's consider the case of generating a new point P. Assume that the first display area 101 has a touch autofocus function. The touch autofocus function is an autofocus function that allows the user to specify the focus position using a touch interface. If the user wants to focus on a certain position (x1, y1) on the first display area 101 at the playback time t1 of the video, they can touch the position (x1, y1) on the first display area 101 (for example, a position within object O1), and object O1 containing that position (x1, y1) will be autofocused. At this time, the focus position of object O1 is updated, so point P1 is generated at the updated focus position of object O1 and the position on the line graph identified at the playback time t1. Then, by associating the position (x1, y1) with point P1, the refocused image and the line graph WF at the playback time t1 can be synchronized.

[0017] Next, we will explain how to adjust the position of point P. Assume that the first display area 101 has a touch autofocus function. Assume that point P2 is generated at playback time t2 on the line graph WF and is not associated with a position on the first display area 101. If the user wants to focus on position (x2, y2) at playback time t2, the user selects point P2 on the line graph WF and then taps or clicks on the subject to be focused on in the first display area 101 (for example, object O3 containing position (x2, y2)). This results in a composite image with object O3 in focus. At this time, the focus position of object O3 is updated, and along with the update of the focus position of point P2 on the line graph WF, point P and position (x2, y2) are associated.

[0018] Note that after the above processing, if the user changes the focus position or time position on the line graph WF for points P1 and P2, the focus will not match the time and coordinates before the change. However, it can be used when (1) the user does not edit point P at all on the line graph WF after plotting with touch focus, or (2) when editing point P on the line graph WF after plotting with touch focus and also retaining the difference information before and after the change. In particular, for (2), it is useful when returning the changed point P to its original position.

[0019] Here, as an example of information showing the relationship between the focus position and the playback position of a series of refocused images, a line graph WF is given as an example. However, any information such as a bar graph that allows the user to visually understand the relationship between the focus position and the playback position is acceptable.

[0020] Figure 2 is an explanatory diagram showing an example of operations on the display screen 100 of the refocused moving image. In (A) to (C), in (A) to (C), a refocused image at the playback position t4 is displayed in the first display area 101. Note that bar B can be slid in the direction of the time axis 121 by the user's operation.

[0021] Point P4 in (A) indicates the focus position d2 at the playback position t4 on bar B as shown in Figure 1. In the case of the focus position d2, when a plurality of images with different parallaxes at the playback position t4 are synthesized, a refocused image RF41 focused on object O2 is generated.

[0022] (B) is an example where the focus position d2 is changed to the focus position d1 (<d2) by the user's operation at the playback position t4. In the case of the focus position d1, when a plurality of images with different parallaxes at the playback position t4 are synthesized, a refocused image RF42 focused on object O1 is generated. Also, the line segment between P4 and P1 is linearly interpolated by the changed point P4 and the point P1 one step before. Similarly, the line segment between P4 and P2 is linearly interpolated by the changed point P4 and the point P2 one step ahead.

[0023] (C) is an example where, at playback position t4, the user changes the focus position d2 to focus position d3 (>d2). When the focus position is d3, when multiple images with different parallax at playback position t4 are combined, a refocused image RF43 in focus on object O3 is generated. Also, the line segment between P4 and P1 is linearly interpolated using the changed point P4 and the point immediately preceding it, P1. Similarly, the line segment between P4 and P2 is linearly interpolated using the changed point P4 and the point immediately following it, P2.

[0024] In this way, the user can freely change the focus point of the refocused image at any playback position. Furthermore, abrupt changes in focus are suppressed by linear interpolation between the point where the focus point is changed and its adjacent points. That is, because the focus point is changed gradually, by combining multiple images with different parallax at each playback position along the linearly interpolated line segment, it is possible to play back a refocused video that does not feel unnatural. Note that while linear interpolation was used as an example above, curve interpolation could also be used.

[0025] Figure 3 is an explanatory diagram showing an example of changing the line width of a line graph. In Figure 3, the original symbol is followed by 'a', and the modified symbol is followed by 'b'. (A) shows the line graph WFa before the line width change, and is the same as (A) in Figure 2. The line width wa of the line graph WFa corresponds to the depth of field. In this case, the image is the refocused image RF4a. For example, by pinching out the line graph WFa through user operation, the line width wa of the line graph WFa widens, resulting in a line graph WFb with a line width wb. Since the depth of field deepens in accordance with the line width w, the refocused image RF4b at playback position t4 will be an image that is in focus within the range of the modified depth of field shown by the dashed line, as shown in (B).

[0026] Furthermore, in state (B), the user can, for example, pinch in on line graph WFb, which narrows the line width w of line graph WF and returns it to line graph WFa with line width wa. As a result, the depth of field becomes shallower, and the refocused image at playback position t4 will be in focus within the range of the changed depth of field, as shown in (A). In other words, in (A), the blurring of objects O1 and O3 increases.

[0027] In this way, the user can change the depth of field by freely changing the line width w of the line graph WF. Although Figure 3 illustrates an example of changing the line width w of the entire line graph WF, the line width w can also be changed for the line segment between two points that enclose the pinch-in or pinch-out position within the line graph WF. For example, if you pinch out between points P1 and P2 in (A), the line width of the line segment between points P1 and P2 in (B) will change from wa to wb. Similarly, if you pinch out between points P1 and P2 in (A), the line width of the line segment between points P1 and P2 in (B) will change from wb to wa.

[0028] <Example of hardware configuration for a playback device> Figure 4 is a block diagram showing an example configuration of a playback device. (A) is a block diagram showing an example hardware configuration of the playback device. The playback device 400 plays back and displays refocused video. The playback device 400 is, for example, a digital camera, a digital video camera, a smartphone, a tablet, a portable game console, or a personal computer. The playback device 400 may also have a function for capturing refocused images and refocused video. In Figure 4, the case where the playback device 400 is a digital camera is used as an example for explanation.

[0029] The playback device 400 includes a lens unit 401 and a camera body 402. The lens unit 401 and the camera body 402 may be detachable or integrated. The lens unit 401 includes an imaging lens 411, a microlens array 412, and a focus adjustment unit 413.

[0030] The imaging lens 411 is an optical system that includes, for example, a focus lens or a zoom lens, and collects light from the subject and emits it to the microlens array 412. The microlens array 412 is a collection of microlenses arranged in a matrix. Each microlens collects light from the pupil region of the corresponding imaging lens 411 and emits it to the corresponding region in the image sensor 421. The focus adjustment unit 413 adjusts the focus position of the imaging lens 411 by user operation or by an autofocus function.

[0031] The camera body 402 includes an image sensor 421, a processor 422, a storage device 423, a processing circuit 424, a drive circuit 425, an input device 427, a display device 428, and a communication interface 429. These are connected by a bus 430. The image sensor 421 receives light incident from the imaging lens 411 via the microlens array 412, converts it into an electrical signal, and stores the RAW data obtained from the electrical signal in the storage device 423. For example, the image sensor 421 may be a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor. The processor 422 provides overall control of the playback device 400. The storage device 423 stores the playback program, and the processor 422 executes the playback program stored in the storage device 423.

[0032] The processing circuit 424 is a circuit that processes data and includes, for example, a compression / decompression processing circuit that compresses and decompresses data, an image processing circuit as an image processing function, and an audio processing circuit that processes audio data. The compression / decompression processing circuit, for example, converts RAW data into JPEG image data, generates compressed video, and decompresses compressed video. The image processing circuit performs, for example, scene analysis, shape change of the subject image, position change of the subject image (subject tracking), exposure correction, face recognition, color temperature detection, and generation of refocused images. The audio processing circuit, for example, analyzes the volume of the audio. The compression / decompression processing circuit, image processing circuit, and audio processing circuit may be implemented in the storage device 423 as a compression / decompression processing program, an image processing program, and an audio processing program, respectively, to be executed by the processor 422.

[0033] The drive circuit 425 drives and controls the shutter, aperture mechanism, and lens unit 401. The recording device stores image data and video data. The recording device is detachable from the camera body 402. The input device 427 accepts user input and external audio input. The input device 427 includes, for example, a shutter button and other buttons necessary for shooting, a touch panel, a dial, a ring, and a lever. The display device 428 displays through images of the subject, captured refocus images, and refocus videos. The communication interface 429 sends and receives data with external devices.

[0034] (B) is a block diagram showing an example of the functional configuration of the playback device 400. The playback device 400 includes an imaging unit 431, a detection unit 432, a sound collection unit 433, a generation unit 434, a playback unit 435, and a display control unit 436. The imaging unit 431 is composed of, for example, a lens unit 401, an image sensor 421, and a processing circuit 424. The imaging unit 431 captures a group of images with different parallax for a subject and stores them as RAW data in the storage device 423.

[0035] The detection unit 432 detects the focus position of the imaging lens 411 by the focus adjustment unit 413. The detection unit 432 also detects the movement of a specific focus position on the line graph WF and changes in the line width of points on the line graph WF. The detection unit 432 also detects the drawing of the line graph WF in coordinate system 120 on the display screen 100. Furthermore, the detection unit 432 detects instructions to play the refocused video file and instructions to display a thumbnail image. The detection unit 432 realizes its functions by causing the processor 422 to execute the playback program stored in the storage device 423. The sound collection unit 433 acquires sound from a sound source using a directional microphone, which is an example of an input device 427.

[0036] The generation unit 434 generates a series of refocus images from RAW data, and also generates a refocus video from the series of refocus images, and stores them in the storage device 423. For example, if the detection unit 432 detects movement of a specific focus position in the direction of the distance axis 122, the generation unit 434 recombines multiple images with different parallax (disparity images described later), which are the source of the refocus image for a specific playback position corresponding to the specific focus position, so that they are in focus on the focus position after the movement of the specific focus position. The generation unit 434 may be composed of a processing circuit 424, or its function may be realized by having a processor 422 execute a playback program stored in the storage device 423.

[0037] The playback unit 435 retrieves refocused video and refocused images stored in the memory device 423 and plays them back on the display screen 100. The display control unit 436 controls the display screen 100, or, in response to input from the input device 427 or user operation on the display screen 100, displays information (for example, a line graph WF or bar B) on the display screen 100 that shows the relationship between the focus position and playback position of a series of refocused images. In this case, the generation unit 434 reconstructs the refocused images and reconstructs the refocused video based on the updated information. The display control unit 436 displays or erases related information 111 on the display screen 100.

[0038] <Example of generating a refocused image> Figure 5 is an explanatory diagram showing the relationship between the microlens array 412 and the image sensor 421. (A) shows the correspondence between the microlens array 412 and the image sensor 421. (B) shows the correspondence between the microlenses Lij (i,j are integers of 1 or more; in this example, 1≦i≦5, 1≦j≦5) and pixels Pijkl (k,l are integers of 1 or more; in this example, 1≦k≦5, 1≦l≦5) that constitute the microlens array 412. Each circular figure represents a microlens Lij. The set of microlenses Lij shown as circular figures is the microlens array 412. The image sensor 421 has multiple pixels Pijkl. For example, the image sensor 421 has 25×25 pixels. The 5×5 pixel area enclosed by the thick frame is called the pixel area Cij. One pixel area Cij corresponds to one microlens Lij. The pixel area Cij is a set of pixels Pijkl.

[0039] Figure 6 is an explanatory diagram showing an example of a disparity image. A disparity image PIkl is each of multiple images taken of the same subject with different disparities. Specifically, a disparity image PIkl is an image in which the pixels Pijkl of each pixel region Cij are arranged according to the arrangement position of the microlens Lij. For example, disparity image PI11 is an image in which the pixels Pij11 of each pixel region Cij are arranged according to the arrangement position of the microlens Lij, and disparity image PI55 is an image in which the pixels Pij55 of each pixel region Cij are arranged according to the arrangement position of the microlens Lij.

[0040] Figure 7 is an explanatory diagram showing an example of refocus image generation. The disparity images PI11 to PI55 are, for example, a group of images in focus on object O1. The processing circuit 424, which is the generation unit 434, or the processor 422 that executes the image processing program, generates a refocus image RF in focus on object O1 by combining the disparity images PI11 to PI55.

[0041] <Example of operation procedure> Figure 8 is a flowchart illustrating an example of the recording process procedure for refocused video by the playback device 400. First, the playback device 400 detects the focus position from the focus adjustment unit 413 using the detection unit 432 and stores it in the storage device 423 (step S801). Next, the playback device 400 stores the RAW data at the time at the detected focus position in the storage device 423, associating it with the detected focus position (step S802). This provides a series of RAW data associated with the focus position for each time point. The playback device 400 determines whether recording is complete or not based on user input or a timer (step S803). If recording is not complete (step S803: No), the process returns to step S801.

[0042] On the other hand, if recording is complete (step S803: Yes), the playback device 400 generates a series of refocus images from the acquired series of RAW data using the generation unit 434 (step S804). Specifically, for example, the playback device 400 converts each of the acquired series of RAW data into image data such as JPEG. Each of the converted series of image data includes multiple image data with different parallax. The playback device 400 generates a refocus image at a certain time by combining these multiple image data at the focus position associated in step S802. This obtains a refocus image for each acquired series of RAW data. The playback device 400 compresses this series of refocus images into a video format (step S805). The playback device 400 then stores the compressed data in video format as a refocus video in the storage device 423 (step S806).

[0043] Figures 9 and 10 are flowcharts illustrating an example of a refocused video image decimation process. In Figure 9, to reduce the load on the processing circuit 424, the playback device 400 arranges the focus positions of selected refocused images in chronological order at predetermined intervals, and performs linear interpolation (or curve interpolation) between the selected refocused images. This makes it possible to generate a line graph WF showing the relationship between the focus position and the playback position.

[0044] First, the playback device 400 sets n=1 (step S901). In Figures 9 and 10, n represents the refocus image appearance position, i.e., the playback position, in the order in which the refocus images appear. The playback device 400 reads the RAW data RDn of appearance position n using the detection unit 432 (step S902). In step S905, the playback device 400 detects the current focus position fn from the focus adjustment unit 413 and stores it in the storage device 423 (step S903).

[0045] The playback device 400 generates a refocus image RFn from the RAW data RDn read in step S902 using the focus position fn detected in step S903 by the generation unit 434 (step S904). The playback device 400 displays the generated refocus image RFn in the first display area 101 of the display screen 100 (step S905). The refocus video being played back is displayed in the first display area 101 until a playback end signal is detected.

[0046] Next, the playback device 400 waits for the detection of a playback start signal (step S906: No). If a playback start signal is detected (step S906: Yes), the playback device 400 associates and stores the refocused image RFn and its focus position fn (step S907). The playback device 400 updates n by adding a to n (step S908). a (≧1) is a pre-set fixed value (integer) that determines the RAW data to be synthesized next. If a is 1, the playback device 400 generates the refocused image RFn without decimation. If a is 2 or greater, the playback device 400 generates the refocused image RFn while decimating.

[0047] Next, in step S908, the playback device 400 reads the RAW data RDn of the updated refocus image position n (step S909). The playback device 400 detects the current focus position fn from the focus adjustment unit 413 using the detection unit 432 and stores it in the storage device 423 (step S910). The playback device 400 generates a refocus image RFn from the RAW data RDn read in step S909 using the focus position fn detected in step S910 using the generation unit 434 (step S910).

[0048] After this, if no playback completion signal is detected (step S912: No), the playback device 400 returns to step S907. On the other hand, if a playback completion signal is detected (step S912: Yes), the device proceeds to step S1001 in Figure 10. This completes the synthesis of the refocused image group.

[0049] In Figure 10, the regeneration device 400 determines whether a is a value of 2 or greater (step S1001). If a is not a value of 2 or greater (step S1001: No), a=1, and the device proceeds to step S1004.

[0050] On the other hand, if a is a value of 2 or more (step S1001: Yes), the playback device 400, using the display control unit 436, interpolates the focus position of the refocused image that has been thinned and not synthesized (thinned refocused image) using the focus position detected in steps S903 and S910 (step S1002). The interpolated focus position is called the interpolated focus position. The playback device 400, using the generation unit 434, generates a refocused image RFn for each RAW data RDn corresponding to the remaining refocused image that has not been thinned (step S1003).

[0051] Then, the playback device 400 compresses the series of refocused images RFn into a video format using the playback unit 435 (step S1004), and stores the compressed refocused video in the storage device 423 (step S1005). This completes the process of downsampling the refocused video.

[0052] Subsequently, when the playback device 400 detects a playback start signal for the refocused video generated in step S1005, the playback unit 435 plays the refocused video in the first display area 101 of the display screen 100, and the display control unit 436 generates a line graph WF showing the relationship between the focus position and the playback position and displays it in the second display area 102.

[0053] <Example of changing a line graph> Figures 11 to 16 are explanatory diagrams showing examples of changes to the line graph WF displayed in the second display area 102 by the display control unit 436. In Figure 11, (A) shows the initial state. The line graph WF includes a group of lines connecting points P10 to P14 with line segments ls1 to ls4. The line graph WF also has a line width w. In addition, a shift button 1100 is displayed in the first display area 101. In Figure 11, the shift button 1100 is in the OFF state. In the OFF state, the positions of points P10 to P14 on the line graph WF can be changed. Therefore, in Figure 11, there is no change in depth of field. In Figures 11 to 16, the code for the line graph WF before change is appended with 'a', and the code for the line graph WF after change is appended with 'b'.

[0054] (B) shows the state in which point P11a is moved upward from the state in (A), i.e., in the depth direction of the distance axis 122, by dragging. By dragging point P11a, the line segment ls1 between points P10 and P11a changes from being parallel to the time axis 121 to sloping upward to the right, becoming the line segment ls1b between points P10 and P11b. Similarly, the line segment ls2 between points P11a and P12 changes from being parallel to the time axis 121 to sloping downward to the right, becoming the line segment ls2b between points P11b and P12. The line width w does not change. That is, there is no change in depth of field.

[0055] (C) shows the state after dragging point P12a to the left parallel to the time axis 121 from the state in (A). By dragging point P12a, the slope of the line segment ls3a between points P12a and P13 changes, becoming the line segment ls3b between points P12b and P13. The line width w does not change. In other words, there is no change in depth of field.

[0056] (D) shows the state after dragging point P12a to the right parallel to the time axis 121 from the state in (A). By dragging point P12a, the slope of the line segment ls3a between points P12a and P13 changes, becoming the line segment ls3b between points P12b and P13. The line width w does not change. In other words, there is no change in depth of field.

[0057] (E) shows the state in which point P11a has been moved to the right parallel to the time axis 121 and upward, i.e., in the depth direction of the distance axis 122, from the state in (A) by dragging. By dragging point P11a, the line segment ls1a between points P10 and P11a changes from being parallel to the time axis 121 to sloping upward to the right. Similarly, the line segment ls2a between points P11a and P12 changes from being parallel to the time axis 121 to sloping downward to the right. The line width w does not change. That is, there is no change in depth of field.

[0058] Figure 12 shows an example of dragging the OFF shift button 1100. In Figure 12, (A) shows the initial state, similar to (A) in Figure 11. (B) shows the state after dragging the shift button 1100 from state (A). By dragging the shift button 1100, points P10a to P14a move in the same direction as the movement of the shift button 1100. In this case, points P10 to P14 move upward, that is, in the depth direction of the distance axis 122, becoming points P10b to P14b. Accordingly, line segments ls1a to ls4a also move upward, becoming line segments ls1b to ls4b. The line width w does not change. As a result, the line graph WFa is changed to line graph WFb, and the focus position can be adjusted all at once over the entire range of the time axis 121. Note that dragging the shift button 1100 downward returns from state (B) to state (A).

[0059] (C) shows the state after moving the shift button 1100 to the right from the state in (A) by dragging. By dragging the shift button 1100, points P10a to P14a move in the same direction as the movement of the shift button 1100. In this case, points P10a to P14a move to the right on the time axis 121 and become points P10b to P14b. Consequently, line segments ls1a to ls4a also move to the right and become line segments ls1b to ls4b. The line width w does not change. As a result, the line graph WFa is changed to line graph WFb, and the focus position can be adjusted all at once across the entire range of the time axis 121. Note that by dragging the shift button 1100 to the left, you can return from (C) to the state in (A).

[0060] Figure 13 shows an example 1 of changing the line width w when the shift button 1100 is ON. (A) shows the state in which the playback device 400 detects that the shift button 1100 has been pressed, and the shift button 1100 has changed to ON, starting from the state in (A) of Figure 11.

[0061] (B) shows the state where, from the state of (A), a drag operation is performed downward, that is, in the forward direction of the distance axis 122 at point P11. By dragging point P11, the line width wa changes so as to become narrower from point P10 toward point P11 between points P10 and P11, and the line width becomes wb (< wa) at point P11. Similarly, the line width wa changes so as to become narrower from point P12 toward point P11 between points P11 and P12, and the line width becomes wb (< wa) at point P11. Thus, it is possible to adjust the depth of field so that it becomes shallower at point P11.

[0062] (C) shows the state where, from the state of (A), a drag operation is performed upward, that is, in the depth direction of the distance axis 122 at point P11. By dragging point P11, the line width wa changes so as to become wider from point P10 toward point P11 between points P10 and P11, and the line width becomes wb (> wa) at point P11. Similarly, the line width wa changes so as to become narrower from point P12 toward point P11 between points P11 and P12, and the line width becomes wb (> wa) at point P11. Thus, it is possible to adjust the depth of field so that it becomes deeper at point P11.

[0063] FIG. 14 shows a second example of the change in the line width w when the shift button 1100 is in the ON state. (A) shows the state where, from the state of (A) in FIG. 13, point P11 is dragged in the left direction of the time axis 121. By dragging in the left direction, the line width wa changes so as to become wider from point P11 toward point P10 between points P10 and P11, and the line width becomes wb (> wa) at point P10.

[0064] (B) shows the state where, from the state of (A) in FIG. 13, point P11 is dragged in the right direction of the time axis 121. By dragging in the right direction, the line width wa changes so as to become wider from point P11 toward point P12 between points P11 and P12, and the line width becomes wb (> wa) at point P12. Also, thereby, the line width wb changes so as to become narrower from point P12 toward point P13 between points P12 and P13, and the line width becomes wa at point P13.

[0065] (C) shows the state where, starting from the state of (A) in FIG. 13, point P11 is dragged leftward along the time axis 121 and then turned back and dragged rightward. By performing such a drag operation, the line width wa between points P10 and P11 changes so as to become narrower from point P11 toward point P10, and the line width wb (< wa) is obtained at point P10.

[0066] (D) shows the state where, starting from the state of (A) in FIG. 13, point P11 is dragged rightward along the time axis 121 and then turned back and dragged rightward. By performing such a drag operation, the line width wa between points P11 and P12 changes so as to become narrower from point P11 toward point P12, and the line width wb (< wa) is obtained at point P12. Further, thereby, the line width wb between points P12 and P13 changes so as to become wider from point P12 toward point P13, and the line width wa is obtained at point P13.

[0067] FIG. 15 shows a third example of changing the line width w when the shift button 1100 is in the ON state. FIG. 15 shows the state where, starting from the state of (A) in FIG. 13, point P11 is dragged in the upper right diagonal direction. By performing the drag operation in the upward component, the line width wa between points P10 and P11 changes so as to become wider from point P10 toward point P11, and the line width wb (> wa) is obtained at point P11. Similarly, the line width wb between points P11 and P12 changes so as to become narrower from point P11 toward point P12, and the line width wbb (> wa) is obtained at point P11. Further, thereby, the line width wbb between points P12 and P13 changes so as to become narrower from point P12 toward point P13, and the line width wa is obtained at point P13.

[0068] FIG. 16 shows an example of dragging the shift button 1100 in the ON state. (A) shows the state where the shift button 1100 is dragged upward starting from the state of (A) in FIG. 13. By dragging the shift button 1100, the line width w of the line graph WF expands from wa to wb, and the line graph WF is changed from the line graph WFa to the line graph WFb. Thereby, the depth of focus can be adjusted over the entire reproduction section.

[0069] (B) shows the state where the shift button 1100 is dragged downward from the state of (A). By dragging the shift button 1100, the line width w of the line graph WF becomes narrower from wa to wb, and the line graph WFa is changed to the line graph WFb. The depth of focus can be adjusted over the entire playback period.

[0070] (C) shows the state where the shift button 1100 is dragged in the right direction of the time axis 121 from the state of (A). By dragging in the right direction, the line width wa changes so that it becomes wider from point P11 to point P14 between points P11 to P14, with the line width wb (>wa) at point P12, the line width wbb (>wb) at point P13, and the line width wbbb (>wbb) at point P14. As a result, the line graph WFa is changed to the line graph WFb.

[0071] (D) shows the state where, after dragging the shift button 1100 in the right direction of the time axis 121 from the state of (A), it is then dragged back and dragged in the left direction. By performing such a drag operation, the line width wa changes so that it becomes narrower from point P11 to point P14 between points P11 to P14, with the line width wb (<wa) at point P12, the line width wbb (<wb) at point P13, and the line width wbbb (<wbb) at point P14. As a result, the line graph WFa is changed to the line graph WFb.

[0072] <Example of the recomposition processing procedure of the refocused image> Next, an example of the recomposition processing of the refocused image when the change operations shown in FIGS. 11 to 16 are performed will be described.

[0073] Figure 17 is a flowchart showing an example of the re-combination process for refocused images. The playback device 400 detects whether the shift button 1100 is ON or OFF (step S1701). If it is OFF (step S1701: OFF), the playback device 400 updates the line graph WF in accordance with the drag operation in the OFF state using the display control unit 436, as shown in Figure 11 or Figure 12 (step S1702). Then, in step S1702, the playback device 400 identifies the playback section in which the focus position has been updated (step S1703).

[0074] Next, in step S1703, the playback device 400 determines whether there is an unselected playback position among the playback positions indicated by points on the line graph within the playback section for which the focus position has been updated (step S1704). If there is an unselected playback position for which the focus position has been updated (step S1704: Yes), the playback device 400 selects the RAW data RDn and the updated focus position for the unselected playback position for which the focus position has been updated (step S1705). Then, the playback device 400 recombines the refocused image with the updated focus position (step S1706) and returns to step S1704.

[0075] In step S1704, if there are no unselected playback positions for which the focus position has been updated (step S1704: No), the playback device 400 reconstructs a compressed refocused video using the recombined refocused image and the remaining refocused image for which the focus position has not been updated (step S1712). With this, the playback device 400 terminates the recombination process of the refocused image.

[0076] Furthermore, if the shift button 1100 is in the ON state in step S1701 (step S1701: OFF), the playback device 400 updates the line graph WF in accordance with the drag operation in the ON state using the display control unit 436, as shown in Figures 13 to 16 (step S1707). Then, in step S1707, the playback device 400 identifies the playback section for which the line width has been updated (step S1703).

[0077] Next, the playback device 400 determines whether there are any unselected updated playback positions (step S1709). If there are any unselected updated playback positions (step S1709: Yes), the playback device 400 selects the RAW data RDn, focus position, and updated line width of the unselected updated playback position (step S1705). Then, the playback device 400, using the generation unit 434, reconstructs the refocused image with the depth of field corresponding to the updated line width (step S1711), and returns to step S1709.

[0078] In step S1709, if there are no unselected playback positions (step S1709: No), the playback device 400 reconstructs the compressed refocused video using the recombined refocused image and the remaining refocused image whose line width was not updated by the generation unit 434 (step S1712). With this, the playback device 400 completes the recombination process of the refocused image.

[0079] <Recombination of refocused images after updating depth of field> Next, we will explain an example of recombining the refocused image generated as described above to achieve pan-focus.

[0080] Figure 18 is an explanatory diagram showing an example of recombining a refocused image after updating the depth of field. In Figure 18, the virtual aperture 1800 restricts the emission of light from the imaging lens 411 to the image sensor 421. In the image sensor 421, hatched pixels are pixels that have received light, and unhatched pixels are pixels that have not received light due to the virtual aperture 1800. Note that, because it is a virtual aperture 1800, even pixels that are not hatched actually receive light, but their output signal is not used.

[0081] In (A), the virtual aperture 1800 allows light from pupil regions PR2 to PR4 of the pupil regions PR1 to PR5 of the imaging lens 411 to pass through. That is, even if the image sensor 421 receives light from pupil regions PR1 and PR5 and outputs an optical signal, the regeneration device 400 does not use it in the refocus image synthesis process. In (B), the virtual aperture 1800 allows light from pupil region PR3 of the pupil regions PR1 to PR5 of the imaging lens 411 to pass through. That is, even if the image sensor 421 receives light from pupil regions PR1, PR2, PR4, and PR5 and outputs an optical signal, the regeneration device 400 does not use it in the refocus image synthesis process.

[0082] Between (A) and (B), the depth of field is deeper in (B) because the aperture 1801 of the virtual aperture 1800 is more closed. In other words, if the user wants to achieve pan-focus, the user can increase the line width w of the line graph WF, causing the playback device 400 to narrow the aperture 1801 of the virtual aperture 1800 using the generation unit 434, as in (B). That is, the user can make the depth of field infinite by increasing the line width w of the line graph. In this way, a refocused image corresponding to the change in depth of field can be generated after shooting.

[0083] <Example of display on other devices> Figure 19 is an explanatory diagram showing an example of display on other devices. Figure 19 shows an example of display on a smartphone or tablet. In the case of smartphones and tablets, due to the limitations of the display screen 100, the adjustment area 1901 is displayed in the first display area 101. After specifying a point, the specified point can be moved in the direction of the distance axis 122 by dragging the slider 1902 of the adjustment area 1901 up or down.

[0084] <Example of 2 screen display> Figure 20 is an explanatory diagram showing an example of a two-screen display. In Figure 20, the first display area 101 has two windows 101A and 101B. The playback device 400, using the playback unit 435, plays the refocused video at the current playback position in the first window 101A. The playback device 400, using the playback unit 435, plays a preceding refocused video that is a predetermined time ahead of the current playback position in the second window 101B. In this case, the playback device 400, using the display control unit 436, displays bar B1 of the first window 101A and bar B2 of the second window 101B in the second display area 102.

[0085] For example, when changing the focus position while playing back a refocused video using the focus adjustment unit 413, if the focus adjustment unit 413 is operated after the subject has moved forward or backward, a focus shift will occur. Therefore, by displaying a refocused video a predetermined time in advance (for example, a few seconds) in the second window 101B, the user can understand whether the subject will move forward or backward next, and thus prevent a focus shift.

[0086] Furthermore, the playback device 400 may superimpose the refocused image at the current playback position and the preceding refocused image on the first display area 101 using the playback unit 435. This allows the user to intuitively grasp the difference in the movement of subjects with a time difference and predict which subject to focus on. In this case, either of the refocused moving images may be displayed, for example, as a colored transparent image. This allows the user to easily distinguish between both refocused moving images.

[0087] Furthermore, the playback unit 435 may, in the second window 101B, convert the series of refocused images constituting the preceding refocused video into a pan-focused refocused image, for example, using the method shown in Figure 18. As a result, the subject of the preceding refocused video is displayed without blurring, allowing the user to refer to the preceding refocused video to confirm the subject to be focused on.

[0088] Furthermore, the playback unit 435 may play back a refocused video image in the first window 101A that has a different focus position than the refocused video image played back in the first window 101A. In this case, the playback unit 435 may play back both refocused videos in sync, that is, to display refocused videos at the same playback position.

[0089] <Subject tracking> Furthermore, in Figure 4(B), the imaging unit 431 may capture images while tracking the subject using the processing circuit 424. In this case, the generation unit 434 generates a series of refocus images corresponding to the focus position at the distance to the subject by combining multiple parallax images PI11 to PI55 for each of the captured subjects. This allows for the automatic generation and display of a refocused video and a time-series line graph WF showing the focus position of the tracked subject. This also allows for confirmation of which subject was being tracked.

[0090] Furthermore, as an example of the input device 427, a sound collection unit 433 consisting of two or more microphones may be used. In this case, the two microphones identify the direction of the subject that is the sound source. The imaging unit 431 may then track the direction of the subject identified by the two microphones. This makes it possible to display a time-series line graph WF that shows the focus position of the sound source. This also makes it possible to confirm which subject was being tracked as the sound source.

[0091] <1 frame extracted> Figure 21 is an explanatory diagram showing an example of displaying a single frame extraction. Single frame extraction is a process in which one of the refocused images that make up a series of refocused images that constitute a refocused video is selected, and multiple parallax images PIkl that will be used as the source for combining the selected refocused image are displayed. The playback device 400 converts the RAW data that will be used as the source for combining the selected refocused image into an image such as JPEG, and displays it as multiple parallax images PI11 to PI55 by the playback unit 435. By displaying multiple parallax images PI11 to PI55 that are at the same playback position for the selected refocused image during playback of the refocused video, it is possible to provide an image that appears to be played in slow motion from different viewing directions. Note that the playback device 400 does not need to display all of the multiple parallax images PI11 to PI55 for the extracted refocused image, and may display a predetermined number of images.

[0092] Figure 22 is a flowchart showing an example of a single-frame extraction process. The playback device 400 specifies one or more playback positions for extracting the refocused image based on user input (step S2201). Next, the playback device 400 waits for a playback start signal via the playback unit 435 (step S2202: No). If a playback start signal is detected (step S2202: Yes), the playback device 400 starts playback of the refocused video via the playback unit 435. If the playback device 400 has not yet reached the extraction playback position (step S2203: No), it proceeds to step S2206.

[0093] On the other hand, if the cropping playback position is reached (step S2203: Yes), the playback device 400, using the playback unit 435, acquires multiple disparity images PI11 to PI55, which will be the source for synthesizing the refocused image at the cropping playback position, from the storage device 423 (step S2204) and displays them on the display screen 100 (for example, the first display area 101) (step S2205). In this case, playback of the refocused video may be stopped for a certain period of time. This allows the user to concentrate on viewing the multiple disparity images PI11 to PI55 for a certain period of time. If the playback device 400 has not detected a playback end signal (step S2206: No), it returns to step S2203.

[0094] Furthermore, when displaying multiple parallax images PI11 to PI55 in step S2205, the playback device 400 may display the multiple parallax images PI11 to PI55 in an order corresponding to the arrangement order of the microlenses Lij in the microlens array 412. This allows multiple parallax images PI11 to PI55 to be displayed so that the parallax changes continuously. In this way, by extracting a single frame, it is possible to provide video that appears to be played back in slow motion from different viewing directions.

[0095] Furthermore, if, during the extraction of a single frame, a user selects an arbitrary parallax image PIkl from among multiple parallax images PI11 to PI55, the playback device 400 may use the generation unit 434 to re-synthesize the selected refocused image with the selected parallax image PIkl as the focus position. This allows the user to obtain a refocused image that is in focus on the intended parallax image.

[0096] <Example of generating refocused video from a drawing line graph> In the example described above, a series of refocused images were generated, and then a line graph WF indicating the playback position and focus position of each refocused image was generated. However, the playback device 400 may first draw the line graph WF through user operation, and then the generation unit 434 may generate a series of refocused images at the focus position at each playback position of the drawn line graph WF. This allows the user to generate a series of refocused images at consecutive focus positions as intended.

[0097] Figure 23 is a flowchart illustrating an example of the process for generating a refocused video from a drawn line graph. First, the playback device 400 detects the drawing of the line graph WF in the second display area 102 (step S2301). Next, the playback device 400 records the combination of playback position and focus position, which are points on the line graph WF (step S2302). Then, the playback device 400 determines whether or not there are any unselected playback positions in that combination (step S2303). If there are unselected playback positions (step S2303: Yes), the playback device 400 selects an unselected playback position (step S2304) and obtains the focus position corresponding to the selected playback position from the drawn line graph WF (step S2305). Then, the playback device 400, using the generation unit 434, reads the RAW data of the selected playback position from the storage device 423, synthesizes the RAW data with the acquired focus position, and generates a refocused image (step S2306).

[0098] Specifically, for example, the playback device 400 converts the acquired RAW data into multiple image data, such as JPEG. These multiple converted image data are a group of image data with different parallax levels. The playback device 400 generates a refocused image by combining these image data groups at their associated focus positions. This allows the refocused image to be obtained from the acquired RAW data. After this, the process returns to step S2302.

[0099] In step S2303, if there are no unselected playback positions (step S2303: No), the playback device 400 compresses the series of refocused images generated in step S2306 into a video format using the playback unit 435 (step S2307). The playback device 400 then stores the compressed data in video format as a refocused video in the storage device 423 (step S2308). Thus, as shown in Figures 22 and 23, a series of refocused images can be generated at a sequence of focus positions intended by the user.

[0100] <Thumbnail display> Figure 24 is an explanatory diagram showing an example of thumbnail display for a video file. In Figure 24(A), the playback device 400, via the display control unit 436, displays the icon IC of the refocused video file in the folder 2400 on the display screen 100. The icon IC is a symbol that accepts playback instructions for the refocused video and instructions to display a thumbnail image. When a user performs a playback instruction (for example, single-click or double-click) on a certain icon IC with the mouse cursor MC, the playback device 400, via the playback unit 435, plays the refocused video.

[0101] On the other hand, when a user controls an icon IC and instructs the display of a thumbnail image using the mouse cursor MC (for example, by pointing at the icon IC), the playback device 400, as shown in (B), displays a thumbnail image S of the subject in the refocused video that is in focus on the subject in the refocused video played back by the instructed icon IC, using the display control unit 436. The thumbnail image S may be a reduced image of the refocused video, or a reduced image of the subject in focus. This allows the user to check what subject is in focus in the refocused video before opening the video file.

[0102] Furthermore, the thumbnail image S to be displayed may be multiple thumbnail images with different focus positions (for example, three thumbnail images Sa, Sb, and Sc). For example, the multiple thumbnail images Sa, Sb, and Sc to be displayed may be thumbnail images of multiple refocused images with different playback positions and focus positions. This allows users to check what kind of refocused image will be displayed with different subjects in focus at different playback positions before opening the video file. Alternatively, the multiple thumbnail images Sa, Sb, and Sc to be displayed may be thumbnail images of multiple refocused images with the same playback position but different focus positions. This allows users to check what kind of refocused image will be produced if the focus position is changed at the same playback position before opening the video file.

[0103] Figure 25 is a flowchart illustrating an example of a thumbnail image display processing procedure. The playback device 400 waits for a thumbnail image display instruction (step S2501). If a display instruction is detected (step S2501: Yes), the playback device 400 identifies the video file corresponding to the icon IC for which the display instruction was issued (step S2502), and displays one or more thumbnail images with different focus positions near the icon IC (step S2503). Then, the process returns to step S2501.

[0104] If no thumbnail display instruction is detected in step S2501 (step S2501: No), the playback device 400 waits for a video file playback instruction (step S2504). If no playback instruction is received (step S2504: No), the process returns to step S2501.

[0105] On the other hand, if a playback command is given (step S2504: Yes), the playback device 400 plays the video file. The playback device 400 then acquires one or more refocused images with different focus positions from the refocused video (step S2505). The playback device 400 then converts the acquired refocused video images into thumbnail images and stores them in the storage device 423 (step S2506). Then, the process returns to step S2501. In this way, the thumbnail display process allows users to check what subjects are in focus in the refocused images before opening the video file.

[0106] Thus, the playback device 400 according to this embodiment includes a playback unit 435 that plays a video on a display screen 100 composed of a series of refocused images, which are a time-series collection of refocused images formed by synthesizing multiple images with different parallaxes (parallax images PI11 to PI55), and a display control unit 436 that controls the display screen 100 to display information indicating the relationship between the focus position and playback position of the series of refocused images on the display screen 100. This makes it possible to intuitively grasp the focus position of each of the played-back series of refocused images.

[0107] Furthermore, the display control unit 436 may display information indicating the playback position of the refocused image in relation to the in-focus position of the refocused image. This makes it possible to identify which subject in which refocused image is at which in-focus position. Therefore, it is possible to identify continuous changes in the in-focus position in the refocused video.

[0108] Furthermore, information showing the relationship between the focus position and playback position of a series of refocused images is, for example, a line graph WF showing the change in the focus position of a series of refocused images due to the playback position in a coordinate system 120 composed of a first axis (time axis 1212) indicating the playback position and a second axis (distance axis 122) indicating the focus position. This allows for an intuitive and explicit understanding of the focus position of each of the playback series of refocused images.

[0109] Furthermore, the playback device 400 includes a detection unit 432 that detects movement of a specific focus position on the line graph, and a generation unit 434 that, when the detection unit 432 detects movement of the specific focus position in the direction of the second axis (distance axis 122), recombines a plurality of specific images (parallax images PI11 to PI55) with different parallax, which are the source images for the refocus image of a specific playback position corresponding to the specific focus position, so that they are in focus on the focus position after the movement.The playback unit 435 then plays the refocus image recombined by the generation unit 434 on the display screen 100, and the display control unit 436 controls the display screen 100 to interpolate the line graph WF based on the focus position after the movement and displays the interpolated line graph WF on the display screen 100.This makes it possible to change the focus position even for refocus images of playback positions that have not been operated by the user.Therefore, it is possible to maintain a continuous change in the focus position.

[0110] Furthermore, the playback unit 435 may play back a first refocused image that constitutes the video on the display screen 100, and a second refocused image that precedes the first refocused image in time on the display screen 100. Specifically, the playback unit 435 plays back the first refocused image and the second refocused image in different areas of the display screen 100. This allows the user to understand whether the subject will move forward or backward next, and prevents focus shifts when changing the line graph WF.

[0111] Furthermore, the playback unit 435 may play back the first refocused image and the second refocused image in the same area of ​​the display screen 100. This allows the user to intuitively grasp the difference in the movement of subjects with a time difference and predict which subject to focus on.

[0112] Alternatively, the generation unit 434 may generate a pan-focus image of the second refocused image by setting the focus position of the second refocused image to infinity, and the playback unit 435 may play the pan-focus image on the display screen 100. As a result, the subject of the preceding refocused video is displayed without blurring, allowing the user to refer to the preceding refocused video to confirm the subject to be focused on.

[0113] Alternatively, the display control unit 436 may control the display screen 100 to display the line width w in the direction of the second axis (distance axis 122) of the line graph WF as the depth of field of the refocused image, and the playback unit 435 may display the refocused image on the display screen 100 based on the depth of field. This allows the user to intuitively understand the depth of field of each of the series of refocused images.

[0114] Furthermore, the playback device 400 includes a detection unit 432 that detects changes in the line width w in the direction of the second axis (distance axis 122) of the line graph WF, and a generation unit 434 that recombines a plurality of disparity images PI11 to PI55, which are the source images for the specific refocus image, by changing the depth of field of the specific refocus image at the playback position where the change in the line width w in the direction of the second axis (distance axis 122) was detected by the detection unit 432, based on the changed width w.The playback unit 435 then plays the specific refocus image after recombination by the generation unit 434 on the display screen 100, and the display control unit 436 controls the display screen 100 to interpolate the line graph WF based on the changed line width w and displays the interpolated line graph WF on the display screen 100.This makes it possible to change the depth of field of a specific refocus image with simple operation.

[0115] Furthermore, the playback device 400 includes a shooting unit 431 that tracks and photographs a subject, and a generation unit 434 that generates a series of refocus images corresponding to the in-focus position at the distance to the subject by combining multiple parallax images PI11 to PI55 for each of the series of subjects photographed by the shooting unit 431. The playback unit 435 then plays back a video (refocus video) composed of the series of refocus images generated by the generation unit 434 on the display screen 100, and the display control unit 436 controls the display screen 100 to display information (for example, a line graph WF) showing the relationship between the in-focus position and the playback position of the series of refocus images on the display screen 100. This makes it possible to automatically generate and display a refocus video and a time-series line graph WF showing the in-focus position of the tracked subject. This also makes it possible to confirm which subject was being tracked.

[0116] Furthermore, the playback device 400 may have a directional sound collection unit 433 that identifies the sound source, and the imaging unit 431 may track the subject that is the sound source identified by the sound collection unit 433. This makes it possible to display a time-series line graph showing the focus position of the sound source. This also makes it possible to confirm which subject was being tracked as the sound source.

[0117] Furthermore, if the playback unit 435 reaches a specific playback position during video playback, it may play back a series of disparity images PI11 to PI55 on the display screen 100, which will be the source for combining the specific refocus image at the specific playback position from among the series of refocus images. This makes it possible to provide the user with video that appears to be played back in slow motion, from different viewing directions.

[0118] Furthermore, if the playback unit 435 reaches a specific playback position during video playback, it may pause video playback for a certain period of time and play multiple parallax images PI11 to PI55 on the display screen 100. This allows the user to concentrate on viewing multiple parallax images PI11 to PI55 for a certain period of time.

[0119] Furthermore, the playback device 400 includes a detection unit 432 that detects the drawing of the line graph WF in the coordinate system 120 on the display screen 100, and a generation unit 434 that, when the detection unit 432 detects the drawing of the line graph WF, generates a series of refocus images so that they are in focus at the focus position indicated by the line graph WF at each playback position of the line graph WF. The playback unit 435 then plays back a video (refocus video) composed of the series of refocus images generated by the generation unit 434 on the display screen 100. This makes it possible to generate a series of refocus images at the user's intended continuous focus positions.

[0120] Furthermore, icons are displayed on the display screen 100 to accept playback instructions for the video (refocused video) and thumbnail display instructions, and the playback device 400 has a detection unit 432 that detects playback instructions and thumbnail display instructions. When a playback instruction is detected by the detection unit 432, the playback unit 435 plays the video on the display screen 100, and when a thumbnail display instruction is detected by the detection unit 432, the display control unit 436 displays a thumbnail image of the subject at the in-focus position. This allows users to check what subject will be in focus in the refocused image before opening the video file.

[0121] Furthermore, the display control unit 436 may display thumbnail images of multiple subjects with different focus positions. For example, if the multiple thumbnail images to be displayed are thumbnail images of multiple refocus images with different playback positions and focus positions, it is possible to check what kind of refocus images will be displayed with different subjects in focus at different playback positions before opening the video file. Also, if the multiple thumbnail images to be displayed are thumbnail images of multiple refocus images with the same playback position but different focus positions, it is possible to check what kind of refocus images will be produced if the focus position is changed at the same playback position before opening the video file.

[0122] In the embodiment described above, a refocused image was generated using multiple parallax images. However, the playback device 400 may generate a refocused image using a color image and a monochrome image of the same subject obtained from a dual-lens camera. More specifically, for example, the playback device 400 calculates the amount of parallax between the color image and the monochrome image, generates a composite image of the color image and the monochrome image using the amount of parallax, calculates the distance to the subject using the amount of parallax, and applies a filter to the composite image according to the distance to the specific subject to be focused on, thereby generating a refocused image in focus on the specific subject.

[0123] Furthermore, the playback device 400 may generate a refocused image by performing ray tracing and rendering. This makes it possible to generate a higher-resolution refocused image than when parallax images are combined. Thus, a refocused image that does not use parallax images or combined images may also be applied to this embodiment. [Explanation of Symbols]

[0124] B Bar, WF Line graph, 100 Display screen, 400 Playback device, 412 Microlens array, 413 Focus adjustment unit, 421 Image sensor, 431 Imaging unit, 432 Detection unit, 433 Sound collection unit, 434 Generation unit, 435 Playback unit, 436 Display control unit

Claims

[Claim 1] A playback unit that plays a video on a display screen composed of composite images whose focus position can be changed by combining multiple images, A display control unit controls the display screen to display information showing the relationship between the focus position and playback position of the composite image, and displays information showing the playback position of the composite image in relation to the focus position of the composite image. A regeneration device characterized by having the following features.

Images