Reproduction device

Abstract

To clarify the relationship between the in-focus position and the reproduction position of each of a series of refocused images.SOLUTION: A reproduction device includes a reproduction unit that reproduces a moving image on a display screen, in which the moving image is composed of a composite image whose focus position can be changed by combining multiple images, and a display control unit that controls the display screen to display information indicating the relationship between the focus position of the composite image and the reproduction position on the display screen.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a playback device. [Background technology]

[0002] Conventionally, an imaging device that generates a refocused image by changing the focus position of the image has been disclosed (for example, see Patent Document 1 below). In this conventional technology, the relationship between the focus position of each of a series of refocused images and the playback position at which the series of refocused images are played back is unclear. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2013-247478 A Summary of the Invention

[0004] A playback device according to one aspect of the invention disclosed in this application has a playback unit that plays a moving image on a display screen, the moving image being composed of a composite image whose focus position can be changed by combining multiple images, and a display control unit that controls the display screen to display information indicating the relationship between the focus position of the composite image and the playback position on the display screen. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 is an explanatory diagram showing an example of a display screen of a refocused moving image. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of an operation on a display screen of a refocused moving image. [Diagram 3] FIG. 3 is an explanatory diagram showing an example of changing the line width of a line graph. [Figure 4] FIG. 4 is a block diagram showing an example of the configuration of a playback device. [Diagram 5] FIG. 5 is an explanatory diagram showing the relationship between the microlens array and the imaging element. [Figure 6]FIG. 6 is an explanatory diagram illustrating an example of a parallax image. [Figure 7] FIG. 7 is an explanatory diagram showing an example of generating a refocused image. [Figure 8] FIG. 8 is a flowchart showing an example of a recording process procedure of a refocused moving image by the playback device. [Figure 9] FIG. 9 is a flowchart 1 showing an example of a procedure for thinning out refocused moving images. [Figure 10] FIG. 10 is a flowchart 2 showing an example of a procedure for thinning out refocused moving images. [Figure 11] FIG. 11 is an explanatory diagram showing a first modification of the line graph displayed in the second display area by the display control unit. [Figure 12] FIG. 12 is an explanatory diagram showing a second modification of the line graph displayed in the second display area by the display control unit. [Figure 13] FIG. 13 is an explanatory diagram showing a third modification of the line graph displayed in the second display area by the display control unit. [Figure 14] FIG. 14 is an explanatory diagram showing a fourth modified example of the line graph displayed in the second display area by the display control unit. [Figure 15] FIG. 15 is an explanatory diagram showing a fifth modified example of a line graph displayed in the second display area by the display control unit. [Figure 16] FIG. 16 is an explanatory diagram showing a sixth modified example of a line graph displayed in the second display area by the display control unit. [Figure 17] FIG. 17 is a flowchart illustrating an example of a procedure for recombining refocused images. [Figure 18] FIG. 18 is an explanatory diagram showing an example of recombination of refocused images after updating the depth of field. [Figure 19] FIG. 19 is an explanatory diagram showing a display example on another device. [Figure 20] FIG. 20 is an explanatory diagram showing an example of a two-screen display. [Figure 21] FIG. 21 is an explanatory diagram showing a display example of one frame cut out. [Figure 22]FIG. 22 is a flowchart illustrating an example of a one-frame extraction process procedure. [Figure 23] FIG. 23 is a flowchart illustrating an example of a process for generating a refocused moving image from a drawing line graph. [Figure 24] FIG. 24 is an explanatory diagram showing an example of thumbnail display for a moving image file. [Diagram 25] FIG. 25 is a flowchart showing an example of a thumbnail image display processing procedure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] <Relationship between the focus position and playback position of a series of frames in a refocused video> The relationship between the focus position and the playback position of a series of frames in a refocused moving image will be described with reference to FIGS.

[0007] FIG. 1 is an explanatory diagram showing an example of a display screen of a refocused moving image. A refocused moving image is a series of refocused images in time series. A refocused image is a composite image obtained by combining a plurality of images of a certain subject. When a parallax image is used, a refocused image is a composite image obtained by combining a plurality of images of a certain subject with different parallax. A refocused image is a composite image that can be recombined by changing the focus position after shooting.

[0008] 1, a display screen 100 is displayed on, for example, a display of a digital camera, a digital video camera, a smartphone, a tablet, a portable game machine, or a personal computer. The display is, for example, a touch panel type.

[0009] The display screen 100 has a first display area 101 and a second display area 102. The first display area 101 is an area for playing back and displaying a series of refocused images. In the first display area 101, a composite image (refocused image RF4) at a certain playback position is displayed that includes objects O1 to O3. Note that, here, of the objects O1 to O3, the object O1 is closest to the original shooting location, and the object O3 is the furthest from the original shooting location.

[0010] The second display area 102 is an area for displaying information indicating the relationship between the focus positions and playback positions of a series of refocused images. The information indicating the relationship between the focus positions and playback positions of a series of refocused images is displayed in a coordinate system 120 configured by, for example, a time axis 121 (t-axis) indicating the playback positions of the series of refocused images and a distance axis 122 (z-axis) indicating the focus positions of the series of refocused images. The distance axis 122 indicates the focus position, for example, the focal length between the imaging element and the imaging lens.

[0011] In the coordinate system 120, a line graph WF showing a change in focus position depending on the playback position of a series of refocus images is displayed as an example of information showing the relationship between the focus position and the playback position of a series of refocus images. That is, the line graph WF shows the focus position of the refocus image at a certain playback position. For example, the operator can change the shape of the line graph WF with a finger through a user interface. Points P1 to P4 are points on the line graph WF (hereinafter collectively referred to as points P), and are generated, for example, every time a predetermined playback position interval or a focus position is changed, or by a user's specification.

[0012] Also, a bar B indicating the playback position of the refocus moving image is displayed in the second display area 102. The bar B moves from left to right as the refocus moving image is played back. The bar B in FIG. 1 indicates the playback position t4. The refocus image RF4 displayed in the first display area 101 is the refocus image at the playback position t4. The focus position at this time is d2, and the focus is on the object O2.

[0013] Also, in the first display area 101, related information 111 indicating which position (coordinate) of the display image was plotted at which time is displayed. For example, related information 111 (P1, time t1) is information corresponding to point P1 on the line graph WF, related information 111 (P4, time t4) is information corresponding to point P4 on the line graph WF, and related information 111 (P2, time t2) is information corresponding to point P2 on the line graph WF. Note that, for the related information 111 corresponding to point P, a mark of the same color or shape as the mark of point P in the second display area 102 may be displayed in the related information 111. For example, if the mark of point P1 is a white circle in the second display area 102, the related information 111 of the object O1 corresponding to point P1 is also a white circle of the same color and shape. In this case, the character string (P1, time t1) indicating the related information 111 may be deleted. Also, point P2 may be a different color or shape from point P1. For example, if the mark of point P2 in the second display area 102 is a white triangle, the related information 111 of the object O2 corresponding to point P2 is also a white triangle of the same color and shape. In this case, the character string (P4, time t4) indicating the related information 111 may be deleted. The same applies to point P2.

[0014] In this way, the relationship between the focus position and the playback position of each of a series of refocused images becomes clear, and the user can intuitively see which subject in which refocused image at which playback position is in focus at which focus position.

[0015] Specifically, for example, the user can generate point P on the line graph WF and adjust the depth position of the pointer point P with touch focus, and then tap or click on the subject to update point P to a focused position on the line graph WF. This eliminates the need to fine-tune the line graph WF each time a subject is to be focused on, improving convenience.

[0016] For example, a case where a point P is newly generated will be described. It is assumed that there is a touch autofocus function on the first display area 101. The touch autofocus function is an autofocus function that allows a user to specify a focus position using a touch interface. When a user wants to focus on a certain position (x1, y1) on the first display area 101 at a playback time t1 of a video, the user touches the position (x1, y1) (for example, a position in the object O1) on the first display area 101, and the object O1 including the position (x1, y1) is autofocused. At this time, the focus position of the object O1 is updated, so that a point P1 is generated at a position on the line graph specified by the updated focus position of the object O1 and the playback time t1. Then, by associating the position (x1, y1) with the point P1, the refocused image at the playback time t1 and the line graph WF can be synchronized.

[0017] A case where the position of point P is adjusted will be described. It is assumed that the first display area 101 has a touch autofocus function. It is assumed that point P2 is generated on the line graph WF at playback time t2 and is not associated with a position on the first display area 101. When the user wants to focus on position (x2, y2) at playback time t2, the user selects point P2 on the line graph WF and then taps or clicks on the subject (for example, object O3 including position (x2, y2)) on which the user wants to focus in the first display area 101. This results in a composite image in which the object O3 is in focus. At this time, the focus position of the object O3 is updated, and the focus position of point P2 on the line graph WF is updated and point P is associated with position (x2, y2).

[0018] Note that after the above processing, if the user changes the focus position and time position on the line graph WF for points P1 and P2, the focus will not match at the time and coordinates before the change. However, it can be used in the following two cases: (1) when the user does not edit point P at all on the line graph WF after plotting with touch focus; or (2) when the user edits point P on the line graph WF after plotting with touch focus and also retains the differential information before and after the change. In particular, for case (2), it is useful when returning the changed point P to its original position.

[0019] Here, as an example of information showing the relationship between the focus position and the playback position of a series of refocused images, a line graph WF is cited. However, any information such as a bar graph can be used as long as the relationship between the focus position and the playback position can be visually understood by the user.

[0020] Figure 2 is an explanatory diagram showing an example of operations on the display screen 100 of the refocused moving image. In (A) to (C), in (A) to (C), a refocused image at the playback position t4 is displayed in the first display area 101. Note that bar B can be slid in the direction of the time axis 121 by the user's operation.

[0021] Point P4 in (A) indicates the focus position d2 at the playback position t4 on bar B as shown in Figure 1. In the case of the focus position d2, when a plurality of images with different parallaxes at the playback position t4 are synthesized, a refocused image RF41 focused on object O2 is generated.

[0022] (B) is an example in which the focus position d2 is changed to the focus position d1 (<d2) by the user's operation at the playback position t4. In the case of the focus position d1, when a plurality of images with different parallaxes at the playback position t4 are synthesized, a refocused image RF42 focused on object O1 is generated. Also, the line segment between P4 and P1 is linearly interpolated by the changed point P4 and the previous point P1. Similarly, the line segment between P4 and P2 is linearly interpolated by the changed point P4 and the next point P2.

[0023] (C) is an example in which the user operates the playback position t4 to change the focus position d2 to focus position d3 (>d2). In the case of the focus position d3, when multiple images with different parallaxes at the playback position t4 are synthesized, a refocused image RF43 focused on the object O3 is generated. In addition, the line segment between P4 and P1 is linearly interpolated between the changed point P4 and the previous point P1. Similarly, the line segment between P4 and P2 is linearly interpolated between the changed point P4 and the next point P2.

[0024] In this way, the user can freely change the focus position of the refocused image at any playback position. In addition, a sudden change in the focus position is suppressed by linearly interpolating between the point where the focus position is changed and the adjacent point. That is, since the focus position is gradually changed, a refocused moving image that does not feel unnatural can be played back by synthesizing multiple images with different parallaxes at each playback position on the linearly interpolated line segment. In the above example, linear interpolation is taken as an example, but curved interpolation may also be used.

[0025] FIG. 3 is an explanatory diagram showing an example of changing the line width of a line graph. In addition, in each symbol shown in FIG. 3, the symbol before the change is suffixed with "a", and the symbol after the change is suffixed with "b". (A) shows the line graph WFa before the change of the line width, and is the same as (A) of FIG. 2. The line width wa of the line graph WFa corresponds to the depth of field. The image in this case is a refocused image RF4a. For example, by pinching out the line graph WFa by a user's operation, the line width wa of the line graph WFa widens, and it becomes a line graph WFb having a line width wb. Since the depth of field becomes deeper corresponding to the line width w, the refocused image RF4b at the playback position t4 becomes an image focused in the range of the depth of field after the change shown by the dashed line, as shown in (B).

[0026] Also, in the state of (B), for example, by pinching in on the line graph WFb, the line width w of the line graph WF narrows, and the line graph WF returns to the line graph WFa with the line width wa. As a result, the depth of field becomes shallower, so that the refocused image at the playback position t4 becomes an image focused within the range of the changed depth of field, as shown in (A). That is, in (A), the objects O1 and O3 become more blurred.

[0027] In this way, the user can change the depth of field by freely changing the line width w of the line graph WF. Although FIG. 3 describes an example of changing the line width w of the entire line graph WF, the line width w of the line segment between two points on either side of the pinched-in or pinched-out position on the line graph WF may be changed. For example, when pinching out between points P1 and P2 in (A), the line width of the line segment between points P1 and P2 in (B) changes from wa to wb. Similarly, when pinching out between points P1 and P2 in (A), the line width of the line segment between points P1 and P2 in (B) changes from wb to wa.

[0028] <Example of hardware configuration of playback device> FIG. 4 is a block diagram showing an example of the configuration of a playback device. (A) is a block diagram showing an example of the hardware configuration of a playback device. The playback device 400 plays and displays refocused moving images. The playback device 400 is, for example, a digital camera, a digital video camera, a smartphone, a tablet, a portable game console, or a personal computer. The playback device 400 may have a function of capturing refocused images and refocused moving images. In FIG. 4, a case where the playback device 400 is a digital camera will be described as an example.

[0029] The playback device 400 has a lens unit 401 and a camera body 402. The lens unit 401 and the camera body 402 may be detachable or may be integrated. The lens unit 401 has an imaging lens 411, a microlens array 412, and a focus adjustment unit 413.

[0030] The imaging lens 411 is an optical system including, for example, a focus lens or a zoom lens, and collects light from a subject and outputs the light to the microlens array 412. The microlens array 412 is a collection of microlenses arranged in a matrix. Each microlens collects light from the pupil region of the corresponding imaging lens 411 and outputs the light to a corresponding region in the image sensor 421. The focus adjustment unit 413 adjusts the focus position of the imaging lens 411 by a user operation or an autofocus function.

[0031] The camera body 402 includes an image pickup element 421, a processor 422, a storage device 423, a processing circuit 424, a drive circuit 425, an input device 427, a display device 428, and a communication I / F 429. These are connected by a bus 430. The image pickup element 421 receives light incident from an image pickup lens 411 through a microlens array 412, converts the light into an electrical signal, and stores RAW data obtained from the electrical signal in a storage device 423. For example, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor is used as the image pickup element 421. The processor 422 controls the playback device 400. The storage device 423 stores a playback program, and the processor 422 executes the playback program stored in the storage device 423.

[0032] The processing circuit 424 is a circuit for processing data, and includes, for example, a compression / decompression processing circuit for compressing and decompressing data, an image processing circuit as an image processing function, and an audio processing circuit for processing audio data. The compression / decompression processing circuit, for example, converts RAW data into image data in JPEG format, generates compressed moving images, and decompresses compressed moving images. The image processing circuit, for example, performs scene analysis, shape change of an image of a subject, position change of an image of a subject (tracking of a subject), exposure correction, face recognition, color temperature detection, and generation of a refocused image. The audio processing circuit, for example, analyzes the volume of audio. The compression / decompression processing circuit, image processing circuit, and audio processing circuit may be implemented in the storage device 423 as a compression / decompression processing program, an image processing program, and an audio processing program to be executed by the processor 422, respectively.

[0033] The driving circuit 425 drives and controls the shutter, the aperture mechanism, and the lens unit 401. The recording device stores image data and video data. The recording device is detachable from the camera body 402. The input device 427 accepts operation input from the user and audio input from the outside world. The input device 427 includes, for example, a shutter button and other buttons necessary for shooting, a touch panel, a dial, a ring, and a lever. The display device 428 displays a through image of the subject, a shot refocused image, and a refocused video image. The communication I / F 429 transmits and receives data to and from an external device.

[0034] 1B is a block diagram showing an example of a functional configuration of the playback device 400. The playback device 400 has an image capturing unit 431, a detection unit 432, a sound collection unit 433, a generation unit 434, a playback unit 435, and a display control unit 436. The image capturing unit 431 is composed of, for example, a lens unit 401, an image sensor 421, and a processing circuit 424. The image capturing unit 431 captures a group of images having different parallaxes of a subject, and stores them in the storage device 423 as RAW data.

[0035] The detection unit 432 detects the focus position of the imaging lens 411 by the focus adjustment unit 413. The detection unit 432 also detects the movement of a specific focus position on the line graph WF and the change in the line width of the points on the line graph WF. The detection unit 432 also detects the drawing of the line graph WF in the coordinate system 120 on the display screen 100. The detection unit 432 also detects an instruction to play the video file of the refocused video image and an instruction to display the thumbnail image. The detection unit 432 realizes its function by having the processor 422 execute a playback program stored in the storage device 423. The sound collection unit 433 acquires sound from a sound source by a directional microphone, which is an example of the input device 427.

[0036] The generating unit 434 generates a series of refocused images from the RAW data, and also generates a refocused moving image from the series of refocused images, and stores the refocused moving image in the storage device 423. For example, when the detecting unit 432 detects a movement of a specific focusing position in the direction of the distance axis 122, the generating unit 434 recomposes a plurality of images (disparity images described later) with different parallaxes, which are the synthesis sources of a refocused image at a specific playback position corresponding to the specific focusing position, so as to focus on the focusing position after the specific focusing position has moved. The generating unit 434 may be configured by the processing circuit 424, or may realize its function by causing the processor 422 to execute a playback program stored in the storage device 423.

[0037] The playback unit 435 acquires the refocused moving image and the refocused image stored in the storage device 423 from the storage device 423 and plays them on the display screen 100. The display control unit 436 controls the display screen 100, or when there is an input from the input device 427 or a user operation on the display screen 100, displays information (for example, a line graph WF or a bar B) indicating the relationship between the in-focus position and the playback position of a series of refocused images on the display screen 100. In this case, the generation unit 434 recomposes the refocused images based on the updated information and reconstructs the refocused moving image. The display control unit 436 displays or erases the related information 111 on the display screen 100.

[0038] <Example of refocused image generation> FIG. 5 is an explanatory diagram showing the relationship between the microlens array 412 and the imaging element 421. (A) shows the correspondence between the microlens array 412 and the imaging element 421. (B) shows the correspondence between the microlenses Lij (i, j are integers equal to or greater than 1. In this example, 1≦i≦5, 1≦j≦5) and the pixels Pijkl (k, l are integers equal to or greater than 1. In this example, 1≦k≦5, 1≦l≦5) constituting the microlens array 412. Each circular figure indicates a microlens Lij. A collection of the microlenses Lij shown by the circular figures is the microlens array 412. The imaging element 421 has a plurality of pixels Pijkl. As an example, the imaging element 421 has 25×25 pixels. Note that the 5×5 pixel area surrounded by a thick frame is referred to as a pixel area Cij. One pixel area Cij corresponds to one microlens Lij. The pixel area Cij is a collection of pixels Pijkl.

[0039] 6 is an explanatory diagram showing an example of a parallax image. A parallax image PIkl is each of a plurality of images with different parallaxes obtained by photographing the same subject. Specifically, a parallax image PIkl is an image in which pixels Pijkl of each pixel region Cij are arranged according to the arrangement positions of the microlenses Lij. For example, a parallax image PI11 is an image in which pixels Pij11 of each pixel region Cij are arranged according to the arrangement positions of the microlenses Lij, and a parallax image PI55 is an image in which pixels Pij55 of each pixel region Cij are arranged according to the arrangement positions of the microlenses Lij.

[0040] 7 is an explanatory diagram showing an example of generating a refocused image. The parallax images PI11 to PI55 are, for example, a group of images in which the object O1 is in focus. The processing circuit 424 serving as the generating unit 434 or the processor 422 executing an image processing program synthesizes the parallax images PI11 to PI55 to generate a refocused image RF in which the object O1 is in focus.

[0041] <Example of operation process> 8 is a flowchart showing an example of a recording process procedure of a refocused moving image by the playback device 400. First, the playback device 400 detects the in-focus position from the focus adjustment unit 413 using the detection unit 432, and stores the detected in-focus position in the storage device 423 (step S801). Next, the playback device 400 associates the RAW data at the detected in-focus position with the detected in-focus position at the corresponding time in the storage device 423 (step S802). This provides a series of RAW data associated with the in-focus position for each time. The playback device 400 determines whether or not recording has ended based on a user operation input or a timer (step S803). If recording has not ended (step S803: No), the process returns to step S801.

[0042] On the other hand, if the recording is ended (step S803: Yes), the playback device 400 generates a series of refocus images from the series of acquired RAW data by the generation unit 434 (step S804). Specifically, for example, the playback device 400 converts each RAW data of the series of acquired RAW data into image data such as JPEG. Each of the converted series of image data includes a plurality of image data with different parallaxes. The playback device 400 generates a refocus image at a certain time by synthesizing the plurality of image data at the focusing position associated in step S802. As a result, a refocus image is obtained for each series of acquired RAW data. The playback device 400 compresses the series of refocus images into a moving image format (step S805). Then, the playback device 400 holds the compressed data compressed into the moving image format as a refocus moving image in the storage device 423 (step S806).

[0043] 9 and 10 are flowcharts showing an example of a procedure for thinning out refocused moving images. In Fig. 9, in order to reduce the load on the processing circuit 424, the playback device 400 arranges the focus positions of refocused images selected at a predetermined interval in a time series, and performs linear interpolation (or curved interpolation) between the selected refocused images. This makes it possible to generate a line graph WF showing the relationship between the focus positions and the playback positions.

[0044] First, the playback device 400 sets n=1 (step S901). In Fig. 9 and Fig. 10, n indicates the appearance position of the refocused image, which is the order of appearance of the refocused image, that is, the playback position. The playback device 400 reads the RAW data RDn at the appearance position n by the detection unit 432 (step S902). The playback device 400 detects the current focus position fn from the focus adjustment unit 413 and stores it in the storage device 423 in order to display it in step S905 (step S903).

[0045] The playback device 400 generates a refocused image RFn from the RAW data RDn read in step S902 at the focusing position fn detected in step S903 by the generation unit 434 (step S904). The playback device 400 displays the generated refocused image RFn in the first display area 101 of the display screen 100 (step S905). The refocused moving image being played back is displayed in the first display area 101 until a playback end signal is detected.

[0046] After that, the playback device 400 waits for detection of a playback start signal (step S906: No). If a playback start signal is detected (step S906: Yes), the playback device 400 associates and holds the refocused image RFn and its focus position fn (step S907). The playback device 400 updates n by adding a to n (step S908). a (≧1) is a preset fixed value (integer) that determines the next RAW data to be synthesized. If a is 1, the playback device 400 generates a refocused image RFn without thinning out. If a is a value of 2 or more, the playback device 400 generates a refocused image RFn while thinning out.

[0047] After that, the playback device 400 reads the RAW data RDn at the refocus image position n updated in step S908 (step S909). The playback device 400 causes the detection unit 432 to detect the current focus position fn from the focus adjustment unit 413 and stores it in the storage device 423 (step S910). The playback device 400 causes the generation unit 434 to generate a refocus image RFn from the RAW data RDn read in step S909 at the focus position fn detected in step S910 (step S910).

[0048] After that, if the playback end signal is not detected (step S912: No), the playback device 400 returns to step S907. On the other hand, if the playback end signal is detected (step S912: Yes), the playback device 400 proceeds to step S1001 in Fig. 10. This completes the composition of the refocused image group.

[0049] 10, playback device 400 determines whether a is equal to or greater than 2 (step S1001). If a is not equal to or greater than 2 (step S1001: No), a=1, and playback device 400 proceeds to step S1004.

[0050] On the other hand, if a is a value of 2 or more (step S1001: Yes), the playback device 400 causes the display control unit 436 to interpolate the focus positions of the refocus images (thinned refocus images) that have not been synthesized due to thinning, using the focus positions detected in steps S903 and S910 (step S1002). Such an interpolated focus position is referred to as an interpolated focus position. The playback device 400 causes the generation unit 434 to generate a refocus image RFn for each RAW data RDn corresponding to the remaining refocus images that have not been thinned (step S1003).

[0051] Then, the playback device 400 compresses the series of refocus images RFn into a moving image format by the playback unit 435 (step S1004), and stores the compressed refocus moving images in the storage device 423 (step S1005). This ends the thinning process of the refocus moving images.

[0052] After that, when the playback device 400 detects a playback start signal for the refocused moving image generated in step S1005, the playback unit 435 plays the refocused moving image in the first display area 101 of the display screen 100, and the display control unit 436 generates a line graph WF showing the relationship between the focus position and the playback position and displays it in the second display area 102.

[0053] <Example of changing line graph> 11 to 16 are explanatory diagrams showing examples of changes to the line graph WF displayed in the second display area 102 by the display control unit 436. In FIG. 11, (A) shows an initial state. The line graph WF includes a group of lines connecting points P10 to P14 with line segments ls1 to ls4. The line graph WF has a line width w. A shift button 1100 is displayed in the first display area 101. In FIG. 11, the shift button 1100 is in an OFF state. In the OFF state, the positions of points P10 to P14 on the line graph WF can be changed. Therefore, in FIG. 11, the depth of field is not changed. In addition, in each of the symbols in FIG. 11 to 16, the symbols before the change of the line graph WF have an a suffix added, and the symbols after the change have an b suffix added.

[0054] (B) shows a state in which point P11a has been moved upward from the state of (A), that is, in the depth direction of the distance axis 122, by a drag operation. By dragging point P11a, the line segment ls1 between points P10 and P11a changes from being parallel to the time axis 121 to rising to the right, becoming a line segment ls1b between points P10 and P11b. Similarly, the line segment ls2 between points P11a and P12 changes from being parallel to the time axis 121 to falling to the right, becoming a line segment ls2b between points P11b and P12. The line width w does not change. In other words, there is no change in the depth of field.

[0055] (C) shows a state in which point P12a has been moved from the state of (A) to the left in parallel with the time axis 121 by a drag operation. By dragging point P12a, the inclination of the line segment ls3a between point P12a and point P13 changes, and it becomes a line segment ls3b between point P12b and point P13. The line width w does not change. In other words, there is no change in the depth of field.

[0056] (D) shows a state in which point P12a has been moved from the state of (A) to the right in parallel with the time axis 121 by a drag operation. By dragging point P12a, the inclination of the line segment ls3a between point P12a and point P13 changes to become a line segment ls3b between point P12b and point P13. The line width w does not change. In other words, there is no change in the depth of field.

[0057] (E) shows a state in which point P11a has been moved from the state of (A) to the right in parallel with the time axis 121 and also moved upward, i.e., in the depth direction of the distance axis 122, by a drag operation. By dragging point P11a, the line segment ls1a between points P10 and P11a changes from being parallel to the time axis 121 to rising to the right. Similarly, the line segment ls2a between points P11a and P12 changes from being parallel to the time axis 121 to falling to the right. The line width w does not change. In other words, there is no change in the depth of field.

[0058] FIG. 12 shows an example of dragging the shift button 1100 in the OFF state. In FIG. 12, (A) shows the initial state, similar to (A) in FIG. 11. (B) shows a state in which the shift button 1100 is moved upward from the state of (A) by a drag operation. By dragging the shift button 1100, the points P10a to P14a move in the same direction as the movement direction of the shift button 1100. In this case, the points P10 to P14 move upward, that is, in the depth direction of the distance axis 122, and become points P10b to P14b. Accordingly, the line segments ls1a to ls4a also move upward and become line segments ls1b to ls4b. The line width w does not change. As a result, the line graph WFa is changed to a line graph WFb, and the focus position can be adjusted collectively over the entire range of the time axis 121. It should be noted that by dragging the shift button 1100 downward, the state returns from (B) to (A).

[0059] (C) shows a state in which the shift button 1100 is moved to the right from the state of (A) by a drag operation. By dragging the shift button 1100, the points P10a to P14a move in the same direction as the movement direction of the shift button 1100. In this case, the points P10a to P14a move to the right on the time axis 121 to become points P10b to P14b. Accordingly, the line segments ls1a to ls4a also move to the right to become line segments ls1b to ls4b. The line width w does not change. As a result, the line graph WFa is changed to a line graph WFb, and the focus position can be adjusted collectively over the entire range of the time axis 121. Note that by dragging the shift button 1100 to the left, the state returns from (C) to (A).

[0060] 13 shows a first example of a change in line width w when the shift button 1100 is in the ON state. (A) shows a state in which the playback device 400 detects the pressing of the shift button 1100 from the state shown in (A) of FIG. 11, and the shift button 1100 changes to ON.

[0061] (B) shows a state where, from the state of (A), a drag operation is performed downward, i.e., in the forward direction with respect to the distance axis 122, at point P11. By dragging point P11, the line width wa changes so as to become narrower from point P10 toward point P11 between points P10 and P11, and the line width becomes wb (< wa) at point P11. Similarly, the line width wa changes so as to become narrower from point P12 toward point P11 between points P11 and P12, and the line width becomes wb (< wa) at point P11. Thereby, it is possible to adjust so that the depth of field becomes shallower at point P11.

[0062] (C) shows a state where, from the state of (A), a drag operation is performed upward, i.e., in the depth direction with respect to the distance axis 122, at point P11. By dragging point P11, the line width wa changes so as to become wider from point P10 toward point P11 between points P10 and P11, and the line width becomes wb (> wa) at point P11. Similarly, the line width wa changes so as to become narrower from point P12 toward point P11 between points P11 and P12, and the line width becomes wb (> wa) at point P11. Thereby, it is possible to adjust so that the depth of field becomes deeper at point P11.

[0063] FIG. 14 shows a second example of the change in the line width w when the shift button 1100 is in the ON state. (A) shows a state where, from the state of (A) in FIG. 13, point P11 is dragged in the left direction along the time axis 121. By dragging in the left direction, the line width wa changes so as to become wider from point P11 toward point P10 between points P10 and P11, and the line width becomes wb (> wa) at point P10.

[0064] (B) shows a state where, from the state of (A) in FIG. 13, point P11 is dragged in the right direction along the time axis 121. By dragging in the right direction, the line width wa changes so as to become wider from point P11 toward point P12 between points P11 and P12, and the line width becomes wb (> wa) at point P12. Further, thereby, the line width wb changes so as to become narrower from point P12 toward point P13 between points P12 and P13, and the line width becomes wa at point P13.

[0065] (C) shows the state where, starting from the state of (A) in FIG. 13, point P11 is dragged to the left along the time axis 121 and then folded back and dragged to the right. By performing such a drag operation, the line width wa between points P10 and P11 changes so as to become narrower from point P11 toward point P10, and the line width becomes wb (<wa) at point P10.

[0066] (D) shows the state where, starting from the state of (A) in FIG. 13, point P11 is dragged to the right along the time axis 121 and then folded back and dragged to the right. By performing such a drag operation, the line width wa between points P11 and P12 changes so as to become narrower from point P11 toward point P12, and the line width becomes wb (<wa) at point P12. Further, thereby, the line width wb between points P12 and P13 changes so as to become wider from point P12 toward point P13, and the line width becomes wa at point P13.

[0067] FIG. 15 shows a third example of changing the line width w when the shift button 1100 is in the ON state. FIG. 15 shows the state where, starting from the state of (A) in FIG. 13, point P11 is dragged in the upper right diagonal direction. By performing the drag operation in the upward component, the line width wa between points P10 and P11 changes so as to become wider from point P10 toward point P11, and the line width becomes wb (>wa) at point P11. Similarly, the line width wb between points P11 and P12 changes so as to become narrower from point P11 toward point P12, and the line width becomes wbb (>wa) at point P11. Further, thereby, the line width wbb between points P12 and P13 changes so as to become narrower from point P12 toward point P13, and the line width becomes wa at point P13.

[0068] FIG. 16 shows an example of dragging the shift button 1100 in the ON state. (A) shows the state where the shift button 1100 is dragged upward starting from the state of (A) in FIG. 13. By dragging the shift button 1100, the line width w of the line graph WF expands from wa to wb, and the line graph is changed from line graph WFa to line graph WFb. Thereby, the depth of field can be adjusted over the entire reproduction section.

[0069] (B) shows the state where the shift button 1100 is dragged downward from the state of (A). By dragging the shift button 1100, the line width w of the line graph WF becomes narrower from wa to wb, and the line graph WFa is changed to the line graph WFb. The depth of field can be adjusted over the entire playback period.

[0070] (C) shows the state where the shift button 1100 is dragged in the right direction of the time axis 121 from the state of (A). By dragging in the right direction, the line width wa changes so as to become wider from point P11 to point P14 between points P11 to P14, the line width is wb (>wa) at point P12, wbb (>wb) at point P13, and wbbb (>wbb) at point P14. As a result, the line graph WFa is changed to the line graph WFb.

[0071] (D) shows the state where, after dragging the shift button 1100 in the right direction of the time axis 121 from the state of (A), it is dragged back in the left direction. By performing such a drag operation, the line width wa changes so as to become narrower from point P11 to point P14 between points P11 to P14, the line width is wb (<wa) at point P12, wbb (<wb) at point P13, and wbbb (<wbb) at point P14. As a result, the line graph WFa is changed to the line graph WFb.

[0072] <Example of refocused image recomposition processing procedure> Next, an example of the refocused image recomposition process when the change operations shown in FIGS. 11 to 16 are performed will be described.

[0073] 17 is a flowchart showing an example of a recombination process procedure for a refocused image. The playback device 400 detects whether the shift button 1100 is ON or OFF (step S1701). If it is OFF (step S1701: OFF), the playback device 400 updates the line graph WF in response to a drag operation in the OFF state by the display control unit 436 as shown in FIG. 11 or 12 (step S1702). Then, the playback device 400 identifies the playback section in which the focus position has been updated in step S1702 (step S1703).

[0074] After that, in step S1703, the playback device 400 determines whether or not there is an unselected playback position whose focus position has been updated among the playback positions indicated by points on the line graph within the playback section (step S1704). If there is an unselected playback position whose focus position has been updated (step S1704: Yes), the playback device 400 selects the RAW data RDn of the unselected playback position whose focus position has been updated and the updated focus position (step S1705). Then, the playback device 400 recomposes the refocused image at the updated focus position (step S1706) and returns to step S1704.

[0075] In step S1704, if there is no unselected playback position whose focus position has been updated (step S1704: No), the playback device 400 causes the generation unit 434 to reconstruct a compressed refocus moving image using the recomposed refocus image and the remaining refocus image whose focus position has not been updated (step S1712). This causes the playback device 400 to end the recomposition process of the refocus image.

[0076] Also, in step S1701, if the shift button 1100 is in the ON state (step S1701: OFF), the playback device 400 causes the display control unit 436 to update the line graph WF in response to the drag operation in the ON state (step S1707), as shown in Figures 13 to 16. Then, in step S1707, the playback device 400 identifies the playback section whose line width has been updated (step S1703).

[0077] After that, the playback device 400 judges whether there is an unselected updated playback position (step S1709). If there is an unselected updated playback position (step S1709: Yes), the playback device 400 selects the RAW data RDn of the unselected updated playback position, the focus position, and the updated line width (step S1705). Then, the playback device 400 causes the generation unit 434 to recombine the refocused image with a depth of field that becomes the updated line width (step S1711), and returns to step S1709.

[0078] In step S1709, if there is no unselected playback position (step S1709: No), the playback device 400 causes the generation unit 434 to reconstruct a compressed refocus moving image using the recomposed refocus image and the remaining refocus image whose line width has not been updated (step S1712). This causes the playback device 400 to end the recomposition process of the refocus image.

[0079] <Recomposition of refocused images after updating depth of field> Next, an example of recombination in the case where the refocused images generated as described above are pan-focused will be described.

[0080] Fig. 18 is an explanatory diagram showing an example of recombination of refocused images after updating the depth of field. In Fig. 18, a virtual aperture 1800 limits emission of light from the imaging lens 411 to the imaging element 421. In the imaging element 421, hatched pixels are pixels that receive light, and non-hatched pixels are pixels that do not receive light by the virtual aperture 1800. Note that because of the virtual aperture 1800, even non-hatched pixels actually receive light, but output signals are not adopted.

[0081] In (A), the virtual diaphragm 1800 passes light from pupil regions PR2 to PR4 among pupil regions PR1 to PR5 of the imaging lens 411. That is, even if the imaging element 421 receives light from pupil regions PR1 and PR5 and outputs an optical signal, the playback device 400 does not use it in the synthesis process of the refocused image. In (B), the virtual diaphragm 1800 passes light from pupil region PR3 among pupil regions PR1 to PR5 of the imaging lens 411. That is, even if the imaging element 421 receives light from pupil regions PR1, PR2, PR4, and PR5 and outputs an optical signal, the playback device 400 does not use it in the synthesis process of the refocused image.

[0082] Between (A) and (B), (B), in which the opening 1801 of the virtual aperture 1800 is more closed, has a deeper depth of field. That is, when pan focus is desired, the user performs an operation to thicken the line width w of the line graph WF, and the playback device 400 narrows the opening 1801 of the virtual aperture 1800 by the generation unit 434 as in (B). That is, the user can make the depth of field infinite by thickening the line width w of the line graph. In this way, after shooting, a refocused image corresponding to the change in the depth of field can be generated.

[0083] <Examples of display on other devices> Fig. 19 is an explanatory diagram showing a display example on another device. Fig. 19 shows a display example on a smartphone or tablet. In the case of a smartphone or tablet, since the size of the display screen 100 is limited, an adjustment area 1901 is displayed in the first display area 101. After specifying a point, the specified point can be moved in the direction of the distance axis 122 by dragging the slider 1902 of the adjustment area 1901 up or down.

[0084] <Example of 2 screen display> Fig. 20 is an explanatory diagram showing an example of a two-screen display. In Fig. 20, the first display area 101 has two windows 101A and 101B. The playback device 400 plays a refocus moving image at a current playback position in the first window 101A by the playback unit 435. The playback device 400 plays a preceding refocus moving image that is a predetermined time ahead of the current playback position in the second window 101B by the playback unit 435. In this case, the playback device 400 displays a bar B1 of the first window 101A and a bar B2 of the second window 101B in the second display area 102 by the display control unit 436.

[0085] For example, when changing the focus position while playing back a refocused moving image using focus adjustment unit 413, a shift in focus occurs if the subject moves forward or backward before operating focus adjustment unit 413. Therefore, by displaying a refocused moving image a predetermined time ahead (for example, several seconds) in advance in second window 101B, the user can know whether the subject will move forward or backward next, and shifting in focus can be prevented.

[0086] Furthermore, the playback device 400 may display the refocus image at the current playback position and the preceding refocus image in a superimposed manner in the first display area 101 by the playback unit 435. This allows the user to intuitively grasp the difference in the movements of the subjects with a time difference, and predict which subject will be focused on. In this case, either one of the refocus moving images may be displayed, for example, in a colored transparent manner. This allows the user to easily distinguish between the two refocus moving images.

[0087] Furthermore, the playback unit 435 may convert a series of refocus images constituting a preceding refocus moving image into pan-focus refocus images in the second window 101B, for example, by the method shown in Fig. 18. This allows the subject of the preceding refocus moving image to be displayed without blurring, allowing the user to check the subject to be focused by referring to the preceding refocus moving image.

[0088] Furthermore, the playback unit 435 may play back a refocus moving image having a different focus position from the refocus moving image played back in the first window 101A. In this case, the playback unit 435 may play back both refocus moving images synchronously, that is, so as to display refocus moving images at the same playback position.

[0089] <Subject tracking> 4B, the photographing unit 431 may capture an image while tracking the subject using the processing circuit 424. In this case, the generating unit 434 generates a series of refocus images with the in-focus position corresponding to the distance to the subject by synthesizing a plurality of parallax images PI11 to PI55 for each of the photographed series of subjects. This makes it possible to automatically generate and display a refocus moving image and a time-series line graph WF indicating the in-focus position of the tracked subject. This also makes it possible to confirm which subject was being tracked.

[0090] As an example of the input device 427, a sound collection unit 433 consisting of two or more microphones may be used. In this case, the two microphones identify the direction of the subject that is the sound source. The photographing unit 431 may then track the direction of the subject identified by the two microphones. This makes it possible to display a time-series line graph WF indicating the focal position of the sound source. This also makes it possible to confirm which subject was being tracked as the sound source.

[0091] <1 frame extraction> FIG. 21 is an explanatory diagram showing a display example of one-frame cutout. One-frame cutout is a process of selecting one of a series of refocus images constituting a refocus moving image, and displaying a plurality of parallax images PIkl that are the synthesis source of the selected refocus image. For example, the playback device 400 converts RAW data that is the synthesis source of the selected refocus image into an image such as jpeg, and displays it as a plurality of parallax images PI11 to PI55 by the playback unit 435. By displaying a plurality of parallax images PI11 to PI55 at the same playback position for the selected refocus image during playback of the refocus moving image, it is possible to provide an image as if it were played in slow motion in different line of sight directions. Note that the playback device 400 does not need to display all of the plurality of parallax images PI11 to PI55 for the cut-out refocus image, and may display a predetermined number of thinned images.

[0092] 22 is a flowchart showing an example of a one-frame cutout process procedure. The playback device 400 specifies one or more playback positions from which to cut out a refocused image through a user's operation input (step S2201). Next, the playback device 400 waits for a playback start signal from the playback unit 435 (step S2202: No). If the playback start signal is detected (step S2202: Yes), the playback device 400 starts playing back the refocused moving image through the playback unit 435. Then, if the playback device 400 has not reached the cutout playback position (step S2203: No), the playback device 400 proceeds to step S2206.

[0093] On the other hand, if the cut-out playback position has been reached (step S2203: Yes), the playback device 400 causes the playback unit 435 to acquire from the storage device 423 the multiple parallax images PI11 to PI55 that are to be the synthesis source of the refocused image at the cut-out playback position (step S2204), and displays them on the display screen 100 (for example, the first display area 101) (step S2205). In this case, the playback of the refocused moving images may be stopped for a certain period of time. This allows the user to concentrate on and view the multiple parallax images PI11 to PI55 for the certain period of time. Then, if the playback end signal has not been detected (step S2206: No), the playback device 400 returns to step S2203.

[0094] When displaying the multiple parallax images PI11 to PI55 in step S2205, the playback device 400 may display the multiple parallax images PI11 to PI55 in an order corresponding to the arrangement order of the microlenses Lij in the microlens array 412. This makes it possible to display the multiple parallax images PI11 to PI55 so that the parallax changes continuously. In this way, by extracting one frame, it is possible to provide a video as if it were being played in slow motion from different viewing directions.

[0095] Furthermore, in the case where an arbitrary parallax image PIkl is selected from among the plurality of parallax images PI11 to PI55 by a user operation in the one-frame cutout, the playback device 400 may re-synthesize a selected refocused image with the selected parallax image PIkl as the focus position by the generation unit 434. This allows the user to obtain a refocused image in which the focus is on the intended parallax image.

[0096] <Example of generating refocused video images from a drawn line graph> In the above example, an example has been described in which a series of refocus images is generated and then a line graph WF indicating the playback position and focus position of each refocus image is generated, but the playback device 400 may first draw a line graph WF by a user operation, and then generate a series of refocus images at focus positions at each playback position of the drawn line graph WF by the generation unit 434. This makes it possible to generate a series of refocus images at continuous focus positions intended by the user.

[0097] FIG. 23 is a flowchart showing an example of a process procedure for generating a refocused moving image from a drawn line graph. First, the playback device 400 detects the drawing of the line graph WF in the second display area 102 (step S2301). Next, the playback device 400 records a combination of a playback position and a focus position, which are points on the line graph WF (step S2302). Then, the playback device 400 judges whether or not there is an unselected playback position in the combination (step S2303). If there is an unselected playback position (step S2303: Yes), the playback device 400 selects the unselected playback position (step S2304) and obtains a focus position corresponding to the selected playback position from the drawn line graph WF (step S2305). Then, the playback device 400 reads the RAW data of the selected playback position from the storage device 423 by the generation unit 434, and synthesizes the RAW data with the obtained focus position to generate a refocused image (step S2306).

[0098] Specifically, for example, the playback device 400 converts the acquired RAW data into a plurality of image data such as JPEG. The converted plurality of image data are a group of image data having different parallaxes. The playback device 400 generates a refocused image by synthesizing the group of image data at the associated focusing position. As a result, the refocused image is obtained from the acquired RAW data. After that, the process returns to step S2302.

[0099] In step S2303, if there is no unselected playback position (step S2303: No), the playback device 400 compresses the series of refocus images generated in step S2306 into a video format by the playback unit 435 (step S2307). Then, the playback device 400 stores the compressed data compressed into the video format as a refocus video image in the storage device 423 (step S2308). In this way, according to Figs. 22 and 23, a series of refocus images can be generated at consecutive focus positions intended by the user.

[0100] <Thumbnail display> Fig. 24 is an explanatory diagram showing an example of thumbnail display of a video file. In (A) of Fig. 24, the playback device 400 displays an icon IC of a video file of a refocused video image in a folder 2400 on the display screen 100 by the display control unit 436. The icon IC is a symbol that accepts an instruction to play a refocused video image and an instruction to display a thumbnail image. When a user operates a mouse cursor MC to give an instruction to play (for example, single-click or double-click) an icon IC, the playback device 400 plays the refocused video image by the playback unit 435.

[0101] On the other hand, when a user instructs to display a thumbnail image on a certain icon IC with the mouse cursor MC (for example, by pointing at the icon IC), as shown in (B), the playback device 400 displays a thumbnail image S related to the subject of a refocus image focused on the subject in the refocus moving image played back with the instructed icon IC by the display control unit 436. The thumbnail image S may be a reduced image of the refocus image, or may be a reduced image of the focused subject. This allows the user to check what kind of subject is in focus in the refocus image to be displayed before opening the moving image file.

[0102] Furthermore, the thumbnail image S to be displayed may be a plurality of thumbnail images with different focus positions (for example, three thumbnail images Sa, Sb, Sc). For example, the plurality of thumbnail images Sa, Sb, Sc to be displayed may be a plurality of thumbnail images with different playback positions and focus positions. This allows the user to check what refocus images are displayed at different playback positions and with what subjects are in focus, respectively, before opening a video file. Furthermore, the plurality of thumbnail images Sa, Sb, Sc to be displayed may be a plurality of thumbnail images with the same playback position and different focus positions. This allows the user to check what refocus images will be displayed if the focus position is changed at the same playback position, before opening a video file.

[0103] 25 is a flowchart showing an example of a thumbnail image display process procedure. The playback device 400 waits for an instruction to display a thumbnail image (step S2501). If a display instruction is detected (step S2501: Yes), the playback device 400 identifies a video file corresponding to the icon IC instructed to be displayed (step S2502), and displays one or more thumbnail images with different focus positions near the icon IC (step S2503). Then, the process returns to step S2501.

[0104] If a thumbnail display instruction has not been detected in step S2501 (step S2501: No), the playback device 400 waits for an instruction to play the video file (step S2504). If there is no instruction to play (step S2504: No), the process returns to step S2501.

[0105] On the other hand, if there is a playback instruction (step S2504: Yes), the playback device 400 plays the video file. Then, the playback device 400 acquires one or more refocus images with different focus positions from the refocus video (step S2505). Then, the playback device 400 converts the acquired refocus video into thumbnail images and stores them in the storage device 423 (step S2506). Then, the process returns to step S2501. In this way, according to the thumbnail display process, it is possible to check what kind of subject is in focus in the refocus image to be displayed before opening the video file.

[0106] As described above, the playback device 400 according to this embodiment includes a playback unit 435 that plays a moving image composed of a series of refocus images, which are a time-series collection of refocus images obtained by combining a plurality of images (parallax images PI11 to PI55) having different parallaxes, on the display screen 100, and a display control unit 436 that controls the display screen 100 to display information indicating the relationship between the focus positions of the series of refocus images and the playback position on the display screen 100. This allows the user to intuitively grasp the focus positions of each of the series of refocus images being played back.

[0107] Furthermore, the display control unit 436 may display information indicating the playback position of a refocused image in association with the focus position of the refocused image. This makes it possible to identify which subject of which refocused image is at which focus position. Therefore, it is possible to identify continuous changes in the focus position in the refocused moving image.

[0108] Moreover, the information indicating the relationship between the focus position and the playback position of the series of refocused images is, for example, a line graph WF indicating the change in focus position depending on the playback position of the series of refocused images in a coordinate system 120 constituted by a first axis (time axis 1212) indicating the playback position and a second axis (distance axis 122) indicating the focus position. This allows the user to intuitively and explicitly grasp the focus position of each of the series of refocused images played back.

[0109] The replay device 400 also includes a detection unit 432 that detects the movement of a specific focus position on the line graph, and a generation unit 434 that, when the detection unit 432 detects the movement of the specific focus position in the direction of the second axis (distance axis 122), recombines a plurality of specific images (parallax images PI11 to PI55) having different parallaxes that are the synthesis sources of a refocus image of a specific playback position corresponding to the specific focus position, so as to focus on the focus position after the movement. The replay unit 435 then reproduces the refocus image recombined by the generation unit 434 on the display screen 100, and the display control unit 436 controls the display screen 100 to interpolate the line graph WF based on the focus position after the movement and display the line graph WF after the interpolation on the display screen 100. This allows the focus position to be changed even for the refocus image of the playback position not operated by the user. Therefore, the continuous change of the focus position can be maintained.

[0110] Furthermore, the playback unit 435 may play back a first refocused image constituting a moving image on the display screen 100, and play back a second refocused image that precedes the first refocused image in time on the display screen 100. Specifically, the playback unit 435 plays back the first refocused image and the second refocused image in different areas of the display screen 100. This allows the user to know whether the subject will move forward or backward next, and prevents the focus from shifting when changing the line graph WF.

[0111] In addition, the playback unit 435 may play back the first refocused image and the second refocused image in the same area of ​​the display screen 100. This allows the user to intuitively grasp the difference in the movements of the subjects with a time difference, and to predict which subject will be focused on.

[0112] Furthermore, the generating unit 434 may generate a deep focus image of the second refocus image by setting the focus position of the second refocus image to infinity, and the reproducing unit 435 may reproduce the deep focus image on the display screen 100. This allows the subject of the preceding refocus moving image to be displayed without blurring, allowing the user to check the subject to be focused by referring to the preceding refocus moving image.

[0113] Furthermore, the display control unit 436 may control the display screen 100 to display the line width w in the direction of the second axis (distance axis 122) of the line graph WF as the depth of field of the refocused image, and the playback unit 435 may display the refocused image on the display screen 100 based on the depth of field. This allows the user to intuitively grasp the extent of the depth of field of each of the series of refocused images.

[0114] The playback device 400 also includes a detection unit 432 that detects a change in the line width w in the direction of the second axis (distance axis 122) of the line graph WF, and a generation unit 434 that recomposes a plurality of parallax images PI11 to PI55, which are the synthesis sources of the specific refocus image, by changing the depth of field of a specific refocus image at a playback position where the change in the line width w in the direction of the second axis (distance axis 122) is detected by the detection unit 432 based on the changed width w. The playback unit 435 then plays the specific refocus image after recomposition by the generation unit 434 on the display screen 100, and the display control unit 436 controls the display screen 100 to interpolate the line graph WF based on the changed line width w and display the interpolated line graph WF on the display screen 100. This allows the depth of field of the specific refocus image to be changed with a simple operation.

[0115] The replay device 400 also includes an image capture unit 431 that tracks and captures an object, and a generation unit 434 that generates a series of refocus images with focus positions corresponding to the distance to the object by synthesizing a plurality of parallax images PI11 to PI55 for each of a series of objects captured by the image capture unit 431. The replay unit 435 plays a moving image (refocus moving image) composed of the series of refocus images generated by the generation unit 434 on the display screen 100, and the display control unit 436 controls the display screen 100 to display information (for example, a line graph WF) indicating the relationship between the focus positions of the series of refocus images and the playback position on the display screen 100. This makes it possible to automatically generate and display the refocus moving image and the time-series line graph WF indicating the focus positions of the tracked object. This also makes it possible to confirm which object was being tracked.

[0116] Furthermore, the playback device 400 may have a directional sound collection unit 433 that identifies a sound source, and the image capture unit 431 may track a subject that is the sound source identified by the sound collection unit 433. This makes it possible to display a time-series line graph showing the focus position of the sound source. This also makes it possible to confirm which subject was being tracked as the sound source.

[0117] Furthermore, when a specific playback position is reached during playback of a video, the playback unit 435 may play back a plurality of parallax images PI11 to PI55 that are the synthesis sources of a specific refocus image at the specific playback position among a series of refocus images on the display screen 100. This makes it possible to provide a user with a video as if it were played back in slow motion in different viewing directions.

[0118] Furthermore, when a specific playback position is reached during playback of the video, the playback unit 435 may stop playback of the video for a certain period of time and play back the multiple parallax images PI11 to PI55 on the display screen 100. This allows the user to concentrate on viewing the multiple parallax images PI11 to PI55 for a certain period of time.

[0119] The playback device 400 also includes a detection unit 432 that detects drawing of the line graph WF in the coordinate system 120 on the display screen 100, and a generation unit 434 that generates a series of refocused images at each playback position of the line graph WF so as to focus on the focus position indicated by the line graph WF when the drawing of the line graph WF is detected by the detection unit 432. The playback unit 435 then plays a moving image (refocused moving image) composed of the series of refocused images generated by the generation unit 434 on the display screen 100. This makes it possible to generate a series of refocused images at consecutive focus positions intended by the user.

[0120] Also, icons for receiving a playback instruction and a thumbnail display instruction for a moving image (refocus moving image) are displayed on the display screen 100, and the playback device 400 has a detection unit 432 that detects a playback instruction and a thumbnail display instruction. When a playback instruction is detected by the detection unit 432, the playback unit 435 plays the moving image on the display screen 100, and when a thumbnail display instruction is detected by the detection unit 432, the display control unit 436 displays a thumbnail image of the subject at the focus position. This makes it possible to check what kind of subject is in focus in the refocus image to be displayed before opening the moving image file.

[0121] The display control unit 436 may also display thumbnail images of multiple subjects with different focus positions. For example, if the multiple thumbnail images to be displayed are thumbnail images of multiple refocus images with different playback positions and focus positions, it is possible to confirm what refocus images with focus on what subjects are displayed at different playback positions before opening a video file. If the multiple thumbnail images to be displayed are thumbnail images of multiple refocus images with the same playback position and different focus positions, it is possible to confirm what refocus images will be if the focus position is changed at the same playback position before opening a video file.

[0122] In the above-described embodiment, a refocused image is generated using a plurality of parallax images, but the refocused image may be generated using a color image and a monochrome image of the same object obtained from a twin-lens camera. More specifically, for example, the refocused image may be generated by the refocused image generating device 400 calculating the amount of parallax between the color image and the monochrome image, generating a composite image of the color image and the monochrome image using the amount of parallax, calculating the distance to the object using the amount of parallax, and applying a filter process to the composite image according to the distance to the specific object to be focused, thereby generating a refocused image in focus on the specific object.

[0123] The playback device 400 may also generate a refocused image by performing ray tracing and rendering. This allows a refocused image with higher resolution to be generated than when a parallax image is synthesized. In this way, a refocused image that does not use a parallax image or a synthesized image may be applied to this embodiment. [Explanation of symbols]

[0124] B bar, WF line graph, 100 display screen, 400 playback device, 412 microlens array, 413 focus adjustment unit, 421 image sensor, 431 shooting unit, 432 detection unit, 433 sound collection unit, 434 generation unit, 435 playback unit, 436 display control unit

Claims

[Claim 1] a playback unit that plays, on a display screen, a moving image that is configured by combining a plurality of images and having a variable focus position; a display control unit that controls the display screen to display information indicating a relationship between a focus position and a playback position of the composite image on the display screen; A playback device comprising:

Images