Image processing device and electronic device

Abstract

To automatically generate a two-dimensional image suitable for viewing from a spherical image.SOLUTION: An image processing device includes an input unit that inputs a panoramic image including a first subject and a second subject captured by an imaging unit, and an image generation unit that generates an image from the panoramic image in which the first subject and the second subject are arranged on the basis of the shortest path from the first subject to the second subject in the panoramic image, and the image generation unit identifies the direction from the first subject to the second subject in a partial image in the panoramic image that includes the first subject, the second subject, and a third subject that is on the shortest path as a first direction, and generates an image from the panoramic image that includes the first subject and the second subject and in which the second subject is arranged on the first direction side of the first subject.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an electronic device. [Background technology]

[0002] BACKGROUND ART There is known a camera that cuts out a part of an image captured by an ultra-wide-angle camera and displays or records it (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2012-119804 Summary of the Invention

[0004] According to a first aspect, an image processing device includes an input unit that inputs a panoramic image including a first subject and a second subject captured by an imaging unit, and an image generation unit that generates an image in which the first subject and the second subject are arranged from the panoramic image based on the shortest path from the first subject to the second subject in the panoramic image. [Brief explanation of the drawings]

[0005] [Figure 1] Block diagram showing the configuration of an image processing system [Figure 2] FIG. 1 is a block diagram illustrating a configuration of an imaging device. [Figure 3] Schematic diagram of the imaging range of the imaging unit and the spherical image [Figure 4] A block diagram showing a schematic configuration of an image processing device and a playback device. [Figure 5] Illustration of the playback process for spherical images [Figure 6] 2D image creation process [Figure 7] A diagram illustrating a two-dimensional image [Figure 8]Flowchart of 2D video creation process [Figure 9] Schematic diagram of a modified example of a spherical image [Figure 10] A block diagram showing a schematic diagram of an electronic device that functions as both an image processing device and a playback device. [Figure 11] 2D image creation process [Figure 12] 2D image creation process [Figure 13] 2D image creation process DETAILED DESCRIPTION OF THE INVENTION

[0006] (First embodiment) 1 is a block diagram schematically illustrating the configuration of an image processing system 1. The image processing system 1 includes an imaging device 2, an image processing device 3, and a playback device 4. The imaging device 2 is an electronic device such as a digital camera, a smartphone, or a tablet terminal. The image processing device 3 is an electronic device such as a digital camera, a smartphone, a tablet terminal, or a personal computer. The playback device 4 is an electronic device such as a digital camera, a smartphone, a tablet terminal, a personal computer, a digital photo frame, or a head-mounted display.

[0007] The imaging device 2 has a still image capturing function and a video capturing function. The still image capturing function is a function for capturing spherical images (described later). The video capturing function is a function for repeatedly capturing spherical images and creating a spherical video, each frame of which is a spherical image. The image processing device 3 creates, from the spherical video created by the imaging device 2, a two-dimensional video (described later), each frame of which is a two-dimensional image with a narrower angle of view than the spherical image. The playback device 4 plays (displays) the spherical images and the two-dimensional video.

[0008] (Explanation of imaging device 2) 2 is a block diagram schematically showing the configuration of the imaging device 2. The imaging device 2 has an imaging section 20, a first imaging optical system 21, a second imaging optical system 22, and a storage section 23. The imaging section 20 has a first imaging element 201 and a second imaging element 202.

[0009] The first imaging optical system 21 and the second imaging optical system 22 are so-called fisheye lenses. The first imaging optical system 21 forms an image of a subject within a range of a hemisphere on the imaging surface of the first imaging element 201. In other words, the first imaging element 201 is configured to be able to capture an image within a range of 360 degrees horizontally and 180 degrees vertically. The imaging range of the first imaging element 201 is referred to as the first hemisphere.

[0010] The second imaging optical system 22 forms an image of a subject within a range of a hemisphere different from the first hemisphere on the imaging surface of the second imaging element 202. In other words, the second imaging element 202 is configured to be able to capture an image within a range of 360 degrees horizontally and 180 degrees vertically. The imaging range of the second imaging element 202 is referred to as the second hemisphere.

[0011] The first and second half spheres form a celestial sphere. That is, the imaging unit 20 captures an image of a celestial sphere of 360 degrees horizontally and 360 degrees vertically using the first imaging element 201 and the second imaging element 202. In the following description, an image captured of the celestial sphere having a field of view of 360 degrees horizontally and 360 degrees vertically is referred to as a celestial sphere image.

[0012] When the user is using the still image capturing function, the storage unit 23 stores a single spherical image captured by the imaging unit 20 in the storage medium 51 (for example, a memory card). When the user is using the video capturing function, the storage unit 23 stores a spherical video including multiple spherical images repeatedly captured by the imaging unit 20 in the storage medium 51. As described above, each frame of the spherical video is a spherical image. Note that although the storage medium 51 is removable from the imaging device 2 in FIG. 2, the imaging device 2 may have the storage medium 51 built in.

[0013] (Explanation of spherical images) Fig. 3(a) is a schematic diagram of the imaging range of the imaging unit 20. The imaging unit 20 captures an image of the range of the celestial sphere 60 shown in Fig. 3(a) with the installation position (camera position) of the imaging unit 20 as the origin O. Fig. 3(b) is a schematic diagram illustrating an example of a celestial sphere image captured by the imaging unit 20. The celestial sphere image 61 illustrated in Fig. 3(b) includes a first semi-spherical image 62 captured by the first imaging element 201 and a second semi-spherical image 63 captured by the second imaging element 202. The first semi-spherical image 62 includes a circular image 64 formed by the first imaging optical system 21. The second semi-spherical image 63 includes a circular image 65 formed by the second imaging optical system 22.

[0014] An image with any angle of view can be obtained by cutting out and deforming a part of the celestial sphere image 61 illustrated in FIG. 3(b). For example, to obtain an image with an angle of view 66 illustrated in FIG. 3(a), an area 67 in FIG. 3(b) can be cut out and deformed into a rectangle. Furthermore, to obtain an image of a range 68, which is a hemisphere in the upper half of the celestial sphere and is shown shaded in FIG. 3(a), that goes around the line segment AB in the horizontal direction, an area 69 in FIG. 3(b) can be cut out and deformed into a rectangle. An example of an image 70 obtained in this case is illustrated in FIG. 3(c). The image 70 is a horizontally long panoramic image. Note that the left end 71 and the right end 72 of the image 70 illustrated in FIG. 3(c) are actually continuous, as shown in FIG. 3(a). That is, the image 70 illustrated in FIG. 3(c) is a 360-degree image captured by the imaging unit 20. The 360-degree image 70 includes a path 600 that goes around the surface of the celestial sphere 60. The path 600 is the circumference of a circle that is centered at the origin O and has the same diameter as the diameter of the celestial sphere 60. Since the origin O is the center of the celestial sphere 60, the circle coincides with the circumference of a cross section of the celestial sphere 60 taken along a plane that passes through the center of the celestial sphere 60.

[0015] The length of the line segment AB can be set arbitrarily. For example, by setting point A to the so-called north pole and point B to the so-called south pole, the range captured in the omnidirectional image 61 and the range captured in the omnidirectional image 70 match. In other words, the omnidirectional image can be said to be a projection (mapping) of the omnidirectional image 61 onto a two-dimensional image.

[0016] 3(c) is a panoramic image captured in a 360-degree range in the horizontal direction of the imaging unit 20. Therefore, the panoramic image 70 includes a path 600 that corresponds to the so-called equator. The panoramic image is not limited to the horizontal direction of the imaging unit 20, and may be an image captured in a 360-degree range in any direction of the imaging unit 20. For example, the image may be an image captured in a 360-degree range around the imaging unit 20 along the meridians of the celestial sphere 60.

[0017] In the following description, for simplicity of explanation, the omnidirectional image will be exemplified as a panoramic image captured over a range of 360 degrees in the horizontal direction, as illustrated in Fig. 3(c) In other words, in the following description, a omnidirectional image illustrated as if it were a panoramic image, such as image 70 in Fig. 3(c), is actually an image captured over the range of the celestial sphere 60 shown in Fig. 3(a), unless otherwise specified.

[0018] Note that the imaging unit 20 may have more imaging elements than the two, the first imaging element 201 and the second imaging element 202. In this way, a spherical image can be obtained even if the imaging range of each imaging element is narrower than a hemisphere. That is, instead of combining two imaging elements that capture a hemisphere to capture the entire celestial sphere 60, three or more imaging elements that capture a narrower range may be combined to capture the entire celestial sphere 60. The imaging ranges of the individual imaging elements may partially overlap each other. For example, the imaging range of the first imaging element 201 and the imaging range of the second imaging element 202 may partially overlap. Similarly, the imaging device 2 may have more imaging optical systems, instead of two, the first imaging optical system 21 and the second imaging optical system 22, each of which forms a subject image of a range narrower than a hemisphere.

[0019] Furthermore, the imaging unit 20 may have a single imaging element instead of the first imaging element 201 and the second imaging element 202. For example, by directing both light from the first imaging optical system 21 and light from the second imaging optical system 22 toward a single imaging element using a mirror or the like, the circular images 64 and 65 can be captured by the single imaging element. In this way, the number of imaging elements can be reduced, and the cost of the imaging unit 20 can be reduced.

[0020] (Explanation of image processing device 3) FIG. 4(a) is a block diagram schematically illustrating a configuration of the image processing device 3. The image processing device 3 includes an image generation unit 30, an input unit 31, and an output unit 32. The input unit 31 reads out a spherical video from a storage medium 51 in which the spherical video is stored, and inputs the video to the image generation unit 30. The image generation unit 30 performs a two-dimensional video creation process (described later) on the input spherical video. The two-dimensional video creation process is a process for creating a two-dimensional video from the spherical video. That is, the image generation unit 30 creates a two-dimensional video from the input spherical video. A two-dimensional video is a video in which each frame is composed of an image with a narrower angle of view than a spherical image. For example, the two-dimensional video has content equivalent to a video captured by placing a video camera with a general angle of view of approximately 50 degrees to 25 degrees at the origin O in FIG. 3(a). The output unit 32 stores the two-dimensional video created by the image generation unit 30 in the storage medium 52. The storage medium 52 may be the same storage medium as the storage medium 51 in which the imaging device 2 stores the spherical video, or may be a different storage medium. Although the storage medium 51 and the storage medium 52 are provided outside the image processing device 3 in FIG. 4( a), the image processing device 3 may incorporate one or both of the storage medium 51 and the storage medium 52. The storage medium 51 and the storage medium 52 may be connected to the image processing device via a wired or wireless network. Instead of the storage medium 51, the spherical video may be directly input from the imaging device 2 via a network.

[0021] Furthermore, each frame of a two-dimensional video created from a spherical video may include not only one image with a narrower angle of view than the spherical image, but also two or more images with a narrower angle of view than the spherical image.

[0022] (Explanation of playback device 4) FIG. 4(b) is a block diagram schematically illustrating the configuration of the playback device 4. The playback device 4 includes a display unit 40, an input unit 41, a control unit 42, and an operation unit 43. The input unit 41 reads out a spherical image from a storage medium 51 in which the spherical image is stored, and inputs the image to the control unit 42. The input unit 41 reads out a two-dimensional video from a storage medium 52 in which a two-dimensional video is stored, and inputs the two-dimensional video to the control unit 42. The control unit 42 controls the display unit 40 to display the input spherical image or two-dimensional video. The display unit 40 has a display screen including, for example, a liquid crystal panel. The display unit 40 displays the spherical image or two-dimensional video on the display screen under the control of the control unit 42. Note that, although the storage medium 51 and the storage medium 52 are provided outside the playback device 4 in FIG. 4(b), the playback device 4 may include one or both of the storage medium 51 and the storage medium 52 built therein. The display unit 40 is, for example, a liquid crystal display of a smartphone, a liquid crystal display of a tablet terminal, or a head-mounted display. Therefore, if the entire image area of ​​the omnidirectional image is displayed on the display unit 40 at once, the 360-degree range is displayed on a two-dimensional display, which is difficult for the user to view. Therefore, a method is known in which a part of the omnidirectional image having a 360-degree angle of view is cut out and displayed on a two-dimensional display screen (display unit 40) to play back the part of the omnidirectional image. The following description is based on the assumption that the method is such that a part of the omnidirectional image is displayed on the display unit 40 to play back the part.

[0023] The operation unit 43 is an operation member into which a user's operation is input. In this embodiment, the operation unit 43 is a touch sensor superimposed on the display screen of the display unit 40. The operation unit 43 detects the position where the user's finger or the like touches the display screen and transmits the position to the control unit 42. That is, the operation unit 43 detects the user's touch operation and inputs it to the control unit 42. An example of a touch operation is a scrolling operation in which the user touches a certain position on the display screen with a finger or the like, slides the finger or the like in any direction up, down, left, or right while maintaining the contact state, and then removes the finger or the like from contact with the display screen. In this embodiment, a scrolling operation in which the finger or the like is moved leftward is a scrolling operation to the left. Here, the scroll operation refers to an operation of moving the image displayed on the display unit 40 in any direction on the display unit 40.

[0024] The operation unit 43 may be an operation member other than a touch sensor. For example, if the playback device 4 is a head-mounted display, the operation of the user shaking their head to the left can be treated as a scrolling operation to the left. In this case, the operation unit 43 is a sensor that detects the displacement (orientation, position, etc.) of the head-mounted display in accordance with the movement of the user's neck. The image displayed on the display unit 40 moves by an amount corresponding to the displacement of the head-mounted display. For example, by shaking their head to the left, the image displayed on the display unit 40 moves to the right. The operating members used in the operating unit 43 are not limited to those described above, as long as they can move the image displayed on the display unit 40 in any direction on the display unit 40 .

[0025] The following describes the playback process (display process) of a spherical image by the playback device 4. FIG. 5 is an explanatory diagram of the playback process of a spherical image. FIGS. 5(a), 5(c), and 5(e) are diagrams illustrating an example of a spherical image 73 to be played back. The spherical image 73 is an image obtained by capturing a subject 74 and a subject 75. FIGS. 5(b), 5(d), and 5(f) are diagrams illustrating an example of the display screen of the display unit 40 playing back the spherical image 73.

[0026] The control unit 42 cuts out a partial area 76 from the omnidirectional image 73 shown in FIG. 5( a) and displays it on the display unit 40 as shown in FIG. 5( b). In FIG. 5( b), the display unit 40 displays an area 76 including a subject 74. When the user performs a scroll operation to the left of the screen, the control unit 42 moves a part of the omnidirectional image 73 displayed on the display unit 40 to the left, as shown in FIG. 5( c) and FIG. 5( d), thereby performing control to display a part of the omnidirectional image 73 that was not displayed on the display unit 40 at the time of FIG. 5( b). That is, the control unit 42 replaces the image displayed on the display unit 40 from the part of the omnidirectional image 73 that was displayed on the display unit 40 with another part of the omnidirectional image 73 that is located further to the right. In other words, the control unit 42 temporarily erases a part of the celestial sphere image 73 currently displayed on the display unit 40, changes the range 76 shown in FIG. 5( a) to the range 77 shown in FIG. 5( c), and displays a new part of the celestial sphere image 73 corresponding to the range 77 on the display unit 40. At this time, to the user, the celestial sphere image 73 appears to move leftward by the distance 78. In other words, to the user, the image displayed on the display unit 40 appears to move leftward by the distance 78. Note that the distance 78 can be measured in units of pixels constituting the display unit 40. For example, by minimizing the scroll operation to the left on the screen, the celestial sphere image 73 moves leftward by one pixel on the display unit 40. By measuring how many pixels the image has moved until the range 77 is displayed on the display unit 40, the distance 78 can be defined in units of pixels. In the following explanation, the change from the display state shown in Figures 5(a) and 5(b) to the display state shown in Figures 5(c) and 5(d) will be expressed as "the image displayed on display unit 40 has moved a distance 78 to the left."

[0027] When the user repeatedly performs a scroll operation to the left on the screen, control unit 42 repeats the above-described control. As a result, as shown in FIG. 5( e) and FIG. 5( f), control unit 42 cuts out range 79 including subject 75 and displays it on display unit 40. As described above, since the right end and left end of spherical image 73 are continuous, when the user further repeats a scroll operation to the left on the screen, control unit 42 again displays subject 74 on display unit 40. That is, the display content on display unit 40 again becomes the content shown in FIG. 5( a) and FIG. 5( b).

[0028] As described above, spherical image 73 is a part of an image obtained by capturing images of subject 74 and subject 75, and is image data used to display subject 74 again on display unit 40 by repeatedly controlling the part of spherical image 73 displayed on display unit 40 to move leftward and display the part of spherical image 73 that is not displayed on display unit 40. This causes subject 74 to be displayed, followed by subject 75, and then subject 74 to be displayed again on display unit 40.

[0029] In this way, the distance 80 (FIG. 5(e)) that the image displayed on display unit 40 moves from when subject 74 is displayed on display unit 40 until subject 75 is displayed on display unit 40 after repeated scrolling operations to the left of the screen is referred to as the distance in the left direction from subject 74 to subject 75. Similarly, the distance 81 (FIG. 5(e)) that the image displayed on display unit 40 moves from when subject 75 is displayed on display unit 40 until subject 74 is displayed on display unit 40 after repeated scrolling operations to the left of the screen is referred to as the distance in the left direction from subject 75 to subject 74.

[0030] 5(a) and 5(b), if the user repeatedly performs a scroll operation to the right of the screen instead of the left, subject 74 is displayed, and then subject 75 is displayed, and then subject 74 is displayed again on display unit 40. However, in the examples of FIGS. 5(a) and 5(b), if subject 75 is located above subject 74 instead of to the right of subject 74, by performing a scroll operation in a direction such as the top of the screen instead of the left or right of the screen, subject 74 is displayed, and then subject 74 disappears from the screen, and then subject 75 is displayed, and then subject 74 is displayed again. In other words, any two subjects captured in spherical image 73 can be displayed as described above by keeping the direction of the scroll operation constant.

[0031] As described above, the playback device 4 of the present embodiment cuts out a part of a spherical image having a 360-degree angle of view in the vertical and horizontal directions, and plays it back on a two-dimensional flat display screen. In the above description, the spherical image has been described as a still image, but it is also possible to play back a spherical video, each frame of which is a spherical image, by using similar processing. In this case, if the spherical video is paused and played back, the process is exactly the same as described above. When the spherical video is played back, the only difference is that the frames (spherical images) constituting the spherical video change over time.

[0032] When playing back a spherical image, a user can perform a scroll operation in any direction to display and visually recognize a main subject on the display unit 40. However, when playing back a spherical video, each frame (spherical image) constituting the spherical video is displayed on the display unit 40 for only a very short time, making it difficult to display a portion of the frame that is not displayed on the display unit 40. Typically, a scroll operation is displayed on the display unit 40 as a control result for the next frame. As a result, the user may miss a scene in which a main subject is making a noteworthy movement in a portion not currently being played on the display screen. Furthermore, the user may not even notice that a main subject that the user is not visually recognizing exists in the spherical video. Furthermore, each time the user watches a video, the user must perform the above-described scroll operation to adjust the display position, which is cumbersome. Furthermore, for example, if two main subjects are making noteworthy movements in different locations, the user must play the video multiple times to view both of them. As such, playing back a spherical video places a heavy burden on the user. Therefore, the image processing system 1 of this embodiment solves the above-mentioned problems by automatically creating a two-dimensional video that focuses on an appropriate subject from a spherical video and playing back the two-dimensional video.

[0033] The playback process (display process) of 2D video by the playback device 4 will now be described. As will be described later, 2D video is made up of multiple 2D images arranged in chronological order. Each of the 2D images that make up a 2D video is called a frame. The control unit 42 plays back the 2D video by displaying the multiple frames in order on the display unit 40.

[0034] Note that the spherical video may be input from the imaging device 2 to the image processing device 3 by a method that does not use the storage medium 51. For example, the imaging device 2 and the image processing device 3 may be electrically connected by a communication cable, and the spherical video may be input to the image processing device 3 by data communication. Alternatively, the imaging device 2 and the image processing device 3 may exchange the spherical video by wireless communication via radio waves. The same applies to the input of the spherical image from the imaging device 2 to the playback device 4 and the input of the 2D video from the image processing device 3 to the playback device 4.

[0035] (Explanation of 2D video creation process) The following describes the two-dimensional video creation process executed by the image generation unit 30. The image generation unit 30 creates a two-dimensional video from the spherical video by executing the two-dimensional video creation process. The two-dimensional video creation process is a process of identifying a main subject from the spherical image and creating a two-dimensional video including the identified main subject.

[0036] The two-dimensional video creation process includes a subject identification process and a two-dimensional image creation process. The subject identification process is a process of identifying a main subject from a spherical image included in the spherical video. The two-dimensional image creation process is a process of creating a two-dimensional image including the main subject identified by the subject identification process from the spherical image. The subject identification process and the two-dimensional image creation process will be described in order below.

[0037] (Explanation of subject identification processing) The image generation unit 30 identifies a main subject from each frame included in one spherical video using well-known techniques such as face recognition and pattern matching. For example, if the main subject is a person, a face included in the spherical image can be detected using human face recognition technology, and the entire body of the person corresponding to the face can be identified from the orientation, position, color, etc. of the face. Note that "identifying a main subject" means recognizing (detecting) the positions and shapes of various subjects captured in the spherical image and selecting the main subject from those subjects. For example, if the main subject is a person and three or more people are detected in the spherical image, the image generation unit 30 identifies all of those people as main subjects.

[0038] The recognition of the main subject can be determined based on various factors (parameters), such as the size and saliency of the subject in the image. Furthermore, by using multiple temporally consecutive images instead of a single image, the main subject can be determined based on the movement of the subject. By quantifying the parameters and using threshold processing, subjects that are equal to or greater than a predetermined threshold can be identified as the main subject. Using threshold processing may result in multiple subjects being recognized as the main subject. There may be one or more main subjects. Since a spherical image captures a 360-degree range, there is a higher likelihood that multiple subjects will be recognized as the main subject compared to images captured with a normal camera.

[0039] (Explanation of 2D image creation process) The two-dimensional image creation process is a process of creating a two-dimensional image including a main subject from each frame of a spherical video. Image processing system 1 of the present embodiment automatically creates a two-dimensional video including a main subject from a spherical video. Each image constituting the two-dimensional video is called a frame. The two-dimensional image creation process is a process of creating a two-dimensional image (frame) including a main subject identified in the subject identification process from a spherical image. In the two-dimensional image creation process, if there is one main subject, a frame including one main subject is created, and if there are two main subjects, a frame including two main subjects is created.

[0040] 7(a) to 7(c) are diagrams illustrating two-dimensional images (frames) generated by a two-dimensional image creation process when two main subjects are recognized. A first subject 201 and a second subject 202 are recognized as the main subjects. The two-dimensional image 610 illustrated in FIG. 7(a) is a two-dimensional image (frame) obtained by cutting out a partial image (angle of view) including the first subject 201 and the second subject 202 from a spherical image. Alternatively, as shown in FIG. 7(b), the first subject 201 may be cut out from the spherical image, the second subject 202 may be cut out from the spherical image, and the two cut-out partial images may be pasted together vertically and horizontally to create a two-dimensional image 611. Alternatively, as shown in FIG. 7(c), a two-dimensional image 612 may be created by superimposing a partial image 613 obtained by cutting out the first subject 201 from the spherical image onto an image of a wide range including the second subject 202 cut out from the spherical image. Hereinafter, the problems in the two-dimensional image creation process will be described using an example in which a volleyball match is captured by the image capture device 2.

[0041] FIG. 6 is an explanatory diagram of the two-dimensional image creation process. FIG. 6(a) is a top view of a volleyball court. In the example of FIG. 6, the image capture device 2 is installed in the center of the court 200. On the left side of the court 200, there is a person who is the main subject (hereinafter referred to as the first subject 201). On the right side of the court 200, there is another person who is the main subject (hereinafter referred to as the second subject 202). In other words, this is a case where two main subjects are recognized. FIG. 6(b) shows the arrangement of the first subject 201 and the second subject 202 in a three-dimensional space centered on the image capture device 2.

[0042] Here, consider a case where image generation unit 30 cuts out a two-dimensional image (frame) including first subject 201 and second subject 202 from a celestial sphere image as shown in FIG. 7(a). If image generation unit 30 generates a two-dimensional image (frame) so as to include path 204 in FIG. 6(b), the generated two-dimensional image 610 will have first subject 201 arranged on the right side and second subject 202 arranged on the left side. On the other hand, if image generation unit 30 generates a two-dimensional image (frame) so as to include path 209 in FIG. 6(b), the generated two-dimensional image 610 will have first subject 201 arranged on the left side and second subject 202 arranged on the right side. The above-mentioned problem of arrangement similarly occurs with two-dimensional image 611 and two-dimensional image 612. That is, due to the nature of a spherical image, image generating unit 30 can arrange first subject 201 and second subject 202 in at least two ways (arranging first subject 201 on the left and second subject 202 on the right, or arranging first subject 201 on the right and second subject 202 on the left). Note that, for ease of understanding, Fig. 7(a) illustrates path 204 shown in Fig. 6(b).

[0043] Incidentally, when a two-dimensional image (or two-dimensional video) generated by the image generation unit 30 is played back, the user's visibility varies greatly depending on the arrangement. For example, in the volleyball example, if the first subject 201 is the receiver and the second subject 202 is the attacker, the arrangement of the first subject 201 and the second subject 202 in relation to the volleyball captured in the two-dimensional image 610 will result in an unnatural image (video). Therefore, the image generation unit 30 needs to generate a two-dimensional image (frame) in which multiple main subjects (two main subjects) are appropriately arranged.

[0044] In the two-dimensional image creation process, image generation unit 30 of this embodiment creates a two-dimensional image including both main subjects, first subject 201 and second subject 202. Using the positions of first subject 201 and second subject 202 identified by the subject identification process, image generation unit 30 determines the angle of view including first subject 201 and second subject 202 as the angle of view of the two-dimensional image. For example, when first subject 201 and second subject 202 are present at the positions shown in FIG. 6( a), image generation unit 30 determines angle of view 203 including first subject 201 and second subject 202 as the angle of view of the two-dimensional image. Image generation unit 30 creates the two-dimensional image by cutting out content corresponding to angle of view 203 from the spherical image and transforming it into a rectangle.

[0045] There are multiple angles of view that include the first subject 201 and the second subject 202. For example, there is an angle of view in which the first subject 201 is located on the left side of the screen and the second subject 202 is located on the right side of the screen, and there is also an angle of view in which the first subject 201 is located on the right side of the screen and the second subject 202 is located on the left side of the screen. From among these multiple angles of view, the image generation unit 30 selects an angle of view that "includes the shortest path 204 connecting the first subject 201 and the second subject 202 in three-dimensional space and includes the first subject 201 and the second subject 202." For example, in FIG. 6( a), there are multiple angles of view that include the first subject 201 and the second subject 202, such as angle of view 203 and angle of view 205. From these angles of view, the image generation unit 30 selects angle of view 203, which includes the shortest path 204 connecting the first subject 201 and the second subject 202 and includes the first subject 201 and the second subject 202. It should be noted that "including the shortest route 204" can also be considered as "including a third subject that is different from the first subject 201 and the second subject 202 that are present on the shortest route 204."

[0046] A method for identifying "the shortest path connecting the first subject 201 and the second subject 202 in the celestial sphere image" will be described. When the celestial sphere 60 is cut by a plane that passes through the center of the celestial sphere 60 and the first subject 201 and the second subject 202, a part of the circumference of the cross section of the celestial sphere 60 is the shortest path connecting the first subject 201 and the second subject 202. In the celestial sphere image 206 shown in FIG. 6(b), if the first subject 201 and the second subject 202 are regarded as points, the circumference of the cross section of the celestial sphere 60 can be said to be a combination of the paths 204 and 209. The shorter of the paths 204 and 209 is the shortest path. In other words, the shortest path connecting the first subject 201 and the second subject 202 in the celestial sphere image is the path 204. Except when the first subject 201 and the second subject 202 are located on opposite sides of the celestial sphere 60, the shortest path can be uniquely identified.

[0047] Image generation unit 30 calculates the shortest path between first subject 201 and second subject 202 as follows. For example, in omnidirectional image (panoramic image) 206 shown in FIG. 6( c), image generation unit 30 places second subject 202 to the right of first subject 201. Alternatively, image generation unit 30 prepares omnidirectional image 206 in which second subject 202 is placed to the right of first subject 201. As a result, a straight line connecting first subject 201 and second subject 202 in omnidirectional image 206 passes through the center of celestial sphere 60 and coincides with the circumference of a cross section of celestial sphere 60 when celestial sphere 60 is cut by a plane that passes through first subject 201 and second subject 202. The image generation unit 30 compares a path 209 from the second subject 202 to the first subject 201 in the left direction with a shortest path 204 from the first subject 201 to the second subject 202 in the left direction. In making the comparison, the image generation unit 30 calculates a distance 208 to the second subject 202 in the left direction of the first subject 201 (hereinafter referred to as a first distance 208). Similarly, the image generation unit 30 calculates a distance 207 to the first subject 201 in the left direction of the second subject 202 (hereinafter referred to as a second distance 207). The distance can be calculated by counting the pixels that make up the spherical image (panoramic image) 206. The image generation unit 30 compares the first distance 208 with the second distance 207. In the example of FIG. 6, the second distance 207 is longer than the first distance 208.

[0048] Next, a method for generating a two-dimensional image (frame) by the image generating unit 30 will be described. As described above, the image generating unit 30 compares the first distance 208 with the second distance 207 and determines that the second distance 207 is longer than the first distance 208. Therefore, the image generating unit 30 generates a two-dimensional image (frame) such that the first object 201 is positioned on the right and the second object 202 is positioned on the left. On the other hand, if the first distance 208 is longer than the second distance 207, the image generating unit 30 generates a two-dimensional image (frame) such that the first object 201 is positioned on the left and the second object 202 is positioned on the right.

[0049] Furthermore, image generation unit 30 may generate a two-dimensional image (frame) as follows. Image generation unit 30 compares first distance 208 and second distance 207, and when it determines that second distance 207 is longer than first distance 208, determines the angle of view so that shortest path 204 is included and that first subject 201 and second subject 202 are included. When a partial image is cut out from omnidirectional image (panoramic image) 206 at the determined angle of view, first subject 201 is positioned to the right of second subject 202 in the partial image. Therefore, image generation unit 30 generates a two-dimensional image (frame) so that first subject 201 is positioned on the right and second subject 202 is positioned on the left.

[0050] The above-described first distance 207 corresponds to a distance that the image displayed on the display unit 40 moves from when the first subject 201 is displayed on the display unit 40 until the second subject 202 is displayed on the display unit 40, when the user repeatedly performs a scroll operation in the left direction in the playback process of the omnidirectional image by the playback device 4. The above-described second distance 208 corresponds to a distance that the image displayed on the display unit 40 moves from when the second subject 202 is displayed on the display unit 40 until the first subject 201 is displayed on the display unit 40, when the user repeatedly performs a scroll operation in the left direction in the playback process of the omnidirectional image by the playback device 4.

[0051] As described above, image generation unit 30 generates, from the spherical image, a two-dimensional image in which first subject 201 and second subject 202 are arranged, based on first distance 207 by which the image displayed on display unit 40 moves from when first subject 201 is displayed on display unit 40 until second subject 202 is displayed on display unit 40, and second distance 208 by which the image displayed on display unit 40 moves from when second subject 202 is displayed on display unit 40 until first subject 201 is displayed on display unit 40. Specifically, when first distance 207 is longer than second distance 208, image generation unit 30 generates, from the spherical image, a two-dimensional image that includes first subject 201 and second subject 202 and in which second subject 202 is arranged to the left of first subject 201.

[0052] Using FIG. 7(d), the meaning of the leftward direction (first direction) of first subject 201 will be described in detail. Note that in FIG. 7(d), for ease of explanation, position 201a of first subject 201 and position 202a of second subject 202 are represented by points. When position 201a of first subject 201 and position 202a of second subject 202 are known, it is possible to determine vector 615 with first subject 201 as the start point and second subject 202 as the end point. Vector 615 is decomposed into component 616 in the leftward direction (horizontal direction) and component 617 in the direction perpendicular to the leftward direction (vertical direction). When leftward component 616 is positive in the leftward direction, second subject 202 is positioned on the leftward side of first subject 201. In other words, "locating second subject 202 to the left (first direction) of first subject 201" means that vector 615, which has position 201a of first subject 201 as its start point and position 202a of second subject 202 as its end point, has a positive component in the left direction (first direction). Note that it does not matter what state component 617 of this vector 615 is in the direction (vertical direction) perpendicular to the left direction (first direction).

[0053] Note that, when image generating unit 30 generates a two-dimensional image (frame), the concept of using first distance 207 and second distance 208 has been described in FIG. 6 , but it is also possible to use angles in three-dimensional space instead of distances. For example, in FIG. 6( b ), consider the angle formed by a vector pointing from origin O to first subject 201 and a vector pointing from origin O to second subject 202. The angle formed by these two vectors can be either an acute angle or an obtuse angle. Of these, an acute angle corresponds to shortest path 204 and angle of view 203, and an obtuse angle corresponds to path 209 and angle of view 205. Therefore, image generating unit 30 determines the angle of view so that the angle formed by these two vectors is the smallest and so that first subject 201 and second subject 202 are included. When a partial image is cut out from omnidirectional image (panoramic image) 206 at the determined angle of view, first subject 201 is positioned to the right of second subject 202 in the partial image. Therefore, the image generating section 30 generates a two-dimensional image (frame) so that the first subject 201 is positioned on the right and the second subject 202 is positioned on the left.

[0054] The image generation unit 30 generates (creates) two-dimensional images (frames) through the processing described above. The image generation unit 30 generates (creates) two-dimensional video including these two-dimensional images and stores the video in the storage medium 52.

[0055] 8 is a flowchart of the two-dimensional video creation process. In step S10, the image generation unit 30 performs a subject identification process on each frame included in the spherical video. As a result, a main subject is identified for each frame.

[0056] In step S30, the image generation unit 30 selects one frame included in the spherical video. The image generation unit 30 acquires the number of main subjects identified in the selected frame. If there is one main subject (step S30: YES), the process proceeds to step S35. In step S35, the image generation unit 30 creates a two-dimensional image (frame) including the main subject based on the spherical image (frame).

[0057] If there are two or more main subjects (Step S30: No), the process proceeds to Step S40. In Step S40, the image generation unit 30 calculates a first distance in the selected frame. That is, the image generation unit 30 sets one of the two main subjects as a first subject and the other as a second subject, and calculates the distance from the first subject to the second subject in the first direction. In Step S50, the image generation unit 30 calculates a second distance in the selected frame. That is, the image generation unit 30 calculates the distance from the second subject to the first direction. Note that the first direction is a direction in which, when some of the frames included in a spherical video are displayed on the display unit 40 and the user repeatedly performs a scroll operation in a certain direction, the first subject 201 is displayed on the display unit 40, then disappears from the display unit 40, and then the second subject 202 is displayed, and then the first subject 201 is displayed on the display unit 40 again.

[0058] In step S60, the image generation unit 30 determines whether the first distance calculated in step S40 is longer than the second distance calculated in step S50. If the first distance is longer than the second distance, the image generation unit 30 proceeds to step S70. In step S70, the image generation unit 30 generates a two-dimensional image in which the second subject is placed on the first direction side of the first subject, based on the spherical image (frame) selected in step S30. On the other hand, if the first distance is equal to or shorter than the second distance in step S60, the image generation unit 30 proceeds to step S80. In step S80, the image generation unit 30 generates a two-dimensional image in which the first subject is placed on the first direction side of the second subject, based on the frame selected in step S30.

[0059] In step S90, the image generation unit 30 determines whether any unselected frames remain in the spherical video. If any unselected frames remain, the image generation unit 30 proceeds to step S30. On the other hand, if all frames have already been selected, the image generation unit 30 proceeds to step S100. In step S100, the image generation unit 30 controls the output unit 32 to store in the storage medium 52 a two-dimensional video made up of the two-dimensional images created in steps S70 and S80.

[0060] According to the above-described embodiment, the following advantages can be obtained: (1) With the configuration of this embodiment, a two-dimensional image suitable for viewing can be automatically generated from a spherical image.

[0061] (Modification of the first embodiment) Note that a single device may have two or more of the imaging unit 20, the image generation unit 30, and the display unit 40. For example, the imaging device 2 may have the image generation unit 30 in addition to the imaging unit 20. In this case, the imaging device 2 also functions as the image processing device 3. Therefore, the image processing device 3 does not need to be included in the image processing system 1. As another example, the image processing device 3 may have the display unit 40 in addition to the image generation unit 30. In this case, the image processing device 3 also functions as the playback device 4. Therefore, the playback device 4 does not need to be included in the image processing system 1. As another example, the imaging device 2 may have the image generation unit 30 and the display unit 40 in addition to the imaging unit 20. In this case, the imaging device 2 also functions as the image processing device 3 and the playback device 4. In other words, the imaging device 2 alone provides functions equivalent to those of the image processing system 1.

[0062] FIG. 10 is a block diagram schematically illustrating an electronic device 1000 that combines an image processing device 3 and a playback device 4. The electronic device 1000 is, for example, a smartphone or a tablet terminal. The electronic device 1000 has an image generation unit 30, an input unit 31, an output unit 32, a display unit 40, a control unit 42, and an operation unit 43. The electronic device 1000 can create two-dimensional moving images, play the created two-dimensional moving images on the display unit 40, store the created two-dimensional moving images in a storage medium 52, and play back spherical images (spherical moving images) on the display unit 40. Note that the operations of the components of the electronic device 1000 are the same as those in the first embodiment, and therefore will not be described again.

[0063] According to the above-described modified example, the following effects can be obtained: (2) The configuration of this embodiment can provide the same effects as those of the above-described embodiment.

[0064] Note that the creation of the 2D video by the image generation unit 30 may be performed in real time in parallel with the creation of the spherical video by the imaging unit 20, or may be started after the creation of the spherical video is completed. Similarly, the display of the 2D video by the display unit 40 may be performed in real time in parallel with the creation of the 2D video by the image generation unit 30, or may be started after the creation of the 2D video is completed.

[0065] In the above-described embodiment, the imaging unit 20 has been described as capturing an image of the celestial sphere. That is, the imaging unit 20 has been described as being capable of capturing an image of a 360-degree range around the imaging unit 20. However, the imaging unit 20 may only be capable of capturing an image of a range narrower than the celestial sphere in the vertical and / or horizontal directions. For example, the imaging unit 20 may be configured to be capable of capturing an image of a hemisphere. Alternatively, the imaging unit 20 may only be capable of capturing an image of an even narrower range than the hemisphere. For example, the imaging unit 20 may only be capable of capturing images of a range 68 shown by the hatching in FIG. 3( a). When the angle of view of the imaging unit 20 is narrower than the celestial sphere, the two-dimensional video will be composed of images with an even narrower angle of view.

[0066] Furthermore, a panoramic image does not necessarily have to be an image capturing the entire 360-degree range. For example, an image capturing a range of about 300 degrees can be treated as a panoramic image with the left and right edges connected. The same applies to a spherical image; an image in which a portion of the celestial sphere is missing can also be treated as a continuous panoramic image.

[0067] In this specification, a spherical image refers to an image in which a part of an image displayed on display unit 40 is moved in a first direction and a control for displaying a part of the image not displayed on display unit 40 is repeated, thereby displaying first subject 201 included in the image, then displaying second subject 202, and then displaying first subject 201 again on display unit 40. An image in which a part of the celestial sphere is missing is also a spherical image because the missing part is made continuous, thereby displaying first subject 201 included in the image, then displaying second subject 202, and then displaying first subject 201 again on display unit 40.

[0068] FIG. 9(a) is a schematic diagram showing an example of a spherical image. Images 620 and 621 corresponding to a hemisphere are images captured over a range smaller than 360 degrees, and a portion of the hemisphere is missing. The image generation unit 30 and the control unit 42 can treat the sides EF and GF as if they were continuous. In other words, the image shown in FIG. 9(a) is a spherical image.

[0069] FIG. 9(b) is a schematic diagram illustrating an example of a panoramic image. FIG. 9(b) illustrates images 622, 623, and 624 captured over a range that is discontinuous in the horizontal direction. These three images are captured over a range smaller than 360 degrees in the horizontal direction, and a portion of the 360 ​​degrees is missing. The image generation unit 30 and the control unit 42 can treat the sides AB, CD, and EF as being continuous. Specifically, the control unit 42 repeatedly controls the image 622 to move horizontally to the left while a portion of the image 622 is displayed on the display unit 40, thereby displaying the portion of the image 622 that is not displayed on the display unit 40. As a result, the side CD is displayed, followed by the side EF. Further repetition of the control displays the side AB, and the state returns to one in which a portion of the image 622 is displayed on the display unit 40. Therefore, the images 622, 623, and 624 shown in FIG. 9(b) are panoramic images. 9(b) (image 622, image 623, image 624) are images that are created by repeatedly controlling the display of the image on display unit 40 to move a portion of the image in the first direction by designating appropriate objects included in the image as first object 201 and second object 202, thereby displaying the portion of the image not displayed on display unit 40. This results in first object 201 included in the image being displayed, followed by second object 202, and then first object 201 being displayed on display unit 40 again. Note that the omnidirectional image (or omnidirectional image) referred to here does not take into consideration the continuity of image content. For example, when an image including a side CD is displayed on display unit 40, a user may visually recognize that the image content on the left side of side CD and the image content on the right side of side CD are discontinuous. However, the continuity of the image content is not an issue, and the important thing is the continuity of the image. In other words, it is sufficient that the image on the left side of side CD and the image on the right side of side CD are continuous.

[0070] For example, in image 622, by treating sides AB and CD as continuous, image 622 can be said to be a omnidirectional image. Also, in image 623, by treating sides CD and EF as continuous, image 623 can also be said to be a omnidirectional image. By treating all images in the same way, a part of the image displayed on display unit 40 is moved in the first direction, and control is repeated to display the part of the image that is not displayed on display unit 40. After first subject 201 included in the image is displayed, second subject 202 is displayed, and first subject 201 is displayed again on display unit 40, so that the image is a celestial sphere image (panoramic image).

[0071] 9(b), even for panoramic images or celestial sphere images (images 622, 623, and 624) that are not actually continuous, the two-dimensional image generation process by image generation unit 30 is no different from that in the above-described embodiment. That is, how to generate a two-dimensional image (frame) in which first subject 201 and second subject 202 are arranged from the panoramic images (images 622, 623, and 624) can be determined in the same way as in the above-described embodiment. For example, the distance the image displayed on the display unit 40 moves from the time the first subject 201 is displayed on the display unit 40 until the second subject 202 is displayed on the display unit 40 after repeated scrolling operations to the left of the screen is compared with the distance the image displayed on the display unit 40 moves from the time the second subject 202 is displayed on the display unit 40 until the first subject 201 is displayed on the display unit 40 after repeated scrolling operations to the left of the screen is compared, and if the former distance is longer than the latter distance, a two-dimensional image can be created so that the second subject 202 is positioned to the left of the first subject 201.

[0072] According to the above-described modified example, the following advantageous effects can be obtained: (3) With the configuration of this embodiment, a two-dimensional image suitable for viewing can be automatically generated from a panoramic image.

[0073] 7(a) to 7(c) may be used to create a two-dimensional image (frame) including the first object 201 and the second object 202. For example, a technique such as seam carving may be used to create a two-dimensional image in which the space between the first object 201 and the second object 202 is compressed. Alternatively, a two-dimensional image may be created by thinning out or reducing the objects located between the first object 201 and the second object 202.

[0074] The image generation unit 30 may perform the subject identification process only on some of the frames, rather than on all of the frames. For example, the image generation unit 30 may identify the main subject every 30 frames, such as the first frame, the 31st frame, and the 61st frame. The image generation unit 30 does not perform the subject identification process on the 29 frames between the first and 31st frames.

[0075] For example, if the frame rate of a spherical video is 60 fps, 30 frames correspond to 0.5 seconds. For a period of about 0.5 seconds, the position of the main subject is expected to remain almost unchanged. In other words, the position of the main subject in these 29 frames can be easily estimated from the position of the main subject in the first frame and the position of the main subject in the 31st frame.

[0076] In this way, by performing the subject identification process on only some of the spherical images (identifying the main subject from only some of the spherical images), it is possible to reduce the amount of calculation required to perform the two-dimensional video creation process.

[0077] (Second embodiment) The image processing system according to the second embodiment differs from that according to the first embodiment in the content of the two-dimensional video creation process executed by the image generation unit 30. Note that the points not mentioned in this embodiment are the same as those described in the first embodiment. In other words, the content described in the first embodiment is entirely incorporated into the second embodiment. Below, the image processing system according to the second embodiment will be described, focusing on the points that differ from the image processing system according to the first embodiment.

[0078] As in the first embodiment, the image generation unit 30 performs a subject identification process for each frame. For the main subject identified thereby, the image generation unit 30 then performs a direction identification process to identify the orientation of the main subject within the frame. In this embodiment, the main subject is a person, and the orientation of the main subject is the orientation of the person's face within the image. In the direction identification process, the image generation unit 30 performs a well-known face recognition process to recognize the face of the main subject and the orientation of that face. The image generation unit 30 identifies the orientation of the face of the main subject within the image as the orientation of the main subject.

[0079] Next, a method for identifying the orientation of a main subject in an image will be described. First, the orientation of the main subject in three-dimensional space is determined. For example, if the main subject is a human, the direction in which the nose is pointing is taken as the orientation of the main subject. In this case, the orientation of a vector starting from the center of the face and ending at the tip of the nose can be taken as the orientation of the main subject. Note that how to determine the orientation of the main subject in three-dimensional space will be described later. Once a vector indicating the orientation of the main subject in three-dimensional space has been identified, this vector is projected onto the image (or imaging surface). As a result, the vector projected onto the two-dimensional image in which the main subject is captured (projection vector) becomes the orientation of the main subject in the image.

[0080] 11 is an explanatory diagram of a two-dimensional image creation process. The spherical image 300 shown in FIG. 11 includes a first subject 301 and a second subject 302, which are main subjects. The distance by which the image displayed on the display unit 40 moves (the distance from the first subject 301 to the second subject 302 in the right direction) from when the first subject is displayed on the display unit 40 until when the second subject is displayed on the display unit 40 due to control to move a part of the spherical image 300 displayed on the display unit 40 in the left direction (the distance from the first subject 301 to the second subject 302 in the right direction) is longer than the distance from the second subject 302 to the first subject 301. Therefore, if a two-dimensional image is created by performing the same process as in the first embodiment, a two-dimensional image will be created in which the second subject 302 is positioned to the left of the first subject 301.

[0081] As described above, due to the nature of a spherical image, image generation unit 30 can arrange first subject 201 and second subject 202 in at least two ways (arranging first subject 201 on the left and second subject 202 on the right, or arranging first subject 201 on the right and second subject 202 on the left). On the other hand, when a two-dimensional image (or two-dimensional video) generated by image generation unit 30 is played back, the user's visibility varies greatly depending on the arrangement. Image generation unit 30 needs to generate a two-dimensional image (frame) in which multiple main subjects (two main subjects) are appropriately arranged.

[0082] Image processing unit 30 according to this embodiment creates a two-dimensional image in which first subject 301 faces toward second subject 302. In the image illustrated in FIG. 11, first subject 301 faces toward the right of the page. By creating a two-dimensional image (frame) in which first subject 301 and second subject 302 are arranged so that first subject 301 faces toward second subject 302, it is possible to create an image (video) that does not feel strange to the user. Image processing unit 30 creates a two-dimensional image in which first subject 301 is arranged to the left of second subject 302.

[0083] Next, the point in the image where first subject 301 faces toward second subject 302 will be described. As described above, when a vector indicating the orientation of the main subject in three-dimensional space is projected onto an image (or imaging plane), the projection vector is the orientation of the main subject in the image. For example, the vector shown in FIG. 11 is the projection vector of first subject 301. In image 300, first subject 301 is set as the origin, and the X-axis is taken as the direction from first subject 301 to second subject 302. In this case, if the component of the X-axis direction of the projection vector of first subject 301 is positive, it can be determined that first subject 301 faces toward second subject 302. Conversely, if the component of the X-axis direction of the projection vector of first subject 301 is negative, it can be determined that first subject 301 does not face toward second subject 302.

[0084] When a first distance moved by the image displayed on the display unit 40 from when the first subject 301 is displayed on the display unit 40 until the second subject 302 is displayed on the display unit 40 is longer than a second distance moved by the image displayed on the display unit 40 from when the second subject 302 is displayed on the display unit 40 until the first subject 301 is displayed on the display unit 40, and when the first subject 301 does not face towards the second subject 302 in the image displayed on the display unit 40 from when the second subject 302 is displayed on the display unit 40 until the first subject 301 is displayed on the display unit 40, the image generation unit 30 generates a two-dimensional image including the first subject 301 and the second subject 302, with the first subject 301 positioned on the first direction side of the second subject 302, from the omnidirectional image.

[0085] Next, a method for determining the orientation of a subject in three-dimensional space will be described. While the orientation of a person's nose has been described as an example, the orientation of the subject's (person's) face may also be used. The orientation of the face may be the direction of the eyes, or, if the face is modeled as a plane, the normal direction of the plane may be used. Furthermore, instead of the orientation of the person's face, the orientation of the person's body may also be used as the orientation of the subject. If the body orientation is used, the chest may be modeled as a plane, and the normal direction of the plane may be used as the orientation of the body. In any case, by pre-determining the orientation of the subject in three-dimensional space, it is possible to uniquely determine the orientation of the subject in three-dimensional space. Furthermore, if a subject other than a person is used as the main subject, the orientation may be appropriately determined according to the subject. For example, if the main subject is a vehicle or other moving object, the direction of travel (direction of movement) of the vehicle may be used as the orientation of the main subject. Furthermore, if the main subject is a building, the direction of the front entrance of the building may be used as the orientation of the main subject.

[0086] Next, a method for acquiring the orientation of a subject in three-dimensional space will be described. As described above, the orientation is defined according to the subject, so that the image generation unit 30 can acquire the orientation of the main subject in three-dimensional space by, for example, performing image analysis on the omnidirectional image, or from the output of a sensor provided separately from the imaging unit 20, or by performing distance measurement calculations using the imaging unit 20. If a vector indicating the orientation of the main subject in three-dimensional space can be acquired, a projection vector can be acquired by projecting the vector. Then, as described above, the image generation unit 30 can calculate the orientation of the main subject in the image based on the projection vector.

[0087] According to the above-described embodiment, the following advantages can be obtained: (1) With the configuration of this embodiment, a two-dimensional image suitable for viewing can be automatically generated from a spherical image.

[0088] (Modification of the second embodiment) Note that a single device may have two or more of the imaging unit 20, the image generation unit 30, and the display unit 40. For example, the imaging device 2 may have the image generation unit 30 in addition to the imaging unit 20. In this case, the imaging device 2 also functions as the image processing device 3. Therefore, the image processing device 3 does not need to be included in the image processing system 1. As another example, the image processing device 3 may have the display unit 40 in addition to the image generation unit 30. In this case, the image processing device 3 also functions as the playback device 4. Therefore, the playback device 4 does not need to be included in the image processing system 1. As another example, the imaging device 2 may have the image generation unit 30 and the display unit 40 in addition to the imaging unit 20. In this case, the imaging device 2 also functions as the image processing device 3 and the playback device 4. In other words, the imaging device 2 alone provides functions equivalent to those of the image processing system 1.

[0089] FIG. 10 is a block diagram schematically illustrating an electronic device 1000 that combines an image processing device 3 and a playback device 4. The electronic device 1000 is, for example, a smartphone or a tablet terminal. The electronic device 1000 has an image generation unit 30, an input unit 31, an output unit 32, a display unit 40, a control unit 42, and an operation unit 43. The electronic device 1000 can create two-dimensional moving images, play the created two-dimensional moving images on the display unit 40, store the created two-dimensional moving images in a storage medium 52, and play back spherical images on the display unit 40. Note that the operations of the various units of the electronic device 1000 are the same as those in the first embodiment, and therefore will not be described again.

[0090] According to the above-described modified example, the following effects can be obtained: (2) The configuration of this embodiment can provide the same effects as those of the above-described embodiment.

[0091] Note that the creation of the 2D video by the image generation unit 30 may be performed in real time in parallel with the creation of the spherical video by the imaging unit 20, or may be started after the creation of the spherical video is completed. Similarly, the display of the 2D video by the display unit 40 may be performed in real time in parallel with the creation of the 2D video by the image generation unit 30, or may be started after the creation of the 2D video is completed.

[0092] According to the above-described modified example, the following advantageous effects can be obtained: (3) With the configuration of this embodiment, a two-dimensional image suitable for viewing can be automatically generated from a panoramic image.

[0093] (Third embodiment) The image processing system according to the third embodiment differs from the first embodiment in the content of the two-dimensional video creation process executed by the image generation unit 30. Note that the points not mentioned in this embodiment are the same as those described in the first embodiment. In other words, the content described in the first embodiment is all incorporated in the third embodiment. The image processing system according to the third embodiment will be described below, focusing on the points that differ from the image processing system according to the first embodiment. The image generation unit 30 of this embodiment executes subject identification processing, similar to the first embodiment. Since the content is the same as that of the first embodiment, a description thereof will be omitted.

[0094] 12 and 13 are explanatory diagrams of the two-dimensional image creation process. FIGS. 12(a) and 13(a) are top views of a volleyball court. In the examples of FIGS. 12 and 13, the imaging device 2 is installed in the center of the court 400. FIG. 12(a) shows the state of the court 400 at time t1, and FIG. 12(b) shows the first frame (hereinafter referred to as the first omnidirectional image 500) captured at time t1. At this time, it is assumed that the image generation unit 30 recognizes a person subject 403 (hereinafter referred to as the third subject 403) as the main subject. Therefore, the image generation unit 30 creates a two-dimensional image (frame) including the third subject 403 in step S35 of the flowchart shown in FIG. 8.

[0095] 13(a) shows the state of court 400 at time t2, which is later than time t1, and FIG. 13(b) shows the 31st frame captured at time t2 (hereinafter referred to as second omnidirectional image 510). At this time, court 400 includes third subject 403, which was the main subject at time t1, main subject 404 (hereinafter referred to as fourth subject 404), which is a person at time t2, and main subject 405 (hereinafter referred to as fifth subject 405), which is also a person. Here, it is assumed that image generation unit 30 has identified fourth subject 404 and fifth subject 405 as the two main subjects. It is assumed that third subject 403 has not been identified as the main subject at time t2.

[0096] 13B, due to control to move a part of the second omnidirectional image 510 displayed on the display unit 40 in a leftward direction, the distance by which the image displayed on the display unit 40 moves from when the fourth subject 404 is displayed on the display unit 40 until the fifth subject 405 is displayed on the display unit 40 (the distance to the fifth subject 405 to the right of the fourth subject 404 (hereinafter referred to as the first distance)) is longer than the distance to the fourth subject 404 to the right of the fifth subject 405 (hereinafter referred to as the second distance). Therefore, if a two-dimensional image is created from the second omnidirectional image 510 by performing the same process as in the first embodiment, a two-dimensional image is created in which the fifth subject 405 is disposed to the left of the fourth subject 404.

[0097] In contrast, image generation unit 30 of the present embodiment generates a two-dimensional image by respecting the position of the main subject in a temporally previous frame, that is, first omnidirectional image 500 captured at time t1 shown in FIG. 12(b). Specifically, image generation unit 30 identifies angle of view 401, which is an angle of view that includes the main subject (third subject 403) at time t1. Then, image generation unit 30 identifies partial image 511 at a position equivalent to angle of view 401 from second omnidirectional image 510 captured at time t2. Then, image generation unit 30 creates a two-dimensional image in which fourth subject 404 and fifth subject 405 are arranged based on the positional relationship between partial image 511, fourth subject 404, and fifth subject 405 in second omnidirectional image 510.

[0098] The two-dimensional image creation process by the image generation unit 30 will be described in detail. The image generation unit 30 assumes a partial image 511 in the second omnidirectional image that is located at a position corresponding to the partial image including the third object 403 in the first omnidirectional image (i.e., a partial image corresponding to the angle of view 401 in the second omnidirectional image; hereinafter referred to as the first partial image 511). The image generation unit 30 assumes a fourth object 404, a fifth object 405, and a partial image in which the first partial image 511 is included between the fourth object 404 and the fifth object 405 (i.e., a partial image corresponding to the angle of view 406 in the second omnidirectional image; hereinafter referred to as the second partial image). The image processing unit 30 creates a two-dimensional image that maintains the left-right positional relationship between the fourth object 404 and the fifth object 405 in the second partial image.

[0099] The reason why a two-dimensional image is generated while respecting the position of the main subject in first omnidirectional image 500 captured at time t1 is as follows: Image generation unit 30 generates a two-dimensional image (frame) including third subject 403 from first omnidirectional image 500 captured at time t1. That is, the user views a direction corresponding to angle of view 401 for a predetermined period of time in terms of a played back two-dimensional video. Then, at time t2, the main subject changes from third subject 403 to fourth subject 404 and fifth subject 405. As a result, image generation unit 30 generates a two-dimensional image (frame) including fourth subject 404 and fifth subject 405 from second omnidirectional image 510 captured at time t2. This is because, for a user who has been viewing a direction corresponding to angle of view 401 as a played-back image of a two-dimensional video for a predetermined period of time, arranging fourth subject 404 and fifth subject 405 based on the position of third subject 403 (i.e., the direction corresponding to angle of view 401) makes it easier to understand the arrangement of the subjects in three-dimensional space. That is, in FIG. 13(a), based on the direction of angle of view 401 as seen from imaging device 2, fourth subject 404 is on the left and fifth subject 405 is on the right. Therefore, when a two-dimensional image (frame) including third subject 403 is switched to a two-dimensional image (frame) including fourth subject 404 and fifth subject 405 during playback of a two-dimensional video, the sense of discomfort felt by the user can be reduced by arranging fourth subject 404 on the left and fifth subject 405 on the right in the two-dimensional image.

[0100] In this way, by determining the angle of view while respecting the angle of view of the previous frame, it is possible to avoid confusion during viewing due to a sudden scene change. In other words, if a direction completely different from that of the previous frame is suddenly played back, it may become difficult to tell which part of the spherical image the screen is cropped from. As described above, the image generation unit 30 of this embodiment determines the angle of view while respecting the angle of view of the previous frame, and therefore it is possible to create a two-dimensional video that is easy to grasp the space and suitable for viewing.

[0101] Note that "the left-right positional relationship is maintained" means that the relationship is maintained when the up-down positional relationship is ignored and only the left-right positional relationship is considered. In other words, no matter how much the up-down positional relationship changes, as long as the left-right positional relationship is maintained, the left-right positional relationship is maintained.

[0102] According to the above-described embodiment, the following advantages can be obtained: (1) With the configuration of this embodiment, a two-dimensional image suitable for viewing can be automatically generated from a spherical image.

[0103] (Modification of the third embodiment) Note that a single device may have two or more of the imaging unit 20, the image generation unit 30, and the display unit 40. For example, the imaging device 2 may have the image generation unit 30 in addition to the imaging unit 20. In this case, the imaging device 2 also functions as the image processing device 3. Therefore, the image processing device 3 does not need to be included in the image processing system 1. As another example, the image processing device 3 may have the display unit 40 in addition to the image generation unit 30. In this case, the image processing device 3 also functions as the playback device 4. Therefore, the playback device 4 does not need to be included in the image processing system 1. As another example, the imaging device 2 may have the image generation unit 30 and the display unit 40 in addition to the imaging unit 20. In this case, the imaging device 2 also functions as the image processing device 3 and the playback device 4. In other words, the imaging device 2 alone provides functions equivalent to those of the image processing system 1.

[0104] FIG. 10 is a block diagram schematically illustrating an electronic device 1000 that combines an image processing device 3 and a playback device 4. The electronic device 1000 is, for example, a smartphone or a tablet terminal. The electronic device 1000 has an image generation unit 30, an input unit 31, an output unit 32, a display unit 40, a control unit 42, and an operation unit 43. The electronic device 1000 can create two-dimensional moving images, play the created two-dimensional moving images on the display unit 40, store the created two-dimensional moving images in a storage medium 52, and play back spherical images on the display unit 40. Note that the operations of the various units of the electronic device 1000 are the same as those in the first embodiment, and therefore will not be described again.

[0105] According to the above-described modified example, the following effects can be obtained: (2) The configuration of this embodiment can provide the same effects as those of the above-described embodiment.

[0106] Note that the creation of the 2D video by the image generation unit 30 may be performed in real time in parallel with the creation of the spherical video by the imaging unit 20, or may be started after the creation of the spherical video is completed. Similarly, the display of the 2D video by the display unit 40 may be performed in real time in parallel with the creation of the 2D video by the image generation unit 30, or may be started after the creation of the 2D video is completed.

[0107] According to the above-described modified example, the following advantageous effects can be obtained: (3) With the configuration of this embodiment, a two-dimensional image suitable for viewing can be automatically generated from a panoramic image.

[0108] Although the above has been described as an example in which subjects exist in the left-right direction, the same applies to cases in which two main subjects exist in directions other than left-right. Furthermore, if the main subject moves not only left-right but also up-down, the two-dimensional image may be created so that the up-down positional relationship is maintained, rather than the left-right positional relationship.

[0109] The disclosures of the following priority applications are incorporated herein by reference: Japanese Patent Application No. 2017-48861 (filed March 14, 2017) [Explanation of symbols]

[0110] 1... image processing system, 2... imaging device, 3... image processing device, 4... playback device, 20... imaging section, 30... image generation section, 31... input section, 40... display section, 42... control section

Claims

1. an input unit that inputs a panoramic image including the first subject and the second subject captured by the imaging unit; an image generating unit that generates an image in which the first subject and the second subject are arranged from the omnidirectional image based on a shortest path from the first subject to the second subject in the omnidirectional image; An image processing device comprising:

2. 2. The image processing device according to claim 1, The image generation unit identifies, as a first direction, a direction from the first subject to the second subject in a partial image in the omnidirectional image that includes the first subject, the second subject, and a third subject that is located on the shortest path, and generates, from the omnidirectional image, an image that includes the first subject and the second subject and in which the second subject is positioned on the first direction side of the first subject.

3. an input unit for inputting first image data used to display a part of an image of a first subject and a second subject on a display unit, the part of the image being a part of an image captured by the image capture unit, the part of the image being displayed on the display unit being moved in a first direction, and a part of the image not being displayed on the display unit being displayed, whereby the second subject is displayed after the first subject is displayed; and a first distance that the image displayed on the display unit moves from when the first subject is displayed on the display unit until when the second subject is displayed on the display unit; and an image generating unit that generates second image data, in which the first subject and the second subject are arranged, from the first image data based on a second distance that the image displayed on the display unit has moved from when the second subject is displayed on the display unit until when the first subject is displayed on the display unit; An image processing device comprising:

4. 4. The image processing device according to claim 3, The control of displaying the portion of the image not being displayed on the display unit is control of displaying at least a part of the portion of the image not being displayed on the display unit.

5. 5. The image processing device according to claim 3, The image processing device, wherein the first image data is spherical image data.

6. 5. The image processing device according to claim 3, The image processing device, wherein the first image data is image data captured within a range smaller than a celestial sphere.

7. 7. The image processing device according to claim 6, The image processing device, wherein the first image data is hemispherical image data.

8. 5. The image processing device according to claim 3, The image processing device, wherein the first image data is image data obtained by capturing an image over a 360-degree range in at least one of the vertical and horizontal directions.

9. 5. The image processing device according to claim 3, An image processing device, wherein the first image data is image data obtained by capturing an image of a range narrower than a 360-degree range in at least one of the vertical direction and the horizontal direction.

10. a display unit that displays an image of the first subject and the second subject; a control unit that repeats control of moving a part of the image displayed on the display unit in a first direction and displaying a part of the image that is not displayed on the display unit, thereby displaying the second subject after displaying the first subject, and then displaying the first subject on the display unit again; a first distance by which the image displayed on the display unit moves from when the first subject is displayed on the display unit until when the second subject is displayed on the display unit; an image generating unit that generates image data including the first subject and the second subject, and in which the second subject is positioned on the first direction side of the first subject, when the image displayed on the display unit is longer than a second distance moved from when the second subject is displayed on the display unit until when the first subject is displayed on the display unit; An electronic device comprising:

11. a display unit that displays first image data obtained by capturing images of the first subject and the second subject; a control unit that repeats control of moving a part of the first image data displayed on the display unit in a first direction to display a part of the first image data that is not displayed on the display unit, thereby displaying the second object after displaying the first object, and then displaying the first object on the display unit again; a first distance that the image displayed on the display unit moves from when the first subject is displayed on the display unit until when the second subject is displayed on the display unit; and an image generating unit that generates second image data, in which the first subject and the second subject are arranged, from the first image data based on a second distance that the image displayed on the display unit has moved from when the second subject is displayed on the display unit until when the first subject is displayed on the display unit; An electronic device comprising:

12. 12. The electronic device according to claim 11, When the first distance is longer than the second distance, the image generation unit generates second image data from the first image data, which includes the first subject and the second subject and positions the second subject on the first direction side of the first subject.

13. 13. The electronic device according to claim 11, The electronic device, wherein the first image data is spherical image data.

14. 14. The electronic device according to claim 11, The electronic device, wherein the first image data is image data captured within an area smaller than a celestial sphere.

15. 14. The electronic device according to claim 13, The electronic device, wherein the first image data is hemispherical image data.

16. 14. The electronic device according to claim 11, The electronic device, wherein the first image data is image data captured over a 360-degree range in at least one of the vertical and horizontal directions.

17. 14. The electronic device according to claim 11, The electronic device, wherein the first image data is image data captured over an area narrower than 360 degrees in at least one of the vertical and horizontal directions.

Images