Representation method

Abstract

To provide a technique that is related to an imaging device and a display device, allows a user to easily grasp the relation among a plurality of images, and can realize the better usability.SOLUTION: An imaging device includes an imaging unit and a display unit, acquires first information including at least one of the azimuth angle and the elevation angle as the imaging direction of an image and at least one of a field angle of the image and field angle relevant information, associates a second image with the first image when a second imaging direction of the second image or a partial field angle is included within a range of the first field angle in a first imaging direction of the first image or within a threshold field angle range extended to the outside thereof for the plurality of captured images, displays a diagram 61 expressing the first image selected on the basis of the operation of a user on a background region 50 in a display screen 401, and displays diagrams 62, 63 expressing the second image associated with the first image in a positional relation corresponding to the imaging directions.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to techniques for imaging devices and display devices, and to a technique for displaying captured images on a screen. [Background technology]

[0002] Conventionally, there are digital cameras and smartphones as electronic devices such as imaging devices having a function of capturing multiple images with different shooting directions and angles of view. In addition, there are television receivers (hereinafter sometimes referred to as "television") and notebook PCs as electronic devices such as display devices having a function of playing and displaying the multiple images. The images are still images or videos. The shooting direction can be defined by, for example, an azimuth angle and an elevation angle. The angle of view represents the image range as an angle, and is also called the viewing angle, and can be defined by, for example, a horizontal angle of view corresponding to the azimuth angle and a vertical angle of view corresponding to the elevation angle.

[0003] Images with different angles of view include an image taken at a relatively wide angle (hereinafter, may be referred to as a "wide-angle image") and an image taken at a relatively narrow angle (hereinafter, may be referred to as a "narrow-angle image"). Examples of wide angles include a horizontal angle of view of 120° or 180° or more. Note that a 180° angle is sometimes called a hemispherical image, and a 360° angle is sometimes called a omnidirectional image. Examples of narrow angles include a horizontal angle of view of 50° or less. In the following, angles are expressed in degrees (°), but can be converted to units such as radians.

[0004] An example of prior art relating to the display of captured images on a screen is JP 2000-92439 A (Patent Document 1). Patent Document 1 describes the following regarding an electronic still camera. When the power is turned off, the electronic still camera generates an image list display file that enables images recorded in a recording memory to be displayed in a list on an external device. A list of multiple reduced images is displayed on a list display screen that is displayed by an image search using the file. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2000-92439 A Summary of the Invention [Problem to be solved by the invention]

[0006] A user who is a photographer may use an imaging device to capture multiple images with different shooting directions and angles of view at the same place and date. For example, a user may capture a wide-angle image with one shooting direction and angle of view and a narrow-angle image with another shooting direction and angle of view in the same park on the same day. In such a case, the multiple images can be said to be a group of related images that have a predetermined relationship with respect to the place, date, shooting direction, angle of view, etc.

[0007] Conventional imaging devices and display devices have a function of displaying a plurality of images on a display screen in a list or parallel arrangement in the order of, for example, the shooting date and time, the file name, etc. For example, in the case of Patent Document 1, a plurality of reduced images are displayed in a parallel arrangement vertically and horizontally on a list display screen.

[0008] However, in the conventional technology, it is difficult for a user to intuitively recognize a plurality of images having a predetermined relationship among all images displayed on the display screen. For example, even when a plurality of images including wide-angle images and narrow-angle images taken at the same place or date are mixed, it is difficult for a user to grasp the interrelationship between the plurality of images on the list display screen. In the conventional technology, there is no function to associate and play and display a plurality of related images. In the conventional technology, there is no function to refer to and display other images having a relationship with an image displayed on the display screen from a state in which the image is displayed. In the conventional technology, editing and management of a plurality of images by a user is also time-consuming.

[0009] That is, the conventional technology makes it difficult to conveniently view a plurality of related images, and there is room for improvement in terms of ease of use.

[0010] An object of the present invention is to provide a technique for an imaging device and a display device that makes it easy to grasp the relationship between a plurality of images and realizes better usability. [Means for solving the problem]

[0011] A representative embodiment of the present invention is an imaging device or the like, characterized by having the following configuration.

[0012] An imaging device in one embodiment includes an imaging unit that captures still or video images to obtain image data, and a display unit that plays and displays the images on a display screen based on the image data, and obtains first information including at least one of azimuth angle and elevation angle as the shooting direction of the image, and at least one of an angle of view of the image or angle-of-view related information by which the angle of view can be calculated. When a second shooting direction or a partial angle of view of a second image is included within a threshold angle of view range extended beyond or within a first angle of view range of a first shooting direction of a first image for the captured multiple images, the imaging device associates the second image with the first image, and displays the first image or a first figure representing the first image selected based on a user's operation on a background area of ​​the display screen, and displays the second image or a second figure representing the second image associated with the first image in a positional relationship corresponding to the shooting direction. Effect of the Invention

[0013] According to a representative embodiment of the present invention, it is possible to easily grasp the relationship between a plurality of images in an imaging device and a display device, thereby achieving better usability. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing a configuration of an imaging and display system including an imaging device and a display device according to a first embodiment of the present invention. [Diagram 2] 1 is a diagram illustrating a configuration of an imaging device according to a first embodiment. [Diagram 3] FIG. 1 is a diagram showing a configuration of a display device according to a first embodiment. [Figure 4] 1 is a diagram showing the appearance of an imaging device according to a first embodiment. [Diagram 5] 3 is a diagram showing a structure of management information and metadata in the imaging device of the first embodiment. [Figure 6] FIG. 2 is a diagram showing management of relationships between a plurality of images in the imaging device of the first embodiment. [Figure 7] 2 is an explanatory diagram showing the shooting direction and angle of view, particularly the azimuth angle and horizontal angle of view, for a plurality of images in the imaging device of the first embodiment. FIG. [Figure 8] 2 is an explanatory diagram showing, in particular, the elevation angle and vertical angle of view of the imaging device according to the first embodiment. FIG. [Figure 9] 4 is an explanatory diagram showing angle-of-view related information in the imaging device according to the first embodiment. FIG. [Figure 10] 4 is a diagram showing transition of display modes of a display screen in the imaging device of the first embodiment. FIG. [Figure 11] FIG. 13 is a diagram showing an example of a screen in a second display mode in the imaging device according to the modified example of the first embodiment. [Figure 12] FIG. 13 is a diagram showing an example of a screen in a second display mode in the imaging device according to the modified example of the first embodiment. [Figure 13] FIG. 13 is a diagram showing another example of an image on the second display mode screen of the imaging device according to the first embodiment. [Figure 14] FIG. 13 is a diagram showing another example of an image on the second display mode screen of the imaging device according to the first embodiment. [Figure 15] FIG. 13 is a diagram showing another example of an image on the second display mode screen of the imaging device according to the first embodiment. [Figure 16] 10A to 10C are diagrams showing an example of scrolling or the like on a second display mode screen in the imaging device of the first embodiment. [Figure 17] 5A to 5C are diagrams showing examples of enlargement, reduction, etc. of a second display mode screen in the imaging device of the first embodiment. [Figure 18] FIG. 4 is a diagram showing an example of a screen in a second display mode in the imaging device according to the first embodiment. [Figure 19] FIG. 4 is a diagram showing an example of a screen in a second display mode in the imaging device according to the first embodiment. [Figure 20]FIG. 13 is a diagram showing an example of a fourth display mode screen in the imaging device of the first embodiment. [Figure 21] 4 is a diagram showing an example of a map display mode screen in the imaging device of the first embodiment. FIG. [Figure 22] FIG. 4 is a diagram showing a flow of control processing during imaging in the imaging device of the first embodiment. [Figure 23] FIG. 4 is a diagram showing a flow of control processing during playback and display in the imaging device and display device of the first embodiment. [Figure 24] FIG. 2 is an explanatory diagram showing a first selection method in the imaging device according to the first embodiment. [Diagram 25] FIG. 11 is an explanatory diagram showing a second selection method in the imaging device according to the first embodiment. [Figure 26] FIG. 11 is an explanatory diagram showing a third selection method in the imaging device according to the first embodiment. [Figure 27] 4 is an explanatory diagram showing association conditions etc. in the imaging device of the first embodiment. FIG. [Figure 28] FIG. 13 is a diagram showing an example of a screen in a second display mode in the imaging device according to the modified example of the first embodiment. [Figure 29] FIG. 13 is a diagram showing an example of a screen in a second display mode in the imaging device according to the modified example of the first embodiment. [Diagram 30] 11 is an explanatory diagram showing shooting modes in an imaging device according to a second embodiment of the present invention. FIG. [Diagram 31] FIG. 11 is a diagram showing an imaging menu screen in the imaging device of the second embodiment. [Diagram 32] FIG. 11 is a diagram showing a display screen of a monitor image in a shooting mode in an imaging device according to a third embodiment of the present invention. [Diagram 33] FIG. 13 is a diagram showing a display screen of a monitor image in a shooting mode in an imaging device according to a modified example of the third embodiment. [Diagram 34] FIG. 13 is a diagram showing an example of a display screen for a first editing function in the imaging device according to the fourth embodiment of the present invention. [Diagram 35] FIG. 13 is a diagram showing an example of a display screen for a second editing function in the imaging device according to the fourth embodiment. [Diagram 36]FIG. 13 is an explanatory diagram showing a spherical image in an imaging device according to a fifth embodiment of the present invention. [Figure 37] FIG. 13 is a diagram showing a configuration relating to an imaging section in an imaging device according to a fifth embodiment. [Figure 38] FIG. 13 is a diagram showing the external configuration of an imaging device according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In addition, in all the drawings for explaining the embodiment, the same parts are basically given the same reference numerals, and the repeated description will be omitted.

[0016] (Embodiment 1) An imaging device and a display device according to a first embodiment of the present invention will be described with reference to Fig. 1 to Fig. 29. The imaging device and the display device according to the first embodiment have a function of displaying a plurality of images captured in different directions or angles of view in an associated state.

[0017] [Image display system] Fig. 1 shows the configuration of an imaging display system which is a system including the imaging device and display device of embodiment 1. In the imaging display system of Fig. 1, a first imaging device 1A and a second imaging device 1B which are the imaging device 1 of embodiment 1, a first display device 2A and a second display device 2B which are the display device 2 of embodiment 1, and a server 3 are connected via a communication network 4. The communication network 4 includes a wireless communication network and the Internet.

[0018] The imaging device 1 is an electronic device having at least an imaging function and a display function. The display device 2 is an electronic device having at least a display function. A user who is a photographer has a first imaging device 1A and a second imaging device 1B, and a first display device 2A and a second display device 2B as electronic devices that the user owns or has the right to use. For example, the first imaging device 1A is a digital camera, the second imaging device 1B is a smartphone, the first display device 2A is a television, and the second display device 2B is a notebook PC.

[0019] The first imaging device 1A and the second imaging device 1B have at least a function of storing image data of an image captured by the device in a storage means in the device. The first imaging device 1A and the second imaging device 1B may also have a function of transmitting image data of an image captured by the device to the server 3 and storing and registering the image data in an image data DB of the server 3. The first imaging device 1A and the second imaging device 1B may also have a function of transmitting image data of an image captured by the device to the first display device 2A and the second display device 2B based on a user operation.

[0020] The first display device 2A and the second display device 2B have a function of receiving and inputting image data from an external device such as the imaging device 1 or the server 3, and storing the image data in a storage means within the device. For example, the first display device 2A receives image data from the first imaging device 1A or the second display device 2B through communication. For example, the second display device 2B receives image data from the first imaging device 1A or the second imaging device 1B through communication. In addition, the first display device 2A and the second display device 2B have an image data display unit. The image data display unit displays an image on a display screen based on the image data. In addition, for example, the second display device 2B has an image data editing unit. The image data editing unit edits information and contents of the image data based on a user operation on the display screen.

[0021] The server 3 is, for example, a server device managed by a business operator, and receives image data from users' devices and stores it in an image data DB. This allows backup and centralized management of image data, etc. The server 3 and the image data DB may be implemented in separate devices.

[0022] [Imaging device] 2 shows the configuration of the imaging device 1. The imaging device 1 includes a control unit 101, a storage unit 102, an imaging unit 103, a display device 104, an operation input unit 105, a recording unit 106, a recording medium 107, a communication interface unit 108, an imaging direction measurement unit 111, an internal clock 112, a position detection unit 113, a sensor group 114, a microphone 115, a speaker 116, an interface circuit unit 131, a signal processing unit 132, a memory 133, and the like. These units are connected to each other via a bus or the like, and cooperate at high speed under the control of the control unit 101.

[0023] The control unit 101 controls the entire imaging device 1. The control unit 101 includes control circuits such as a processor, a ROM, and a RAM. The control unit 101 has an imaging control unit 11, a display control unit 12, a touch detection unit 13, a setting unit 14, and an association unit 15 as processing units realized by processing of the control circuit.

[0024] The control unit 101 controls each unit such as the imaging unit 103 according to the operation mode to realize imaging and display. The operation modes include an imaging mode, a playback display mode, etc. The control unit 101 controls the imaging unit 103, the display device 104, the recording unit 106, etc. via the signal processing unit 132 and the memory 133 according to the operation mode and state.

[0025] The control unit 101 controls settings such as the shooting direction and angle of view of the image based on a user input operation via the operation input unit 105 or the touch sensor 42. When an image is captured via the imaging unit 103, the control unit 101 creates metadata 22 together with image data 23, and stores the metadata 22 and the image data 23 in the storage unit 102 or records the metadata 22 and the image data 23 in the recording medium 107 of the recording unit 106 as an image file 24. The control unit 101 also creates and manages management information 21 related to the image file 24.

[0026] The storage unit 102 stores various data and information including a plurality of image files 24 and management information 21 under the control of the control unit 101. Note that the storage unit 102 may be integrated into the control unit 101, or the storage unit 102 and the recording unit 106 may be integrated into one unit, etc.

[0027] The imaging unit 103 captures an image based on control from the imaging control unit 11. The imaging unit 103 includes a drive unit 31 and a lens unit 32. The drive unit 31 includes a drive circuit, and drives the imaging element and lens of the lens unit 32 based on drive control from the imaging control unit 11. The lens unit 32 includes an optical system with multiple lenses including a lens compatible with electronic zoom or optical zoom. For example, the drive unit 31 performs a focusing operation that controls the position of a focus lens to change the focal length. The lens of the lens unit 32 may be a zoom lens type compatible with electronic zoom that is fixed and does not need to be replaced, or may be an optical lens type that is removable and replaceable.

[0028] The imaging unit 103 includes an imaging element such as a CMOS or a CCD. Photoelectric conversion elements are arranged two-dimensionally on the imaging surface of the imaging element. The imaging unit 103 photoelectrically converts an optical image of a subject that is incident through the lens unit 32 and formed on the imaging surface of the imaging element into an imaging signal. The imaging unit 103 has an AD conversion circuit built in that converts an analog signal into a digital signal, and outputs the digitized imaging signal. The imaging unit 103 may use an imaging element in which pixels for phase difference autofocus are arranged to speed up autofocus, or may have a built-in memory to speed up input and output of imaging signals. An AD conversion circuit or the like may be provided outside the imaging unit 103. The imaging method of the imaging unit 103 is not limited.

[0029] The display device 104 is, for example, a touch panel, and includes a display unit 41 and a touch sensor 42. The touch panel is a display means and an operation input means. The display unit 41 includes a display driver, and displays a display screen to the user. The display unit 41 is, for example, a liquid crystal display unit, but may be an organic EL display unit or the like. There are no limitations on the type of the display unit 41 or the type of the touch sensor 42. A plurality of display devices 104 may be provided, or the display device 104 may not include the touch sensor 42. In the first embodiment, in addition to the touch panel that is the main display device 104, it has a display device for a shooting monitor, which will be described later.

[0030] The touch sensor 42 is disposed in correspondence with an area of ​​the display screen of the display unit 41, and accepts a touch input operation on the display screen. The touch sensor 42 is, for example, a capacitance type, detects a change in capacitance due to the proximity or contact of a finger or the like as an electric signal, and outputs a touch detection signal to the touch detection unit 13 of the control unit 101. The touch panel has the touch sensor 42 built in, for example, a liquid crystal display unit, a glass protective cover arranged on the front side, and a backlight arranged on the rear side. The display device 104 may be controlled so that the backlight is automatically turned off to be in a non-display state based on a certain period of non-use, etc. The touch panel may be a pressure-sensitive type, etc.

[0031] The touch detection unit 13 detects the presence or absence of a touch on the display screen, the touch position coordinates, etc., based on a touch detection signal obtained from the touch sensor 42, and also detects operations such as tapping, swiping, pinching, etc. A user can perform various operations such as inputting instructions to the imaging device 1 and focusing on an arbitrary position on the display screen by performing a touch input operation on the display screen of the touch panel.

[0032] The interface circuit unit 131 transfers a drive control signal from the control unit 101 to the imaging unit 103 based on the control of the control unit 101 in accordance with the operation mode. Also, the interface circuit unit 131 outputs an imaging signal from the imaging unit 103 to the signal processing unit 132 or the memory 133 based on the control. Note that a configuration in which a plurality of imaging units 103 are connected to the interface circuit unit 131 may be used.

[0033] The signal processing unit 132 includes an image and audio signal processing circuit and an encoding / decoding circuit, and has the function of performing various types of signal processing such as image signal processing, audio signal processing, encoding processing, and decoding processing under the control of the control unit 101. The signal processing by the signal processing unit 132 includes signal processing for the imaging signal from the imaging unit 103, signal processing for the image data 23, and signal processing when displaying an image on the display device 104. The signal processing unit 132 generates a video signal for recording or display by signal processing. This video signal includes still images and moving images of the image data 23.

[0034] The signal processing unit 132 performs signal processing such as filtering, amplification according to a sensitivity setting, white balance correction, etc. on the imaging signal. When recording the image data 23 on the recording medium 107 as an image file 24, the signal processing unit 132 performs encoding processing on the original data in a predetermined format for that purpose. This encoding includes data compression. When reading the image data 23 from the recording medium 107, the signal processing unit 132 performs a corresponding decoding processing to restore the original data. When reproducing and displaying an image on the display device 104 based on the image data 23, the signal processing unit 132 performs signal processing for that purpose. The signal processing unit 132 performs audio signal processing on an audio input signal from the microphone 115 and an audio output signal to the speaker 116.

[0035] The memory 133 is used as a buffer memory and stores various signals and data when the signal processing unit 132 processes signals. The memory 133 is configured by, for example, a DRAM or a flash memory.

[0036] The operation input unit 105 includes various hardware buttons that enable the user to perform input operations, etc. The user can input instructions through the operation input unit 105 and can set information related to the angle of view, etc., in the setting unit 14.

[0037] The setting unit 14 accepts settings input by the user via the operation input unit 105 or the touch panel, and sets information related to shooting and display. The setting unit 14 can set the electronic zoom magnification, focal length, angle of view, etc., related to the lens of the imaging unit 103 as setting information related to shooting. The imaging control unit 11 drives and controls the imaging unit 103 based on the setting information of the setting unit 14 so as to automatically adjust the angle of view, etc.

[0038] The recording unit 106 includes a recording and playback circuit and the like, and is provided with a removable recording medium 107. The recording medium 107 is, for example, an SD card. The recording unit 106 records an image file 24 including image data 23 in the storage unit 102, memory 133, or the like, on the recording medium 107 under the control of the control unit 101. The recording unit 106 reads out the image file 24 from the recording medium 107 and stores it in the storage unit 102, memory 133, or the like under the control of the control unit 101. The user can remove the recording medium 107 to carry or replace it. The image file 24 may include audio data along with the image data 23.

[0039] The communication interface unit 108 includes a wireless communication interface unit. The communication interface unit 108 connects to the communication network 4 via a wireless access point or the like under the control of the control unit 101, and performs communication processing with an external device. For example, as shown in FIG. 1, the imaging device 1 communicates with a server 3, a display device 2, or an imaging device 1 other than the imaging device itself via the communication network 4, and transmits an image file 24 of the imaging device itself, or receives an image file 24 from an external device. At that time, the communication interface unit 108 transfers communication data of the image file 24. The communication interface unit 108 may have a telephone network communication function, a LAN communication function, a short-distance communication function, or the like.

[0040] The shooting direction measuring unit 111 includes sensors such as an electronic compass, a gyro sensor, and an acceleration sensor, and uses them to measure the shooting direction of the imaging device 1 and output information about the shooting direction to the control unit 101. The shooting direction measuring unit 111 measures the azimuth angle and elevation angle as the shooting direction. The shooting direction measuring unit 111 also detects the rotation angle of the image at the time of imaging based on the detection of the attitude of the imaging device 1. The rotation angle of the image is represented by the angle formed by the horizontal and vertical lines of a rectangle in the image with respect to the horizontal and vertical directions.

[0041] The internal clock 112 measures the current date and time. This allows the control unit 101 to obtain the shooting date and time. The position detection unit 113 includes a GPS receiving unit, and detects the position of the imaging device 1 using signals from satellites. The position detection unit 113 detects {latitude, longitude, altitude (altitude above sea level)} as position information. The altitude may be omitted. The sensor group 114 is a group of other sensors, such as a proximity sensor and an illuminance sensor. The microphone 115 is an audio input device, and the speaker 116 is an audio output device.

[0042] The imaging control unit 11 controls the imaging unit 103 and the like during imaging in accordance with the operation mode of the imaging device 1 to capture an image and obtain image data 23. The imaging control unit 11 stores the image data 23 in the storage unit 102 or the like as an image file 24. The imaging unit 103 and the imaging control unit 11 have a function that enables both wide-angle images and narrow-angle images to be captured. The imaging control unit 11 performs zoom control regarding the optical system of the imaging unit 103. The zoom control is control of electronic zoom or optical zoom. The imaging control unit 11 grasps and adjusts the electronic zoom magnification, focal length, angle of view, and the like regarding the lens of the imaging unit 103 based on setting information such as angle-of-view related information of the setting unit 14.

[0043] When the imaging unit 103 is replaceable with an optical lens having a different focal length, for example, the user replaces the lens with an optical lens having a relatively long focal length when capturing a narrow-angle image, and replaces the lens with an optical lens having a relatively short focal length when capturing a wide-angle image. When the user replaces the lens of the imaging unit 103, the setting unit 14 may automatically detect the replacement and detect the focal length, etc. of the replaced lens. Alternatively, after the replacement, the user may input and set the focal length, etc. of the replaced lens to the setting unit 14. Then, the imaging control unit 11 obtains the updated setting information from the setting unit 14 and grasps the focal length, etc.

[0044] The display control unit 12 controls the display unit 41 of the display device 104 and the like to display images and information on the display screen of the display unit 41. The display control unit 12 displays folder information and images on the display screen based on the management information 21 and the image files 24. The display control unit 12 provides various screens that serve as a graphical user interface for the user. The display control unit 12 controls switching of the screen according to the playback display mode. The display control unit 12 also provides a menu screen, a setting screen, and the like.

[0045] The associating unit 15 performs an associating process for associating a plurality of images having a predetermined relationship in cooperation with the imaging control unit 11 and the display control unit 12. The associating unit 15 performs the associating process at a predetermined timing and opportunity such as when capturing images or when playing and displaying images according to an associating method described later. As a first process in the associating process, the associating unit 15 performs a judgment for associating a plurality of image data 23 in the storage unit 102, etc. according to a selection method described later. In the judgment of the first process, an image that is a candidate for associating is selected based on conditions such as a shooting location according to a selection method described later. As a second process, the associating unit 15 determines a first image and a second image to be associated among the plurality of candidate images based on conditions such as a shooting direction and an angle of view. As a result of the judgment, the associating unit 15 creates metadata 22 or management information 21 so as to associate the first image and the second image.

[0046] When playing and displaying images, the display control unit 12 identifies a first image and a second image having a predetermined relationship based on the determination result of the associating unit 15 or based on a reference check of the metadata 22 or management information 21 of the associated image file 24. The display control unit 12 displays a plurality of images including the first image and the second image in an associated state on the display screen.

[0047] In addition, the hardware configuration of each processing unit and each function, such as the control unit 101 and the signal processing unit 132 of the imaging device 1, may be such that each processing unit and each function is implemented in an individual LSI or other circuit, or such that each processing unit and each function is implemented in a single integrated LSI or other circuit.

[0048] [Display device] 3 shows the configuration of the display device 2. The main difference between the display device 2 and the imaging device 1 is that the display device 2 does not include the imaging unit 103 and a processing unit related to imaging. A description of the components of the display device 2 that are similar to the components of the imaging device 1 will be omitted. The display device 2 includes a control unit 201, a storage unit 202, a display unit 204, an operation input unit 205, a recording unit 206, a recording medium 207, a communication interface unit 208, a group of sensors, a microphone, a speaker, etc. These units are connected to each other via a bus or the like, and work together at high speed based on the control of the control unit 201.

[0049] The control unit 201 controls the entire display device 2. The control unit 201 has a display control unit 212 and an association unit 215 as processing units realized by processing of a control circuit. The control unit 201 controls the display on the display screen of the display unit 204 based on a user input operation via the operation input unit 205.

[0050] The storage unit 202 stores various data and information including a plurality of image files 24 and management information 21 under the control of the control unit 201. The image files 24 and management information 21 may be created by the device itself or may be obtained from an external device.

[0051] The display unit 204 includes a display driver and displays a display screen to the user. The display unit 204 may be a touch panel.

[0052] The operation input unit 205 includes various hardware buttons that enable the user to perform input operations. The operation input unit 205 may include a remote control and a remote control light receiving unit. The user can input instructions and perform user settings through the operation input unit 205.

[0053] The recording unit 206 includes a recording and playback circuit and the like, and is provided with a detachable recording medium 207. Based on the control of the control unit 201, the recording unit 206 records an image file 24 including image data 23 in the storage unit 202 onto the recording medium 207. Based on the control of the control unit 201, the recording unit 206 reads out the image file 24 from the recording medium 207 and stores it in the storage unit 202.

[0054] The communication interface unit 208 connects to the communication network 4 and performs communication processing with an external device based on the control of the control unit 201. For example, as shown in Fig. 1, the display device 2 communicates with the server 3, the imaging device 1, or a display device 2 other than the display device 2 via the communication network 4, and transmits an image file 24 of the display device 2, or receives an image file 24 from an external device. At that time, the communication interface unit 208 transfers communication data of the image file 24.

[0055] The display control unit 212 controls the display unit 204 and the like to display images and information on the display screen of the display unit 204. The display control unit 212 reproduces and displays folder information and images on the display screen of the display unit 204 based on the image files 24 and management information 21. The display control unit 212 controls switching of the screen according to the reproduction display mode.

[0056] The associating unit 215 performs an associating process for associating a plurality of images having a predetermined relationship with each other in cooperation with the display control unit 212. The associating unit 215 can also perform the associating process for an image file 24 acquired from an external device. The associating process of the associating unit 215 is similar to the associating process of the associating unit 15.

[0057] When playing and displaying images, the display control unit 212 identifies a first image and a second image having a predetermined relationship based on the determination result of the associating unit 215 or based on a reference check of the metadata 22 or management information 21 of the associated image file 24. The display control unit 212 displays a plurality of images including the first image and the second image in an associated state on the display screen.

[0058] The following description will be mainly focused on the imaging device 1, but functions and operations other than those related to imaging can be similarly realized by focusing on the display device 2.

[0059] [Imaging device - appearance] Fig. 4 shows an example of the external configuration of the imaging device 1. The left side of Fig. 4 shows the configuration as seen from the back of the housing 400 of the imaging device 1, and the right side shows the configuration as seen from the right side of the housing 400. The back side is the side on which the display screen 401 of the display unit 41 is provided, and the front side is the side on which the lens 402 of the imaging unit 103 is provided.

[0060] In the side configuration, a lens 402 of the lens unit 32 is disposed at the center position on the front side of the housing 400. Also, an electronic viewfinder (EVF) 403 is provided, for example, protruding upward from the main housing 400. The EVF 403 is one of the display devices 104.

[0061] In the configuration of the rear side, a rectangular display screen 401 corresponding to the above-mentioned touch panel is arranged, and information display and touch input operation are possible. In addition, the EVF 403 has a display screen 404 and a proximity sensor 405. The EVF 403 displays an image of a subject as a monitor image based on a signal input through a lens and an image sensor during shooting. A user can shoot while checking the monitor image of the EVF 403. Information such as an aperture value may be displayed superimposed on the display screen 404. The proximity sensor 405 detects the proximity of an object. The proximity sensor 405 may be a capacitance type, an optical type, or the like. For example, the control unit 101 determines whether or not the user is using the EVF 403 by detecting the presence or absence of proximity through the proximity sensor 405. The control unit 101 automatically switches between display on the display screen 401 and display of a monitor image on the display screen 404 of the EVF 403 according to the determination. Note that the EVF 403 is not limited to the EVF 403, and an optical viewfinder may be installed. It is also possible to set a mode in which a monitor image is displayed on the display screen 401 .

[0062] The housing 400 is provided with a mode setting dial 431, a shutter button 432, a cross key 433, a playback display mode button 434, a power button 435, and the like as the operation input unit 105. The cross key 433 includes up, down, left, right, and center buttons, and can be used to select information at a desired position on the display screen 401, and to input confirmation, cancellation, and the like. On the display screen 401, a menu can also be displayed based on a predetermined input operation.

[0063] The mode setting dial 431 is used to set the operation mode. The shutter button 432 is pressed when shooting. The playback display mode button 434 is used when playing back and displaying an image on the display screen 401, and can also switch the playback display mode. The power button 435 is used to turn the power on and off, etc. The user can perform various input operations, such as displaying a monitor image on the display screen 401 or the display screen 404 and touching a desired position to adjust the focus.

[0064] [Imaging device - Operation mode] The operation of the shooting mode as the operation mode of the imaging device 1 is outlined below. When the user presses the power button 435 to turn on the power, the normal shooting mode is started. The control unit 101 drives and controls the imaging unit 103 and the like according to the normal shooting mode. In the normal shooting mode, still images and videos can be captured as images. During shooting, the control unit 101 reads out an imaging signal from the imaging unit 103 at a predetermined cycle, and the signal is passed through the interface circuit unit 131 and subjected to predetermined signal processing in the signal processing unit 132 to convert it into a video signal in a display format. Based on the video signal, the control unit 101 displays a monitor image in real time on the display screen 401 of the touch panel of the display device 104 or the display screen 404 of the EVF 403. The user shoots as follows while checking the monitor image on the display screen 401 or the display screen 404.

[0065] The user sets information related to the angle of view, etc., as shooting conditions in the setting unit 14 through the operation input unit 105. When shooting a still image, the user presses the shutter button 432. The imaging control unit 11 detects the pressing of the shutter button 432, and determines the aperture value, shutter speed, focus, etc., according to the shooting conditions that have been set. The shutter button 432 has, for example, a half-pressed state and a fully-pressed state. The imaging control unit 11 sets the focus, etc., when the button is half-pressed, and controls to start imaging when the button is fully pressed.

[0066] The imaging signal of the still image captured by the imaging unit 103 is subjected to predetermined signal processing for still images using the signal processing unit 132 and the memory 133. As the encoding method, for example, JPEG is applied, but other methods such as MPEG or high-quality RAW format may also be applied. An image file 24 corresponding to the encoded still image data is recorded on the recording medium 107 by the recording unit 106, for example.

[0067] When shooting a moving image, the user presses the moving image shooting button to start shooting the moving image, and presses the moving image shooting button again to stop shooting the moving image. The captured image signal of the moving image is subjected to signal processing in a predetermined format for moving images. For example, MPEG such as H264 or H265, or other formats are applied as the format. The encoded moving image data is similarly recorded, for example, on the recording medium 107.

[0068] The operation of the playback display mode is outlined below. When the control unit 101 detects that the playback display mode has been set by the playback display mode button 434, it reads out the image file 24 from the recording medium 107 of the recording unit 106 and stores it in the memory unit 102. The display control unit 12 restores the original image data from the image data 23 by decoding processing using the signal processing unit 132, and generates video data for displaying the image, etc. on the display screen. The display control unit 12 controls the display device 104 based on the video data, and displays the image, etc. on the display screen.

[0069] [Management information and image files] 5 shows an example of the configuration of management information 21 and image files 24 in storage unit 102. Management information 21 includes image file management information and association management information. Image file management information is information managed by a known file system, and includes information such as file name, update date and time, type, and size. Association management information is specific information created to manage the relationships between multiple images. Association management information includes, for example, a type identifier, a reference identifier, and a setting character string.

[0070] The type identifier is an identifier for identifying the type of the image for each image file 24. There are two types of images, for example, a wide-angle image and a narrow-angle image. The reference identifier is an identifier for referring to associations between a plurality of images. For example, when there is a first image and a second image associated with it, the reference identifier of the first image is the file name of the image file 24 of the referenced second image. Also, the reference identifier of the second image is the file name of the image file 24 of the reference source first image. The reference identifier may have an identifier of a group to which the first image and the second image belong. The setting character string is a character string that can be arbitrarily set by the user based on a user operation. When editing and managing an image, the user can set a character string representing, for example, the shooting location to the image. Although the setting character string is set in the association management information, the setting character string may be set in the metadata 22.

[0071] The image file 24 includes metadata 22 and image data 23. The image file 24 has the metadata 22 in a header section and the image data 23 in a body section. The image data 23 is still image or video data, and is encoded data when used for recording. The metadata 22 is data that manages information including various attribute values ​​related to the image data 23, in other words, auxiliary information and attribute information. The metadata 22 includes information items such as shooting date and time, shooting location, model, resolution, shooting direction, angle of view, information related to the angle of view, rotation angle, and thumbnail.

[0072] The shooting date and time is, for example, year, month, day, hour, minute, and second. The shooting location includes location information or a set character string. The location information includes the position coordinates {latitude, longitude, altitude} detected by the position detection unit 113. The set character string is a character string representing the shooting location set by the user. The model is information that uniquely represents the type of the imaging device 1. The model may include the lens type, etc. The resolution represents the resolution of the image of the image data 23.

[0073] The shooting direction is information including an azimuth angle (θ) and an elevation angle (φ). The angle of view is information including a horizontal angle of view (Ah) and a vertical angle of view (Av). The angle of view related information is information such as an electronic zoom magnification (n), focal length (f), and dimensions of the imaging surface of the imaging element. The focal length f is the focal length converted into a 35 mm film. The dimensions are {horizontal size sw, vertical size sh, diagonal size sd}. Note that the metadata 22 may have a format in which only one of the angle of view item and the angle of view related information item is provided.

[0074] In other words, the rotation angle is the rotation around the optical axis of the lens during imaging, and is the angle that represents the rotation of the image during imaging. For example, as shown in FIG. 4, the angle or direction is based on the state in which the horizontal direction of the housing 400 of the imaging device 1 coincides with the horizontal plane. In the case of so-called landscape shooting, the rotation angle is close to 0 degrees corresponding to the reference state. In the case of an image in which the horizontal angle of view is larger than the vertical angle of view as a ratio of the angles of view, the image is long in the landscape direction. In the case of so-called portrait shooting, the rotation angle is close to 90 degrees relative to the reference state.

[0075] A thumbnail is a reduced image based on the original data of an image. A thumbnail is configured according to the resolution, type, etc. of an image, and is recorded with or without encoding. If a thumbnail is created and stored in advance, the thumbnail can be displayed quickly during playback display. A plurality of types of thumbnails with different resolutions, etc. may be created and stored. Thumbnails may be omitted, or may be managed separately from the metadata 22. The display control unit 12 may create and display a thumbnail each time playback display is performed.

[0076] The metadata 22 is in a predetermined format, for example, an extended format based on Exif (Exchangeable image file format), but is not limited to this. Exif is a format for digital camera photos. In Exif, information on the model and shooting conditions is embedded in image data as metadata. Image display and image editing software can refer to this metadata, and users can edit the metadata information. The metadata of conventional Exif image files does not include information such as the angle of view. In the first embodiment, as an extended format, items such as the angle of view and information related to the angle of view are provided as unique information items in the metadata 22.

[0077] The metadata 22 is created and managed as a header section of the image file 24, but may be created and managed as a metadata file separate from the image data file. Furthermore, management of specific information using the metadata 22 is not limited to a form in which similar information is collectively managed in the management information 21. In the case of a moving image, various pieces of information such as the metadata 22 may be described, for example, for each image frame at a predetermined time interval. The time interval may be constant, or may be only when there is a change in the angle of view, etc.

[0078] [Manage relationships between multiple images] FIG. 6 shows association management regarding the relationship between a plurality of images. The control unit 101 manages the relationship as shown in FIG. 6, for example, in the association management information in the management information 21 in FIG. 5. For the sake of explanation, the base side image is the first image, and the reference side image having a relationship with the first image is the second image. In the example of FIG. 6, the first images include images G1, G2, G3, G4, etc. The first image may be a wide-angle image or a narrow-angle image. The second image may be a wide-angle image or a narrow-angle image.

[0079] Image G1 is a narrow-angle image of narrow angle type, and has an azimuth angle θ1, an elevation angle φ1, a horizontal angle of view Ah1, a vertical angle of view Av1, etc. As second images associated with image G1, there are images g11, g12, and g13. Images g11, g12, and g13 are narrow-angle images of narrow angle type. For example, image g11 has an azimuth angle θ11, an elevation angle φ11, a horizontal angle of view Ah11, a vertical angle of view Av11, etc. These images G1 and images g11, g12, and g13 are associated as a plurality of images having a predetermined relationship. These images are associated as a group of images belonging to the same group, for example, the first group.

[0080] Image G2 is a wide-angle image of wide angle type, and has an azimuth angle θ2, an elevation angle φ2, a horizontal angle of view Ah2, a vertical angle of view Av2, etc. As second images associated with image G2, there are images g21, g22, and g23. Images g21, g22, and g23 are narrow-angle images. For example, image g21 has an azimuth angle θ21, an elevation angle φ21, a horizontal angle of view Ah21, a vertical angle of view Av21, etc. These images belong to the second group.

[0081] Image G3 is a narrow-angle image, and has an azimuth angle θ3, an elevation angle φ3, a horizontal angle of view Ah3, and a vertical angle of view Av3. Image G3 has images g31 and g32 as second images associated with it. Image g31 is a wide-angle image, and image g32 is a narrow-angle image. These images belong to a third group.

[0082] Image G4 is a wide-angle image, and has an azimuth angle θ4, an elevation angle φ4, a horizontal angle of view Ah4, and a vertical angle of view Av4. Image G4 has images g41 and g42 as second images associated with it. Image g41 is a wide-angle image, and image g42 is a narrow-angle image. These images belong to a fourth group.

[0083] For example, when image g11 is regarded as the first image on the reference side, image G1 is the second image on the reference side. Also, for example, image g11 may be regarded as the first image on the reference side, and another image may be associated with image g11 as the second image.

[0084] An example of the association information is a reference identifier for the image file of the first image, which has the image file name of each second image. Also, a reference identifier for each image file of the second image, which has the image file name of the first image. Also, a group identifier, which is a reference identifier for each associated image. These allow mutual reference between related images.

[0085] As one display mode, information showing the relationship between a plurality of images may be displayed on the display screen as shown in Fig. 6. In this case, the first image and the second image may be displayed in a tree structure as shown in Fig. 6, or the node of the second image may be displayed around the node of the first image with a link line connecting them (not shown).

[0086] In the first embodiment, the image types are classified into wide-angle and narrow-angle in terms of the angle of view. The imaging device 1 classifies the angle of view by an absolute value based on a threshold. A threshold value for the angle of view is set. When the angle of view of the captured image is equal to or greater than the threshold angle of view, the control unit 101 classifies the image as a wide-angle image, and when the angle of view is less than the threshold angle of view, the control unit 101 classifies the image as a narrow-angle image. Without being limited to this, three or more types of angle of view may be provided.

[0087] [Image shooting direction and angle of view] FIG. 7 shows a simple concept of the relationship between the shooting direction and the angle of view of a wide-angle image and a narrow-angle image as a specific example of a plurality of images. FIG. 7 shows the azimuth angle and the horizontal angle of view in particular. In this example, there are images 6a, 6b, 6c, and 6d. Images 6a, 6b, and 6c are narrow-angle images, and image 6d is a wide-angle image. These images are taken by a user at the same shooting point P0 using the imaging device 1. The images are taken at the closest dates and times in the order of image 6d, image 6a, image 6b, and image 6c. The shooting dates and times of the images are, for example, 13:10:00 on January 5, 2016 for image 6d, 13:11:30 on the same day for image 6a, 13:14:40 on the same day for image 6b, and 13:15:50 on the same day for image 6c.

[0088] Image 6a has a shooting direction 7a indicated by azimuth angle θa and a shooting range 701 corresponding to horizontal angle of view Aha. Image 6b has a shooting direction 7b indicated by azimuth angle θb and a shooting range 702 corresponding to horizontal angle of view Ahb. Image 6c has a shooting direction 7c indicated by azimuth angle θc and a shooting range 703 corresponding to horizontal angle of view Ahc. Image 6d has a shooting direction that is the same as shooting direction 7a of image 6a and has a shooting range 704 corresponding to horizontal angle of view Ahd. In FIG. 7, the shooting direction of the image, which is defined by azimuth angle θ, is indicated by a dashed line.

[0089] Point P0 particularly indicates the center point of lens 402 of imaging unit 103. Spherical surface 600 indicates the spherical surface of the entire celestial sphere, and is a virtual spherical surface with the point at infinity ignored. Spherical surface 600 particularly indicates the azimuth angle θ and the part of the circumference corresponding to the XY plane. N indicates the north direction, E indicates the east direction, S indicates the south direction, and W indicates the west direction. The X direction corresponds to the east direction, the Y direction corresponds to the north direction, and the Z direction corresponds to the zenith direction. The azimuth angle θ is expressed as N being 0° as the reference, E is +90°, S is +180°, and W is +270° (-90°) with the clockwise direction being the positive direction.

[0090] A shooting range 701 indicated by an arc on the spherical surface 600 indicates a portion of the image 6a that corresponds to the horizontal angle of view Aha, and corresponds to the area projected onto the spherical surface 600 as viewed from point P0. In this example, a case is simply shown in which a mountain 611, the sky, a tower 612, a building 613, etc. are shown as scenery captured in the image on the spherical surface 600. Image 6b is an example of an image of the tower 612, and image 6c is an example of an image of the building 613.

[0091] Specific examples of values ​​of the azimuth angle θ, elevation angle φ, horizontal angle of view Ah, and vertical angle of view Av of each image are as follows: Image 6a has an azimuth angle θa, an elevation angle φa, a horizontal angle of view Aha, and a vertical angle of view Ava; image 6b has an azimuth angle θb, an elevation angle φb, a horizontal angle of view Ahb, ​​and a vertical angle of view Avb; image 6c has an azimuth angle θc, an elevation angle φc, a horizontal angle of view Ahc, and a vertical angle of view Avc; and image 6d has an azimuth angle θd=θa, an elevation angle φd=φa, a horizontal angle of view Ahd, and a vertical angle of view Avd. θa=+10°, φa=0°, Aha=21°, Ava=14° θb=+45°, φb=0°, Ahb=21°, Avb=14° θc=-20°, φc=-10°, Ahc=21°, Avc=14° θd=+10°, φd=0°, Ahd=120°, Avd=80°

[0092] FIG. 8 is an explanatory diagram showing, like FIG. 7, particularly the elevation angle φ and the vertical angle of view Av. FIG. 8 shows an example of a narrow-angle image, which is another image 6e. In FIG. 8, the X direction corresponds to the horizontal direction, and the Z direction corresponds to the vertical direction. The positive direction in the Z direction corresponds to the head and zenith direction, and the negative direction corresponds to the foot direction. The elevation angle φ is set to 0° based on the horizontal direction, +90° in the zenith direction in the Z direction, and -90° in the foot direction. The azimuth angle θe, the elevation angle φe, the horizontal angle of view Ave, and the vertical angle of view Ahe of the image 6e are shown. The shooting range 705 corresponding to the vertical angle of view Ave of the image 6e is shown. The shooting direction 7e of the image 6e shows the part corresponding to the elevation angle φe.

[0093] [Angle of view and related information] FIG. 9 shows the angle of view and the angle-of-view related information in a simplified manner. FIG. 9 also shows a case where the angle of view and the angle-of-view related information are stored as the setting information in the setting unit 14. {horizontal angle of view Ah, vertical angle of view Av} is shown as the angle of view of the image. Also shown is the horizontal size Sh corresponding to the horizontal angle of view Ah of the image, and the vertical size Sv corresponding to the vertical angle of view Av. The horizontal size Sh is the size from the right end to the left end in the horizontal direction in the image frame. The vertical size Sv is the size from the top end to the bottom end in the vertical direction in the image frame. Note that in the first embodiment, the horizontal angle of view Ah and the vertical angle of view Av are used, but this is not limiting, and at least one angle of view including the diagonal angle of view Ad may be used.

[0094] The information related to the angle of view is the focal length f of the lens considering the optical zoom, the electronic zoom magnification n, and the dimensions of the imaging surface of the image sensor {horizontal size sw, vertical size sh, diagonal size sd}. Instead of the dimensions, the model may be used. The focal length f is the focal length converted into a 35 mm film.

[0095] The angle of view can also be obtained by calculation based on a relational expression using angle-of-view related information. When there is enough angle-of-view related information to calculate the angle of view, the control unit 101 may calculate the angle of view from the angle-of-view related information. The angle of view can be calculated using, for example, the dimensions of the image sensor, the focal length f, and the electronic zoom magnification n. For example, the angle of view can be calculated from a relational expression that is uniquely determined as an arctangent using information such as the 35 mm equivalent focal length or the dimensions of the image sensor.

[0096] The angle of view is calculated from the angle-of-view related information as follows. The imaging control unit 11 refers to the angle-of-view related information related to the image. The imaging control unit 11 can grasp the angle-of-view related information from the setting information of the setting unit 14 described above, or information described in the metadata 22 or management information 21. The angle of view {horizontal angle of view Ah, vertical angle of view Av, diagonal angle of view Ad} can be calculated, for example, from the following formula. Ah=2×arctan(sw / (2×n×f)) ···(1) Av=2×arctan(sh / (2×n×f)) ···(2) Ad=2×arctan(sd / (2×n×f)) ···(3)

[0097] For example, when electronic zoom magnification n, focal length f, and dimensions {horizontal size sw, vertical size sh} are given, horizontal angle of view Ah1 and vertical angle of view Av1 are determined. The control unit 101 describes the information on the angle of view in the angle of view item of the metadata 22 or in the management information 21. Since the angle of view can be calculated from the angle of view related information, the imaging device 1 may store the angle of view related information as one of the metadata 22, etc., instead of storing the angle of view. Since the dimensions of the imaging element of the angle of view related information can be grasped from the model, the imaging device 1 may store the model as one of the metadata 22, etc., instead of the dimensions of the imaging element. For example, the imaging device 1 holds information indicating the correspondence between the model and the angle of view related information in advance. Alternatively, the imaging device 1 can refer to the information indicating the correspondence from an external device through communication. The imaging device 1 can obtain the angle of view related information from the model based on the information.

[0098] When the imaging device 1 captures an image, it creates information including the shooting date and time, the shooting direction {azimuth angle θ, elevation angle φ}, the angle of view {horizontal angle of view Ah, vertical angle of view Av}, etc. The imaging control unit 11 obtains the shooting date and time from the internal clock 112. The imaging control unit 11 obtains the position information {latitude, longitude, altitude} from the position detection unit 113. The imaging control unit 11 obtains the shooting direction {azimuth angle θ and elevation angle φ} from the imaging direction measurement unit 111. The imaging control unit 11 obtains the angle of view {horizontal angle of view Ah, vertical angle of view Av} from the setting information of the setting unit 14 or by calculation. Then, the imaging device 1 describes the information in the corresponding items of the metadata 22 in FIG. 5 and creates it as an image file 24 together with the image data 23.

[0099] [Display screen - Display mode] 10 shows the display modes and their transitions of the display screen 401 of the display unit 41. The imaging device 1 has at least a first display mode, a second display mode, and a third display mode as display modes that are playback display modes. For example, the imaging device 1 initially displays in the first display mode, and switches the display mode based on a user operation.

[0100] As will be described later, after shooting or during playback and display, the imaging device 1 associates multiple images based on the shooting location, shooting date and time, etc., shooting direction, angle of view, etc., sets the multiple images into groups, and creates association management information. When playing back and displaying images, the imaging device 1 displays multiple images of the same group that are related to each other on the display screen 401 in an associated state as follows, based on the metadata 22 (at least shooting direction information) and the association management information.

[0101] [1st display mode - 1st screen] The first display mode is a mode for displaying a list of information on all images, and is a mode for displaying a folder screen. In the first screen of the first display mode, information on a plurality of images in a folder is displayed in a list in an order using, for example, file names and dates and times as keys. In this example, information on images j1, j2, j3, etc., as a plurality of images, is displayed in order from top to bottom. In each row of images, an icon or thumbnail of the image, file name, shooting date and time, type, and other information are displayed. In this example, the type of images j1, j2, j3 is narrow angle. As other information, attribute values ​​such as shooting direction {azimuth angle θ, elevation angle φ} and angle of view {horizontal angle of view Ah, vertical angle of view Av} may be displayed. The user can display information that cannot be displayed on the first screen by scrolling, etc. The first screen is not limited to such a folder screen, and various known methods are possible. For example, a screen in which thumbnails of a plurality of images are arranged in parallel in a predetermined direction or in a two-dimensional shape such as vertical and horizontal is possible.

[0102] On the first screen, a user selects a desired first image, for example, image j1. Image j1 corresponds to image G1 in FIG. 6 or image 6a in FIG. 7. When a second image associated with the selected first image exists, the imaging device 1 transitions to the second screen in the second display mode. Image j1 is associated with images j2, j3, etc., for example. Images j2, j3, etc. correspond to images g11, g12 in FIG. 6 or images 6b, 6c in FIG. 7. The relationship between images j1 to j3 corresponds to the first group in FIG. 6. [2nd display mode - 2nd screen (1) - thumbnail, 1st group]

[0103] The second display mode is a unique mode in which multiple images are displayed in association with each other. The display control unit 12 displays the second screen based on the image file 24 and the management information 21. On the second screen in the second display mode, a predetermined background area 50 is provided over the entire screen. The background area 50 is a black area or the like, and can be set by the user. The background area 50 has a size equal to or larger than the vertical and horizontal sizes of the display screen 401, and is large enough to encompass the range of the angle of view of multiple images.

[0104] The display control unit 12 displays a plurality of images related to each other on the background area 50 in the display screen 401 in an associated state. The center point of the display screen 401 and the background area 50 is set to point p0. A figure 61 representing an image j1, which is a selected first image, is displayed at point p0 on the background area 50. The center point p1 of the figure 61 is placed at point p0. In the first embodiment, the figure 61 of the first image is a thumbnail. Around the figure 61 of the image j1 on the background area 50, a figure 62 representing an image j2 and a figure 63 representing an image j3 are displayed as figures representing related second images. The center point p2 of the figure 62 and the center point p3 of the figure 63 are shown. The figures 62 and 63 of the second image are, for example, thumbnails. The display positions of the figures 62 and the like of the second image are determined according to the relationship between the shooting direction and the angle of view with respect to the display position of the figure 61 of the first image. The display size of each thumbnail is determined to a size corresponding to the angle of view.

[0105] On the second screen, the display state can be changed by operations such as scrolling and zooming in and out according to user operations. For example, by an operation such as swiping on the second screen, scrolling and zooming in and out can be performed within a range corresponding to the background region 50. When the display state is changed, the positional relationship between the multiple images is maintained.

[0106] A desired first image or a figure of a second image is selected by the user on the second screen. The selection operation is, for example, tapping on the figure. In this case, the imaging device 1 transitions to the third screen of the third display mode on which the selected image is displayed. In this example, the case where the figure 62 of the image j2, which is the second image, is selected is shown. Also, when a predetermined operation for returning to the first display mode is performed on the second screen, the imaging device 1 returns to the first screen of the first display mode. The predetermined operation is, for example, inputting designation of the first display mode, pressing the back button, tapping on the background area 50, etc.

[0107] In this way, on the second screen, multiple related images are displayed as thumbnails so that the relationships, including the relative positions, shooting direction, angle of view, etc., can be clearly seen. The user can intuitively grasp the relationship between the multiple images and can easily refer to the second image from the first image. Note that the center points, center lines, etc. shown in the figure for the purpose of explanation are not displayed, but they may be displayed on the screen as viewing aid information.

[0108] [Third display mode - Third screen] The third display mode is a mode for displaying details of a specified single image, a narrow-angle image or a wide-angle image. In this example, an example is shown in which an image j2, which is a narrow-angle image, is enlarged and displayed on the entire third screen. Display of a single wide-angle image is also possible. Initially, a central point p2 corresponding to the shooting direction (θ, φ) of the image j2, which is the target image, is displayed at a point p0 at the center of the display screen 401. When the entire image cannot be displayed within the display screen 401, the display control unit 12 may display only a part of the image, i.e., a part of the angle of view, or may display the entire image by reducing it. On the third screen, the display state can be changed by scrolling, zooming, etc. according to a user operation. In the case of displaying a wide-angle image, if the entire wide-angle image cannot be displayed, first, the shooting direction of the wide-angle image is aligned with point p0 to display a part of the wide-angle image, and then the display state is changed according to scrolling, etc. Similarly, when a figure 61 of an image j1, which is a first image, is selected on the second screen, the imaging device 1 displays only the image j1 in detail on the third screen.

[0109] When a predetermined operation is performed on the third screen, such as inputting designation of the second display mode, pressing the back button, tapping the image area, etc., the imaging device 1 returns to the second screen in the second display mode. Note that the transition is not limited to from the second display mode to the third display mode. An operation mode in which the first display mode is directly transitioned to the third display mode by a predetermined operation without passing through the second display mode is also possible, and the operation mode can be selected according to user settings, etc.

[0110] [Second display mode - Second screen (2) - Mark] FIG. 11 shows another display example of the second screen in the second display mode as a modified example. In this second screen, the figure 61 of the first image is also a thumbnail, and a predetermined mark is used as a figure representing the second image. The mark is particularly shown as a balloon shape. The mark is variable based on user settings. For example, a mark 72 representing image j2 is displayed at the center point p2, and a mark 73 representing image j3 is displayed at the center point p3. When the user selects a mark, an image corresponding to the mark is displayed on the third screen in the third display mode. Alternatively, a thumbnail or the like may be displayed at the position according to the mark selection operation. The display position of each mark is determined according to the image shooting direction (azimuth angle θ, elevation angle φ). In this manner, the screen display content is simplified.

[0111] [Second display mode - Second screen (3) - Transparent frame] FIG. 12 shows another display example of the second screen in the second display mode as a modified example. In this second screen, the figure 61 of the first image is also displayed as a thumbnail, and the figure representing the second image is displayed as a transparent frame. The transparent frame is displayed as a rectangle with four sides displayed with lines and colors, and the inside is a transparent area, and the background area 50 is displayed as it is. For example, a transparent frame 82 representing the image j2 is displayed at the center point p2, and a transparent frame 83 representing the image j3 is displayed at the center point p3. When the user selects a transparent frame, an image corresponding to the transparent frame is displayed on the third screen in the third display mode. Alternatively, a thumbnail or the like may be displayed at the position according to the selection operation of the transparent frame. The display position of the transparent frame is determined according to the shooting direction of the image, and the display size of the transparent frame is determined according to the angle of view of the image.

[0112] When the second screen is enlarged or reduced, the display control unit 12 updates the display size of the thumbnails and transparent frames so as to correspond to the changed display state. In this manner, when thumbnails and transparent frames are displayed, the relationship between the angles of view of multiple images can be easily understood.

[0113] 12, an example of the rotation angle of the second image is shown by a transparent frame 84 representing a narrow-angle image at point p4. This narrow-angle image has an angle α with respect to the x direction as its rotation angle.

[0114] Furthermore, in response to a predetermined operation on the figure of the second image within the second screen, such as a double tap, the display control unit 12 may switch the display state so that the second image is positioned at the center of the display screen 401 as a new first image, and may also display the new second image with reference to it.

[0115] [2nd display mode - 2nd screen (4) - 2nd group] FIG. 13 shows another example of the display of a plurality of images on the second screen. In this example, the first image j5 selected on the first screen is a wide-angle image. The second images associated with the first image are images j6 and j7, which are narrow-angle images. The relationship of the images in this example corresponds to the second group in FIG. 6.

[0116] A thumbnail of figure 65 representing image j5 is displayed with a center point p5 corresponding to the shooting direction of image j5 positioned at a center point p0 on background region 50. The display size of the thumbnail of figure 65 is determined according to the angle of view of the wide-angle image. In this example, the thumbnail of figure 65 fits entirely within display screen 401 and background region 50. If the thumbnail of figure 65 does not fit entirely within display screen 401 and background region 50, a part of the thumbnail corresponding to the angle of view of a portion of the wide-angle image is displayed. The thumbnail of figure 65 can be displayed entirely by scrolling, etc.

[0117] A transparent frame 86 representing image j6 is displayed so as to overlap center point p6 on figure 65, and a transparent frame 87 representing image j7 is displayed so as to overlap center point p7 on figure 65. In this way, even if the first image is a wide-angle image and the second image is a narrow-angle image, the relationship between these multiple images can be grasped on the second screen.

[0118] [2nd display mode - 2nd screen (5) - 3rd group] FIG. 14 shows another example of the display of the second screen, in which another image is displayed. In this example, image j1, which is the first image selected in the first screen, is a narrow-angle image. Images j8 and j9 are associated with the first image as second images, and image j8 is a wide-angle image. The relationship between the images in this example corresponds to the third group in FIG. 6.

[0119] A thumbnail, which is a graphic 61 representing image j1, is displayed at a central point p0 on the background region 50. A transparent frame 88, which is a graphic representing image j8, is displayed at a central point p8 on the background region 50. A transparent frame 89, which is a graphic representing image j9, is displayed at a central point p9. In this example, the transparent frame 88 of image j8 extends outside the thumbnail of image j1. This shows a case where the transparent frame 88 of image j8 does not fit within the display screen 401, and a part of it is displayed on the background region 50. Another part of the transparent frame 88 can be displayed according to scrolling, etc. In this way, even when the first image is narrow-angle and the second image is wide-angle, the relationship between the multiple images can be grasped on the second screen.

[0120] [2nd display mode - 2nd screen (6) - 4th group] FIG. 15 shows another example of the display of the second screen, in which another image is displayed. In this example, the first image selected in the first screen, image j5, is a wide-angle image. Images j8 and j9 are associated with the first image, and image j8 is a wide-angle image. The relationship between the images in this example corresponds to the fourth group in FIG. 6.

[0121] A thumbnail, which is graphic 65 representing image j5, is displayed at center point p0 on background region 50. A transparent frame 88, which is a graphic representing image j8, is displayed at center point p8 on background region 50. In this example, the thumbnail of image j5 and the transparent frame 88 of image j8 partially overlap. This shows a case where transparent frame 88 of image j8 does not fit within display screen 401, and a portion of it is displayed on background region 50. In this way, even when the first image and the second image are both wide-angle, the relationship between these multiple images can be grasped on the second screen.

[0122] [Second display mode - Second screen (7) - Scroll] FIG. 16 shows a scrolling example as another display example of the second screen in the second display mode. In this second screen, the background area 50 has a width in the x direction corresponding to a range of 360° as the azimuth angle θ and the horizontal angle of view Ah. A part corresponding to a part of the angle of view range of the background area 50 is displayed in the display screen 401, and the display state can be changed by scrolling in the x direction, etc. A plurality of images are arranged in the background area 50 at positions according to the shooting direction and angle of view. Initially, the figure 61 of the first image is arranged and displayed at the point p0. The right end of the background area 50 in FIG. 16 corresponds to θ=180°, and the left end corresponds to -180°, and the left and right ends may be connected so that the display is looped. The display screen 401b shows an example after scrolling in the x direction. As a result, the figure of the image included in a part of the background area 50, such as the figure 161, is displayed on the display screen 401b.

[0123] [Second display mode - Second screen (8) - Zoom in / out] FIG. 17 shows an example of enlargement / reduction as another display example of the second screen in the second display mode. The upper side of FIG. 17 shows the state before enlargement, and the lower side shows the state after enlargement. In the upper second screen, a thumbnail is displayed as a figure 1701 of the image r1, which is the first image, at the center point p0 on the background area 50. As second images associated with the first image, there are, for example, images r2, r3, and r4. Image r2 is a wide-angle image shot in the same direction as image r1, and a transparent frame is placed at point p0 as a figure 1702. Image r3 is a narrow-angle image shot in a direction close to image r1, and a figure 1703 is placed at center point pr3. Since the overlapping portion of image r3 with respect to the thumbnail of the first image becomes large, the figure 1703 of image r3 is made a transparent frame. A transparent frame is placed at center point pr4 as a figure 1704 of image r4. The display size of each figure is determined based on the angle of view and the enlargement / reduction ratio.

[0124] From the above state, the desired portion is enlarged and displayed in response to an operation such as a pinch by the user. The portion to be enlarged is indicated by a frame 1700. In the second screen at the bottom, the portion of the frame 1700 is displayed over the entire display screen 401. Each image is enlarged and displayed while maintaining the relationship of the angle of view, etc. Due to the enlargement, the figure 1701 of the image r1 is displayed as an enlarged image 1701b rather than a thumbnail. The enlarged image 1701b is generated based on the original image data 23 or thumbnail of the image r1, and is generated using the original image data 23 in the case of high-definition display.

[0125] [2nd display mode - 2nd screen (9) - spherical background] FIG. 18 shows another display example of the second screen in the second display mode of the first embodiment. In this second screen, a circular area corresponding to the spherical surface of a hemisphere, rather than a rectangular area, is provided as the background area 50 in the display screen 401. A plurality of images having a relationship are displayed in the circular background area 50 of the second screen. Examples of a plurality of images having a relationship include images J1 to J5. The center point of the circle of the background area 50 is defined as point O. At point O, a thumbnail is initially displayed as a figure 601 representing a first image, for example, image J1. Point O is a point corresponding to the shooting direction (azimuth angle θ, elevation angle φ) of the first image. A point at any position on the spherical surface of the hemisphere of the background area 50 can be expressed by (azimuth angle θ, elevation angle φ). In this example, a case where point O is (θ=0°, φ=0°) is shown. Also, although orientation information is displayed at the top, bottom, left, and right ends of the circle of the background area 50 to assist viewing, this may be omitted. In this example, the orientation information displayed is the orientation angle that changes depending on the viewpoint direction within the sphere. The top point corresponds to (0°, 90°), the bottom point to (0°, -90°), the right point to (90°, 0°), and the left point to (0°, -90°).

[0126] The imaging device 1 generates a projected image area projected from the shooting position onto the spherical surface of the background area 50 according to the viewpoint direction for each image in a plurality of related images, based on information such as the shooting direction and angle of view, as a graphic representing the image. In this example, there are images J2 to J5 as second images associated with image J1, and graphics 602 to 605 are shown representing the second images. The graphics 602 to 605 are, for example, thumbnails, but may also be the above-mentioned transparent frames, marks, etc.

[0127] The user can view the spherical surface of the celestial sphere from any viewpoint direction according to the operation on the second screen. The imaging device 1 changes the display state of multiple images in the hemisphere of the background area 50 in real time by known image processing according to the viewpoint change operation. The viewpoint change operation may be scrolling by swiping or the like in the background area 50, or may be an operation of a scroll button outside the background area 50 in the up, down, left, and right directions or a similar hardware button operation. In addition, the image in the background area 50 can be enlarged or reduced according to an operation such as pinching. On this second screen, the user can view the relationship between multiple images while viewing the spherical surface of the celestial sphere of the background area 50 from a desired line of sight direction.

[0128] At the top left of the display screen 401, information is displayed indicating that the current display mode is the second display mode and that the first image at the center is image J1. At the bottom left of the display screen 401, an item for switching the display of the second image, which is a related image, is provided. When this item is in the on state, the first and second images are displayed as shown in the figure, and when it is in the off state, the second image is not displayed and only the first image is displayed. At the bottom right of the display screen 401, a back button and a display mode switching button are provided. The back button allows the user to return to the original state of the first screen, and the display mode switching button allows the user to select and transition between display modes. Even on this second screen, a detailed display can be performed on the third screen in the third display mode in response to a selection operation of a single image, and the user can return to the original state of the second screen in response to a predetermined operation on the third screen.

[0129] Specific examples of the shooting directions and angles of view of the images J1 to J5 are as follows. The image J1 has (θ,φ)=(0°,0°), {horizontal angle of view Ah, vertical angle of view Av}={74°,53°}. Similarly, the image J2 has (0°,40°), {14°,7°}. The image J3 has (45°,0°), {54°,38°}. The image J4 has (340°,-30°), {27°,40°}. The image J5 has (310°,20°), {17°,11°}. The image J1 is an image taken facing north in FIG. 7 and in the horizontal direction in FIG. 8, and is an image taken horizontally (rotation angle=0°) with a focal length f equivalent to 24 mm in 35 mm conversion. The image J4 is an image taken vertically (rotation angle=90°).

[0130] The display size of the thumbnail and transparent frame of each image displayed on the background area 50 is determined based on the angle of view information of the image and a predetermined relational expression, such as a size proportional to the angle of view. However, as the focal length increases and the angle of view decreases, the size of the thumbnail also decreases. Therefore, a minimum value may be set in advance for the display size of thumbnails, etc., and the display size may be determined to be equal to or larger than the minimum value.

[0131] The figure of the projected image displayed on the hemispherical surface of the background region 50 is not exactly a rectangle, but a figure distorted according to the radial direction, that is, a figure having a circular arc in the circumferential direction. In this example, the figure of the projected image is illustrated as a rectangle for simplicity. The figure of the projected image may be displayed as a rectangle as in this example, or may be displayed as a figure with accurate distortion. The processing method for displaying a figure with distortion is as follows. When generating a figure to be displayed on the hemispherical surface of the background region 50, the display control unit 12 reads out the image data 23 of the target image or a thumbnail in the metadata 22 into the memory 133. Based on the image data or thumbnail, the display control unit 12 uses the signal processing unit 132 to generate a projection image onto the hemispherical surface by known image processing. The display control unit 12 displays the projection image or a transparent frame corresponding to the outer frame of the projection image. The processing method for displaying the figure of the projected image as a rectangle for simplicity is as follows. The display control unit 12 omits the generation process of the projection image and calculates the position coordinates of the center point for displaying the figure of the image on the hemispherical surface of the background region 50 based on the shooting direction (θ, φ) of the image. The display control unit 12 displays a thumbnail or a transparent frame of the image by pasting it at the calculated center point.

[0132] FIG. 19 shows a display state in which the viewing direction is set to the zenith direction as another display example of the second screen having the circular background region 50. Examples of multiple images having a relationship include images K1, K2, and K3. For example, the azimuth angle θ of the shooting direction of image K1 is 0°, that of image K2 is θ=90°, and that of image K3 is θ=225°. In this second screen, a thumbnail of the selected first image is initially displayed at a position corresponding to θ=0° facing north, for example. When image K2 is selected as the first image, it is displayed in a state in which the azimuth angle θ is rotated by -90° while maintaining the relationship between the angles of view of the multiple images so that the top end of the circular background region 50 is θ=90°. In this display state, the relationship between the azimuth angles θ of the multiple images is easy to recognize.

[0133] The above-mentioned configurations of the plurality of display examples may be appropriately combined, or may be configured to transition between the plurality of display examples. For example, the display example of scrolling on the second screen in the second display mode in FIG. 16 and the display example of zooming in and out in FIG. 17 can be applied to any screen of the second screen or the third screen. In addition, by performing a zoom-out display operation on the second screen, the display may gradually transition from a narrow-angle range display to a wide-angle range display, and further continuously transition to a display state of the celestial sphere background on the second screen in FIG. 19. Conversely, the display state of the celestial sphere background in FIG. 19 may gradually transition to a wide-angle range display state in accordance with a zoom-in display operation.

[0134] In addition, in the second screen of Fig. 16, an image that is not displayed on the screen can be displayed on the screen by scrolling, but instead of this, the following may be done. The shooting direction measuring unit 111 etc. is operated even in the display mode to detect the orientation of the imaging device 1 or the display device 2. As a result, the image is displayed in a display state in which the orientation of the device and the shooting direction of the image match according to the orientation of the device. In other words, scrolling is automatically performed to match the orientation of the device.

[0135] [4th display mode - 4th screen] FIG. 20 shows a fourth screen in the fourth display mode as a modified example. The fourth display mode is a modified example of the first display mode. For example, when the fourth display mode is designated by a predetermined operation such as a display mode switching button on the second screen such as FIG. 18, the fourth screen is displayed. This fourth screen displays a list of multiple images having a relationship, for example thumbnails of images J1 to J5, arranged vertically and horizontally. For example, the thumbnail of image J1 is displayed in the upper left, and the thumbnail of image J2 is displayed to the right of it. On the first screen described above, information on all images is displayed in a list regardless of the relationship between the images, but on the fourth screen, only multiple images having the same group relationship are displayed in a list. The fourth screen can be transitioned to the first screen, second screen, third screen, etc. according to a user operation.

[0136] Furthermore, when wide-angle images and narrow-angle images are mixed on the first screen or the fourth screen, thumbnails of the wide-angle images and thumbnails of the narrow-angle images may be arranged vertically and horizontally and displayed. In this case, the display size of the thumbnails may be different depending on the size of the angle of view of the images. For example, when the wide-angle image is a panoramic image with a large horizontal angle of view of 180° or more, the thumbnail of the wide-angle image may be displayed in one row, and thumbnails of multiple narrow-angle images may be displayed in parallel in the row below. The number of images arranged in parallel in one direction is variable, such as two, three, four, etc.

[0137] [Map display mode] FIG. 21 shows a display screen in a modified example in which a map display mode is provided as one of the display modes. The imaging device 1 acquires map information from the outside through the communication network 4, or uses map information previously stored inside. The imaging device 1 displays map information in the background area 50 in the display screen 401 as a screen corresponding to the map display mode. The imaging device 1 displays figures representing each target image on the map in the background area 50 based on the position information of the metadata 22 of the first image and the second image which are target images. For example, at the central point p0, a thumbnail is displayed as a figure 611 representing the image m1 which is the first image whose position is the shooting point. This figure 611 may be a transparent frame or a mark. Also, at each point near the point p0, a figure representing the second image taken at that position is displayed as, for example, a mark. For example, at the point pm2, a mark 612 representing the image m2 which is the second image is displayed. These multiple images are associated images within a predetermined distance range. On this screen, it is possible to switch to a screen of another display mode in response to a predetermined operation by the user. Furthermore, when a figure at a desired position is selected on this screen, it may be displayed in detail on the third screen in the third display mode, or a thumbnail may be displayed at that position.

[0138] [Association method] The association method includes a first method and a second method with respect to the timing of association. The first method is a method in which association is performed at the time of image capture, and the second method is a method in which association is performed at the time of playback display. In the first embodiment, the image capture device 1 uses the first method, and the display device 2 uses the second method.

[0139] In the first method, immediately after capturing an image, the imaging device 1 performs an association process using the associating unit 15, and describes the resulting association information in the metadata 22 or management information 21. When playing back and displaying the image, the imaging device 1 can grasp the relationship between multiple images by referring to the association information in the metadata 22 or management information 21, and displays the images on the display screen in the second display mode.

[0140] In the second method, when playing back and displaying an image on the display screen based on the image file 24, the display device 2 performs an associating process using the associating unit 15, and displays the result on the display screen in the second display mode. In the second method, the display device 2 may omit describing the associating information in the metadata 22 and the management information 21 by performing the associating process each time an image is played back and displayed. Alternatively, in the second method, the display device 2 may perform the associating process the first time an image is played back and displayed, describe the associating information in the metadata 22 and the management information 21, and thereafter, when playing back and displaying the image for the second or subsequent times, omit describing the associating process by referring to the associating information.

[0141] In the imaging device 1 of another embodiment, the second method may be adopted. The association method to be adopted may be fixed in the implementation, or may be selectable and configurable according to a user setting.

[0142] [Processing flow (1)] Fig. 22 and Fig. 23 show processing flows of the imaging device 1 and the display device 2 of the first embodiment. Fig. 22 shows a flow of control processing when an image is captured in the imaging device 1. Fig. 22 has steps S101 to S106. The same processing is repeated for each image capture. The steps will be described below in order.

[0143] (S101) Based on a user operation, the control unit 101 sets the angle of view and the like in the setting unit 14. The imaging control unit 11 grasps the angle of view and the like based on the setting information of the setting unit 14.

[0144] (S102) Based on a user operation, the imaging control unit 11 controls the imaging unit 103 to capture an image, obtains image data 23, and stores the image data in the storage unit 102. The imaging control unit 11 describes the angle of view or information related to the angle of view in the metadata 22 or management information 21 of the image file 24.

[0145] (S103) Branching is performed depending on whether the first method is used, which performs association during image capture, or the second method is used, which performs association during playback and display. If the first method is used, the process proceeds to S104, and if the second method is used, the process proceeds to S106. That is, in the case of the image capture device 1 of the first embodiment, the process proceeds to S104 since the first method is adopted, and in the case of the image capture device 1 of the other embodiments, the process proceeds to S106 if the second method is adopted.

[0146] (S104) The associating unit 15 performs a predetermined associating process, thereby associating a plurality of images including the first image and the second image.

[0147] (S105) The associating unit 15 creates association information as a result of the associating process, and describes the association information in the metadata 22 or management information 21 of the image file 24. The imaging control unit 11 stores the image file 24 in the memory unit 102, and causes the recording unit 106 to record the image file 24 on the recording medium 107.

[0148] (S106) The imaging control unit 11 does not perform association at this point, and therefore creates metadata 22 and management information 21 that do not include association information. The control unit 101 stores the image file 24 in the storage unit 102, and causes the recording unit 106 to record it on the recording medium 107.

[0149] [Processing flow (2)] Fig. 23 shows a flow of control processing when an image is played back and displayed in the imaging device 1 or the display device 2. The control in Fig. 23 corresponds to the control for transitioning the display mode as in Fig. 10. Fig. 23 has steps S201 to S209. The steps will be explained below in order. The following explanation will be given for the case where the imaging device 1 is the subject, but the same applies when the display device 2 is the subject.

[0150] (S201) Based on a user operation, the control unit 101 causes the display control unit 12 to display the first screen in the first display mode on the display screen 401 of the display unit 41 of the display device 104 as shown in Fig. 10. The display control unit 12 accepts the selection of an image in the first screen in response to a user input operation. The selected image is set as the first image, which is the reference side.

[0151] (S202) The display control unit 12 branches depending on whether the method is the first method or the second method. If the method is the second method, the process proceeds to S203, and if the method is the first method, the process proceeds to S205. That is, in the case of the imaging device 1 of the first embodiment, the process proceeds to S205 since the method is the first method, and in the case of the display device 2, the process proceeds to S203 since the method is the second method.

[0152] (S203) Immediately before playback and display, the display control unit 12 uses the associating unit 15 to perform an association process for a plurality of images.

[0153] (S204) If the result of S203 shows that there is a second image associated with the first image, the display control unit 12 reads out image data 23 of the second image.

[0154] (S205) In the case of the first method, the association process has already been performed. The display control unit 12 checks the association by referring to the association information in the metadata 22 or management information 21 of the image file 24 of the first image. The display control unit 12 reads out the image data 23 of the second image associated with the first image.

[0155] (S206) The display control unit 12 uses the image data 23 resulting from S204 or S205 to display a plurality of images including the first image and the second image on the display screen in the second display mode. That is, as in the example of the second screen in FIG. 10, the display control unit 12 displays the figure of the first image at the center position on the background area 50, and displays the figure of the second image around it. The display control unit 12 accepts selection of the figure of the first image or the second image in the second screen in response to a user input operation. For example, when only the first image or only the second image is viewed, a selection operation of the corresponding figure is performed. When returning to the first screen, a predetermined operation is performed.

[0156] (S207) The display control unit 12 branches depending on whether the first image or the second image figure is selected on the second screen in the second display mode, or whether a back operation or the like is performed. If the first image or the second image is selected, the process proceeds to S208. If another operation is performed, such as pressing the back button, specifying the first display mode, tapping the background area 50, or the like, the process returns to S201.

[0157] (S208) The display control unit 12 uses the image data 23 of the image corresponding to the figure selected in S207 to display the image in detail on the third screen in the third display mode. The display control unit 12 accepts a user input operation on the third screen. For example, in order to return to the second screen, the back button is pressed, the second display mode is specified, the image area is tapped, or the like is performed.

[0158] (S209) If an operation such as "Back" is performed on the third screen of the third display mode, the display control unit 12 returns to S206 as shown by A1, and if not, continues the display in the third display mode in S208. Note that the display may be returned from the third display mode to the first screen of the first display mode in response to a predetermined operation.

[0159] [Association process (1)] The associating unit 15 performs an associating process for a plurality of images. The associating process includes a first process and a second process. The first process is a process for selecting images that are candidates for association based on a selection condition of a predetermined selection method. The predetermined selection method includes a first selection method, a second selection method, a third selection method, and the like, which are shown below. In the first embodiment, the first selection method is used. In the modified example, the second selection method or the third selection method is used. The selection method to be adopted may be fixed in implementation, or may be variably selectable by user settings.

[0160] [First Selection Method - Shooting Location] FIG. 24 is an explanatory diagram showing the first selection method. The first selection method uses a condition related to the shooting location as a selection condition. In the first selection method, images that are determined to have been shot at roughly the same location are selected as candidates. In the first selection method, the location information {latitude, longitude, altitude} from the location detection unit 113 is used to determine the shooting location. When associating, the association unit 15 refers to the location information {latitude, longitude} described in the shooting location item of the metadata 22, for example, at the time of image capture. {latitude, longitude} is represented as {LA, LO}.

[0161] As shown in FIG. 24, the associating unit 15 takes the first image as a reference and takes a position 241 representing the shooting location of the first image as a center point in space, for example, an XY plane. The first image is image k1, and the images that are the second images are images k2 to k4. The position 241 is expressed as {LA1, LO1}. The position range 240 corresponds to the threshold value of the condition, and indicates a circular area with a predetermined radial distance with the position 241 as a center point. The radial distance that defines the position range 240 may be a fixed setting value in design, or may be a setting value that can be changed by a user setting. The position range 240 is not limited to a circle.

[0162] The associating unit 15 compares the position of each image, which is the second image, with the position 241 of the first image. In this example, a case is shown in which the position 242 {LA2, LO2} of image k2, the position 243 {LA3, LO3} of image k3, and the position 244 {LA4, LO4} of image k4 are around the first image. The associating unit 15 searches for an image included within the position range 240 for the position 241 of the first image. In this example, the position 242 of image k2 and the position 243 of image k3 are included within the position range 240, but the position 244 of image k4 is not included. The associating unit 15 determines that the second image, the position of which is included within the position range 240, is an image captured at roughly the same shooting location as the first image, and selects it as a candidate for association. An example of the same shooting location is when a user takes pictures in each direction of 360° while going around a circle along a handrail at an observation deck.

[0163] The means for determining the same shooting location is not limited to a method using GPS location information, but may also use a method using location information from a short-range communication interface such as BlueTooth (registered trademark) or an indoor positioning system. For example, a device compatible with a short-range communication interface is installed in a facility, and the imaging device 1 acquires location information by communicating with the device. This allows for highly accurate location determination. [Second selection method - Photo date and time]

[0164] 25 is an explanatory diagram showing the second selection method. The second selection method uses a condition related to the shooting date and time as a selection condition. In the second selection method, images that are determined to have roughly the same shooting date and time are selected as candidates. When associating, the associating unit 15 refers to the shooting date and time (year, month, day, hour, minute, and second) described in the shooting date and time item of the metadata 22, for example.

[0165] The associating unit 15 takes the time t1 representing the shooting date and time of the first image based on the first image in the time series of shooting dates and times. In this example, the time t1 is the date and time DT3. The date and time range 250 corresponds to the threshold value of the condition and indicates a range of a predetermined time centered on the time t1 and including both before and after the time t1. The time that defines the date and time range 250 may be a fixed setting value set by design, or may be a setting value that can be changed by the user. The date and time range 250 can be set, for example, to one day (24 hours), one hour, five minutes, etc.

[0166] The associating unit 15 searches for a second image included within a predetermined date and time range 250 with respect to the time t1 of the photographed date and time of the first image. The associating unit 15 compares the photographed date and time of each image, which is the second image, with the time t1 of the photographed date and time of the first image. In this example, a case is shown in which the photographed date and time of image k2, the time t3 of image k3, and the time t4 of image k4 are near the time t1. In this example, the date and time range 250 includes the time t2 of image k2 and the time t3 of image k3, but does not include the time t4 of image k4. The control unit 101 determines that the second image, whose time is included within the date and time range 250, is an image captured at roughly the same photographed date and time as the first image, and selects it as a candidate for association.

[0167] [Third selection method - setting string] Fig. 26 is an explanatory diagram showing the third selection method. The third selection method uses, as a selection condition, a condition related to a set character string that can be arbitrarily set by the user. In the third selection method, images that are determined to have roughly the same set character string are selected as candidates. When associating, the associating unit 15 refers to a character string that is set as a set character string in the association management information after, for example, capturing an image.

[0168] FIG. 26 shows an example of a list display of information on a plurality of images on the first screen in the first display mode. In this example, the file name, shooting date and time, tag, type, etc. are displayed as information for each image. The "tag" item corresponds to a set character string. The user can set any character string in the "tag" item during editing. In this example, the user sets a character string representing the shooting location in the "tag" item. For example, images k1, k2, and k3 are images taken in the same park, and the character string "Park A" is set. Images k4 and k5 are images taken in the same observation deck, and the character string "Observation Deck B" is set. The "Type" item displays "Wide Angle" or "Narrow Angle" as the type of the angle of view of the image.

[0169] The associating unit 15 refers to the setting string of the "tag" item of a first image selected by a user, for example, image k1, as a reference. The associating unit 15 refers to and compares the setting string of the first image with the setting string of each of the other second images, and searches for images having setting strings that include the same string, for example. For example, the setting strings of images k1, k2, and k3 are the same. The associating unit 15 determines that these images were taken at the same shooting location, and selects them as candidates for association.

[0170] As a modified example, the control unit 101 may automatically set a character string representing the shooting location without depending on a user operation when capturing an image, etc. For example, the control unit 101 may determine a character string representing the shooting location based on location information {latitude, longitude}. The control unit 101 may set a character string in the "file name" item of the image, etc.

[0171] The above selection methods can also be applied in combination. For example, the first selection method's shooting location condition and the second selection method's shooting date and time condition can be combined. For example, the condition can be specified as "same shooting location up to one year ago."

[0172] [4th Selection Method - Type of Subject] As another selection method, the following fourth selection method is also possible. The imaging device 1 determines the type of subject by image recognition processing. The types of subjects include landscapes, buildings, people, animals, etc. The type of subject that can be determined depends on the accuracy of the image recognition processing, etc. Depending on the determined type of subject, the imaging device 1 presumes that multiple images having the same subject type are a related group and selects them as candidates.

[0173] Furthermore, the imaging device 1 may use the determined type of subject as a criterion for determining the relationship between a plurality of images. The imaging device 1 adjusts the position range, which is, for example, the threshold value of the first selection method, according to the type of subject. For example, when the type of subject is a landscape, even if the focal length is large and the difference in the position of the shooting point is large, the effect of the image change is small. Therefore, in this case, the position range is set relatively large. As a result, a group of images included within the large position range are associated. When the type of subject is a person, when the focal length is small and the difference in the position of the shooting point is large, the effect of the image change is large. Therefore, in this case, the position range is set relatively small.

[0174] Similarly, the imaging device 1 may adjust thresholds such as the position range according to the distance to the subject and the shooting mode. For example, when the shooting mode is set to a macro mode suitable for landscape photography, the position range, which is the threshold, is set relatively large.

[0175] [Association process (2)] The second process in the association process of the association unit 15 is as follows. In the second process, the image selected as a candidate in the first process is judged to be associated using information on the shooting direction and the angle of view, and the image is associated according to the judgment result. The association unit 15 refers to the information on the shooting direction and the angle of view in the metadata 22 of each image file 24 or the management information 21. As a first condition, the association unit 15 judges whether or not the shooting direction (azimuth angle θ, elevation angle φ) of the candidate image is included within the range of the angle of view {horizontal angle of view Ah, vertical angle of view Av} of the first image serving as a reference. The association unit 15 determines the image satisfying this first condition as the image to be associated. In addition, the association unit 15 may determine, as a second condition, whether or not the shooting direction (azimuth angle θ, elevation angle φ) of the candidate image is included within a predetermined angle of view range extended outside the range of the angle of view {horizontal angle of view Ah, vertical angle of view Av} of the first image serving as a reference. The association unit 15 determines the image satisfying this second condition as the image to be associated. In the second association process, the first condition or the second condition is applied.

[0176] An example that satisfies the first condition is as follows. As an example of the first image, the image 6d, which is a wide-angle image in FIG. 7, is taken. The horizontal angle of view Ahd of the image 6d is 120°, and the shooting range 704 of the horizontal angle of view Ahd has a right end position of 70° and a left end position of -50°. The shooting range 704 includes an azimuth angle θb=45° of the shooting direction 7b of the image 6b and an azimuth angle θc=-20° of the shooting direction 7c of the image 6c. Although not shown, the elevation angle φb of the shooting direction 7b of the image 6b and the elevation angle φc of the shooting direction 7c of the image 6c are included within the range of the vertical angle of view Avd of the image 6d. This satisfies the first condition, and the image 6d is associated with the images 6b and 6c.

[0177] 27 is an explanatory diagram of the first condition, second condition, etc., when determining the relationship between a plurality of images. In the entire background area 50 within the display screen 401, the horizontal direction within the screen corresponding to the azimuth angle θ is the x direction, and the vertical direction within the screen corresponding to the elevation angle φ is the y direction. The width of the display screen 401 in the x direction is W, and the height in the y direction is H. The center point p0 of the display screen 401 is expressed by the position coordinates (x0, y0). It is assumed that background areas 50 of the same size or larger are set throughout the entire display screen 401.

[0178] As the first image, which is the reference, the image q1, which is the first narrow-angle image, is taken as an example. The point corresponding to the shooting direction (θ,φ) of the image q1 is the center point p1. In this example, the (θ,φ) of the center point p1 is (0°, 0°). The figure 271 representing the image q1 is arranged so that the center point p1 is aligned with the point p0. The figure 271 is rectangular here and is shown by a solid line frame. The horizontal angle of view Ah1 of the image q1 is 21°, and the vertical angle of view Av1 is 14°. The figure 271 of the image q1 has a horizontal size Sh1 corresponding to the horizontal angle of view Ah1 and a vertical size Sv1 corresponding to the vertical angle of view Av1. The right end of the figure 271 in the x direction is indicated by the position x11, and the left end is indicated by the position x12, which are 10.5° and -10.5° in azimuth angle θ, respectively. The upper end of the figure 271 in the y direction is indicated by position y11, and the lower end is indicated by position y12, which are 7° and −7° in terms of elevation angle φ, respectively.

[0179] As the second image, image q2, which is the second narrow-angle image, is taken as an example. The point corresponding to the shooting direction (θ, φ) of image q2 is set as center point p2. In this example, (θ, φ) of center point p2 is set as (16°, -6°). A figure 272 representing image q2 is placed at center point p2 and is also indicated by a solid frame. The horizontal angle of view Ah2 and vertical angle of view Av2 of image q2 are 21° and 14°, the same as those of image q1. The figure 272 of image q2 has a horizontal size Sh2 corresponding to the horizontal angle of view Ah2 and a vertical size Sv2 corresponding to the vertical angle of view Av2. The right end of the figure 272 in the x direction is indicated as position x21, and the left end is indicated as position x22, which are 26.5° and 5.5° in azimuth angle θ, respectively. The upper end of the figure 272 in the y direction is indicated by position y21, and the lower end is indicated by position y22, which correspond to 1° and −13° in terms of elevation angle φ, respectively.

[0180] An example of the first condition is that the shooting direction (θ, φ) of image q2 is included within the range of horizontal angle of view Ah1 and vertical angle of view Av1 centered on the shooting direction of image q1, in other words, that the center point p2 of image q2 is included within the range of horizontal size and vertical size of image q1. In particular, it is assumed that both the azimuth angle θ and the elevation angle φ must satisfy the condition. In this example, the azimuth angle θ=16° of image q2 is outside the range of the horizontal angle of view Ah1 of image q1 (-10.5° to 10.5°), and the elevation angle φ=-6° is within the range of the vertical angle of view Av1 (-7° to 7°). Therefore, image q2 does not satisfy the first condition.

[0181] For the second condition, an extended threshold range 273 is set outside the range of the horizontal angle of view Ah1 and the vertical angle of view Av1 centered on the shooting direction of the image q1, which is the first image. The threshold range 273 is indicated by a dashed frame. The threshold range 273 may be a fixed setting value in implementation, or may be a user-set value. The threshold range 273 may be particularly set to match the size of the background region 50. In particular, the threshold range 273 may be a range obtained by a predetermined calculation formula based on the size of the angle of view of the first image, for example, a range obtained by adding a predetermined angle of view or a range obtained by multiplying a predetermined rate. In this example, the threshold range 273 is set to a range three times larger than the range of the angle of view of the first image, with the horizontal angle of view Aht=63° and the vertical angle of view Avt=42°. The right end of the threshold range 273 in the x direction is position x31, the left end is position x32, and the azimuth angle θ is 31.5°, -31.5°. The upper end of threshold range 273 in the y direction is position y31, and the lower end is position y32, with elevation angles φ of 21° and −21°.

[0182] An example of the second condition is that the shooting direction (θ, φ) of image q2 is included within threshold range 273 based on the angle of view of image q1. In particular, it is assumed that both the azimuth angle θ and the elevation angle φ must satisfy the condition. In this example, the azimuth angle θ=16° of image q2 is within the range of the horizontal angle of view Aht of threshold range 273, and the elevation angle φ=-6° is within the range of the vertical angle of view Avt. Therefore, image q2 satisfies the second condition. As a result, when the second condition is used, image q2 is associated with image q1.

[0183] Other possible conditions are as follows. A third condition is that the angle of view range of the second image overlaps part of the angle of view range of the first image. A fourth condition is that the angle of view range of the second image overlaps part of the angle of view range of the first image within a threshold range outside the angle of view range of the first image. For example, the angle of view range of image q2 overlaps part of the angle of view range of image q1. In other words, the position x22 of the left edge of image q2 is to the left of the position x11 of the right edge of image q1. Therefore, image q2 is associated with image q1.

[0184] As a fifth condition, the difference between the angle (θ or φ) of the shooting direction of the first image and the corresponding angle (θ or φ) of the shooting direction of the second image is within a predetermined threshold angle. In other words, the distance between the position coordinates of the center point of the first image and the center point of the second image is calculated, and the distance is within a predetermined threshold distance. For example, the difference value dθ of the azimuth angle θ between the center point p1 of image q1 and the center point p2 of image q2 is 16°, and the difference value dφ of the elevation angle φ is -6°. The threshold angles are (dθt, dφt). The condition is satisfied if dθ≦dθt or if dφ≦dφt.

[0185] When the first image is a wide-angle image or when the second image is a wide-angle image, the above conditions can be similarly applied to determine the association. In particular, when the first image is a wide-angle image, the first condition may be applied, and when the first image is a narrow-angle image, the second condition may be applied. In this way, a condition may be selected according to the type of image, or a combination of multiple conditions may be applied.

[0186] [movie] Although the above mainly describes the case of still images, the same is basically applicable to the case of moving images. When shooting a moving image, the imaging control unit 11 shoots the image while changing the electronic zoom magnification, the angle of view, etc. in a continuous shooting mode to obtain image data 23 of the moving image. For an image file 24 of the moving image, the imaging control unit 11 describes information such as the electronic zoom magnification and the angle of view according to the time in the metadata 22 or management information 21.

[0187] When playing and displaying a video on the second screen in the second display mode, the display control unit 12 determines the display position of the figure based on the shooting date and time, location information, and the shooting direction that changes depending on the time, which are described in the metadata 22, etc. The display position may be, for example, a single fixed display position depending on the shooting direction of the first image frame that constitutes the video, or may be a display position determined from statistics of multiple image frames. The display position may be a display position corresponding to the shooting direction that changes depending on the image frame.

[0188] [Effects, etc.] As described above, the imaging device 1 and the display device 2 of the first embodiment make it easy to grasp the relationship between multiple images including a wide-angle image and a narrow-angle image, and can realize better usability. Since a related narrow-angle image is automatically superimposed and displayed near a wide-angle image on the display screen, for example, the user can easily recognize and intuitively grasp the related multiple images and can view them suitably. The user can also easily grasp the relationship between the wide-angle image and the narrow-angle image. While viewing a first image, the user can quickly refer to a related second image. The user can view a large number of images suitably by switching the display mode.

[0189] The following are examples of modifications of the first embodiment. As a modification, the relationship between the angle of view and the type of image may be determined relatively. That is, the imaging device 1 or the display device 2 refers to the angles of view of multiple images, and determines the image with the largest angle of view among them as a wide-angle image, and determines the images other than the wide-angle image as narrow-angle images.

[0190] [Variation (1) - 2nd screen] FIG. 28 shows an example of the second screen in the second display mode in the imaging device 1 of the modified example of the first embodiment. The display screen 401 has a main area 281 and areas other than the main area 281, such as a right side area 282 and a bottom side area 283 in this example. In the main area 281, a thumbnail is displayed as a figure 284 of the first image at a central position on the background area, and a figure of the associated second image is displayed around it. Different figures may be displayed for the first image and the second image depending on the type of image, the shooting date and time, etc. In this example, circular, rhombus, and star marks are shown as figures. As the type of image, for example, a type according to the difference in the angle of view may be set based on a user setting. In addition, information may be added to the figures and displayed. In this example, an identification number is added to the mark. A mark with an identification number of 1 is displayed together with the thumbnail of the first image. A mark with an identification number of 2 or less is displayed as the figure of the second image. In addition, an attribute value based on the metadata 22 or management information 21 may be displayed as information on the figure, or only the attribute value may be displayed.

[0191] In area 282 on the right side, information of all of the associated images, such as marks and attribute values, is displayed in parallel. In area 281, the display state based on a user operation is such that the figures of some images are displayed and the figures of other images are not displayed. In area 282, information of images that are not displayed in area 281 is also displayed. When the figure of an image is selected in area 281 or area 282, a transition to the third display mode may be made. Also, in the lower right area, a thumbnail of the last selected image is displayed as a preview.

[0192] In the area 283 on the bottom side, information on the display mode and information on the attribute value of the first image are displayed. Also, an operation button for switching between displaying and hiding the mark in the second display mode is provided. In the ON state, the mark is displayed, and in the OFF state, the mark is hidden. Also, a back button for returning to the first display mode and the like are provided.

[0193] In addition, in the case where multiple associated second images are densely present at roughly the same position or in the same area on the background area in the second screen, one representative mark is displayed instead of displaying individual figures overlapping each other. In this example, a star is used as the representative mark. When the user selects the representative mark, information on the multiple second images associated with the position of the representative mark is displayed. In this case, for example, a pop-up area 285 is displayed, and information on the multiple second images is displayed in the pop-up area 285. When the user selects a desired second image in the pop-up area 285, a transition is made to a third display mode in which the selected second image is displayed in detail.

[0194] As another method, information on a plurality of second images associated with the representative mark may be displayed in the area 282 on the right side, etc. Also, in response to the selection of the representative mark, the associated plurality of second images may be displayed in sequence, i.e., displayed as a slide show, in a predetermined area. For example, when there are a plurality of images taken at the same location, in the same shooting direction, but at different shooting dates and times, these images can be associated and continuously played and displayed. Also, in response to the selection operation of the representative mark, an area where a plurality of second images are densely packed may be enlarged and displayed so that the figures of the plurality of second images can be distinguished.

[0195] [Variation (2)-Tilt Operation] As a modified example, the tilt angle indicating the tilt and rotation of the housing of the imaging device 1 is detected using a sensor or the like of the imaging direction measuring unit 111. The imaging device 1 controls switching of the display mode and scrolling of multiple images in the second screen, etc., according to the state of the detected tilt angle. This function can be switched between an enabled display mode and an disabled display mode according to a predetermined operation. Normally, in the disabled display mode, the display content of the second screen is fixed and displayed regardless of the tilt angle of the imaging device 1. Then, the display state can be changed by scrolling, zooming in and out, etc., according to the above-mentioned predetermined operation.

[0196] Fig. 29 shows a state in which the display screen 401 of the housing 400 of the imaging device 1 is arranged on the XZ plane in the valid display mode. With respect to the orientation of the housing 400 and the display screen 401, a tilt operation about the y axis and its tilt angle 291, and a tilt operation about the x axis and its tilt angle 292 are shown. As an example of the second screen, an example similar to the second screen of Fig. 10 is shown. On the background area 50 of the second screen, a figure 61 of the first image is displayed in the center, and a figure 62 of the second image is displayed nearby.

[0197] In this state, the housing 400 is tilted as a user operation. For example, the housing 400 is tilted in the positive right direction with respect to the y-axis. The imaging device 1 detects the tilt angle 291 at that time, and changes the display state in the second screen by scrolling to the right according to the tilt angle 291. As a result, each image in the second screen is moved to the left, and at the same time, information on the right outside the screen is displayed on the screen. In this display mode, the user can easily change the display state by only tilting, and can view multiple images with little effort. Similarly, the zooming may be controlled according to the tilting operation.

[0198] [Variation (3) - Linking with external devices] The imaging device 1 and the display device 2 may store the image file 24 in an external device, or may read the image file 24 from the external device and play and display it. The imaging device 1 and the display device 2 may perform an association process on the image file 24 to be transmitted to the external device, and transmit the image file 24 including the association information to the external device. The imaging device 1 and the display device 2 may also perform an association process on the image file 24 received from the external device, and play and display the image file 24 in an associated state based on the association information. That is, as an association method other than the first method and the second method described above, a method of performing association before transmitting the image file 24 to the external device, or a method of performing association after receiving the image file 24 from the external device may be adopted. When transmitting a plurality of image files 24 to an external device, the imaging device 1 and the display device 2 may describe the angle of view information and the association information as the metadata 22 for each image file 24, or may provide the information by describing it in a management file separate from the plurality of image files 24.

[0199] 1, for example, the imaging device 1 performs an association process on multiple captured images before transmitting them to the server 3, creates an image file 24 including the angle of view and association information, and transmits the image file 24 to the server 3 via the communication network 4 to be stored in the image data DB. The imaging device 1 or the display device 2 accesses the server 3 via the communication network 4 and acquires the image file 24 from the image data DB of the server 3. At that time, it is possible to acquire multiple image files 24 collectively by specifying, for example, the shooting date and time or the shooting location as a search condition. The imaging device 1 or the display device 2 plays and displays the image data 23 of the acquired multiple image files 24 in an associated state based on reference to the association information.

[0200] Also, for example, the imaging device 1 does not perform association processing for the captured multiple images, but transmits an image file 24 that includes the angle of view but does not include association information to the server 3 to be stored in the image data DB. The imaging device 1 or the display device 2 accesses the server 3 through the communication network 4, and acquires the image file 24 from the image data DB of the server 3. The imaging device 1 or the display device 2 performs association processing for the acquired multiple image files 24, and plays and displays the images in an associated state based on the angle of view and the association information.

[0201] The server 3 may also perform an association process on multiple image data 23 in the image data DB. The server 3 performs the association process and stores the result as an image file 24 including association information. The imaging device 1 or the display device 2 plays and displays the multiple image files 24 acquired from the server 3 based on the association information.

[0202] Also, for example, the imaging device 1 performs an association process on a plurality of captured images, and transmits an image file 24 including the angle of view and association information to the display device 2. The display device 2 acquires the image file 24, and plays back and displays the image data 23 of the acquired image file 24 in an associated state based on reference to the association information.

[0203] Also, for example, the imaging device 1 does not associate multiple captured images, and transmits an image file 24 that includes an angle of view but does not include association information to the display device 2. The display device 2 acquires the image file 24, performs an association process on the image data 23 of the acquired image file 24, and plays and displays the images in an associated state based on the association information.

[0204] Furthermore, the imaging device 1 and the display device 2 may read image files 24 from multiple devices of the same user and manage them in an integrated manner. For example, the first imaging device 1A and the second imaging device 1B in Fig. 1 each transmit image files 24 of captured images to the first display device 2A without associating the images. The first display device 2A performs an association process on the image files 24 and plays and displays the images in an associated state.

[0205] According to this modification, an association display function can be realized in a system in which a plurality of devices work together, and a user can associate and save a large number of images.

[0206] (Embodiment 2) An imaging device and a display device according to a second embodiment of the present invention will be described with reference to Fig. 30 and Fig. 31. The following describes the differences between the configuration of the second embodiment and the configuration of the first embodiment. The imaging device 1 according to the second embodiment is provided with a unique shooting mode for associating multiple images.

[0207] [Shooting mode] The imaging device 1 of the second embodiment has, as imaging modes, a first imaging mode which is a normal imaging mode, and a second imaging mode which is a group imaging mode for performing association. The imaging device 1 switches the imaging mode based on a user input operation or a user setting. The user performs a predetermined operation to switch between the first imaging mode and the second imaging mode at any timing through the operation input unit 105 or the like. The predetermined operation may be, for example, pressing a mode switching button.

[0208] 30 shows switching of shooting modes on the time axis in the imaging device 1 of the second embodiment. The imaging device 1 is initially in the first shooting mode, and does not perform association processing for each image captured in the first shooting mode. For example, an image captured at time t11 is not associated, and association information is not described in the metadata 22, etc.

[0209] The user performs an operation to switch from the first shooting mode to the second shooting mode at a desired timing, for example, at time T2. In the second shooting mode, the imaging device 1 automatically associates the individual images captured during that period as a group. For example, images are captured at times t1 to t4 during the second shooting mode. The control unit 101 performs an association process on those images using the association unit 15, and describes the association information in the metadata 22 or management information 21. In this example, the images at times t1 to t4 are associated as one group. A group identifier is described in the image files 24 or management information 21 of those images. The group identifier may be a character string using the start date and time and the end date and time of the period of the second shooting mode, etc.

[0210] The user performs an operation to switch from the second shooting mode to the first shooting mode at a desired timing, for example, at time T4. The imaging device 1 does not associate individual images captured in the first shooting mode, for example, the image captured at time t12. The control unit 101 may execute the association process when returning to the first shooting mode. When playing back and displaying, the display control unit 12 checks the association based on reference to the group identifier in the image file 24 or the management information 21, and performs the association display.

[0211] [Menu screen] Fig. 31 shows a menu screen relating to the shooting mode as an example of a display screen of the imaging device 1. This menu screen is displayed, for example, by pressing the center button of the cross key 433 in Fig. 4. This menu screen has a shooting menu 311, a playback menu 312, and a setting menu 313. The shooting menu 311 has setting items relating to the shooting mode, the playback menu 312 has setting items relating to the playback display mode, and the setting menu 313 has setting items for the setting mode. Each setting item in each mode can be set by the user. For example, the shooting menu 311 has setting items such as group shooting 314, size, image quality, focus mode, and image stabilization. Size, image quality, and the like are well-known setting items.

[0212] The user selects the setting item of group shooting 314 in the shooting menu 311 by tapping or other operations on the menu screen, and selects the ON button, thereby setting the group shooting mode as the shooting mode. When the OFF button is selected, the group shooting mode is cancelled. Turning the group shooting mode on / off may be achieved by a dedicated hardware button. Furthermore, the imaging device 1 may automatically switch the group shooting mode to the OFF state when the power is turned off while the group shooting mode is on. Furthermore, the imaging device 1 may automatically switch the group shooting mode to the OFF state when a predetermined time has elapsed since the group shooting mode was turned on or since an image was shot in the shooting mode.

[0213] [Effects, etc.] As described above, according to the second embodiment, the user can easily associate multiple images by switching the shooting mode at his / her own discretion. The second embodiment corresponds to a form in which the judgment regarding the shooting date and time in the second selection method described above is automated by using a specific shooting mode.

[0214] (Embodiment 3) An imaging device and a display device according to the third embodiment of the present invention will be described with reference to FIG. 32. Differences between the configuration of the third embodiment and the configuration of the first embodiment will be described below. The imaging device 1 according to the third embodiment has a function of displaying a captured image in association with a monitor image displayed on the display screen in the shooting mode. This function realizes an assist function that assists the user in the shooting operation, including the determination of the shooting direction, etc. In particular, the third embodiment also uses the group shooting mode according to the second embodiment. During the group shooting mode as the shooting mode, the monitor image is displayed, and the captured image is displayed in association with the second image.

[0215] [Shooting monitor] FIG. 32 shows the display screen 404 of the EVF 403 in FIG. 4 in the first monitor mode in the third embodiment. The display control unit 12 displays a monitor image in the shooting mode on the display screen 404, as well as an associated image. In the first monitor mode, the monitor image is regarded as the background area 50 and the first image, and a figure of the second image is displayed on the monitor image. The imaging device 1 displays a monitor image 411 according to the shooting direction and angle of view during shooting, corresponding to the entire background area 50 of the display screen 404. The center point p0 of the display screen 404 and the center point of the monitor image 411 coincide with each other.

[0216] The user switches to group shooting mode and starts shooting. The imaging device 1 switches to group shooting mode in response to a predetermined operation. As in the second embodiment, the imaging device 1 associates images captured in the group shooting mode as a group, and stores information on the associated second images in the storage unit 102 or the like.

[0217] The display control unit 12 determines a second image to be associated with the monitor image 411 as a first image. In particular, a second image captured during group shooting mode is associated with the monitor image 411. The display control unit 12 determines conditions related to the shooting direction and angle of view for the monitor image and the second image, and decides whether to display the second image. The conditions at that time can be the same as those in the first embodiment. For example, the display control unit 12 uses the first condition, and determines the second image to be displayed if the shooting direction (θ, φ) of the second image is included within the angle of view range of the monitor image 411.

[0218] The display control unit 12 superimposes and displays a figure representing the second image to be displayed within the area of ​​the monitor image 411. In this example, the figure is a mark. Point ps1 indicates the shooting direction of image s1, which is the second image, and a mark 321 representing image s1 is displayed. Point ps2 indicates the shooting direction of image s2, which is the second image, and a mark 322 representing image s2 is displayed. The user can easily recognize the presence or absence and position of a captured image within the angle of view of the monitor image 411 by the mark in the monitor image 411. Based on this recognition, the user can determine a suitable shooting direction, angle of view, etc. for a new image to be captured.

[0219] FIG. 33 shows an example of a display screen of the second monitor method as a modified example. In the second monitor method, a monitor image 412 is displayed as a first image in the background area 50 of the display screen 404, and a figure of the second image is displayed in and around the monitor image 412. A monitor image 412 smaller in size than the display screen 404 is displayed at point p0 at the center of the display screen 404. The size of the monitor image 412 may be changed according to the state of the angle of view, etc. The display control unit 12 similarly determines the association and conditions of the second image with respect to the monitor image 412, and displays the figure of the second image. The display control unit 12 uses, for example, the second condition, and when a part of the shooting direction or angle of view of the second image is included within a threshold range extended outside the angle of view range of the monitor image 412, the second image is made the display target. In this example, the shooting direction of the image s3, which is the second image, is at point ps3, and a transparent frame 331 representing the image s3 is displayed. Furthermore, at a point not shown on the background region 50 outside the display screen 404, there is an image s4 which is a second image, and a part of a transparent frame 332 representing the image s4 is displayed within the display screen 404. The image s4 is a wide-angle image. The transparent frame near the monitor image 412 allows the user to easily recognize the presence or absence, position, and angle of view of previously captured images near the angle of view of the monitor image 412, and allows the user to determine a suitable shooting direction and angle of view for a new image.

[0220] Also, after deciding the shooting direction etc. in the display state of the monitor image 412 as shown in Fig. 33, when shooting is executed, a transition may be made to a state in which the monitor image 411 is displayed over the entire display screen 404 as shown in Fig. 32 based on a predetermined input operation. Also, in response to a predetermined input operation, the display and non-display of the figure of the second image may be switched. Also, whether or not to display the associated image on the monitor image in this way may be changed according to a user setting. A display mode switching button etc. may be provided within the display screen of the monitor image.

[0221] Also, a command to re-shoot may be given by selecting a figure representing a captured second image displayed in association with a monitor image on the display screen 404 by a predetermined operation such as tapping or pinching. The imaging device 1 enlarges the second image for which re-shooting has been instructed and displays it as a new monitor image on the display screen 404, and automatically performs zoom control to adjust the focus, etc. The imaging device 1 performs shooting with the shooting direction and angle of view in that state.

[0222] [Effects, etc.] As described above, according to the third embodiment, the assist function allows the user to easily recognize the presence or absence and position of a captured image on the monitor image, and the user can easily determine a suitable shooting direction and angle of view for the next image to be captured by referring to the captured image. The user can, for example, select a shooting direction and angle of view that have not yet been captured and capture the image.

[0223] The following are examples of modified examples of the third embodiment. As a modified example, the same as above can be applied to the display screen 401 of a touch panel. Also, as a modified example, it is possible to realize a mode other than the combined use with the group shooting mode, and while the monitor image is being displayed, a process of confirming the association may be performed in the same manner as in the first embodiment. Also, the association between the monitor image and the second image may be a temporary association, and the association may be automatically released after the display of the monitor image is terminated.

[0224] (Embodiment 4) An imaging device and a display device according to a fourth embodiment of the present invention will be described with reference to Figs. 34 and 35. The following describes the differences between the configuration of the fourth embodiment and the configuration of the first embodiment. The fourth embodiment has an image editing function. This editing function allows the user to manually set and edit the association of multiple images as determined by the user. The user can use the editing function to add, delete, or change images to be associated on the display screen. The editing function also has a function of generating a composite image by trimming.

[0225] [Editing function (1)] FIG. 34 shows an example of the second screen in the second display mode of the imaging device 1. The following shows the case of the imaging device 1 having an editing function, but the same applies to the case of the display device 2 having an editing function. Depending on the user's operation, it is possible to switch between an ON state in which editing is enabled and an OFF state in which editing is disabled. In the ON state, the imaging device 1 variably sets the association of the second image with respect to the first image, the display position of the figure of the second image, etc. on the second screen in the second display mode based on the user's input operation. The control unit 101 saves and manages the display position of the figure of the second image as one of the metadata 22 or management information 21. When playing and displaying, the control unit 101 refers to the information to determine the display position of the figure.

[0226] In this example, a case where a transparent frame is displayed as a figure of the second image is shown. First, the imaging device 1 automatically determines and displays the display positions of the figures of the first image and the second image on the background area 50, as in the first embodiment. For example, a thumbnail of the figure 341 of the first image is displayed at the center point p0. Also, as the second image, a transparent frame of the figure 342a of a certain narrow-angle image is displayed at point p2a. When the user wants to change the display position of the figure of the second image, the user touches the figure, for example, to move it to a desired position. The control unit 101 updates the display state so as to move the display position of the figure according to the user operation. In this example, the state after the transparent frame of the figure 342a is moved is shown as the figure 342b at the point p2b after the movement. The control unit 101 saves the position after the movement of the figure of the second image as a new display position.

[0227] The editing functions include a function to add a second image to be associated and a function to cancel the association of the second image. When adding, the user touches a desired position on the background area 50, for example, point p3, presses the add button in the pop-up area, and selects the second image to be added. The control unit 101 displays a figure related to the selected second image at that position, saves the display position, and associates the first image with the second image.

[0228] In the case of cancellation, the user selects the figure of the second image displayed on the background area 50 and specifies cancellation. For example, the user touches the figure 344 at point p4 and presses the cancellation button in the pop-up area. The control unit 101 cancels the association between the second image, for which cancellation has been specified, and the first image, and erases the display of the figure.

[0229] [Editing function (2)] Furthermore, as an editing function, it has a function of generating a composite image by trimming. This function makes it possible to generate a panoramic image having a larger angle of view than the original image, or an image having a smaller angle of view than the original image. First, as in the first embodiment, the imaging device 1 obtains image files 24 and the like of a plurality of captured images. The imaging device 1 displays the plurality of images in association with each other on the second screen of the above-mentioned second display mode, and turns on the trimming function of the editing function in response to a user operation. Switching is possible by using a display mode switching button, selecting a mode in the playback menu, or the like.

[0230] The upper part of Fig. 35 is a display example of the second screen when the trimming function of the editing function is on, and four images v1, v2, v3, and v4 are displayed. A thumbnail which is a figure 351 representing image v1 is displayed so that the center point pv1 of image v1 which is the first image is placed at point p0 which is the center of background area 50. Around the thumbnail of image v1, a thumbnail which is a figure 352 of image v2 which is the second image is displayed at point pv2, a thumbnail which is a figure 353 of image v3 is displayed at point pv3, and a thumbnail which is a figure 354 of image v4 is displayed at point pv4.

[0231] In this example, the thumbnails of each image have overlapping portions, and the borders of the overlapping portions are indicated by dashed lines. For the overlapping portions, the thumbnail portion of the image on the upper side is displayed according to the vertical arrangement of the multiple images. For example, the first image is displayed in the top layer, and the second image is displayed in the lower layer. The multiple second images are arranged in the upper and lower layers in an order according to the shooting date and time or an order specified by the user. The vertical arrangement relationship of the multiple images can be changed according to the image selection operation, etc. In this example, images v1, v2, v3, and v4 are arranged in order from the top layer. In this state, desired scrolling and zooming are performed according to the user operation. For example, it is assumed that the entire image is moved by scrolling so that point pv1 of image v1 comes to point pv1b.

[0232] The lower side of FIG. 35 shows the second screen after the display state is changed. In this second screen, a trimming frame 350 is displayed on the background area 50, in a layer above the multiple images. The position, vertical and horizontal sizes, rotation angle, etc. of the trimming frame 350 can be changed according to a user operation, for example, a touch and drag operation. The user places the trimming frame 350 in a desired state at a desired position on the image. In this state, the image can be trimmed by a predetermined operation. The predetermined operation is, for example, pressing the shutter button, inputting a command to execute trimming, etc. As a result, the imaging device 1 newly generates image data of a trimmed image including the angle of view range of the image included in the trimming frame 350, and stores it in the storage unit 102, etc.

[0233] When performing trimming, if the angle of view ranges of multiple images are included within the trimming frame 350 as in this example, the imaging device 1 synthesizes the angle of view ranges of the multiple images to generate one trimmed image. When generating the trimmed image, the imaging device 1 reads the image data 23 corresponding to each image (v1, v2, v3) into the memory 133 and synthesizes them by a known image processing such as stitching processing. This results in a composite image corresponding to the trimming frame 350. In this example, a panoramic image having a horizontal angle of view corresponding to the horizontal size of the trimming frame 350 is obtained as the composite image. Not limited to the above example, an image in which a desired angle of view range is extracted from the angle of view range of a wide-angle image can be generated by trimming.

[0234] [Effects, etc.] As described above, according to the fourth embodiment, the editing function allows the user to freely edit the associated display of multiple images in the second display mode so that the images are easy to view and understand. Conventionally, for example, when a user wants to generate a panoramic image, the user needs to perform an operation such as panning the camera while shooting. In contrast, according to the fourth embodiment, the editing function allows the user to easily obtain a desired panoramic image or an image with a desired angle of view while checking multiple images after shooting.

[0235] (Embodiment 5) An imaging device and a display device according to a fifth embodiment of the present invention will be described with reference to Figs. 36 to 38. Differences between the configuration of the fifth embodiment and the configuration of the first embodiment will be described below. The imaging device 1 according to the fifth embodiment has a function of capturing a celestial sphere image as a wide-angle image, and a function of displaying the wide-angle image in association with the first image or the second image.

[0236] [Spherical image] In this specification, a virtual spherical surface 600 in which the positions of the imaging device and the user who is the photographer are set as center points, as in Fig. 7 etc., is referred to as a celestial sphere. In this specification, a celestial sphere image is an image having at least one of a horizontal angle of view and a vertical angle of view of 360°.

[0237] FIG. 36 shows the celestial sphere image in the fifth embodiment in terms of horizontal angle of view. A spherical surface 600 corresponding to the celestial sphere includes a first wide-angle image 6A, a second wide-angle image 6B, and a narrow-angle image 6C. The first wide-angle image 6A is a wide-angle image having a shooting range 361 of a horizontal angle of view of 180° when viewed from the front camera side of the imaging device 1 in a forward shooting direction (for example, facing north). The second wide-angle image 6B is a wide-angle image having a shooting range 362 of a horizontal angle of view of 180° when viewed from the rear camera side in a rear shooting direction (for example, facing south). The narrow-angle image 6C is a narrow-angle image having a shooting range 363 of a horizontal angle of view Ah when viewed from the front camera side in a desired shooting direction (for example, facing north). The first wide-angle image 6A and the second wide-angle image 6B are combined to form a celestial sphere image that is a wide-angle image having a shooting range of a horizontal angle of view of 360°. Although not shown, the same is true for the vertical angle of view Av; for example, the first wide-angle image 6A has a shooting range of 180° in terms of the vertical angle of view.

[0238] [Imaging device] 37 shows a configuration of a portion related to the imaging unit 103 of the imaging device 1. The imaging device 1 has a first imaging unit 103A, a second imaging unit 103B, and a third imaging unit 103C as the imaging unit 103, which are connected to an interface circuit unit 131.

[0239] The first imaging unit 103A is an imaging unit corresponding to the front camera, and includes a drive unit 31a and a lens unit 32a. The lens unit 32a has a first lens as an ultra-wide-angle lens capable of capturing a first wide-angle image 6A in a front shooting range of 180° in horizontal and vertical angles of view (i.e., a hemisphere). The second imaging unit 103B is an imaging unit corresponding to the rear camera, and includes a drive unit 31b and a lens unit 32b. The lens unit 32b has a second lens as an ultra-wide-angle lens capable of capturing a second wide-angle image 6B in a rear shooting range of 180° in horizontal and vertical angles of view (i.e., a hemisphere).

[0240] The third imaging unit 103C has the same configuration as the imaging unit 103 of the first embodiment, and includes a driving unit 31c and a lens unit 32c. The lens unit 32c includes a third lens capable of capturing a narrow-angle image 6C at a set angle of view (e.g., a general angle of view of about 50 degrees) less than 180° in the horizontal and vertical directions on the front side. The angle of view when capturing an image by the imaging unit 103C can be set based on a user operation within a predetermined angle of view range. For example, assume that a narrow-angle image 6C having a horizontal angle of view Ah of 50° is captured.

[0241] The control unit 101 includes an omnidirectional imaging control unit 11A and an imaging control unit 11B. The omnidirectional imaging control unit 11A drives and controls the first imaging unit 103A and the second imaging unit 103B to capture an omnidirectional image. The omnidirectional imaging control unit 11A controls the first wide-angle image 6A in the first imaging unit 103A and the second wide-angle image 6B in the second imaging unit 103B to capture the omnidirectional image at the same timing. The omnidirectional imaging control unit 11A inputs and synthesizes the first wide-angle image 6A and the second wide-angle image 6B via the signal processing unit 132, etc., to capture an omnidirectional image having a 360° angle of view range. Note that the configuration of the imaging unit 103, the control unit 101, etc., including an optical system that can capture an omnidirectional image in this way, can be realized using a known technology.

[0242] Omnidirectional imaging control unit 11A stores image data 23 of the captured omnidirectional image together with metadata 22 including information such as the angle of view as image file 24 in storage unit 102, similar to embodiment 1. Imaging control unit 11B stores image data 23 of the captured narrow-angle image together with metadata 22 including information such as the angle of view as image file 24 in storage unit 102, similar to embodiment 1.

[0243] In the shooting mode of the imaging device 1, it is possible to capture an image using any of the imaging units according to the control of the control unit 101. Based on the control from the control unit 101, the first imaging unit 103A, the second imaging unit 103B, and the third imaging unit 103C can simultaneously capture images at one time. Based on the control from the control unit 101, the first imaging unit 103A, the second imaging unit 103B, and the third imaging unit 103C can also capture images at individual timings. A celestial sphere image can be captured by simultaneous shooting and synthesis of the first imaging unit 103A and the second imaging unit 103B. Also, a hemispherical image of the first wide-angle image 6A or the second wide-angle image 6B can be captured by shooting using one of the first imaging unit 103A and the second imaging unit 103B. Also, simultaneous shooting of one of the first imaging unit 103A or the second imaging unit 103B and the third imaging unit 103C is also possible. In this case, the one wide-angle image and narrow-angle image 6C have the same shooting date and time, and can be automatically associated with each other. The control unit 101 may add information such as a type indicating a celestial sphere image, a hemispherical image, or a narrow-angle image 6C to the metadata 22 or management information 21 of the captured image. The control unit 101 may add association information to the metadata 22 or the like for multiple images captured simultaneously.

[0244] [Imaging device - appearance] FIG. 38 shows the appearance of an implementation example of the imaging device 1. In this example, a case where the imaging device 1 is a smartphone is shown. The left side of FIG. 38 shows the front camera side of the housing 380 of the imaging device 1, and the right side shows the rear camera side. On the front camera side, a first lens 381 is provided at a predetermined position of the housing 380, and a third lens 383 is also provided at a predetermined position. On the rear camera side, a second lens 382 is provided at a predetermined position on the opposite side corresponding to the position of the first lens 381 on the front camera side. Also, on the rear camera side, a display screen 384 of the display device 104, an operation button 385, etc. are provided.

[0245] [Second display mode - second screen] In the fifth embodiment, similarly to the first embodiment and the like, the first image and the second image are displayed in association with each other on the background region 50 on the second screen in the second display mode. At this time, in the fifth embodiment, a celestial sphere image or a hemispherical image can be applied as the first image or the second image. For example, in the case of display using the second screen having the same format as that of FIG. 18, if the first image is a celestial sphere image, a hemispherical portion corresponding to the shooting direction of the celestial sphere image is displayed as it is in the circular background region 50.

[0246] [Effects, etc.] As described above, according to the fifth embodiment, when a celestial sphere image or the like is displayed as a wide-angle image, a plurality of images including related narrow-angle images can be displayed in association with each other, so that the user can easily grasp the relationship between the plurality of images.

[0247] The following modifications of the embodiment are possible. The second imaging unit 103B on the rear camera side may be omitted, and a hemispherical image having a forward angle of view of 180° may be captured using the first imaging unit 103A. When the omnidirectional image is displayed on the second screen, it may be converted into a rectangular area having a horizontal angle of view of 360° as in FIG. 16, instead of a circular image, and displayed. As a modification, the omnidirectional image may be displayed in the background area 50 as a reduced circular shape.

[0248] Although the present invention has been specifically described based on the embodiment, the present invention is not limited to the above embodiment and can be modified in various ways without departing from the gist of the present invention. The functions of the present invention may be realized in part or in whole by hardware such as an integrated circuit, or by software program processing. [Explanation of symbols]

[0249] 1...imaging device, 2...display device, 3...server, 4...communication network, 11...imaging control unit, 12...display control unit, 13...touch detection unit, 14...setting unit, 15...association unit, 21...management information, 22...metadata, 23...image data, 24...image file, 50...background area, 61, 62, 63...figure, 101...control unit, 102...memory unit, 103...imaging unit, 104...display device, 105...operation input unit, 106...recording unit, 107...recording medium, 108...communication interface unit, 111...shooting direction measurement unit, 112...internal clock, 113...position detection unit, 131...interface circuit unit, 132...signal processing unit, 133...memory, 401...display screen.

Claims

1. A display method comprising: An imaging step of capturing an image to obtain data; a display step of reproducing and displaying an image based on the data; Equipped with In the imaging step, first information is acquired, the first information including the data obtained by imaging, at least one of an azimuth angle and an elevation angle as an imaging direction, and at least one of an angle of view or information related to an angle of view that allows the angle of view to be calculated; When a second shooting direction or a part of an angle of view of a second image is included within a range of a first angle of view of a first shooting direction of a first image or within a threshold angle of view range extended therefrom, the second image is associated with the first image; In the display step, displaying the first image or a first figure representing the first image, and displaying the second image associated with the first image or a second figure representing the second image in a positional relationship corresponding to the shooting direction; a display size of the first image or the first figure is displayed so as to correspond to the angle of view of the first image; a display size of the second image or the second figure is displayed so as to correspond to the angle of view of the second image; When there are a plurality of second images associated with the first figure, a representative figure is displayed as the second figure; displaying information about the associated second images in response to an operation on the graphic representing the representative; A display method comprising:

2. A display method comprising: An imaging step of capturing an image to obtain data; a display step of reproducing and displaying an image based on the data; Equipped with In the imaging step, first information is acquired, the first information including the data obtained by imaging, at least one of an azimuth angle and an elevation angle as an imaging direction, and at least one of an angle of view or information related to an angle of view that allows the angle of view to be calculated; When a second shooting direction or a part of an angle of view of a second image is included within a range of a first angle of view of a first shooting direction of a first image or within a threshold angle of view range extended therefrom, the second image is associated with the first image; In the display step, displaying the first image or a first figure representing the first image, and displaying the second image associated with the first image or a second figure representing the second image in a positional relationship corresponding to the shooting direction; a display size of the first image or the first figure is displayed so as to correspond to the angle of view of the first image; a display size of the second image or the second figure is displayed so as to correspond to the angle of view of the second image; When the first figure is present, displaying information about a plurality of images including the first image and the second image as a first information display mode; as a second information display mode, displaying the first image or the first figure selected in the first information display mode, and displaying the second image or the second figure associated with the first image; As a third information display mode, the first image or the second image selected in the second information display mode is displayed in detail; Displaying information based on a user operation-based information display mode; A display method comprising:

3. The display method according to claim 1 or 2, The background region is a rectangular or circular region corresponding to a range of angles of view of 180° or more on the surface of the celestial sphere, In the display step, an image of the first image projected onto the spherical surface, reduced, or enlarged is displayed on the background area at a position corresponding to the shooting direction of the first image, and an image of the second image projected onto the spherical surface, reduced, or enlarged is displayed at a position corresponding to the shooting direction of the second image. A display method comprising:

4. The display method according to claim 1 or 2, a storage step of storing the data and the first information, The information relating to the angle of view includes an electronic zoom magnification, a focal length, and a size or model of an image sensor, When the first information includes the angle-of-view-related information, the angle of view is calculated from the angle-of-view-related information, and the angle of view is stored in the first information. A display method comprising:

5. The display method according to claim 1 or 2, The second figure has a reduced image, an enlarged image, or a predetermined figure including a mark of the second image, a transparent frame, or an attribute value of the second image. A display method comprising:

6. The display method according to claim 1 or 2, creating and storing the first information as metadata or management information of the data in the imaging step; When the second video is associated with the first video in the association, association information between the first video and the second video is created and stored in the metadata or the management information; In the display step, the association information is confirmed, and the first image and the second image are displayed. A display method comprising:

7. The display method according to claim 1 or 2, creating and storing the first information as metadata or management information of the data in the imaging step; In the display step, the association is performed, and when the second image is associated with the first image, the first image and the second image are displayed. A display method comprising:

8. The display method according to claim 1 or 2, Switch between the first and second shooting modes, The association is not performed for the image captured in the first imaging mode, The association is performed on a group of a plurality of images captured in the second imaging mode. A display method comprising:

9. The display method according to claim 1 or 2, In the imaging step, a monitor image being captured is displayed as the first image, and when there is a second image associated with the first image, the second figure is displayed. A display method comprising:

10. The display method according to claim 1 or 2, In the imaging step, location information indicating a shooting location is created and stored as metadata or management information of the data; In the association, when the position information of the second video is included within a predetermined position range with respect to the position information of the first video, the second video is selected as a candidate video for the association. A display method comprising:

11. The display method according to claim 1 or 2, In the imaging step, information including the image capturing date and time is created and stored as metadata or management information of the data; In the association, when the shooting date and time of the second video is included within a predetermined time range with respect to the shooting date and time of the first video, the second video is selected as a candidate video for the association. A display method comprising:

12. The display method according to claim 1 or 2, creating and storing the set character string as metadata or management information of the data in the imaging step or the display step; In the association, when the character string in the second video includes the same character string as the character string in the first video, the second video is selected as a candidate video for the association. A display method comprising:

13. The display method according to claim 1 or 2, In the imaging step or the display step, a type of a subject is determined by a recognition process; In the association, when the object type of the second image includes the same object type as the object type of the first image, the second image is selected as a candidate image for the association. A display method comprising:

14. The display method according to claim 1 or 2, The imaging step can capture a wide-angle image having an angle of view of 180° or more and 360° or less, and at least one of the first image and the second image is the wide-angle image. A display method comprising:

Images