IMAGE PROCESSING APPARATUS AND IMAGE DISPLAY METHOD

Abstract

To allow for appropriately and efficiently displaying an image corresponding to a location.SOLUTION: An image processing device displays an image 100 captured at a location specified by a user's operation in response to a user operation. When the user performs a predetermined operation on the image 100, the image processing device displays a situation selection icon 102 for selecting a shooting situation of the same location as that of the image 100. When the user operates the situation selection icon 102 to select a shooting situation, the image processing device switches the display to an image 104 captured in the selected situation.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image display method for displaying an image on a display. [Background technology]

[0002] Advances in networks and information processing devices have made it easier to access necessary information regardless of time or place. For example, a map application allows you to immediately view information about facilities at a specified location on the map and images of that location, making it possible to understand the information in relation to the location information inherent in the map. In addition, by displaying images in any direction using 360° images, or by sequentially switching the display to images taken from the viewpoint in the direction of travel, it is possible to create a sense of realism as if you are looking around the world of the image or walking through it. Summary of the Invention [Problem to be solved by the invention]

[0003] In order to provide the above-mentioned environment stably, it is desirable to prepare as many captured images as possible regardless of the location. For this reason, measures are sometimes taken in which users share images via a network by widely soliciting image data from the general public or allowing individual users to associate them with locations on a map. However, as the number of images increases, images that are completely different in appearance even in the same place are treated equally, which increases the possibility of confusing the viewer or requiring a great deal of effort to reach the desired image. In particular, when the displayed images are switched sequentially, as in the above-mentioned technology of moving the viewpoint of the omnidirectional image, the difference in appearance of the images before and after the switch may cause a loss of realism or make it difficult to recognize the same place.

[0004] The present invention has been made in view of the above problems, and has a general purpose to provide a technique capable of suitably and efficiently displaying an image corresponding to a location. [Means for solving the problem]

[0005] One aspect of the present invention relates to an image processing device, which includes an input information acquisition unit that accepts a user operation related to an image to be displayed on a display device, and an image display unit that displays one of a plurality of images on the display device in response to the user operation and further displays a situation selection icon for selecting a shooting situation, and when the input information acquisition unit accepts an operation on the situation selection icon, displays another image captured in the selected situation on the display device, and is characterized in that when a user operation is performed to continuously point within a range regarded as the same position as the position where the situation selection icon is displayed for a predetermined period of time or more, the image display unit displays a situation selection icon representing another option, or expands the area of ​​the situation selection icon to increase the number of options displayed.

[0006] Another aspect of the present invention relates to an image display method, the image display method including a step of an image processing device receiving a user operation related to an image to be displayed on a display device, a step of displaying one of a plurality of images on the display device in response to the user operation, a step of displaying a situation selection icon for selecting a shooting situation, and a step of receiving an operation on the situation selection icon and displaying another image captured in the selected situation on the display device, the step of displaying the situation selection icon being characterized in that, when a user operation is performed to continuously point within a range regarded as the same position as the position where the situation selection icon is displayed for a predetermined period of time or more, a situation selection icon representing another option is displayed, or an area of ​​the situation selection icon is expanded to increase the number of options represented.

[0007] Any combination of the above components, and any transformation of the present invention into a method, device, system, computer program, or recording medium having a computer program recorded thereon, are also effective aspects of the present invention. Effect of the Invention

[0008] According to the present invention, an image corresponding to a location can be displayed preferably and efficiently. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image display system to which the first embodiment can be applied. [Diagram 2] 1 is a diagram showing an internal circuit configuration of an image processing device in accordance with a first embodiment. [Diagram 3] FIG. 2 is a diagram showing a configuration of functional blocks of an image processing device and an image providing server according to the first embodiment. [Figure 4] 4 is a diagram showing an example of a functional block configuration of an image providing server when constructing an image database in the first embodiment. FIG. [Diagram 5] FIG. 2 is a diagram showing an example of a screen displayed on a display device in the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of a screen when a selection of a shooting situation is received in the first embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of a database structure stored in an image database in the first embodiment. [Figure 8] 4 is a diagram illustrating an example of a structure of classification information of taking conditions used in the first embodiment. FIG. [Figure 9] 10 is a flowchart showing a processing procedure in which the image processing device displays a captured image corresponding to a location in cooperation with the image providing server in the first embodiment. [Figure 10] FIG. 11 is a diagram illustrating an example of the structure of data stored in an image database in the second embodiment. [Figure 11] 11 is a diagram for explaining data stored in an image database when a position in a hierarchy is further associated with the image data column in the image data information of FIG. 10. FIG. [Figure 12] FIG. 13 is a diagram showing an example of an image displayed on a display device in the second embodiment; [Figure 13] FIG. 13 is a diagram showing another manner for accepting an operation for selecting a floor number in the present embodiment. [Figure 14] 13 is a flowchart showing a processing procedure in which an image processing device displays a captured image corresponding to a location in cooperation with an image providing server in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] First embodiment Fig. 1 shows an example of the configuration of an image display system to which this embodiment can be applied. The image display system 1 includes image processing devices 10a, 10b that display images in response to user operations, and an image providing server 20 that provides image data used for display. Input devices 14a, 14b for user operations and display devices 16a, 16b for displaying images are connected to the image processing devices 10a, 10b, respectively. The image processing devices 10a, 10b and the image providing server 20 establish communication via a wide area communication network such as the Internet, or a local network such as a LAN (Local Area Network). Alternatively, the image data may be provided from the image providing server 20 to the image processing devices 10a, 10b via a recording medium.

[0011] The image processing devices 10a and 10b may be connected to the display devices 16a and 16b and the input devices 14a and 14b either by wire or wirelessly. Alternatively, two or more of these devices may be integrally formed. For example, in the illustrated example, the image processing device 10b is integrated with the input device 14b, which is a touch pad covering the screen of the display device 16b, to form a mobile terminal. The display device 16a is not limited to a flat display, and may be a wearable display such as a head-mounted display, a projector, or the like.

[0012] In this way, the external shapes and connection forms of the illustrated devices are not limited. The number of image processing devices 10a, 10b connected to the image providing server 20 is also not limited. Hereinafter, the image processing devices 10a, 10b will be collectively referred to as the image processing device 10, the input devices 14a, 14b as the input device 14, and the display devices 16a, 16b as the display device 16.

[0013] The image processing device 10 displays on the display device 16 an image corresponding to a user operation on the input device 14. Specifically, at least an image taken at a location specified by the user among images taken at various locations is displayed. The display purpose is not particularly limited here, and in order to achieve the purpose, the image processing device 10 may simultaneously perform general processing such as progressing an electronic game or displaying images from an Internet site using a browser. In this embodiment, an electronic map application, which has a high affinity as a means for specifying a location, will be described as an example.

[0014] The input device 14 may be any one or a combination of general input devices such as a controller, keyboard, mouse, touch pad, joystick, etc., and in this embodiment, in particular, the input device 14 accepts from the user a location specification, a movement of the viewpoint, a change of line of sight, and a selection of the image capture conditions. The display device 16 may be a general display such as a liquid crystal display, a plasma display, an organic EL display, etc., and displays the image output from the image processing device 10. As described above, the display device 16 may be a head-mounted display or a projector.

[0015] The image providing server 20 establishes communication with the image processing device 10 and transmits captured image data in response to a request from the image processing device 10. For this purpose, the image providing server 20 holds a database in which captured image data is associated with a shooting location and shooting conditions. It then accepts a designation of a shooting location and shooting conditions from the image processing device 10 and transmits data of the matching image. Here, the "shooting conditions" may be, for example, a time situation such as the date and time of shooting, time period, day of the week, month, season, year, or era, or a subject situation such as the name of an event taking place at the time of shooting, event classification, number of people in the image, or weather. However, the specific items are not particularly limited as long as they are factors that change the appearance or impression of the place.

[0016] 2 shows the internal circuit configuration of the image processing device 10. The image processing device 10 includes a CPU (Central Processing Unit) 23, a GPU (Graphics Processing Unit) 24, and a main memory 26. These components are connected to each other via a bus 30. An input / output interface 28 is further connected to the bus 30. The input / output interface 28 is connected to a communication unit 32, which is made up of a peripheral device interface such as USB or IEEE1394, or a network interface for wired or wireless LAN, and which establishes communication with the image providing server 20, a storage unit 34 such as a hard disk drive or nonvolatile memory, an output unit 36 ​​which outputs data to the display device 16, an input unit 38 which inputs data from the input device 14, and a recording medium drive unit 40 which drives a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory.

[0017] The CPU 23 executes an operating system stored in the storage unit 34 to control the entire image processing device 10. The CPU 23 also executes various programs read from a removable recording medium and loaded into the main memory 26, or downloaded via the communication unit 32. The GPU 24 has a geometry engine function and a rendering processor function, performs drawing processing according to drawing commands from the CPU 23, and outputs the results to the output unit 36. The main memory 26 is composed of a RAM (Random Access Memory), and stores programs and data required for processing. The image providing server 20 may have a similar internal circuit configuration.

[0018] Fig. 3 shows the functional block configuration of the image processing device 10 and the image providing server 20. Each functional block shown in Fig. 3 and Fig. 4 described later can be realized in terms of hardware by the various circuits shown in Fig. 2, and in terms of software by a program that performs various functions such as an image analysis function, an information processing function, an image drawing function, and a data input / output function, which are loaded from a recording medium into a main memory. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by only hardware, only software, or a combination of these, and are not limited to any one of them.

[0019] The image processing device 10 includes an input information acquisition unit 50 that acquires information related to user operations, a data request unit 52 that requests data corresponding to the user operations from the image providing server 20, a data acquisition unit 54 that acquires data transmitted from the image providing server 20, and an image display unit 56 that generates a display image and displays it on the display device 16. The input information acquisition unit 50 is realized by the input unit 38 in FIG. 2, the CPU 23, etc., and acquires information related to user operations from the input device 14. Specifically, as described above, in addition to designation of the shooting location and shooting conditions of the image, the input information acquisition unit 50 acquires operation contents such as start and end of display processing, change of line of sight to the panoramic image, and movement of the viewpoint. When associated with a map application, the input information acquisition unit 50 also acquires contents of various operations related to map display.

[0020] The data request unit 52 is realized by the CPU 23, the communication unit 32, and the like in FIG. 2, and requests image data corresponding to a user operation from the image providing server 20. The data acquisition unit 54 is realized by the CPU 23, the communication unit 32, the main memory 26, and the like in FIG. 2, and acquires image data transmitted in response to a request from the image providing server 20. The data acquisition unit 54 may include a recording medium drive unit 40 and read image data from a recording medium. The data acquisition unit 70 may also acquire information on options indicating what kind of images of what shooting conditions can be selected for a specified location from the image providing server 20. The data acquisition unit 70 decodes and expands the acquired image data as necessary.

[0021] The image display unit 56 is realized by the CPU 23, the GPU 24, the main memory 26, etc. in FIG. 2, generates a display image including the image acquired by the data acquisition unit 54, and outputs it to the display device 16. In detail, the image display unit 56 includes an icon display unit 64, an image switching unit 66, and a viewpoint change processing unit 68. The icon display unit 64 displays at least a situation selection icon for selecting the shooting situation of the image to be displayed. By superimposing the situation selection icon on the image currently being displayed, it is possible to switch the display to an image of a different shooting situation while keeping the image being displayed. Note that, since the shooting situations that can be selected vary depending on the image prepared for the specified location, the icon display unit 64 determines the configuration of the situation selection icon based on the information of the options transmitted from the image providing server 20 as described above.

[0022] In addition, the icon display unit 64 displays a movement icon for moving the viewpoint in the full-sky image display, superimposed on the image being displayed as necessary. The image switching unit 66 switches the display target to at least an image of a shooting situation selected by an operation on the situation selection icon. The image switching unit 66 also appropriately performs display of thumbnail images, enlarged display of an image selected from the displayed thumbnail images, display of a map or information of a searched location, etc. In the full-sky image display, the viewpoint change processing unit 68 switches the displayed image in response to an operation on the movement icon so that the viewpoint moves in a corresponding direction in the image world, and changes the field of view of the displayed target so that the corresponding direction can be seen in response to an operation on the up, down, left, right, or diagonal direction designation operation on the screen.

[0023] A general technique can be used for the control of display in response to changes in viewpoint or line of sight, which is performed by the viewpoint change processing unit 68. However, in this embodiment, when switching images by moving the viewpoint, an image with the same shooting conditions as before the switching is preferentially selected and displayed. Alternatively, the user is allowed to select the shooting conditions on the spot. Note that if the display target is not a panoramic image, the function of the viewpoint change processing unit 68 can be omitted.

[0024] The image providing server 20 includes a request acquisition unit 58 that acquires a request for image data from the image processing device 10, a data transmission unit 62 that transmits image data corresponding to the request to the image processing device 10, and an image database 60 that stores the image data in association with the shooting location and shooting conditions. The request acquisition unit 58 is realized by the communication unit 32, CPU 23, etc. in Fig. 2, and acquires a data request for an image specifying the location and shooting conditions from the image processing device 10.

[0025] The data transmission unit 62 is realized by the CPU 23, the communication unit 32, etc., and extracts image data and option information corresponding to the request from the image database 60, and transmits them to the image processing device 10. When realizing a map application, data generally provided as a map application, such as map images and facility information, may be included as the object of a request from the image processing device 10 or the object of transmission from the image providing server 20. In this case, a database for such data is prepared separately, but is not shown in FIG. 3.

[0026] The image database 60 is realized by the storage unit 34 in Fig. 2, and stores data of images taken at various locations in association with the location and circumstances of the images. As described above, the circumstances of the images can be considered to be the time circumstances and the circumstances of the subject. As options for selecting the former, time ranges of various granularities are prepared, such as the time of day when the image was taken, the month or season when the image was taken, and the year when the image was taken. As options for selecting the latter, classifications and number ranges of various granularities are prepared for events that were held, the number of people, the weather, and the like.

[0027] Fig. 4 shows an example of the functional block configuration of the image providing server 20 when constructing the image database 60. The functional blocks of the image providing server 20a shown in the figure may be included in the image providing server 20 in Fig. 3, or may be different devices. The image providing server 20a includes a data acquisition unit 70 that acquires data of captured images from an external device, an attribute acquisition unit 72 that acquires attributes of the captured images such as the shooting location and shooting conditions, and the image database 60 shown in Fig. 3.

[0028] The data acquisition unit 70 is realized by the CPU 23, the communication unit 32, the input unit 38, etc. in Fig. 2, and acquires captured image data from an external device such as the image processing device 10. For example, a general user may specify a location on a map for a map application executed in the image processing device 10, and upload data of an image that he or she has captured at that location. In this case, the data acquisition unit 70 acquires the uploaded data together with information about the specified location via a network. The source of image data may also be an imaging device or information processing device directly connected to the image providing server 20a, or another image providing server connected via a network.

[0029] The attribute acquisition unit 72 acquires information related to the location and circumstances of the image by a predetermined means. In the case of an image captured by a digital camera, the date and time of capture, location information at the time of capture, and the like are recorded as additional information such as Exif (Exchangeable Image File Format). Therefore, the attribute acquisition unit 72 reads the additional information from the image data acquired by the data acquisition unit 70 and acquires necessary information. Alternatively, the attribute acquisition unit 72 may acquire information related to the attributes inputted in the image processing device 10 by the user of the image providing source. Alternatively, the attribute acquisition unit 72 may acquire information such as the number of people in the image and the weather by analyzing the captured image using a general method. Such information may be directly confirmed and inputted by the user operating the image providing server 20a.

[0030] The attribute acquisition unit 72 associates the information related to the photography location and photography conditions thus acquired with the photographed image data, and stores the associated data in the image database 60. By carrying out such association for data of images with no uniformity, such as those collected from general users, and storing the data in a database, it is possible to easily extract, from the abundant photographed images, the images most suitable for the viewing conditions.

[0031] For example, when a panoramic image of a certain place is displayed and an attempt is made to move in the direction of travel in the image world by moving the viewpoint, if there is insufficient image data and no images taken from the corresponding viewpoint, it is impossible to switch the display. On the other hand, even if there are a wealth of images taken from the corresponding viewpoint, if one of them is displayed out of context, even a slight movement is likely to cause a large change in the appearance of the displayed image, which can cause discomfort and confusion.

[0032] As described above, by creating an image database 60 in association with attributes other than location, and extracting images of the same shooting conditions in response to the operation of moving the viewpoint, it is possible to continuously display images with a consistent feel, allowing the user to appreciate and confirm the atmosphere and structure of the place without confusion. Also, by allowing the user to select the shooting conditions, it is possible to continuously present images suited to the individual purpose of the user. When the image data is provided by a recording medium, it is stored in the recording medium in the form of the image database 60.

[0033] 5 shows an example of a screen displayed on the display device 16 in this embodiment. Note that the changes in the display screen described below are actually made through a process in which the input information acquisition unit 50 of the image processing device 10 acquires the contents of a user operation, and the data request unit 52 makes a request to the image providing server 20, and as a result, the data acquisition unit 54 acquires the corresponding image data, but the description will be omitted as appropriate.

[0034] 13A shows a screen in which an electronic map is displayed by a map application. In this example, the display screen includes a map field 80, a search field 82, and an information field 84. When a user inputs an address or facility name into the search field 82, the image switching unit 66 of the image display unit 56 displays a map of the surrounding area in the map field 80. When a facility name is input, or when a facility is designated by operating a cursor 88 on a displayed map, the image switching unit 66 displays text information such as basic information about the facility and the URL of its homepage in the information field 84.

[0035] In the illustrated example, the name of a facility, "City Center," is entered in search field 82, and a map of the surrounding area is displayed in map field 80, with basic information displayed in information field 84. Furthermore, image switching unit 66 displays thumbnails 86a, 86b of images taken near the facility. The images displayed as thumbnails 86a, 86b are extracted from image data stored in image database 60 using the location of the facility as a key. When the user designates one of these images with cursor 88 or the like, a transition to the captured image display screen of (b) is made. In this example, the captured image display screen includes an image field 90 and a thumbnail field 92.

[0036] The image switching unit 66 displays an enlarged image of the thumbnail 86a selected by the user on the map display screen (a) in the image field 90, and displays thumbnails of other images taken at the same location in the thumbnail field 92. Note that the "same location" here is not limited to the scale of the area, and may include the same store or room, as well as a building, facility, or site. When one of the thumbnails displayed in the thumbnail field 92 is designated, the image switching unit 66 switches the display in the image field 90 to an enlarged image of that thumbnail.

[0037] When a panoramic image is displayed in the image field 90, the icon display unit 64 of the image display unit 56 displays a movement icon 94 for moving the viewpoint on the image. When the movement icon 94 is pointed to by a cursor or the like, the viewpoint change processing unit 68 moves the viewpoint in the direction of the arrow indicated by the icon (toward the back in the figure) and switches the display to the panoramic image captured there. When an operation is performed to point to an arbitrary position on the screen and slide the screen, the viewpoint change processing unit 68 moves the field of view in the opposite direction so that the image moves in conjunction with the slide. This allows the user to view the specified location from any viewpoint or line of sight. The screen configuration is not limited to that shown in FIG. 5, and the process procedure for displaying the captured image is not limited either.

[0038] In this way, in an environment where a location is selected and images taken there are displayed, in addition to appreciating the images themselves, the purpose is often to understand the state of the location in relation to facilities and locations. However, as described above, the more images taken of the location, the more various images can be viewed, but it is likely to be difficult to access the desired image. In addition, when moving the viewpoint using the movement icon 94, if the state of the image changes significantly before and after the movement, it is considered that the sense of movement is not obtained and it is difficult to grasp the space.

[0039] Therefore, as described above, the captured images in a situation suited to the user's purpose are extracted and displayed. FIG. 6 shows an example of a screen when the selection of the capture situation is accepted. The image 100 shown in (a) may be either a thumbnail displayed on the map display screen in (a) or the captured image display screen in (b) of FIG. 5, or an enlarged image displayed on the captured image display screen in (b). Alternatively, it may be a captured image displayed on a homepage or the like, unrelated to a map. When such an image 100 is displayed and the user performs a predetermined operation on the image 100, the icon display unit 64 displays a situation selection icon 102 representing a selection of the capture situation superimposed on the image 100.

[0040] In general image display technology, it is often set up so that a displayed image 100 can be selected and some action can be initiated by pointing at the image with a cursor or the like. For example, in the example of Fig. 5, as described above, pointing to thumbnail 86a causes an enlarged image to be displayed in image field 90. Alternatively, a pointing operation on an image displayed in image field 90 can be used to move the viewpoint or change the line of sight.

[0041] In order to be compatible with such general operations, the icon display unit 64 displays the situation selection icon 102 at the indicated position on the condition that an indication within a predetermined range that can be regarded as the same position on the image 100 continues for a time equal to or longer than a predetermined threshold value t1. Here, the "indication" may be an operation of a displayed cursor or a touch operation on a touch pad. In the former case, the duration of the indication corresponds to, for example, the press time of a button on a controller or a mouse. In the latter case, the duration of the indication corresponds to the duration of the contact. These operations are generally called "long press". However, the operation for calling up the situation selection icon 102 is not limited to a long press, and for example, another GUI (Graphical User Interface) may be displayed and the situation selection icon 102 may be displayed in response to an operation on the GUI.

[0042] As an example, the illustrated situation selection icon 102 is a circular icon with daytime and nighttime ("day", "night") and four seasons ("spring", "summer", "autumn", "winter") as shooting situation options, each of which is assigned an area. In this case, for example, when a long press is determined, the situation selection icon 102 is displayed with the position as the center, and if the designated position is slid to any of the areas, it is determined that the option of the area including the arrival point has been selected. Also, if the designated position does not move even when the situation selection icon 102 is displayed and this state continues for a time equal to or longer than a predetermined threshold value t2, a situation selection icon 102 representing another option may be displayed.

[0043] Instead of displaying a new icon 102 for selecting a situation, the icon may be expanded according to the duration of the instruction to increase the number of options. For example, when the icon 102 for selecting a situation is displayed for the first time, only the options arranged in the central area (the options "day" and "night" in the illustrated example) are displayed. If the instruction continues for a time equal to or longer than a threshold value t2 without the indicated position moving even after the option is displayed, the options in the outer area (the options "spring", "summer", "autumn", and "winter" in the illustrated example) may appear. When one of the shooting situations is selected, the image switching unit 66 switches the display to an image taken at the same location in the selected shooting situation, as shown in (b).

[0044] For example, by switching from an image taken at night to an image taken during the day, the number of people in the photograph increases as shown in the figure. Among the photographing situation options shown by the situation selection icon 102, options for which no image data exists are indicated as not selectable by being displayed in a different color from the other options. The image after the switch may be displayed in the same position as the image before the switch, or may be displayed in a different position depending on the purpose. For example, if the image 100 is the thumbnail 86a shown in the information column 84 on the map display screen in FIG. 5(a), the image of the selected photographing situation may be enlarged and displayed in the image column 90 on the photographed image display screen in (b).

[0045] In this case, when the user moves the viewpoint by operating the movement icon 94, all images of the destination that are switched and displayed by the viewpoint change processing unit 68 may be those of the initially selected shooting situation. This unifies the shooting situations of the images that are switched sequentially in response to the movement, and as described above, it is possible to avoid a situation in which a completely different image is displayed due to a slight movement, causing confusion. On the other hand, even if the image 100 in FIG. 6 is an image that is enlarged and displayed in the image field 90, by calling the situation selection icon 102 and switching the display to an image of the selected shooting situation, it is possible to easily check how the same place looks in a different situation.

[0046] For example, you can check how the same place changes during the day and night, or depending on the season, or how the place looked in the past. You can also check how an event you are planning to attend looked the last time. By displaying the situation selection icon 102 on an image, you can switch images directly, eliminating the trouble and distraction of first displaying another image such as a search screen or thumbnail. You can also easily switch and compare images even with a limited display area.

[0047] 7 illustrates an example of the structure of a database stored in the image database 60. In this example, the image data information 110 includes a location column 112, an address column 114, a facility ID column 116, an image data column 118, a shooting date and time column 120, and an event column 122. The location column 112 indicates location information such as the latitude and longitude of the shooting location, the address column 114 indicates the address of the shooting location, and the facility ID column 116 indicates identification information such as the name of the facility at the shooting location. The image data column 118 indicates data of images taken at the location or facility or their identification information. The shooting date and time column 120 indicates the date and time when the image was taken, and the event column 122 indicates the name of the event that was being held at the location at the time of the image taking.

[0048] That is, the image data information 110 associates each image data shown in the image data column 118 with the "location information" described in the position column 112, address column 114, and facility ID column 116, as well as with the "shooting situation" described in the shooting date and time column 120 and event column 122. However, it is not intended to limit the items of "location information" and "shooting situation" and the data structure of the image data information 110 to the illustrated example. Here, the shooting situation indicated by the image data information 110 is information specific to the image data, and is obtained by Exif, manual input, or the like. In order to efficiently narrow down this using the situation selection icon 102 in FIG. 6, classification information on the shooting date and time and events is prepared.

[0049] Fig. 8 illustrates an example of the structure of classification information of shooting conditions used in this embodiment. In this example, classification information 130 includes a location column 132, an address column 134, a facility ID column 136, a time period column 138, a season column 140, and an event classification column 142. The location column 132, the address column 134, and the facility ID column 136 indicate the same information as the location column 112, the address column 114, and the facility ID column 116 shown in Fig. 7. The time period column 138, the season column 140, and the event classification column 142 indicate the classification of the time period, season, and event that can be presented as options for the location, respectively.

[0050] For example, for the "City Center" shown in the figure, there are images taken during the day and at night, while there are images for the seasons "spring," "autumn," and "winter," but no images for "summer." There are also images of events classified as "exhibitions" and "concerts." However, the classification items for the shooting conditions are not limited to these. By preparing classifications of various granularities for each of the time conditions and subject conditions, it is possible to efficiently narrow down the images. The classification information 130 is stored in the image database 60 together with the image data information 110 shown in FIG. 7.

[0051] Each time a new captured image is acquired, the attribute acquisition unit 72 of the image providing server 20 registers the data in the image data information 110 in Fig. 7 and updates the classification information 130 as necessary based on the attributes. When viewing an image, the icon display unit 64 of the image processing device 10 acquires the classification information 130 of the specified location from the image providing server 20 and displays selectable shooting situation classifications as options on the situation selection icon 102.

[0052] When the user selects one of the options, the image switching unit 66 acquires from the image providing server 20 the data corresponding to the selected shooting situation among the image data associated with the location in the image data information 110, and displays it. If there are multiple corresponding images, one of them may be displayed in an enlarged manner, and the other images may be displayed as thumbnails. Note that the image data information 110 shown in Fig. 7 and the classification information 130 shown in Fig. 8 may actually be a single table, or image data may be associated with each classification.

[0053] Next, the image display processing procedure of this embodiment that can be realized by the above configuration will be described. Fig. 9 is a flowchart showing the processing procedure in which the image processing device 10 displays a captured image corresponding to a location in cooperation with the image providing server 20. This flowchart assumes a mode in which the captured image is displayed as part of a map application, but as described above, the processing performed prior to image display is not limited to map display. In addition, the image displayed by the image display unit 56 is basically the image data requested by the data request unit 52 and transmitted from the image providing server 20 and acquired by the data acquisition unit 54, but the procedure is omitted.

[0054] First, when a map application is started by a user operation on the image processing device 10, the image switching unit 66 of the image display unit 56 displays an image of an electronic map on the display device 16 (S10). Until the user inputs the name of a facility to be searched for or indicates a location on the map, the system waits while performing a process (not shown) corresponding to the operation on the map (N in S12). Once the location is specified (Y in S12), the image switching unit 66 displays a thumbnail of an image taken of the location together with information about the location, as shown in FIG. 5(a) (S14).

[0055] The image displayed here may be one selected according to a predetermined rule. In the example of the map display screen in Fig. 5(a), the thumbnail 86a plays the role of a GUI for switching the main display from the map to the captured image, so in some cases, a GUI for switching may be displayed instead of the thumbnail. If no operation is performed on the displayed thumbnail image or GUI, the image display unit 56 waits while performing a process (not shown) corresponding to the operation on the map (N in S16).

[0056] If an operation is performed on the thumbnail image or GUI (Y in S16), and if it is not the above-mentioned long press operation (Y in S18), the image switching unit 66 executes the display set for the operation, such as enlarging and displaying the operated thumbnail in the image field 90 shown in FIG. 5(b) (S20). If the GUI is operated, an image selected according to a predetermined rule may be enlarged and displayed. If the operation on the thumbnail image or GUI is a long press (N in S18), the icon display unit 64 displays the situation selection icon 102 for selecting a shooting situation at the indicated position, as shown in FIG. 6(a) (S22). Then, if a certain shooting situation is selected by sliding the indicated position, the image switching unit 66 displays an image corresponding to the shooting situation in the image field 90 shown in FIG. 5(b) (S24).

[0057] If there is no need to end the display due to a user operation or the like, the processes of S16 to S24 are repeated (N in S26). However, from the second round onwards, the operation target in S16 is not limited to the thumbnail image, and may be an image enlarged and displayed in the image field 90. The process performed in S20 may be an enlarged display of the thumbnail by the image switching unit 66, or a process corresponding to a change in the viewpoint or line of sight by the viewpoint change processing unit 68. Here, when switching the display to an image to which the viewpoint has been moved, an image with the same shooting conditions as the one most recently selected is selected and displayed. If it becomes necessary to end the display, all processes are terminated (Y in S26).

[0058] According to the present embodiment described above, when or while displaying images taken at a location specified by the user, a situation selection icon is displayed for selecting a shooting situation by a specific operation. Images taken in the situation selected by the icon operation are then displayed. This allows images of a desired state to be presented efficiently even at the same location, so that even if images are collected without setting any policy or restriction, inconvenience due to a large number of images is unlikely to occur. As a result, the user can conveniently view images taken at any location and check the state of the location.

[0059] In addition, because the image itself is switched by the icon displayed on the image, it is not distracting and it is easy to compare images in an enlarged state, compared to displaying a separate selection menu or rearranging thumbnail images. Furthermore, in a mode in which the omnidirectional image allows the image of a place to be viewed from a free viewpoint or line of sight, it is possible to continue to display a consistent image in response to the movement of the viewpoint, so that the sense of realism is not easily lost and the surrounding situation can be confirmed without any sense of incongruity.

[0060] Embodiment 2 The first embodiment is a technology for easily selecting and unifying the shooting conditions of images taken at the same location. In the present embodiment, images of different floors among images taken at the same address (same point on the ground) can be distinguished and selected. Maps are generally expressed as a bird's-eye view of the ground so that the horizontal positional relationship of each facility and road can be seen. Therefore, it is difficult to understand the number of floors of a building and the state of each floor from the map alone. Although maps by floor level are sometimes prepared, this is limited to large-scale facilities where there is a need for them.

[0061] Also, as shown in the first embodiment, when the captured images are displayed based on the map, images associated with horizontal positions such as latitude and longitude and the name of the entire building are detected at once, and images of various floors are displayed in a mixed state, which requires a lot of effort to reach the image of the desired floor. In particular, when creating a sense of realism with a panoramic image, it is as if the user is on one floor and has moved to a completely different floor by switching the image, which causes discomfort and confusion. Therefore, by classifying the captured images in association with the floors of the building, the display is also managed by floor, and a user can move to a different floor of the same building.

[0062] The system configuration in this embodiment may be the same as the image display system 1 shown in Fig. 1, and the configurations of the image processing device 10 and the image providing server 20 may be the same as those shown in Figs. 2 to 4. Fig. 10 illustrates an example of the structure of data stored in the image database 60 in this embodiment. In this example, image data information 150 includes a location column 152, an address column 154, a facility ID column 156, a floor column 158, and an image data column 160. The location column 152, the address column 154, and the facility ID column 156 indicate the same information as the location column 112, the address column 114, and the facility ID column 116 shown in Fig. 7. The floor column 158 indicates the number of floors of a building in a corresponding location, and the image data column 160 indicates the data of an image taken on that floor or its identification information.

[0063] In this case, the attribute acquisition unit 72 shown in FIG. 4 may identify the floor on which the image was taken from information such as altitude included in the additional information of the acquired image data, or may have the user who provided the image input the floor on which the image was taken, or may identify the floor from information such as the store name associated with the user. The attribute acquisition unit 72 may also identify the total number of floors of the building shown in the facility ID column 156 from official information of the building, and separately associate the building with the image data. The image data information 150 shown in the figure may be integrated with the image data information 110 shown in FIG. 7 to form one database. In this case, the first and second embodiments can be realized at the same time.

[0064] Fig. 11 is a diagram for explaining the data stored in the image database 60 when the image data column 160 in the image data information 150 in Fig. 10 is further associated with a position in a hierarchy. A schematic diagram of the hierarchy of a building is shown on the left side of the diagram. In this example, the same positions (same positions in the horizontal direction on the floor) a, b, c, d, e, and f are defined on each of the first to third floors. Of these, black circles indicate positions where there is image data, and white circles indicate positions where there is no image data.

[0065] For example, image data does not exist at positions c, e, and f on the second floor because there is no floor due to the atrium structure. There may be cases where there is a floor but no image data, such as position a on the third floor. When managing image data at defined positions, table 162 shown on the right side of FIG. 11 is associated instead of image data column 160 shown in FIG. 10. Table 162 indicates the presence or absence of image data for each level and position. Also, in this example, when image data exists, its identification information, for example "001", "002", ... is directly associated, but the association with the image data may be performed in a separate table.

[0066] By managing image data for each floor position in this way, when an operation to move from the displayed floor to another floor is performed, the image can be switched to the same position on the selected floor. For example, when an image of position a on the first floor is displayed, if an operation to move to the second floor is performed, the display is switched to the image of position a on the second floor. In this way, it is easy to know which position on the selected floor the switched image corresponds to, and there is no confusion when moving thereafter.

[0067] Fig. 12 shows examples of images displayed on the display device 16 in this embodiment. (a) shows a state in which a captured image of a certain floor in a building is displayed. These images are displayed in the image field 90 of the captured image display screen in (b) of Fig. 5, for example, when an electronic map is displayed by a map application and a specific building is specified by searching or cursor operation, as shown in Fig. 5. However, the images are not limited to those switched from the map, so long as they are the result of specifying a location on the ground, such as a building, address, latitude and longitude, etc.

[0068] The image 170 on the left side of (a) is a part of the panoramic image, and as in the case of (b) of FIG. 5, a movement icon 94 for moving the viewpoint is displayed. That is, when this is pointed to, the viewpoint moves in the direction of the arrow (toward the back in the figure), and the panoramic image ahead is displayed. In this embodiment, furthermore, when the movement icon 94 is pressed and held, a floor selection icon 172 for selecting a floor is displayed. As an example, the floor selection icon 172 shown in the figure is a rectangular icon with options from the second basement floor ("B2F") to the third floor above ground ("3F"), each of which is assigned an area.

[0069] The icon display unit 64 determines the configuration of the floor selection icon 172 by acquiring information on all floors of the building and information on selectable floors for which images exist from the image providing server 20. For example, an area for all floors is provided on the floor selection icon 172, and the areas for floors that cannot be selected are made different from the others to indicate that they cannot be selected. As shown in the figure, the currently displayed image, in this example, the first floor ("1F"), is also made a different color from the other floors on the floor selection icon 172, making it unselectable.

[0070] As shown in FIG. 11, when managing image data by position, floors that do not have image data at the same position as the displayed position may be included in the hierarchical levels that cannot be selected. Alternatively, for floors where image data exists at other positions, a different display color may be used or a text may be used to indicate this, so that the floors can be selected with the user's consent. The image providing server 20 refers to the table 162 of the image database 60 and transmits information regarding the selectability of such positions to the image processing device 10. The user selects a desired floor by indicating one of the options of the floor selection icon 172. However, the shape and configuration of the floor selection icon 172 are not limited to this. Furthermore, the operation for displaying the floor selection icon 172 is not limited.

[0071] The image 174 on the right side of (a) assumes a case where a touch pad compatible with multi-touch operation is provided as the input device 14 so as to cover the screen of the display device 16. In this case, the viewpoint change processing unit 68 of the image display unit 56 changes the direction of the line of sight with respect to the panoramic image in response to a swipe operation in which, for example, a finger that has been touched with a single touch (one contact point) is slid. As with the image 170, a movement icon 94 may be displayed, and the viewpoint may be advanced in response to a touch operation thereon.

[0072] Furthermore, in this embodiment, in response to a swipe operation in which a finger touching the screen with multi-touch (multiple contact points) is slid in a direction that can be considered as the up or down direction on the screen, the display is switched to the next floor up or down corresponding to that direction. In the illustrated example, as shown by the arrow, an upward swipe operation switches the display to the image of the next floor down. This allows for an intuitive operation that feels like moving the displayed floor up. By distinguishing between viewpoint or line of sight operations and floor change operations according to the number of contact points, it is possible to easily achieve both.

[0073] (b) shows an image during a transition period when the display is switched to a lower floor by the above-mentioned operation on image 170 or image 174. In this example, an image obtained by combining pre-switching image 176 and post-switching image 178 is shown as a transition period. Here, the images of each floor are arranged so as to correspond to the hierarchical relationship of the floors of the building. In other words, if the post-switching floor is lower than the pre-switching floor, the post-switching image is arranged below, and if the post-switching floor is higher, the post-switching image is arranged above. Then, by moving the boundary between the two so that the area of ​​the post-switching image gradually increases, the post-switching image is presented as appearing from a direction corresponding to the floor.

[0074] For example, both images can be alpha blended and the area in which the post-switching image 178 is transparent (α=0) can be narrowed vertically. The boundary between the two images can be semi-transparently composited by appropriately controlling the α value. However, the way in which the images are displayed during the transition period is not limited to this; for example, the images before and after the switch can be slid upward or downward, or both can be enlarged or reduced vertically. In any case, by making the image of the post-switching floor appear from a direction corresponding to the floor of the building, the user can get the feeling that he or she is moving to a different floor of the building.

[0075] After the transition period shown in (b), an image captured at the switched floor is displayed as shown in (c). If multiple images are associated with the switched floor or the same position thereon, for example, by allowing the user to select the shooting situation as in the first embodiment, it becomes easy to access an image of a desired floor and a desired situation. If the image being displayed includes an elevator, and the image of the switched floor also includes the same elevator, the user can easily determine the direction of movement of the viewpoint thereafter, as if he or she were actually traveling in an elevator.

[0076] Therefore, if the image being displayed includes an elevator, when the image of the elevator is pressed and held instead of the movement icon 94 shown in the image 170, the floor selection icon 172 may be displayed. In this case, the input information acquisition unit 50 recognizes that the area of ​​the elevator image has been pressed and held, and the image switching unit 66 determines the field of view so that the image of the elevator is also included in the image after switching. In cases where the same equipment, such as not only elevators but also escalators, stairs, and toilets, is located in the same position on multiple floors, it becomes easier to grasp the location by displaying the image of the corresponding equipment even after switching.

[0077] In this case, the equipment that accepts the display operation of the floor selection icon 172 is set in advance, and when any of the equipment is included in the image being displayed, the input information acquisition unit 50 detects a pointing operation on the area of ​​the image. The operation here may be a long press or other predetermined operation. When switching the image to the floor selected by the floor selection icon, the image switching unit 66 determines the field of view so that the image of the same equipment is included. Note that the illustrated example is an image taken inside a building, but the display target of this embodiment is not limited to this, and an image of the outside of a building or land with an elevation difference may be used as long as it can be associated with height data that is difficult to represent on a two-dimensional map.

[0078] FIG. 13 shows another mode for accepting an operation to select a floor number. In this example, the display image 180 shows the interior of an open-ceiling building. That is, floors other than the currently displayed floor (the floor that accepts the horizontal viewpoint movement operation) are also included in the field of view. In this case, it is more efficient and easier to understand if the destination floor can be directly selected. On the other hand, to avoid confusion with the horizontal viewpoint movement operation on the currently displayed floor, the movement to another floor is also performed by a long press operation in this case. In the example shown in the figure, the third floor is pressed and held by a finger, and the input information acquisition unit 50 detects the operation, and the image switching unit 66 switches the display to the third floor.

[0079] Since the destination is directly pressed and held, the image after switching in this case is preferably an image of the selected floor near the position where it was pressed and held. This allows the line of sight to be moved freely within the building, including up and down. This aspect is not limited to an atrium, and can be realized similarly as long as a part of another floor is displayed in the image of the displayed floor, such as near an escalator or stairs. The image transition may be the same as that shown in FIG. 12, or the viewpoint may be gradually moved to the indicated position. Note that the operation of moving to another floor is not limited to a long press, as long as it can be distinguished from the horizontal movement operation on the same floor. For example, multi-touch or double tap may be used. This aspect may be combined with an aspect of displaying a floor selection icon, or may be realized independently.

[0080] Next, the image display processing procedure of this embodiment that can be realized by the above configuration will be described. Fig. 14 is a flowchart showing the processing procedure in which the image processing device 10 displays a captured image corresponding to a location in cooperation with the image providing server. This flowchart assumes a mode in which the captured image is displayed as part of a map application, but as described above, the processing performed prior to image display is not limited to map display. In addition, the image displayed by the image display unit 56 is basically the image data requested by the data request unit 52 and transmitted from the image providing server 20 and acquired by the data acquisition unit 54, but the procedure is omitted.

[0081] First, when a map application is started by a user operation on the image processing device 10, the image switching unit 66 of the image display unit 56 displays an image of an electronic map on the display device 16 (S30). Until the user inputs the name of a facility to be searched for or indicates a location on the map, the system waits while performing a process (not shown) corresponding to the operation on the map (N in S32). Once the location is specified (Y in S32), the image switching unit 66 enlarges and displays an image selected according to a predetermined rule from among images of buildings in the location (S34), as shown in FIG. 5(b).

[0082] At this time, as in the first embodiment, thumbnails may be displayed once and the user may select an image to be enlarged or the image providing server 20 may select from a database according to a predetermined rule. Then, the icon display unit 64 displays an icon for moving the viewpoint for moving the viewpoint superimposed on the captured image (S36). If no operation is performed on the displayed icon for moving the viewpoint, the image display unit 56 waits while performing a process (not shown) corresponding to a change of line of sight or a selection operation of another image (N in S38). If an operation is performed on the icon for moving the viewpoint (Y in S38) and this is not a long press operation (Y in S40), the viewpoint change processing unit 68 switches the display to the image to which the viewpoint is moved (S42).

[0083] If the operation on the viewpoint movement icon is a long press (N in S40), the icon display unit 64 displays a floor selection icon 172, as in image 170 in Fig. 12 (S44). Then, when a certain floor is selected from the floor selection icon 172, the image switching unit 66 displays an image of the selected floor (S46). At this time, as shown in Fig. 12(b), a transition period occurs in which the switched image appears from a direction corresponding to the hierarchical relationship of the floors, and then the switched floor image as shown in Fig. 12(c) is displayed. Also, if image data is managed by position, the image is displayed in the same position as before the switch.

[0084] If a multi-touch compatible touchpad is introduced, instead of the processes of S38, S40, and S44, a multi-touch swipe operation displays an image of the next floor in the opposite direction to the sliding direction. Also, when a long press operation on a specific facility image is accepted, instead of the determination of the long press operation on the viewpoint movement icon in S38 and S40, a pointing operation on the facility image is determined. Then, in S46, the image of the same facility is included in the display.

[0085] Furthermore, if a part of another floor appears in the image displayed in S34, instead of judging the long press operation on the viewpoint movement icon in S38 and S40, the system judges the operation of pointing to the image of the other floor. In this case, the display process of the floor selection icon in S44 is omitted, and in S46, the image of the corresponding position among the images of the pointed to other floor is displayed.

[0086] If there is no need to end the display due to a user operation or the like, the processes of S36 to S46 are repeated (N in S48). At this time, the image after switching in the viewpoint movement operation in S42 and the candidate images displayed as thumbnails are limited to the image of the most recently selected floor, thereby preventing an image of another floor from being displayed out of context. If it becomes necessary to end the display, all processes are terminated (Y in S48).

[0087] According to the embodiment described above, while a photographed image of a location specified by a user is displayed, a floor selection icon for selecting a floor number of a building is displayed by a specific operation. Then, the display is switched to an image of the floor selected by the icon operation. As a result, even if images correspond to the same location on a two-dimensional map, they can be displayed separately for each floor, and the user can check the situation and understand the structure by narrowing down to the floor of his / her business or interest.

[0088] In the aspect of viewing the image of a place from a free viewpoint or line of sight by using the panoramic image, the floors can be changed with the same operation feeling as moving the viewpoint or changing the line of sight. Furthermore, by providing a transition period in which the image of the selected floor appears from a direction following the hierarchical relationship of the floors, a sense of realism as if exploring a building can be given. In addition, by limiting the images to be displayed after the selection of a floor to the same position on the selected floor, an image of an unrelated floor does not suddenly appear, and the sense of realism is less likely to be lost. Furthermore, by superimposing a floor selection icon on the displayed image, it is possible to easily change between floors compared to displaying a separate selection menu.

[0089] Furthermore, in cases where common equipment is located in the same place on multiple floors, by accepting pointing operations to that image and making that image appear in the field of view before and after switching the displayed floor, it becomes easier to understand the direction even on the switched floor. Also, by being able to directly specify another floor or its location included in the displayed image, the user can easily move to the desired location even if it is a different floor.

[0090] The present invention has been described above based on the embodiments. The above embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0091] 1 image display system, 10 image processing device, 14 input device, 16 display device, 20 image providing server, 23 CPU, 24 GPU, 26 main memory, 32 communication unit, 34 storage unit, 36 output unit, 38 input unit, 40 recording medium drive unit, 50 input information acquisition unit, 52 data request unit, 54 data acquisition unit, 56 image display unit, 58 request acquisition unit, 60 image database, 62 data transmission unit, 64 icon display unit, 66 image switching unit, 68 viewpoint change processing unit, 70 data acquisition unit, 72 attribute acquisition unit.

Claims

1. an input information acquisition unit that accepts a user operation related to an image to be displayed on the display device; an image display unit that displays one of the plurality of images on a display device in response to the user operation, and further displays a situation selection icon for selecting a shooting situation, and when the input information acquisition unit accepts an operation on the situation selection icon, displays another image captured in the selected situation on the display device; Equipped with When a user operation is performed to continuously point within a range that is regarded as the same position as the position at which the situation selection icon is displayed for a predetermined period of time or more, the image display unit displays the situation selection icon representing a different option, or expands the area of ​​the situation selection icon to increase the options represented.

2. 2. The image processing device according to claim 1, wherein when the input information acquisition unit receives an operation to move a viewpoint for the panoramic image being displayed, the image display unit switches the display to a panoramic image from the destination viewpoint that was captured in a most recently selected situation.

3. 3. The image processing apparatus according to claim 1, wherein the image display unit displays the situation selection icon which indicates, as a choice, at least one of a situation related to a shooting time and a situation related to a subject.

4. 4. The image processing apparatus according to claim 1, wherein the image display unit displays the situation selection icon on a thumbnail image being displayed, and displays an enlarged image taken in a selected situation.

5. The image processing device according to any one of claims 1 to 4, characterized in that when a user points at a location on the displayed image and the pointing continues for a predetermined period of time within a range that is considered to be the same location, the image display unit superimposes and displays the situation selection icon at the pointed-to location on the image.

6. 6. The image processing apparatus according to claim 1, wherein the image display unit displays a map on the display device, and displays an image of a location corresponding to a position specified by a user on the map.

7. receiving a user operation related to an image to be displayed on a display device; displaying one of the plurality of images on a display device in response to the user operation; displaying a situation selection icon for selecting a photographing situation; accepting an operation on the situation selection icon and displaying, on the display device, another image captured in the selected situation; Including, The step of displaying the situation selection icon is characterized in that, when a user operation is performed to continuously point within a range regarded as the same position as the position at which the situation selection icon is displayed for a predetermined period of time or more, the situation selection icon representing a different option is displayed, or the area of ​​the situation selection icon is expanded to increase the options represented.

8. A function of accepting a user operation related to an image to be displayed on the display device; a function of displaying one of a plurality of images on a display device in response to the user operation; A function to display icons for selecting shooting conditions, a function of accepting an operation on the situation selection icon and displaying, on the display device, another image captured in the selected situation; This is realized on a computer. A computer program characterized in that the function of displaying the situation selection icon displays the situation selection icon representing a different option, or expands the area of ​​the situation selection icon to increase the options displayed, when a user operation is performed to continuously point within a range that is regarded as the same position as the position at which the situation selection icon is displayed for a predetermined period of time or more.

Images