Image processing device, method, and program

Abstract

Provided are an image processing device, method, and program capable of providing a good image viewing space. The image processing device acquires position information in a virtual space in which a real space is reproduced, and displays a target image indicating a first image in the virtual space on the basis of the first image related to the position information in a group of images captured in the real space. And displays, on the display unit, a second image of the virtual space observed from a viewpoint corresponding to the position information.

Description

Technical Field

[0001] The present invention relates to an image processing device, method and program, and in particular to an image processing device, method and program that provides a virtual space. Background Art

[0002] Patent Documents 1 to 4 describe a technology for viewing an image captured in a real space in a virtual space.

[0003] Previous technical literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2009-296248

[0006] Patent Document 2: Japanese Patent Application Publication No. 2011-186565

[0007] Patent Document 3: Japanese Patent Application Publication No. 2012-252716

[0008] Patent Document 4: Japanese Patent Application Publication No. 2022-68642 Summary of the invention

[0009] One embodiment of the technology according to the present invention provides an image processing device, method, and program that can provide a good image viewing space.

[0010] Means for solving technical problems

[0011] (1) An image processing device comprising a processor, wherein the processor is configured as follows:

[0012] Acquire position information in a virtual space that reproduces the real space; display a first image related to the position information in a group of images shot in the real space, and an object image representing the first image in the virtual space; and display a second image of the virtual space observed from a viewpoint corresponding to the position information on a display unit.

[0013] (2) The image processing device according to (1), wherein:

[0014] The processor consists of:

[0015] The virtual space is changed to one corresponding to the image information of the first image.

[0016] (3) The image processing device according to (1) or (2), wherein:

[0017] The first image is an image captured within a range of a first distance from a position in the real space corresponding to the position information in the virtual space.

[0018] (4) The image processing device according to (3), wherein:

[0019] The first image is also an image captured within a range in the real space corresponding to the range of the second image in the virtual space.

[0020] (5) The image processing device according to any one of (1) to (4), wherein:

[0021] The processor consists of:

[0022] The target image is displayed at a position in the virtual space corresponding to the position in the real space where the first image was captured.

[0023] (6) The image processing device according to (2), wherein:

[0024] The processor consists of:

[0025] When the position indicated by the position information moves to a position within the range of the second distance from the display position of the target image, the virtual space is changed to the virtual space corresponding to the image information of the first image.

[0026] (7) The image processing device according to any one of (1) to (6), wherein:

[0027] The processor consists of:

[0028] The target image is displayed so as to be aligned with the direction in which the first image was captured.

[0029] (8) The image processing device according to any one of (1) to (7), wherein:

[0030] The processor consists of:

[0031] The target image is displayed in a size corresponding to the angle of view of the first image.

[0032] (9) The image processing device according to any one of (1) to (7), wherein:

[0033] The processor consists of:

[0034] When the viewing angle of the first image exceeds a threshold, the target image is displayed in a size corresponding to the viewing angle; and when the viewing angle of the first image is below the threshold, the target image is displayed in a specified size.

[0035] (10) The image processing device according to (2), wherein:

[0036] The processor consists of:

[0037] The photographic environment of the first image is determined based on the image information of the first image; and the virtual space is changed to correspond to the determined photographic environment.

[0038] (11) The image processing device according to (10), wherein:

[0039] The processor consists of:

[0040] At least one of season, weather, and time period is determined as the photographing environment.

[0041] (12) The image processing device according to (2), (10) or (11), wherein:

[0042] The processor consists of:

[0043] The virtual space data is changed to construct a different virtual space.

[0044] (13) The image processing device according to (2), (10), (11) or (12), wherein:

[0045] The processor consists of:

[0046] Apply effects to change virtual space.

[0047] (14) The image processing device according to (2), (10), (11), (12) or (13), wherein:

[0048] The processor consists of:

[0049] When there are a plurality of first images, a plurality of object images are displayed in the virtual space; a selection of an object image is received; and the virtual space is changed to correspond to the image information of the first image represented by the selected object image.

[0050] (15) The image processing device according to (14), wherein:

[0051] The processor consists of:

[0052] An object image existing in the line of sight direction is regarded as a selected object image, and the selection of the object image is received.

[0053] (16) The image processing device according to (2), (10), (11), (12), (13), (14) or (15), wherein:

[0054] The first image is an image captured within a range of a first distance from a position in the real space corresponding to the position information in the virtual space.

[0055] (17) The image processing device according to (16), wherein:

[0056] The first image is also an image captured within a range in the real space corresponding to the range of the second image in the virtual space.

[0057] (18) The image processing device according to (2), (10), (11), (12), (13), (14), (15), (16) or (17), wherein:

[0058] The processor consists of:

[0059] The selected object image and other object images are displayed in different display modes.

[0060] (19) An image processing method comprising the following steps:

[0061] Acquire position information in a virtual space that reproduces the real space; display an object image representing the first image in the virtual space based on the first image related to the position information in an image group shot in the real space; and display a second image of the virtual space observed from a viewpoint corresponding to the position information on a display unit.

[0062] (20) An image processing program that enables a computer to implement the following functions:

[0063] Acquire position information in a virtual space that reproduces the real space; display an object image representing the first image in the virtual space based on the first image related to the position information in an image group shot in the real space; and display a second image of the virtual space observed from a viewpoint corresponding to the position information on a display unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a diagram showing an overview of an image viewing system.

[0065] Figure 2 This is a diagram showing an example of a system configuration of an image viewing system.

[0066] Figure 3 This is a block diagram showing a structural example of a display terminal.

[0067] Figure 4 This is a block diagram of the main functions of the control unit of the display terminal.

[0068] Figure 5 This is a diagram showing an example of the hardware configuration of an image processing device.

[0069] Figure 6 This is a block diagram of the main functions of the image processing device.

[0070] Figure 7 It is a conceptual diagram of image retrieval.

[0071] Figure 8 This is a diagram showing an example of virtual space data stored in a virtual space database.

[0072] Fig. 9 This is a diagram showing an example of an image of a virtual space displayed on the display unit of the display terminal.

[0073] Fig.10 This is a flowchart showing the procedure of providing a virtual space by an image processing device.

[0074] Fig.11 This is a flowchart showing the procedure of providing a virtual space by an image processing device.

[0075] Fig.12 This is a conceptual diagram of the display of photographic images.

[0076] Fig.13 This is a flowchart showing the processing steps for displaying a photographic image.

[0077] Fig.14 This is a conceptual diagram of the display of photographic images.

[0078] Fig.15 This is a flowchart showing the processing steps for displaying a photographic image.

[0079] Fig.16 This is a diagram showing another example of photographic image display.

[0080] Fig.17 This is a diagram showing another example of photographic image display.

[0081] Fig.18 This is a block diagram of main functions of an image processing device related to changing a virtual space.

[0082] Fig.19 This is a conceptual diagram for judging photographic images during appreciation.

[0083] Fig. 20 This is a flowchart of processing steps related to changes in virtual space.

[0084] Fig.21 This is a flowchart showing the steps of the photographic environment determination process.

[0085] Fig. 22 This is a conceptual diagram for judging photographic images during appreciation.

[0086] Fig.23 This is a diagram showing an example of an image selection operation performed by a user.

[0087] Fig.24 This is a diagram showing another example of display of photographic images in a virtual space.

[0088] Fig.25 This is a diagram showing another example of display of photographic images in a virtual space. DETAILED DESCRIPTION

[0089] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0090] [First embodiment]

[0091] In recent years, with the popularization of social networking services (SNS), images taken at tourist attractions are posted in a near-real-time timeline. Therefore, even if you do not actually visit the tourist attractions, you can indirectly confirm the local conditions by browsing these images.

[0092] On the other hand, in recent years, with the advancement of technologies such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality), digital twins that reproduce real space (real space, actually existing space) in virtual space are gradually becoming a reality.

[0093] In the present invention, a new image appreciation system using digital twins is provided.

[0094] [summary]

[0095] Figure 1 It is a diagram showing an overview of the image viewing system of the present invention.

[0096] For example, a virtual space that reproduces a real space such as a tourist attraction is provided to the user via a head mounted display (HMD). The user can move freely in the virtual space as in the real space. When there is an image captured in the real space, if the user approaches a position in the virtual space corresponding to the position in the real space where the image was captured, the image will be displayed in the virtual space.

[0097] [Image Appreciation System]

[0098] Figure 2 This is a diagram showing an example of the system configuration of an image viewing system.

[0099] like Figure 2As shown, the image appreciation system 1 of the present embodiment includes: a display terminal 10 that displays an image in a virtual space; an image processing device 100 that provides the image in the virtual space to the display terminal 10; and an image transmission terminal 200 that transmits an image captured in a real space to the image processing device 100. The display terminal 10 and the image processing device 100 are connected in a communicable manner via a network 2. Furthermore, the image transmission terminal 200 and the image processing device 100 are connected in a communicable manner via the network 2.

[0100] [Image transmission terminal]

[0101] The image transmission terminal 200 is composed of, for example, a computer with a communication function (e.g., a personal computer, etc.), a mobile terminal (e.g., a smart phone, a mobile phone, a tablet terminal, etc.), a photographic device with a communication function (e.g., a digital camera, etc.), etc. These structures themselves are well-known contents, and therefore the detailed description thereof is omitted.

[0102] As described above, the image transmitting terminal 200 transmits (uploads) an image captured in a real space to the image processing device 100. The transmitted image may be an image captured by another device. When the image transmitting terminal itself has a photographing function (e.g., a smartphone with a camera function, etc.), the image captured by the terminal can be transmitted to the image processing device 100.

[0103] The image sent by the image sending terminal 200 to the image processing device 100 is accompanied by at least information indicating the shooting position (shooting position information). The shooting position information is composed of information that can uniquely determine the position in the real space. For example, the shooting position information can be composed of information on the latitude and longitude of the place where the photo was taken. The shooting position information can also include altitude information. Preferably, in addition to the shooting position information, the image is also accompanied by information indicating the shooting date and time (shooting date and time information), information indicating the shooting direction (shooting orientation, optical axis direction) (shooting direction information), etc.

[0104] The manner in which photographic location information and the like are attached to an image is not particularly limited. For example, it can be attached to an image in the form of metadata. In a device that records images in the EXIF ​​(Exchangeable image file format) format, photographic location information and the like can be attached to an image in the form of tag information. In particular, in digital cameras and mobile terminals equipped with GPS (Global Positioning System), the GPS information (latitude, longitude, altitude, etc.) at the time of shooting is automatically attached to the captured image as photographic location information. Furthermore, in digital cameras and mobile terminals equipped with an electronic compass (geomagnetic sensor), the captured orientation information is automatically attached to the image as photographic direction information.

[0105] [Display Terminal]

[0106] In this embodiment, the display terminal 10 is, for example, a non-transmissive HMD that is worn on the user's head and covers the user's field of view with a display unit. By setting the HMD to block the outside world when worn on the head, the sense of virtual reality and immersion during viewing can be increased.

[0107] Figure 3 This is a block diagram showing a structural example of a display terminal.

[0108] like Figure 3 As shown, the display terminal 10 includes a control unit 11 , a communication unit 12 , an operation unit 13 , a sensor unit 14 , a display unit 15 , a sound input unit 16 , and a sound output unit 17 .

[0109] The control unit 11 functions as a processing unit and a control unit, and centrally controls the overall operation of the display terminal 10 according to various programs. The control unit 11 is composed of, for example, a computer having a processor and a memory. The processor is implemented by an electronic circuit such as a CPU (Central Processing Unit). The memory includes a ROM (Read Only Memory) for storing programs and various data, a RAM (Random Access Memory) used as a work area, a flash memory, and the like.

[0110] The communication unit 12 is connected to the network 2 by wire or wireless, and communicates with the image processing device 100 on the network. The communication unit 12 is connected to the network 2 for communication, for example, by wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), mobile communication network 5G (5th Generation / 5th generation mobile communication system), 4G (4th Generation / 4th generation mobile communication system), LTE (Long Term Evolution), etc.

[0111] The operation unit 13 receives the user's operation instructions and outputs the operation content to the control unit 11. In this embodiment, movement in the virtual space is performed by operating the operation unit 13. The movement operation can be configured to be performed by an operation device that can input a direction. For example, a hand controller, a joystick, etc. can be used. In addition, the operation unit 13 can include a known operation device such as a push switch, a rod, a volume controller, a pedal switch, a keyboard, a mouse, a touch pad, a trackball, a gesture input device, etc.

[0112] The sensor unit 14 at least detects the inclination (inclination of XYZ3 axes) (3 Degree of Freedom; 3DoF / 3 degrees of freedom) of the headset or goggles (mounting portion for the head). That is, the orientation of the head is detected. The sensor unit 14 can also be configured to detect the position of the headset or goggles in three-dimensional space (6DoF / 6 degrees of freedom). That is, in addition to the orientation, the position of the head is also detected. In addition, since this sensing technology is well-known, the detailed description thereof is omitted. As an example, a structure that uses a gyroscope sensor, an acceleration sensor, and a geomagnetic sensor to detect the inclination and the position in three-dimensional space of the headset or goggles can be adopted. The sensor unit 14 can also include a living body sensor that detects the user's living body information (for example, pulse, heartbeat, sweating, blood pressure, body temperature, breathing, electromyographic value, brain wave, etc.), a line of sight detection sensor that detects the user's line of sight in the headset or goggles, etc. The sensing information detected by the sensor unit 14 is output to the control unit 11.

[0113] When the display terminal 10 is composed of an HMD, the display unit 15 has a left-eye screen and a right-eye screen corresponding to the left and right eyes of the user, and displays images corresponding to each screen (left-eye image and right-eye image). The screen of the display unit 15 is composed of a display panel such as an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display, or a laser scanning display such as a retinal direct scanning display.

[0114] The voice input unit 16 is constituted by, for example, a microphone, and collects voice uttered by the user.

[0115] The audio output unit 17 is composed of, for example, headphones or earphones, and reproduces audio signals.

[0116] Figure 4 This is a block diagram of the main functions of the control unit of the display terminal.

[0117] like Figure 4 As shown, the control unit 11 of the display terminal 10 functions as a position recognition unit 11A, a line of sight direction recognition unit 11B, a communication control unit 11C, a display control unit 11D, and the like.

[0118] The position recognition unit 11A recognizes the current position of the user (user position) in the virtual space based on the operation information of the operation unit 13. For example, the user position is recognized based on the moving direction and amount from the origin position set in the virtual space. The position in the virtual space corresponds one-to-one to the position in the real space.

[0119] The gaze direction recognition unit 11B recognizes the current gaze direction of the user in the virtual space based on the head state (direction and position, etc.) detected by the sensor unit 14 .

[0120] The communication control unit 11C controls the communication with the image processing device 100. The current user position information (user position information) recognized by the position recognition unit 11A and the information of the current line of sight direction of the user recognized by the line of sight direction recognition unit (user line of sight information) are sent to the image processing device 100 via the communication unit 12 under the control of the communication control unit 11C. As described later, the image processing device 100 generates an image displayed on the display unit 15 based on the information sent from the display terminal 10. The communication control unit 11C receives the display image sent from the image processing device 100 via the communication unit 12.

[0121] The display control unit 11D controls display on the display unit 15. The display image received from the image processing apparatus 100 is displayed on the display unit 15 under the control of the display control unit 11D.

[0122] [Image processing device]

[0123] Figure 5 This is a diagram showing an example of the hardware configuration of an image processing device.

[0124] like Figure 5 As shown, the image processing apparatus 100 includes a CPU 111, a ROM 112, a RAM 113, an auxiliary storage device 114, an input device 115, an output device 116, and a communication interface (Interface: I / F) 117. Generally, such a configuration can be realized by a computer.

[0125] The image processing apparatus 100 functions as an image processing apparatus when the CPU 111 as a processor executes a predetermined program (image processing program). The program executed by the CPU 111 is stored in the ROM 112 or the auxiliary storage device 114 .

[0126] The auxiliary storage device 114 constitutes a storage unit of the image processing apparatus 100. The auxiliary storage device 114 is constituted by, for example, a HDD (hard disk drive) or an SSD (solid state drive).

[0127] The input device 115 constitutes an operation unit of the image processing apparatus 100. The input device 115 is constituted by, for example, a keyboard, a mouse, a touch panel, or the like.

[0128] The output device 116 constitutes a display unit of the image processing device 100. The output device 116 is constituted by, for example, a KCD, an OLED display, or the like.

[0129] The communication interface 117 is connected to the network 2 via wire or wirelessly, and communicates with the image processing apparatus 100 and the image transmission terminal 200 on the network.

[0130] Figure 6 This is a block diagram of the main functions of the image processing device.

[0131] The image processing device 100 mainly has a function of collecting photographic images from the image transmission terminal 200 and a function of providing a virtual space to the display terminal 10 .

[0132] As for the function of collecting photographic images from the image transmission terminal 200 , the image processing device 100 functions as a photographic image acquisition unit 100A and a photographic image management unit 100B.

[0133] The photographed image acquisition unit 100A acquires a photographed image transmitted from the image transmission terminal 200 via the network 2 .

[0134] The photographic image management unit 100B stores the photographic images acquired by the photographic image acquisition unit 100A in a photographic image database (Data Base; DB) 114A. The photographic image database 114A is stored in, for example, the auxiliary storage device 114. As described above, information on the photographic position (photographic position information) is added to the photographic image sent from the image sending terminal 200. The photographic image management unit 100B associates the photographic image with the photographic position information and stores it in the photographic image database 114A. In addition, the photographic image management unit 100B numbers each photographic image and stores it in the photographic image database 114A. The multiple photographic images stored in the photographic image database 114A are an example of a group of images taken in a real space.

[0135] Regarding the function of providing a virtual space to the display terminal 10 , the image processing device 100 functions as a user information acquisition unit 100C, a view area calculation unit 100D, a photographic image search unit 100E, a virtual space data selection unit 100F, and a display image generation unit 100G.

[0136] The user information acquisition unit 100C acquires information about the current state of the user (user information). The user information includes user position information (the current user position information in the virtual space) and user line of sight information (the current user line of sight direction information in the virtual space). These information are acquired from the display terminal 10 via the network 2. The acquired information is added to the photographic image retrieval unit 100E and the viewing area calculation unit 100D. The user position information is an example of position information in the virtual space that reproduces the real space.

[0137] The viewing area calculation unit 100D calculates the viewing area (field of view) of the user in the virtual space based on the user information (user position information and user line of sight information). More specifically, the viewing area of ​​the user's avatar in the virtual space is calculated. The viewing area corresponds to the display range of the image of the virtual space displayed on the display unit 15. The calculated viewing area information (display range information) is added to the photographic image retrieval unit 100E and the display image generation unit 100G.

[0138] The photographic image search unit 100E searches the photographic image database 114A for photographic images (related images) related to the user's current position based on the user's current position information (user position information) in the virtual space and information on the user's current view area.

[0139] Figure 7 It is a conceptual diagram of image retrieval.

[0140] Figure 7This is a diagram that looks down on the virtual space (Virtual Space) and the real space (Real Space) corresponding to the virtual space. The symbol Pv0 in the figure represents the position of the user (avatar) in the virtual space. The symbol Pr0 represents the position in the real space corresponding to the position Pv0 in the virtual space. The symbols Pr1 to Pr8 represent the positions where the images are taken in the real space (the positions where the images taken at the positions exist). The symbols Pv1 to Pv8 represent the positions in the virtual space corresponding to the positions Pr1 to Pr8 in the real space. The arrow DG represents the line of sight of the user in the virtual space. The area FV indicated by the oblique lines represents the area within the range of the radius R1 based on the current position Pv0 of the user in the virtual space and within the range of the current user's viewing angle area (field of view). In this example, the range of 100° in each left and right direction (200° as a whole) based on the line of sight direction DG is used as the horizontal viewing angle area of ​​the user in the virtual space. The area FR indicated by the oblique lines is the area of ​​the real space corresponding to the area FV in the virtual space.

[0141] In the present embodiment, the photographic image retrieval unit 100E retrieves images that are taken within a range of radius R1 (below a distance threshold R1) based on a position Pr0 in the real space corresponding to the current position Pv0 of the user in the virtual space and that are taken within the current field of view. For example, when the current position of the user in the virtual space is XX degrees XX minutes XX seconds north latitude and XX degrees XX minutes XX seconds east longitude, images that are taken within a circle of radius R1 at a position of XX degrees XX minutes XX seconds north latitude and XX degrees XX minutes XX seconds east longitude and that are taken within the current field of view are retrieved (images that are taken within the range of area FR in the real space corresponding to area FV in the virtual space are retrieved). Figure 7 In the example shown, the images captured at positions Pr1, Pr3, Pr5, and Pr7 are images captured within the range of radius R1. Among them, the images captured at positions Pr1 and Pr3 are images captured within the range corresponding to the current field of view. Therefore, the images captured at positions Pr1 and Pr3 are retrieved.

[0142] The retrieved photographic image is added to the display image generation unit 100G. The retrieved photographic image is an example of the first image related to the position information. Furthermore, the range of the radius R1 is an example of the range of the first distance.

[0143] The range of the region FR is an example of a range in the real space that is a range of the first distance from the position in the real space corresponding to the position information in the virtual space and corresponds to the range of the second image in the virtual space.

[0144] The virtual space data selection unit 100F selects virtual space data to be used. A plurality of virtual space data are stored in the virtual space database 114B in advance. Therefore, the virtual space data selection unit 100F selects the virtual space data to be used from the plurality of virtual space data stored in the virtual space database 114B. The virtual space database 114B is stored in the auxiliary storage device 114, for example.

[0145] Figure 8 This is a diagram showing an example of virtual space data stored in a virtual space database. Figure 8 An example of virtual space data for providing a plurality of virtual spaces having different environments with respect to one real space is shown.

[0146] like Figure 8 As shown, multiple virtual space data are prepared in combination of seasons (spring, summer, autumn, winter), weather (sunny, cloudy, rainy) and time periods (morning, noon, evening, night). Each virtual space data is data that reproduces the same real space, but has different seasons, weather and time periods.

[0147] In this way, the virtual space database 114B is configured to record a plurality of virtual space data for one real space so as to be able to construct virtual spaces of different environments. Virtual space data that can reproduce at least one real space is recorded in the virtual space database 114B. Each virtual space data is recorded by associating it with the information of the reproduced real space.

[0148] The real space to be reproduced (real space provided to the user as a virtual space) is selected by the user, for example. The virtual space data to be used is selected, for example, according to the current date and time of the region where the display terminal 10 is used. In the case of Japan, March to May is set as spring, June to August is set as summer, September to November is set as autumn, and December to February is set as winter, and the season of the virtual space data to be used is selected according to the current date and time. In addition, 6:00 to 9:00 is set as morning, 9:00 to 15:00 is set as noon, 15:00 to 18:00 is set as evening, and 18:00 to 6:00 the next day is set as night, and the time period of the virtual space data to be used is selected according to the current date and time. For example, if the region where the display terminal 10 is used is Japan and the current date and time is 12:00 on January 1, the virtual space data corresponding to "noon" of "winter" is selected. In addition, with regard to weather, the default setting weather (for example, sunny) is selected. Therefore, in this case, "JP040102" is selected as the virtual space data (the default setting of "weather" is "sunny"). In addition, the weather may be randomly selected. The current date and time information is acquired from the system or the display terminal 10. The selected virtual space data is added to the display image generation unit 100G. Furthermore, it may be configured to select pre-set virtual space data or virtual space data selected by the user.

[0149] The display image generation unit 100G generates an image of the virtual space (display image) provided to the display terminal 10 based on the virtual space data selected by the virtual space data selection unit 100F and the information of the viewing angle area calculated by the viewing angle area calculation unit 100D. The display image is an image of the virtual space observed from the user's viewpoint at the user's (avatar's) current position in the virtual space.

[0150] In this embodiment, when there is a photographic image (related image) related to the user's current position, the photographic image is displayed in the virtual space. Therefore, when there is a photographic image related to the user's current position, the display image generation unit 100G generates a display image that displays the photographic image in the virtual space.

[0151] Fig. 9 This is a diagram showing an example of an image of a virtual space displayed on the display unit of the display terminal.

[0152] The image of the virtual space (display image) IMV displayed on the display unit 15 of the display terminal 10 is an image obtained by cutting out a view area from the image of the virtual space constructed by the virtual space data.

[0153] In the present embodiment, the photographic image IMO is displayed at a position Pv in the virtual space corresponding to the photographic position Pr in the real space. Furthermore, the photographic image IMO is displayed in the virtual space with a preset size. That is, a planar object of a specified size is arranged in the virtual space and can be visually recognized by the user. Therefore, the closer the user is to the display position (photographing position), the larger it is displayed. Furthermore, the photographic image IMO is displayed at a specified height from the ground (for example, a position approximately at the eye level of an adult). Therefore, in the virtual space, it is displayed in a suspended state. Furthermore, the photographic image IMO is displayed in a manner facing the user (avatar). In the present embodiment, the photographic image IMO displayed in the virtual space is an example of an object image.

[0154] The display image generated by the display image generation unit 100G is transmitted to the display terminal 10 via the network 2. The display terminal 10 receives the display image transmitted from the image processing device 100 and displays it on the display unit 15. Thus, the user can appreciate the image of the virtual space that changes in conjunction with the user's movement. In this embodiment, the display image is an example of the second image.

[0155] [Operation of the Image Processing Device (Image Processing Method)]

[0156] Fig.10 and Fig.11 This is a flowchart showing the procedure of providing a virtual space by an image processing device. Fig.10 The steps until an image of a virtual space is displayed on the display unit 15 of the display terminal 10 are shown. Fig.11 The operation steps after the display starts are shown.

[0157] like Fig.10 As shown, first, the information of the real space to be provided is obtained (step S1). In this embodiment, for example, the information of the real space that can be provided as the virtual space is displayed in a list on the display unit 15 of the display terminal 10. The user selects the desired real space from the list display. The selected information is sent to the image processing device 100 as the information of the real space to be provided.

[0158] Next, the information of the current date and time is acquired (step S2). As described above, the information of the current date and time is acquired from the system or the display terminal 10.

[0159] Next, virtual space data of the virtual space to be provided is selected based on the acquired real space information and the current date and time information (step S3). As described above, for example, if the current date and time is 12:00 on January 1, virtual space data corresponding to "noon" in "winter" is selected. In addition, regarding the weather, the default setting weather is selected.

[0160] Next, a display image is generated based on the selected virtual space data (step S4). That is, an image of the virtual space displayed on the display unit 15 of the display terminal 10 is generated. The image is generated based on a reference position (origin position or starting position) pre-set in the virtual space. And, it is generated based on a reference line of sight direction. That is, it is generated as an image observed when facing a predetermined direction at the reference position.

[0161] The generated display image is provided to the display terminal 10 via the network 2 and displayed on the display unit 15 of the display terminal 10 (step S5 ).

[0162] If the display starts, then Fig.11 As shown, it is determined whether the user state has changed (step S11). That is, it is determined whether the user has moved or changed the direction of sight.

[0163] If a change in state is detected, a photographic image (related image) related to the user's current position is retrieved and the retrieved related image is displayed in the virtual space. The related image is retrieved from the photographic image database 114A. The retrieval and display are performed according to the following steps.

[0164] First, it is assumed that i=1 (step S12), and the i-th photographic image is selected from the photographic image database 114A as a processing target (step S13).

[0165] Next, determine whether the selected photographic image (selected image) is a related image (step S14). Determine whether it is an image that was shot within a radius R1 based on a position in the real space corresponding to the current position of the user in the virtual space and shot within the current field of view. In this case, first determine whether the selected image is an image shot within a radius R1. When it is not an image shot within a radius R1, it is determined that it is not a related image. On the other hand, when it is an image shot within a radius R1, then determine whether the image is an image shot within the current field of view. When it is not an image shot within the current field of view, it is determined that it is not a related image. On the other hand, when it is an image shot within the current field of view, it is determined that it is a related image.

[0166] If it is determined that the selected image is a related image, it is determined whether the selected image is being displayed (step S15). In other words, it is determined whether the selected image is being displayed in the virtual space.

[0167] If the selected image is already being displayed, the display continues. On the other hand, if the selected image is not being displayed, the selected image is displayed in the virtual space (step S16). The image is displayed at a position corresponding to the shooting position. That is, it is displayed at a position in the virtual space corresponding to the shooting position in the real space. Furthermore, the image is displayed in the virtual space with a predetermined size and in a manner facing the user (refer to FIG. 1 ). Fig. 9 ).

[0168] In step S14, if it is determined that the selected image is not a related image, it is determined whether the selected image is being displayed (step S20). That is, it is determined whether the selected image is displayed in the virtual space even though it is not a related image. If the selected image is being displayed, the display is terminated (step S21).

[0169] Then, i=i+1 is set, and the value of i is incremented (step S18), and it is determined whether i exceeds N (i>N) (step S19). That is, it is determined whether all the photographic images in the photographic image database 114A have been processed. N is the total number of photographic images in the photographic image database 114A.

[0170] When i>N is not satisfied, that is, when there is a photographic image to be processed next, the process returns to step S13 and the above-mentioned process is performed again.

[0171] On the other hand, when i>N, that is, when all photographic images in the photographic image database 114A have been processed, it is determined whether the user has exited the virtual space (step S19). For example, it is determined whether the end of the display is indicated by turning off the power supply, etc. In step S11, it is also determined whether the user has exited the virtual space in the same manner when it is determined that there is no state change. If it is determined that the user has exited the virtual space, the processing is terminated. On the other hand, if it is determined that the user has not exited the virtual space, the process returns to step S11 and it is determined again whether there is a state change.

[0172] According to the above structure, if the user moves in the virtual space, the photographic images are displayed one by one in the virtual space in conjunction with the movement. Each photographic image is displayed at a position corresponding to the position in the real space where the image was taken. In this way, it is possible to indicate how each photographic image was taken in an easily understandable form. In addition, the user can deepen his understanding of the taken images and enjoy a better viewing experience. In addition, if the user is separated from the display position (photographing position) by more than a specified distance, each photographic image will disappear, so the display in the virtual space will not become complicated.

[0173] [Modifications]

[0174] "Image display in a way that is aligned with the direction of photography"

[0175] In the above embodiment, when the photographic image is displayed in the virtual space, it is displayed facing the user. When the information of the photographing direction is acquired from the photographic image, the photographic image may be displayed in a manner aligned with the direction in which the photographic image was taken.

[0176] Fig.12 This is a conceptual diagram of the display of photographic images.

[0177] Fig.12 (A) is a conceptual diagram of display of the photographic image IMO when information on the photographic direction cannot be acquired from the photographic image IMO. Fig.12 (B) is a conceptual diagram of display of the photographic image IMO when information on the photographic direction can be acquired from the photographic image IMO.

[0178] like Fig.12 As shown in (A), when the information of the photographing direction cannot be obtained from the photographic image IMO, the photographic image IMO is displayed so as to face the user (avatar) U. More specifically, the photographic image IMO is displayed so as to be perpendicular to the depth direction of the screen.

[0179] On the other hand, when the information of the photographic direction can be obtained from the photographic image IMO, the photographic image IMO is displayed in the virtual space in a manner aligned with the photographic direction when the image was taken. Specifically, the photographic image IMO is displayed in a manner orthogonal to the direction of the optical axis L when the image was taken.

[0180] The information of the photographing direction is obtained from the information attached to the photographed image (for example, tag information). Alternatively, the photographing direction can be inferred through image recognition.

[0181] Fig.13 This is a flowchart showing the processing steps for displaying a photographic image.

[0182] First, the information of the photographic direction of the photographic image to be displayed is obtained (step S31). As described above, the information of the photographic direction is obtained from the information attached to the photographic image. Alternatively, the information of the photographic direction is obtained by image recognition.

[0183] Next, it is determined whether there is information on the photographing direction (step S32). That is, it is determined whether the information on the photographing direction is acquired.

[0184] When there is information about the photographing direction (when the information about the photographing direction is acquired), the photographed image is displayed in the virtual space in a manner aligned with the photographed direction (step S33). Fig.12 As shown in (B), the photographic image IMO is displayed in a manner perpendicular to the photographing direction (the direction of the optical axis L).

[0185] On the other hand, when there is no information on the photographing direction (when the information on the photographing direction cannot be obtained), the photographed image is displayed in the virtual space in a normal display manner (step S34). Fig.12 As shown in (A), the photographic image IMO is displayed facing the user.

[0186] In this way, when the information of the photographing direction can be obtained, the photographed image is displayed in the virtual space in a manner aligned with the photographed direction. This enables a deeper understanding of how the displayed photographic image was taken.

[0187] Furthermore, when it is possible to obtain information on the height from the ground when the photographic image was taken, the photographic image can be displayed in the virtual space in a manner aligned with the height at the time of taking the image.

[0188] In addition, some users may find that the normal display method is easier to see. Therefore, the user may be able to arbitrarily select whether to display in a method aligned with the shooting direction.

[0189] [Image display in a way that is aligned with the photographic field of view]

[0190] In the above embodiment, when the photographic image is displayed in the virtual space, it is displayed in a predetermined size. When the information of the viewing angle can be obtained from the photographic image, the photographic image can be displayed in the virtual space in a size corresponding to the viewing angle.

[0191] Fig.14 This is a conceptual diagram of the display of photographic images.

[0192] Fig.14 (A) and (B) show display examples of two images captured at different viewing angles. Fig.14 (A) shows the Fig.14 (B) An example of shooting with a wider field of view. More specifically, Fig.14 The viewing angle of the photographic image IMO1 shown in (A) is ω1 and Fig.14 (B) shows an example of a case where, when the viewing angle of the photographic image IMO2 is ω2, ω1>ω2.

[0193] like Fig.14 As shown in (A) and (B), when viewed from the same position, the photographic image IMO1 photographed at the wide viewing angle ω1 is displayed in a larger size than the photographic image IMO2 photographed at the narrow viewing angle ω2.

[0194] For images captured by cameras with the same sensor size, the shorter the focal length, the larger the field of view. That is, the wider the field of view is for images captured with a wide-angle lens, the wider (larger) the field of view is, and the narrower the field of view is for images captured with a telephoto lens.

[0195] When a photographic image is accompanied by information about the angle of view, the information about the angle of view of the photographic image is directly obtained from the accompanying information. When a photographic image is not accompanied by information about the angle of view, the information is obtained using other information accompanying the photographic image. The angle of view can be calculated based on the size of the image sensor and the focal length of the lens. And, for example, when the accompanying information contains information about the focal length of a 35mm conversion lens, the angle of view can be calculated based on the information. For example, when the focal length of a 35mm conversion lens is f=50mm, the angle of view is approximately 47°. In addition, in this case, obtaining information about the focal length of the 35mm conversion lens is actually equivalent to obtaining information about the angle of view.

[0196] Fig.15 This is a flowchart showing the processing steps for displaying a photographic image.

[0197] First, the information of the viewing angle of the photographic image to be displayed is obtained (step S41). As described above, the information of the viewing angle is obtained from the information attached to the photographic image. At this time, when the information of the viewing angle cannot be directly obtained from the attached information, it is obtained through calculation.

[0198] Next, it is determined whether there is information on the viewing angle (step S42). That is, it is determined whether the information on the viewing angle is acquired.

[0199] When there is information on the angle of view (when the information on the angle of view is acquired), the captured image is displayed in the virtual space in a size corresponding to the angle of view (step S43).

[0200] On the other hand, when there is no information about the angle of view (when the information about the angle of view cannot be obtained), the photographic image is displayed in the virtual space at a predetermined display size (default display size) (step S44). That is, the image is displayed in the virtual space at a preset size.

[0201] In this way, when the information of the viewing angle can be obtained, the photographic image is displayed in the virtual space with a size corresponding to the viewing angle. Thus, it is possible to understand at which viewing angle the displayed photographic image was taken, and thus it is possible to correctly understand the size of the subject.

[0202] In addition, as shown in this example, when the display size of a photographic image is changed according to the field of view angle, the display size may sometimes become too large or too small depending on the image. For example, with respect to an image captured using an ultra-telephoto lens, the displayed image may sometimes become too small. And, for example, with respect to an image captured using an ultra-wide-angle lens (including a fisheye lens), the displayed image may sometimes become too large. Therefore, it is more preferable to set restrictions on the display size. For example, it can be configured so that images with a field of view angle below a threshold value (lower limit position) are uniformly displayed at a specified size (minimum display size). Similarly, it can be configured so that images with a field of view angle above a threshold value (above an upper limit value) are uniformly displayed at a specified size (maximum display size).

[0203] Furthermore, some users may find it easier to see if all images are displayed in the same size. Therefore, the user may be able to arbitrarily select whether to change the display size according to the viewing angle.

[0204] In addition, the photographic image may be displayed in a manner aligned with the direction in which it was photographed, similarly to the above-mentioned modification. In this case, the photographic image is displayed in a manner aligned with the direction in which it was photographed and in a size corresponding to the angle of view.

[0205] [Environmental settings of virtual space]

[0206] In the above-mentioned embodiment, the environment of the virtual space provided to the user is determined based on the information of the current date and time of the region where the display terminal 10 is used, but the method of determining the environment of the virtual space provided to the user is not limited to this. It can be configured so that a virtual space of a preset environment is provided to the user. In addition, it can be configured so that the user can arbitrarily select and set the environment of the virtual space to be displayed. For example, it can be configured so that the user can select and set all of the season, weather, and time period of the virtual space to be provided.

[0207] [Display of photographic images in virtual space]

[0208] In the above embodiment, when a photographic image (related image) related to the user's current position exists in the virtual space, the photographic image is displayed at a position corresponding to the photographing position. However, the method of displaying the photographic image in the virtual space is not limited to this.

[0209] Fig.16 This is a diagram showing another example of photographic image display.

[0210] Fig.16 This shows an example of a case where, when related images exist, they are collectively displayed at a predetermined position in the virtual space. Fig.16 In the example shown, the photographic images IMO are displayed in a vertical row in the right corner. Fig.16 As shown, a mark Mv may be indicated at a position in the virtual space corresponding to the photographic position of each photographic image IMO so that the photographic position can be known. In this case, each mark Mv may be indicated with a different color, and the corresponding photographic image IMO may be indicated with a frame of the same color so that the correspondence between each photographic image IMO and each mark Mv may be known.

[0211] Fig.17 This is a diagram showing another example of photographic image display.

[0212] Fig.17 This is an example of a case where an image captured within a radius R1 based on a position in the real space corresponding to the current position of the user in the virtual space is extracted as a related image and displayed in the virtual space.

[0213] The images captured within the current visual field are displayed in the virtual space in accordance with the shooting position, while the images captured outside the visual field are displayed in a predetermined area. Fig.17 In the example shown, the photographic images IMO are displayed in a vertical row in the right corner. Fig.17 As shown, the orientation map Mp can be displayed in the virtual space, and the approximate display position of each photographic image can be shown on the orientation map Mp.

[0214] [Second embodiment]

[0215] In the image viewing system 1 of the above-described embodiment, when there is an image captured in the real space, if the user approaches the shooting position of the image in the virtual space, the captured image is displayed in the virtual space.

[0216] In the image viewing system 1 of this embodiment, the virtual space also changes according to the photographic image viewed by the user. For example, if the image viewed by the user is an image of autumn, the season of the virtual space will also change to autumn.

[0217] In addition, except for the point that the virtual space is changed, the image viewing system is the same as the image viewing system of the above-mentioned embodiment. Therefore, only the differences will be described below.

[0218] Fig.18 This is a block diagram of main functions of an image processing device related to changing a virtual space.

[0219] like Fig.18 As shown in FIG. 1 , the image processing device 100 of the present embodiment further has the functions of an appreciation image determination unit 100H and an image analysis unit 100I.

[0220] The viewed image determination unit 100H determines the photographic image that the user is viewing from among the photographic images displayed in the virtual space. The viewed image determination unit 100H determines the photographic image being viewed based on the user's current position information (user position information) in the virtual space and information on the photographic images being displayed in the virtual space.

[0221] Fig.19 This is a conceptual diagram for judging photographic images during appreciation.

[0222] Fig.19 This is a diagram looking down on the virtual space. The symbol Pv0 in the figure indicates the current position of the user (avatar) in the virtual space. The symbols Pi1 to Pi3 indicate the display positions of the photographed images in the virtual space. Fig.19 The example shown is an example of a case where three photographic images are displayed in a virtual space.

[0223] In this embodiment, images within a radius R2 (below a distance threshold R2) based on the user's current position Pv0 are considered to be images being viewed by the user and extracted from the photographic images being displayed. The radius R2 is set to a value smaller than the radius R1 (R2 < R1). That is, for a photographic image displayed in a virtual space, when it is closer than a specified distance, it is considered to be viewed.

[0224] When a plurality of photographic images exist within the range of the radius R2, a photographic image displayed at a position closest to the user's current position Pv0 among the plurality of photographic images displayed is regarded as an image being viewed by the user.

[0225] Fig.19 In the example shown, two photographic images (the photographic image at position Pi2 and the photographic image at position Pi3) are displayed within the radius R2. Position Pi2 is located closer to the user than position Pi3. Therefore, in this case, the photographic image displayed at position Pi2 is regarded as the photographic image being viewed.

[0226] Information on the photographic image determined to be the one being viewed is added to the image analyzing unit 1001. In the present embodiment, the range of the radius R2 is an example of the range of the second distance.

[0227] Furthermore, judging the photographic image being viewed is substantially the same as judging the photographic image being selected by the user, that is, judging the image being selected for viewing is substantially the same as judging the image being viewed.

[0228] The image analysis unit 100I analyzes the image information of the photographic image. That is, the photographed content (so-called contents) is analyzed. In particular, in this embodiment, the image information is analyzed to determine the photographic environment. The photographic environment here refers to the environment of the real space photographed. The photographic environment includes at least one of the season, weather and time period. In this embodiment, the season, weather and time period of the photographic image are determined.

[0229] The analysis of image information includes not only image analysis but also analysis of incidental information. That is, it includes analyzing incidental information to determine the photographic environment. The analysis of image information by image analysis adopts, for example, a method of determining the photographic environment by image recognition. In this embodiment, the photographic environment is determined by analyzing incidental information.

[0230] Here, the season, weather and time period of the captured image can be roughly determined based on the shooting location and the shooting date and time. For example, if the shooting location is Japan, if 6:00 to 9:00 is set as morning, 9:00 to 15:00 is set as noon, 15:00 to 18:00 is set as evening, and 18:00 to 6:00 the next day is set as night, the time period can be determined based on the shooting date and time. Similarly, if the shooting location is Japan, if March to May is set as spring, June to August is set as summer, September to November is set as autumn, and December to February is set as winter, the season can be determined based on the shooting date and time. Regarding the weather, for example, it can be determined by referring to a database (weather database) that records the weather of various places in the past. Therefore, if the shooting location and shooting date and time information are attached to the photographic image, the shooting environment (season, weather and time period) of the photographic image can be determined based on this information.

[0231] The image analysis unit 100I analyzes information (for example, meta information) accompanying the photographic image to determine the photographic environment, and adds the determined photographic environment information to the virtual space data selection unit 100F.

[0232] In addition, when the additional information cannot be obtained, the photographic environment cannot be determined, so in this case, it is considered that it cannot be determined. In addition, when a part of the items cannot be determined, the item is considered to be unable to be determined (no information). For example, when the weather cannot be determined, the weather is considered to be unable to be determined.

[0233] The virtual space data selection unit 100F selects the virtual space data to be used. As described above, standard virtual space data is selected based on the current date and time of the region where the display terminal 10 is used. On the other hand, when there is a photographic image being viewed by the user, the virtual space data corresponding to the photographic image being viewed is selected. The virtual space data selection unit 100F selects the virtual space data to be used based on the analysis result of the photographic environment of the photographic image by the image analysis unit 100I. That is, the virtual space data to be used is selected based on the season, weather and time period of the determined photographic image. The virtual space data selection unit 100F retrieves the corresponding virtual space data from the virtual space database 114B and selects the virtual space data to be used. For example, when the season determined based on the photographic image is "spring", the weather is "sunny", and the time period is "noon", as Figure 8 As shown, the virtual space data of "JP010102" is selected.

[0234] If the shooting environment cannot be determined for the photographic image being viewed by the user, the virtual space is not changed. Therefore, in this case, the virtual space data is not selected.

[0235] When the photographic environment cannot be determined for a part of the items (when there is no information), the items are searched with the pre-set default settings. For example, when the "season" cannot be determined (when there is no information), the default season information (such as "spring") is used to search for the corresponding virtual space data. The default settings can be configured so that the user can set them arbitrarily. In addition, for items that cannot be determined, the current settings can be inherited. For example, when the "season" cannot be determined, the virtual space data is searched using the season information of the currently displayed virtual space.

[0236] The display image generation unit 100G generates an image of the virtual space (display image) provided to the display terminal 10 based on the virtual space data selected by the virtual space data selection unit 100F and the information of the viewing angle area calculated by the viewing angle area calculation unit 100D. In the present embodiment, when there is a photographic image being viewed by the user, an image of the virtual space matching the environment of the photographic image is generated.

[0237] [Operation of the image appreciation system]

[0238] Fig. 20 This is a flowchart of processing steps related to changes in virtual space.

[0239] First, it is determined whether there is a photographed image being displayed (step S51). That is, it is determined whether there is a photographed image being displayed in the virtual space currently being displayed.

[0240] If there is a photographic image being displayed, it is determined whether there is a photographic image being viewed (step S52). In this embodiment, the presence of an image within a radius R2 relative to the current position of the user is determined to determine whether there is a photographic image being viewed.

[0241] Furthermore, when there are a plurality of photographic images within the radius R2, the photographic image closest to the user's current position is regarded as the photographic image being viewed.

[0242] If there is a photographic image being viewed, the image information of the photographic image being viewed is analyzed (step S53). In this embodiment, the photographic environment is determined by analyzing the accompanying information of the photographic image being viewed.

[0243] Fig.21 This is a flowchart showing the steps of the photographic environment determination process.

[0244] First, it is determined whether the season can be determined (step S53_1). In this embodiment, it is determined whether the season can be determined based on the presence or absence of information on the shooting location and the shooting date and time. When the information on the shooting location and the shooting date and time can be obtained, it is determined that the season can be determined.

[0245] When the season can be determined, the season data is set as the determined content (step S53_2). On the other hand, when the season cannot be determined, the season data is set to "none" (step S53_3).

[0246] Next, it is determined whether the weather can be determined (step S53_4). In this embodiment, it is determined whether the weather can be determined based on the presence or absence of information about the shooting location and the shooting date and time. When the information about the shooting location and the shooting date and time can be obtained, it is determined that the weather can be determined. As described above, the weather is determined based on the shooting location and the shooting date and time by referring to the weather database.

[0247] When the weather can be determined, the weather data is set as the determined content (step S53_5). On the other hand, when the weather cannot be determined, the weather data is set to "none" (step S53_6).

[0248] Next, it is determined whether the time period can be determined (step S53_7). In the present embodiment, it is determined whether the time period can be determined based on the presence or absence of information on the photographing date and time.

[0249] When the time period can be determined, the time period data is set to the determined content (step S53_8). On the other hand, when the time period cannot be determined, the time period data is set to "none" (step S53_9).

[0250] The photographic environment of the photographic image being observed is determined through the above series of steps. The virtual space data is selected according to the determined photographic environment (step S54). That is, according to the set season, weather and time period data, the corresponding virtual space data is retrieved from the virtual space database 114B, and the virtual space data of the virtual space to be changed is selected. For example, when the season data is set to "spring", the weather data is set to "sunny", and the time period data is set to "noon", the virtual space data is selected. Figure 8 As shown, the virtual space data of "JP010102" is selected. For items with data of "none", the default setting data is used. For example, when the season data is "spring", the weather data is "none", and the time period data is "noon", the weather data is set to the default setting data (for example, "sunny") to retrieve the virtual space data.

[0251] If you select the virtual space data you want to change, Fig. 20 As shown, it is determined whether the virtual space needs to be changed (step S56). When the selected virtual space data is the same as the virtual space data of the currently provided virtual space, it is determined that no change is required. In this case, the virtual space is not changed and the current display is continued. On the other hand, when the selected virtual space data is different from the virtual space data of the currently provided virtual space, it is determined that a change is required.

[0252] If it is determined that the virtual space needs to be changed, the provided virtual space is changed (step S56). In this case, an image of the virtual space provided to the display terminal 10 (display image) is generated based on the selected virtual space data and provided to the display terminal 10. The provided image of the virtual space is an image that reproduces the environment of the image being viewed by the user. This can improve the sense of reality.

[0253] Then, it is determined whether the user has exited the virtual space (step S57). If it is determined that the user has exited the virtual space, the process ends. On the other hand, if it is determined that the user has not exited the virtual space, the process returns to step S51 and the processes after step S51 are performed again.

[0254] Furthermore, when it is determined in step S51 that there is no photographic image being displayed and in step S52 that there is no photographic image being viewed, it is determined whether the virtual space being displayed is a standard virtual space (step S58). Here, the standard virtual space refers to the environment of the virtual space at the start of the display. As described above, in this embodiment, the virtual space provided is set based on the information of the current date and time of the region where the display terminal 10 is used. Therefore, it is determined whether the image of the virtual space being displayed is an image of the virtual space set based on the current date and time of the region where the display terminal 10 is used.

[0255] When the image of the virtual space being displayed is not a standard virtual space, it is changed to a standard virtual space (step S59). Therefore, when the user stops viewing the photographic image displayed in the virtual space (when the distance from the photographic image being viewed is greater than the threshold value R2), it is restored to the standard virtual space.

[0256] As described above, according to the image appreciation system of this embodiment, when there is an image shot in a real space, if the user approaches the shooting position of the photographic image, the photographic image is displayed at the shooting position. If the user approaches the displayed photographic image, the display is switched to a virtual space corresponding to the shooting environment of the photographic image. That is, the display is switched to a virtual space that reproduces the shot environment.

[0257] Furthermore, if the user moves away from the photographic image being viewed, the environment of the virtual space returns to the original environment. In addition, if the user leaves, the display of the photographic image disappears.

[0258] Thus, according to this embodiment, the virtual space provided changes according to the photographic image being viewed, thereby improving the sense of reality when viewing the photographic image, and thus enabling a better viewing experience.

[0259] [Modifications]

[0260] [Determination of photography environment]

[0261] As described above, the photographic environment can also be determined by image recognition. In this case, for example, the photographic environment can be determined using a pre-trained model that has been machine-learned to determine the photographic environment from an image.

[0262] Furthermore, it is also possible to configure the photographic environment of the photographic image to be determined by using both determination of the photographic environment by image recognition and determination of the photographic environment by analysis of incidental information.

[0263] Furthermore, in the above-mentioned embodiment, the season, weather and time period are determined as the photographic environment, but the information used for photographic environment determination is not limited to these. For example, wind direction, wind speed, etc. can be determined. Furthermore, each item can be classified in more detail for determination.

[0264] [Changes in Virtual Space]

[0265] In the above-mentioned embodiment, the environment of the virtual space is changed by switching the virtual space data used, but the method of changing the virtual space is not limited to this. In addition, for example, it is possible to change the environment of the virtual space by processing the image of the provided virtual space. For example, it is possible to change the environment of the virtual space by adding an effect to the image. For example, it is possible to prepare a plurality of effects corresponding to representative weather in advance, and add the weather effect corresponding to the weather of the photographic image being appreciated to the image of the virtual space to change the weather of the virtual space. In addition, the effect itself is a well-known technology, so its detailed description is omitted. For example, it is possible to add an effect to the image through filter processing.

[0266] Furthermore, the environment of the provided virtual space can be changed by switching the virtual space data used and combining the effects. For example, the season can be changed by switching the virtual space data, and the time period and weather can be changed by the effects.

[0267] [Method for judging images during appreciation]

[0268] [Judgment based on line of sight]

[0269] In the above embodiment, the photographic image displayed at the position closest to the user is regarded as the image being viewed, but the method of determining the image being viewed is not limited to this. For example, the photographic image existing in front of the user's line of sight may be regarded as the image being viewed and detected.

[0270] Fig. 22 This is a conceptual diagram for judging photographic images during appreciation.

[0271] Fig. 22 This is a diagram looking down at the virtual space. The symbol Pv0 in the figure represents the current position of the user (avatar) in the virtual space. The arrow DG represents the direction of the user's line of sight in the virtual space. The area FD represented by the oblique lines represents the user's gaze area. In other words, it represents the area where the user's line of sight is directed. In this example, the range of angles of ±α° in the horizontal direction based on the line of sight direction DG is set as the gaze area FD. The symbols Pi1~Pi3 represent the display positions of the photographic images in the virtual space. Fig. 22 The example shown is an example of a case where three photographic images are displayed in a virtual space.

[0272] In this example, the image within the attention area FD is regarded as the image being viewed by the user and is extracted from the captured image being displayed. Fig. 22 In the example shown, the photographic image displayed at the position Pi2 is regarded as the photographic image being viewed.

[0273] Furthermore, when there are a plurality of photographic images within the range of the radius R2, the photographic image displayed at the position closest to the user's current position Pv0 is regarded as the image being viewed by the user.

[0274] In this way, it is also possible to extract a photographic image that exists in front of the user's line of sight as an image being viewed.

[0275] In addition, in this example, the image in the fixation area FD (the image in the direction of the user's line of sight) is regarded as the image being viewed, but it can be further limited to the image in the specified range from the current position of the user. That is, it can be configured so that the image in the direction of the user's line of sight and within the specified distance range is regarded as the image being viewed.

[0276] [Judgment based on user selection operation]

[0277] It is also possible to configure such that the image being viewed is determined based on the selection operation of the user. In other words, it is possible to configure such that the image selected by the user's operation is determined as the image being viewed.

[0278] Fig.23 This is a diagram showing an example of an image selection operation performed by a user.

[0279] Fig.23 This is an example of a case where the hand Ha of the avatar, which is the user's avatar, is displayed in the virtual space and an image is selected with the hand Ha. The user selects an image by touching the photographic image IMO displayed in the virtual space with the hand Ha.

[0280] In this case, if the user touches the photographic image IMO with the hand Ha, the virtual space is changed to the virtual space corresponding to the photographic image IMO. If the user touches the photographic image IMO with the hand Ha again, the virtual space returns to the original environment. Alternatively, if the user moves away from the selected photographic image by a predetermined distance, the virtual space returns to the original environment. Alternatively, if another photographic image is selected, the virtual space is changed to the virtual space corresponding to the newly selected photographic image.

[0281] In this case, the image being viewed can be operated by the hand Ha displayed in the virtual space. For example, the image can be enlarged or reduced by performing an operation (gesture) of enlarging or reducing the image displayed.

[0282] [Display of photographic images]

[0283] Fig.24 This is a diagram showing another example of display of photographic images in a virtual space.

[0284] Fig.24An example is shown in which two photographic images IMO3 and IMO4 exist in association with the current position of the user. One photographic image IMO3 is an image being viewed, and the other photographic image IMO4 is an image not being viewed.

[0285] like Fig.24 As shown, in this example, the photographic image IMO3 being viewed and the photographic image IMO4 not being viewed are displayed in different display modes. Fig.24 In the example shown, an example is shown in which the image other than the photographic image being viewed is displayed in a semi-transparent state (the photographic image IMO4 is displayed in a semi-transparent state).

[0286] In this way, by displaying the photographic image IMO3 being viewed and the photographic image IMO4 not being viewed in different display modes, it is possible to easily distinguish the photographic images being viewed. Fig.24 As shown in FIG. 1 , by displaying the image other than the photographed image being viewed in a semi-transparent state, the image being viewed can be more easily seen.

[0287] Fig.25 This is a diagram showing another example of display of photographic images in a virtual space.

[0288] When a plurality of photographic images are displayed, it is also possible to configure the images to be grouped according to the photographed contents and displayed in groups.

[0289] Fig.25 This is an example of grouping images with the same photographing environment. An example of classifying into two groups is shown.

[0290] In this case, the position of the displayed image can be set to the shooting position of the image representing each group, for example. The image representing each group can be, for example, the image with the latest shooting date and time, the image with the earliest shooting date and time, the image with the most viewing times, etc. The user can set these arbitrarily.

[0291] In this way, by collectively displaying images of the same or similar photographic environments, it is possible to suppress frequent changes in the virtual space.

[0292] [Other embodiments]

[0293] [Analysis of image information]

[0294] It is possible to analyze the image information of the photographic image to determine the time when the photographic image was taken, and change to a virtual space corresponding to the time when the photographic image was taken. In this way, various information can be obtained from the image, and the virtual space can be changed based on the obtained information. In addition, as described above, the method of analyzing the image is not particularly limited, and various methods can be used.

[0295] Furthermore, when the display terminal 10 has a sound output function, it can be configured to output sound corresponding to the image information of the photographed image. For example, it can be configured to output BGM (Background Music) corresponding to the content of the photograph. For example, when a river is reflected in the image being appreciated, the gurgling sound of the river can be output. Furthermore, for example, it can be configured to prepare BGM for each virtual space data, and change the BGM in conjunction with the change of the virtual space.

[0296] [Display Terminal]

[0297] In the above embodiment, the display terminal 10 is described as an example in which the display terminal 10 is constituted by an HMD, but the structure of the display terminal is not limited thereto and may be configured so as to present an image of a virtual space on a non-wearable display such as a flat panel display.

[0298] Furthermore, in addition to the configuration in which the operation on the display terminal 10 is performed using a controller, it is also possible to set a configuration in which the operation is performed by gestures, voice input, etc.

[0299] [System Structure]

[0300] In the above embodiment, the image processing device 100 is configured to be arranged on a network, but the configuration of the image processing device 100 is not limited thereto. For example, the image processing device 100 may be directly connected to the display terminal 10 via a wire or wireless connection. Furthermore, the functions of the image processing device 100 may be given to the display terminal 10.

[0301] [Collection of photographic images]

[0302] In the above embodiment, the captured images are collected from a plurality of image transmission terminals 200 via the network 2, but the method of collecting or acquiring the captured images is not limited thereto. It is also possible to acquire images posted on SNS or the like.

[0303] [Photographic images]

[0304] Photographic images are not limited to so-called still images, but also include dynamic images. In the case of dynamic images, the photographic position is determined by using the accompanying information in the same way as for still images, and the image is displayed at a position corresponding to the determined photographic position. Furthermore, in the case of dynamic images, for example, the image of the first frame is displayed. Furthermore, in the case of dynamic images, it can be configured so that if it is determined to be in viewing, playback starts. The determination of whether it is in viewing is the same as that of still images. Furthermore, in the case of dynamic images, for example, it can be configured so that the virtual space is changed at the same time (including almost at the same time) as the playback of the image starts, and the virtual space is restored to the original space at the end of playback.

[0305] Furthermore, when both a static image and a dynamic image are displayed in a virtual space as photographic images related to the current position of the user, it is preferable to display the two images separately. For example, regarding dynamic images, they can be distinguished by displaying them with a predetermined mark. Furthermore, preview playback or summary playback can be performed. Preview playback refers to a function of playing the first few seconds. Summary playback refers to a function of shortening a dynamic image and playing it.

[0306] [Hardware Structure of Image Processing Device]

[0307] The functions implemented by the image processing device are implemented by various processors. The various processors include general-purpose processors such as CPU and / or GPU (Graphic Processing Unit), FPGA (Field Programmable Gate Array) and other processors that can change the circuit structure after manufacturing, such as programmable logic devices (PLD), ASIC (Application Specific Integrated Circuit) and other processors with circuit structures specially designed for performing specific processing, such as dedicated circuits. Program and software have the same meaning.

[0308] One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types. For example, one processing unit may be composed of multiple FPGAs or a combination of a CPU and an FPGA. In addition, multiple processing units may be composed of one processor. As an example of multiple processing units composed of one processor, first, there are the following methods: as represented by a computer used for a client and a server, one processor is composed of a combination of one or more CPUs and software, and the processor functions as multiple processing units. Second, there are the following methods: as represented by a system on chip (SoC), a processor that uses one IC (Integrated Circuit) chip to implement the functions of the entire system including multiple processing units. In this way, various processing units are composed of more than one of the above-mentioned various processors in terms of hardware structure.

[0309] Explanation of symbols

[0310] 1-image appreciation system, 2-network, 10-display terminal, 11-control unit, 11A-position recognition unit, 11B-line of sight direction recognition unit, 11C-communication control unit, 11D-display control unit, 12-communication unit, 13-operation unit, 14-sensor unit, 15-display unit, 16-sound input unit, 17-sound output unit, 100-image processing device, 100A-photographic image acquisition unit, 100B-photographic image management unit, 100C-user information acquisition unit, 100D-viewing area calculation unit, 100E-photographic image retrieval unit, 100F-virtual space data selection unit, 100G-display image generation unit, 100H-appreciation image determination unit, 100I-image analysis unit, 111-CPU, 112-ROM, 113-RAM, 114-auxiliary storage device, 114A-photographic image database, 114B-virtual space database, 115- Input device, 116-output device, 117-communication interface, 200-image sending terminal, DG-line of sight direction, FD-gaze area, FV-area in virtual space, FR-area in real space corresponding to area FV, IMO-photographic image, IMO1-photographic image, IMO2-photographic image, IMO3-photographic image, IMO4-photographic image, IMV-image of virtual space (displayed image), L-optical axis, Mp-azimuth map, Mv-mark indicating photographic position, Pr-photographic position, Pv0-current position, U-user (avatar), Ha-hand of user (avatar), S1~S5-virtual space providing step, S11~S21-virtual space providing step, S31~S34-processing step of displaying photographic image, S41~S44-processing step of displaying photographic image, S51~S59-processing step related to change of virtual space.

Claims

1. An image processing device comprising a processor, The processor is composed of: Acquiring position information in a virtual space that reproduces the real space; Based on a first image associated with the position information in a group of images captured in the real space, displaying an object image representing the first image in the virtual space; and A second image of the virtual space viewed from a viewpoint corresponding to the position information is displayed on a display unit.

2. The image processing device according to claim 1, wherein: The processor is composed of: The virtual space is changed to correspond to the image information of the first image.

3. The image processing device according to claim 1, wherein: The first image is an image captured within a range of a first distance from a position in the real space corresponding to the position information in the virtual space.

4. The image processing device according to claim 3, wherein: The first image is also an image captured within a range in the real space corresponding to a range of the second image in the virtual space.

5. The image processing device according to claim 1, wherein: The processor is composed of: The object image is displayed at a position in the virtual space corresponding to the position in the real space where the first image was captured.

6. The image processing device according to claim 2, wherein: The processor is composed of: When the position indicated by the position information moves to a position within a second distance range from the display position of the target image, the virtual space is changed to the virtual space corresponding to the image information of the first image.

7. The image processing device according to any one of claims 1 to 6, wherein: The processor is composed of: The target image is displayed in a manner aligned with the direction in which the first image was captured.

8. The image processing device according to any one of claims 1 to 6, wherein: The processor is composed of: The target image is displayed in a size corresponding to the viewing angle of the first image.

9. The image processing apparatus according to claim 8, wherein: The processor is composed of: When the viewing angle of the first image exceeds a threshold, displaying the object image in a size corresponding to the viewing angle; and When the viewing angle of the first image is equal to or smaller than the threshold, the target image is displayed in a predetermined size.

10. The image processing apparatus according to claim 2, wherein: The processor is composed of: determining a photographing environment of the first image based on the image information of the first image; and The virtual space is changed to correspond to the determined photographic environment.

11. The image processing apparatus according to claim 10, wherein: The processor is composed of: At least one of season, weather, and time period is determined as the photographing environment.

12. The image processing device according to claim 2, wherein: The processor is composed of: The virtual space data is changed to construct a different virtual space.

13. The image processing apparatus according to claim 2, wherein: The processor is composed of: Applying effects to modify the virtual space.

14. The image processing apparatus according to claim 2, wherein: The processor is composed of: When there are a plurality of the first images, displaying a plurality of the object images in the virtual space; receiving a selection of the object image; and The virtual space is changed to the virtual space corresponding to the image information of the first image represented by the selected target image.

15. The image processing apparatus according to claim 14, wherein: The processor is composed of: The object image existing in the line of sight direction is regarded as the selected object image, and the selection of the object image is received.

16. The image processing device according to claim 14 or 15, wherein: The first image is an image captured within a range of a first distance from a position in the real space corresponding to the position information in the virtual space.

17. The image processing apparatus according to claim 16, wherein: The first image is also an image captured within a range in the real space corresponding to a range of the second image in the virtual space.

18. The image processing device according to claim 14 or 15, wherein: The processor is composed of: The selected object image and the other object images are displayed in different display modes.

19. An image processing method comprising the following steps: Acquiring position information in a virtual space that reproduces a real space; Based on a first image associated with the position information in a group of images captured in the real space, displaying an object image representing the first image in the virtual space; and A second image of the virtual space viewed from a viewpoint corresponding to the position information is displayed on a display unit.

20. An image processing program that enables a computer to implement the following functions: Acquiring position information in a virtual space that reproduces a real space; Based on a first image associated with the position information in a group of images captured in the real space, displaying an object image representing the first image in the virtual space; and A second image of the virtual space viewed from a viewpoint corresponding to the position information is displayed on a display unit. 21 . A recording medium which is a non-transitory computer-readable recording medium and has recorded thereon the program according to claim 20 .

Images