Image processing device, method, and program

Abstract

Provided are an image processing device, a method, and a program that make it possible to provide a good image viewing space. The image processing device acquires position information in a virtual space in which a real space is reproduced. The image processing device displays, on the basis of a first image related to the position information among an image group captured in the real space, an object image representing the first image in the virtual space, and displays on a display unit a second image of the virtual space as observed from a viewpoint corresponding to the position information.

Description

Image processing device, method and program

[0001] The present invention relates to an image processing device, method, and program, and more particularly to an image processing device, method, and program for providing a virtual space.

[0002] Patent Documents 1 to 4 describe techniques for viewing images captured in real space in a virtual space.

[0003] JP 2009-296248 A JP 2011-186565 A JP 2012-252716 A JP 2022-68642 A

[0004] One embodiment of the technique of the present disclosure provides an image processing device, method, and program that can provide a good image viewing space.

[0005] (1) An image processing device comprising a processor, the processor being configured to acquire position information within a virtual space that reproduces a real space, and based on a first image associated with the position information among a group of images taken in the real space, display an object image representing the first image within the virtual space, and display a second image of the virtual space observed from a viewpoint corresponding to the position information on a display unit.

[0006] (2) The image processing device of (1), wherein the processor is configured to change the virtual space to correspond to the image information of the first image.

[0007] (3) An image processing device according to (1) or (2), wherein the first image is an image captured within a first distance from a position in real space corresponding to the position information in the virtual space.

[0008] (4) The image processing device of (3), wherein the first image is further an image captured within a range in real space corresponding to the range of the second image in virtual space.

[0009] (5) An image processing device according to any one of (1) to (4), wherein the processor is configured to display an object image at a position in a virtual space corresponding to the position in a real space where the first image was taken.

[0010] (6) An image processing device according to (2), wherein the processor is configured to change the virtual space to correspond to the image information of the first image when the position indicated by the position information moves to a position within a second distance range from the display position of the object image.

[0011] (7) The image processing device according to any one of (1) to (6), wherein the processor is configured to display the object image in accordance with the orientation in which the first image was captured.

[0012] (8) The image processing device according to any one of (1) to (7), wherein the processor is configured to display the object image at a size corresponding to the angle of view of the first image.

[0013] (9) An image processing device according to any one of (1) to (7), wherein the processor is configured to display the object image at a size corresponding to the angle of view of the first image when the angle of view of the first image exceeds a threshold, and to display the object image at a specified size when the angle of view of the first image is equal to or less than the threshold.

[0014] (10) An image processing device according to (2), wherein the processor is configured to determine the shooting environment of the first image based on image information of the first image and change the virtual space to correspond to the determined shooting environment.

[0015] (11) The image processing device according to (10), wherein the processor is configured to determine at least one of the season, weather, and time of day as the shooting environment.

[0016] (12) The image processing device of (2), (10), or (11), wherein the processor is configured to convert the virtual space data into virtual space data that constructs a different virtual space.

[0017] (13) The image processing device of (2), (10), (11), or (12), wherein the processor is configured to apply effects to modify the virtual space.

[0018] (14) An image processing device according to (2), (10), (11), (12) or (13), wherein the processor is configured to, when there are multiple first images, display multiple object images in the virtual space, accept selection of an object image, and change the virtual space to correspond to the image information of the first image represented by the selected object image.

[0019] (15) The image processing device of (14), wherein the processor is configured to accept a selection of an object image by regarding an object image present in the line of sight direction as the selected object image.

[0020] (16) An image processing device according to (2), (10), (11), (12), (13), (14) or (15), wherein the first image is an image taken within a first distance from a position in real space corresponding to position information in virtual space.

[0021] (17) The image processing device of (16), wherein the first image is further an image captured within a range in real space corresponding to the range of the second image in virtual space.

[0022] (18) An image processing device according to (2), (10), (11), (12), (13), (14), (15), (16) or (17), wherein the processor is configured to display the selected object image and other object images in different display modes.

[0023] (19) An image processing method including the steps of: acquiring position information within a virtual space that reproduces a real space; displaying an object image representing a first image within the virtual space based on a first image associated with the position information from among a group of images taken in the real space; and displaying a second image of the virtual space observed from a viewpoint corresponding to the position information on a display unit.

[0024] (20) An image processing program that causes a computer to realize the following functions: a function of acquiring position information within a virtual space that reproduces real space; a function of displaying an object image representing a first image within the virtual space based on a first image associated with the position information from a group of images taken in the real space; and a function of displaying a second image of the virtual space observed from a viewpoint corresponding to the position information on a display unit.

[0025] FIG. 1 is a diagram showing an overview of an image viewing system. FIG. 1 is a diagram showing an example of the system configuration of an image viewing system. Block diagram showing an example of the configuration of a display terminal. Block diagram of main functions of a control unit of a display terminal. FIG. 1 is a diagram showing an example of the hardware configuration of an image processing device. Block diagram of main functions of an image processing device. Conceptual diagram of image search. FIG. 1 is a diagram showing an example of virtual space data stored in a virtual space database. FIG. 1 is a diagram showing an example of an image of a virtual space displayed on the display unit of a display terminal. Flowchart showing the procedure for providing a virtual space by an image processing device. Flowchart showing the procedure for providing a virtual space by an image processing device. Conceptual diagram of displaying a captured image. Flowchart showing the procedure for processing the display of a captured image. Conceptual diagram of displaying a captured image. Flowchart showing the procedure for processing the display of a captured image.

[0026] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0027] [First Embodiment] In recent years, social networking services (SNS) have become widespread, and images taken at tourist spots and the like are now published on a timeline close to real time. Therefore, even if you do not actually visit a tourist spot, you can indirectly check the local conditions by viewing these images.

[0028] Meanwhile, in recent years, advances in technologies such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality) have made it possible to create digital twins, which recreate real spaces (actual spaces, real-world spaces) in virtual spaces.

[0029] This disclosure provides a new image viewing system that utilizes digital twins.

[0030] [Overview] FIG. 1 is a diagram showing an overview of an image viewing system according to the present disclosure.

[0031] For example, a virtual space that reproduces a real space such as a tourist spot is provided to a user via a head-mounted display (HMD) or the like. The user can move freely in the virtual space just as in the real space. If an image taken in the real space exists, when the user approaches a position in the virtual space that corresponds to the position in the real space where the image was taken, the image is displayed in the virtual space.

[0032] [Image Viewing System] FIG. 2 is a diagram showing an example of the system configuration of an image viewing system.

[0033] As shown in the figure, the image viewing system 1 of this embodiment is configured to include a display terminal 10 that displays images of a virtual space, an image processing device 100 that provides images of the virtual space to the display terminal 10, and an image sending terminal 200 that sends images captured in real space to the image processing device 100. The display terminal 10 and the image processing device 100 are connected to each other so as to be able to communicate with each other via a network 2. The image sending terminal 200 and the image processing device 100 are also connected to each other so as to be able to communicate with each other via the network 2.

[0034] [Image Transmission Terminal] The image transmission terminal 200 is configured, for example, by a computer with a communication function (for example, a personal computer), a mobile terminal (for example, a smartphone, a mobile phone, a tablet terminal), an imaging device with a communication function (for example, a digital camera), etc. These configurations themselves are well known, so detailed explanations thereof will be omitted.

[0035] As described above, the image sending terminal 200 sends (uploads) images taken in real space to the image processing device 100. The images to be sent may be images taken by other devices. If the image sending terminal itself has a shooting function (for example, a smartphone with a camera function), it can send images taken by the device itself to the image processing device 100.

[0036] The image transmitted from the image transmitting terminal 200 to the image processing device 100 is accompanied by at least information indicating the shooting location (shooting location information). The shooting location information is composed of information that can uniquely identify a location in real space. For example, the shooting location information can be composed of information on the latitude and longitude of the point where the image was taken. The shooting location information can further include altitude information. In addition to the shooting location information, it is preferable that the image be accompanied by information indicating the shooting date and time (shooting date and time information), information indicating the shooting direction (the direction in which the image was taken, the direction of the optical axis) (shooting direction information), etc.

[0037] The manner in which the shooting location information and the like are attached to an image is not particularly limited. For example, the shooting location information and the like can be attached to the image as metadata. In a device that records images in EXIF ​​(Exchangeable Image File Format) format, the shooting location information and the like can be attached to the image as tag information. In particular, in a digital camera, mobile terminal, etc. equipped with a GPS (Global Positioning System), GPS information (latitude, longitude, altitude, etc.) at the time of shooting is automatically added to the captured image as shooting location information. In addition, in a digital camera, mobile terminal, etc. equipped with an electronic compass (geomagnetic sensor), information on the direction in which the image was taken is automatically added as shooting direction information.

[0038] [Display Terminal] In the present embodiment, the display terminal 10 is configured as, for example, a non-transparent HMD that is worn on the user's head and covers the user's field of vision with a display unit. By configuring the HMD to block the outside world when worn on the head, it is possible to increase the sense of virtual reality and immersion when viewing.

[0039] FIG. 3 is a block diagram showing an example of the configuration of the display terminal.

[0040] As shown in the figure, the display terminal 10 includes a control unit 11 , a communication unit 12 , an operation unit 13 , a sensor unit 14 , a display unit 15 , an audio input unit 16 , and an audio output unit 17 .

[0041] The control unit 11 functions as an arithmetic processing unit and a control device, and controls the overall operation of the display terminal 10 in accordance with various programs. The control unit 11 is configured, for example, by a computer equipped with a processor and memory. The processor is realized by an electronic circuit such as a CPU (Central Processing Unit). The memory includes a ROM (Read Only Memory) that stores programs and various data, a RAM (Random Access Memory) used as a work area, a flash memory, etc.

[0042] The communication unit 12 is connected to the network 2 by wire or wirelessly, and communicates with the image processing device 100 on the network. The communication unit 12 is connected to the network 2 by, for example, a wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network 5G (5th Generation / fifth generation mobile communication system), 4G (4th Generation / fourth generation mobile communication system), LTE (Long Term Evolution), or the like.

[0043] The operation unit 13 accepts operation instructions from the user and outputs the operation content to the control unit 11. In this embodiment, movement within the virtual space is performed by operating the operation unit 13. The movement operation can be configured to be performed, for example, by an operation device that can input directions. For example, a hand controller, a joystick, or the like can be used. In addition, the operation unit 13 can include known operation devices such as a push switch, a lever, a volume control, a pedal switch, a keyboard, a mouse, a trackpad, a trackball, and a gesture input device.

[0044] The sensor unit 14 detects at least the tilt (tilt on the X, Y, and Z axes) of the headset or goggles (the part worn on the head) (3 Degrees of Freedom; 3 DoF / 3 degrees of freedom). In other words, it detects the orientation of the head. The sensor unit 14 may also be configured to detect the position of the headset or goggles in three-dimensional space (6 DoF / 6 degrees of freedom). In other words, it detects the position of the head in addition to the orientation. Note that this type of sensing technology is well known, so detailed description thereof will be omitted. As an example, a configuration may be adopted in which a gyro sensor, an acceleration sensor, a geomagnetic sensor, etc. are used to detect the tilt and position of the headset or goggles in three-dimensional space. The sensor unit 14 may also include a biosensor that detects the user's biometric information (e.g., pulse, heart rate, sweat, blood pressure, body temperature, respiration, electromyography, brain waves, etc.), a gaze detection sensor that detects the user's gaze inside the headset or goggles, etc. The sensing information detected by the sensor unit 14 is output to the control unit 11.

[0045] When the display terminal 10 is configured as an HMD, the display unit 15 has a left-eye screen and a right-eye screen corresponding to the user's left and right eyes, and displays images corresponding to the respective screens (left-eye image and right-eye image). The screen of the display unit 15 is configured, for example, by a display panel such as an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display, or a laser scanning display such as a retinal direct imaging display.

[0046] The voice input unit 16 is configured by, for example, a microphone, and collects voices uttered by the user.

[0047] The audio output unit 17 is configured by, for example, headphones, earphones, etc., and reproduces audio signals.

[0048] FIG. 4 is a block diagram showing the main functions of the control unit of the display terminal.

[0049] As shown in the figure, the control unit 11 of the display terminal 10 functions as a position recognition unit 11A, a line-of-sight direction recognition unit 11B, a communication control unit 11C, a display control unit 11D, and the like.

[0050] The position recognition unit 11A recognizes the user's current position (user position) in the virtual space based on operation information from the operation unit 13. For example, the position recognition unit 11A recognizes the user's position from the direction and amount of movement from an origin position set in the virtual space. Positions in the virtual space correspond one-to-one to positions in the real space.

[0051] The gaze direction recognition unit 11B recognizes the current gaze direction of the user in the virtual space based on the state of the head (orientation, position, etc.) detected by the sensor unit 14.

[0052] The communication control unit 11C controls communication with the image processing device 100. Current position information of the user recognized by the position recognition unit 11A (user position information) and information on the user's current gaze direction recognized by the gaze direction recognition unit (user gaze information) are transmitted to the image processing device 100 via the communication unit 12 under the control of the communication control unit 11C. As will be described later, the image processing device 100 generates an image to be displayed on the display unit 15 based on information transmitted from the display terminal 10. The communication control unit 11C receives a display image transmitted from the image processing device 100 via the communication unit 12.

[0053] The display control unit 11D controls the display of the display unit 15. The display image received from the image processing device 100 is displayed on the display unit 15 under the control of the display control unit 11D.

[0054] [Image Processing Apparatus] FIG. 5 is a diagram illustrating an example of the hardware configuration of an image processing apparatus.

[0055] As shown in the figure, the image processing apparatus 100 includes a CPU 111, a ROM 112, a RAM 113, an auxiliary storage device 114, an input device 115, an output device 116, and a communication interface (I / F) 117. Generally, this type of configuration can be realized by a computer.

[0056] The image processing apparatus 100 functions as an image processing apparatus when a CPU 111, which is a processor, executes a predetermined program (image processing program). The program executed by the CPU 111 is stored in the ROM 112 or the auxiliary storage device 114.

[0057] The auxiliary storage device 114 constitutes a storage unit of the image processing device 100. The auxiliary storage device 114 is constituted by, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0058] The input device 115 constitutes an operation unit of the image processing device 100. The input device 115 is constituted by, for example, a keyboard, a mouse, a touch panel, and the like.

[0059] The output device 116 constitutes a display unit of the image processing device 100. The output device 116 is constituted by, for example, a KCD, an OLED display, or the like.

[0060] The communication interface 117 is connected to the network 2 by wire or wirelessly, and communicates with the image processing device 100 and the image sending terminal 200 on the network.

[0061] FIG. 6 is a block diagram showing the main functions of the image processing apparatus.

[0062] The image processing device 100 mainly has a function of collecting captured images from the image sending terminal 200 and a function of providing a virtual space to the display terminal 10 .

[0063] With regard to the function of collecting photographed images from the image sending terminal 200, the image processing device 100 functions as a photographed image acquiring section 100A and a photographed image managing section 100B.

[0064] The photographed image acquiring unit 100A acquires photographed images transmitted from the image transmitting terminal 200 via the network 2.

[0065] The captured image management unit 100B stores the captured images acquired by the captured image acquisition unit 100A in a captured image database (Data Base; DB) 114A. The captured image database 114A is saved in, for example, the auxiliary storage device 114. As described above, information on the shooting location (shooting location information) is added to the captured images sent from the image sending terminal 200. The captured image management unit 100B stores the captured images in the captured image database 114A in association with the shooting location information. The captured image management unit 100B also numbers each captured image and stores them in the captured image database 114A. The multiple captured images stored in the captured image database 114A are an example of a group of images captured in real space.

[0066] Regarding the function of providing a virtual space to the display terminal 10, the image processing device 100 functions as a user information acquisition unit 100C, a field of view calculation unit 100D, a captured image search unit 100E, a virtual space data selection unit 100F, and a display image generation unit 100G, etc.

[0067] The user information acquisition unit 100C acquires information on the user's current state (user information). The user information includes user position information (information on the user's current position in the virtual space) and user line of sight information (information on the direction of the user's current line of sight in the virtual space). This information is acquired from the display terminal 10 via the network 2. The acquired information is added to the captured image search unit 100E and the field of view calculation unit 100D. The user position information is an example of position information in a virtual space that reproduces a real space.

[0068] The visual field calculation unit 100D calculates the visual field (field of view) of the user in the virtual space based on user information (user position information and user line of sight information). More specifically, it calculates the visual field of an avatar, which is the user's alter ego in the virtual space. This visual field corresponds to the display range of the image of the virtual space to be displayed on the display unit 15. Information on the calculated visual field (display range information) is sent to the captured image search unit 100E and the display image generation unit 100G.

[0069] The captured image search unit 100E searches the captured image database 114A for captured images (related images) related to the user's current location based on the user's current location information (user location information) in the virtual space and information on the user's current field of view.

[0070] FIG. 7 is a conceptual diagram of image search.

[0071] The figure shows a planar view of a virtual space and a real space corresponding to the virtual space. In the figure, the symbol Pv0 indicates the position of the user (avatar) in the virtual space. The symbol Pr0 indicates a position in the real space corresponding to the position Pv0 in the virtual space. The symbols Pr1 to Pr8 indicate positions in the real space where images were captured (the positions where the images captured at those positions exist). The symbols Pv1 to Pv8 indicate positions in the virtual space corresponding to the positions Pr1 to Pr8 in the real space. The arrow DG indicates the direction of the user's line of sight in the virtual space. The hatched area FV indicates an area within a radius R1 based on the user's current position Pv0 in the virtual space, and within the user's current field of view (field of vision). In this example, the user's horizontal field of view in the virtual space is a range of 100° to the left and right (200° overall) based on the line of sight direction DG. The shaded area FR is an area in real space that corresponds to the area FV in virtual space.

[0072] In this embodiment, the captured image search unit 100E searches for images captured within a radius R1 (less than or equal to the distance threshold R1) based on a position Pr0 in real space corresponding to the user's current position Pv0 in the virtual space, and within the current field of view. For example, if the user's current position in the virtual space is XX degrees XX minutes XX seconds north latitude and XX degrees XX minutes XX seconds east longitude, the unit searches for images captured within a circle of radius R1 from the position XX degrees XX minutes XX seconds north latitude and XX degrees XX minutes XX seconds east longitude, and within the current field of view (searching for images captured within the real space region FR corresponding to the virtual space region FV). In the example shown in FIG. 7, the images captured at positions Pr1, Pr3, Pr5, and Pr7 are images captured within the radius R1. Among these, the images captured at positions Pr1 and Pr3 are images captured within the range corresponding to the current field of view. Therefore, images taken at positions Pr1 and Pr3 are searched for.

[0073] The retrieved captured image is added to the display image generating unit 100G. The retrieved captured image is an example of a first image related to the location information. The range of radius R1 is also an example of a range of the first distance.

[0074] Furthermore, the range of region FR is an example of a range of a first distance from a position in real space corresponding to the position information in virtual space, and a range in real space corresponding to the range of a second image in virtual space.

[0075] The virtual space data selection unit 100F selects the virtual space data to be used. A plurality of pieces of virtual space data are stored in advance in the virtual space database 114B. Therefore, the virtual space data selection unit 100F selects the virtual space data to be used from the plurality of pieces of virtual space data stored in the virtual space database 114B. The virtual space database 114B is stored in the auxiliary storage device 114, for example.

[0076] 8 is a diagram showing an example of virtual space data stored in the virtual space database, which shows an example of virtual space data for providing multiple types of virtual spaces with different environments for one real space.

[0077] As shown in the figure, multiple sets of virtual space data are prepared for different combinations of seasons (spring, summer, autumn, winter), weather conditions (sunny, cloudy, rainy), and time periods (morning, noon, evening, night). Each set of virtual space data is a reproduction of the same real space, but with different seasons, weather conditions, and time periods.

[0078] In this way, the virtual space database 114B is configured to record multiple types of virtual space data for one real space, allowing virtual spaces with different environments to be constructed. The virtual space database 114B records virtual space data that can reproduce at least one real space. Each piece of virtual space data is recorded in association with information about the real space to be reproduced.

[0079] The real space to be reproduced (the real space provided to the user as a virtual space) is selected by the user, for example. The virtual space data to be used is selected, for example, based on the current date and time of the region where the display terminal 10 is used. In Japan, the season of the virtual space data to be used is selected based on the current date and time, with March to May being spring, June to August being summer, September to November being autumn, and December to February being winter. The time period of the virtual space data to be used is also selected based on the current date and time, with 6:00 AM to 9:00 AM being morning, 9:00 AM to 3:00 PM being daytime, 3:00 PM to 6:00 PM being evening, and 6:00 PM to 6:00 AM the next day being night. For example, if the region where the display terminal 10 is used is Japan and the current date and time is 12:00 PM on January 1st, virtual space data corresponding to "daytime" in "winter" is selected. The default weather setting (e.g., sunny) is selected. Therefore, in this case, "JP040102" is selected as the virtual space data (assuming the default setting for "weather" is "sunny"). The weather may also be selected randomly. The current date and time information is obtained from the system or the display terminal 10. The selected virtual space data is added to the display image generation unit 100G. Alternatively, predetermined virtual space data or virtual space data selected by the user may be selected.

[0080] The display image generation unit 100G generates an image of the virtual space (display image) to be provided to the display terminal 10 based on the virtual space data selected by the virtual space data selection unit 100F and the information on the visual field calculated by the visual field calculation unit 100D. This display image is an image of the virtual space observed from the viewpoint of the user at the user's (avatar's) current position in the virtual space.

[0081] In this embodiment, if a captured image (related image) related to the user's current location exists, the captured image is displayed in the virtual space. Therefore, if a captured image related to the user's current location exists, the display image generating unit 100G generates a display image in which the captured image is displayed in the virtual space.

[0082] FIG. 9 is a diagram showing an example of an image of the virtual space displayed on the display unit of the display terminal.

[0083] The image (display image) IMV of the virtual space displayed on the display unit 15 of the display terminal 10 is an image obtained by cutting out the field of view area from the image of the virtual space constructed from the virtual space data.

[0084] In this embodiment, the captured image IMO is displayed at a position Pv in the virtual space corresponding to the shooting position Pr in the real space. The captured image IMO is displayed in the virtual space at a predetermined size. That is, it is placed in the virtual space as a planar object of a predetermined size and is visually recognized by the user. Therefore, the closer the user is to the display position (shooting position), the larger the image is displayed. The captured image IMO is displayed at a predetermined height from the ground (for example, approximately the height of an adult's line of sight). Therefore, the captured image IMO is displayed floating in the air in the virtual space. The captured image IMO is displayed facing the user (avatar). In this embodiment, the captured image IMO displayed in the virtual space is an example of an object image.

[0085] The display image generated by the display image generation unit 100G is transmitted to the display terminal 10 via the network 2. The display terminal 10 receives the display image transmitted from the image processing device 100 and displays it on the display unit 15. This allows the user to view an image of the virtual space that changes in conjunction with the user's movements. In this embodiment, the display image is an example of a second image.

[0086] [Operation of Image Processing Device (Image Processing Method)] Figures 10 and 11 are flowcharts showing the procedure for providing a virtual space by an image processing device. Figure 10 shows the procedure up to displaying an image of the virtual space on the display unit 15 of the display terminal 10. Figure 11 shows the operation procedure after display has started.

[0087] 10 , first, information on the real space to be provided is acquired (step S1). In this embodiment, for example, information on real spaces that can be provided as virtual spaces is displayed in a list on the display unit 15 of the display terminal 10. The user selects a desired real space from the displayed list. The selected information is transmitted to the image processing device 100 as information on the real space to be provided.

[0088] Next, information on the current date and time is acquired (step S2). As described above, information on the current date and time is acquired from the system or the display terminal 10.

[0089] Next, virtual space data for the virtual space to be provided is selected based on the acquired real space information and current date and time information (step S3). As described above, for example, if the current date and time is 12:00 on January 1st, virtual space data corresponding to "daytime" in "winter" is selected. Note that the weather setting is selected as a default.

[0090] Next, a display image is generated based on the selected virtual space data (step S4). That is, an image of the virtual space to be displayed on the display unit 15 of the display terminal 10 is generated. This image is generated based on a reference position (origin position or start position) previously set in the virtual space. It is also generated based on a reference line of sight direction. That is, it is generated as an image that is observed when facing a predetermined direction at the reference position.

[0091] The generated display image is provided to the display terminal 10 via the network 2 and displayed on the display unit 15 of the display terminal 10 (step S5).

[0092] When the display starts, it is determined whether or not the state of the user has changed (step S11), as shown in Fig. 11. That is, it is determined whether or not the user has moved or changed the direction of their line of sight.

[0093] When a change in state is detected, a captured image (related image) related to the user's current location is searched for, and the searched related image is displayed in the virtual space. The related image is searched for in the captured image database 114A. The search and display are performed in the following procedure.

[0094] First, i=1 is set (step S12), and the i-th photographed image is selected from the photographed image database 114A as a processing target (step S13).

[0095] Next, it is determined whether the selected captured image (selected image) is a related image (step S14). It is determined whether the image was captured within a radius R1 based on a position in real space corresponding to the user's current position in the virtual space, and whether it was captured within the current field of view. In this case, it is first determined whether the selected image was captured within the radius R1. If the image was not captured within the radius R1, it is determined that it is not a related image. On the other hand, if the image was captured within the radius R1, it is next determined whether the image was captured within the current field of view. If the image was not captured within the current field of view, it is determined that it is not a related image. On the other hand, if the image was captured within the current field of view, it is determined that it is a related image.

[0096] If it is determined that the selected image is a related image, it is then determined whether or not the selected image is currently being displayed (step S15), i.e., whether or not the selected image is already being displayed in the virtual space.

[0097] If the selected image is already being displayed, it continues to be displayed. On the other hand, if the selected image is not being displayed, the selected image is displayed in the virtual space (step S16). The image is displayed at a position corresponding to the image capture position. That is, the image is displayed at a position in the virtual space corresponding to the image capture position in the real space. The image is also displayed in the virtual space at a predetermined size and facing the user (see FIG. 9).

[0098] If it is determined in step S14 that the selected image is not a related image, it is then determined whether the selected image is currently being displayed (step S20). That is, it is determined whether the selected image is currently being displayed in the virtual space despite not being a related image. If the selected image is currently being displayed, its display is terminated (step S21).

[0099] Thereafter, the value of i is incremented as i=i+1 (step S18), and it is determined whether i exceeds N (i>N) (step S19). That is, it is determined whether all the photographed images in the photographed image database 114A have been processed. N is the total number of photographed images in the photographed image database 114A.

[0100] If i>N is not true, that is, if there is a next photographed image to be processed, the process returns to step S13, and the above process is carried out again.

[0101] On the other hand, if i>N, that is, if all captured images in the captured image database 114A have been processed, it is determined whether or not the user has exited the virtual space (step S19). For example, it is determined whether or not an instruction to end the display has been issued by turning off the power, etc. Similarly, if it is determined in step S11 that there has been no change in the state, it is determined whether or not the user has exited the virtual space. If it is determined that the user has exited the virtual space, the processing ends. On the other hand, if it is determined that the user has not exited the virtual space, the process returns to step S11, and it is again determined whether or not there has been a change in the state.

[0102] With the above configuration, when a user moves within the virtual space, captured images are displayed one after another in the virtual space in conjunction with the user's movement. Each captured image is displayed at a position corresponding to the position in real space where the image was captured. This allows each captured image to be presented in a way that makes it easy to understand how it was captured. This also allows the user to deepen their understanding of the captured images and enjoy a better viewing experience. Furthermore, each captured image disappears when the user moves away from the display position (capture position) by more than a certain distance, so the display in the virtual space does not become cluttered.

[0103] [Modification] "Displaying an image according to the shooting direction" In the above embodiment, a captured image is displayed in a virtual space facing the user. If information about the shooting direction can be obtained from the captured image, the captured image may be displayed according to the direction in which it was taken.

[0104] FIG. 12 is a conceptual diagram of the display of a captured image.

[0105] 1A is a conceptual diagram of the display of a captured image IMO when information on the shooting direction cannot be obtained from the captured image IMO, and FIG. 1B is a conceptual diagram of the display of a captured image IMO when information on the shooting direction can be obtained from the captured image IMO.

[0106] 12A , when information about the shooting direction cannot be acquired from the captured image IMO, the captured image IMO is displayed facing the user (avatar) U. More specifically, the captured image IMO is displayed perpendicular to the depth direction of the screen.

[0107] On the other hand, if information on the shooting direction can be acquired from the captured image IMO, the captured image IMO is displayed in the virtual space in accordance with the shooting direction when the image was captured. Specifically, the captured image IMO is displayed perpendicular to the direction of the optical axis L when the image was captured.

[0108] The information on the shooting direction is acquired from information attached to the captured image (for example, tag information). Alternatively, the shooting direction may be estimated by image recognition.

[0109] FIG. 13 is a flowchart showing the procedure for processing the display of a captured image.

[0110] First, information on the shooting direction of the captured image to be displayed is acquired (step S31). As described above, the information on the shooting direction is acquired from information attached to the captured image. Alternatively, the information on the shooting direction is acquired by image recognition.

[0111] Next, it is determined whether or not there is information about the photographing direction (step S32), that is, whether or not the information about the photographing direction has been acquired.

[0112] If information on the shooting direction exists (if information on the shooting direction has been acquired), the captured image is displayed in the virtual space in accordance with the shooting direction (step S33). That is, as shown in Fig. 12(B), the captured image IMO is displayed perpendicular to the shooting direction (the direction of the optical axis L).

[0113] On the other hand, if there is no information about the shooting direction (if the information about the shooting direction cannot be acquired), the captured image is displayed in the virtual space in the normal display form (step S34). That is, as shown in FIG. 12A, the captured image IMO is displayed facing the user.

[0114] In this way, if information on the shooting direction can be obtained, the captured image can be displayed in a virtual space in accordance with the shooting direction, allowing users to gain a deeper understanding of how the displayed image was taken.

[0115] In addition, if information on the height from the ground when the photographed image was taken can be acquired, the photographed image may be displayed in the virtual space at the height when it was taken.

[0116] Note that some users may find the normal display format easier to view, so the user may be allowed to select whether or not to display the image in accordance with the shooting direction.

[0117] [Displaying an image according to the angle of view of the image] In the above embodiment, the image is displayed in a virtual space at a preset size. If information about the angle of view can be obtained from the image, the image may be displayed in the virtual space at a size corresponding to the angle of view.

[0118] FIG. 14 is a conceptual diagram of the display of a captured image.

[0119] 1A and 1B show examples of displaying two images captured at different angles of view. 1A shows an example of a case where the image is captured at a wider angle of view than 1B. More specifically, 1B shows an example where ω1>ω2, where ω1 is the angle of view of the captured image IMO1 shown in 1A and ω2 is the angle of view of the captured image IMO2 shown in 1B.

[0120] As shown in Figures 14(A) and (B), when viewed from the same position, the captured image IMO1 captured at a wide angle of view ω1 appears larger than the captured image IMO2 captured at a narrow angle of view ω2.

[0121] For images taken with cameras of the same sensor size, the shorter the focal length, the wider the angle of view. In other words, images taken with a wide-angle lens will have a wider (larger) angle of view, and images taken with a telephoto lens will have a narrower (smaller) angle of view.

[0122] If information on the angle of view is attached to the captured image, the information on the angle of view of the captured image is directly obtained from the attached information. If information on the angle of view is not attached to the captured image, the information is obtained using other information attached to the captured image. The angle of view can be calculated from the size of the image sensor and the focal length of the lens. Furthermore, for example, if information on the 35mm equivalent lens focal length is included in the attached information, the angle of view can be calculated from that information. For example, if the 35mm equivalent lens focal length is f=50mm, the angle of view is approximately 47°. Note that in this case, obtaining information on the 35mm equivalent lens focal length essentially means obtaining information on the angle of view.

[0123] FIG. 15 is a flowchart showing the procedure for processing the display of a captured image.

[0124] First, information on the angle of view of the captured image to be displayed is obtained (step S41). As described above, the information on the angle of view is obtained from information attached to the captured image. At this time, if the information on the angle of view cannot be obtained directly from the attached information, it is obtained by calculation.

[0125] Next, it is determined whether or not information on the angle of view exists (step S42), that is, it is determined whether or not the information on the angle of view has been acquired.

[0126] If the information on the angle of view exists (if the information on the angle of view can be acquired), the photographed image is displayed in the virtual space at a size according to the angle of view (step S43).

[0127] On the other hand, if the information on the angle of view does not exist (if the information on the angle of view cannot be acquired), the captured image is displayed in the virtual space at a specified display size (default display size) (step S44). That is, the captured image is displayed in the virtual space at a predetermined size.

[0128] In this way, if information on the angle of view can be obtained, the captured image is displayed in the virtual space at a size that corresponds to the angle of view. This allows you to understand the angle of view at which the displayed captured image was taken, and therefore you can correctly understand the size of the subject.

[0129] Note that, when the display size of a captured image is changed according to the angle of view, as in this example, the display size may become too large or too small depending on the image. For example, an image captured with a super telephoto lens may be displayed too small. Furthermore, an image captured with a super wide-angle lens (including a fisheye lens) may be displayed too large. Therefore, it is more preferable to set a limit on the display size. For example, images with an angle of view equal to or smaller than a threshold (lower limit position) may be displayed at a uniformly specified size (minimum display size). Similarly, images with an angle of view equal to or larger than a threshold (upper limit position) may be displayed at a uniformly specified size (maximum display size).

[0130] Also, some users may find it easier to view images when they are all displayed at the same size, so the user may be allowed to select whether or not to change the display size depending on the angle of view.

[0131] As in the above-described modified example, the captured image may be displayed in accordance with the orientation in which it was taken. In this case, the captured image is displayed in accordance with the orientation in which it was taken, and at a size that corresponds to the angle of view.

[0132] [Virtual Space Environment Setting] In the above embodiment, the environment of the virtual space to be provided to the user is determined based on information about the current date and time of the region in which the display terminal 10 is used. However, the method of determining the environment of the virtual space to be provided to the user is not limited to this. A virtual space with a predetermined environment may be provided to the user. Alternatively, the environment of the virtual space to be displayed may be freely selected and set by the user. For example, the season, weather, and time zone of the virtual space to be provided may all be selected and set by the user.

[0133] [Displaying a captured image in a virtual space] In the above embodiment, if a captured image (related image) related to the user's current location exists in the virtual space, the captured image is displayed at a position corresponding to the captured location, but the manner in which a captured image is displayed in a virtual space is not limited to this.

[0134] FIG. 16 is a diagram showing another example of the display of a captured image.

[0135] The figure shows an example in which, if related images exist, they are displayed together at a predetermined position in the virtual space. The example shown in the figure shows an example in which the captured images IMO are displayed in a vertical row in the right corner. In this case, as shown in the figure, marks Mv may be displayed at positions in the virtual space corresponding to the capture positions of each captured image IMO, so that the capture positions can be identified. In this case, each mark Mv may be displayed in a different color, and the corresponding captured image IMO may be displayed in a frame of the same color, so that the correspondence between each captured image IMO and each mark Mv can be identified.

[0136] FIG. 17 is a diagram showing another example of the display of a captured image.

[0137] The figure shows an example in which images taken within a radius R1 based on a position in real space corresponding to the user's current position in the virtual space are extracted as related images and displayed in the virtual space.

[0138] Images captured within the current field of view are displayed in the virtual space according to the capture position. On the other hand, images captured outside the field of view are displayed in a predetermined area. In the example shown in FIG. 17, the captured images IMO are displayed in a vertical row in the right corner. In this case, as shown in FIG. 17, a map MP may be displayed in the virtual space, and the approximate display position of each captured image may be indicated on the map MP.

[0139] Second Embodiment In the image viewing system 1 of the above embodiment, if an image captured in real space exists, and the user approaches the capture position of the image in virtual space, the captured image is displayed in the virtual space.

[0140] In the image viewing system 1 of this embodiment, the virtual space changes in accordance with the captured image that the user is viewing. For example, if the image that the user is viewing is an image of the autumn season, the season in the virtual space also changes to autumn.

[0141] Note that the image viewing system is the same as the image viewing system of the above embodiment except for the change in the virtual space, so only the differences will be described below.

[0142] FIG. 18 is a block diagram of the main functions of the image processing device regarding changes to the virtual space.

[0143] As shown in the figure, the image processing device 100 of this embodiment further includes the functions of a viewing image determination unit 100H and an image analysis unit 100I.

[0144] The viewing image determination unit 100H determines the captured image being viewed by the user from among the captured images displayed in the virtual space. The viewing image determination unit 100H determines the captured image being viewed based on the user's current position information (user position information) in the virtual space and information on the captured image being displayed in the virtual space.

[0145] FIG. 19 is a conceptual diagram of determination of a captured image during viewing.

[0146] The figure shows a plan view of the virtual space. The symbol Pv0 in the figure indicates the current position of the user (avatar) in the virtual space. The symbols Pi1 to Pi3 indicate the display positions of the captured images in the virtual space. The example shown in the figure is an example in which three captured images are displayed in the virtual space.

[0147] In this embodiment, images located within a radius R2 (less than or equal to the distance threshold R2) based on the user's current position Pv0 are considered to be images currently being viewed by the user, and are extracted from the displayed captured images. Radius R2 is set to a value smaller than radius R1 (R2<R1). In other words, an image displayed in virtual space is considered to be currently being viewed when it is closer than a certain distance.

[0148] If there are multiple captured images within the range of radius R2, the captured image displayed closest to the user's current position Pv0 among the multiple captured images displayed is considered to be the image the user is viewing.

[0149] 19, two captured images (a captured image at position Pi2 and a captured image at position Pi3) are displayed within a range of radius R2. Position Pi2 is closer to the user than position Pi3. Therefore, in this case, the captured image displayed at position Pi2 is the captured image currently being viewed.

[0150] Information about the captured image that is determined to be being viewed is added to the image analysis unit 1001. In this embodiment, the range of radius R2 is an example of the range of the second distance.

[0151] Note that determining the captured image being viewed is essentially the same as determining the captured image being selected by the user, that is, determining the image selected for viewing.

[0152] The image analysis unit 100I analyzes the image information of the captured image. That is, it analyzes the content of the captured image. In particular, in this embodiment, the image information is analyzed to determine the capture environment. The capture environment here refers to the environment of the real space in which the image was captured. The capture environment includes at least one of the season, weather, and time of day. In this embodiment, the season, weather, and time of day of the captured image are determined.

[0153] The analysis of image information includes not only image analysis but also analysis of incidental information. That is, it includes analyzing incidental information to determine the shooting environment. The analysis of image information by image analysis employs, for example, a method of determining the shooting environment by image recognition. In this embodiment, the shooting environment is determined by analyzing incidental information.

[0154] The season, weather, and time of day of a captured image can be roughly determined from the shooting location and shooting date and time. For example, if the image was taken in Japan, the time period can be determined from the shooting date and time by defining morning as 6:00 AM to 9:00 AM, daytime as 9:00 AM to 3:00 PM, evening as 3:00 PM to 6:00 PM, and night as 6:00 PM to 6:00 AM the next day. Similarly, if the image was taken in Japan, the season can be determined from the shooting date and time by defining spring as March to May, summer as June to August, autumn as September to November, and winter as December to February. The weather can be determined, for example, by referencing a database (weather database) that records past weather conditions in various locations. Therefore, if the captured image is accompanied by information about the shooting location and shooting date and time, the shooting environment (season, weather, and time of day) of the captured image can be determined from this information.

[0155] The image analysis unit 100I analyzes information (for example, meta-information) attached to the captured image and determines the image capturing environment. The determined information on the image capturing environment is sent to the virtual space data selection unit 100F.

[0156] If additional information cannot be acquired, the shooting environment cannot be determined, and in this case, it is deemed impossible to determine. Also, if a determination cannot be made for some items, it is deemed impossible to determine (no information) for that item. For example, if the weather cannot be determined, it is deemed impossible to determine the weather.

[0157] The virtual space data selection unit 100F selects the virtual space data to be used. As described above, the standard virtual space data is selected based on the current date and time of the area where the display terminal 10 is used. On the other hand, if there is a photographed image currently being viewed by the user, virtual space data corresponding to the photographed image is selected. The virtual space data selection unit 100F selects the virtual space data to be used based on the analysis results of the photographed image shooting environment by the image analysis unit 100I. In other words, the virtual space data to be used is selected based on the determined season, weather, and time of day of the photographed image. The virtual space data selection unit 100F searches the virtual space database 114B for the corresponding virtual space data and selects the virtual space data to be used. For example, if the season determined from the photographed image is "spring," the weather is "sunny," and the time of day is "daytime," the virtual space data "JP010102" is selected, as shown in FIG. 8.

[0158] If the shooting environment of the captured image being viewed by the user cannot be determined, the virtual space is not changed, and therefore, in this case, virtual space data is not selected either.

[0159] If the shooting environment cannot be determined for some items (if there is no information), a search is performed using a predetermined default setting for that item. For example, if the "season" cannot be determined (if there is no information), the corresponding virtual space data is searched for using the seasonal information set by default for the "season" (for example, "spring"). The default setting may be configured to be set arbitrarily by the user. Alternatively, for items that cannot be determined, the current setting may be retained. For example, if the "season" cannot be determined, the seasonal information of the currently displayed virtual space may be used to search for virtual space data.

[0160] Display image generation unit 100G generates an image of the virtual space (display image) to be provided to display terminal 10 based on the virtual space data selected by virtual space data selection unit 100F and the information on the field of view calculated by field of view calculation unit 100D. In this embodiment, if there is a captured image that the user is currently viewing, an image of the virtual space that matches the environment of the captured image is generated.

[0161] [Operation of Image Viewing System] FIG. 20 is a flowchart showing the procedure of processing related to changes in the virtual space.

[0162] First, it is determined whether or not there is a photographed image being displayed (step S51). That is, it is determined whether or not there is a photographed image being displayed in the currently displayed virtual space.

[0163] If there is a captured image being displayed, it is determined whether there is a captured image being viewed (step S52). In this embodiment, the presence or absence of an image being viewed is determined by determining whether there is an image within a radius R2 of the user's current position.

[0164] If there are multiple captured images within the range of radius R2, the captured image closest to the user's current position is determined to be the captured image currently being viewed.

[0165] If there is a photographed image being viewed, the image information of the photographed image being viewed is analyzed (step S53). In this embodiment, the photographing environment is determined by analyzing the supplementary information of the photographed image being viewed.

[0166] FIG. 21 is a flowchart showing the procedure for determining the shooting environment.

[0167] First, it is determined whether or not the season can be determined (step S53_1). In this embodiment, whether or not the season can be determined is determined based on the presence or absence of information on the photographing location and the photographing date and time. If the information on the photographing location and the photographing date and time can be acquired, it is determined that the season can be determined.

[0168] If the season can be determined, the season data is set to the determined content (step S53_2).On the other hand, if the season cannot be determined, the season data is set to "none" (step S53_3).

[0169] Next, it is determined whether or not the weather can be determined (step S53_4). In this embodiment, whether or not the weather can be determined is determined based on the presence or absence of information on the shooting location and shooting date and time. If the information on the shooting location and shooting date and time can be acquired, it is determined that the weather can be determined. As described above, the weather is determined from the shooting location and shooting date and time by referring to the weather database.

[0170] If the weather can be determined, the weather data is set to the determined content (step S53_5).On the other hand, if the weather cannot be determined, the weather data is set to "none" (step S53_6).

[0171] Next, it is determined whether or not the time period can be determined (step S53_7). In this embodiment, it is determined whether or not the time period can be determined depending on whether or not information on the shooting date and time is available.

[0172] If the time period can be determined, the time period data is set to the determined content (step S53_8).On the other hand, if the time period cannot be determined, the time period data is set to "none" (step S53_9).

[0173] Through the above series of steps, the shooting environment of the captured image being observed is determined. Virtual space data is selected based on the determined shooting environment (step S54). That is, based on the set season, weather, and time zone data, the corresponding virtual space data is searched for in the virtual space database 114B, and the virtual space data for the virtual space to be changed is selected. For example, if the season data is set to "spring," the weather data is set to "sunny," and the time zone data is set to "daytime," the virtual space data "JP010102" is selected, as shown in FIG. 8. For items with "none" data, default data is used. For example, if the season data is "spring," the weather data is "none," and the time zone data is "daytime," the default weather data (e.g., "sunny") is used to search for virtual space data.

[0174] When the virtual space data to be changed is selected, as shown in Fig. 20, it is determined whether or not a change to the virtual space is necessary (step S56). If the selected virtual space data is the same as the virtual space data of the currently provided virtual space, it is determined that no change is necessary. In this case, the virtual space is not changed and the current display continues. On the other hand, if the selected virtual space data is different from the virtual space data of the currently provided virtual space, it is determined that a change is necessary.

[0175] If it is determined that the virtual space needs to be changed, the virtual space to be provided is changed (step S56). In this case, an image of the virtual space to be provided to the display terminal 10 (display image) is generated based on the selected virtual space data and provided to the display terminal 10. The image of the virtual space provided is an image that reproduces the environment of the image the user is viewing. This can improve the sense of realism.

[0176] Thereafter, it is determined whether the user has exited the virtual space (step S57). If it is determined that the user has exited the virtual space, the process ends. On the other hand, if it is determined that the user has not exited the virtual space, the process returns to step S51, and the processes from step S51 onward are performed again.

[0177] Furthermore, if it is determined in step S51 that no captured image is being displayed, or if it is determined in step S52 that no captured image is being viewed, it is determined whether the virtual space being displayed is a standard virtual space (step S58). Here, the standard virtual space refers to the virtual space environment when the display is started. As described above, in this embodiment, the virtual space to be provided is set based on information about the current date and time of the area where the display terminal 10 is used. Therefore, it is determined whether the image of the virtual space being displayed is an image of the virtual space set based on the current date and time of the area where the display terminal 10 is used.

[0178] If the image of the virtual space being displayed is not the standard virtual space, it is changed to the standard virtual space (step S59). Therefore, when the user stops viewing the captured image displayed in the virtual space (when the user moves away from the captured image being viewed by more than the distance threshold R2), the virtual space is returned to the standard virtual space.

[0179] As described above, according to the image viewing system of this embodiment, when an image captured in a real space exists, and the user approaches the capture position of the captured image, the captured image is displayed at the capture position. When the user approaches the displayed captured image, the display switches to a virtual space corresponding to the capture environment of the captured image. In other words, the display switches to a virtual space that reproduces the capture environment.

[0180] Furthermore, when the user moves away from the captured image being viewed, the environment of the virtual space is returned to the original environment, and when the user moves away, the display of the captured image is erased.

[0181] In this way, according to the present embodiment, the virtual space provided changes in accordance with the captured image being viewed, thereby improving the sense of realism when viewing the captured image, thereby providing an even better viewing experience.

[0182] [Modification] [Determination of the shooting environment] As described above, the shooting environment can also be determined by image recognition. In this case, for example, the shooting environment can be determined using a trained model that has been machine-learned to determine the shooting environment from an image.

[0183] Furthermore, it is also possible to configure the system so that the shooting environment of a captured image is determined by using both the determination of the shooting environment by image recognition and the determination of the shooting environment by analysis of the supplementary information.

[0184] In the above embodiment, the information used to determine the shooting environment is the season, weather, and time of day, but the information used to determine the shooting environment is not limited to these. For example, wind direction, wind speed, etc. may also be determined. Each item may also be further classified and determined.

[0185] [Changing the Virtual Space] In the above embodiment, the environment of the virtual space is changed by switching the virtual space data used, but the method of changing the virtual space is not limited to this. Alternatively, for example, the environment of the virtual space can be changed by processing the image of the virtual space provided. For example, the environment of the virtual space can be changed by adding an effect to the image. For example, a plurality of effects corresponding to typical weather conditions can be prepared in advance, and a weather effect corresponding to the weather in the captured image being viewed can be added to the image of the virtual space to change the weather in the virtual space. Note that the effects themselves are well-known technology, so detailed description thereof will be omitted. For example, an effect can be added to an image by filter processing.

[0186] Furthermore, the environment of the virtual space provided can be changed by combining switching of the virtual space data to be used with effects. For example, the season can be handled by switching the virtual space data, and the time of day and weather can be handled by effects.

[0187] [Method for Determining the Image Being Viewed] [Determination Based on the Line of Sight] In the above embodiment, the captured image displayed closest to the user is deemed to be the image being viewed, but the method for determining the image being viewed is not limited to this. For example, a captured image that is in the user's line of sight may be deemed to be the image being viewed and detected.

[0188] FIG. 22 is a conceptual diagram of determination of a captured image during viewing.

[0189] The figure is a plan view of the virtual space. The symbol Pv0 in the figure indicates the current position of the user (avatar) in the virtual space. The arrow DG indicates the direction of the user's line of sight in the virtual space. The hatched area FD indicates the user's gaze area. In other words, it indicates the area where the user is directing their gaze and gazing. In this example, the gaze area FD is a range of ±α degrees in the horizontal direction based on the line of sight direction DG. The symbols Pi1 to Pi3 indicate the display positions of the captured images in the virtual space. The example shown in the figure is an example where three captured images are displayed in the virtual space.

[0190] In this example, the image located within the gaze area FD is regarded as the image currently being viewed by the user, and is extracted from the captured images currently being displayed. In the example shown in Figure 22, the captured image displayed at position Pi2 is regarded as the captured image currently being viewed.

[0191] If a plurality of captured images exist within the range of radius R2, the captured image displayed closest to the user's current position Pv0 is considered to be the image the user is viewing.

[0192] In this way, a captured image that is in the user's line of sight can be extracted and used as the image being viewed.

[0193] In this example, the image within the gaze area FD (the image the user is directing his / her gaze at) is considered to be the image being viewed, but the image being viewed may be determined by limiting it to images located within a predetermined range from the user's current position. In other words, the image being viewed may be considered to be an image that exists in the direction of the user's gaze and within a predetermined distance.

[0194] [Determination Based on User's Selection Operation] A configuration may be adopted in which the image being viewed is determined based on the user's selection operation. That is, a configuration may be adopted in which the image selected by the user's operation is determined to be the image being viewed.

[0195] FIG. 23 is a diagram showing an example of an image selection operation by the user.

[0196] The figure shows an example in which the hand Ha of an avatar representing the user is displayed in a virtual space and an image is selected with the hand Ha. The user selects an image by touching a captured image IMO displayed in the virtual space with the hand Ha.

[0197] In this case, when the user touches the photographed image IMO with his / her hand Ha, the virtual space changes to one corresponding to the photographed image IMO. When the user touches the photographed image IMO again with his / her hand Ha, the virtual space changes to the original environment. Alternatively, when the user moves a certain distance away from the selected photographed image, the virtual space changes to the original environment. Alternatively, when another photographed image is selected, the virtual space changes to one corresponding to the newly selected photographed image.

[0198] In this case, the image being viewed may be manipulated by a hand Ha displayed in the virtual space. For example, the image may be enlarged or reduced by performing a pinch-out or pinch-in operation (gesture) on the displayed image.

[0199] [Display of Photographed Image] FIG. 24 is a diagram showing another example of display of a photographed image in a virtual space.

[0200] This figure shows an example in which two captured images IMO3 and IMO4 exist in relation to the user's current location. One captured image IMO3 is the image being viewed, and the other captured image IMO4 is the image not being viewed.

[0201] As shown in the figure, in this example, a photographed image IMO3 being viewed and a photographed image IMO4 not being viewed are displayed in different display modes. In particular, the example shown in Fig. 24 shows an example in which a photographed image other than the one being viewed is displayed semi-transparently (the photographed image IMO4 is displayed semi-transparently).

[0202] In this way, by displaying the photographed image IMO3 being viewed and the photographed image IMO4 not being viewed in different display modes, it becomes possible to easily distinguish the photographed image being viewed. In particular, as shown in Fig. 24, by displaying the photographed image other than the photographed image being viewed in a semi-transparent state, the image being viewed can be made easier to see.

[0203] FIG. 25 is a diagram showing another example of displaying a captured image in a virtual space.

[0204] When displaying a plurality of photographed images, the images may be grouped according to the photographed content and displayed in groups.

[0205] 25 shows an example of grouping images taken in the same environment into two groups.

[0206] In this case, the position at which the image is displayed may be, for example, the shooting position of the image that represents each group. The image that represents each group may be, for example, the image with the most recent shooting date and time, the image with the oldest shooting date and time, the image that has been viewed the most, etc. The user may be able to set these as desired.

[0207] By displaying images of the same or similar shooting environments together in this way, frequent changes in the virtual sky can be suppressed.

[0208] [Other Embodiments] [Analysis of Image Information] The image information of a captured image may be analyzed to determine the era in which the captured image was taken, and the virtual space may be changed to reflect the era in which the captured image was taken. In this way, various information can be obtained from the image, and the virtual space can be changed based on the obtained information. Furthermore, as described above, the method for analyzing the image is not particularly limited, and various methods can be adopted.

[0209] Furthermore, if the display terminal 10 has an audio output function, it may be configured to output audio corresponding to the image information of the captured image. For example, it may be configured to output background music (BGM) corresponding to the content of the image being captured. For example, if a river is shown in the image being viewed, the sound of a babbling river may be output. Furthermore, for example, BGM may be prepared for each piece of virtual space data, and the BGM may be changed in conjunction with changes in the virtual space.

[0210] [Display Terminal] In the above embodiment, the display terminal 10 is configured as an HMD, but the configuration of the display terminal is not limited to this. An image of the virtual space may be presented on a non-wearable display such as a flat panel display.

[0211] Furthermore, the operation on the display terminal 10 may be performed using a controller, or may be performed using gestures, voice input, or the like.

[0212] [System Configuration] In the above embodiment, the image processing device 100 is configured to be placed on a network, but the configuration of the image processing device 100 is not limited to this. For example, the image processing device 100 may be configured to be directly connected to the display terminal 10 via a wired or wireless connection. Furthermore, the display terminal 10 may be provided with the functions of the image processing device 100.

[0213] [Collection of Photographed Images] In the above embodiment, photographed images are collected from a plurality of image sending terminals 200 via the network 2, but the method for collecting or acquiring photographed images is not limited to this. Images posted on SNS or the like may also be acquired.

[0214] [Photographed Image] Photographed images are not limited to so-called still images, but also include videos. In the case of videos, similar to still images, the shooting position is determined using accompanying information, and the video is displayed at a position corresponding to the determined shooting position. In the case of videos, for example, the image of the first frame is displayed. In the case of videos, if it is determined that the video is being viewed, playback can be started. The determination of whether or not the video is being viewed is made in the same manner as in the case of still images. In the case of videos, for example, the virtual space can be changed simultaneously (including almost simultaneously) with the start of image playback, and the virtual space can be returned to the original space when playback ends.

[0215] Furthermore, when both still images and videos are displayed in the virtual space as captured images related to the user's current location, it is preferable to distinguish between them. For example, videos can be distinguished by adding a specific mark to the display. Preview playback or digest playback is also possible. Preview playback is a function that plays the first few seconds of a video. Digest playback is a function that plays a shortened version of a video.

[0216] [Hardware Configuration of Image Processing Device] The functions of the image processing device are realized by various processors. The various processors include a CPU and / or a GPU (Graphic Processing Unit), which are general-purpose processors that execute programs and function as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically to execute specific processing. A program is synonymous with software.

[0217] A single processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same or different types. For example, a single processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA. Furthermore, multiple processing units may be composed of a single processor. Examples of multiple processing units composed of a single processor include: a first configuration, as typified by computers used as clients or servers, in which a single processor is composed of a combination of one or more CPUs and software, and this processor functions as multiple processing units; a second configuration, as typified by system-on-chip (SoC), in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip; and a hardware structure, in which the various processing units are composed of one or more of the above-mentioned various processors.

[0218] 1 Image viewing system 2 Network 10 Display terminal 11 Control unit 11A Position recognition unit 11B Gaze direction recognition unit 11C Communication control unit 11D Display control unit 12 Communication unit 13 Operation unit 14 Sensor unit 15 Display unit 16 Audio input unit 17 Audio output unit 100 Image processing device 100A Photographed image acquisition unit 100B Photographed image management unit 100C User information acquisition unit 100D Viewing area calculation unit 100E Photographed image search unit 100F Virtual space data selection unit 100G Display image generation unit 100H Viewing image determination unit 100I Image analysis unit 111 CPU 112 ROM 113 RAM 114 Auxiliary storage device 114A Photographed image database 114B Virtual space database 115 Input device 116 Output device 117 Communication interface 200 Image transmission terminal DG Gaze direction FD Gaze area FV Area in virtual space FR Area in real space corresponding to area FV IMO Photographed image IMO1 Photographed image IMO2 Photographed image IMO3 Photographed image IMO4 Photographed image IMV Image of virtual space (display image) L Optical axis Mp Map Mv Mark indicating photographing position Pr Photographing position Pv0 Current position U User (avatar) Ha Hand of user (avatar) S1 to S5 Procedure for providing virtual space S11 to S21 Procedure for providing virtual space S31 to S34 Processing procedure for displaying photographed image S41 to S44 Processing procedure for displaying photographed image S51 to S59 Processing procedure for changing virtual space

Claims

1. A processor is provided. The processor, Decide on the virtual environment to be provided, Constructing the virtual space that reproduces the real space in the determined environment; Acquire position information in a virtual space that reproduces the real space; displaying an object image representing a first image, the first image being captured in the real space and associated with the position information, in the virtual space; changing the virtual space to correspond to image information of the first image; A second image of the virtual space observed from a viewpoint corresponding to the position information is displayed on a display unit. Image processing device.

2. The processor determines the environment of the virtual space to be provided based on information of a current date and time. The image processing device according to claim 1 .

3. The processor accepts a selection of an environment of the virtual space to be provided and determines the environment of the virtual space to be provided. The image processing device according to claim 1 .

4. The processor accepts at least one selection of a season, a weather, and a time period and determines an environment of the virtual space to be provided. The image processing device according to claim 3 .

5. The first image is an image captured within a range of a first distance from a position in the real space corresponding to the position information in the virtual space. The image processing device according to claim 1 .

6. The first image is further an image captured within a range in the real space corresponding to a range of the second image in the virtual space. The image processing device according to claim 5 .

7. the processor is configured to display the object image at a position in the virtual space corresponding to a position in the real space at which the first image was taken; The image processing device according to claim 1 .

8. The processor is configured to change the virtual space to a space corresponding to the image information of the first image when the position indicated by the position information is moved to a position within a range of a second distance from the display position of the object image. The image processing device according to claim 1 .

9. The processor is configured to display the object image in a direction consistent with the direction in which the first image was captured. The image processing device according to claim 1 .

10. The processor is configured to display the object image at a size corresponding to an angle of view of the first image. The image processing device according to claim 1 .

11. The processor, When the angle of view of the first image exceeds a threshold, the object image is displayed at a size corresponding to the angle of view; When the angle of view of the first image is equal to or smaller than the threshold, the object image is displayed at a specified size. The image processing device according to claim 10.

12. The processor, determining a capture environment of the first image based on the image information of the first image; The virtual space is changed to a space corresponding to the determined shooting environment. The image processing device according to claim 1 .

13. The processor is configured to determine at least one of a season, a weather, and a time of day as the shooting environment. The image processing device according to claim 12.

14. The processor is configured to change the virtual space data to construct a different virtual space. The image processing device according to claim 1 .

15. The processor is configured to apply effects to modify the virtual space. The image processing device according to claim 1 .

16. The processor, When a plurality of first images are present, a plurality of object images are displayed in the virtual space; Accepting a selection of the object image; The virtual space is changed to a virtual space corresponding to the image information of the first image represented by the selected object image. The image processing device according to claim 1 .

17. The processor is configured to accept a selection of the object image by regarding the object image present in a line of sight direction as the selected object image. The image processing device according to claim 16.

18. The first image is an image captured within a range of a first distance from a position in the real space corresponding to the position information in the virtual space. The image processing device according to claim 16.

19. The first image is further an image captured within a range in the real space corresponding to a range of the second image in the virtual space. The image processing device according to claim 18.

20. The processor is configured to display the selected object image and the other object images in different display modes. The image processing device according to claim 16.

21. A processor is provided. The processor, Decide on the virtual environment to be provided, Constructing the virtual space that reproduces the real space in the determined environment; Acquire position information in a virtual space that reproduces the real space; displaying an object image representing a first image, the object image being captured in the real space and being associated with the position information, in the virtual space at a size corresponding to an angle of view of the first image; A second image of the virtual space observed from a viewpoint corresponding to the position information is displayed on a display unit. Image processing device.

22. determining an environment of a virtual space to be provided; constructing the virtual space that reproduces the real space in the determined environment; acquiring position information within the virtual space; displaying, in the virtual space, an object image representing a first image that is captured in the real space and is associated with the position information; changing the virtual space to correspond to image information of the first image; displaying, on a display unit, a second image of the virtual space observed from a viewpoint corresponding to the position information; An image processing method comprising:

23. determining an environment of a virtual space to be provided; constructing the virtual space that reproduces the real space in the determined environment; acquiring position information within the virtual space; displaying an object image representing a first image captured in the real space and associated with the position information in the virtual space at a size corresponding to an angle of view of the first image; displaying, on a display unit, a second image of the virtual space observed from a viewpoint corresponding to the position information; An image processing method comprising:

24. A function for determining the virtual environment to be provided; A function of constructing the virtual space that reproduces the real space in the determined environment; A function of acquiring position information within the virtual space; a function of displaying an object image representing a first image, the first image being captured in the real space and associated with the position information, in the virtual space; a function of changing the virtual space to correspond to image information of the first image; a function of displaying, on a display unit, a second image of the virtual space observed from a viewpoint corresponding to the position information; An image processing program that enables a computer to achieve this.

25. A function for determining the virtual environment to be provided; A function of constructing the virtual space that reproduces the real space in the determined environment; A function of acquiring position information within the virtual space; a function of displaying an object image representing a first image captured in the real space and associated with the position information in the virtual space at a size corresponding to an angle of view of the first image; a function of displaying, on a display unit, a second image of the virtual space observed from a viewpoint corresponding to the position information; An image processing program that enables a computer to achieve this.

26. A non-transitory computer-readable recording medium having the program according to claim 24 or 25 recorded thereon.