Image processing apparatus, method, and program

Abstract

Provided are an image processing device, a method, and a program that make it possible to provide a good image viewing space. The image processing device acquires position information in a virtual space in which a real space is reproduced. The image processing device displays, on the basis of a first image related to the position information among an image group captured in the real space, an object image representing the first image in the virtual space, and displays on a display unit a second image of the virtual space as observed from a viewpoint corresponding to the position information.

Description

Technical Field

[0004] ,

[0006] , , , , , , , , , , ,

[0005] ,

[0001] The present invention relates to an image processing apparatus, method, and program, and particularly to an image processing apparatus, method, and program that provide a virtual space.

Background Art

[0002] Patent Documents 1 to 4 describe techniques for viewing an image captured in the real space within a virtual space.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

[0004] One embodiment of the technology according to the present disclosure provides an image processing apparatus, method, and program that can provide a good image viewing space.

Means for Solving the Problems

[0005] (1) An image processing apparatus comprising a processor configured to acquire position information within a virtual space that reproduces the real space, and based on a first image related to the position information among a group of images captured in the real space, display an object image representing the first image within the virtual space and display a second image of the virtual space observed from a viewpoint corresponding to the position information on a display unit.

[0006] (2) The image processing apparatus according to (1), wherein the processor is configured to change to a virtual space corresponding to the image information of the first image. [[ID=--]]

[0007] (3) The image processing apparatus of (1) or (2), wherein the first image is an image taken within a range of a first distance from a position in real space corresponding to positional information in virtual space.

[0008] (4) The image processing apparatus of (3), wherein the first image is an image taken within a range in real space that corresponds to the range of the second image in virtual space.

[0009] (5) An image processing device, any one of (1) to (4), wherein the processor is configured to display an object image at a position in a virtual space corresponding to the position in real space where the first image was taken.

[0010] (6) The image processing device according to (2), wherein the processor is configured to change the virtual space corresponding to the image information of the first image when the position indicated by the position information moves to a position within a second distance from the display position of the object image.

[0011] (7) An image processing device, any one of (1) to (6), wherein the processor is configured to display an object image in the orientation from which the first image was taken.

[0012] (8) An image processing device, any one of (1) to (7), wherein the processor is configured to display an object image at a size corresponding to the field of view of the first image.

[0013] (9) An image processing device, any one of (1) to (7), wherein the processor is configured to display an object image at a size corresponding to the field of view if the field of view of the first image exceeds a threshold, and to display an object image at a predetermined size if the field of view of the first image is less than or equal to the threshold.

[0014] (10) The image processing device according to (2), wherein the processor is configured to determine the shooting environment of the first image based on the image information of the first image and change it to a virtual space corresponding to the determined shooting environment.

[0015] (11) The image processing apparatus (10), wherein the processor is configured to determine at least one of the season, weather and time of day as the shooting environment.

[0016] (12) An image processing apparatus according to (2), (10), or (11), wherein the processor is configured to change virtual space data to construct different virtual spaces.

[0017] (13) An image processing apparatus according to (2), (10), (11), or (12), wherein the processor is configured to apply effects and modify the virtual space.

[0018] (14) An image processing apparatus according to (2), (10), (11), (12), or (13), wherein the processor is configured to display multiple object images in a virtual space when multiple first images exist, accept the selection of an object image, and change the virtual space to one corresponding to the image information of the first image represented by the selected object image.

[0019] (15) The image processing device of (14), wherein the processor is configured to accept the selection of an object image, with the object image present in the line of sight being the selected object image.

[0020] (16) An image processing apparatus according to (2), (10), (11), (12), (13), (14), or (15), wherein the first image is an image taken within a range of a first distance from a position in real space corresponding to positional information in virtual space.

[0021] (17) The image processing apparatus of (16), wherein the first image is an image taken within a range in real space that corresponds to the range of the second image in virtual space.

[0022] (18) An image processing apparatus according to (2), (10), (11), (12), (13), (14), (15), (16), or (17), wherein the processor is configured to display the selected object image and other object images in different display modes.

[0023] (19) A step of obtaining position information in a virtual space that reproduces the real space, a step of displaying an object image representing the first image in the virtual space based on the first image related to the position information among the image group captured in the real space, and a step of displaying a second image of the virtual space observed from a viewpoint corresponding to the position information on a display unit. An image processing method including these steps.

[0024] (20) A function of obtaining position information in a virtual space that reproduces the real space, a function of displaying an object image representing the first image in the virtual space based on the first image related to the position information among the image group captured in the real space, and a function of causing a second image of the virtual space observed from a viewpoint corresponding to the position information to be displayed on a display unit. An image processing program for causing a computer to realize these functions.

Brief Description of Drawings

[0025] [Figure 1] A diagram showing an overview of an image appreciation system [Figure 2] A diagram showing an example of the system configuration of an image appreciation system [Figure 3] A block diagram showing an example of the configuration of a display terminal [Figure 4] A block diagram showing the main functions of the control unit of a display terminal [Figure 5] A diagram showing an example of the hardware configuration of an image processing device [Figure 6] A block diagram showing the main functions of an image processing device [Figure 7] A conceptual diagram of image search [Figure 8] A diagram showing an example of virtual space data stored in a virtual space database [Figure 9] A diagram showing an example of an image of a virtual space displayed on a display unit of a display terminal [Figure 10] A flowchart showing the procedure for providing a virtual space by an image processing device [Figure 11] A flowchart showing the procedure for providing a virtual space by an image processing device [Figure 12]Conceptual diagram of image display [Figure 13] A flowchart showing the steps for processing and displaying captured images. [Figure 14] Conceptual diagram of image display [Figure 15] A flowchart showing the steps for processing and displaying captured images. [Figure 16] This diagram shows another example of how captured images are displayed. [Figure 17] This diagram shows another example of how captured images are displayed. [Figure 18] Block diagram of the main functions of the image processing device regarding changes to the virtual space. [Figure 19] Conceptual diagram for determining the quality of images taken during viewing. [Figure 20] Flowchart of the procedure for processing changes to the virtual space [Figure 21] A flowchart showing the procedure for determining the shooting environment. [Figure 22] Conceptual diagram for determining the quality of images taken during viewing. [Figure 23] A diagram illustrating an example of a user's image selection operation. [Figure 24] This diagram shows another example of displaying captured images in a virtual space. [Figure 25] This diagram shows another example of displaying captured images in a virtual space. [Modes for carrying out the invention]

[0026] Preferred embodiments of the present invention will be described in detail below with reference to the attached drawings.

[0027] [First Embodiment] In recent years, with the widespread adoption of social networking services (SNS), images taken at tourist destinations are being shared in near real-time. Therefore, even without actually visiting a tourist spot, it's now possible to indirectly get a sense of what the area is like by viewing these images.

[0028] On the other hand, in recent years, advancements in technologies such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality) have made digital twins, which recreate real space (actual space, existing space) in a virtual space, a reality.

[0029] This disclosure provides a new image viewing system that utilizes digital twins.

[0030] [overview] Figure 1 shows an overview of the image viewing system disclosed herein.

[0031] For example, a virtual space that replicates a real-world location, such as a tourist spot, is provided to the user via a head-mounted display (HMD). The user can move freely within the virtual space, just as they would in the real world. If an image taken in the real world exists, when the user approaches the location in the virtual space that corresponds to the location in the real world where the image was taken, that image will be displayed in the virtual space.

[0032] [Image viewing system] Figure 2 shows an example of the system configuration of an image viewing system.

[0033] As shown in the figure, the image viewing system 1 of this embodiment includes a display terminal 10 that displays images in a virtual space, an image processing device 100 that provides images in the virtual space to the display terminal 10, and an image transmission terminal 200 that transmits images captured in real space to the image processing device 100. The display terminal 10 and the image processing device 100 are connected to each other via a network 2 so as to be able to communicate. The image transmission terminal 200 and the image processing device 100 are also connected to each other via a network 2 so as to be able to communicate.

[0034] [Image transmission terminal] The image transmission terminal 200 consists of, for example, a computer with communication capabilities (e.g., a personal computer), a mobile terminal (e.g., a smartphone, mobile phone, tablet device), or a photographic device with communication capabilities (e.g., a digital camera). Since these configurations are publicly known, a detailed explanation will be omitted.

[0035] As described above, the image transmission terminal 200 transmits (uploads) images captured in real space to the image processing device 100. The images transmitted may be captured by other devices. If the image transmission terminal itself has a camera function (for example, a smartphone with a camera function), it can transmit images captured by itself to the image processing device 100.

[0036] The image transmitted by the image transmission terminal 200 to the image processing device 100 includes at least information indicating the shooting location (shooting location information). The shooting location information consists of information that can uniquely identify the location in real space. For example, the shooting location information can consist of information on the latitude and longitude of the point where the image was taken. The shooting location information may further include altitude information. In addition to the shooting location information, it is preferable to include information indicating the date and time of shooting (shooting date and time information), information indicating the shooting direction (direction of shooting, direction of the optical axis) (shooting direction information), etc., in the image.

[0037] The form in which location information and other data are attached to an image is not particularly limited. For example, it can be attached to the image as metadata. In devices that record images in EXIF ​​(Exchangeable image file format), location information and other data can be attached to the image as tag information. In particular, in digital cameras and mobile devices equipped with GPS (Global Positioning System), GPS information (latitude, longitude, altitude, etc.) at the time of shooting is automatically attached to the captured image as location information. Also, in digital cameras and mobile devices equipped with an electronic compass (geomagnetic sensor), information on the direction in which the image was taken is automatically attached as shooting direction information.

[0038] [Display terminal] In this embodiment, the display terminal 10 is composed of, for example, an opaque HMD that is worn on the user's head and covers the user's field of vision with the display unit. By configuring the HMD to block out the outside world when worn on the head, the sense of virtual reality and immersion during viewing can be enhanced.

[0039] Figure 3 is a block diagram showing an example of the configuration of a display terminal.

[0040] As shown in the figure, the display terminal 10 includes a control unit 11, a communication unit 12, an operation unit 13, a sensor unit 14, a display unit 15, an audio input unit 16, and an audio output unit 17.

[0041] The control unit 11 functions as an arithmetic processing unit and control unit, and comprehensively controls the overall operation of the display terminal 10 according to various programs. The control unit 11 is composed of, for example, a computer equipped with a processor and memory. The processor is implemented by electronic circuits such as a CPU (Central Processing Unit). The memory includes ROM (Read Only Memory) for storing programs and various data, and RAM (Random Access Memory) and flash memory used as work areas, etc.

[0042] The communication unit 12 connects to the network 2 by wire or wireless connection and communicates with the image processing device 100 on the network. The communication unit 12 communicates with the network 2 by, for example, wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), 5G (5th Generation), 4G (4th Generation), LTE (Long Term Evolution), etc.

[0043] The operation unit 13 receives operation instructions from the user and outputs the operation details to the control unit 11. In this embodiment, movement within the virtual space is performed by operating the operation unit 13. Movement operations can be performed using, for example, an operation device capable of directional input. For example, a hand controller or joystick can be used. In addition, the operation unit 13 can include known operation devices such as push switches, levers, volume controls, pedal switches, keyboards, mice, trackpads, trackballs, and gesture input devices.

[0044] The sensor unit 14 detects at least the tilt (tilt in the XYZ 3 axes) of the headset or goggles (the part worn on the head) (3 Degrees of Freedom; 3DoF / 3 degrees of freedom). In other words, it detects the orientation of the head. The sensor unit 14 may also be configured to detect the position of the headset or goggles in 3D space (6DoF / 6 degrees of freedom). In other words, it detects the position of the head in addition to the orientation. Since this type of sensing technology is publicly known, a detailed explanation will be omitted. As an example, a configuration can be adopted that uses a gyro sensor, an accelerometer, and a geomagnetic sensor to detect the tilt and position of the headset or goggles in 3D space. The sensor unit 14 may also include a biosensor that detects the user's biometric information (e.g., pulse, heart rate, sweat, blood pressure, body temperature, respiration, electromyography, electroencephalogram, etc.), an eye-tracking sensor that detects the user's gaze within the headset or goggles, etc. The sensing information detected by the sensor unit 14 is output to the control unit 11.

[0045] When the display terminal 10 is configured as an HMD, the display unit 15 includes a left-eye screen and a right-eye screen corresponding to the user's left and right eyes, and displays images corresponding to each screen (left-eye image and right-eye image). The screen of the display unit 15 is composed of a display panel such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diode) display, or a laser scanning display such as a retinal direct-drawing display.

[0046] The audio input unit 16 is, for example, composed of a microphone and collects the voice spoken by the user.

[0047] The audio output unit 17 is composed of, for example, headphones, earphones, etc., and reproduces an audio signal.

[0048] Figure 4 is a block diagram of the main functions of the control unit of the display terminal.

[0049] As shown in the figure, the control unit 11 of the display terminal 10 functions as a position recognition unit 11A, a gaze direction recognition unit 11B, a communication control unit 11C, and a display control unit 11D, etc.

[0050] The position recognition unit 11A recognizes the user's current position (user position) in the virtual space based on the operation information from the operation unit 13. For example, it recognizes the user position from the direction and amount of movement from the origin position set in the virtual space. The position in the virtual space corresponds one-to-one with the position in the real space.

[0051] The gaze direction recognition unit 11B recognizes the current direction of the user's gaze in the virtual space based on the state of the head (orientation and position, etc.) detected by the sensor unit 14.

[0052] The communication control unit 11C controls communication with the image processing device 100. The current user's location information (user location information) recognized by the location recognition unit 11A, and the user's current gaze direction information (user gaze direction information) recognized by the gaze direction recognition unit are transmitted to the image processing device 100 via the communication unit 12 under the control of the communication control unit 11C. As will be described later, the image processing device 100 generates an image to be displayed on the display unit 15 based on the information transmitted from the display terminal 10. The communication control unit 11C receives the display image transmitted from the image processing device 100 via the communication unit 12.

[0053] The display control unit 11D controls the display of the display unit 15. The display image received from the image processing device 100 is displayed on the display unit 15 under the control of the display control unit 11D.

[0054] [Image processing device] Figure 5 shows an example of the hardware configuration of an image processing device.

[0055] As shown in the figure, the image processing device 100 includes a CPU 111, ROM 112, RAM 113, auxiliary storage device 114, input device 115, output device 116, and communication interface (I / F) 117, etc. Generally, this type of configuration can be implemented using a computer.

[0056] The image processing device 100 functions as an image processing device when the CPU 111, which is a processor, executes a predetermined program (image processing program). The program executed by the CPU 111 is stored in the ROM 112 or the auxiliary storage device 114.

[0057] The auxiliary storage device 114 constitutes the storage unit of the image processing device 100. The auxiliary storage device 114 is composed of, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc.

[0058] The input device 115 constitutes the operating section of the image processing device 100. The input device 115 is composed of, for example, a keyboard, mouse, touch panel, etc.

[0059] The output device 116 constitutes the display unit of the image processing device 100. The output device 116 is, for example, L It consists of a CD player, an OLED display, and other components.

[0060] The communication interface 117 connects to network 2 by wire or wireless connection, and on the network Display terminal 10 It also communicates with the image transmission terminal 200.

[0061] Figure 6 is a block diagram of the main functions of the image processing device.

[0062] The image processing device 100 primarily has the function of collecting captured images from the image transmission terminal 200 and the function of providing a virtual space to the display terminal 10.

[0063] Regarding the function of collecting captured images from the image transmission terminal 200, the image processing device 100 functions as a captured image acquisition unit 100A and a captured image management unit 100B.

[0064] The captured image acquisition unit 100A acquires captured images transmitted from the image transmission terminal 200 via the network 2.

[0065] The captured image management unit 100B stores the captured images acquired by the captured image acquisition unit 100A in the captured image database (Data Base; DB) 114A. The captured image database 114A is stored, for example, in the auxiliary storage device 114. As described above, the captured images transmitted from the image transmission terminal 200 are accompanied by information about the shooting location (shooting location information). The captured image management unit 100B stores the captured images in the captured image database 114A in association with the shooting location information. The captured image management unit 100B also numbers each captured image and stores it in the captured image database 114A. The multiple captured images stored in the captured image database 114A are an example of a group of images taken in real space.

[0066] With regard to the function of providing a virtual space to the display terminal 10, the image processing device 100 functions as a user information acquisition unit 100C, a field of view area calculation unit 100D, a captured image search unit 100E, a virtual space data selection unit 100F, and a display image generation unit 100G, etc.

[0067] The user information acquisition unit 100C acquires information about the user's current state (user information). User information includes user location information (the user's current location in the virtual space) and user gaze information (information about the direction of the user's gaze in the virtual space). This information is acquired from the display terminal 10 via the network 2. The acquired information is added to the captured image search unit 100E and the field of view area calculation unit 100D. User location information is an example of location information in a virtual space that reproduces real space.

[0068] The field of view calculation unit 100D calculates the user's field of view in the virtual space based on user information (user position information and user gaze information). More specifically, it calculates the field of view of the avatar, which is a representation of the user in the virtual space. This field of view corresponds to the display range of the virtual space image to be displayed on the display unit 15. The calculated field of view information (display range information) is added to the captured image search unit 100E and the display image generation unit 100G.

[0069] The captured image search unit 100E searches the captured image database 114A for captured images (related images) that are associated with the user's current location, based on the user's current location information in the virtual space (user location information) and information about the user's current field of view.

[0070] Figure 7 is a conceptual diagram of image search.

[0071] The figure shows a plan view of the virtual space and the corresponding real space. The symbol Pv0 in the figure indicates the position of the user (avatar) in the virtual space. The symbol Pr0 indicates the position in real space corresponding to the position Pv0 in the virtual space. The symbols Pr1 to Pr8 indicate the positions in real space where the image was taken (the positions where the image taken at that position exists). The symbols Pv1 to Pv8 indicate the positions in the virtual space corresponding to the positions Pr1 to Pr8 in real space. The arrow DG indicates the direction of the user's line of sight in the virtual space. The shaded area FV indicates the area within radius R1 relative to the user's current position Pv0 in the virtual space, and within the user's current field of view. In this example, the user's horizontal field of view in the virtual space is defined as a range of 100° to the left and right (200° in total), relative to the line of sight direction DG. The region FR, indicated by the diagonal lines, is the region in real space that corresponds to the region FV in virtual space.

[0072] In this embodiment, the captured image search unit 100E searches for images that were captured within a radius R1 range (less than or equal to the distance threshold R1) based on the position Pr0 in real space corresponding to the user's current position Pv0 in virtual space, and that were captured within the current field of view. For example, if the user's current position in virtual space is XX degrees XX minutes XX seconds north latitude and XX degrees XX minutes XX seconds east longitude, the unit searches for images that were captured within a circle of radius R1 from the position XX degrees XX minutes XX seconds north latitude and XX degrees XX minutes XX seconds east longitude, and that were captured within the current field of view (searching for images captured within the real space region FR corresponding to the virtual space region FV). In the example shown in Figure 7, the images captured at positions Pr1, Pr3, Pr5, and Pr7 are images captured within the radius R1 range. Among these, the images captured at positions Pr1 and Pr3 are images captured within the range corresponding to the current field of view. Therefore, the images captured at positions Pr1 and Pr3 are searched.

[0073] The retrieved captured images are added to the display image generation unit 100G. The retrieved captured images are an example of a first image associated with location information. Furthermore, the area within radius R1 is an example of the area within the first distance.

[0074] Furthermore, the range of region FR is an example of a range in real space that corresponds to a first distance from a position in real space that corresponds to positional information in virtual space, and also corresponds to a range in real space that corresponds to the range of the second image in virtual space.

[0075] The virtual space data selection unit 100F selects the virtual space data to be used. Multiple virtual space data sets are pre-stored in the virtual space database 114B. Therefore, the virtual space data selection unit 100F selects the virtual space data to be used from among the multiple virtual space data sets stored in the virtual space database 114B. The virtual space database 114B is stored, for example, in the auxiliary storage device 114.

[0076] Figure 8 shows an example of virtual space data stored in a virtual space database. This figure illustrates an example of virtual space data used to provide multiple types of virtual spaces with different environments for a single physical space.

[0077] As shown in the figure, multiple virtual space data sets are prepared, each based on a combination of season (spring, summer, autumn, winter), weather (sunny, cloudy, rainy), and time of day (morning, noon, evening, night). Each virtual space data set reproduces the same real space, but differs in season, weather, and time of day.

[0078] Thus, the virtual space database 114B is configured to record multiple types of virtual space data for a single real space, enabling the construction of virtual spaces with different environments. The virtual space database 114B records virtual space data that can reproduce at least one real space. Each virtual space data is recorded in association with information about the real space to be reproduced.

[0079] The real space to be reproduced (the real space provided to the user as a virtual space) is selected by the user, for example. The virtual space data to be used is selected, for example, according to the current date and time in the region where the display terminal 10 is being used. In the case of Japan, the season of the virtual space data to be used is selected from the current date and time, with March to May being spring, June to August being summer, September to November being autumn, and December to February being winter. In addition, the time of day of the virtual space data to be used is selected from the current date and time, with 6:00 to 9:00 being morning, 9:00 to 15:00 being noon, 15:00 to 18:00 being evening, and 18:00 to 6:00 the next day being night. For example, if the region where the display terminal 10 is being used is Japan, and the current date and time is 12:00 on January 1st, the virtual space data corresponding to "winter" and "noon" will be selected. The weather will be selected based on the default setting (for example, sunny). Therefore, in this case, "JP040102" will be selected as the virtual space data (if the default setting for "weather" is "sunny"). The weather may also be configured to be selected randomly. The current date and time information is obtained from the system or the display terminal 10. The selected virtual space data is added to the display image generation unit 100G. Alternatively, the system may be configured to select either predefined virtual space data or virtual space data selected by the user.

[0080] The display image generation unit 100G generates a virtual space image (display image) to be provided to the display terminal 10 based on the virtual space data selected by the virtual space data selection unit 100F and the field of view area information calculated by the field of view area calculation unit 100D. This display image is an image of the virtual space as observed from the user's viewpoint at the user's (avatar's) current position in the virtual space.

[0081] In this embodiment, if there is a captured image (related image) associated with the user's current location, that captured image is displayed in the virtual space. Therefore, the display image generation unit 100G generates a display image in which the captured image is displayed in the virtual space if there is a captured image associated with the user's current location.

[0082] Figure 9 shows an example of a virtual space image displayed on the display unit of a display terminal.

[0083] The virtual space image (display image) IMV displayed on the display unit 15 of the display terminal 10 is an image obtained by extracting the field of view region from the image of the virtual space constructed from virtual space data.

[0084] In this embodiment, the captured image IMO is displayed at a virtual space position Pv corresponding to the real-world shooting position Pr. The captured image IMO is displayed in the virtual space at a predetermined size. That is, it is placed in the virtual space as a planar object of a predetermined size and is visible to the user. Therefore, it is displayed larger the closer the user gets to the display position (shooting position). The captured image IMO is also displayed at a predetermined height from the ground (for example, at approximately the eye level of an adult). Therefore, in the virtual space, it is displayed as if floating in the air. Furthermore, the captured image IMO is displayed facing the user (avatar). In this embodiment, the captured image IMO displayed in the virtual space is an example of an object image.

[0085] The display image generated by the display image generation unit 100G is transmitted to the display terminal 10 via the network 2. The display terminal 10 receives the display image transmitted from the image processing device 100 and displays it on the display unit 15. This allows the user to view images of a virtual space that change in conjunction with their movements. In this embodiment, the display image is an example of a second image.

[0086] [Operation of image processing device (image processing method)] Figures 10 and 11 are flowcharts illustrating the procedure for providing a virtual space using an image processing device. Figure 10 shows the procedure for displaying the image of the virtual space on the display unit 15 of the display terminal 10. Figure 11 shows the operation procedure after the display has started.

[0087] As shown in Figure 10, first, information about the real space to be provided is acquired (step S1). In this embodiment, for example, information about real spaces that can be provided as virtual spaces is displayed in a list on the display unit 15 of the display terminal 10. The user selects the desired real space from the displayed list. The selected information is transmitted to the image processing device 100 as information about the real space to be provided.

[0088] Next, the current date and time information is obtained (step S2). As described above, the current date and time information is obtained from the system or the display terminal 10.

[0089] Next, based on the acquired real-world information and the current date and time, virtual space data for the virtual space to be provided is selected (step S3). As described above, for example, if the current date and time is 12:00 on January 1st, virtual space data corresponding to "winter" and "daytime" will be selected. The weather will be selected based on the default setting.

[0090] Next, a display image is generated based on the selected virtual space data (step S4). That is, an image of the virtual space to be displayed on the display unit 15 of the display terminal 10 is generated. This image is generated based on a reference position (origin position or starting position) set in advance within the virtual space. It is also generated based on the reference line of sight direction. That is, it is generated as an image that would be observed when facing a predetermined direction at the reference position.

[0091] The generated display image is provided to the display terminal 10 via the network 2 and displayed on the display unit 15 of the display terminal 10 (step S5).

[0092] Once the display starts, it is determined whether the user's state has changed, as shown in Figure 11 (step S11). That is, it is determined whether the user has moved or changed the direction of their gaze.

[0093] When a change in state is detected, images related to the user's current location (related images) are searched for, and the searched related images are displayed in the virtual space. Related images are searched from the image database 114A. The search and display are performed in the following steps.

[0094] First, i=1 is set (step S12), and the i-th captured image is selected from the captured image database 114A to be processed (step S13).

[0095] Next, it is determined whether the selected image is a related image (step S14). It is determined whether the image was taken within a radius R1 range based on the user's current position in real space corresponding to their current position in virtual space, and within the current field of view. In this case, first, it is determined whether the selected image was taken within a radius R1 range. If it was not taken within a radius R1 range, it is determined not to be a related image. On the other hand, if the image was taken within a radius R1 range, it is then determined whether it was taken within the current field of view. If it was not taken within the current field of view, it is determined not to be a related image. On the other hand, if it was taken within the current field of view, it is determined to be a related image.

[0096] If the selected image is determined to be a related image, it is determined whether the selected image is currently displayed or not (step S15). In other words, it is determined whether the selected image is already displayed in the virtual space or not.

[0097] If the selected image is already displayed, its display continues. On the other hand, if the selected image is not already displayed, it is displayed in the virtual space (step S16). The image is displayed at a position corresponding to its shooting location. That is, it is displayed at a position in the virtual space corresponding to the shooting location in the real space. Furthermore, the image is displayed in the virtual space at a predetermined size and is displayed facing the user directly (see Figure 9).

[0098] In step S14, if it is determined that the selected image is not an associated image, it is determined whether the selected image is currently being displayed (step S20). That is, it is determined whether the selected image is being displayed in the virtual space despite not being an associated image. If the selected image is currently being displayed, the display is terminated (step S21).

[0099] After this, we set i = i + 1 and increment the value of i (step S1 7 ), it is determined whether i exceeds N (i>N) (Step S1 8 ). In other words, it is determined whether or not all captured images in the captured image database 114A have been processed. N is the total number of captured images in the captured image database 114A.

[0100] If i > N, i.e., if there is another image to be processed, the process returns to step S13 and the above process is repeated.

[0101] On the other hand, if i > N, that is, if all captured images in the captured image database 114A have been processed, it is determined whether or not the user has left the virtual space (step S19). For example, it is determined whether or not the display has been instructed to end, such as by turning off the power. Similarly, if it is determined in step S11 that there has been no change in state, it is determined whether or not the user has left the virtual space. If it is determined that the user has left the virtual space, the process ends. On the other hand, if it is determined that the user has not left the virtual space, the process returns to step S11, and it is determined again whether or not there has been a change in state.

[0102] With the above configuration, as the user moves within the virtual space, captured images are displayed one after another in conjunction with their movement. Each captured image is displayed at a position corresponding to the real-world location where it was taken. This makes it easy to understand how each image was taken. Furthermore, users can deepen their understanding of the captured images and enjoy a better viewing experience. In addition, each captured image disappears when the user moves beyond a certain distance from the display position (shooting position), so the display within the virtual space does not become cluttered.

[0103] [Differentiation] "Displaying images aligned with the shooting direction" In the above embodiment, when displaying the captured image in the virtual space, the configuration is such that it is displayed facing the user. If information about the direction of capture can be obtained from the captured image, the configuration may be such that the captured image is displayed in the orientation in which it was taken.

[0104] Figure 12 is a conceptual diagram of how captured images are displayed.

[0105] Figure (A) is a conceptual diagram of how the captured image IMO is displayed when information about the shooting direction cannot be obtained from the captured image IMO. Figure (B) is a conceptual diagram of how the captured image IMO is displayed when information about the shooting direction can be obtained from the captured image IMO.

[0106] As shown in Figure 12(A), if information about the shooting direction cannot be obtained from the captured image IMO, the captured image IMO is displayed facing the user (avatar) U. More specifically, the captured image IMO is displayed orthogonally to the depth direction of the screen.

[0107] On the other hand, if information about the shooting direction can be obtained from the captured image IMO, the captured image IMO will be displayed in the virtual space in accordance with the shooting direction at the time the image was taken. Specifically, the captured image IMO will be displayed orthogonally to the direction of the optical axis L at the time of capture.

[0108] Information about the shooting direction is obtained from information attached to the captured image (for example, tag information). Alternatively, the shooting direction may be estimated using image recognition.

[0109] Figure 13 is a flowchart showing the steps for processing and displaying captured images.

[0110] First, information about the shooting direction of the captured image to be displayed is obtained (step S31). As described above, the shooting direction information is obtained from the information attached to the captured image. Alternatively, the shooting direction information is obtained by image recognition.

[0111] Next, it is determined whether or not information about the shooting direction exists (step S32). That is, it is determined whether or not information about the shooting direction was acquired.

[0112] If information about the shooting direction exists (if information about the shooting direction can be obtained), the captured image is displayed in the virtual space in accordance with the direction in which it was captured (step S33). That is, as shown in Figure 12(B), the captured image IMO is displayed perpendicular to the shooting direction (direction of the optical axis L).

[0113] On the other hand, if information about the shooting direction is unavailable (i.e., information about the shooting direction cannot be obtained), the captured image is displayed in the virtual space in the normal display format (step S34). That is, as shown in Figure 12(A), the captured image IMO is displayed facing the user.

[0114] In this way, if information about the shooting direction can be obtained, the captured image is displayed in the virtual space according to the orientation in which it was captured. re This makes it possible to gain a deeper understanding of how the displayed image was taken.

[0115] In addition, if information about the height from the ground at the time the image was taken can be obtained, the image may be displayed in the virtual space according to the height at which it was taken.

[0116] Furthermore, some users may find the standard display format easier to view. Therefore, it would be beneficial to allow users to choose whether or not to display the image according to the shooting direction.

[0117] [Displaying images aligned with the shooting angle] In the above embodiment, when displaying the captured image in the virtual space, the configuration is such that it is displayed at a predetermined size. If field of view information can be obtained from the captured image, the configuration may be such that the captured image is displayed in the virtual space at a size corresponding to the field of view.

[0118] Figure 14 is a conceptual diagram of how captured images are displayed.

[0119] Figures (A) and (B) show examples of displaying two images taken at different angles of view. Figure (A) shows an example where the image was taken at a wider angle of view than in Figure (B). More specifically, if the angle of view of the captured image IMO1 shown in Figure (A) is ω1 and the angle of view of the captured image IMO2 shown in Figure (B) is ω2, then this shows an example where ω1 > ω2.

[0120] As shown in Figures 14(A) and (B), when viewed from the same position, the image IMO1, taken with a wider field of view ω1, is displayed at a larger size than the image IMO2, taken with a narrower field of view ω2.

[0121] For images taken with cameras of the same sensor size, the shorter the focal length, the wider the field of view. In other words, images taken with a wide-angle lens have a wider (larger) field of view, while images taken with a telephoto lens have a narrower (smaller) field of view.

[0122] If field of view information is attached to the captured image, the field of view information of the captured image is obtained directly from the attached information. If field of view information is not attached to the captured image, it is obtained using other information attached to the captured image. The field of view can be calculated from the size of the image sensor and the focal length of the lens. Also, for example, if the attached information includes information on the 35mm equivalent lens focal length, the field of view can be calculated from that information. For example, if the 35mm equivalent lens focal length is f=50mm, the field of view is approximately 47°. In this case, obtaining the 35mm equivalent lens focal length information is essentially the same as obtaining the field of view information.

[0123] Figure 15 is a flowchart showing the steps for processing and displaying captured images.

[0124] First, information about the field of view of the captured image to be displayed is obtained (step S41). As described above, the field of view information is obtained from the information attached to the captured image. In this case, if the field of view information cannot be obtained directly from the attached information, it is obtained by calculation.

[0125] Next, it is determined whether or not field of view information exists (step S42). That is, it is determined whether or not field of view information was obtained.

[0126] If field of view information exists (if field of view information can be obtained), the captured image is displayed in the virtual space at a size corresponding to the field of view (step S43).

[0127] On the other hand, if field of view information is unavailable (if field of view information cannot be obtained), the captured image is displayed in the virtual space at a predetermined display size (default display size) (step S44). In other words, it is displayed in the virtual space at a predetermined size.

[0128] In this way, when field of view information can be obtained, the captured image is displayed in the virtual space at a size corresponding to the field of view. This allows the user to understand the field of view at which the displayed image was taken, thus enabling a correct understanding of the subject's size.

[0129] Furthermore, as in this example, when changing the display size of a captured image according to the field of view, the display size may become too large or too small depending on the image. For example, images taken with a super telephoto lens may appear too small. Also, for example, images taken with an ultra-wide-angle lens (including a fisheye lens) may appear too large. Therefore, it is preferable to set a limit on the display size. For example, the field of view may be below a threshold (lower limit). below Images of the specified size can be uniformly displayed at a predetermined size (minimum display size). Similarly, images with a field of view greater than or equal to a threshold (above or equal to an upper limit) can be uniformly displayed at a predetermined size (maximum display size).

[0130] Furthermore, some users may find it easier to view everything at the same size. Therefore, it would be beneficial to allow users to choose whether or not to change the display size according to the viewing angle.

[0131] Furthermore, similar to the above modification, the system may be configured to display the captured image according to the orientation in which it was taken. In this case, the captured image is displayed according to the orientation in which it was taken, and the image is displayed at a size corresponding to the field of view.

[0132] [Virtual space environment settings] In the above embodiment, the virtual space environment provided to the user is determined based on the current date and time information of the region where the display terminal 10 is used. However, the method for determining the virtual space environment provided to the user is not limited to this. The system may be configured to provide the user with a virtual space with a predetermined environment. Alternatively, the system may be configured to allow the user to arbitrarily select and set the environment of the virtual space to be displayed. For example, the system may be configured so that the user can select and set all of the season, weather, and time of day for the virtual space provided.

[0133] [Displaying images taken in a virtual space] In the above embodiment, if a captured image (related image) related to the user's current location exists in the virtual space, the configuration displays that captured image at a position corresponding to the location where it was taken. However, the manner in which captured images are displayed in the virtual space is not limited to this.

[0134] Figure 16 shows another example of how captured images are displayed.

[0135] The figure shows an example of displaying related images together in a predetermined location within a virtual space. In the example shown, the captured images (IMOs) are displayed in a vertical column in the right corner. In this case, as shown in the figure, a mark Mv may be displayed in the virtual space at a location corresponding to the shooting position of each captured image (IMO) to indicate the shooting position. Alternatively, in this case, each mark Mv may be displayed in a different color, and the corresponding captured image (IMO) may be displayed with a frame of the same color, so that the correspondence between each captured image (IMO) and each mark Mv is clear.

[0136] Figure 17 shows another example of how captured images are displayed.

[0137] The figure shows an example of how images taken within a radius R1 range, based on the user's current location in the virtual space, are extracted as related images and displayed in the virtual space.

[0138] Images captured within the current field of view are displayed in the virtual space corresponding to the capture location. On the other hand, images captured outside the field of view are displayed in a predetermined area. In the example shown in Figure 17, Upper to beside This shows an example of displaying captured images (IMO) in a single row. In this case, as shown in Figure 17, a map (Mp) may be displayed in a virtual space, and the approximate display position of each captured image may be indicated on the map (Mp).

[0139] [Second Embodiment] In the image viewing system 1 of the above embodiment, if an image taken in real space exists, when the user approaches the location where the image was taken in the virtual space, the taken image is displayed in the virtual space.

[0140] In the image viewing system 1 of this embodiment, the virtual space further changes according to the captured image that the user is viewing. For example, if the image the user is viewing is of the autumn season... of In the case of the image, the season in the virtual space also changes to autumn.

[0141] Aside from the fact that it alters the virtual space, it is the same as the image viewing system of the above embodiment. Therefore, only the differences will be explained below.

[0142] Figure 18 is a block diagram of the main functions of the image processing device regarding the modification of the virtual space.

[0143] As shown in the figure, the image processing apparatus 100 of this embodiment further includes the functions of an image viewing determination unit 100H and an image analysis unit 100I.

[0144] The viewing image determination unit 100H determines which image the user is viewing from among the images displayed in the virtual space. The viewing image determination unit 100H determines which image the user is viewing based on the user's current location information in the virtual space (user location information) and information on the images displayed in the virtual space.

[0145] Figure 19 is a conceptual diagram of the determination of images taken during viewing.

[0146] This figure is a plan view of the virtual space. The symbol Pv0 in the figure indicates the current position of the user (avatar) within the virtual space. Symbols Pi1 to Pi3 indicate the display positions of the captured images within the virtual space. The example shown in the figure illustrates a case where three captured images are displayed within the virtual space.

[0147] In the present embodiment, based on the current position Pv0 of the user, an image located within the range of radius R2 (distance threshold R2 or less) is regarded as an image being viewed by the user and extracted from the displayed captured images. The radius R2 is set to a value smaller than the radius R1 (R2 < R1). That is, when approaching a captured image displayed in the virtual space by a certain distance or more, it is regarded as being viewed.

[0148] When there are a plurality of captured images within the range of the radius R2, among the plurality of displayed captured images, the captured image displayed at the position closest to the current position Pv0 of the user is regarded as the image being viewed by the user.

[0149] In the example shown in FIG. 19, two captured images (the captured image at the position Pi2 and the captured image at the position Pi3) are displayed within the range of the radius R2. The position Pi2 exists closer to the user than the position Pi3. Therefore, in this case, the captured image displayed at the position Pi2 is regarded as the captured image being viewed.

[0150] The information of the captured image determined to be being viewed is added to the image analysis unit 100I. In the present embodiment, the range within the radius R2 is an example of the range of the second distance.

[0151] Note that determining the captured image being viewed is substantially synonymous with determining the captured image selected by the user. That is, it is synonymous with determining the image selected to be viewed.

[0152] The image analysis unit 100I analyzes the image information of the captured image. That is, it analyzes the content being captured (so-called content). In particular, in the present embodiment, the image information is analyzed to determine the shooting environment. The shooting environment here is the environment of the real space where the shooting is performed. The shooting environment includes at least one of season, weather, and time zone. In the present embodiment, the season, weather, and time zone of the captured image are determined.

[0153] Image information analysis includes not only image analysis but also analysis of associated information. That is, it includes analyzing associated information to determine the shooting environment. Image information analysis using image analysis may employ methods such as determining the shooting environment through image recognition. In this embodiment, the shooting environment is determined by analyzing associated information.

[0154] Here, the season, weather, and time of day of a captured image can be roughly determined from the shooting location and date / time. For example, if the shooting location is Japan, the time of day can be determined from the shooting date and time if we define morning as 6:00 to 9:00, noon as 9:00 to 15:00, evening as 15:00 to 18:00, and night as 18:00 to 6:00 the next day. Similarly, if the shooting location is Japan, the season can be determined from the shooting date and time if we define spring as March to May, summer as June to August, autumn as September to November, and winter as December to February. The weather can be determined, for example, by referring to a database of past weather records for various locations (weather database). Therefore, if the captured image is accompanied by information on the shooting location and date / time, the shooting environment (season, weather, and time of day) of the image can be determined from this information.

[0155] The image analysis unit 100I analyzes information attached to the captured image (for example, metadata) and determines the shooting environment. The determined shooting environment information is added to the virtual space data selection unit 100F.

[0156] If supplementary information cannot be obtained, the shooting environment cannot be determined, and in this case, the determination will be considered impossible. Also, if some items cannot be determined, the determination for those items will be considered impossible (no information available). For example, if the weather cannot be determined, the weather will be considered impossible to determine.

[0157] The virtual space data selection unit 100F selects the virtual space data to be used. As described above, the standard virtual space data is selected based on the current date and time in the region where the display terminal 10 is used. On the other hand, if there is a captured image that the user is currently viewing, the virtual space data corresponding to that captured image is selected. The virtual space data selection unit 100F selects the virtual space data to be used based on the analysis results of the image analysis unit 100I's analysis of the shooting environment of the captured image. That is, it selects the virtual space data to be used based on the determined season, weather, and time of day of the captured image. The virtual space data selection unit 100F searches the virtual space database 114B for the corresponding virtual space data and selects the virtual space data to be used. For example, if the season determined from the captured image is "spring," the weather is "sunny," and the time of day is "daytime," the virtual space data "JP010102" is selected, as shown in Figure 8.

[0158] If the shooting environment of an image the user is viewing cannot be determined, no changes will be made to the virtual space. Therefore, in this case, no selection of virtual space data will be made.

[0159] If the shooting environment cannot be determined for certain items (i.e., no information is available), the search for that item will be performed using a predetermined default setting. For example, if the "season" cannot be determined (i.e., no information is available), the system will use the default season information (e.g., "spring") to search for the corresponding virtual space data. The default settings may be configured to be arbitrarily set by the user. Alternatively, for items that cannot be determined, the system may use the current settings. For example, if the "season" cannot be determined, the system may use the season information of the currently displayed virtual space to search for virtual space data.

[0160] The display image generation unit 100G generates a virtual space image (display image) to be provided to the display terminal 10 based on the virtual space data selected by the virtual space data selection unit 100F and the field of view area information calculated by the field of view area calculation unit 100D. In this embodiment, if there is a captured image that the user is viewing, a virtual space image that matches the environment of that captured image is generated.

[0161] [Image viewing system operation] Figure 20 is a flowchart showing the procedure for processing changes related to the virtual space.

[0162] First, it is determined whether or not the currently displayed captured image exists (step S51). That is, it is determined whether or not the currently displayed captured image exists within the currently displayed virtual space.

[0163] If there is a captured image being displayed, it is determined whether or not there is a captured image being viewed (step S52). In this embodiment, the presence or absence of images within a radius R2 relative to the user's current position is determined to determine whether or not there is a captured image being viewed.

[0164] If multiple images are captured within a radius of R2, the image closest to the user's current location will be considered the image being viewed.

[0165] If there are images taken during viewing, the image information of the images taken during viewing is analyzed (step S53). In this embodiment, the shooting environment is determined by analyzing the supplementary information of the images taken during viewing.

[0166] Figure 21 is a flowchart showing the procedure for determining the shooting environment.

[0167] First, it is determined whether or not the season can be determined (step S53_1). In this embodiment, it is determined whether or not the season can be determined based on the availability of information on the shooting location and the date and time of shooting. If information on the shooting location and the date and time of shooting can be obtained, it is determined that the season can be determined.

[0168] If the season can be determined, the season data is set to the determined value (step S53_2). On the other hand, if the season cannot be determined, the season data is set to "none" (step S53_3).

[0169] Next, it is determined whether or not weather conditions can be determined (step S53_4). In this embodiment, it is determined whether or not weather conditions can be determined based on the availability of information on the shooting location and shooting date and time. If information on the shooting location and shooting date and time can be obtained, it is determined that weather conditions can be determined. As described above, the weather is determined from the shooting location and shooting date and time by referring to the weather database.

[0170] If the weather can be determined, the weather data is set to the determined value (step S53_5). On the other hand, if the weather cannot be determined, the weather data is set to "none" (step S53_6).

[0171] Next, it is determined whether or not it is possible to determine the time period (step S53_7). In this embodiment, it is determined whether or not it is possible to determine the time period based on whether or not information about the shooting date and time is available.

[0172] If the time period can be determined, the time period data is set to the determined value (step S53_8). On the other hand, if the time period cannot be determined, the time period data is set to "none" (step S53_9).

[0173] Through the above series of steps, the shooting environment of the captured image being observed is determined. Based on the determined shooting environment, virtual space data is selected (step S54). That is, based on the set season, weather, and time of day data, the corresponding virtual space data is searched from the virtual space database 114B, and the virtual space data to be changed is selected. For example, if the season data is set to "spring", the weather data to "sunny", and the time of day data to "daytime", the virtual space data "JP010102" is selected as shown in Figure 8. For items where the data is "none", the default data is used. For example, if the season data is "spring", the weather data to "none", and the time of day data to "daytime", the virtual space data is searched using the default weather data (for example, "sunny").

[0174] Once the virtual space data to be modified is selected, it is determined whether or not the virtual space needs to be modified, as shown in Figure 20 (Step S5). 5 If the selected virtual space data is the same as the virtual space data of the currently provided virtual space, it is determined that no change is necessary. In this case, no changes will be made to the virtual space, and the current display will continue. On the other hand, if the selected virtual space data is different from the virtual space data of the currently provided virtual space, it is determined that a change is necessary.

[0175] If it is determined that a change to the virtual space is necessary, the virtual space provided is changed (step S56). In this case, an image of the virtual space to be provided to the display terminal 10 (display image) is generated based on the selected virtual space data and provided to the display terminal 10. The provided image of the virtual space is an image that reproduces the environment of the image the user is viewing. This improves the sense of realism.

[0176] Next, it is determined whether the user has left the virtual space (step S57). If it is determined that the user has left the virtual space, the process ends. On the other hand, if it is determined that the user has not left the virtual space, the process returns to step S51, and the processes from step S51 onwards are executed again.

[0177] Furthermore, if it is determined in step S51 that there is no captured image being displayed, and if it is determined in step S52 that there is no captured image being viewed, it is determined whether the virtual space being displayed is the standard virtual space (step S58). Here, the standard virtual space is the virtual space environment at the time the display was started. As described above, in this embodiment, the virtual space to be provided is set based on the current date and time information of the region where the display terminal 10 is used. Therefore, it is determined whether the image in the virtual space being displayed is the image of the virtual space set based on the current date and time of the region where it is used.

[0178] If the image in the currently displayed virtual space is not from the standard virtual space, it is changed to the standard virtual space (step S59). Therefore, if the user stops viewing the captured image displayed in the virtual space (if they move more than the distance threshold R2 away from the captured image being viewed), the virtual space is returned to the standard virtual space.

[0179] As described above, according to the image viewing system of this embodiment, when an image taken in real space exists, the image is displayed at the location where the image was taken when the user approaches that location. When the user approaches the displayed image, the display switches to a virtual space corresponding to the environment in which the image was taken. In other words, it switches to a virtual space that reproduces the environment in which the image was taken.

[0180] Furthermore, when the user moves away from the captured image they are viewing, the virtual environment returns to its original state. Additionally, when the user moves away again, the display of the captured image disappears.

[0181] Thus, according to this embodiment, the virtual space provided changes in accordance with the captured image being viewed. This enhances the sense of realism when viewing the captured image, resulting in an even better viewing experience.

[0182] [Differentiation] [Determining the shooting environment] As described above, the shooting environment can also be determined by image recognition. In this case, for example, a pre-trained model that has been machine-learned to determine the shooting environment from an image can be used to determine the shooting environment.

[0183] Furthermore, the system can be configured to determine the shooting environment of a captured image by using both image recognition to determine the shooting environment and analysis of accompanying information to determine the shooting environment.

[0184] Furthermore, while the above embodiment is configured to determine the shooting environment as season, weather, and time of day, the information to be determined as the shooting environment is not limited to these. For example, wind direction and wind speed may also be determined. In addition, each item may be further classified and determined in more detail.

[0185] [Change virtual space] In the above embodiment, the virtual space environment is changed by switching the virtual space data used, but the method of changing the virtual space is not limited to this. In addition, for example, the virtual space environment can be changed by processing the images of the provided virtual space. For example, the virtual space environment can be changed by adding effects to the images. For example, several effects corresponding to typical weather conditions can be prepared in advance, and the weather effect corresponding to the weather of the captured image during viewing can be added to the virtual space image to change the weather of the virtual space. Note that the effects themselves are known technologies, so a detailed explanation will be omitted. For example, effects can be added to images by filtering.

[0186] Furthermore, the environment of the virtual space can be changed by switching the virtual space data used and combining it with effects. For example, the season can be handled by switching the virtual space data, while the time of day and weather can be handled with effects.

[0187] [How to determine the image being viewed] [Decision based on gaze] In the above embodiment, the captured image displayed closest to the user is considered the image being viewed. However, the method for determining the image being viewed is not limited to this. For example, the system may be configured to consider a captured image located in the user's line of sight as the image being viewed and to detect it.

[0188] Figure 22 is a conceptual diagram of the determination of images taken during viewing.

[0189] The figure shows a plan view of the virtual space. The symbol Pv0 in the figure indicates the current position of the user (avatar) in the virtual space. The arrow DG indicates the direction of the user's gaze in the virtual space. The shaded area FD indicates the user's gaze area; that is, the area the user is looking at and fixating on. In this example, the gaze area FD is defined as an angle range of ±α° horizontally, relative to the gaze direction DG. The symbols Pi1 to Pi3 indicate the display positions of the captured images in the virtual space. The example shown in the figure is an example where three captured images are displayed in the virtual space.

[0190] In this example, the image located within the gaze area FD is considered the image the user is currently viewing and is extracted from the currently displayed captured images. In the example shown in Figure 22, the captured image displayed at position Pi2 is considered the captured image being viewed.

[0191] If multiple captured images exist within a radius R2, the image displayed closest to the user's current location Pv0 will be considered the image the user is currently viewing.

[0192] In this way, it is possible to extract a captured image that is in the user's line of sight and use it as the image being viewed.

[0193] In this example, the system is configured to consider images within the gaze area FD (the image the user is looking at) as the image being viewed. However, the system may further limit the determination of the image being viewed to images located within a predetermined range from the user's current position. In other words, the system may be configured to consider images that are in the direction of the user's gaze and within a predetermined distance as the image being viewed.

[0194] [Decision based on user selection] The system can also be configured to determine the image being viewed based on the user's selection. In other words, the system can be configured to determine the image being viewed as the image selected by the user.

[0195] Figure 23 shows an example of a user's image selection operation.

[0196] The diagram shows an example where the user's avatar hand, Ha, is displayed in the virtual space, and an image is selected using this hand. The user selects an image by touching the captured image IMO displayed in the virtual space with hand Ha.

[0197] In this case, when the user touches the captured image IMO with their hand, the virtual space changes to one corresponding to that image IMO. When the user touches the captured image IMO again with their hand, the virtual space changes back to the original environment. Alternatively, when the user moves a certain distance away from the selected image, the virtual space changes back to the original environment. Or, when another image is selected, the virtual space changes to one corresponding to the newly selected image.

[0198] In this case, the image being viewed may also be manipulated using a hand (Ha) displayed in the virtual space. For example, the image may be enlarged or reduced by performing pinch-out and pinch-in gestures on the displayed image.

[0199] [Display captured images] Figure 24 shows another example of how captured images are displayed in a virtual space.

[0200] The figure illustrates an example where two captured images, IMO3 and IMO4, exist in relation to the user's current location. One captured image, IMO3, is the image being viewed, while the other captured image, IMO4, is the image not being viewed.

[0201] As shown in the figure, in this example, the captured image IMO3 being viewed and the captured image IMO4 not being viewed are displayed in different display modes. In particular, the example shown in Figure 24 shows an example where images other than the captured image being viewed are displayed in a semi-transparent state (captured image IMO4 is displayed in a semi-transparent state).

[0202] In this way, by displaying the image being viewed (IMO3) and the image not being viewed (IMO4) in different display modes, it becomes easy to distinguish between the image being viewed and the image not being viewed. In particular, as shown in Figure 24, displaying images other than the one being viewed in a semi-transparent state makes the image being viewed easier to see.

[0203] Figure 25 shows another example of how captured images are displayed in a virtual space.

[0204] When displaying multiple captured images, it is also possible to group them by the content they contain and display them in groups.

[0205] Figure 25 shows an example of grouping images taken in the same environment. It illustrates an example where images are classified into two groups.

[0206] In this case, the position where the images are displayed can be, for example, the shooting location of the image representing each group. The image representing each group could be, for example, the image with the most recent shooting date and time, the image with the most views, etc. These could also be set arbitrarily by the user.

[0207] By grouping images taken in the same or similar shooting environments in this way, you can frequently view virtual images. between This can suppress changes in it.

[0208] [Other embodiments] [Image processing analysis] The image information of the captured image can be analyzed to determine the year the image was taken, and the virtual space can be transformed according to the year the image was taken. In this way, various information can be obtained from the image, and the virtual space can be modified based on the obtained information. Furthermore, as mentioned above, the method of analyzing the image is not particularly limited, and various methods can be employed.

[0209] Furthermore, if the display terminal 10 has an audio output function, it may be configured to output audio corresponding to the image information of the captured image. For example, it may be configured to output background music (BGM) corresponding to the content being captured. For example, if a river is shown in the image being viewed, the sound of a flowing river can be output. Alternatively, for example, BGM may be prepared for each virtual space data, and the BGM may be changed in conjunction with changes in the virtual space.

[0210] [Display terminal] In the above embodiment, the case in which the display terminal 10 is composed of an HMD was described as an example, but the configuration of the display terminal is not limited to this. It may also be a configuration in which images of the virtual space are presented on a non-wearable display such as a flat panel display.

[0211] Furthermore, in addition to being operated using a controller, the operation of the display terminal 10 may also be performed using gestures, voice input, etc.

[0212] [System Configuration] In the above embodiment, the image processing device 100 is configured to be located on a network, but the configuration of the image processing device 100 is not limited to this. For example, it may be configured to be directly connected to the display terminal 10 by wired or wireless connection. Alternatively, the display terminal 10 may be equipped with the functions of the image processing device 100.

[0213] [Collection of captured images] In the above embodiment, images are collected from multiple image transmission terminals 200 via the network 2. However, the method of collecting or acquiring images is not limited to this. It is also possible to configure the system to acquire images posted on social media, etc.

[0214] [Captured image] The captured images include not only still images but also videos. In the case of videos, as with still images, the shooting position is determined using supplementary information and displayed at the position corresponding to the determined shooting position. Also, in the case of videos, for example, the image of the first frame is displayed. Furthermore, in the case of videos, the system can be configured to start playback when it is determined that viewing is in progress. The determination of whether or not viewing is in progress is the same as in the case of still images. Also, in the case of videos, for example, the system can be configured to change the virtual space simultaneously with (or almost simultaneously with) the start of image playback and return the virtual space to the original space when playback ends.

[0215] Furthermore, when displaying both still images and videos in the virtual space as images related to the user's current location, it is preferable to display them separately. For example, videos can be distinguished by adding a predetermined mark. Preview playback or digest playback is also possible. Preview playback is a function that plays the first few seconds. Digest playback is a function that plays a shortened version of the video.

[0216] [Hardware configuration of the image processing unit] The functions implemented by image processing devices are realized by various types of processors. These processors include general-purpose processors such as CPUs and / or GPUs (Graphics Processing Units) that execute programs and function as various processing units; programmable logic devices (PLDs) such as FPGAs (Field Programmable Gate Arrays) whose circuit configurations can be changed after manufacturing; and dedicated electrical circuits such as ASICs (Application Specific Integrated Circuits) that have circuit configurations specifically designed to perform particular processing. A program is synonymous with software.

[0217] A single processing unit may be composed of one of these various processors, or it may be composed of two or more processors of the same or different type. For example, a single processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA. Alternatively, multiple processing units may be composed of a single processor. Examples of composing multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software are combined to form a single processor, and this processor functions as multiple processing units, as is typical of computers used as clients or servers. Secondly, a configuration where a processor is used that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as is typical of a System on Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned various processors. [Explanation of Symbols]

[0218] 1. Image viewing system 2 Network 10 Display terminal 11 Control Unit 11A Position recognition part 11B Gaze direction recognition unit 11C Communication Control Unit 11D Display Control Unit 12 Communications Department 13 Control section 14 Sensor section 15 Display 16. Voice input section 17 Audio output section 100 Image Processing Devices 100A Image Acquisition Unit 100B Image Management Department 100C User Information Acquisition Unit 100D Field of View Calculation Unit 100E Image Search Unit 100F Virtual Space Data Selection Unit 100G display image generation section 100H Viewing Image Judgment Unit 100I Image Analysis Department 111 CPU 112 ROM 113 RAM 114 Auxiliary storage 114A Image Database 114B Virtual Space Database 115 Input device 116 Output device 117 Communication Interface 200 Image transmission terminals DG line of sight direction FD gaze area FV (Final Vinyl) - Domain within virtual space FR region FV corresponding to the region in real space IMO (Information Management Organization) captured images IMO1 captured image IMO2 captured images IMO3 captured images IMO4 captured images IMV (Image of a virtual space) L optical axis MP Map Marks indicating the filming location of the music video. Pr shooting position Pv0 Current position U User (Avatar) Ha User (Avatar) Hand Procedure for providing the virtual space (S1-S5) S11-S21 Procedure for providing the virtual space S31-S34 Procedure for displaying captured images S41-S44 Procedure for displaying captured images Procedures for processing changes to the virtual space (S51-S59)

Claims

1. Equipped with a processor, The aforementioned processor, Determine the virtual environment to be provided. The virtual space is constructed by reproducing the real space in the determined environment. By acquiring positional information within a virtual space that replicates the aforementioned real space, Based on the first image captured in the real space and associated with the location information, an object image representing the first image is displayed in the virtual space. The virtual space is changed to correspond to the image information of the first image. The system is configured to display a second image of the virtual space observed from a viewpoint corresponding to the aforementioned positional information on the display unit. Image processing device.

2. The processor determines the environment of the virtual space to be provided based on the current date and time information. The image processing apparatus according to claim 1.

3. The processor accepts the selection of the virtual space environment to be provided and determines the virtual space environment to be provided. The image processing apparatus according to claim 1.

4. The processor accepts a selection of at least one of season, weather, and time of day to determine the environment of the virtual space to be provided. The image processing apparatus according to claim 3.

5. The first image is an image taken within a range of a first distance from the position in the real space corresponding to the position information in the virtual space. The image processing apparatus according to claim 1.

6. The first image is further an image taken within the range in the real space that corresponds to the range of the second image in the virtual space. The image processing apparatus according to claim 5.

7. The processor is configured to display the object image at a position in the virtual space corresponding to the position in the real space where the first image was captured. The image processing apparatus according to claim 1.

8. The processor is configured to change the virtual space corresponding to the image information of the first image when the position indicated by the position information moves to a position within a second distance range from the display position of the object image. The image processing apparatus according to any one of claims 1 to 7.

9. The processor is configured to display the object image in the orientation from which the first image was captured. The image processing apparatus according to any one of claims 1 to 7.

10. The processor is configured to display the object image with a size corresponding to the field of view of the first image. The image processing apparatus according to any one of claims 1 to 7.

11. The aforementioned processor, If the field of view of the first image exceeds a threshold, the object image is displayed with a size corresponding to the field of view. If the field of view of the first image is less than or equal to the threshold, the system is configured to display the object image at a predetermined size. The image processing apparatus according to claim 10.

12. The aforementioned processor, Based on the image information of the first image, the shooting environment of the first image is determined. It is configured to change to the virtual space corresponding to the determined shooting environment. The image processing apparatus according to any one of claims 1 to 7.

13. The processor is configured to determine at least one of the season, weather, and time of day as the shooting environment. The image processing apparatus according to claim 12.

14. The processor is configured to change the virtual space data to construct a different virtual space. The image processing apparatus according to any one of claims 1 to 7.

15. The processor is configured to apply effects to modify the virtual space. The image processing apparatus according to any one of claims 1 to 7.

16. The aforementioned processor, If multiple first images exist, the object images are displayed multiple times within the virtual space. The selection of the aforementioned object image is accepted. The selected object image is configured to change the virtual space to one that corresponds to the image information of the first image represented by the selected object image. The image processing apparatus according to any one of claims 1 to 7.

17. The processor is configured to accept the selection of an object image, with the object image located in the line of sight being the selected object image. The image processing apparatus according to claim 16.

18. The first image is an image taken within a range of a first distance from the position in the real space corresponding to the position information in the virtual space. The image processing apparatus according to claim 16.

19. The first image is further an image taken within the range in the real space that corresponds to the range of the second image in the virtual space. The image processing apparatus according to claim 18.

20. The processor is configured to display the selected object image and other object images in different display modes. The image processing apparatus according to claim 16.

21. Equipped with a processor, The aforementioned processor, Determine the virtual environment to be provided. The virtual space is constructed by reproducing the real space in the determined environment. By acquiring positional information within a virtual space that replicates the aforementioned real space, Based on a first image captured in the real space and associated with the position information, an object image representing the first image is displayed in the virtual space at a size corresponding to the field of view of the first image. The system is configured to display a second image of the virtual space observed from a viewpoint corresponding to the aforementioned positional information on the display unit. Image processing device.

22. The steps include determining the virtual environment to be provided, The steps include constructing the virtual space that reproduces the real space in the determined environment, The steps include: acquiring location information within the virtual space; The steps include: displaying an object image representing the first image in the virtual space based on the first image captured in the real space and associated with the location information; The steps include changing the virtual space to one corresponding to the image information of the first image, The steps include: displaying a second image of the virtual space observed from a viewpoint corresponding to the aforementioned location information on the display unit; Image processing methods including [specific details omitted].

23. The steps include determining the virtual environment to be provided, The steps include constructing the virtual space that reproduces the real space in the determined environment, The steps include: acquiring location information within the virtual space; The steps include: displaying an object image representing the first image in the virtual space, based on a first image captured in the real space and associated with the position information, with a size corresponding to the field of view of the first image; The steps include: displaying a second image of the virtual space observed from a viewpoint corresponding to the aforementioned location information on the display unit; Image processing methods including [specific details omitted].

24. A function to determine the environment of the virtual space to be provided, A function to construct the virtual space that reproduces the real space in the determined environment, A function to acquire location information within the aforementioned virtual space, A function to display an object image representing a first image in the virtual space based on a first image captured in the real space and associated with the location information, A function to change the virtual space to one corresponding to the image information of the first image, A function to display a second image of the virtual space observed from a viewpoint corresponding to the aforementioned position information on the display unit, An image processing program that enables computers to perform image processing.

25. A function to determine the environment of the virtual space to be provided, A function to construct the virtual space that reproduces the real space in the determined environment, A function to acquire location information within the aforementioned virtual space, A function to display an object image representing the first image in the virtual space, based on a first image captured in the real space and associated with the position information, with a size corresponding to the field of view of the first image; A function to display a second image of the virtual space observed from a viewpoint corresponding to the aforementioned position information on the display unit, An image processing program that enables computers to perform image processing.

26. A non-temporary and computer-readable recording medium on which the program described in claim 24 or 25 is recorded.

Images