Information processing device, information processing method, and program

Abstract

An information processing device equipped with an acquisition unit, a generation unit and an output unit. The acquisition unit acquires, from an image server, a viewpoint image which corresponds to the viewpoint of a first user, on the basis of location information which expresses the location of a mobile terminal which moves with the first user. The image server stores a plurality of viewpoint images which are obtained by imaging in a plurality of directions from each of a plurality of locations. By using the acquired viewpoint image, the generation unit generates a free viewpoint rendering image, which is specified by a second user in a location which differs from that of the first user, as a display image. The output unit outputs the generated display image.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, monitoring services have become known that notify guardians of the location and movements of a user being monitored, such as a child, based on location information from a mobile device such as a mobile phone or a BLE (Bluetooth (registered trademark) Low Energy) tag that moves with the user being monitored.

[0003] Also, a service that uses AR (Augmented Reality) technology is known that provides route guidance to a user wearing an HMD (Head Mounted Display). A device that realizes AR technology such as an HMD can display an image that indicates the position and direction according to the movement of the user's viewpoint, for example, by using a 360-degree image captured by a 360-degree camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2018-502360 Summary of the Invention [Problem to be solved by the invention]

[0005] However, image data showing a 360-degree image is large in volume, so for example, when the field of view of a user of a mobile terminal is displayed on a display terminal such as an HMD used by a user in another position, the volume of data transferred from the mobile terminal to the display terminal becomes large, which can cause delays in interaction.

[0006] Therefore, the present disclosure proposes an information processing device, an information processing method, and a program that can reduce interaction delays when the field of view of a user of a mobile terminal is displayed on a display terminal used by a user in another location. [Means for solving the problem]

[0007] According to the present disclosure, an information processing device includes an acquisition unit, a generation unit, and an output unit. The acquisition unit acquires a viewpoint image according to a viewpoint of a first user from an image server based on location information indicating a location of a mobile terminal moving with the first user. The image server stores a plurality of viewpoint images obtained by photographing a plurality of directions from each of a plurality of positions. The generation unit uses the acquired viewpoint images to generate a rendering image of a free viewpoint designated by a second user at a position different from that of the first user as a display image. The output unit outputs the generated display image. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an information processing system according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a schematic diagram illustrating an overview of information processing according to an embodiment of the present disclosure. [Diagram 3] 10 is a schematic diagram showing an overview of information processing for transmitting image data captured by a mobile terminal, which is different from information processing according to an embodiment of the present disclosure. FIG. [Figure 4] 1A to 1C are diagrams for explaining generation of a display image in information processing according to an embodiment of the present disclosure. [Diagram 5] 11A to 11C are diagrams for explaining updating of a display image accompanying movement of the position of an object or a free viewpoint in information processing according to an embodiment of the present disclosure. [Figure 6] 11A to 11C are diagrams for explaining updating of viewpoint images according to the amount of movement of an object in information processing according to an embodiment of the present disclosure. [Figure 7]11A to 11C are diagrams for explaining updating of a display image accompanying rotation at a free viewpoint in information processing according to an embodiment of the present disclosure. [Figure 8] 11A to 11C are diagrams for explaining updating of a display image in response to movement between indoors and outdoors in information processing according to an embodiment of the present disclosure. [Figure 9] 11A and 11B are diagrams for explaining generation of a display image without using viewpoint image data in information processing according to an embodiment of the present disclosure. [Figure 10] 1 is a flowchart illustrating an example of information processing according to an embodiment of the present disclosure. [Figure 11] FIG. 1 is a diagram for explaining a usage example of an information processing system according to an embodiment of the present disclosure. [Figure 12] FIG. 2 is a block diagram illustrating an example of a hardware configuration of each device in the information processing system according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0010] The present disclosure will be described in the following order. 1. Embodiment 1-1. Functional configuration of the information processing system according to the embodiment 1-2. Example of information processing according to the embodiment 1-2-1. Updating the displayed image according to the movement of the object or free viewpoint 1-2-2. Updating the displayed image according to the movement of the subject indoors and outdoors 1-2-3.Generation of display images without using viewpoint images 1-3. Information Processing Procedure According to the Embodiment 1-4. Example of use of information processing system according to embodiment 1-5.Hardware configuration 2. Effects of the information processing system according to the present disclosure

[0011] (1. EMBODIMENT) This embodiment illustrates a monitoring service that detects the location and movement of a first user by acquiring location information of a mobile terminal that moves with the first user. Specifically, this embodiment illustrates an information processing system that displays an image corresponding to the field of view of a first user to a second user at a different location from the first user, based on the location information of the mobile terminal, without receiving image data from the mobile terminal. In the following description, it is assumed that the first user is a child who moves with the mobile terminal. It is also assumed that the second user is a guardian who uses the monitoring service realized by the information processing system according to the embodiment to monitor the location and movement of the child.

[0012] [1-1. Functional configuration of the information processing system according to the embodiment] An example of a functional configuration of an information processing system 1 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of a configuration of the information processing system 1 according to an embodiment of the present disclosure.

[0013] 1, an information processing system 1 according to the embodiment includes a mobile terminal 10, a street image server 20, an information processing device 30, and a display terminal 40. The mobile terminal 10, the street image server 20, and the information processing device 30 are communicatively connected via a network N. The information processing device 30 and the display terminal 40 are communicatively connected. As the network N, for example, a telecommunication line such as the Internet can be used.

[0014] The mobile terminal 10 and the street image server 20 may be connected to the information processing device 30 via different networks N. The information processing device 30 and the display terminal 40 may be connected to each other via the network N so as to be able to communicate with each other.

[0015] The mobile terminal 10 is a communication terminal that moves with a child. The mobile terminal 10 functions as the mobile terminal 10 according to the present disclosure by executing a program (application) for realizing information processing according to the present disclosure. As the mobile terminal 10, a dedicated terminal or a general communication terminal such as a smartphone may be used.

[0016] As shown in FIG. 1, the mobile terminal 10 has the functions of a communication unit 11, a sensor unit 12, a detection unit 13, and an input / output unit 14.

[0017] The communication unit 11 is wirelessly connected to a network N and communicates with the information processing device 30 via the network N. As shown in FIG. 1 , the communication unit 11 includes a cellular module 111, a Wi-Fi (registered trademark) module 112, and a BLE (Bluetooth (registered trademark) Low Energy) module 113. The cellular module 111 is a communication circuit compatible with various standards such as 3G, 4G, and 5G. The Wi-Fi module 112 is a communication circuit compatible with the Wi-Fi standard. The BLE module 113 is a communication circuit compatible with the BLE standard.

[0018] As an example, the communication unit 11 establishes communication with a mobile phone base station by the Cellular module 111, and outputs information indicating the mobile phone base station with which communication has been established or its position to the detection unit 13. The communication unit 11 establishes communication with a Wi-Fi access point by the Wi-Fi module 112, and outputs information indicating the Wi-Fi access point with which communication has been established or its position to the detection unit 13. The communication unit 11 receives a BLE beacon from a BLE tag or the like by the BLE module 113, and outputs information indicating the radio wave intensity of the received BLE beacon and the BLE tag or the like that transmitted the received BLE beacon or its position to the detection unit 13. In addition, the communication unit 11 transmits information indicating the position, direction, and movement of the mobile terminal 10 detected by the detection unit 13 to the information processing device 30. Here, the information indicating the position of the mobile terminal 10 is, for example, latitude and longitude values. The information indicating the movement of the mobile terminal 10 is, for example, information indicating the start of movement of the mobile terminal 10 or information indicating the stationary state of the mobile terminal 10.

[0019] The sensor unit 12 detects the position, direction, and movement of the mobile terminal 10. Furthermore, the sensor unit 12 outputs information indicating the detected position, direction, and movement of the mobile terminal 10 to the detection unit 13. As shown in FIG. 1 , the sensor unit 12 includes a GPS (Global Positioning System) sensor 121, a geomagnetic sensor 122, and an IMU (Inertial Measurement Unit) 123.

[0020] As an example, the sensor unit 12 detects the latitude and longitude of the location of the mobile terminal 10 using the GPS sensor 121, and outputs information indicating the detected latitude and longitude to the detection unit 13. The sensor unit 12 detects the direction of the mobile terminal 10 using the geomagnetic sensor 122, and outputs information indicating the detected direction to the detection unit 13. The sensor unit 12 detects the acceleration and angular velocity of the mobile terminal 10 using the IMU 123, and outputs information indicating the detected acceleration and angular velocity to the detection unit 13. Here, the information indicating the acceleration and angular velocity can also be expressed as information indicating changes in the position and attitude of the mobile terminal 10.

[0021] The detection unit 13 detects information indicating the position, direction, and movement of the mobile terminal 10 based on the outputs of the communication unit 11 and the sensor unit 12. Furthermore, the detection unit 13 outputs information indicating the detected position, direction, and movement of the mobile terminal 10 to the communication unit 11. As shown in Fig. 1, the detection unit 13 has functions as a position detection unit 131, a direction detection unit 134, and a movement detection unit 135. The position detection unit 131 has functions as an outdoor position detection unit 132 and an indoor position detection unit 133.

[0022] The outdoor position detection unit 132 detects information indicating the position of the mobile terminal 10 based on the output of at least one of the cellular module 111, the GPS sensor 121, and the Wi-Fi module 112. The indoor position detection unit 133 detects information indicating the indoor position of the mobile terminal 10 based on the output of at least one of the Wi-Fi module 112 and the BLE module 113. The indoor position of the mobile terminal 10 includes the height (floor, story) at which the mobile terminal 10 is located. The direction detection unit 134 detects information indicating the direction of the mobile terminal 10 based on the output of the geomagnetic sensor 122. The movement detection unit 135 detects information indicating the direction of the mobile terminal 10 based on the output of the IMU 123.

[0023] In addition, the position detection unit 131 may be configured to estimate the current position of the mobile terminal 10 based on information indicating the position of the mobile terminal 10 detected at the previous point in time and information indicating the direction and movement of the mobile terminal 10 detected at the current point in time.

[0024] The input / output unit 14 accepts voice input from the user. The input / output unit 14 outputs the accepted voice data to the communication unit 11. The input / output unit 14 also outputs voice corresponding to the voice data received from the information processing device 30 or the display terminal 40 by the communication unit 11. The input / output unit 14 includes an input device such as a microphone and an output device such as a speaker.

[0025] The input and output of voice via the input / output unit 14 is, for example, a phone call between a user (guardian) using the display terminal 40, but it may also be the recording and playback of a voice message, or the conversion of voice to text data and the conversion of text data to voice.

[0026] In addition, if the child has a separate communication terminal such as a smartphone, for example, and the child can use the function of the input / output unit 14 through another device, the mobile terminal 10 does not need to have the input / output unit 14.

[0027] The street image server 20 is an image server (server device) that stores a database of multiple viewpoint images (street images, 360-degree images) obtained by photographing multiple orientations (directions) from multiple positions. Each of the multiple viewpoint images stored in the street image server 20 is associated with information indicating the position and direction at which it was photographed. Also, each of the multiple viewpoint images stored in the street image server 20 is an image that has been subjected to appropriate processing regarding reflected subjects. For example, Street View (registered trademark) by Google Inc. can be used as the street image server 20.

[0028] The information processing device 30 generates an image corresponding to the field of view of the child from a free viewpoint specified by the guardian, without receiving image data from the mobile terminal 10, based on information indicating the position of the mobile terminal 10 moving with the child, and outputs the image to a display terminal 40 used by the guardian. The information processing device 30 functions as the information processing device 30 according to the present disclosure by executing a program (application) for realizing information processing according to the present disclosure. As the information processing device 30, for example, a general communication terminal such as a smartphone, a personal computer (PC), a tablet PC, or other device can be appropriately used.

[0029] As shown in FIG. 1, the information processing device 30 has functions as a communication unit 31, a position information update management unit 32, a 360-degree image rendering unit 33, and a free viewpoint image rendering unit .

[0030] The communication unit 31 is wirelessly connected to the network N, and communicates with each of the mobile terminal 10 and the street image server 20 via the network N. The communication unit 31 receives information indicating the position, direction, and movement of the mobile terminal 10 from the mobile terminal 10. The communication unit 31 transmits a viewpoint image request from the position information update management unit 32 to the street image server 20. The communication unit 31 receives a viewpoint image corresponding to the viewpoint image request from the street image server 20. The communication unit 31 also transmits a free viewpoint image generated by the free viewpoint image rendering unit 34 to the display terminal 40. The communication unit 31 receives information indicating the movement of the display terminal 40 in 6DoF (degrees of freedom) from the display terminal 40. The communication unit 31 includes a communication circuit for wired communication, a communication circuit for wireless communication, and a combination thereof. As the communication circuit for wireless communication, a communication circuit corresponding to various standards such as 3G, 4G, 5G, Wi-Fi, and BLE can be appropriately used. Here, the communication unit 31 is an example of an acquisition unit and an output unit.

[0031] The location information update management unit 32 outputs a viewpoint image request to the communication unit 31, requesting a viewpoint image from the street image server 20, based on information indicating the position, direction, and movement of the mobile terminal 10 received by the communication unit 31. The location information update management unit 32 outputs a viewpoint image request when the amount of movement of the mobile terminal 10 exceeds a predetermined threshold. The location information update management unit 32 also outputs a viewpoint image request when the free viewpoint moves with the movement of the display terminal 40, for example, when the guardian wearing the display terminal 40 moves. Here, the location information update management unit 32 is an example of an acquisition unit.

[0032] The 360-degree image rendering unit 33 performs rendering processing to generate a 360-degree image for display based on the viewpoint image from the street image server 20. The 360-degree image rendering unit 33 outputs the generated 360-degree image for display to the free viewpoint image rendering unit 34. Here, the 360-degree image rendering unit 33 is an example of a generating unit.

[0033] The free viewpoint image rendering unit 34 performs a rendering process to generate a rendering image of a free viewpoint according to the field of view of the display terminal 40 based on the 360-degree image generated by the 360-degree image rendering unit 33 and the designation of the free viewpoint from the display terminal 40. Hereinafter, the rendering image of the free viewpoint generated by this process is described as a free viewpoint image. Note that the designation of the free viewpoint from the display terminal 40 indicates, as an example, that the free viewpoint is designated based on the output of the IMU 42 and / or the camera 43 of the display terminal 40. In other words, designating the free viewpoint includes recognizing the movement of the display terminal 40 in 6DoF (degrees of freedom) in the three-dimensional space. The free viewpoint image rendering unit 34 outputs the generated free viewpoint image to the display terminal 40 via the communication unit 31. Here, the free viewpoint image rendering unit 34 is an example of a generating unit.

[0034] The display terminal 40 displays a free viewpoint image that expresses the field of view of the child at a free viewpoint, which is generated based on the position information of the mobile terminal 10. The display terminal 40 is a display terminal device for realizing a service using AR (Augmented Reality) technology. As an example, the display terminal 40 is a HMD (Head Mounted Display) worn by a guardian. The display terminal 40 is assumed to be, for example, an optical see-through type HMD. As shown in FIG. 1, the display terminal 40 has functions as a communication unit 41, a sensor unit, and an input / output unit 44. The sensor unit includes an IMU 42 and a camera 43.

[0035] The communication unit 41 receives a free viewpoint image from the information processing device 30. The communication unit 41 also transmits information indicating 6DoF movement of the display terminal 40 to the information processing device 30. The communication unit 41 includes a communication circuit for wired communication, a communication circuit for wireless communication, and a combination of these. As the communication circuit for wireless communication, a communication circuit compatible with various standards such as 3G, 4G, 5G, Wi-Fi, and BLE can be appropriately used.

[0036] The sensor unit detects information indicating 6DoF movement of the display terminal 40. Specifically, the sensor unit detects the acceleration and angular velocity of the display terminal 40 by the IMU 42, and transmits information indicating the detected acceleration and angular velocity to the information processing device 30 via the communication unit 41. The sensor unit also detects the movement or rotation of the display terminal 40 based on, for example, the movement or deformation of a feature point on an image obtained by the camera 43, and transmits information indicating the detected movement or rotation to the information processing device 30 via the communication unit 41. In other words, the guardian wearing the display terminal 40 can specify a free viewpoint by moving or rotating himself / herself without any special operation.

[0037] The input / output unit 44 accepts voice input from the user. The input / output unit 44 outputs the accepted voice data to the communication unit 41. The input / output unit 44 also outputs voice corresponding to the voice data received from the mobile terminal 10 or the information processing device 30 by the communication unit 41. The input / output unit 44 includes an input device such as a microphone and an output device such as a speaker. The input / output unit 44 also includes a light guide plate lens 441 as shown in FIG. 1. In the display terminal 40 as an optical see-through type HMD, for example, a virtual image optical system using a half mirror or a transparent light guide plate lens 441 is held in front of the user's eyes, and an image is displayed inside the virtual image optical system. Therefore, a user wearing the optical see-through type HMD can see the scenery around the user while looking at the image displayed inside the virtual image optical system.

[0038] The display terminal 40 may be, for example, a video see-through type HMD. The video see-through type HMD is worn by the user so as to cover the user's eyes, and a display of the video see-through type HMD is held in front of the user's eyes. The video see-through type HMD also has a camera 43 for capturing images of the scenery in front of the user, and an image of the scenery captured by the camera 43 is displayed on the display. For this reason, it is difficult for a user wearing a video see-through type HMD to directly view the scenery in front of the user, but the user can check the scenery in front of the user from the image displayed on the display.

[0039] The information processing device 30 and the display terminal 40 may be integrated into one device.

[0040] [1-2. Example of information processing according to the embodiment] Here, an overview of the information processing according to the embodiment will be described with reference to FIG. 2 and FIG.

[0041] In recent years, monitoring services have become known that notify parents of the location and movements of a monitored user, such as a child, based on location information of a mobile terminal 10, such as a mobile phone or a BLE tag, that moves along with the monitored user.

[0042] Also, a service that uses AR technology is known that provides route guidance to a user wearing an HMD. A device that realizes AR technology such as an HMD can display an image (free viewpoint image) that indicates the position and direction according to the user's viewpoint movement, for example, by using a 360-degree image taken by a 360-degree camera.

[0043] 2 is a schematic diagram showing an overview of information processing according to an embodiment of the present disclosure. The mobile terminal 10 is carried by a child U who is a user to be watched over, and moves with the child U. The mobile terminal 10 transmits GEO data including position information of the mobile terminal 10 to the information processing device 30 via the network N. Based on the GEO data from the mobile terminal 10, the information processing device 30 acquires viewpoint images from the street image server 20 at timing according to the movement of the target or the free viewpoint, and generates a free viewpoint image. The display terminal 40 displays the free viewpoint image from the information processing device 30.

[0044] On the other hand, consider a case where a free viewpoint image is generated based on a 360-degree image taken by a child U to be watched. FIG. 3 is a schematic diagram showing an overview of information processing for transmitting image data taken by a mobile terminal 10a', which is different from the information processing according to the embodiment of the present disclosure. In the example shown in FIG. 3, the mobile terminal 10a' is a 360-degree camera. Also, the mobile terminal 10b' is a smartphone that transmits 360-degree video data taken by the mobile terminal 10a' to an information processing device 30' via a network N. In the information processing realized by such a configuration, as shown in FIG. 3, an actual video can be transferred to the display terminal 40 in real time. Also, a free viewpoint image can be generated without using the street image server 20. Therefore, an information processing system that is not affected by changes, such as the disappearance of a building, after the street image server 20 is prepared can be realized.

[0045] However, in the example shown in FIG. 3, unlike the information processing according to the embodiment, image data is also transmitted in the communication between the mobile terminal 10b' and the information processing device 30'. In general, image data showing a 360-degree image has a large amount of data. For example, the amount of data in the communication is large, about several hundred Mbps. For this reason, for example, when the field of view of the user on the mobile terminal 10 side is displayed on a display terminal 40 such as an HMD used by a user in another position, the amount of data transferred from the mobile terminal 10 to the display terminal 40 becomes large, and a delay in the interaction may occur.

[0046] Therefore, in the present disclosure, an information processing device 30, an information processing system 1, an information processing method, and a program capable of suppressing delays in interactions when the field of view of the user on the mobile terminal 10 side is displayed on a display terminal 40 used by a user at another location are proposed. Specifically, in the information processing according to the embodiment, as shown in FIG. 2, 360-degree image data is not transmitted from the mobile terminal 10 to the information processing device 30. For example, the amount of GEO data (location information) transmitted from the mobile terminal 10 to the information processing device 30 is small, about several Kbps. Therefore, even when the field of view of the user on the mobile terminal 10 side is displayed on a display terminal 40 such as an HMD used by a user at another location, the amount of data transferred from the mobile terminal 10 to the display terminal 40 is small, so that delays in interactions can be suppressed.

[0047] Furthermore, capturing a 360-degree image may involve complicated operations for the child U, who is the user to be watched over. However, in the information processing according to the embodiment, the child U, who is the user to be watched over, does not need to perform any shooting operations on the mobile terminal 10.

[0048] In the information processing according to the embodiment, the free viewpoint image displayed on the display terminal 40 is generated based on the viewpoint image acquired from the street image server 20. Here, the street image server 20 is an image server prepared in advance prior to the generation of the free viewpoint image. In other words, it is possible to perform image processing such as blurring in advance on a reflected subject, which is a subject that is unintentionally reflected within the angle of view.

[0049] [1-2-1. Updating the displayed image according to the movement of the object or free viewpoint] Generation and updating of a display image in information processing according to the embodiment will be described with reference to FIGS. 4, 5, 6 and 7. FIG.

[0050] 4 is a diagram for explaining generation of a display image in information processing according to an embodiment of the present disclosure. The information processing according to the embodiment includes a process of acquiring a viewpoint image according to information indicating the position and direction of the mobile terminal 10, and a process of generating a 360-degree image that reproduces the field of view of the child U to be watched over based on the acquired viewpoint image. The information processing according to the embodiment also includes a process S1 of generating, as a display image, a 360-degree image (free viewpoint image) at a free viewpoint that can be moved arbitrarily on the 360-degree image. Here, the position and direction of the free viewpoint in the virtual space VS change according to the movement of the guardian wearing the display terminal 40.

[0051] 5 is a diagram for explaining updating of a display image accompanying movement of the position of an object or a free viewpoint in information processing according to an embodiment of the present disclosure. The information processing according to the embodiment includes a process S2 for updating a display image according to movement of the free viewpoint. The process S2 is a process executed when movement of the free viewpoint is detected. The process S2 includes the process of acquiring a viewpoint image, the process of generating a 360-degree image from the acquired viewpoint image, and the process S1 for generating the free viewpoint image as a display image, which are described above with reference to FIG. 4. In addition, the information processing according to the embodiment includes a process S3 for updating a display image according to movement of a child U.

[0052] Fig. 6 is a diagram for explaining updating of viewpoint images according to the amount of movement of a target in information processing according to an embodiment of the present disclosure. The upper part of Fig. 6 shows viewpoint images IMG1, IMG2, IMG3, IMG4, and IMG5 corresponding to the respective positions of a target child U moving along a sidewalk toward the line of sight.

[0053] As an example, process S3 may be a process of sequentially acquiring viewpoint images according to the position of the child U in response to the movement of the child U, and generating a free viewpoint image using each viewpoint image. However, if viewpoint images are acquired from the street image server 20 every time the position of the child U changes, the amount of data transferred from the street image server 20 to the information processing device 30 may become large, which may cause a delay in interaction.

[0054] Therefore, as another example, the process S3 can be a process of acquiring a viewpoint image IMG1 according to the position of the child U, and generating a free viewpoint image according to the position of the child U using the viewpoint image IMG1 until the movement amount of the child U exceeds a predetermined threshold. The predetermined threshold can be set arbitrarily, and can be set to, for example, 30 m. The lower part of FIG. 6 illustrates each free viewpoint image IMG1-1, IMG1-2, IMG1-3, IMG1-4 generated using the viewpoint image IMG1. Specifically, in the process S3, each free viewpoint image IMG1-2, IMG1-3, IMG1-4 is generated by superimposing an icon UV indicating the position of the child U on the same viewpoint image IMG1 until the movement amount of the child U exceeds a predetermined threshold. In other words, the process S3 can be a process of generating each free viewpoint image according to the position of the child U by superimposing an icon UV on the same viewpoint image or moving the icon UV instead of updating the viewpoint image until the movement amount of the child U exceeds a predetermined threshold. On the other hand, when the amount of movement of the child U exceeds a predetermined threshold, the process S3 can be a process of acquiring a viewpoint image IMG5 according to the position of the child U from the street image server 20 and generating a free viewpoint image IMG5-1 using the viewpoint image IMG5. In this way, the process S3 is a process of changing the update frequency of the viewpoint image according to the amount of movement of the target. This makes it possible to suppress an increase in the amount of data transferred from the street image server 20 to the information processing device 30.

[0055] 7 is a diagram for explaining updating of a display image accompanying rotation at a free viewpoint in information processing according to an embodiment of the present disclosure. The information processing according to the embodiment includes a process S4 for updating a display image in accordance with rotation of the display terminal 40. The process S4 is a process for the case where the display terminal 40 rotates while the free viewpoint does not move. The process S4 is a process for generating a free viewpoint image in accordance with the rotation of the display terminal 40 from an already acquired viewpoint image without updating the viewpoint image.

[0056] [1-2-2. Updating the displayed image according to the movement of the subject indoors and outdoors] Using FIG. 8, updating of a display image in accordance with indoor / outdoor movement of a target in information processing according to the embodiment will be described.

[0057] FIG. 8 is a diagram for explaining the update of a display image according to movement between indoors and outdoors in the information processing according to the embodiment of the present disclosure. There are also cases where the child U enters and leaves the building B, that is, moves from indoors to outdoors or from outdoors to indoors. In the information processing according to the embodiment, when a viewpoint image inside the building B is stored in the street image server 20, a free viewpoint image is generated as a display image based on indoor position information in the same manner as described above. On the other hand, there may be cases where a viewpoint image inside the building B is not stored in the street image server 20. In this case, as shown in FIG. 8, the information processing according to the embodiment generates a free viewpoint image IMG6 as a display image based on a viewpoint image including the building B at an arbitrary position outside the building B. Specifically, the information processing according to the embodiment generates a free viewpoint image IMG6 in which the child U is represented as being inside the building B by superimposing an icon UV indicating the child U on the position of the building B. As the viewpoint image including the building B, for example, a viewpoint image acquired just before the child U enters the building B can be used.

[0058] It should be noted that it is sufficient to show that child U is inside building B. For example, a transparency process may be applied to the area of ​​building B in the viewpoint image, and then an icon UV may be placed in the area where the transparency process has been applied.

[0059] The position of the icon UV superimposed on the viewpoint image may be moved based on indoor position information of the child U. For example, as the child U moves from floor to floor indoors, the position where the icon UV is superimposed is moved in the height direction of the building B. Also, the viewpoint image on which the icon UV is superimposed may be updated based on indoor position information of the child U.

[0060] [1-2-3. Generation of display images without using viewpoint images] Generation of a display image without using a viewpoint image in the information processing according to the embodiment will be described with reference to FIG.

[0061] FIG. 9 is a diagram for explaining the generation of a display image without using a viewpoint image in the information processing according to the embodiment of the present disclosure. There are cases where the child U moves using a vehicle (moving object) such as a train, a bus, or a bicycle. In such a case, the frequency of acquiring viewpoint images from the street image server 20 increases according to the movement of the child U, and there is a risk of a delay in the interaction. In addition, the viewpoint image stored in the street image server 20 cannot express the field of view of the child U in the train. For these reasons, when the information processing according to the embodiment detects that the child U is moving using a vehicle such as a train or a bus, it generates a display image IMG7 using a map without using a viewpoint image, as shown in FIG. 9. At this time, the position of the child U may be expressed by a marker P or the like, as shown in FIG. 9. The form of movement of the child U can be detected by, for example, activity recognition based on position information or the like.

[0062] Depending on the position of the child U, there may be cases where the viewpoint image is not stored in the street image server 20. Even in this case, the display image can be generated using a map as described above.

[0063] [1-3. Information Processing Procedure According to the Embodiment] Next, the procedure of information processing according to the embodiment will be described with reference to FIG.

[0064] FIG. 10 is a flowchart illustrating an example of information processing according to an embodiment of the present disclosure.

[0065] The communication unit 31 acquires information (position information) indicating the position of a child U who is a target to be watched over from the mobile terminal 10 (S101). In addition, the communication unit 31 acquires information (free viewpoint position information) indicating the position of a free viewpoint specified on the display terminal 40, which is an AR glass (HMD), from the display terminal 40 (S102).

[0066] The position information update management unit 32 determines whether the position of the free viewpoint has moved, that is, whether the AR glasses have moved (S103).

[0067] <Updating the displayed image according to the movement of the free viewpoint> When the AR glasses are moved (S103: Yes), the position information update management unit 32 generates a viewpoint image request based on the free viewpoint position information after the movement, and outputs it to the communication unit 31. Here, the movement of the AR glasses is generated means that the position of the free viewpoint has moved in 6DoF. The communication unit 31 transmits the viewpoint image request to the street image server 20, and receives a viewpoint image corresponding to the viewpoint image request from the street image server 20 (S104). The 360-degree image rendering unit 33 performs a rendering process to generate a 360-degree image for display based on the viewpoint image from the street image server 20 (S105). The 360-degree image rendering unit 33 outputs the generated 360-degree image for display to the free viewpoint image rendering unit 34. The free viewpoint image rendering unit 34 performs a rendering process to generate a rendering image of the free viewpoint corresponding to the field of view of the display terminal 40 based on the 360-degree image generated by the 360-degree image rendering unit 33 and the designation of the free viewpoint from the display terminal 40. The free viewpoint image rendering unit 34 outputs the generated free viewpoint image to the display terminal 40 via the communication unit 31. The display terminal 40 displays the generated free viewpoint image (S106). After that, the position information update management unit 32 determines whether or not to end the information processing (S107). When it is determined not to end the information processing (S107: No), the flow of FIG. 10 returns to the processing of S101. On the other hand, when it is determined to end the information processing (S107: Yes), the flow of FIG. 10 ends.

[0068] <Update the displayed image according to the movement of the target indoors or outdoors> When no movement of the AR glasses has occurred (S103: No), the position detection unit 131 determines whether the subject has moved between indoors and outdoors, that is, whether there has been an indoor / outdoor switch (S108). Here, no movement of the AR glasses means that the position of the free viewpoint has not moved or has moved with 3DoF.

[0069] When indoor / outdoor switching occurs (S108: Yes), the flow of Fig. 10 proceeds to the process of S104. In other words, when the AR glasses have not moved, the display image is updated in the same manner as described above.

[0070] If indoor / outdoor switching has not occurred (S108: No), the position detection unit 131 determines whether or not floor switching has occurred indoors (S109).

[0071] If the floor change is indoors (S109: Yes), the flow of Fig. 10 proceeds to the process of S104. That is, if the floor change is indoors, the display image is updated in the same manner as described above.

[0072] <Update the displayed image according to the movement of the target> If the floor change is not indoors (S109: No), the location information update management unit 32 calculates the amount of movement of the target from the difference in location information (S110). After that, the location information update management unit 32 determines whether or not a certain amount of movement of the target has occurred based on the calculated amount of movement (S111).

[0073] When a certain amount of movement of the object occurs (S111: Yes), the flow of Fig. 10 proceeds to the process of S104. In other words, when an amount of movement that exceeds the threshold for switching the viewpoint occurs, the display image is updated in the same manner as described above.

[0074] If the target has not moved a certain amount (S111: No), the 360-degree image rendering unit 33 generates a 360-degree image depicting the target's position by superimposing a marker or icon on the created 360-degree image (S112). Then, the process of FIG. 10 proceeds to S106. In other words, if the target has moved within a certain range, the display image is updated without receiving a viewpoint image from the street image server 20.

[0075] [1-4. Examples of use of the information processing system according to the embodiment] Here, a usage example of the information processing system 1 according to the embodiment will be described with reference to FIG.

[0076] FIG. 11 is a diagram for explaining a usage example of the information processing system 1 according to an embodiment of the present disclosure. In the example shown in FIG. 11, the orientation of the display image corresponds to the facing direction of the guardian wearing the AR glasses (display terminal 40). For example, a left-facing display image indicates an image displayed on the display terminal 40 when the guardian faces left. For example, a downward-facing display image indicates an image displayed on the display terminal 40 when the guardian looks back. Suppose that when the guardian is watching the field of view of the child U moving with the mobile terminal 10 in a stationary state, for example, the child U is in trouble at a dead end.

[0077] First, a display image IMG7 (360-degree image) centered on the position of the child U is displayed on the display terminal 40. For example, the direction of the child U in the display image IMG7 is in the initial state. This allows the guardian to easily grasp the situation the child U is facing, such as being stuck at a dead end. After that, the guardian starts moving in 6DoF while wearing the display terminal 40. At this time, the display image is updated with the movement of the position of the free viewpoint. The display image IMG8 is a display image when the guardian turns left and looks toward the dead end in a situation where the guardian's free viewpoint has moved to the opposite side of the dead end. In this way, an icon UV indicating the position of the child U may be superimposed and displayed based on the position information of the child U. This allows the guardian to easily grasp the relative positional relationship with the child U when moving virtually. In addition, the guardian can see the opposite side of the dead end by the display image IMG9 displayed when the guardian turns right, so that the guardian can search for a route for the child U to get out of the dead end. Based on the display image IMG9, the guardian can provide specific route guidance to the child U, for example, "When you reach the place where you can see the green house on your left, turn right." After that, based on the display image IMG10 updated as the position of the free viewpoint moves, the guardian can provide specific route guidance to the child U, for example, "Take the next road to the left." In this way, the guardian can check the bus stops from the display image IMG11 updated as the free viewpoint moves, without the child U moving.

[0078] In the embodiment, a monitoring service for a child U by a guardian is exemplified, but the present disclosure is not limited thereto. The target of the monitoring service according to the present disclosure may be, for example, an elderly person. Furthermore, the technology according to the present disclosure is not limited to a monitoring service, and may be realized as a route guidance service for delivery drivers, tourists, and the like.

[0079] [1-5. Hardware configuration] Fig. 12 is a block diagram showing an example of a hardware configuration of each device of the information processing system 1 according to the embodiment of the present disclosure. Information devices such as each device (mobile terminal 10, information processing device 30, and display terminal 40) of the information processing system 1 according to the embodiment described above are realized by a computer 1000 having a configuration as shown in Fig. 12, for example.

[0080] The computer 1000 includes a CPU 1100, a RAM 1200, a read only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected via a bus 1050 so as to be able to communicate with each other.

[0081] The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400 and controls each unit. For example, the CPU 1100 loads a program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs (information processing according to the present disclosure).

[0082] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000.

[0083] The HDD 1400 is a computer-readable recording medium that non-temporarily records the programs executed by the CPU 1100 and the data used by the programs.

[0084] The communication interface 1500 is an interface for connecting the computer 1000 to the network N or an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0085] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading a program or the like recorded on a predetermined recording medium. The medium may be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0086] For example, CPU 1100 of computer 1000 realizes each function of each device by executing a program loaded onto RAM 1200. Note that CPU 1100 reads program data 1450 from HDD 1400 and executes it, but as another example, these programs may be obtained from other devices via external network 1550.

[0087] (2. Effects of the Information Processing System According to the Present Disclosure) The information processing device 30 includes an acquisition unit (a communication unit 31, a position information update management unit 32), a generation unit (a 360-degree image rendering unit 33, a free viewpoint image rendering unit 34), and an output unit (the communication unit 31). The acquisition unit acquires a viewpoint image according to the viewpoint of the first user from a street image server 20 that stores a plurality of viewpoint images obtained by photographing a plurality of directions from each of a plurality of positions, based on position information indicating the position of a mobile terminal 10 moving with a first user. The generation unit uses the acquired viewpoint image to generate a rendering image of a free viewpoint designated by a second user at a position different from that of the first user as a display image. The output unit outputs the generated display image.

[0088] As a result, even when the information processing device 30 displays the field of view of the first user on the mobile terminal 10 on the display terminal 40 used by the second user, the amount of data transferred from the mobile terminal 10 to the display terminal 40 can be reduced. Therefore, according to the information processing device 30, the field of view of the first user on the mobile terminal 10 can be displayed at a free viewpoint on the display terminal 40 used by the second user at another position along with the movement of the first user, so that the second user can easily grasp the situation of the first user, while reducing the amount of data transfer and suppressing delays in interaction. Furthermore, taking a 360-degree image may involve complicated operations for the first user who is the target of watching. However, according to the information processing device 30, it is possible to eliminate the need for the first user who is the target of watching to take a 360-degree image on the mobile terminal 10. Furthermore, the display image output from the information processing device 30 is generated based on a viewpoint image acquired from a street image server 20 prepared in advance. Therefore, according to the information processing device 30, by using a viewpoint image to which image processing such as blurring has been applied in advance for the reflected subject, which is a subject that is unintentionally reflected within the angle of view, it is possible to reduce security risks caused by the reflected subject.

[0089] In the information processing device 30, the acquisition unit acquires a viewpoint image when the amount of movement of the mobile terminal 10 exceeds a predetermined threshold.

[0090] This allows the information processing device 30 to reduce the frequency with which the viewpoint images are updated.

[0091] In the information processing device 30, the acquisition unit does not acquire a viewpoint image when the change in the position of the mobile terminal 10 is equal to or smaller than a predetermined threshold. Also, in the information processing device 30, the generation unit generates a display image by superimposing an icon or marker indicating the position of the first user on the acquired viewpoint image.

[0092] This allows the information processing device 30 to represent the position of the first user in the display image even when the viewpoint image is not updated.

[0093] In the information processing device 30, the acquisition unit acquires a viewpoint image when the free viewpoint is moved by the second user.

[0094] This allows the information processing device 30 to update the viewpoint image when the free viewpoint moves.

[0095] In the information processing device 30, the acquisition unit acquires a viewpoint image when the location of the mobile terminal 10 switches between indoors and outdoors.

[0096] This allows the information processing device 30 to update the viewpoint image when the position of the mobile terminal 10 switches between indoors and outdoors.

[0097] In the information processing device 30, when there is no indoor viewpoint image where the mobile terminal 10 is located, the generation unit generates a display image by superimposing an icon or marker indicating the position of the first user on the viewpoint image acquired before the position of the mobile terminal switches from outdoors to indoors.

[0098] This allows the information processing device 30 to represent the position of the first user in the display image even if the first user enters a building for which no viewpoint image is prepared in the street image server 20.

[0099] In the information processing device 30, the acquisition unit acquires a viewpoint image when the mobile terminal 10 is located indoors and the floor is changed.

[0100] This allows the information processing device 30 to generate a display image using an appropriate viewpoint image even when the first user is indoors.

[0101] In the information processing device 30, when movement of the first user by a moving body (vehicle) is detected, the generation unit generates a display image by superimposing an icon or marker indicating the position of the first user on a map.

[0102] This allows the information processing device 30 to represent the position of the first user in the display image even in a situation where a viewpoint image is not prepared in the street image server 20. Furthermore, the information processing device 30 can suppress unnecessary updates even when updating the display image according to the amount of movement.

[0103] The information processing system 1 includes a mobile terminal 10, a street image server 20, an information processing device 30, and a display terminal 40. The mobile terminal 10 moves with a first user and outputs position information. The street image server 20 stores viewpoint images. The display terminal 40 moves with a second user and displays a display image.

[0104] This allows the information processing system 1 to suppress delays in interaction when the field of view of the first user on the mobile terminal 10 is displayed on the display terminal 40 used by the second user at another location. It also makes it unnecessary for the first user to perform a 360-degree image shooting operation on the mobile terminal 10, which is the target of monitoring. It also makes it possible to reduce security risks caused by reflected subjects.

[0105] In the information processing system 1, the display terminal 40 is a head-mounted display that is worn on the head of the second user.

[0106] This allows the information processing system 1 to specify a free viewpoint without any particular operation by the second user wearing the display terminal 40.

[0107] The information processing method includes acquiring a viewpoint image corresponding to the viewpoint of the first user from a street image server 20 that stores a plurality of viewpoint images obtained by photographing a plurality of directions from each of a plurality of positions based on position information indicating the position of a mobile terminal 10 moving with the first user, generating a rendering image of a free viewpoint specified by a second user at a position different from the first user as a display image using the acquired viewpoint image, and outputting the generated display image.

[0108] As a result, the information processing method can suppress delays in interaction when the field of view of the first user on the mobile terminal 10 is displayed on the display terminal 40 used by the second user at another location. Also, it is possible to eliminate the need for the first user who is the target of monitoring to perform a shooting operation of a 360-degree image on the mobile terminal 10. Also, it is possible to reduce security risks caused by reflected subjects.

[0109] The information processing method further includes acquiring a viewpoint image when the amount of movement of the mobile terminal 10 exceeds a predetermined threshold.

[0110] This makes it possible for the information processing method to reduce the frequency with which the viewpoint images are updated.

[0111] The information processing method further includes, when the change in position of the mobile terminal 10 is equal to or less than a predetermined threshold, not acquiring a viewpoint image, and generating a display image by superimposing an icon or marker indicating the position of the first user on the acquired viewpoint image.

[0112] This makes it possible for the information processing method to represent the position of the first user in the display image even when the viewpoint image is not updated.

[0113] The information processing method further includes acquiring a viewpoint image when the free viewpoint is moved by the second user.

[0114] Thereby, the information processing method can update the viewpoint image when the free viewpoint moves.

[0115] The information processing method further includes acquiring viewpoint images when the location of the mobile terminal 10 switches between indoors and outdoors.

[0116] In this way, the information processing method can update the viewpoint image when the position of the mobile terminal 10 switches between indoors and outdoors.

[0117] The information processing method further includes, when there is no indoor viewpoint image where the mobile terminal 10 is located, generating a display image by superimposing an icon or marker indicating the position of the first user on a viewpoint image acquired before the position of the mobile terminal switches from outdoors to indoors.

[0118] As a result, the information processing method can represent the position of the first user in the display image even if the first user enters a building for which the street image server 20 does not have a viewpoint image.

[0119] The program causes a computer to execute the following steps: acquire a viewpoint image corresponding to the viewpoint of the first user from a street image server 20 that stores a plurality of viewpoint images obtained by photographing a plurality of directions from each of a plurality of positions based on position information indicating the position of a mobile terminal 10 traveling with the first user; use the acquired viewpoint image to generate a rendering image of a free viewpoint specified by a second user at a position different from the first user as a display image; and output the generated display image.

[0120] This allows the program to suppress delays in interaction when the field of view of the first user on the mobile terminal 10 is displayed on the display terminal 40 used by the second user at another location. Also, it is possible to eliminate the need for the first user to perform a 360-degree image capture operation on the mobile terminal 10, which is the target of monitoring. Also, it is possible to reduce security risks caused by subjects that are caught in the image.

[0121] The program further causes the computer to acquire a viewpoint image when the amount of movement of the mobile terminal 10 exceeds a predetermined threshold.

[0122] This allows the program to update the viewpoint images less frequently.

[0123] The program further causes the computer to perform the following steps when the change in the position of the mobile terminal 10 is below a predetermined threshold: not acquiring a viewpoint image, and superimposing an icon or marker indicating the position of the first user on the acquired viewpoint image to generate a display image.

[0124] This allows the program to represent the position of the first user in the display image even when the viewpoint image is not updated.

[0125] The program further causes the computer to acquire a viewpoint image when the free viewpoint is moved by the second user.

[0126] This allows the program to update the viewpoint image when the free viewpoint moves.

[0127] The program further causes the computer to acquire viewpoint images when the location of the mobile terminal 10 switches between indoors and outdoors.

[0128] This allows the program to update the viewpoint image when the location of the mobile terminal 10 switches between indoors and outdoors.

[0129] The program further causes the computer to generate a display image when there is no indoor viewpoint image where the mobile terminal 10 is located by superimposing an icon or marker indicating the position of the first user on a viewpoint image acquired before the position of the mobile terminal switches from outdoors to indoors.

[0130] This allows the program to represent the position of the first user in the display image even if the first user enters a building for which the street image server 20 does not have a viewpoint image.

[0131] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0132] The present technology can also be configured as follows. (1) an acquisition unit that acquires a viewpoint image corresponding to a viewpoint of the first user from an image server that stores a plurality of viewpoint images obtained by photographing a plurality of directions from each of a plurality of positions based on position information indicating a position of a mobile terminal that moves together with the first user; a generating unit that generates, as a display image, a rendering image of a free viewpoint designated by a second user at a position different from that of the first user, using the acquired viewpoint image; an output unit that outputs the generated display image; An information processing device comprising: (2) The information processing device according to (1), wherein the acquisition unit acquires the viewpoint image when an amount of movement of the mobile terminal exceeds a predetermined threshold. (3) The acquisition unit does not acquire the viewpoint image when a change in the position of the mobile terminal is equal to or smaller than a predetermined threshold value, The generation unit generates the display image by superimposing an icon or a marker indicating the position of the first user on the acquired viewpoint image. The information processing device according to (1) or (2). (4) The information processing device according to any one of (1) to (3), wherein the acquisition unit acquires the viewpoint image when the free viewpoint is moved by the second user. (5) The information processing device according to any one of (1) to (4), wherein the acquisition unit acquires the viewpoint image when a position of the mobile terminal is switched between indoors and outdoors. (6) The information processing device described in (5), wherein, when there is no viewpoint image of indoors where the mobile terminal is located, the generation unit generates the display image by superimposing an icon or marker indicating the position of the first user on the viewpoint image acquired before the position of the mobile terminal switches from outdoors to indoors. (7) The information processing device according to any one of (1) to (6), wherein the acquisition unit acquires the viewpoint image when the mobile terminal is located indoors and the floor is changed. (8) The information processing device according to any one of (1) to (7), wherein the generation unit generates the display image by superimposing an icon or marker indicating the position of the first user on a map when movement of the first user by a moving body is detected. (9) the mobile terminal that moves with the first user and outputs the location information; The image server for storing the viewpoint images; the display terminal that moves with the second user and displays the display image; The information processing device according to any one of (1) to (8) above; An information processing system comprising: (10) The information processing system according to (9), wherein the display terminal is a head-mounted display worn on the head of the second user. (11) acquiring a viewpoint image corresponding to a viewpoint of the first user from an image server that stores a plurality of viewpoint images obtained by photographing a plurality of directions from each of a plurality of positions based on position information indicating a position of a mobile terminal that moves together with the first user; Using the acquired viewpoint images, a rendering image of a free viewpoint designated by a second user at a position different from that of the first user is generated as a display image; outputting the generated display image; An information processing method comprising: (12) The information processing method according to (11), further comprising acquiring the viewpoint image when an amount of movement of the mobile terminal exceeds a predetermined threshold. (13) The information processing method according to (11) or (12), further comprising: when a change in the position of the mobile terminal is equal to or less than a predetermined threshold, not acquiring the viewpoint image, and superimposing an icon or marker indicating the position of the first user on the acquired viewpoint image to generate the display image. (14) The information processing method according to any one of (11) to (13), further comprising acquiring the viewpoint image when the free viewpoint is moved by the second user. (15) The information processing method according to any one of (11) to (14), further comprising acquiring the viewpoint image when a position of the mobile terminal is switched between indoors and outdoors. (16) The information processing method described in (15), further comprising: when there is no viewpoint image of indoors where the mobile terminal is located, generating the display image by superimposing an icon or marker indicating the position of the first user on the viewpoint image acquired before the position of the mobile terminal switches from outdoors to indoors. (17) acquiring a viewpoint image corresponding to a viewpoint of the first user from an image server that stores a plurality of viewpoint images obtained by photographing a plurality of directions from each of a plurality of positions based on position information indicating a position of a mobile terminal that moves together with the first user; Using the acquired viewpoint images, a rendering image of a free viewpoint designated by a second user at a position different from that of the first user is generated as a display image; outputting the generated display image; A program for causing a computer to execute the following. (18) The program according to (17), further causing the computer to acquire the viewpoint image when the amount of movement of the mobile terminal exceeds a predetermined threshold. (19) The program described in (17) or (18), which further causes a computer to execute the following: when a change in the position of the mobile terminal is equal to or less than a predetermined threshold, not acquiring the viewpoint image, and superimposing an icon or marker indicating the position of the first user on the acquired viewpoint image to generate the display image. (20) The program according to any one of (17) to (19), further causing a computer to acquire the viewpoint image when the free viewpoint is moved by the second user. (twenty one) The program according to any one of (17) to (20), further causing a computer to acquire the viewpoint image when the position of the mobile terminal is switched between indoors and outdoors. (twenty two) The program described in (21), further causing a computer to execute, when there is no viewpoint image of indoors where the mobile terminal is located, generating the display image by superimposing an icon or marker indicating the position of the first user on the viewpoint image acquired before the position of the mobile terminal switches from outdoors to indoors. [Explanation of symbols]

[0133] 1. Information Processing Systems 10 Mobile terminals 11 Communications Department 111 Cellular module 112 Wi-Fi Module 113 BLE Module 12 Sensor section 121 GPS sensor 122 Geomagnetic Sensor 123 IMU 13 Detection section 131 Position detection unit 132 Outdoor position detection unit 133 Indoor position detection unit 134 Direction detection unit 135 Motion detection unit 14 Input / output section 20 Street Image Server 30 Information processing device 31 Communication unit (acquisition unit, output unit) 32 Location information update management section (acquisition section) 33 360-degree image rendering unit (generation unit) 34 Free viewpoint image rendering unit (generation unit) 40 Display terminal 41 Communications Department 42 IMU 43 Camera 44 Input / output section 441 Light guide plate lens N Network

Claims

1. an acquisition unit that acquires a viewpoint image based on location information indicating a location of a mobile terminal that moves together with the first user from an image server that stores a plurality of viewpoint images obtained by photographing a plurality of directions from each of a plurality of positions; a generating unit that generates, as a display image, a rendering image of a free viewpoint designated by a second user at a position different from that of the first user, using the acquired viewpoint image; Equipped with The generation unit sets an initial state of the display image based on a direction of the mobile terminal; The display image is updated by the generation unit in response to a change in the position of the mobile terminal and a designation of a free viewpoint by the movement or rotation of the second user. Information processing device.

2. The acquisition unit acquires the viewpoint image when a movement amount of the mobile terminal exceeds a predetermined threshold. The information processing device according to claim 1 .

3. The acquisition unit does not acquire the viewpoint image when a change in the position of the mobile terminal is equal to or smaller than a predetermined threshold value, The generation unit generates the display image by superimposing an icon or a marker indicating a position of the first user on the acquired viewpoint image. The information processing device according to claim 1 .

4. The acquisition unit acquires the viewpoint image when a position of the mobile terminal is switched between indoors and outdoors. The information processing device according to claim 1 .

5. When the viewpoint image of the indoor location where the mobile terminal is located does not exist, the generation unit generates the display image by superimposing an icon or a marker indicating the position of the first user on the viewpoint image acquired before the position of the mobile terminal is switched from outdoors to indoors. The information processing device according to claim 4.

6. the acquisition unit acquires the viewpoint image when the mobile terminal is located indoors and the floor is changed. The information processing device according to claim 1 .

7. the acquisition unit acquires the viewpoint image when the free viewpoint is moved by the second user.

7. The information processing device according to claim 2, 3, 4 or 6.

8. the generation unit generates the display image by superimposing an icon or a marker indicating a position of the first user on a map when movement of the first user by a moving body is detected.

7. The information processing device according to claim 2, 3, 4 or 6.

9. acquiring, from an image server storing a plurality of viewpoint images obtained by photographing a plurality of directions from each of a plurality of positions, viewpoint images based on position information indicating a position of a mobile terminal moving together with the first user; Using the acquired viewpoint image, a rendering image of a free viewpoint designated by a second user at a position different from that of the first user is generated as a display image; Including, an initial state of the display image is set based on a bearing of the mobile terminal; The display image is updated in response to a change in the position of the mobile terminal and a designation of a free viewpoint by a movement or rotation of the second user. Information processing methods.

10. and acquiring the viewpoint image when a movement amount of the mobile terminal exceeds a predetermined threshold. The information processing method according to claim 9.

11. and when a change in the position of the mobile terminal is equal to or smaller than a predetermined threshold, not acquiring the viewpoint image, and superimposing an icon or a marker indicating the position of the first user on the acquired viewpoint image to generate the display image. The information processing method according to claim 9.

12. The method further includes acquiring the viewpoint image when the location of the mobile terminal is switched between indoors and outdoors. The information processing method according to claim 9.

13. When the viewpoint image of the indoor location where the mobile terminal is located does not exist, the display image is generated by superimposing an icon or a marker indicating the position of the first user on the viewpoint image acquired before the position of the mobile terminal is switched from outdoors to indoors. The information processing method according to claim 12.

14. acquiring the viewpoint image when the free viewpoint is moved by the second user.

13. The information processing method according to claim 10, 11 or 12.

15. acquiring, from an image server storing a plurality of viewpoint images obtained by photographing a plurality of directions from each of a plurality of positions, viewpoint images based on position information indicating a position of a mobile terminal moving together with the first user; Using the acquired viewpoint image, a rendering image of a free viewpoint designated by a second user at a position different from that of the first user is generated as a display image; setting an initial state of the display image based on a bearing of the mobile terminal; updating the display image in response to a change in the position of the mobile terminal and a designation of a free viewpoint by a movement or rotation of the second user; A program for causing a computer to execute the following.

16. acquiring the viewpoint image when a movement amount of the mobile terminal exceeds a predetermined threshold; The program according to claim 15, further causing a computer to execute the steps.

17. when a change in the position of the mobile terminal is equal to or smaller than a predetermined threshold, the viewpoint image is not acquired, and an icon or marker indicating the position of the first user is superimposed on the acquired viewpoint image to generate the display image; The program according to claim 15, further causing a computer to execute the steps.

18. acquiring the viewpoint image when the position of the mobile terminal is switched between indoors and outdoors; The program according to claim 15, further causing a computer to execute the steps.

19. and causing the computer to further execute, when there is no viewpoint image of the indoor location where the mobile terminal is located, generating the display image by superimposing an icon or marker indicating the position of the first user on the viewpoint image acquired before the location of the mobile terminal is switched from outdoors to indoors.

20. The program of claim 18.

20. and causing the computer to further execute acquiring the viewpoint image when the free viewpoint is moved by the second user.

19. A program according to claim 16, 17 or 18.

Images