Information processing device, information processing method, and program

Abstract

An information processing device includes an acquisition unit, a generation unit, and an output unit. The acquisition unit acquires a viewpoint image from an image server based on position information indicating the position of a mobile terminal traveling with a first user. The image server stores a plurality of viewpoint images obtained by capturing images from a plurality of orientations at each of a plurality of positions. The generation unit uses the acquired viewpoint images to generate a rendered image as a display image, rendered from a free viewpoint specified by a second user located at a position different from the first user's position. The output unit outputs the generated display image.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. Background Art

[0002] Recently known monitoring services are based on location information about a mobile terminal, such as a mobile phone or a Bluetooth (registered trademark) Low Energy (BLE) tag that moves with a user to be monitored (such as a child), to inform a guardian of the location and movement of the user to be monitored.

[0003] In addition, a service using augmented reality (AR) technology is known that guides users wearing head-mounted displays (HMDs) along a route. Devices implementing AR technology (such as HMDs) can display images indicating the user's position and direction based on the movement of the user's viewpoint, using, for example, 360-degree images captured by a 360-degree camera.

[0004] Citation List

[0005] Patent Literature

[0006] Patent Document 1: JP 2018-502360 A Summary of the Invention

[0007] Technical issues

[0008] However, 360-degree images require a large amount of image data. Therefore, when displaying the field of view of a mobile terminal user on a display terminal such as an HMD used by a user at a different location, the amount of data transmitted from the mobile terminal to the display terminal becomes large, and in some cases, the large amount of data may cause interaction delays.

[0009] Therefore, the present disclosure proposes an information processing device, an information processing method, and a program configured to suppress interaction delay when displaying a field of view of a user of a mobile terminal on a display terminal used by a user located at a different location.

[0010] Solution to the problem

[0011] According to the present disclosure, an information processing device includes an acquisition unit, a generation unit, and an output unit. The acquisition unit acquires a viewpoint image from an image server based on position information indicating the position of a mobile terminal moving with the first user. The image server stores a plurality of viewpoint images obtained by capturing images from a plurality of orientations at each of a plurality of positions. The generation unit uses the acquired viewpoint images to generate a rendered image as a display image, rendered from a free viewpoint specified by a second user located at a position different from the first user's position. The output unit outputs the generated display image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram illustrating a configuration example of an information processing system according to an embodiment of the present disclosure.

[0013] Figure 2 is a schematic diagram illustrating an overview of information processing according to an embodiment of the present disclosure.

[0014] Figure 3 is a diagram illustrating an overview of other information processing for transmitting image data captured by a mobile terminal that is different from the information processing according to an embodiment of the present disclosure.

[0015] Figure 4 is a diagram illustrating generation of a display image in information processing according to an embodiment of the present disclosure.

[0016] Figure 5 is a diagram illustrating updating of a display image as a target or a free viewpoint position moves in information processing according to an embodiment of the present disclosure.

[0017] Figure 6 is a diagram illustrating updating of viewpoint images according to a movement amount of an object in information processing according to an embodiment of the present disclosure.

[0018] Figure 7 is a diagram illustrating updating of a display image in response to rotation at a free viewpoint in information processing according to an embodiment of the present disclosure.

[0019] Figure 8 is a diagram illustrating updating of a display image according to movement indoors and outdoors in information processing according to an embodiment of the present disclosure.

[0020] Figure 9 is a diagram illustrating generation of a display image without using viewpoint image data in information processing according to an embodiment of the present disclosure.

[0021] Figure 10 is a flowchart illustrating an example of information processing according to an embodiment of the present disclosure.

[0022] Figure 11 is a diagram illustrating a usage example of the information processing system according to an embodiment of the present disclosure.

[0023] Figure 12 is a block diagram illustrating an example of a hardware configuration of each device of the information processing system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in the following embodiments, identical parts will be denoted by identical reference numerals and symbols, and further repeated description thereof will be omitted.

[0025] The present disclosure will be described in the order of the items shown below.

[0026] 1. Example

[0027] 1-1. Functional Configuration of Information Processing System According to Embodiment

[0028] 1-2. Example of information processing according to the embodiment

[0029] 1-2-1. Update the display image according to the movement of the target or free viewpoint

[0030] 1-2-2. Update the display image based on the movement of indoor and outdoor objects

[0031] 1-2-3. Generating display images without using viewpoint images

[0032] 1-3. Information Processing Process According to the Embodiment

[0033] 1-4. Use Example of Information Processing System According to Embodiment

[0034] 1-5. Hardware Configuration

[0035] 2. Effects of the Information Processing System According to the Present Disclosure

[0036] (1. Example)

[0037] This embodiment illustrates a monitoring service that obtains the location information of a mobile terminal that moves with a first user as a target to detect the location and movement of the first user. Specifically, in the embodiment, an information processing system is illustrated that, based on the location information of the mobile terminal, displays an image corresponding to the first user's field of view for a second user located at a different location from the first user, without receiving image data from the mobile terminal. In the following description, it is assumed that the first user is a child who moves with the mobile terminal. In addition, it is assumed that the second user is a guardian who monitors the location and movement of the child by using a monitoring service implemented by the information processing system according to the embodiment.

[0038] [1-1. Functional Configuration of Information Processing System According to Embodiment]

[0039] Will refer to Figure 1 An example of the functional configuration of the information processing system 1 according to the embodiment is described. Figure 1 is a block diagram illustrating a configuration example of the information processing system 1 according to an embodiment of the present disclosure.

[0040] like Figure 1As shown, the information processing system 1 according to the embodiment includes a mobile terminal 10, a street image server 20, an information processing device 30, and a display terminal 40. The mobile terminal 10, the street image server 20, and the information processing device 30 are communicably connected to each other via a network N. In addition, the information processing device 30 and the display terminal 40 are communicably connected to each other. As the network N, a telecommunication line such as the Internet can be used.

[0041] Note that the mobile terminal 10 and the street image server 20 may be connected to the information processing device 30 via different networks N. Furthermore, the information processing device 30 and the display terminal 40 may be communicably connected via the network N.

[0042] The mobile terminal 10 is a communication terminal that moves with the child. The mobile terminal 10 executes a program (application) for implementing information processing according to the present disclosure and functions as the mobile terminal 10 according to the present disclosure. As the mobile terminal 10, a dedicated terminal or a general-purpose communication terminal such as a smartphone can be used.

[0043] like Figure 1 As shown, the mobile terminal 10 has functions as a communication unit 11 , a sensor unit 12 , a detection unit 13 , and an input / output unit 14 .

[0044] The communication unit 11 is wirelessly connected to the network N to communicate with the information processing device 30 via the network N. Figure 1 As shown, the communication unit 11 includes a cellular module 111, a Wi-Fi (registered trademark) module 112, and a Bluetooth (registered trademark) low energy (BLE) module 113. The cellular module 111 is a communication circuit compatible with various standards such as 3G, 4G, and 5G. The Wi-Fi module 112 is a communication circuit compatible with the Wi-Fi standard. The BLE module 113 is a communication circuit compatible with the BLE standard.

[0045] In one example, the communication unit 11 establishes communication with a mobile phone base station via the cellular module 111 and outputs information indicating the mobile phone base station with which communication has been established or its location to the detection unit 13. The communication unit 11 establishes communication with a Wi-Fi access point via the Wi-Fi module 112 and outputs information indicating the Wi-Fi access point with which communication has been established or its location to the detection unit 13. The communication unit 11 receives a BLE beacon from a BLE tag or the like via the BLE module 113 and outputs information indicating the radio wave strength of the received BLE beacon, the BLE tag that transmitted the received BLE beacon, or the location of the BLE tag to the detection unit 13. Furthermore, the communication unit 11 transmits information indicating the position, direction, and movement of the mobile terminal 10 detected by the detection unit 13 to the information processing device 30. Here, the information indicating the position of the mobile terminal 10 includes, for example, latitude and longitude values. The information indicating the movement of the mobile terminal 10 is, for example, information indicating the start of movement of the mobile terminal 10 or information indicating the stationary state of the mobile terminal 10.

[0046] The sensor unit 12 detects the position, direction, and movement of the mobile terminal 10. In addition, the sensor unit 12 outputs detection information indicating the detected position, direction, and movement of the mobile terminal 10 to the detection unit 13. Figure 1 As shown, the sensor unit 12 includes a global positioning system (GPS) sensor 121 , a geomagnetic sensor 122 , and an inertial measurement unit (IMU) 123 .

[0047] In one example, the sensor unit 12 detects the latitude and longitude of the mobile terminal 10 through the GPS sensor 121 and outputs information indicating the detected latitude and longitude to the detection unit 13. The sensor unit 12 detects the direction of the mobile terminal 10 through the geomagnetic sensor 122 and outputs information indicating the detected direction to the detection unit 13. The sensor unit 12 detects the acceleration and angular velocity of the mobile terminal 10 through the IMU 123 and outputs information indicating the detected acceleration and angular velocity to the detection unit 13. Here, the information indicating the acceleration and angular velocity can also be expressed as information indicating a change in the position and attitude of the mobile terminal 10.

[0048] The detection unit 13 detects information indicating the position, direction, and movement of the mobile terminal 10 based on the outputs of the communication unit 11 and the sensor unit 12. In addition, the detection unit 13 outputs detection information indicating the detected position, direction, and movement of the mobile terminal 10 to the communication unit 11. Figure 1 As shown, the detection unit 13 has functions as a position detection unit 131, a direction detection unit 134, and a motion detection unit 135. The position detection unit 131 has functions as an outdoor position detection unit 132 and an indoor position detection unit 133.

[0049] The outdoor position detection unit 132 detects information indicating the location of the mobile terminal 10 based on the output of at least one of the cellular module 111, the GPS sensor 121, and the Wi-Fi module 112. The indoor position detection unit 133 detects information indicating the indoor location of the mobile terminal 10 based on the output of at least one of the Wi-Fi module 112 and the BLE module 113. The indoor location of the mobile terminal 10 includes the height (floor, level) at which the mobile terminal 10 is located. The direction detection unit 134 detects information indicating the direction of the mobile terminal 10 based on the output of the geomagnetic sensor 122. The motion detection unit 135 detects information indicating the direction of the mobile terminal 10 based on the output of the IMU 123.

[0050] Note that the position detection unit 131 may be configured to estimate the current position of the mobile terminal 10 based on information indicating the last detected position of the mobile terminal 10 and information indicating the direction and movement of the currently detected mobile terminal 10 .

[0051] The input / output unit 14 receives input of user voice. The input / output unit 14 outputs the received voice data to the communication unit 11. In addition, the input / output unit 14 outputs voice based on the voice data received by the communication unit 11 from the information processing device 30 or the display terminal 40. The input / output unit 14 includes an input device such as a microphone and an output device such as a speaker.

[0052] Note that the input / output of voice via the input / output unit 14 represents, for example, a call with a user (guardian) using the display terminal 40, but may include recording and playback of voice messages, or may include conversion of voice to text data and conversion of text data to voice.

[0053] Note that when the child can use the function as the input / output unit 14 through another device, for example, when the child has another communication terminal such as a smartphone, the mobile terminal 10 may not have the input / output unit 14 .

[0054] The street image server 20 is an image server (server device) configured to store a database of multiple viewpoint images (street images, 360-degree images) obtained by capturing images from multiple orientations (directions) at each of multiple locations. Each of the multiple viewpoint images stored in the street image server 20 is assumed to be associated with information indicating the position and direction of the captured image. Furthermore, each of the multiple viewpoint images stored in the street image server 20 is assumed to be an image in which unintentionally captured objects have been appropriately processed. For example, Google Street View (registered trademark) can be used as the street image server 20.

[0055] The information processing device 30 generates an image corresponding to the child's field of view visible from a free viewpoint designated by the guardian, based on information indicating the position of the mobile terminal 10 that is moving with the child, without receiving image data from the mobile terminal 10, and outputs the image to the display terminal 40 used by the guardian. The information processing device 30 executes a program (application) for implementing information processing according to the present disclosure and functions as the information processing device 30 according to the present disclosure. As the information processing device 30, a general-purpose communication terminal such as a smartphone, or a device such as a personal computer (PC) or a tablet PC can be appropriately used.

[0056] like Figure 1 As shown, the information processing device 30 has functions as a communication unit 31 , a position information update management unit 32 , a 360-degree image rendering unit 33 , and a free viewpoint image rendering unit 34 .

[0057] The communication unit 31 is wirelessly connected to the network N to communicate with the mobile terminal 10 and the street image server 20 via the network N. The communication unit 31 receives information indicating the position, direction, and movement of the mobile terminal 10 from the mobile terminal 10. The communication unit 31 sends a viewpoint image request from the position information update management unit 32 to the street image server 20. The communication unit 31 receives a viewpoint image from the street image server 20 according to the viewpoint image request. In addition, the communication unit 31 sends the free viewpoint image generated by the free viewpoint image rendering unit 34 to the display terminal 40. The communication unit 31 receives information indicating the 6DoF (degrees of freedom) movement of the display terminal 40 from the display terminal 40. The communication unit 31 includes a communication circuit for wired communication, a communication circuit for wireless communication, and a combination thereof. As the communication circuit for wireless communication, a communication circuit compatible with various standards such as 3G, 4G, 5G, Wi-Fi, and BLE can be appropriately used. Here, the communication unit 31 is an example of an acquisition unit and an output unit.

[0058] Based on the information indicating the detected position, direction, and movement of the mobile terminal 10 received by the communication unit 31, the position information update management unit 32 outputs a viewpoint image request to the street image server 20 for requesting viewpoint images. The position information update management unit 32 outputs the viewpoint image request when the amount of movement of the mobile terminal 10 exceeds a predetermined threshold. Furthermore, the position information update management unit 32 outputs the viewpoint image request when the free viewpoint moves with the movement of the display terminal 40, for example, when a guardian wearing the display terminal 40 moves. Here, the position information update management unit 32 is an example of an acquisition unit.

[0059] The 360-degree image rendering unit 33 performs rendering processing for generating a 360-degree image for display based on the viewpoint image from the street image server 20. The 360-degree image rendering unit 33 outputs the generated 360-degree image for display to the free viewpoint image rendering unit 34. Here, the 360-degree image rendering unit 33 is an example of a generating unit.

[0060] The free viewpoint image rendering unit 34 performs rendering processing to generate a rendered image rendered from a free viewpoint according to the field of view of the display terminal 40 based on the 360-degree image generated by the 360-degree image rendering unit 33 and the designation of the free viewpoint from the display terminal 40. Hereinafter, the rendered image rendered from the free viewpoint generated in this processing will be described as a free viewpoint image. Note that, in one example, the designation of the free viewpoint from the display terminal 40 means the designation of the free viewpoint based on the output of the IMU 42 and / or the camera 43 of the display terminal 40. In other words, the designation of the free viewpoint includes the perception of the 6DoF (degrees of freedom) movement of the display terminal 40 in the three-dimensional space. The free viewpoint image rendering unit 34 outputs the generated free viewpoint image to the display terminal 40 via the communication unit 31. Here, the free viewpoint image rendering unit 34 is an example of a generating unit.

[0061] The display terminal 40 displays a free viewpoint image that is generated based on the location information of the mobile terminal 10 and expresses the child's field of view at the free viewpoint. The display terminal 40 is a display terminal device configured to implement a service using augmented reality (AR) technology. In one example, the display terminal 40 is a head-mounted display (HMD) worn by a guardian. The display terminal 40 is, for example, an optical see-through HMD. Figure 1 As shown, the display terminal 40 has functions as a communication unit 41, a sensor unit, and an input / output unit 44. The sensor unit includes an IMU 42 and a camera 43.

[0062] The communication unit 41 receives the free viewpoint image from the information processing device 30. Furthermore, the communication unit 41 transmits information indicating the 6DoF movement of the display terminal 40 to the information processing device 30. The communication unit 41 includes a communication circuit for wired communication, a communication circuit for wireless communication, and a combination thereof. As the communication circuit for wireless communication, a communication circuit compatible with various standards such as 3G, 4G, 5G, Wi-Fi, and BLE can be appropriately used.

[0063] The sensor unit detects information indicating the 6DoF movement of the display terminal 40. Specifically, the sensor unit detects the acceleration and angular velocity of the display terminal 40 via the IMU 42, and transmits information indicating the detected acceleration and angular velocity to the information processing device 30 via the communication unit 41. Furthermore, the sensor unit detects the movement or rotation of the display terminal 40 based on, for example, the movement or deformation of feature points on an image obtained by the camera 43, and transmits information indicating the detected movement or rotation to the information processing device 30 via the communication unit 41. In other words, a guardian wearing the display terminal 40 can specify a free viewpoint through his / her own movement or rotation without any special operation.

[0064] The input / output unit 44 receives input of user voice. The input / output unit 44 outputs the received voice data to the communication unit 41. In addition, the input / output unit 44 outputs voice based on the voice data received from the mobile terminal 10 or the information processing device 30 through the communication unit 41. The input / output unit 44 includes an input device such as a microphone and an output device such as a speaker. In addition, Figure 1 As shown, the input / output unit 44 includes a light guide lens 441. In the display terminal 40, which is an optical see-through HMD, a virtual image optical system using, for example, a half mirror or a transparent light guide lens 441 is held in front of the user's eyes, and an image is displayed within the virtual image optical system. Therefore, the user wearing the optical see-through HMD can see the scenery around the user even while viewing the image displayed within the virtual image optical system.

[0065] Note that display terminal 40 may be, for example, a video see-through HMD. A video see-through HMD is worn by the user so as to cover the user's eyes, and its display is held in front of the user's eyes. Furthermore, the video see-through HMD includes a camera 43 for capturing images of the scenery in front of the user, and the scenery images captured by camera 43 are displayed on the display. Therefore, while it is difficult for a user wearing the video see-through HMD to directly see the scenery in front of them, the user can confirm the scenery in front of them through the image on the display.

[0066] Note that the information processing apparatus 30 and the display terminal 40 may be configured integrally.

[0067] [1-2. Example of information processing according to the embodiment]

[0068] Here, we will refer to Figure 2 and Figure 3 An overview of information processing according to an embodiment is described.

[0069] In a recently known monitoring service, a guardian is notified of the location or movement of a target user (such as a child) based on location information of a mobile terminal 10 (such as a mobile phone or a BLE tag) that moves with the target user.

[0070] In addition, services using AR technology are known that guide users wearing HMDs along a route. Devices implementing AR technology, such as HMDs, can display images indicating the position and direction (free viewpoint images) according to the movement of the user's viewpoint by using, for example, 360-degree images captured by a 360-degree camera.

[0071] Figure 2 1 is a schematic diagram illustrating an overview of information processing according to an embodiment of the present disclosure. A mobile terminal 10 is held by a child U, a user who is the target of monitoring, and moves with the child U. Mobile terminal 10 transmits GEO data including the location information of mobile terminal 10 to information processing device 30 via network N. Based on the GEO data from mobile terminal 10, information processing device 30 acquires viewpoint images from street image server 20 at the moment the target or free viewpoint moves, and generates free viewpoint images. Display terminal 40 displays the free viewpoint images from information processing device 30.

[0072] Meanwhile, it is considered that a free viewpoint image is generated based on a 360-degree image captured by a child U who is a target to be monitored. Figure 3 1 is a schematic diagram illustrating an overview of other information processing for transmitting image data captured by the mobile terminal 10a' that is different from the information processing according to an embodiment of the present disclosure. Figure 3 In the example shown, the mobile terminal 10a' is a 360-degree camera. In addition, the mobile terminal 10b' is a smartphone that transmits 360-degree moving video data captured by the mobile terminal 10a' to the information processing device 30' via the network N. In the information processing implemented by this configuration, Figure 3 As shown, the actual image can be transmitted in real time to the display terminal 40. In addition, the free viewpoint image can be generated without using the street image server 20. Therefore, after the street image server 20 is prepared, an information processing system that is not affected by changes such as the disappearance of buildings can be realized.

[0073] However, in Figure 3 In the illustrated example, unlike the information processing according to the embodiment, image data is also transmitted during communication between the mobile terminal 10b' and the information processing device 30'. Generally speaking, image data displaying a 360-degree image is large in size. For example, the amount of data in such communication can reach several hundred megabits per second. Therefore, when displaying the field of view of a user of the mobile terminal 10 on a display terminal 40, such as an HMD used by a user in a different location, for example, the amount of data transmitted from the mobile terminal 10 to the display terminal 40 becomes large, and in some cases, this large amount of data may cause interaction delays.

[0074] Therefore, the present disclosure proposes an information processing device 30, an information processing system 1, an information processing method, and a program configured to suppress interaction delay when displaying the visual field of a user of a mobile terminal 10 on a display terminal 40 used by a user at a different location. Specifically, in the information processing according to the embodiment, 360-degree image data is not transmitted from the mobile terminal 10 to the information processing device 30, as Figure 2 As shown. For example, the amount of GEO data (position information) transmitted from the mobile terminal 10 to the information processing device 30 is as low as several thousand bits per second. Therefore, for example, when displaying the field of view of the user of the mobile terminal 10 on the display terminal 40 such as an HMD used by a user at a different location, the amount of data transmitted from the mobile terminal 10 to the display terminal 40 is small, and the small amount of data makes it possible to suppress interaction delays.

[0075] In addition, in some cases, capturing a 360-degree image may require complex operations of the user's child U, who is the target of monitoring. However, in the mobile terminal 10 of the user's child U, who is the target of monitoring, the information processing according to the embodiment does not require an operation of capturing an image.

[0076] Furthermore, in the information processing according to this embodiment, a free viewpoint image is generated for display on the display terminal 40 based on the viewpoint image acquired from the street image server 20. Here, the street image server 20 is an image server that is prepared in advance before the free viewpoint image is generated. In other words, image processing such as blurring can be performed in advance on objects that are not intended to be captured, i.e., unexpected objects within the viewing angle.

[0077] [1-2-1. Update the display image according to the movement of the target or free viewpoint]

[0078] Will refer to Figure 4 、 Figure 5 、 Figure 6 and Figure 7 Generation and updating of a display image in information processing according to an embodiment will be described.

[0079] Figure 4is a diagram illustrating the generation of a display image in the information processing according to an embodiment of the present disclosure. The information processing according to the embodiment includes a process for acquiring a viewpoint image based on information indicating the position and direction of the mobile terminal 10, and a process for generating a 360-degree image that reproduces the field of view of the child U as a target to be monitored based on the acquired viewpoint image. In addition, the information processing according to the embodiment includes a process S1 for generating a 360-degree image (free viewpoint image) visible from a free viewpoint that can move freely on the 360-degree image as a display image. Here, the position and direction of the free viewpoint on the virtual space VS change according to the movement of the guardian wearing the display terminal 40.

[0080] Figure 5 This is a diagram illustrating updating of a display image as the position of a target or free viewpoint moves in information processing according to an embodiment of the present disclosure. The information processing according to the embodiment includes processing S2 for updating a display image according to the movement of the free viewpoint. Processing S2 is a process performed when the movement of the free viewpoint is detected. Processing S2 includes processing for acquiring a viewpoint image, processing for generating a 360-degree image from the acquired viewpoint image, and processing S1 for generating the free viewpoint image as a display image, which refers to Figure 4 Furthermore, the information processing according to the embodiment includes processing S3 for updating the display image according to the movement of the child U.

[0081] Figure 6 1 is a diagram illustrating updating of a viewpoint image according to a movement amount of a target in information processing according to an embodiment of the present disclosure. Figure 6 On the upper side of FIG, viewpoint images IMG1, IMG2, IMG3, IMG4, and IMG5 are shown corresponding to the position of a child U who is a target moving in the visual field direction along the sidewalk.

[0082] In one example, in process S3, in response to the movement of the child U, viewpoint images are sequentially acquired according to the position of the child U, and free viewpoint images are generated using the viewpoint images. However, when viewpoint images are acquired from the street image server 20 each time the position of the child U changes, the amount of data transmitted from the street image server 20 to the information processing device 30 becomes large, and the large amount of data may cause interaction delays.

[0083] Therefore, in another example, in processing S3, a viewpoint image IMG1 according to the position of the child U is acquired, and a free viewpoint image according to the position of the child U is generated by using the viewpoint image IMG1 until the movement amount of the child U exceeds a predetermined threshold. The predetermined threshold can be freely set to, for example, 30 m. Figure 6The lower side of the figure shows free viewpoint images IMG1-1, IMG1-2, IMG1-3, and IMG1-4 generated using viewpoint image IMG1. Specifically, in process S3, an icon UV indicating the position of child U is superimposed on the same viewpoint image IMG1 until the amount of movement of child U exceeds a predetermined threshold, thereby generating free viewpoint images IMG1-2, IMG1-3, and IMG1-4. In other words, in process S3, the icon UV is superimposed or moved on the same viewpoint image until the amount of movement of child U exceeds a predetermined threshold, rather than updating the viewpoint image. This allows the generation of corresponding free viewpoint images based on the position of child U. Simultaneously, in process S3, when the amount of movement of child U exceeds a predetermined threshold, a viewpoint image IMG5 based on the position of child U is obtained from the street image server 20, and free viewpoint image IMG5-1 is generated using viewpoint image IMG5. As described above, in process S3, the update frequency of the viewpoint images changes according to the amount of movement of the target. This makes it possible to suppress an increase in the amount of data transmitted from the street image server 20 to the information processing device 30 .

[0084] Figure 7 This figure illustrates updating a display image in response to rotation at a free viewpoint in information processing according to an embodiment of the present disclosure. The information processing according to this embodiment includes process S4 for updating a display image in response to the rotation of display terminal 40. Process S4 is performed when display terminal 40 rotates without moving the free viewpoint. In process S4, a free viewpoint image corresponding to the rotation of display terminal 40 is generated from the acquired viewpoint image without updating the viewpoint image.

[0085] [1-2-2. Update the display image based on the movement of indoor and outdoor objects]

[0086] Will refer to Figure 8 The description is given of updating a display image according to the movement of indoor and outdoor objects in the information processing according to the embodiment.

[0087] Figure 8 is a diagram illustrating updating of a display image according to movement indoors and outdoors in information processing according to an embodiment of the present disclosure. In some cases, the child U enters and leaves the building B, that is, moves outdoors or moves indoors. In the information processing according to the embodiment, when the street image server 20 stores viewpoint images of the interior of the building B, a free viewpoint image is generated as a display image based on the indoor position information as described above. On the other hand, the street image server 20 may not store viewpoint images of the interior of the building B. In this case, in the information processing according to the embodiment, a free viewpoint image IMG6 is generated as a display image based on viewpoint images of the building B including those captured from any position outside the building B, as Figure 8Specifically, in the information processing according to the embodiment, a free viewpoint image IMG6 expressing the child U in the building B is generated by superimposing an icon UV indicating the child U at the position of the building B. The viewpoint image including the building B can be, for example, a viewpoint image acquired immediately before the child U enters the building B.

[0088] Note that it is preferable to express that the child U is inside the building B. For example, it is possible to make the area of ​​the building B in the viewpoint image transparent and then allow the icon UV to be placed in the area.

[0089] Note that the position of the icon UV superimposed on the viewpoint image can be moved based on the indoor position information of the child U. For example, the position of the superimposed icon UV moves in the height direction of the building B according to the movement of the child U between floors in the building. In addition, the viewpoint image on which the icon UV is to be superimposed can be updated based on the indoor position information of the child U.

[0090] [1-2-3. Display image generation without using viewpoint images]

[0091] Will refer to Figure 9 A case where a display image is generated without using viewpoint images in information processing according to the embodiment is described.

[0092] Figure 9 is a diagram illustrating the generation of a display image without using a viewpoint image in the information processing according to an embodiment of the present disclosure. In some cases, the child U travels in a vehicle (moving object) such as a train, a bus, or a bicycle. In this case, the frequency of acquiring viewpoint images from the street image server 20 increases in response to the travel of the child U, and there is a possibility that the interaction may be delayed. In addition, the viewpoint images stored in the street image server 20 cannot express the field of view of the child U in the train. Therefore, in the information processing according to the embodiment, when it is detected that the child U is traveling in a vehicle such as a train or a bus, a display image IMG7 is generated by using a map without using a viewpoint image, as shown in FIG. Figure 9 At this time, the position of the child U is preferably represented by a mark P, etc., as shown in FIG. Figure 9 Note that the movement pattern of the child U can be detected by, for example, activity recognition based on position information or the like.

[0093] Note that the street image server 20 may not store the viewpoint image depending on the position of the child U. In this case as well, the display image may be generated using the map, as described above.

[0094] [1-3. Information processing process according to the embodiment]

[0095] Next, we will refer to Figure 10The procedure of information processing according to the embodiment is described.

[0096] Figure 10 is a flowchart illustrating an example of information processing according to an embodiment of the present disclosure.

[0097] The communication unit 31 acquires information indicating the position of the child U as a target to be monitored (position information) from the mobile terminal 10 (S101). In addition, the communication unit 31 acquires information indicating the position of the free viewpoint specified by the display terminal 40 as AR glasses (HMD) (free viewpoint position information) from the display terminal 40 (S102).

[0098] The position information update management unit 32 determines whether the position of the free viewpoint has moved, that is, whether the AR glasses have moved ( S103 ).

[0099] <Update the display image according to the movement of the free viewpoint>

[0100] If the movement of the AR glasses has occurred (S103: Yes), the position information update management unit 32 generates a viewpoint image request based on the free viewpoint position information after the movement of the AR glasses, and outputs the viewpoint image request to the communication unit 31. Here, the movement of the AR glasses means that the position of the free viewpoint has moved by 6DoF. The communication unit 31 sends a viewpoint image request to the street image server 20 and receives a viewpoint image according to the viewpoint image request from the street image server 20 (S104). The 360-degree image rendering unit 33 performs rendering processing based on the viewpoint image from the street image server 20 to generate a 360-degree image for display (S105). The 360-degree image rendering unit 33 outputs the generated 360-degree image for display to the free viewpoint image rendering unit 34. The free viewpoint image rendering unit 34 performs rendering processing based on the 360-degree image generated by the 360-degree image rendering unit 33 and the designation of the free viewpoint from the display terminal 40 to generate a rendered image rendered from the free viewpoint according to the field of view of the display terminal 40. Furthermore, the free viewpoint image rendering unit 34 outputs the generated free viewpoint image to the display terminal 40 via the communication unit 31. Furthermore, the display terminal 40 displays the generated free viewpoint image (S106). Then, the position information update management unit 32 determines whether to end the information processing (S107). If it is determined that the information processing is not ended (S107: No), then Figure 10 On the other hand, when it is determined that the information processing is completed (S107: Yes), Figure 10 The processing is completed.

[0101] <Update the display image based on indoor and outdoor target movement>

[0102] If the AR glasses do not move (S103: No), the position detection unit 131 determines whether the target moves indoors or outdoors, that is, whether to switch between indoor and outdoor movement (S108). Here, the AR glasses do not move means that the position of the free viewpoint does not move or there is 3DoF movement.

[0103] If the indoor movement and outdoor movement are switched (S108: Yes), then Figure 10 The process proceeds to the process of S104. In other words, when the AR glasses do not move, the display image is updated as described above.

[0104] If the indoor movement and the outdoor movement are not switched (S108: No), the position detection unit 131 determines whether the indoor movement is performed between floors (S109).

[0105] If the indoor movement is performed between floors (S109: Yes), then Figure 10 The process proceeds to the process of S104. In other words, when the indoor movement is performed between floors, the display image is updated as described above.

[0106] <Update the displayed image according to the target's movement>

[0107] If the indoor movement is not performed between floors (S109: No), the position information update management unit 32 calculates the target's movement amount based on the difference in position information (S110). Then, the position information update management unit 32 determines whether a certain movement amount of the target has occurred based on the calculated movement amount (S111).

[0108] If a certain amount of movement of the target has occurred (S111: YES), then Figure 10 The process proceeds to the process of S104. In other words, when the movement amount exceeding the threshold value of viewpoint switching is generated, the display image is updated as described above.

[0109] If a certain amount of movement of the target does not occur (S111: No), the 360-degree image rendering unit 33 superimposes a mark or icon on the created 360-degree image to generate a 360-degree image in which the position of the target is drawn (S112). Figure 10 The process proceeds to S106. In other words, when the movement amount of the target is within a certain range, the display image is updated without receiving the viewpoint image from the street image server 20.

[0110] [1-4. Use Example of Information Processing System According to Embodiment]

[0111] Here, we will refer to Figure 11 A usage example of the information processing system 1 according to the embodiment is described.

[0112] Figure 11 1 is a diagram illustrating a usage example of the information processing system 1 according to an embodiment of the present disclosure. Figure 11 In the example shown, the orientation of the displayed image corresponds to the orientation of the guardian wearing AR glasses (display terminal 40). For example, a left-facing display image represents an image displayed on display terminal 40 when the guardian turns left. For example, a downward-facing display image represents an image displayed on display terminal 40 when the guardian looks back. Assume that a guardian, who is stationary, is looking into the field of view of a child U who is moving with mobile terminal 10, and that child U is, for example, stuck in a dead end.

[0113] First, a display image IMG7 (360-degree image) showing the location and surroundings of child U is displayed on the display terminal 40. For example, in display image IMG7, the direction of child U is in an initial state. This helps the guardian understand the situation, for example, if child U is stuck in a dead end. The guardian then begins 6DoF movement using the installed display terminal 40. At this point, the display image updates as the free viewpoint position moves. Display image IMG8 is a display image showing the direction of the dead end, in which the guardian's free viewpoint moves to the opposite side of the dead end, and in this state, the guardian faces left to see the direction of the dead end. In this way, an icon UV indicating the position of child U can be displayed in a superimposed manner based on the position information of child U. This makes it easy for the guardian to understand the relative position relationship with child U during the virtual movement. In addition, display image IMG9, which is displayed when the guardian faces right, shows the opposite side of the dead end, allowing the guardian to search for a route to get child U out of the dead end. The guardian can guide the child U along a specific route based on the displayed image IMG9, such as "Go to the point where you can see a greenhouse on the left and turn right." The guardian can then further guide the child U along a specific route based on the displayed image IMG10, which is updated as the free viewpoint moves, such as "Turn left on the next road." In this way, the guardian can confirm the bus stop through the displayed image IMG11, which is updated as the free viewpoint moves, without the child U having to move.

[0114] Note that this embodiment illustrates, but is not limited to, a monitoring service in which a guardian monitors a child U. The target of the monitoring service according to the present disclosure may be, for example, an elderly person. Furthermore, the technology according to the present disclosure is not limited to monitoring services, but may be implemented as a route guidance service for delivery service drivers, tourists, and the like.

[0115] [1-5. Hardware Configuration]

[0116] Figure 121 is a block diagram illustrating an example of the hardware configuration of each device of the information processing system 1 according to an embodiment of the present disclosure. Information devices such as the respective devices (mobile terminal 10, information processing device 30, and display terminal 40) of the information processing system 1 according to the above-described embodiment are configured by, for example, having Figure 12 The configuration shown is implemented by a computer 1000 .

[0117] The computer 1000 includes a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. The respective component units of the computer 1000 are communicably connected via a bus 1050.

[0118] The CPU 1100 operates and controls each component unit based on a program stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 to execute processing corresponding to the various programs (information processing according to the present disclosure).

[0119] The ROM 1300 stores a boot program such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is booted, and programs depending on the hardware of the computer 1000 .

[0120] The HDD 1400 is a computer-readable recording medium that non-transitorily records programs executed by the CPU 1100 , data used by the programs, and the like.

[0121] The communication interface 1500 is an interface configured to connect the computer 1000 to the network N and an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from another device via the communication interface 1500 or transmits data generated by the CPU 1100 to another device.

[0122] The input / output interface 1600 is an interface configured to connect an input / output device 1650 to the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. In addition, the CPU 1100 sends data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. In addition, the input / output interface 1600 can be used as a medium interface configured to read a program recorded on a predetermined recording medium. The medium includes, for example, an optical recording medium such as a digital versatile disc (DVD) or a phase-change rewritable disc (PD), a magneto-optical recording medium such as a magneto-optical disc (MO), a tape medium, a magnetic recording medium, a semiconductor memory, etc.

[0123] For example, CPU 1100 of computer 1000 executes the program loaded into RAM 1200 to realize each function of each device. Note that CPU 1100 reads program data 1450 from HDD 1400 to execute program data 1450, but in another example, these programs may be acquired from another device via external network 1550.

[0124] (2. Effects of the Information Processing System According to the Present Disclosure)

[0125] The information processing device 30 includes an acquisition unit (communication unit 31, location information update management unit 32), a generation unit (360-degree image rendering unit 33, free viewpoint image rendering unit 34), and an output unit (communication unit 31). The acquisition unit acquires a viewpoint image according to the viewpoint of the first user from the street image server 20 based on the location information indicating the location of the mobile terminal 10 moving with the first user. The street image server 20 is configured to store multiple viewpoint images obtained by capturing images from multiple directions at each of multiple locations. The generation unit uses the acquired viewpoint image to generate a rendered image specified by a free viewpoint at a location different from the first user's location as a display image. The output unit outputs the generated display image.

[0126] This configuration enables the information processing device 30 to reduce the amount of data transmitted from the mobile terminal 10 to the display terminal 40, even when the view of the first user of the mobile terminal 10 is displayed on the display terminal 40 used by the second user. This allows the information processing device 30 to display the view of the first user of the mobile terminal 10, visible from a free viewpoint, and then display it on the display terminal 40 used by the second user at a different location as the first user moves. This reduces the amount of data transmitted by the second user, while also making it easier for the second user to understand the first user's situation and suppressing interaction delays. Furthermore, in some cases, capturing a 360-degree image requires complex operations by the first user, who is the target of surveillance. However, the information processing device 30 eliminates the need for the first user, who is the target of surveillance, to perform the operation of capturing a 360-degree image using the mobile terminal 10. Furthermore, the display images output by the information processing device 30 are generated based on viewpoint images acquired from the pre-prepared street image server 20. Therefore, according to the information processing device 30, the use of viewpoint images on which image processing, such as blurring, has been pre-processed for unintentionally captured objects within the viewpoint reduces the safety risks posed by unintentionally captured objects.

[0127] In the information processing apparatus 30 , when the movement amount of the mobile terminal 10 exceeds a predetermined threshold, the acquisition unit acquires a viewpoint image.

[0128] Therefore, the information processing device 30 can reduce the frequency of updating viewpoint images.

[0129] In the information processing device 30, when the position change of the mobile terminal 10 is equal to or less than a predetermined threshold, the acquisition unit does not acquire the viewpoint image. In addition, in the information processing device 30, the generation unit superimposes an icon or mark indicating the position of the first user on the acquired viewpoint image and generates a display image.

[0130] Therefore, even when the viewpoint image is not updated, the information processing apparatus 30 is allowed to express the position of the first user in the display image.

[0131] In the information processing device 30 , when the free viewpoint is moved by the second user, the acquisition unit acquires the viewpoint image.

[0132] Therefore, when the free viewpoint moves, the information processing device 30 is allowed to update the viewpoint image.

[0133] In the information processing apparatus 30 , when the position of the mobile terminal 10 changes between the indoor position and the outdoor position, the acquisition unit acquires viewpoint images.

[0134] Therefore, when the position of the mobile terminal 10 switches between the indoor position and the outdoor position, the information processing device 30 is allowed to update the viewpoint image.

[0135] In the information processing device 30, when there is no viewpoint image of the indoor location where the mobile terminal 10 is located, the generation unit generates a display image by superimposing an icon or mark indicating the location of the first user on the viewpoint image acquired before the location of the mobile terminal is switched from the outdoor location to the indoor location.

[0136] Therefore, even when the first user enters a building for which no viewpoint image is prepared in the street image server 20, the information processing device 30 is allowed to express the position of the first user in the display image.

[0137] In the information processing apparatus 30 , when the mobile terminal 10 is located indoors and moves between floors, the acquisition unit acquires viewpoint images.

[0138] Therefore, even when the first user is indoors, the information processing device 30 is allowed to generate a display image using appropriate viewpoint images.

[0139] In the information processing device 30 , when the travel of the first user by the mobile object (vehicle) is detected, the generation unit generates a display image by superimposing an icon or a mark indicating the position of the first user on the map.

[0140] Therefore, the information processing device 30 is allowed to express the position of the first user in the display image even when no viewpoint image is prepared for this purpose in the street image server 20. In addition, even when the display image is configured to be updated according to the amount of movement, the information processing device 30 is allowed to suppress unnecessary updates.

[0141] Information processing system 1 includes a mobile terminal 10, a street image server 20, an information processing device 30, and a display terminal 40. Mobile terminal 10 moves with a first user and outputs location information. Street image server 20 stores viewpoint images. Display terminal 40 moves with a second user and displays a display image.

[0142] Therefore, when the field of view of a first user using mobile terminal 10 is displayed on display terminal 40 used by a second user at a different location, information processing system 1 is allowed to suppress interaction delays. Furthermore, this eliminates the need for the first user, who is the target of surveillance, to capture a 360-degree image using mobile terminal 10. Furthermore, safety risks caused by accidentally capturing an object can be reduced.

[0143] In the information processing system 1 , the display terminal 40 is a head-mounted display configured to be mounted on the head of the second user.

[0144] Therefore, the information processing system 1 is allowed to specify a free viewpoint without any special operation by the second user wearing the display terminal 40 .

[0145] The information processing method includes: acquiring, based on position information indicating the position of a mobile terminal 10 that moves with a first user, a viewpoint image according to a viewpoint of the first user from a street image server 20, the street image server 20 being configured to store a plurality of viewpoint images obtained by capturing images from a plurality of orientations at each of a plurality of positions; generating, as a display image, a rendered image rendered from a free viewpoint specified by a second user located at a position different from the position of the first user using the acquired viewpoint image; and outputting the generated display image.

[0146] This information processing method allows for suppressing interaction delays when displaying the field of view of a first user using mobile terminal 10 on display terminal 40 used by a second user in a different location. Furthermore, this eliminates the need for the first user, who is the target of surveillance, to capture a 360-degree image using mobile terminal 10. Furthermore, safety risks resulting from accidentally capturing an object can be reduced.

[0147] The information processing method further includes acquiring a viewpoint image when the movement amount of the mobile terminal 10 exceeds a predetermined threshold.

[0148] Therefore, the information processing method is allowed to reduce the frequency of updating viewpoint images.

[0149] The information processing method also includes generating a display image by superimposing an icon or mark indicating the position of the first user on the acquired viewpoint image without acquiring the viewpoint image when the position change of the mobile terminal 10 is equal to or lower than a predetermined threshold.

[0150] Therefore, even when the viewpoint image is not updated, the information processing method is allowed to express the position of the first user in the display image.

[0151] The information processing method further includes acquiring a viewpoint image when the free viewpoint is moved by the second user.

[0152] Therefore, when the free viewpoint moves, the information processing method is allowed to update the viewpoint image.

[0153] The information processing method also includes acquiring viewpoint images when the position of the mobile terminal 10 changes between the indoor position and the outdoor position.

[0154] Therefore, when the position of the mobile terminal 10 switches between an indoor position and an outdoor position, the information processing method is allowed to update the viewpoint image.

[0155] The information processing method also includes: when there is no indoor viewpoint image of the location of the mobile terminal 10, generating a display image by superimposing an icon or mark indicating the location of the first user on a viewpoint image acquired before the location of the mobile terminal is switched from the outdoor location to the indoor location.

[0156] Therefore, even when the first user enters a building for which no viewpoint image is prepared in the street image server 20, this information processing method is allowed to express the position of the first user in the display image.

[0157] The program causes the computer to execute, based on position information indicating the position of the mobile terminal 10 moving with the first user, acquiring a viewpoint image according to the viewpoint of the first user from the street image server 20, the street image server 20 being configured to store a plurality of viewpoint images obtained by capturing images from a plurality of orientations at each of a plurality of positions; generating, as a display image, a rendered image rendered from a free viewpoint specified by a second user located at a position different from the position of the first user by using the acquired viewpoint image; and outputting the generated display image.

[0158] This allows the program to suppress interaction delays when displaying the field of view of a first user using mobile terminal 10 on display terminal 40 used by a second user in a different location. Furthermore, this eliminates the need for the first user, who is the target of surveillance, to capture a 360-degree image using mobile terminal 10. Furthermore, safety risks caused by accidentally capturing an object can be reduced.

[0159] The program also causes the computer to execute acquisition of viewpoint images when the movement amount of the mobile terminal 10 exceeds a predetermined threshold value.

[0160] Therefore, the program is allowed to reduce the frequency of updating viewpoint images.

[0161] The program also causes the computer to generate a display image by superimposing an icon or mark indicating the position of the first user on the acquired viewpoint image without acquiring the viewpoint image when the position change of the mobile terminal 10 is equal to or lower than a predetermined threshold.

[0162] Therefore, even when the viewpoint image is not updated, the program is allowed to express the position of the first user in the display image.

[0163] The program also causes the computer to acquire viewpoint images when the free viewpoint is moved by the second user.

[0164] Therefore, the program is allowed to update the viewpoint image as the free viewpoint moves.

[0165] The program also causes the computer to acquire viewpoint images when the position of the mobile terminal 10 changes between an indoor position and an outdoor position.

[0166] Therefore, when the location of the mobile terminal 10 switches between an indoor location and an outdoor location, the program is allowed to update the viewpoint image.

[0167] The program also causes the computer to generate a display image by superimposing an icon or mark indicating the location of the first user on a viewpoint image acquired before the location of the mobile terminal is switched from an outdoor location to an indoor location when there is no indoor viewpoint image of the location where the mobile terminal 10 is located.

[0168] Therefore, even when the first user enters a building for which no viewpoint image is prepared in the street image server 20, the program is allowed to express the position of the first user in the display image.

[0169] Note that the effects described herein are merely examples, and the present invention is not limited to these effects but may also provide other effects.

[0170] Note that the present technology can also have the following configurations. (1)

[0172] An information processing device, comprising:

[0173] an acquisition unit configured to acquire, based on position information indicating a position of a mobile terminal moving with the first user, a viewpoint image according to a viewpoint of the first user from an image server configured to store a plurality of viewpoint images obtained by capturing images from a plurality of directions at each of a plurality of positions;

[0174] a generating unit configured to generate, as a display image, a rendered image rendered from a free viewpoint designated by a second user located at a position different from that of the first user using the acquired viewpoint image; and

[0175] The output unit is configured to output the generated display image. (2)

[0177] The information processing apparatus according to (1), wherein the acquisition unit acquires the viewpoint image when the movement amount of the mobile terminal exceeds a predetermined threshold value. (3)

[0179] The information processing device according to (1) or (2), wherein

[0180] When the position change of the mobile terminal is equal to or lower than a predetermined threshold, the acquiring unit does not acquire the viewpoint image, and

[0181] The generation unit superimposes an icon or a mark indicating the position of the first user on the acquired viewpoint image and generates a display image. (4)

[0183] The information processing apparatus according to any one of (1) to (3), wherein the acquisition unit acquires the viewpoint image when the free viewpoint is moved by the second user. (5)

[0185] The information processing apparatus according to any one of (1) to (4), wherein the acquisition unit acquires the viewpoint image when the position of the mobile terminal changes between an indoor position and an outdoor position. (6)

[0187] An information processing device according to (5), wherein when there is no viewpoint image of the indoor location where the mobile terminal is located, the generation unit generates a display image by superimposing an icon or mark indicating the location of the first user on the viewpoint image acquired before the location of the mobile terminal is switched from the outdoor location to the indoor location. (7)

[0189] The information processing apparatus according to any one of (1) to (6), wherein the acquisition unit acquires the viewpoint image when the mobile terminal is located indoors and moves between floors. (8)

[0191] The information processing device according to any one of (1) to (7), wherein when the travel of the first user by the mobile object is detected, the generation unit generates the display image by superimposing an icon or a mark indicating the position of the first user on the map. (9)

[0193] An information processing system comprising:

[0194] a mobile terminal that moves with the first user and is configured to output location information,

[0195] an image server configured to store viewpoint images,

[0196] a display terminal that moves with the second user and is configured to display a display image, and

[0197] The information processing apparatus according to any one of (1) to (8). (10)

[0199] The information processing system according to (9), wherein the display terminal is a head-mounted display mounted on a head of the second user. (11)

[0201] An information processing method, comprising:

[0202] acquiring, based on position information indicating a position of a mobile terminal moving with the first user, a viewpoint image according to a viewpoint of the first user from an image server configured to store a plurality of viewpoint images obtained by capturing images from a plurality of directions at each of a plurality of positions;

[0203] generating, by using the acquired viewpoint image, a rendered image rendered from a free viewpoint designated by a second user located at a position different from that of the first user as a display image; and

[0204] Output the generated display image. (12)

[0206] The information processing method according to (11) further includes acquiring a viewpoint image when the movement amount of the mobile terminal exceeds a predetermined threshold. (13)

[0208] The information processing method according to (11) or (12) further includes, when the position change of the mobile terminal is equal to or lower than a predetermined threshold, generating a display image by superimposing an icon or mark indicating the position of the first user on the acquired viewpoint image without acquiring the viewpoint image. (14)

[0210] The information processing method according to any one of (11) to (13), further comprising acquiring a viewpoint image when the free viewpoint is moved by the second user. (15)

[0212] The information processing method according to any one of (11) to (14), further comprising acquiring a viewpoint image when the position of the mobile terminal changes between an indoor position and an outdoor position. (16)

[0214] The information processing method according to (15) further includes generating a display image by superimposing an icon or mark indicating the location of the first user on a viewpoint image acquired before the location of the mobile terminal is switched from the outdoor location to the indoor location when there is no viewpoint image of the indoor location where the mobile terminal is located. (17)

[0216] A program that causes a computer to:

[0217] acquiring, based on position information indicating a position of a mobile terminal moving with the first user, a viewpoint image according to a viewpoint of the first user from an image server configured to store a plurality of viewpoint images obtained by capturing images from a plurality of directions at each of a plurality of positions;

[0218] generating, using the acquired viewpoint image, a rendered image rendered from a free viewpoint designated by a second user located at a position different from that of the first user as a display image; and

[0219] Output the generated display image. (18)

[0221] The program according to (17), further causing the computer to execute acquisition of viewpoint images when the movement amount of the mobile terminal exceeds a predetermined threshold. (19)

[0223] The program according to (17) or (18) further causes the computer to generate a display image by superimposing an icon or mark indicating the position of the first user on the acquired viewpoint image when the position change of the mobile terminal is equal to or lower than a predetermined threshold, without acquiring the viewpoint image. (20)

[0225] The program according to any one of (17) to (19), further causing the computer to execute acquiring the viewpoint image when the free viewpoint is moved by the second user. (twenty one)

[0227] The program according to any one of (17) to (20), further causing the computer to execute acquiring viewpoint images when the position of the mobile terminal changes between the indoor position and the outdoor position. (twenty two)

[0229] The program described in (21) also causes the computer to generate a display image by superimposing an icon or mark indicating the location of the first user on a viewpoint image acquired before the location of the mobile terminal is switched from an outdoor location to an indoor location when there is no viewpoint image of the indoor location where the mobile terminal is located.

[0230] Reference designator list

[0231] 1 Information Processing System

[0232] 10 Mobile Terminals

[0233] 11 Communication Unit

[0234] 111 cellular module

[0235] 112Wi-Fi module

[0236] 113 BLE module

[0237] 12 sensor units

[0238] 121 GPS sensor

[0239] 122 geomagnetic sensor

[0240] 123IMU

[0241] 13 Detection Unit

[0242] 131 Position detection unit

[0243] 132 Outdoor location detection device

[0244] 133 Indoor location detection unit

[0245] 134 direction detection unit

[0246] 135 Motion Detection Unit

[0247] 14 input / output units

[0248] 20 Street Image Server

[0249] 30 Information processing device 31 Communication unit (acquisition unit, output unit) 32 Position information update management unit (acquisition unit) 33 360-degree image rendering unit (generation unit)

[0250] 34 Free viewpoint image rendering unit (generation unit) 40 Display terminal

[0251] 41 Communication Unit 42 IMU

[0252] 43 cameras

[0253] 44 input / output units

[0254] 441 light guide lens

[0255] N Network

Claims

1. An information processing device, comprising: an acquisition unit configured to acquire, from an image server configured to store a plurality of viewpoint images obtained by capturing images from a plurality of directions at each of a plurality of positions, viewpoint images based on position information indicating a position of a mobile terminal moving with a first user, the position information being transmitted from the mobile terminal; a generating unit configured to generate, as a display image, a rendered image rendered from a free viewpoint designated by a second user located at a position different from that of the first user, using the acquired viewpoint image; as well as an output unit configured to output the generated display image, wherein the generating unit sets the initial state of the display image based on the direction of the mobile terminal, information indicating the direction of the mobile terminal is sent from the mobile terminal, and The generating unit updates the display image in response to a change in position of the mobile terminal and designation of a free viewpoint according to movement or rotation of the second user. 2 . The information processing apparatus according to claim 1 , wherein the acquisition unit acquires the viewpoint image when the movement amount of the mobile terminal exceeds a predetermined threshold value.

3. The information processing device according to claim 1, wherein When the position change of the mobile terminal is equal to or lower than a predetermined threshold, the acquiring unit does not acquire the viewpoint image, and The generation unit superimposes an icon or a mark indicating the position of the first user on the acquired viewpoint image and generates a display image. 4 . The information processing apparatus according to claim 1 , wherein the acquisition unit acquires the viewpoint image when the free viewpoint is moved by the second user. 5 . The information processing apparatus according to claim 1 , wherein the acquisition unit acquires the viewpoint image when the position of the mobile terminal changes between an indoor position and an outdoor position.

6. An information processing device according to claim 5, wherein when there is no viewpoint image of the indoor location where the mobile terminal is located, the generation unit generates a display image by superimposing an icon or mark indicating the location of the first user on the viewpoint image acquired before the position of the mobile terminal is switched from the outdoor location to the indoor location. 7 . The information processing apparatus according to claim 1 , wherein the acquisition unit acquires the viewpoint image when the mobile terminal is located indoors and moves between floors. 8 . The information processing apparatus according to claim 1 , wherein when the travel of the first user by the mobile object is detected, the generation unit generates the display image by superimposing an icon or a mark indicating the position of the first user on the map.

9. An information processing method comprising: acquiring, from an image server configured to store a plurality of viewpoint images obtained by capturing images from a plurality of orientations at each of a plurality of positions, viewpoint images based on position information indicating a position of a mobile terminal moving with a first user, the position information being transmitted from the mobile terminal; generating, by using the acquired viewpoint image, a rendered image rendered from a free viewpoint designated by a second user located at a position different from that of the first user as a display image; as well as Output the generated display image, wherein the initial state of the displayed image is set based on the direction of the mobile terminal, information indicating the direction of the mobile terminal is sent from the mobile terminal, and The display image is updated in response to a change in position of the mobile terminal and designation of a free viewpoint according to movement or rotation of the second user. 10 . The information processing method according to claim 9 , further comprising acquiring the viewpoint image when the movement amount of the mobile terminal exceeds a predetermined threshold.

11. The information processing method according to claim 9 further includes not acquiring a viewpoint image when the position change of the mobile terminal is equal to or lower than a predetermined threshold, but generating a display image by superimposing an icon or mark indicating the position of the first user on the acquired viewpoint image. 12 . The information processing method according to claim 9 , further comprising acquiring a viewpoint image when the free viewpoint is moved by the second user. 13 . The information processing method according to claim 9 , further comprising acquiring the viewpoint image when the position of the mobile terminal changes between an indoor position and an outdoor position.

14. The information processing method according to claim 13 also includes generating a display image by superimposing an icon or mark indicating the location of the first user on a viewpoint image obtained before the position of the mobile terminal is switched from an outdoor position to an indoor position when there is no viewpoint image of the indoor location where the mobile terminal is located.

15. A computer program product causing a computer to execute: acquiring, from an image server configured to store a plurality of viewpoint images obtained by capturing images from a plurality of orientations at each of a plurality of positions, viewpoint images based on position information indicating a position of a mobile terminal moving with a first user, the position information being transmitted from the mobile terminal; generating, using the acquired viewpoint image, a rendered image rendered from a free viewpoint designated by a second user located at a position different from that of the first user as a display image; as well as Output the generated display image, wherein the initial state of the displayed image is set based on the direction of the mobile terminal, information indicating the direction of the mobile terminal is sent from the mobile terminal, and The display image is updated in response to a change in position of the mobile terminal and designation of a free viewpoint according to movement or rotation of the second user. 16 . The computer program product according to claim 15 , further causing a computer to execute: acquiring a viewpoint image when the movement amount of the mobile terminal exceeds a predetermined threshold.

17. The computer program product according to claim 15 further causes the computer to execute: when the position change of the mobile terminal is equal to or lower than a predetermined threshold, not acquiring a viewpoint image, but generating a display image by superimposing an icon or mark indicating the position of the first user on the acquired viewpoint image. 18 . The computer program product according to claim 15 , further causing a computer to execute: acquiring a viewpoint image when the free viewpoint is moved by the second user.

19. The computer program product according to claim 15, further causing a computer to execute: acquiring a viewpoint image when the position of the mobile terminal changes between an indoor position and an outdoor position.

20. The computer program product according to claim 19 further causes the computer to execute: when there is no viewpoint image of the indoor location where the mobile terminal is located, generating a display image by superimposing an icon or mark indicating the location of the first user on the viewpoint image acquired before the location of the mobile terminal is switched from the outdoor location to the indoor location.

Images