Information processing apparatus, information processing method, and program

Abstract

An information processing apparatus includes a rendering unit and a reproduction unit. A rendering unit generates rendering data of a subsequent scene expected to appear in a virtual space of a receiver at a subsequent time after a current time on the basis of sensor information of the current time acquired from a sender. The reproduction unit reproduces the rendering data in the virtual space of the receiver at the subsequent time.

Description

Technical Field

[0001] The present technical content relates to an information processing device, an information processing method and a program. Background Art

[0002] There is an increasing need for remote communication. Floor mounted tactile presentation devices and wearable tactile presentation devices like gloves or clothing can be used to communicate with remote partners in the XR space.

[0003] Prior art literature

[0004] Patent Literature

[0005] Japanese Patent Application Laid-Open No. 2015-111417A Summary of the invention

[0006] Technical Problems to be Solved by the Invention

[0007] In the case of communicating with a remote party, there is a possibility that a deviation from the reproduction timing occurs due to communication delay, etc. For example, although image and sound playback is allowed even with a slight delay, in a case where it is very important that the tactile presentation timing of both parties is the same, such as when directly touching the fingertips of the other party on VR, there is a possibility that discomfort may occur due to communication delay.

[0008] Therefore, the present disclosure proposes an information processing device, an information processing method, and a program capable of suppressing a reproduction timing deviation on the receiving side.

[0009] Means of solving the problem

[0010] According to the present disclosure, there is provided an information processing device, comprising: a rendering unit, which generates rendering data of a subsequent scene expected to appear in a virtual space of a receiver at a subsequent moment after the current moment based on sensor information at the current moment acquired from a sender; and a reproduction unit, which reproduces the rendering data in the virtual space of the receiver at the subsequent moment. According to the present disclosure, there is provided an information processing method in which a computer performs information processing of an information processing device, and a program for causing a computer to perform information processing of an information processing device. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a diagram of a telecommunication system according to a first embodiment.

[0012] Figure 2 is a view showing an example of a haptic device.

[0013] Figure 3is a view showing an example of a haptic device.

[0014] Figure 4 is a view showing an example of interaction using ripples.

[0015] Figure 5 is a diagram showing an example of processing operations on the transmission side and the reception side.

[0016] Figure 6 is a diagram showing an example of processing operations on the transmission side and the reception side.

[0017] Figure 7 It is a view comparing the conventional method and the method of the present disclosure.

[0018] Figure 8 It is a view comparing the conventional method and the method of the present disclosure.

[0019] Fig. 9 is a view showing an example of a processing flow on the receiving side.

[0020] Fig.10 It is an explanatory diagram of the first modification example.

[0021] Fig.11 It is an explanatory diagram of the second modification example.

[0022] Fig.12 It is an explanatory diagram of the third modification example.

[0023] Fig.13 It is an explanatory diagram of the fourth modification example.

[0024] Fig.14 is a view showing a telecommunication system according to a second embodiment.

[0025] Fig.15 This is a diagram for explaining the process of generating and reproducing rendering data.

[0026] Fig.16 This is a diagram for explaining the process of generating and reproducing rendering data.

[0027] Fig.17 It is an explanatory diagram of the fifth modification example.

[0028] Fig.18 is a diagram showing an example of the hardware configuration of an information processing device. DETAILED DESCRIPTION

[0029] Hereinafter, embodiments of the present disclosure will be described in detail based on the accompanying drawings. In the following embodiments, the same parts are represented by the same reference numerals, and redundant descriptions are omitted.

[0030] Note that the description will be given in the following order.

[0031] [1. First embodiment]

[0032] [1-1. Configuration of remote communication system]

[0033] [1-2. Generation and reproduction of rendering data]

[0034] [1-3. Information processing method]

[0035] [1-4. Effect]

[0036] [1-5. First Modification Example]

[0037] [1-6. Second Modification Example]

[0038] [1-7. Third Modification]

[0039] [1-8. Fourth Modification]

[0040] [2. Second Embodiment]

[0041] [2-1. Generation and reproduction of rendering data]

[0042] [2-2. Fifth Modification]

[0043] [3. Hardware configuration example]

[0044] [1. First embodiment]

[0045] [1-1. Configuration of remote communication system]

[0046] Figure 1 is a view showing a telecommunication system CS according to the first embodiment.

[0047] The remote communication system CS includes a server SV and a plurality of reproduction systems RS. The plurality of reproduction systems RS are connected via the server SV. The reproduction system RS presents a virtual space to a user US using reproduction data related to images, sounds, and tactile sensations. The user US communicates with another user US via the virtual space.

[0048] The reproduction system RS includes a camera CM, a projector PJC, a screen SCR, a projector PJR, a haptic device HD, a sensor unit SE, and an information processing apparatus PD.

[0049] The information processing device PD generates rendering data of various scenes reproduced by the reproduction system RS. The rendering data includes image, sound and tactile data. The sensor unit SE acquires sensor information required for rendering. For example, the sensor unit SE includes a camera CM, a tactile device HD and a distance sensor DS (see Figure 4). The sensor unit SE outputs information detected by each device as sensor information.

[0050] The camera DM captures an image of the space in the room where the user US is located. The camera DM outputs the captured room image as a monitor image CIM. The monitor image CIM is transmitted to another reproduction system RS via the server SV. The projector PJC displays the monitor image CIM of another reproduction system RS received via the server SV on a screen SCR.

[0051] The projector PJR reproduces rendering data related to images and sounds. Figure 1 In the example of , rendering data related to an image (rendering image RIM) is reproduced on the ground via a projector PJR. In each reproduction system RS, a rendering image RIM about the same content is reproduced. The monitor image CIM and the rendering image RIM are presented as one image extending from the ground to the screen SCR. The user US can feel as if the virtual space of another reproduction system RS is extended on the other side of the screen SCR.

[0052] exist Figure 1 In the example of FIG. 1 , a user US of one reproduction system RS communicates with a user US of another reproduction system RS via a network (such as the Internet). Each reproduction system RS generates rendering data using sensor information acquired by a sensor unit SE and content information (content storage unit CTS) acquired from a server SV. The sensor information is also transmitted to another reproduction system RS via the server SV and used for rendering processing of the other reproduction system RS.

[0053] In the following, a reproduction system RS that transmits sensor information to another reproduction system RS via a server SV is described as a sender TR. A virtual space presented by the sender TR is described as a virtual space A. A reproduction system RS that receives sensor information of another reproduction system RS from the server SV is described as a receiver RE. A virtual space presented by the receiver RE is described as a virtual space B. Each reproduction system RS can serve as a sender TR and a receiver RE. A common virtual object VOB is presented in the virtual space A and the virtual space B. A user US on the sender RT side and a user US on the receiver RE side can interact via the virtual object VOB.

[0054] The sender TR transmits the monitor image CIM captured by the camera CM and the sensor information at the current moment acquired by the sensor unit SE to the receiver RE via the server SV. The sensor information includes various types of information used to render the reproduced scene. For example, the sensor information includes information related to the interaction between the user US and the virtual object VOB (see Figure 4 and Fig.14). The time information may be obtained from a clock in the computer of the sender TR or from a clock in the server SV.

[0055] Figure 2 and Figure 3 is a view showing an example of a haptic device HD.

[0056] The tactile device HD has a tactile presentation function and a pressure-sensitive detection function. Figure 2 In the example of , the haptic device HD is configured as a plurality of tile units TU placed on the ground. The tile unit TU includes a pressure-sensitive sensor PS and a plurality of haptic units HU. The surface of the tile unit TU is used as a screen for displaying an image (rendered image RIM) of the projector PJR.

[0057] The haptic device HD detects the stepping position and strength of the user US as sensor information by using a plurality of two-dimensionally arranged pressure-sensitive sensors PS. The detected sensor information is used to generate image, sound, and tactile rendering data. The haptic device HD reproduces rendering data about tactile sensation using the tactile unit HU near the stepping position.

[0058] Note that when the LED display is built into the tile unit TU, the rendered image RIM can be displayed directly on the LED display. In this case, the projector PJR can be omitted. When the haptic unit HU is configured as a broadband actuator, the haptic unit HU can also be used as a sound element.

[0059] exist Figure 3 In the example of , the haptic device HD is configured as a wearable device WD. The pressure-sensitive sensor PS and the haptic unit HU are built into clothes. The user US can input operation information by touching the pressure-sensitive sensor PS built into the clothes. The haptic device HD uses the haptic unit HU built into the clothes to reproduce rendering data about touch.

[0060] Back to Figure 1 , the information processing device PD includes a monitor image transmitting unit CT, a monitor image receiving unit CR, a sensor information acquiring unit SI, a delay time acquiring unit DT, a rendering unit RD and a reproduction unit RP.

[0061] The monitor image sending unit CT acquires the monitor image CIM from the camera CA and transmits the monitor image CIM to the server SV. The server SV stores the monitor image CIM received from the monitor image sending unit CT in the monitor image storage unit CIS. The monitor image receiving unit CR receives the monitor image CIM of another reproduction system RS from the monitor image storage unit CIS and provides the monitor image CIM to the projector PJC.

[0062] The sensor information acquisition unit SI acquires sensor information from the sensor unit SE and provides the sensor information to the rendering unit RD. The sensor information acquisition unit SI transmits the sensor information to the server SV. The server SV stores the sensor information received from the sensor information acquisition unit SI in the sensor information storage unit SIS.

[0063] The delay time acquisition unit DT acquires the communication delay time between the transmitter TR and the receiver RE. The communication delay time refers to the time required for communication. For example, the communication delay time is 500 ms. The communication delay time can be measured using a known measuring device. The communication delay time may be different between the uplink direction (the direction from the transmitter TR toward the receiver RE) and the downlink direction (the direction from the receiver RE toward the transmitter TR). However, in this embodiment, for simplicity, it is assumed that the communication delay time in the uplink direction is equal to the communication delay time in the downlink direction.

[0064] The rendering unit RD generates rendering data for reproducing the scene using the sensor information. The sensor information includes sensor information acquired from the sensor information acquisition unit SI and sensor information acquired from the sensor information storage unit SIS. Both pieces of sensor information are input to the rendering unit RD of the receiving party RE, but each piece of sensor information is rendered using a different rendering method.

[0065] For example, the rendering unit RD of the receiving side RE uses the sensor information acquired from the sensor information acquisition unit SI of the receiving side RE (hereinafter, referred to as the receiving side sensor information) to generate rendering data of a scene caused by factors on the receiving side RE side (such as the behavior of the user US on the receiving side RE side). The rendering unit RD of the receiving side RE uses the sensor information transmitted from the sensor information storage unit SIS (hereinafter, referred to as the sending side sensor information) to generate rendering data of a scene caused by factors on the sending side TR side (such as the behavior of the user US on the sending side TR side).

[0066] In the case of rendering using the sensor information on the sending side, the rendering unit RD of the receiving side RE generates rendering data for a scene slightly ahead (in the future) in consideration of the communication delay. For example, the rendering unit RD generates rendering data for a subsequent scene on the receiving side RE based on the sensor information at the current moment obtained from the sending side TR. The subsequent scene refers to a future scene that is expected to appear in the virtual space of the receiving side RE at a subsequent moment after the current moment. For example, the subsequent scene is set to a scene that appears after a moment that is later than the current moment by the communication extension time.

[0067] When rendering using the receiving side sensor information, the rendering unit RD of the receiving side RE generates rendering data of the current scene on the receiving side RE side based on the sensor information at the current moment directly acquired from the sensor information acquisition unit SI. The current scene refers to the scene expected to appear in the virtual space B of the receiving side RE at the current moment. The time information can be obtained from the clock in the computer of the receiving side RE or from the clock in the server SV.

[0068] The reproduction unit RP transmits the rendering data to the reproduction system RS. The reproduction unit RP reproduces the rendering data via the projector PJR, the haptic device HD, etc. of the reproduction system RS.

[0069] As described above, the rendering unit RD changes the predetermined reproduction time of the scene to be rendered between the case where the rendering is performed using the sending side sensor information and the case where the rendering is performed using the receiving side sensor information. Therefore, the reproduction unit RP reproduces the rendering data according to the predetermined reproduction time of the scene. For example, for the rendering data generated using the sending side sensor information, the reproduction unit RP reproduces the rendering data in the virtual space B of the receiving party RE at a subsequent time. For the rendering data generated using the receiving side sensor information, the reproduction unit RP reproduces the rendering data in the virtual space B of the receiving party RE at the current time.

[0070] [1-2. Generation and reproduction of rendering data]

[0071] Hereinafter, a specific example of generation and reproduction of rendering data will be described. Figure 4 is a diagram showing an example of interaction using ripple RL.

[0072] The user US on the sender TR can recognize the virtual space B of the receiver RE through the monitor image CIM on the screen SCR. The image of the virtual space B on the screen SCR (monitor image CIM) is an image when the image delay time is advanced from the current moment. The rendering image RIM reproduced in the virtual space A of the sender TR is generated based on the sensor information at the current moment (sensor information directly input from the sensor information acquisition unit SI of the sender TR to the rendering unit RD of the sender TR). The reproduction unit RP of the sender TR reproduces the wide-area scene WS extending from the sender TR side to the receiver RE side in the virtual space A of the sender TR by using the rendering image RIM and the monitor image CIM.

[0073] The boundary between the rendered image RIM and the monitor image CIM is the boundary BD between the virtual space A and the virtual space B (see Figure 6 ).exist Figure 4 In the example of , the plane including the screen SCR is the boundary BD between the virtual space A and the virtual space B.

[0074] exist Figure 4 In the example of FIG. 1 , the image of the ripple RL is presented as a rendered image RIM. The wide area scene WS includes a scene where the ripple RL extends from the sender TR side to the receiver RE side. The rendering unit RD of the receiver RE generates rendering data of a subsequent scene that is expected to appear after the ripple RL reaches the boundary BD.

[0075] When the user US touches the ripple RL, wave vibration is applied from the haptic device HD to the position on the ground where the user US is located. As a result, the tactile sensation of the wave is reproduced on the sole of the foot of the user US. By stepping on the ground, the user US can generate the ripple RL having an amplitude and a speed according to the stepping strength. Although the shape of the ripple RL is circular, the ripple RL having a long axis in the stepping direction can be generated by strongly stepping on an inclined direction (refer to Fig.13 ). In this case, the stepping direction is also detected as sensor information by the pressure sensitive sensor PS.

[0076] The position of the user US is detected using a pressure-sensitive sensor PS installed on the ground. Figure 4 In the example of FIG. 4 , a distance sensor DS (such as a time-of-flight (ToF) camera) may be used to detect the location of the user US.

[0077] Figure 5 and Figure 6 is a diagram showing an example of processing operations of the transmission side TR and the reception side RE.

[0078] The transmitter TR detects the operation of the user US stepping on the floor based on the detection information of the haptic device HD. Before the ripple RL caused by the stepping reaches the boundary BD between the virtual space A and the virtual space B, the transmitter TR transmits sensor information about the stepping position and strength to the receiver RE via the server SV.

[0079] The receiver starts rendering based on the sensor information (sender side sensor information) received from the transmitter TR. The scene to be rendered is a future scene after the moment when the ripple RL reaches the boundary BD. The receiver RE completes the rendering before the ripple RL reaches the boundary BD. Through the rendering process, time-series image, sound, and tactile rendering data indicating the state of the ripple RL propagating through the virtual space B on the receiver RE side are generated. The receiver RE reproduces the rendering data according to the timing of the ripple RL reaching the boundary BD.

[0080] The receiving side RE anticipates the future scene (subsequent scene) after the ripple RL reaches the boundary BD based on the transmitting side sensor information, and prepares the rendering data of the subsequent scene in advance. The receiving side RE reproduces the rendering data prepared in advance when the ripple RL reaches the boundary BD. As a result, appropriate reproduction processing is performed without being affected by communication delay.

[0081] Figure 7 and Figure 8 It is a view comparing the conventional method and the method of the present disclosure. Figure 7 is a view showing a conventional method as a comparative example. Figure 8 is a view illustrating a method of the present disclosure.

[0082] In the conventional method, rendering of the ripple RL in the virtual space B starts after the ripple RL reaches the boundary BD. Since a communication delay occurs between the sender TR and the receiver RE, the ripple RL presented in the virtual space B is a ripple (past ripple RL) based on past sensor information (sending side sensor information) according to the communication delay. The expansion of the ripple RL observed from the virtual space B side is smaller than the expansion of the ripple RL observed from the virtual space A side. The difference in the size of the ripple RL is recognized as a deviation in the reproduction timing.

[0083] In the method of the present disclosure, when the ripple RL enters the virtual space B, the ripple RL rendered in advance in the virtual space B is reproduced. Therefore, the extension of the ripple RL viewed from the virtual space B side is equal to the extension of the ripple RL viewed from the virtual space A side, and no deviation in the reproduction timing occurs (see Figure 8 upper side).

[0084] Even in the case of interaction between the ripple RL and the user US, similar processing can be performed. Figure 8 In the example on the lower side, the ripple RL collides with the user US to generate a reflected wave. The receiver RE anticipates a future scene (subsequent scene) after the reflected wave reaches the boundary BD based on the information of the reflection position, and prepares rendering data for the subsequent scene in advance. The receiver RE reproduces the rendering data prepared in advance when the reflected wave reaches the boundary BD. As a result, appropriate reproduction processing is performed without being affected by communication delays.

[0085] [1-3. Information processing method]

[0086] Fig. 9 is a view showing an example of a processing flow of a receiving-side RE.

[0087] The delay time acquisition unit DT acquires the communication delay time (step S1), and determines whether the communication delay time is greater than the allowable value (step S2). The allowable value refers to the communication delay time. Figure 7 In the conventional method shown above, the small communication delay time at which the deviation of the reproduction timing hardly causes any problem can be set arbitrarily by the system developer.

[0088] In the case where the communication delay time is greater than the allowable value (step S2: Yes), the rendering unit RD generates rendering data of a subsequent scene on the receiving side RE side based on the sensor information at the current moment (sending side sensor information) acquired from the sending side TR (step S3). The subsequent scene is a future scene expected to appear in the virtual space B of the receiving side RE at a subsequent moment after the current moment. The reproduction unit RP reproduces the generated rendering data of the subsequent scene at the subsequent moment (step S4).

[0089] In the case where the communication delay time is equal to or less than the allowable value (step S2: No), the rendering unit RD generates rendering data of the current scene on the receiving side RE side based on the sensor information at the current moment (sending side sensor information) acquired from the sending side TR (step S5). The current scene is a scene expected to appear in the virtual space B of the receiving side RE at the current moment. The reproducing unit RP reproduces the generated rendering data of the current scene at the current moment (step S6).

[0090] [1-4. Effect]

[0091] The information processing device PD includes a rendering unit RD and a reproduction unit RP. The rendering unit RD generates rendering data of a subsequent scene expected to appear in the virtual space B of the receiver RE at a subsequent time after the current time based on the sensor information at the current time acquired from the transmitter TR. The reproduction unit RP reproduces the rendering data in the virtual space B of the receiver RE at the subsequent time. In the information processing method of the present disclosure, the processing of the information processing device PD is performed by the computer 1000 (see Fig.18 ). The program of the present disclosure causes the computer 1000 to implement the processing of the information processing device PD.

[0092] According to this configuration, rendering data of a subsequent scene that is not the current scene is generated based on the sensor information at the current moment obtained from the transmitter TR. Since the rendering data of the subsequent scene generated in advance is reproduced at the time of the scheduled reproduction in the subsequent scene, the deviation of the reproduction timing on the receiving side RE due to communication delay is suppressed.

[0093] The reproduction data includes tactile data.

[0094] According to this configuration, the deviation of the timing of the tactile presentation between the sender and the receiver is suppressed. In the interaction using the sense of touch, it is easier to detect the deviation of the reproduction timing than in the interaction using images or sounds. The deviation of the reproduction timing causes discomfort. By using the method of the present disclosure, interaction with less discomfort can be performed.

[0095] The subsequent scene is set as a scene that appears after a time that is later than the current time by the image delay time.

[0096] According to this configuration, an appropriate scene is reproduced on the receiving side RE side without being affected by communication delay.

[0097] The reproduction unit RP reproduces the wide-area scene WS extending from the sender TR side to the receiver RE side in the virtual space A of the sender TR by using the rendering image RIM generated based on the sensor information at the current moment and the monitor image CIM obtained from the receiver RE at a time earlier than the current moment by the communication delay time.

[0098] According to this configuration, deviation of the image at the boundary between the rendering image RIM and the monitor image CIM is suppressed.

[0099] The wide area scene WS includes a scene in which the ripple RL extends from the transmitter TR side to the receiver RE side. The rendering unit RD generates rendering data of a subsequent scene expected to appear after the moment when the ripple RL reaches the boundary BD.

[0100] According to this configuration, occurrence of deviation in the spreading timing of the ripple RL between the transmission side TR side and the reception side RE side is suppressed.

[0101] It should be noted that the effects described in this specification are merely examples and are not limited, and other effects may be provided. In the above description, the receiver RE anticipates the time when the ripple RL reaches the boundary BD based on the sensor information of the sender TR at the current moment. However, this anticipation may be made by the sender TR. In this case, the receiver RE may render based on the expected information received from the sender TR.

[0102] [1-5. First Modification Example]

[0103] Hereinafter, a modification example of information processing performed on the receiving side RE will be described. Fig.10 It is an explanatory diagram of the first modification example.

[0104] As described above, the rendering image RIM reproduced in the virtual space B on the receiving side RE is an appropriate reproduction image that is not affected by the communication delay. However, the monitor image CIM of the virtual space B that has reproduced the rendering image RIM is transmitted to the sending side TR via the server SV. The rendering image RIM of the virtual space B on the monitor image CIM becomes a past image delayed by the image delay time, and the expansion of the ripple RL is also reduced. For this reason, a deviation occurs between the rendering image RIM reproduced in the virtual space of the sending side TR and the rendering image RIM of the virtual space B on the monitor image CIM (see Fig.10 upper side).

[0105] Therefore, the reproduction unit RP corrects a portion of the image of the monitor image CIM adjacent to the boundary BD, thereby eliminating the discontinuity of the image at the boundary between the rendering image RIM and the monitor image CIM. Fig.10 In the example of , in the monitor image CIM, the image area with a predetermined width constituting the lower end portion of the screen SCR is replaced with the correction image CIR. According to this configuration, a high-quality wide-area scene WS in which the rendering image RIM and the monitor image CIM are smoothly connected is reproduced.

[0106] [1-6. Second Modification Example]

[0107] Fig.11 It is an explanatory diagram of the second modification example.

[0108] exist Fig.11 In the example of , a stepping operation for generating a ripple RL is performed near the screen SCR. Depending on the stepping position of the user US, the propagation time before the ripple RL reaches the boundary BD may be shorter than the communication delay time. In this case, the receiving side RE cannot complete the rendering before the ripple RL reaches the boundary BD. For this reason, a deviation in the reproduction timing occurs between the sending side TR side and the receiving side RE side.

[0109] In order to solve this problem, in the present modification, the propagation speed of the ripple RL is adjusted on the transmission side TR side. In the case where the propagation time of the ripple RL from the ripple generation source (step position) to the boundary BD is shorter than the communication delay time, the reproduction unit RP of the transmission side TR adjusts the propagation speed of the ripple RL so that the propagation time is equal to or greater than the communication delay time. According to this configuration, the deviation of the expansion timing of the ripple RL between the transmission side TR side and the reception side RE side is suppressed.

[0110] [1-7. Third Modification]

[0111] Fig.12 It is an explanatory diagram of the third modification example.

[0112] exist Fig.12 In the example of FIG. 1 , a step is detected on the transmitter TR side, and the sensor information is transmitted to the receiver RE, but the ripple RL disappears on the way. In the receiver RE, rendering has been completed based on the sensor information received from the transmitter TR. If no measures are taken, the rendering data will be erroneously reproduced on the receiver RE side at the scheduled arrival time of the ripple RL to be reproduced.

[0113] In order to solve this problem, in the present modification, based on the disappearance information of the ripple RL, the reproduction stop processing of the rendering data is performed. When it is detected that the ripple RL has disappeared on the way, the transmission side TR transmits information indicating that the ripple RL has disappeared (disappearance information) to the reception side RE. When the ripple RL disappears before the ripple RL reaches the boundary BD, the reproduction unit RP of the reception side RE stops reproducing the ripple RL on the reception side RE side based on the rendering data. According to this configuration, the image of the ripple RL scheduled to be reproduced is prevented from being erroneously reproduced on the reception side RE side.

[0114] [1-8. Fourth Modification]

[0115] Fig.13 It is an explanatory diagram of the fourth modification example.

[0116] Although the shape of the ripple RL is a circle, the ripple RL having a long axis in the stepping direction can be generated by strongly performing an inclined stepping. In this case, the rendering unit RD determines the timing at which the sensor information should be acquired from the transmitter TR and the start timing of the rendering process for generating rendering data based on the positional relationship between the ripple generation source GS and the boundary BD and the propagation direction (long axis direction) of the ripple RL. According to this configuration, rendering can be performed according to the timing at which the ripple RL reaches the boundary BD.

[0117] exist Fig.13 In the example of , the propagation direction (the direction of the long axis) of the ripple RL is determined by the angle of the foot when stepping and the order of the stepped pressure-sensitive sensors PS. The time from the generation of the ripple RL to the arrival of the ripple RL at the user US on the receiving side RE depends on the angle θ of the stepping direction, the distance L1 from the ripple generation source GS to the boundary BD, the distance L2 from the boundary BD to the user US on the receiving side RE, and the propagation speed v of the ripple RL. Therefore, considering this information and the communication delay, it is possible to determine how early the sensor information should be transmitted and how early the rendering should be started.

[0118] [2. Second Embodiment]

[0119] Fig.14 is a view showing a telecommunication system according to a second embodiment.

[0120] In this embodiment, the tactile sensation when touching the virtual object VOB is reproduced by the glove-type tactile device HD. The rendering data of the virtual object VOB is reproduced by an image reproduction device (such as a head-mounted display or augmented reality (AR) glasses). The user US can interact with the user US of another reproduction system RS via the virtual object VOB.

[0121] [2-1. Generation and reproduction of rendering data]

[0122] Fig.15and Fig.16 This is a diagram for explaining the process of generating and reproducing rendering data.

[0123] exist Fig.15 and Fig.16 In the example of , the virtual object VOB is the hand of the user US who is interacting. The sender TR transmits information related to the position and movement speed of the hand of the user US of the sender TR as sensor information. The receiver RE anticipates how the position and movement speed of the hand of the user US of the sender TR change at a moment later than the current moment by using the sensor information received from the sender TR. The receiver RE generates expected rendering data related to the image and tactile sense of the hand of the user US of the sender TR as rendering data of a subsequent scene. The timing and intensity of the generated tactile sense are obtained from the timing of the contact of the hands of the user US with each other and the movement speed of the hands at the time of contact.

[0124] For example, the communication delay data is 100 ms. The receiving side RE receives the sensor information 100 ms after the sending side TR starts transmitting the sensor information at the current moment. The receiving side RE generates rendering data of a subsequent scene expected to appear in the virtual space B of the receiving side RE at a moment later than the current moment by the communication delay time based on the sensor information at the current moment acquired from the sending side TR. As a result, in real time, the hand of the sending side TR is displayed at the same position in the virtual space A and the virtual space B, and there is no deviation in the timing of tactile presentation.

[0125] In the above description, the position and movement speed of the hand of the user US of the transmitter TR at the subsequent time have been anticipated by the receiver RE. However, the anticipation may be made by the sender TR. In this case, the receiver RE may render based on the expected information received from the sender TR.

[0126] [2-2. Fifth Modification]

[0127] Fig.17 It is an explanatory diagram of the fifth modification example.

[0128] In the above-mentioned embodiment, a description has been given assuming that the communication delay speeds in the uplink direction and the downlink direction are the same. However, when the communication delay time between the uplink direction and the downlink direction is different, the position information expected after how long can be determined according to each communication delay speed. For example, in a case where the communication delay time in the uplink direction is 100 ms, the position and movement speed of the sensor information at the current moment after 100 ms can be expected, and rendering can be performed based on the expected information. In a case where the communication delay time in the downlink direction is 50 ms, the position and movement speed of the sensor information at the current moment after 50 ms can be expected, and rendering can be performed based on the expected information.

[0129] [3. Hardware configuration example]

[0130] Fig.18 is a view showing an example of the hardware configuration of the information processing device PD.

[0131] For example, information processing of the information processing device PD is implemented by a computer 1000. The computer 1000 includes a central processing unit (CPU) 1100, a random access memory (RAM) 1200, a read only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

[0132] The CPU 1100 operates and controls each unit based on a program (program data 1450) stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 expands the program stored in the ROM 1300 or the HDD 1400 in the RAM 1200 and executes processing corresponding to various programs.

[0133] The ROM 1300 stores a startup program such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, a hardware program depending on the computer 1000 , and the like.

[0134] The HDD 1400 is a non-transitory computer-readable recording medium that non-transitorily records a program executed by the CPU 1100 , data used by the program, and the like. Specifically, the HDD 1400 is a recording medium that records an information processing program according to the embodiment as an example of the program data 1450 .

[0135] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from another device via the communication interface 1500 or transmits data generated by the CPU 1100 to another device.

[0136] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. In addition, the CPU 1100 transmits data to an output device such as a display device, a speaker, or a printer via the input / output interface 1600. In addition, the input / output interface 1600 can be used as a medium interface for reading a program recorded in a predetermined recording medium (medium). For example, the medium is an optical recording medium such as a digital versatile disk (DVD) or a phase-change rewritable disk (PD), a magneto-optical recording medium such as a magneto-optical disk (MO), a magnetic tape medium, a magnetic recording medium, a semiconductor memory, etc.

[0137] For example, in the case where the computer 1000 is used as the information processing device PD according to the embodiment, the CPU 1100 of the computer 1000 executes the information processing program loaded on the RAM 1200 to realize the functions of each unit described above. In addition, the HDD 1400 stores the information processing program according to the present disclosure, various models, and various data. Note that the CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program data, but as another example, these programs may be acquired from another device via the external network 1550.

[0138] [Supplementary explanation]

[0139] Note that the present technology can also adopt the following configurations.

[0140] (1) An information processing device comprising:

[0141] a rendering unit that generates rendering data of a subsequent scene that is expected to appear in the virtual space of the receiver at a subsequent time after the current time based on sensor information at the current time acquired from the sender; and

[0142] A reproduction unit is configured to reproduce the rendering data in a virtual space of the recipient at the subsequent time.

[0143] (2) The information processing device according to (1), wherein:

[0144] The rendering data includes haptic data.

[0145] (3) The information processing device according to (2), wherein:

[0146] The subsequent scene is set as a scene that appears after a time later than the current time by a communication delay time.

[0147] (4) The information processing device according to (3), wherein:

[0148] The reproduction unit reproduces a wide-area scene extending from the sender side to the receiver side in the sender's virtual space by using a rendering image generated based on the sensor information at the current moment and a monitor image acquired from the receiver at a time earlier than the current moment by the communication delay time.

[0149] (5) The information processing device according to (4), wherein:

[0150] The reproduction unit corrects an image of a portion of the monitor image adjacent to a boundary between the rendering image and the monitor image to eliminate discontinuity of the image at the boundary.

[0151] (6) The information processing device according to (5), wherein:

[0152] The wide area scenario includes a scenario in which ripples spread from the sender side to the receiver side, and

[0153] The rendering unit generates rendering data of the subsequent scene expected to appear after the moment when the ripple reaches the boundary.

[0154] (7) The information processing device according to (6), wherein:

[0155] In a case where a propagation time of the ripple from a ripple generation source to the boundary is shorter than the communication delay time, the reproduction unit adjusts a propagation speed of the ripple so that the propagation time is equal to or greater than the communication delay time.

[0156] (8) The information processing device according to (6), wherein:

[0157] The reproduction unit stops reproducing the ripple on the receiving side based on the rendering data, if the ripple disappears before reaching the boundary.

[0158] (9) The information processing device according to (6), wherein:

[0159] The rendering unit determines a timing at which the sensor information should be acquired from the sender and a start timing of a rendering process for generating the rendering data based on a positional relationship between a ripple generation source and the boundary and a propagation direction of the ripple.

[0160] (10) An information processing method executed by a computer, comprising:

[0161] Based on the sensor information at the current moment acquired from the sender, generating rendering data of a subsequent scene expected to appear in the virtual space of the receiver at a subsequent moment after the current moment; and

[0162] The rendering data is reproduced in the virtual space of the recipient at the subsequent time.

[0163] (11) A program that causes a computer to execute:

[0164] Based on the sensor information at the current moment acquired from the sender, generating rendering data of a subsequent scene expected to appear in the virtual space of the receiver at a subsequent moment after the current moment; and

[0165] The rendering data is reproduced in the virtual space of the recipient at the subsequent time.

[0166] Reference numerals

[0167] BD Boundary

[0168] CIM Monitor Image

[0169] GS Ripple Source

[0170] PD Information Processing Device

[0171] RD Rendering Unit

[0172] RE Receiver

[0173] RIM Rendered Image

[0174] RL Ripple

[0175] RP Reproduction Unit

[0176] TR Sender

[0177] WS Wide Area Scenario

Claims

1. An information processing device, comprising: a rendering unit that generates rendering data of a subsequent scene that is expected to appear in the virtual space of the receiver at a subsequent time after the current time based on the sensor information at the current time acquired from the sender; as well as A reproduction unit is configured to reproduce the rendering data in a virtual space of the recipient at the subsequent time.

2. The information processing device according to claim 1, wherein The rendering data includes haptic data.

3. The information processing device according to claim 1, wherein The subsequent scene is set as a scene that appears after a time later than the current time by a communication delay time.

4. The information processing device according to claim 3, wherein The reproduction unit reproduces a wide-area scene extending from the sender side to the receiver side in the sender's virtual space by using a rendering image generated based on the sensor information at the current moment and a monitor image acquired from the receiver at a time earlier than the current moment by the communication delay time.

5. The information processing device according to claim 4, wherein The reproduction unit corrects an image of a portion of the monitor image adjacent to a boundary between the rendering image and the monitor image to eliminate discontinuity of the image at the boundary.

6. The information processing device according to claim 5, wherein The wide area scenario includes a scenario in which ripples spread from the sender side to the receiver side, and The rendering unit generates rendering data of the subsequent scene expected to appear after the moment when the ripple reaches the boundary.

7. The information processing device according to claim 6, wherein In a case where a propagation time of the ripple from a ripple generation source to the boundary is shorter than the communication delay time, the reproduction unit adjusts a propagation speed of the ripple so that the propagation time is equal to or greater than the communication delay time.

8. The information processing device according to claim 6, wherein: The reproduction unit stops reproducing the ripple on the receiving side based on the rendering data, if the ripple disappears before reaching the boundary.

9. The information processing device according to claim 6, wherein: The rendering unit determines a timing at which the sensor information should be acquired from the sender and a start timing of a rendering process for generating the rendering data based on a positional relationship between a ripple generation source and the boundary and a propagation direction of the ripple.

10. An information processing method executed by a computer, comprising: Based on the sensor information at the current moment acquired from the sender, generating rendering data of a subsequent scene expected to appear in the virtual space of the receiver at a subsequent moment after the current moment; as well as The rendering data is reproduced in the virtual space of the recipient at the subsequent time.

11. A program causing a computer to execute: generating rendering data of a subsequent scene expected to appear in the virtual space of the receiver at a subsequent time after the current time based on the sensor information at the current time acquired from the sender; and The rendering data is reproduced in the virtual space of the recipient at the subsequent time.

Images