Information Processing Apparatus, Information Processing Method, and Computer Program Product

Abstract

An information processing apparatus includes a control unit that is configured to display a virtual object on three-dimensional coordinates associated with a real space and cause the virtual object to perform a predetermined operation according to a predetermined sound reproduced from a real sound source located in the real space. The control unit performs a delay amount setting process for gradually increasing a delay amount of an operation start timing of the predetermined operation performed by the virtual object with respect to a reproduction timing of the predetermined sound reproduced from the real sound source in correspondence with an increase in a distance between a position of the real sound source in the real space and a position of the virtual object on the three-dimensional coordinates.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program. Background Art

[0002] Conventionally, schemes for controlling the reproduction timing of content have been proposed (for example, see Patent Documents 1 and 2 below).

[0003] Citation List

[0004] Patent Documents

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2018-183483

[0006] Patent Document 2: Japanese Patent Application Laid-Open No. 2015-073182 Summary of the Invention

[0007] Problems to be Solved by the Invention

[0008] Incidentally, at a live venue, generally, the longer the distance to the stage, the later the sound emitted from the stage reaches the user. In recent years, a live format has been proposed in which, while reproducing sound from speakers on the stage, a video of a virtual singer is reproduced instead of a video of an actual singer singing or dancing on the stage. Even in this live format, it is necessary to deal with the reality related to the delay of the sound reproduced from the speakers, but it is difficult to appropriately recreate the reality related to the delay of the sound using the techniques described in the above patent documents.

[0009] An object of the present disclosure is to provide an information processing apparatus, an information processing method, and a program capable of appropriately recreating the reality related to the delay of sound reproduced from a real sound source.

[0010] Solution to the Problem

[0011] The present disclosure is, for example,

[0012] An information processing apparatus including:

[0013] a control unit that displays a virtual object on a three-dimensional coordinate space associated with a real space and causes the virtual object to perform a predetermined action according to a predetermined sound reproduced from a real sound source provided in the real space,

[0014] wherein the control unit performs a delay amount setting process in which, as the position of the real sound source in the real space and the position of the virtual object on the three-dimensional coordinate space are separated further from each other, the delay amount of the action start timing of the virtual object performing the predetermined action with respect to the reproduction timing of the predetermined sound reproduced from the real sound source increases.

[0015] The present disclosure is, for example,

[0016] An information processing method executed by a control unit, comprising:

[0017] Displaying a virtual object on a three-dimensional coordinate space associated with a real space, and causing the virtual object to perform a predetermined action according to a predetermined sound reproduced from a real sound source provided in the real space,

[0018] wherein the control unit performs a delay amount setting process in which, as the positions of the real sound source in the real space and the virtual object on the three-dimensional coordinate space are separated farther from each other, the delay amount of the action start timing of the virtual object performing the predetermined action with respect to the reproduction timing of the predetermined sound reproduced from the real sound source increases.

[0019] The present disclosure is, for example,

[0020] A program for causing a computer to execute an information processing method executed by a control unit, wherein the method comprises:

[0021] Displaying a virtual object on a three-dimensional coordinate space associated with a real space, and causing the virtual object to perform a predetermined action according to a predetermined sound reproduced from a real sound source provided in the real space,

[0022] wherein the control unit performs a delay amount setting process in which, as the positions of the real sound source in the real space and the virtual object on the three-dimensional coordinate space are separated farther from each other, the delay amount of the action start timing of the virtual object performing the predetermined action with respect to the reproduction timing of the predetermined sound reproduced from the real sound source increases. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a diagram for describing a reproduction system according to one embodiment.

[0024] Figure 2 is a diagram to be referred to when describing problems to be considered in one embodiment.

[0025] Figure 3 is a diagram for describing an overview of one embodiment.

[0026] Figure 4 A to D in are diagrams for describing an overview of one embodiment.

[0027] Figure 5 is a diagram for describing a specific example of a reproduction system.

[0028] Figure 6 is a diagram for describing a specific example of processing executed by a reproduction system according to one embodiment.

[0029] Figure 7 It is a diagram for describing a specific example of processing performed by a reproduction system according to an embodiment.

[0030] Figure 8 It is a diagram for describing a specific example of processing performed by a reproduction system according to an embodiment.

[0031] Figure 9 It is a diagram for describing a specific example of processing performed by a reproduction system according to an embodiment.

[0032] Figure 10 It is a diagram for describing a specific example of processing performed by a reproduction system according to an embodiment.

[0033] Figure 11 It is a diagram for describing a specific example of processing performed by a reproduction system according to an embodiment.

[0034] Figure 12 It is a table indicating that the reality related to sound delay can be achieved by the processing performed by a reproduction system according to an embodiment.

[0035] Figure 13 It is a diagram to be referred to when describing an example of a mathematical expression for calculating a delay amount for obtaining an action start timing in an embodiment.

[0036] Figure 14 It is a diagram showing a configuration example of a system including multiple devices in an embodiment.

[0037] Figure 15 It is a diagram for describing respective configuration examples of a server, a smart display, and a speaker according to an embodiment.

[0038] Figure 16 It is a flowchart to be referred to when describing a processing flow performed in an embodiment.

[0039] Figure 17 It is a flowchart to be referred to when describing a processing flow performed in an embodiment.

[0040] Figure 18 It is a flowchart to be referred to when describing a processing flow performed in an embodiment.

[0041] Figure 19 It is a diagram to be referred to when describing a modification example.

[0042] Figure 20 It is a diagram to be referred to when describing a modification example. Detailed Description

[0043] In the following, embodiments of the present disclosure are described with reference to the accompanying drawings. Note that the description is given in the following order.

[0044] <One embodiment>

[0045] <Modification example>

[0046] The embodiments and the like described below are preferred specific examples of the present disclosure, and the present disclosure is not limited to such embodiments and the like.

[0047] <One embodiment>

[0048] [Processing performed in the embodiment]

[0049] First, the processing performed in the present embodiment is described. In the present embodiment, a relatively large-scale reproduction system, such as a live venue, is assumed.

[0050] (Overview of the reproduction system)

[0051] Figure 1 is a diagram schematically showing a reproduction system (reproduction system 1) according to the present embodiment. The reproduction system 1 includes, for example, a stage 2, and speakers 3 as examples of real sound sources are provided on or around the stage 2. Actual sounds (music, sound effects, notification sounds, etc.) are reproduced from the speakers 3. Although Figure 1 an example in which the speakers 3 are provided on both the left and right sides of the stage 2 is shown, the number and arrangement pattern of the speakers 3 can be appropriately changed.

[0052] Facing the stage 2, there are users as the audience. The users watch a video in which a virtual object imitating a singer or a dancer is displayed, while listening to the sound reproduced from the speakers 3. The virtual object can be a real person such as a singer, rather than a virtual object such as an artificially created virtual singer, and the video can be a video in which such a real person is displayed. The video is displayed on a smart display carried or worn by the user. The smart display is a display capable of presenting virtual information in the real space, and specifically, examples thereof can include the display of a smart phone owned by the user, a glasses-type display (also referred to as augmented reality (AR) glasses, etc.), a head-up display (HUD), etc. Each user can watch the same video content by, for example, installing the same application in a device including the smart display and reproducing the video content. As described above, the reproduction system 1 according to the present embodiment is a system that performs on-site reproduction in which a real stage and a virtual object are integrated.

[0053] (Problems to be considered)

[0054] Next, problems to be considered in the above-described reproduction system 1 will be described. When attempting to recreate the reality related to the delay at which music reproduced from a real sound source reaches the user while a video is being displayed on the smart display, the appropriate timing for the virtual object to perform a predetermined action (motion) varies according to the distance between the user and the stage 2 and the position of the virtual object.

[0055] Figure 2 is a diagram for describing an example of the reality corresponding to the user's position and the position of the virtual object. In Figure 2 the table shown, the position of the virtual object as the virtual object is defined in the vertical direction. Specifically, the mode of the position of the virtual object on the stage and the mode of the position of the virtual object directly in front of the user are defined. Further, in Figure 2 the table shown, the relationship between the position of the user and the position of the stage is defined in the horizontal direction. Specifically, the mode in which the position of the user is close to the stage and the mode in which the position of the user is far from the stage are defined.

[0056] In the case of the mode where the position of the virtual object is on the stage and the position of the user is close to the stage (hereinafter appropriately referred to as "mode P1"), reality can be felt when the timing of the video coincides with the timing of the sound. In the case of the mode where the position of the virtual object is directly in front of the user and the position of the user is close to the stage (hereinafter appropriately referred to as "mode P2"), reality can be felt when the timing of the video coincides with the timing of the sound. According to mode P1 and mode P2, reality can be felt when the timing of the video always coincides with the timing of the sound.

[0057] In the case of the mode where the position of the virtual object is on the stage and the position of the user is far from the stage (hereinafter appropriately referred to as "mode P3"), reality can be felt when the sound reaches the user later than the video. In the case of the mode where the position of the virtual object is directly in front of the user and the position of the user is far from the stage (hereinafter appropriately referred to as "mode P4"), reality can be felt when the timing of the video coincides with the timing of the sound. According to mode P3 and mode P4, reality can be felt when the display timing varies according to the position of the virtual object.

[0058] Further, according to mode P1 and mode P3, reality can be felt when the display timing of the virtual object is the same regardless of the distance between the user and the stage. Further, according to mode P2 and mode P4, reality can be felt when the display timing of the virtual object varies according to the distance between the user and the stage. The technique described in the above patent document has the problem of not being able to recreate reality.

[0059] (Overview of an Embodiment)

[0060] Next, referring to Figure 3 andFigure 4 Describe the overview of this embodiment. Figure 3 is a diagram schematically showing the above four modes (mode P1, mode P2, mode P3, and mode P4), and in Figure 3 , the user close to the stage 2 is user UA and the user far from the stage 2 is user UB. In addition, Figure 3 shows the virtual object C displayed on the smart display according to each mode.

[0061] Figure 4 A to D in Figure 4 are diagrams for describing the overview of the processing performed according to each mode.

[0062] Figure 4 A in

[0063] Figure 4 shows the processing performed in the case of mode P1. In the case of mode P1, a process is performed to make the timing (hereinafter appropriately referred to as the "action start timing") at which the virtual object performs a predetermined action close to the reproduction timing of the sound reproduced from the real sound source.

[0064] Figure 4 B in

[0065] shows the processing performed in the case of mode P2. In the case of mode P2, a process is performed to make the action start timing at which the virtual object performs a predetermined action close to the reproduction timing of the sound reproduced from the real sound source.

[0066] C in Figure 5 shows the processing performed in the case of mode P3. In the case of mode P3, a process is performed to make the action start timing at which the virtual object performs a predetermined action close to the reproduction timing of the sound reproduced from the real sound source.

[0067] The position PO2 in the reproduction system 1 indicates the position of the user UA near the stage 2. For example, the position of the user UA corresponds to the position of the smart display held or worn by the user UA. The distance between the user UA and the stage 2 is, for example, 10 m. Regarding sound, the user UA listens to the sound of the music reproduced from the speaker 3, and visually recognizes the virtual object through the glasses-type AR glasses 11A as an example of the smart display. Note that when the virtual object comes directly in front of the user UA, that is, when the virtual object is displayed on the AR glasses 11A so as to be located directly in front of the user UA, the position of the virtual object substantially coincides with the position PO2.

[0068] In addition, the position PO3 in the reproduction system 1 indicates the position of the user UB far from the stage 2. For example, the position of the user UB corresponds to the position of the smart display held or worn by the user UB. The distance between the user UB and the stage 2 is, for example, 200 m. Regarding sound, the user UB listens to the sound of the music reproduced from the speaker 3, and visually recognizes the virtual object through the glasses-type AR glasses 11B as an example of the smart display. Note that when the virtual object comes directly in front of the user UB, that is, when the virtual object is displayed on the AR glasses 11B so as to be located directly in front of the user UB, the position of the virtual object substantially coincides with the position PO3.

[0069] In the reproduction system 1 in the above real space, the distance between the speaker 3 and the position of the virtual object is defined as D (m). In this case, the action start timing at which the virtual object performs a predetermined action after the sound corresponding to the predetermined action is reproduced from the speaker 3 is set to the timing delayed by the following from the predetermined timing

[0070] D×1000 / 340 (milliseconds)…(Mathematical expression 1)

[0071] (Here, it is assumed that the speed of sound is 340 m / sec).

[0072] Therefore, as the position of the speaker 3 in the real space and the position of the virtual object in the three-dimensional coordinate space are separated from each other farther, the delay amount of the action start timing at which the virtual object performs a predetermined action with respect to the reproduction timing of the speaker 3 reproducing the predetermined sound is set to be larger. From the set action start timing, a video in which the virtual object performs a predetermined action is presented on the AR glasses.

[0073] Specifically, the delay amount from the reproduction of the sound from the speaker 3 to the start of the predetermined action of the virtual object is set as follows.

[0074] (1) In the case of the user UA

[0075] The time taken for the sound reproduced from the speaker 3 to reach the user UA: approximately 30 milliseconds

[0076] The delay amount when the virtual object is on stage 2: 0 milliseconds (since the distance between the speaker 3 and the virtual object is 0 m, and D = 0 in (Mathematical Expression 1))

[0077] The delay amount of the action start timing when the virtual object is directly in front of the user UA: approximately 30 milliseconds (since the distance between the speaker 3 and the virtual object is 10 m, and D = 10 in (Mathematical Expression 1)).

[0078] (2) In the case of user UB

[0079] The time taken for the sound reproduced from the speaker 3 to reach the user UB: approximately 600 milliseconds

[0080] The delay amount when the virtual object is on stage 2: 0 milliseconds (since the distance between the speaker 3 and the virtual object is 0 m, and D = 0 in (Mathematical Expression 1))

[0081] The delay amount of the action start timing when the virtual object is directly in front of the user UB: approximately 600 milliseconds (since the distance between the speaker 3 and the virtual object is 200 m, and D = 200 in (Mathematical Expression 1)).

[0082] Based on the above, specifically describe the action start timing, etc.

[0083] Figure 6 It is a timing diagram in the case where the user is the user UA at a position close to the speaker 3 and the position of the virtual object C is on stage 2. As an example of the timing of the sound reproduced from the speaker 3 as a real sound source, the timings T1, T2, and T3 are defined in chronological order. Without considering the delay of the sound, etc., the virtual object C performs a predetermined action at the timing T1. When the timing of the sound for performing the predetermined action is the timing t (in this example, t = T1), the sound reproduced from the speaker 3 reaches the user UA after (t + 30) milliseconds. Note that the action performed by the virtual object C is an example of an action including actions M1, M2, and M3. In addition, in Figure 6 etc., the same note markings represent the same sound.

[0084] For the convenience of understanding the processing performed in this embodiment (hereinafter also referred to as this solution), Reference Examples 1 and 2 will be described together with this solution. Reference Example 1 is an example where the action start timing always coincides with the reproduction timing of a predetermined sound. Reference Example 2 is an example where the action start timing always coincides with the timing when the predetermined sound reaches the user UA.

[0085] As Figure 6As shown, in Reference Example 1, the virtual object C starts operation M1 at timing t when the sound is reproduced from the speaker 3, and then executes operation M2 and operation M3. That is, the operation start timing is set to timing t. In this case, although there is a 30 - millisecond difference between the timing when the sound reaches the user UA and the operation start timing, the 30 - millisecond difference is at a level where the user will not feel uncomfortable aurally. Therefore, according to Reference Example 1, a reality related to sound delay can be achieved.

[0086] In addition, in Reference Example 2, the virtual object C starts operation M1 at the timing (t + 30 milliseconds) when the sound reproduced from the speaker 3 reaches the user UA, and then executes operation M2 and M3. That is, the operation start timing is set to (t + 30 milliseconds). In this case, the timing when the sound reaches the user UA is the same as the operation start timing. Since the user UA is at a position close to the stage 2, according to Reference Example 2, a reality related to sound delay can be achieved.

[0087] Furthermore, in this solution, as described above, the delay amount of the operation start timing is set to 0. Therefore, similar to Reference Example 1, this solution can achieve a reality related to sound delay.

[0088] Figure 7 It is a timing diagram in the case where the user is the user UB at a position far from the speaker 3 and the position of the virtual object C is on the stage 2. In this case, the sound reproduced from the speaker 3 reaches the user UB after (t + 600 milliseconds).

[0089] In Reference Example 1, the virtual object C starts operation M1 at timing t when the sound is reproduced from the speaker 3, and then executes operation M2 and M3. That is, the operation start timing is set to timing t. In addition, the sound reproduced from the speaker 3 reaches the user UB after (t + 600 milliseconds). According to Reference Example 1, a reality related to sound delay can be achieved.

[0090] In Reference Example 2, the virtual object C starts operation M1 at the timing (t + 600 milliseconds) when the sound reproduced from the speaker 3 reaches the user UB, and then executes operation M2 and M3. In Reference Example 2, although the sound recognized aurally should reach the user UB later than the video recognized visually, the timing when the sound reaches the user UB is the same as the operation start timing. Therefore, according to Reference Example 2, a reality related to sound delay cannot be achieved.

[0091] According to this solution, since the virtual object is on stage 2, the delay amount is set to 0, that is, the action start timing is set to timing t. In addition, the sound reproduced from the speaker 3 reaches the user UB after (t + 600 milliseconds). Therefore, according to this solution, the reality related to sound delay can be achieved.

[0092] Figure 8 It is a timing diagram when the user is user UA at a position close to the speaker 3 and the position of the virtual object C is directly in front of the user. Since the virtual object C is an object displayed on the smart display, the position of the virtual object C can be freely changed in the content, which is different from the actual scene. For example, the virtual object C can be displayed as if the virtual object C flies from stage 2 to a position directly in front of the user UA. In this case, as described above, since the virtual object C is directly in front of the user, when the action start timing coincides with the timing when the user UA hears the sound, the reality related to sound delay can be felt.

[0093] In Reference Example 1, the virtual object C starts action M11 at the timing t when the sound is reproduced from the speaker 3, and then performs actions M12 and M13. That is, the action start timing is set to timing t. In this case, the timing when the sound reaches the user UA is basically the same as the action start timing (a difference of 30 milliseconds) to the extent that there is no auditory discomfort. Therefore, according to Reference Example 1, the reality related to sound delay can be achieved.

[0094] In addition, in Reference Example 2, the virtual object C starts action M11 at the timing (t + 30 milliseconds) when the sound reproduced from the speaker 3 reaches the user UA, and then performs actions M12 and M13. That is, the action start timing is set to timing (t + 30 milliseconds). In this case, the timing when the sound reaches the user UA coincides with the action start timing. Based on the above, according to Reference Example 2, the reality related to sound delay can be achieved.

[0095] In addition, in this solution, as described above, the action start timing is set to (t + 30 milliseconds). Therefore, similar to Reference Example 2, according to this solution, the reality related to sound delay can be achieved.

[0096] Figure 9 It is a timing diagram when the user is user UB at a position far from the speaker 3 and the position of the virtual object C is directly in front of the user. In this case, the sound reproduced from the speaker 3 reaches the user UB after (t + 600 milliseconds). As described above, since the virtual object C is directly in front of the user, when the action start timing coincides with the timing when the user UB hears the sound, the reality related to sound delay can be felt.

[0097] In Reference Example 1, the virtual object C starts operation M11 at timing t when the sound is reproduced from the speaker 3, and then performs operations M12 and M13. That is, the operation start timing is set to timing t. On the other hand, the sound reaches the user UB after (t + 600 msec), and the user UB recognizes the sound. Therefore, according to Reference Example 1, the reality related to sound delay cannot be achieved.

[0098] In addition, in Reference Example 2, the virtual object C starts operation M11 at the timing (t + 600 milliseconds) when the sound reproduced from the speaker 3 reaches the user UB, and then performs operations M12 and M13. Since the timing when the user UB hears the sound coincides with the operation start timing, the reality related to sound delay can be achieved according to Reference Example 2.

[0099] In addition, in this solution, as described above, since the operation start timing is set to (t + 600 milliseconds), the timing when the user UB hears the sound coincides with the operation start timing. Therefore, similar to Reference Example 2, the reality related to sound delay can be achieved in this solution.

[0100] Figure 10 It is a timing chart in the middle case where the user is the user UA at a position close to the speaker 3 and the position of the virtual object C is between the speaker 3 and the user UA (the distance between the position of the speaker 3 and the position of the virtual object C is 5 m). Since the position of the virtual object C can be considered to be substantially the same as the position on the stage 2, when the timing of hearing the sound substantially coincides with the operation start timing, the reality related to sound delay can be felt.

[0101] In Reference Example 1, timing t is the operation start timing. Therefore, the reality related to sound delay can be achieved. According to Reference Example 2, the operation start timing is set to timing (t + 30 milliseconds). As described above, since it is considered that a difference of 30 milliseconds basically does not affect audibility, the reality related to sound delay can also be achieved according to Reference Example 2.

[0102] In this solution, by substituting D = 5 into (Mathematical Expression 1), the set delay amount is calculated to be 15 milliseconds. The operation start timing is set to timing (t + 15 milliseconds). Since it is considered that a difference of 15 milliseconds basically does not affect audibility, the reality related to sound delay can also be achieved according to this solution. When the operation start timing arrives, the video of the virtual object C performing a series of operations M21, 22, and 23 is presented on the AR glasses 11A.

[0103] Figure 11This is a timing diagram in the case where the user is user UB at a position far from the speaker 3 and the position of the virtual object C is in the middle between the speaker 3 and the user UB (when the distance between the position of the speaker 3 and the position of the virtual object C is 100 m and the distance between the virtual object C and the user UB is 100 m). In this case, since the distance between the virtual object C and the user UB is 100 m, when the sound reaches the user UB 300 milliseconds after the reproduction of the virtual object C, the reality related to the sound delay can be felt.

[0104] In Reference Example 1, the action start timing is set to t. Then, the sound reproduced from the speaker 3 reaches the user UB 600 milliseconds after starting from t. Therefore, in Reference Example 1, the reality related to the sound delay cannot be felt.

[0105] In Reference Example 2, the action start timing is set to (t + 600 milliseconds). The timing at which the sound reaches the user UB is the same as the action start timing. Therefore, in Reference Example 2, the reality related to the sound delay cannot be felt.

[0106] In this solution, by substituting D = 100 into (Mathematical Expression 1), the set delay amount is calculated to be 300 milliseconds. The action start timing is set to the timing (t + 300 milliseconds). Then, the sound reaches the user UB 300 milliseconds after the timing (t + 300 milliseconds). Therefore, in this solution, the reality related to the sound delay is felt. When the action start timing arrives, the video of the virtual object C performing a series of actions M21, 22, and 23 is presented on the AR glasses 11B.

[0107] Figure 12 This is a table summarizing the results of the above specific processing. In this table, the ○ mark indicates that the reality related to the sound delay can be achieved. In addition, in this table, the × mark indicates that the reality related to the sound delay cannot be achieved. Compared with Reference Examples 1 and 2, according to this solution, the reality related to the sound delay can be achieved in any case.

[0108] Figure 13 This is a diagram for describing the parameters required for setting the delay amount of the action start timing and a calculation example using this parameter.

[0109] As Figure 13 shown, the following expressions are provided:

[0110] Absolute coordinates of the virtual object: AC U +RC O ; and

[0111] Distance between the real sound source (speaker 3) and the virtual object: Distance(AC R ,AC U+RC O ) × α(m),

[0112] Among them, the position on the three-dimensional coordinate space (absolute coordinates) of the real sound source (loudspeaker 3) associated with the said real space: AC R ,

[0113] The position of the user associated with the real space on the three-dimensional coordinate space (absolute coordinates): AC U ,

[0114] The relative coordinates of the virtual object with respect to the user's position: RC O ,

[0115] The distance between the coordinates and the transformation coefficient for measuring the distance (m): α. Then, the delay amount at the start timing of the action can be expressed as follows.

[0116] Distance (AC R , AC U +RC O ) × α × 1000 / 340 (milliseconds) … (Mathematical expression 2)

[0117] (Configuration example of the system)

[0118] Figure 14 FIG. is a diagram showing a configuration example of a system (system 100) including a plurality of devices in one embodiment. The system 100 includes, for example, a server 30, a smart display (for example, the above-mentioned AR glasses 11A and 11B) 40, and a loudspeaker (for example, the above-mentioned loudspeaker 3) 50. Various types of data and commands can be exchanged between the devices through communication.

[0119] (Configuration example of each device)

[0120] "Configuration example of the server"

[0121] Refer to Figure 15 Describe the configuration example of each of the server 30, the smart display 40 as an example of an information processing device, and the loudspeaker 50 according to the present embodiment.

[0122] The server 30 includes, for example, a server control unit 301, a sound source position coordinate holding unit 302, a progress information holding unit 303, and a server communication unit 304. The server control unit 301 controls each unit of the server 30 as a whole. Unless otherwise specified, the processing performed by the server 30 described below is executed under the control of the server control unit 301.

[0123] The sound source position coordinate holding unit 302 is a memory that holds the position (absolute position) of the speaker 50, which is an example of a real sound source, in a three-dimensional coordinate space associated with the real space. The progress information holding unit 303 is a memory that holds content progress information (e.g., reproduction time). Note that the sound source position coordinate holding unit 302 and the progress information holding unit 303 may be the same memory or different memories.

[0124] The server communication unit 304 is configured to communicate with the smart display 40 and the speaker 50, and includes a modulation / demodulation circuit, an antenna, etc. compatible with the communication standard. The server communication unit 304 may be a communication unit compatible with a network such as the Internet, may be a communication unit compatible with short-range wireless communication such as a local area network (LAN), or may be a communication unit compatible with both of the above.

[0125] "Configuration Example of Smart Display"

[0126] The smart display 40 includes a terminal control unit 401, a terminal communication unit 402, a sound source position information holding unit 403, a progress information holding unit 404, a self-position information acquisition unit 405, a content presentation unit 406, and a display 407.

[0127] The terminal control unit 401 controls the entire smart display 40 integrally. The terminal control unit 401 includes, for example, a virtual object position information holding unit 401A, a display timing holding unit 401B, a virtual object absolute position calculation unit 401C, a distance calculation unit 401D, a start time calculation unit 401E, a start time comparison unit 401F, and an animation control unit 401G.

[0128] The virtual object position information holding unit 401A holds the position information of the virtual object. In addition, the display timing holding unit 401B holds the display timing set by default for the virtual object. Note that, for example, when an application including video content is installed in the smart display 40, the position information and the display timing of the virtual object are acquired. The position information and the display timing of the virtual object can be acquired through communication or the like. The position information and the display timing of the virtual object are defined in chronological order. The position information of the virtual object is information defined as a relative position with respect to the smart display 40. In addition, the display timing of the virtual object is the timing for displaying a video in which the virtual object performs a predetermined action, and the display timing of the virtual object is defined corresponding to the elapse of the reproduction time of the video content. The position information of the virtual object is provided from the virtual object position information holding unit 401A to the virtual object absolute position calculation unit 401C. The display timing of the virtual object is provided from the display timing holding unit 401B to the start time calculation unit 401E.

[0129] The virtual object absolute position calculation unit 401C calculates the absolute position of the virtual object based on the position information of the virtual object provided by the own position information acquisition unit 405 and the position information of the smart display 40. The absolute position of the virtual object is a predetermined position on a three-dimensional coordinate space associated with the real space. The virtual object is displayed (reproduced) at a position corresponding to the absolute position calculated by the virtual object absolute position calculation unit 401C on the display 407.

[0130] The distance calculation unit 401D calculates the distance between the speaker 50 and the virtual object based on the sound source position information that is the absolute position of the speaker 50 transmitted from the server 30 and the absolute position of the virtual object provided by the virtual object absolute position calculation unit 401C.

[0131] The start time calculation unit 401E calculates the start time, that is, the action start timing when the virtual object performs a predetermined action. More specifically, the start time calculation unit 401E performs a delay amount setting process in which, as the positions of the speaker 50 in the real space and the virtual object on the three-dimensional coordinate space are separated further from each other, the delay amount of the action start timing of the virtual object performing the predetermined action with respect to the reproduction timing of the speaker 50 reproducing the predetermined sound increases. The action start timing calculated by the start time calculation unit 401E is provided to the start time comparison unit 401F.

[0132] The start time comparison unit 401F determines whether the timing defined by the action start timing exceeds the time indicated by the progress information (hereinafter appropriately referred to as "progress time") based on the progress information provided by the progress information holding unit 404 and the action start timing provided by the start time calculation unit 401E. When the action start timing exceeds the progress time, the start time comparison unit 401F notifies the animation control unit 401G of this effect.

[0133] When the action start timing exceeds the progress time, the animation control unit 401G generates an animation in which the virtual object performs a series of predetermined actions. The animation generated by the animation control unit 401G is provided to the content presentation unit 406.

[0134] The terminal communication unit 402 communicates with an external device such as the server 30. Through the communication performed by the terminal communication unit 402, the smart display 40 acquires sound source position information and progress information provided from the server 30. The sound source position information acquired through the communication performed by the terminal communication unit 402 is held (stored) in the sound source position information holding unit 403, and the progress information is held in the progress information holding unit 404. The sound source position information held in the sound source position information holding unit 403 is provided to the distance calculation unit 401D. In addition, the progress information held in the progress information holding unit 404 is provided to the start time comparison unit 401F.

[0135] The own position information acquisition unit 405, which is a position information acquisition unit, acquires its own position information that is the current position of the smart display 40. The own position information can be directly input by the user, or can be acquired by the user performing an operation of reading a QR code (registered trademark) assigned to the location (such as a seat) where the user is located. In addition, the own position information can be acquired via a position sensor such as a global navigation satellite system (GNSS), or can be acquired by estimating the own position using simultaneous localization and mapping (SLAM) or a feature amount map obtained by pre-imaging. The position of the user holding the smart display 40 can be variable. In this case, the own position information also changes according to the change in the user's position.

[0136] The content presentation unit 406 performs a process of presenting an animation provided from the animation control unit 401G on the display 407. Specifically, a virtual object singing on the stage 2 or a virtual object flying in the venue is presented on the display 407. Accordingly, the user visually recognizes the virtual object displayed on the display 407.

[0137] The display 407 is, for example, an optical see-through display. The user recognizes the virtual object displayed on the display 407 while visually recognizing a real object such as the stage 2 through the display 407.

[0138] "Configuration Example of Speaker"

[0139] The speaker 50 includes a speaker communication unit 501, a music reproduction unit 502, and a speaker unit 503. The speaker communication unit 501 communicates with the server 30 and the smart display 40. The speaker communication unit 501 receives, for example, a sound reproduction command transmitted from the server 30 or the smart display 40.

[0140] The music reproduction unit 502 performs known sound signal processing for reproducing music data. Note that the music data may be stored in an appropriate memory of the speaker 50, or may be acquired through communication via the speaker communication unit 501. The music data processed by the music reproduction unit 502 is reproduced from the speaker unit 503.

[0141] Note that in the case where a plurality of virtual objects are included in the video content, position information and display timing are defined for each virtual object. In addition, the processing described below is also performed for each virtual object. The display timing is not necessarily set, and the virtual object may be pre-programmed to perform a predetermined action.

[0142] The progress information may be managed by the smart display 40 instead of the server 30. In addition, the smart display 40 may obtain the current progress information not from the server 30, but by using marker recognition, recognition by a camera using feature quantity mapping, or audio fingerprint. In addition, the sound source position information of the speaker 50 may be directly provided from the speaker 50 to the smart display 40.

[0143] "Processing flow"

[0144] Next, with reference to Figure 16 、 Figure 17 and Figure 18 the flow of the processing (processing mainly performed in the smart display 40) performed in the present embodiment is described. Figure 16 is a flowchart showing the flow of the overall processing. Figure 17 is shown Figure 16 the flowchart of the virtual object display timing control processing in Figure 18 is shown Figure 17 the flowchart of the action start timing calculation processing in

[0145] In Figure 16 In step ST11 in the flowchart shown, the user position is acquired by the own position information acquisition unit 405, in other words, the position of the smart display 40 held or worn by the user. As described above, the user position may be input by a user operation, or may be acquired by a sensor or the like. Then, the processing proceeds to step ST12.

[0146] In step ST12, the sound source position information of the speaker 50 as an example of the real sound source is acquired from the server 30, and the acquired sound source position information is held in the sound source position information holding unit 403. Then, the processing proceeds to step ST13.

[0147] In step ST13, the content starts to be reproduced. For example, in response to a content reproduction operation on the smart display 40, the content reproduction starts by a command from the server 30. When the content reproduction command from the server 30 is provided to the speaker 50, music data is reproduced from the speaker unit 503. When the content reproduction starts, the smart display 40 periodically obtains progress information from the server 30. Then, the process proceeds to step ST14.

[0148] In step ST14, the control related to the content reproduction is executed together with the content reproduction. Then, the process proceeds to step ST15.

[0149] In step ST15, it is determined whether X seconds or more have elapsed since the previous acquisition of the progress information. This determination is performed, for example, by the terminal control unit 401. For example, the terminal control unit 401 determines whether X seconds have elapsed since the progress information held in the progress information holding unit 404 was updated to the progress information transmitted from the server 30. X seconds can be appropriately set to several seconds to 5 seconds or the like. If X seconds have not elapsed since the previous acquisition of the progress information, the process proceeds to step ST18. If X seconds have elapsed since the previous acquisition of the progress information, the process proceeds to step ST16.

[0150] In step ST16, the smart display 40 obtains progress information from the server 430. Then, the process proceeds to step ST17.

[0151] In step ST17, when the progress information holding unit 404 holds the progress information obtained in the process of step ST16, the progress information is updated. Then, the process proceeds to step ST18.

[0152] In the present embodiment, the user position can change. When the user position changes, the process according to step ST18 is executed to provide a new user position from the own position information acquisition unit 405 to the terminal control unit 401, thereby updating the user position. Then, the process based on the updated user position is executed. Then, the process proceeds to step ST19.

[0153] In step ST19, the virtual object display timing control process is executed. The details of the virtual object display timing control process are described below. The action start timing of the virtual object is obtained through the virtual object display timing control process. Then, the process proceeds to step ST20.

[0154] In step ST20, a virtual object that performs a predetermined action according to the passage of the action start timing is presented on the display 407. Note that in the case of a virtual object that does not depend on the action start timing, for example, a virtual object is always presented on the display 407 when the content is reproduced.

[0155] Next, the process of virtual object display timing control in this embodiment will be described with reference to Figure 17 the flowchart shown in

[0156] For example, after the process of step ST18 shown in Figure 16 is executed, the virtual object display timing control process starts at step ST31. Then, the process proceeds to step ST32.

[0157] At step ST32, an action start timing calculation process is executed. The action start timing of the virtual object is obtained through the action start timing calculation process. Note that the details of the action start timing calculation process will be described below. Then, the process proceeds to step ST33.

[0158] At step ST33, the start time comparison unit 401F determines whether the action start timing obtained in step ST32 has exceeded the progress time by comparing the action start timing with the progress time. If the action start timing has not exceeded the progress time, the process proceeds to step ST35. If the action start timing has exceeded the progress time, the process proceeds to step ST34.

[0159] At step ST34, the animation of the target predetermined action of the virtual object is started. This process is executed by the animation control unit 401G. Then, the process proceeds to step ST35.

[0160] At step ST35, the terminal control unit 401 determines whether the entire timing control of the action with an action start timing at or after the progress time has been completed. If the entire timing control of the action has not been completed, the process returns to step ST32. If the entire timing control of the action has been completed, the process proceeds to step ST36.

[0161] At step ST36, the virtual object display timing control process is completed. Subsequently, the process of step ST20 shown in Figure 16 is executed, and the video of the virtual object performing the predetermined action generated by the animation control unit 401G is displayed on the display 407.

[0162] Next, the process of the action start timing calculation process in this embodiment will be described with reference to Figure 18 the flowchart shown in

[0163] At step ST41, the start time of the predetermined action to be performed by the virtual object is set to t seconds (in milliseconds) after the start of reproduction. The value of t is defined by the installed application. Then, the process proceeds to step ST42.

[0164] In step ST42, the position of the virtual object is converted to an absolute position based on the user's position, and this absolute position is a position in the three-dimensional coordinate space associated with the real space. That is, based on the position of the smart display 40 provided by the own position information acquisition unit 405, the relative position of the virtual object with respect to the smart display 40 defined by the application is converted to an absolute position. Then, the process proceeds to step ST43.

[0165] In step ST43, the distance calculation unit 401D calculates the distance D between the absolute position of the virtual object calculated in step ST42 and the position indicated by the sound source position information of the speaker 50. Then, the process proceeds to step ST44.

[0166] In step ST44, the start time calculation unit 401E performs a delay amount setting process for calculating the delay amount based on the above (mathematical expression 2). As described above, in the delay amount setting process, the larger the distance D, the larger the delay amount is set. Then, the process proceeds to step ST45.

[0167] In step ST45, the action start timing is determined by adding the delay amount to the predefined t. Then, the process of step ST33 and subsequent processes shown Figure 17 is executed.

[0168] In addition, processing such as the above-described delay amount setting process is performed for each virtual object or each action sequence of the virtual object.

[0169] According to the present embodiment described above, it is possible to appropriately set the action start timing for causing the virtual object to perform a predetermined action.

[0170] <Modification Example>

[0171] Although one embodiment of the present disclosure has been specifically described above, the present disclosure is not limited to the above embodiment, and various modification examples can be made based on the technical idea of the present disclosure. Hereinafter, modification examples will be described.

[0172] "Modification Example 1"

[0173] Predetermined synchronization information can be set for the content to be reproduced. The synchronization information is, for example, a flag indicating synchronization or non-synchronization. For example, in the case of "synchronization", the logical value "1" is set as the synchronization information, and in the case of "non-synchronization", the logical value "0" is set as the synchronization information. Of course, the logical values can be interchanged. The synchronization information is information that can be set for each action of the virtual object or for each virtual object in the content segment. For example, there is a scenario in which it is desired to display the same video to multiple users at the same timing (e.g., countdown). In this case, "1" is set as the synchronization information.

[0174] The processing indicated by the patterns P11 and P12 shown Figure 19 is the processing in the case where "0", that is, "out of sync", is set as the synchronization information. In the case where "0" is set as the synchronization information, similarly to the above-described one embodiment, the start timing of the action of the virtual object is set based on the delay amount setting process. Figure 19 The processing indicated by the pattern P13 shown

[0175] is the processing in the case where "1", that is, "synchronized", is set as the synchronization information. In the case where "1" is set as the synchronization information, a synchronization process is performed to prevent display delay regardless of the distance between the stage and the virtual object, in other words, to set the start timing of the action of the virtual object for each user to the same predetermined timing. This synchronization process is performed, for example, under the control of the terminal control unit 401. By performing the synchronization process, the video of a predetermined scene can be reproduced for multiple users at the same timing. This timing is the common timing of a predetermined smart display (for example, the smart display held by the user UA) and another smart display (for example, the smart display held by the user UB) operated by the same application.

[0176] "Modification Example 2"

[0177] Predetermined shared information can be set for the content to be reproduced. The shared information is, for example, a flag indicating sharing or non-sharing. For example, in the case of "sharing", "1" as a logical value is set as the shared information, and in the case of "non-sharing", "0" as a logical value is set as the shared information. Of course, the logical values can be interchanged. The shared information is information that can be set for each action of the virtual object or for each virtual object in a content segment.

[0178] In the case where "1" is set as the shared information, the predetermined actions and their positions of the virtual object are shared among the users who hold smart displays operated by the same application. In the case where "0" is set as the shared information, the predetermined actions of the virtual object are shared among the users, but the position of the virtual object varies according to each user. A specific example of the latter is, for example, the case where the flying action of the virtual object is shared among the users, but, for example, for the user UA, the position is in front of the user, and for the user UB, the position is close to the stage 2.

[0179] Refer to Figure 20 the specific example in the case where "1" is set as the shared information. As Figure 20As shown, when the virtual object moves to the position PO8 (a position 150 m from the stage) between user UA and user UB, a video showing the back of the virtual object is presented on the smart display of user UA, and a video showing the front of the virtual object is presented on the smart display of user UB. That is, when "sharing" is set as the predetermined shared information, the virtual object is presented in a mode corresponding to the positional relationship between the position of the smart display and the position of the virtual object in the three-dimensional coordinate space. In addition, when "non-sharing" is set as the predetermined shared information, the virtual object is displayed in a mode corresponding to each information processing device.

[0180] Note that the delay amount in the case shown is set in the same manner as in the case of non-sharing (i.e., the case of the above-described embodiment). Figure 20 Specifically, D is calculated as D = 150. Then, in the case of user UA, the sound of the real sound source reaches user UA after 30 milliseconds, and the virtual object starts to move 440 milliseconds after the sound is generated at the real sound source. In addition, in the case of user UB, the sound of the real sound source reaches user UB after 30 milliseconds, and the virtual object starts to move 440 milliseconds after the sound is generated at the real sound source.

[0181] Synchronization information and sharing information are set at the time of content creation. Both synchronization information and sharing information can be set. After content creation, the synchronization information and sharing information can be variable.

[0182] "Other modification examples"

[0183] In the above-described embodiment, when the virtual object comes to the front of the user from the stage, the action from jumping out of the stage to coming to the front of the user can be presented as a series of animations. In this case, the timing of jumping out of the stage is determined considering the time required for the action, or the speed from jumping out of the stage to coming to the front of the user is appropriately adjusted.

[0184] In addition to the real audio from the real sound source, the user can also listen to the music reproduced from the smart display while wearing open headphones. The experience of hearing sound from a position different from the real sound source can be provided to the user by a sound source located at the position of the virtual object and using 3D sound reproduction.

[0185] The real sound source can use a microphone to collect the sound in the concert venue and reproduce the sound. This embodiment can be applied not only to live events but also to parades and the like for updating virtual objects.

[0186] The configurations, methods, steps, shapes, materials, numerical values, etc. described in the above embodiments and modified examples are merely examples, and configurations, methods, steps, shapes, materials, numerical values, etc. different from the above can be used as needed, or the configurations, methods, steps, shapes, materials, numerical values, etc. described in the above embodiments and modified examples can be replaced with known ones. In addition, the configurations, methods, steps, shapes, materials, numerical values, etc. in the embodiments and modified examples can be combined with each other within the range where there is no technical inconsistency. In addition, the present disclosure can be implemented in any form such as a manufacturing apparatus such as a control method and an electronic device.

[0187] Note that the effects exemplified in this specification are not limited to the interpretation of the content of the present disclosure.

[0188] The present disclosure may also have the following configurations. (1)

[0190] An information processing apparatus, comprising:

[0191] A control unit that displays a virtual object on a three-dimensional coordinate space associated with a real space, and causes the virtual object to perform a predetermined action according to a predetermined sound reproduced from a real sound source provided in the real space;

[0192] Wherein, the control unit performs a delay amount setting process, in which as the positions of the real sound source in the real space and the virtual object on the three-dimensional coordinate space are separated farther from each other, the delay amount of the action start timing of the virtual object performing the predetermined action with respect to the reproduction timing of the real sound source reproducing the predetermined sound increases. (2)

[0194] The information processing apparatus according to (1),

[0195] Wherein, the control unit obtains the position of the virtual object on the three-dimensional coordinate space based on the relative position defined by the position of the virtual object with respect to the information processing apparatus. (3)

[0197] The information processing apparatus according to (2),

[0198] Wherein, the control unit calculates the distance between the position of the real sound source and the position of the virtual object on the three-dimensional coordinate space, and sets the delay amount based on the calculation result. (4)

[0200] The information processing apparatus according to (3),

[0201] Among them, the position of the real sound source provided by the external device in the three-dimensional coordinate space is provided. (5)

[0203] The information processing apparatus according to any one of (2) to (4),

[0204] wherein the position of the information processing apparatus is variable. (6)

[0206] The information processing apparatus according to any one of (2) to (5) further includes a position information acquisition unit that acquires the position of the information processing apparatus. (7)

[0208] The information processing apparatus according to any one of (1) to (6),

[0209] wherein the control unit performs synchronization processing for setting the action start timing to a predetermined timing regardless of the delay amount. (8)

[0211] The information processing apparatus according to (7),

[0212] wherein the control unit determines the action start timing based on the synchronization processing when synchronization is set to predetermined synchronization information, and determines the action start timing based on the delay amount setting processing when desynchronization is set to predetermined synchronization information. (9)

[0214] The information processing apparatus according to (8),

[0215] wherein the synchronization information is set for each of a plurality of virtual objects. (10)

[0217] The information processing apparatus according to (7) or (8),

[0218] wherein the predetermined timing is a timing common to another information processing apparatus operated by the same application. (11)

[0220] The information processing apparatus according to any one of (7) to (10),

[0221] wherein, when performing the synchronization processing, the control unit displays information corresponding to the synchronization processing. (12)

[0223] The information processing apparatus according to any one of (1) to (12),

[0224] Among them, when sharing is set to predetermined shared information, the control unit displays the virtual object corresponding to the positional relationship between the position of the information processing device and the position of the virtual object in the three-dimensional coordinate space, and when not sharing is set to the predetermined shared information, the control unit displays the virtual object corresponding to each of the plurality of information processing devices. (13)

[0226] The information processing device according to (12),

[0227] Among them, the shared information is set for each of the plurality of virtual objects. (14)

[0229] The information processing device according to any one of (1) to (13),

[0230] Among them, the control unit performs the delay amount setting process for each of the plurality of virtual objects to be displayed. (15)

[0232] The information processing device according to (1) to (14),

[0233] Among them, the device displays the virtual object and is configured as a portable wearable display. (16)

[0235] The information processing device according to any one of (1) to (15),

[0236] Among them, the delay amount setting process is performed in each of the plurality of information processing devices operated by the same application. (17)

[0238] An information processing method executed by a control unit, including:

[0239] Displaying a virtual object in a three-dimensional coordinate space associated with the real space, and causing the virtual object to perform a predetermined action according to a predetermined sound reproduced from a real sound source provided in the real space,

[0240] Among them, the control unit performs a delay amount setting process, in which as the position of the real sound source in the real space and the position of the virtual object in the three-dimensional coordinate space are separated farther from each other, the delay amount of the action start timing of the virtual object performing the predetermined action with respect to the reproduction timing of the real sound source reproducing the predetermined sound increases. (18)

[0242] A program for causing a computer to execute an information processing method performed by a control unit, wherein the method includes:

[0243] Displaying a virtual object on a three-dimensional coordinate space associated with a real space, and causing the virtual object to perform a predetermined action according to a predetermined sound reproduced from a real sound source provided in the real space,

[0244] wherein the control unit performs a delay amount setting process in which, as the position of the real sound source in the real space and the position of the virtual object on the three-dimensional coordinate space are separated farther from each other, the delay amount of the action start timing of the virtual object performing the predetermined action with respect to the reproduction timing of the real sound source reproducing the predetermined sound increases.

[0245] List of reference numerals

[0246] 1…Reproduction system

[0247] 2…Stage

[0248] 3, 50…Speaker

[0249] 30…Server

[0250] 40…Smart display

[0251] 401…Control unit

[0252] 401C…Virtual object absolute position calculation unit

[0253] 401D…Distance calculation unit

[0254] 401E....Start time calculation unit

[0255] 405…Self-position information acquisition unit

[0256] C…Virtual object

[0257] UA, UB…User

Claims

1. An information processing apparatus, comprising: A control unit that displays a virtual object on a three-dimensional coordinate space associated with the real space and causes the virtual object to perform a predetermined action according to a predetermined sound reproduced from a real sound source provided in the real space; Wherein, the control unit performs a delay amount setting process in which, as the positions of the real sound source and the virtual object on the three-dimensional coordinate space associated with the real space are separated farther from each other, the delay amount of the action start timing of the virtual object with respect to the reproduction timing of the real sound source reproducing the predetermined sound increases.

2. The information processing apparatus according to claim 1, wherein, The control unit obtains the position of the virtual object on the three-dimensional coordinate space based on the relative position defined by the position of the virtual object with respect to the information processing device.

3. The information processing apparatus according to claim 2, wherein, The control unit calculates the distance between the position of the real sound source and the position of the virtual object on the three-dimensional coordinate space, and sets the delay amount based on the calculation result.

4. The information processing apparatus according to claim 3, wherein, The position of the real sound source on the three-dimensional coordinate space is provided from an external device.

5. The information processing apparatus according to claim 2, wherein, The position of the information processing device is variable.

6. The information processing apparatus according to claim 2, further comprising a position information acquisition unit, and the position information acquisition unit acquires the position of the information processing apparatus.

7. The information processing apparatus according to claim 1, wherein, The control unit performs a synchronization process of setting the action start timing to a predetermined timing regardless of the delay amount.

8. The information processing apparatus according to claim 7, wherein, The control unit determines the action start timing based on the synchronization process when synchronization is set to predetermined synchronization information, and determines the action start timing based on the delay amount setting process when non-synchronization is set to predetermined synchronization information.

9. The information processing apparatus according to claim 8, wherein, The synchronization information is set for each of the plurality of virtual objects.

10. The information processing apparatus according to claim 7, wherein, The predetermined timing is a timing common to another information processing device operated by the same application.

11. The information processing apparatus according to claim 7, wherein, When performing the synchronization process, the control unit displays information corresponding to the synchronization process.

12. The information processing apparatus according to claim 1, wherein, When sharing is set to predetermined sharing information, the control unit displays the virtual object corresponding to the positional relationship between the position of the information processing device and the position of the virtual object on the three-dimensional coordinate space, and when non-sharing is set to predetermined sharing information, the control unit displays the virtual object corresponding to each of the plurality of information processing devices.

13. The information processing apparatus according to claim 12, wherein, The sharing information is set for each of the plurality of virtual objects.

14. The information processing apparatus according to claim 1, wherein, The control unit performs the delay amount setting process for each of the plurality of virtual objects to be displayed.

15. The information processing apparatus according to claim 1, wherein, The device displays the virtual object and is configured as a portable wearable display.

16. The information processing apparatus according to claim 1, wherein, The delay amount setting process is performed in each of the plurality of information processing devices operated by the same application.

17. An information processing method executed by a control unit, comprising: A virtual object is displayed on a three-dimensional coordinate space associated with the real space, and the virtual object is caused to perform a predetermined action according to a predetermined sound reproduced from a real sound source provided in the real space, Among them, the control unit performs a delay amount setting process. In the delay amount setting process, as the position of the real sound source in the three-dimensional coordinate space associated with the real space and the position of the virtual object in the three-dimensional coordinate space are separated further from each other, the delay amount of the action start timing for the virtual object to perform the predetermined action with respect to the reproduction timing of the real sound source reproducing the predetermined sound increases.

18. A computer program product that causes a computer to execute the information processing method executed by the control unit, wherein, The method includes: displaying a virtual object in a three-dimensional coordinate space associated with a real space, and causing the virtual object to perform a predetermined action according to a predetermined sound reproduced from a real sound source provided in the real space, Among them, the control unit performs a delay amount setting process. In the delay amount setting process, as the position of the real sound source in the three-dimensional coordinate space associated with the real space and the position of the virtual object in the three-dimensional coordinate space are separated further from each other, the delay amount of the action start timing for the virtual object to perform the predetermined action with respect to the reproduction timing of the real sound source reproducing the predetermined sound increases.

19. A removable medium having a program recorded thereon, the program causing a computer to execute the information processing method executed by the control unit, wherein, The method includes: displaying a virtual object in a three-dimensional coordinate space associated with a real space, and causing the virtual object to perform a predetermined action according to a predetermined sound reproduced from a real sound source provided in the real space, Among them, the control unit performs a delay amount setting process. In the delay amount setting process, as the position of the real sound source in the three-dimensional coordinate space associated with the real space and the position of the virtual object in the three-dimensional coordinate space are separated further from each other, the delay amount of the action start timing for the virtual object to perform the predetermined action with respect to the reproduction timing of the real sound source reproducing the predetermined sound increases.

Images