Program and information processing device

Abstract

To provide a program and an information processing apparatus that improves the acoustic experience of a user.SOLUTION: A program of an information processing device 2 in a system 1 causes a control unit 10 to function as determination means 102 that determines sound source coordinates, which are the coordinates of a sound source object in a virtual space, on the basis of a first operation of the user, generation means 104 that generates arriving sound information 148 corresponding to arriving sound virtually arriving at observation coordinates in the virtual space, on the basis of the sound source coordinates and outgoing sound information 144 corresponding to outgoing sound virtually emitted from the sound source object, and audio output means 110a that outputs an output sound to the user on the basis of the arriving sound information 148.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a program and an information processing device. [Background technology]

[0002] Conventionally, there are known technologies for providing content that realizes a highly realistic viewing experience. For example, Patent Document 1 discloses a viewing system that provides a viewing user with the experience of viewing content that presents a performer character whose behavior is controlled based on the performer's physical movements. This viewing system acquires motion information indicating the performer's physical movements, controls the behavior of the performer character, acquires information on operation inputs made by the viewing user, and presents, based on information on the determined viewpoint, a rendered image related to binocular stereoscopic content that presents a performance in a three-dimensional space in which the performer character related to the performer is placed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-173870 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the audio appreciation system disclosed in Patent Document 1 cannot sufficiently improve the sound that the user experiences.

[0005] In view of the above problems, an object of the present invention is to improve the sound experienced by a user. [Means for solving the problem]

[0006] A program according to one embodiment of the present invention causes a computer to function as a determination means for determining sound source coordinates, which are the coordinates of a sound source object in a virtual space, based on a first operation by a user; a generation means for generating arriving sound information corresponding to arriving sound that virtually reaches observation coordinates in the virtual space, based on the sound source coordinates and outgoing sound information corresponding to outgoing sound virtually emitted from the sound source object; and an audio output means for outputting output sound to the user based on the arriving sound information.

[0007] A program according to another aspect of the present invention causes a computer to function as: a receiving means for receiving a first operation from a user regarding determining sound source coordinates, which are the coordinates of a sound source object in a virtual space; an acquisition means for acquiring arriving sound information corresponding to arriving sound virtually reaching observation coordinates in the virtual space, wherein the arriving sound information is generated based on the sound source coordinates and outgoing sound information corresponding to outgoing sound virtually emitted from the sound source object; and an audio output means for outputting output sound to the user based on the arriving sound information.

[0008] An information processing device according to another aspect of the present invention includes a determination unit that determines sound source coordinates, which are the coordinates of a sound source object in a virtual space, based on a first operation by a user; a generation unit that generates arriving sound information corresponding to arriving sound that virtually arrives at observation coordinates in the virtual space, based on the sound source coordinates and outgoing sound information corresponding to outgoing sound virtually emitted from the sound source object; and an audio output unit that outputs output sound to the user based on the arriving sound information. [Effects of the Invention]

[0009] According to the present disclosure, the acoustics experienced by the user can be improved. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a functional block diagram of a system according to an embodiment of the present invention. [Figure 2]FIG. 10 is a diagram illustrating an example of an operation of the information processing device according to the present embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of an operation of the information processing device according to the present embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of an operation of the information processing device according to the present embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device according to the present embodiment. [Figure 6] FIG. 10 is a diagram illustrating a modified example of the operation of the information processing device according to the embodiment. [Figure 7] FIG. 10 is a diagram illustrating a modified example of the operation of the information processing device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] A preferred embodiment of the present invention (hereinafter referred to as the present embodiment) will be described with reference to the accompanying drawings. In each drawing, components with the same reference numerals have the same or similar configurations.

[0012] In this invention, "unit," "means," "device," and "system" do not simply mean physical means, but also include cases where the functions of the "unit," "means," "device," and "system" are realized by software. Furthermore, the functions of one "unit," "means," "device," and "system" may be realized by two or more physical means, devices, and software, and the functions of two or more "units," "means," "device," and "system" may be realized by one physical means, device, and software.

[0013] <1. Overview of System 1> The system 1 according to this embodiment is a system that provides a user with an audio listening experience via a virtual space. The virtual space is a space in virtual reality (VR). The virtual reality is a digital environment generated using a computer. The virtual space can also be called a metaverse.

[0014] A user can act in a virtual space as an object (hereinafter referred to as an "avatar") that represents the user in the virtual space. The user can recognize other objects in the virtual space through the avatar.

[0015] The other object may be, for example, a sound source object corresponding to a sound source such as a musical instrument in the virtual space. The user can experience the image of the sound source object and the sound virtually emitted from the sound source object through the avatar. The user's avatar can also be considered an observation object that observes the other object.

[0016] Typically, a user experiences video and audio using a VR device as a terminal device 3. The VR device outputs three-dimensional images (including three-dimensional video) and stereophonic audio to the user. By using the VR device, the user can experience video and audio as if they were inside an avatar. An example of a VR device is a head-mounted display (HDM). Hereinafter, the three-dimensional images and stereophonic audio that the user can experience using the VR device will be referred to as "output images" and "output audio," respectively.

[0017] When a user moves an avatar in a virtual space, the images and sounds experienced by the user change to those related to the coordinates of the destination. For example, when a user moves the avatar in a direction closer to a sound source object, the volume of the sound experienced by the user increases, and the size of the sound source object experienced by the user increases. In addition, for example, when a user directs the avatar's gaze toward a sound source object, the user can experience the sound of the sound source object as if it were resonating from directly in front of the user. In addition, when a user moves the avatar to coordinates farther away from the sound source object, the user can experience the sound emitted from the sound source object as if it were resonating from a distance.

[0018] The output image and output sound are typically determined based on the coordinates and gaze direction of the avatar, and the type and coordinates of the sound source object. Of these, the type and coordinates of the sound source object may be particularly important in order to enhance the sense of realism in the virtual space. For example, when placing a sound source object in a virtual space that recreates a concert hall with multiple seats, the sound source object needs to be placed so that the sound flowing from the sound source object can be observed at a certain level of quality from all seats.

[0019] <2. System 1 Configuration> An example of the configuration of the system 1 will be described with reference to Fig. 1. The system 1 includes an information processing device 2, a terminal device 3, a voice input device 4, and a communication network 5.

[0020] [Information processing device 2] The information processing device 2 includes a control unit 10, a storage unit 14, and a network interface unit 18. The control unit 10, the storage unit 14, and the network interface unit 18 are electrically connected to each other via a bus 16.

[0021] The storage unit 14 stores various information necessary for the operation of the information processing device 2. The stored information includes spatial information 140, object information 142, outgoing voice information 144, propagation information 146, arriving voice information 148, recorded voice information 150, and placement information 152.

[0022] The space information 140 is information relating to the shape, etc. of a virtual space. The shape, etc. of a virtual space includes information relating to the size (i.e., height, width, depth, etc.) of the virtual space. If the virtual space is the inside of a building, the space information 140 includes information relating to the shape, position, color, pattern, etc. of the ceiling, stairs, floor, corridor, and walls of the building. The space information 140 may be, for example, three-dimensional structural information such as CAD (Computer Aided Design) data.

[0023] The information processing device 2 can generate a virtual space by reading the spatial information 140. When the information processing device 2 generates a virtual space based on the spatial information 140 including information on the shape of a concert hall, for example, the user can experience images and sounds as if they were virtually in the concert hall via an avatar.

[0024] The virtual space generated by the spatial information 140 may be a reproduction of a space that actually exists (for example, NHK (registered trademark) Hall, etc.) or a space that existed in the past (for example, the school building of the Japan Music School (now the former Tokyo Music School Concert Hall, etc.)), or it may be created based on the image of a space that does not actually exist (for example, a space from a fantasy world, etc.).

[0025] An example of a space that existed in the past is the space inside or around a feature that existed in the past. The space information 140 for generating such a virtual space can be generated based on, for example, images of the feature that existed in the past, blueprints of the feature, maps of the inside or around the feature, and information on the building materials used when the feature was constructed.

[0026] The object information 142 is information about the shape of an object, etc. The shape of an object includes information about the size, color, and pattern of the object. Although an object does not have a physical entity in real space, a user can recognize the object as if it were physically present through an avatar. The object information 142 may be generated based on three-dimensional structural information such as CAD data.

[0027] Examples of objects are sound source objects, observation objects, and furniture objects. Sound source objects are, for example, musical instrument objects (hereinafter referred to as "musical instrument objects") and singer objects (hereinafter referred to as "singer objects"). An example of an observation object is a user's avatar. Furniture objects are objects other than sound source objects and observation objects, such as installed objects such as seats in a concert hall.

[0028] The observation object and the sound source object may be the same object. For example, if a user makes his / her avatar play a musical instrument object, the avatar is both an observation object and a sound source object.

[0029] The object information 142 further includes information about the coordinates of the object in the virtual space, such as sound source coordinates that are the coordinates of a sound source object, observation coordinates that are the coordinates of an observation object, and fixture coordinates that are the coordinates of a fixture object.

[0030] The information processing device 2 generates a virtual space in which objects are arranged, based on the space information 140 and the object information 142.

[0031] The outgoing voice information 144 is information corresponding to an outgoing voice virtually emitted from a sound source object. The outgoing voice information 144 typically differs depending on the type of sound source object.

[0032] If the sound source object is, for example, a musical instrument object, the outgoing voice information 144 corresponds to the performance of the musical instrument. Hereinafter, the outgoing voice information 144 corresponding to the musical instrument object will be referred to as outgoing performance information 144a.

[0033] If the sound source object is, for example, a singer object, the outgoing voice information 144 is information corresponding to the singing of the singer. Hereinafter, the outgoing voice information 144 corresponding to the singer object will be referred to as outgoing singing information 144b.

[0034] The outgoing voice information 144 is stored in the format of, for example, a voice file (such as an MP3 (MPEG-1 Audio Layer-3) file or a WAV (waveform audio format) file).

[0035] The propagation information 146 is information about the acoustic propagation characteristics between any two coordinates in the virtual space. The acoustic propagation characteristics are parameters that indicate how the transmitted sound sounds at the observation coordinates. The acoustic propagation characteristics include, for example, parameters related to the reverberation and spaciousness caused by sound delay, reflection, diffraction, etc.

[0036] The propagation information 146 may be obtained by computer simulation based on the space information 140 and the object information 142. Alternatively, the propagation information 146 may be obtained by analyzing sounds recorded in the real space that serves as a model for the virtual space (for example, in the case where the virtual space is created based on an existing concert hall, the concert hall).

[0037] The arriving sound information 148 is information corresponding to the arriving sound that virtually arrives at the observation coordinates in the virtual space. The arriving sound information 148 typically differs depending on the type of sound source object, similar to the outgoing sound information 144.

[0038] If the sound source object is, for example, a musical instrument object, the reaching sound information 148 also corresponds to the performance of the musical instrument. Hereinafter, the reaching sound information 148 corresponding to the musical instrument object will be referred to as reaching performance information 148a.

[0039] If the sound source object is, for example, a singer object, the reached voice information 148 corresponds to the singing of the singer. Hereinafter, the reached voice information 148 corresponding to the singer object will be referred to as reached singing information 148b.

[0040] The arrival sound information 148 is stored, for example, in the form of a sound file (such as an MP3 file or a WAV file).

[0041] The recorded audio information 150 is information about audio recorded by a user in real space. The recorded audio information 150 may be audio information in a state where recording has been completed, or audio information in a state where recording is in progress live.

[0042] The recorded audio information 150 is stored in the form of, for example, an audio file (such as an MP3 file or a WAV file).

[0043] The arrangement information 152 is information about the arrangement of sound sources in real space. The arrangement information 152 may be, for example, information about an image showing one or more musical instruments arranged on a stage. The arrangement information 152 may also be, for example, information about a layout diagram of one or more musical instruments.

[0044] The placement information 152 is stored in a format such as an image file (such as a JPEG (Joint Photographic Experts Group) file) and a text file (such as a CSV (Comma Separated Values) file) in which the coordinates of one or more sound sources are associated with each sound source.

[0045] The control unit 10 executes various programs stored in the storage unit 14, thereby functioning as a determination means 102, a generation means 104, a reading means 106, a change means 108, and an output means 110. Each function will be described below.

[0046] -Decision Method 102- The determining means 102 determines the sound source coordinates based on a first operation related to determining the sound source coordinates of the user. The first operation is typically an operation accepted by the terminal device 3 from the user, and the information processing device 2 receives information related to the first operation. The first operation may be an operation of inputting coordinates in a three-dimensional space as values, or an operation of selecting coordinates using a GUI (Graphical User Interface). The sound source coordinates may also be referred to as the sound source position.

[0047] Furthermore, the first operation may include reading the placement information 152, and the determining means 102 may determine the sound source coordinates based on the placement information 152. If the placement information 152 is an image file, the determining means 102 may read the placement of the sound source in the image by performing image recognition processing on the image file, and may determine the sound source coordinates for each of the one or more sound source objects based on the read placement.

[0048] Furthermore, when the sound source objects include a first sound source object and a second sound source object, the determining means 102 may determine first sound source coordinates that are the coordinates of the first sound source object and second sound source coordinates that are the coordinates of the second sound source object based on the first operation. That is, the user can determine the sound source coordinates of each of the multiple sound source objects by the first operation.

[0049] Furthermore, the determining means 102 may further determine the observation coordinates based on a second operation of the user. The second operation is, for example, an operation in which the user moves his / her avatar. The observation coordinates can also be referred to as an observation position.

[0050] - Generation means 104 - The generating means 104 generates the arriving sound information 148 based on the sound source coordinates determined by the determining means 102 and the outgoing sound information 144. At this time, the generating means 104 may generate the arriving sound information 148 by referring to the propagation information 146. The generating means 104 can obtain the arriving sound information 148 by, for example, applying a frequency filter corresponding to the acoustic propagation characteristics to the outgoing sound information 144.

[0051] -Reading method 106- The reading means 106 reads the recorded voice information 150. Typically, the reading means 106 reads the recorded voice information 150 by receiving the recorded voice information 150 from the terminal device 3 or the voice input device 4. In this embodiment, reading information means making the information processable in the control unit 10. The reading means 106 can also be called an acquisition means for acquiring the recorded voice information 150.

[0052] -Method of change 108- The change means 108 changes the outgoing voice information 144 based on a third operation by the user. The third operation is typically an operation related to changing the settings of the sound source object. The settings of the sound source object include, for example, the type of instrument or the like that the sound source object corresponds to, the volume of the outgoing voice, the scale of the outgoing voice, and the rhythm of the outgoing voice.

[0053] The third operation may be a user reading the recorded voice information 150 into the information processing device 2. In this case, the reading means 106 reads the recorded voice information 150, and the changing means 108 changes the outgoing voice information 144 based on the read recorded voice information 150. Specifically, the changing means 108 may change the setting of the sound source object so that the recorded voice is used as the outgoing voice. This allows the user to experience the recorded voice as if it were being emitted from the sound source object.

[0054] -Output means 110- The output means 110 outputs output sound to the user based on the arrival sound information 148 generated by the generation means 104. The output sound is typically output to the user as stereo sound from the terminal device 3, which is a VR device. It can also be said that the output means 110 includes sound output means 110a.

[0055] The output means 110 also outputs to the user an image corresponding to the field of view when the sound source object is viewed from the observation coordinates. The image is typically output to the user as a stereoscopic image or a stereoscopic video from the terminal device 3, which is a VR device. It can also be said that the output means 110 includes image output means 110b.

[0056] As exemplified above, outputting a sound or an image by the output means 110 includes causing another device (for example, the terminal device 3) to output the sound or the image.

[0057] The network interface unit 18 realizes communication with the terminal device 3 and the voice input device 4 via the communication network 5 .

[0058] [Terminal device 3] The terminal device 3 is a terminal operated by a user, and includes a control unit 20, a storage unit (not shown), and a network interface unit (not shown).

[0059] The control unit 20 executes various programs stored in the storage unit of the terminal device 3, thereby functioning as a receiving unit 200, an acquiring unit 202, and an output unit 204. Each function will be described below.

[0060] - Reception method 200 - The receiving means 200 receives from the user a first operation related to determining sound source coordinates. The receiving means 200 also receives a second operation related to determining observation coordinates. The receiving means 200 also receives a third operation related to changing the outgoing voice information 144. Information related to the first operation to the third operation is transmitted to the information processing device 2 via the network interface unit of the terminal device 3.

[0061] -Acquisition method 202- The acquiring means 202 acquires the arriving voice information 148. The acquiring means 202 typically receives the arriving voice information 148 from the information processing device 2 via the network interface unit of the terminal device 3.

[0062] -Output means 204- The output means 204 outputs an output sound to the user based on the arriving sound information 148. It can also be said that the output means 204 includes sound output means 204a. The output means 204 also outputs to the user an image corresponding to the field of view when the sound source object is viewed from the observation coordinates. It can also be said that the output means 204 includes image output means 204b.

[0063] [Voice Input Device 4] The audio input device 4 is a device that records audio in real space and generates recorded audio information 150. The audio input device 4 can also be called an audio interface. The audio input device 4 can transmit the recorded audio information 150 to the information processing device 2 via a network interface unit (not shown).

[0064] <3. Operation of information processing device 2> An example of the operation of the information processing device 2 will be described with reference to Figs. 2 to 4. Fig. 2 is a flowchart showing the operation of the information processing device 2 from determining the sound source coordinates of the sound source objects to outputting the output sound when a musical instrument object 142a and a singer object 142b are placed as sound source objects. Fig. 3 is a conceptual diagram overlooking the virtual space to assist in the explanation of Fig. 2. Fig. 4 is a diagram showing an example of propagation information 146.

[0065] First, the information processing device 2 determines the sound source coordinates of each of the musical instrument object 142a and the singer object 142b and the observation coordinates of the observation object 142c (S10). In this example, the information processing device 2 determines the sound source coordinates p x The instrument object 142a is placed at the sound source coordinate p x The singer object 142b is placed in the observation coordinate q z Assume that the observer 142 places an observation object 142c (i.e., his / her own avatar) at

[0066] 3, the observation object 142c is placed on the opposite side to the instrument object 142a and the singer object 142b. Between the side where the instrument object 142a and the singer object 142b are placed and the side where the observation object 142c is placed, the furniture object 142d and the furniture object 142e are placed.

[0067] Next, the information processing device 2 generates the transmission performance information 144a (S12). In this example, the information processing device 2 generates the transmission performance information 144a by reading out an audio file related to the performance of a musical instrument that has been stored in advance in the storage unit 14.

[0068] Next, the information processing device 2 calculates the sound source coordinates p of the musical instrument object 142a by referring to the propagation information 146. x 4, the propagation information 146 in this example is a sound source coordinate p m(where m is an integer between 1 and M) and the observation coordinate q n (where n is an integer between 1 and N) nm That is, the information processing device 2 associates the sound source coordinates p x and observation coordinate q z The frequency filter f specified by the combination of xz By applying the above, the reached performance information 148a is generated.

[0069] In FIG. 3, the transmitted performance information 144a is filtered by a frequency filter f xz The performance information 148a obtained by applying z The sound source coordinates p x and the observation coordinate q z Since the furniture object 142d is located between xz reflects these effects (e.g., attenuation of sound arriving from the front and increased effects of diffraction).

[0070] In parallel with S12, the information processing device 2 generates the transmitted singing information 144b (S16). In this example, the information processing device 2 generates the transmitted singing information 144b by reading out an audio file related to singing stored in advance in the storage unit 14.

[0071] Next, the information processing device 2 calculates the sound source coordinates p y and the transmitted singing information 144b, the information processing device 2 generates the reached singing information 148b (S18). y and observation coordinate q z The frequency filter f specified by the combination of yz By applying the above, the reached singing information 148b can be generated.

[0072] In FIG. 3, the frequency filter f yzThe singing information 148b obtained by applying z The sound source coordinates p y and the observation coordinate q z The furniture object 142d and the furniture object 142e are not placed between them, but are placed around them. Therefore, the frequency filter f yz The influence of this (for example, a decrease in the sense of sound spread due to the attenuation of sounds arriving from the left and right) is reflected in the

[0073] Next, the information processing device 2 outputs an output sound based on the reached performance information 148a and the reached singing information 148b (S20).

[0074] To summarize the above, a program according to one embodiment of the present invention causes a computer 70 to function as a determination means 102 that determines sound source coordinates, which are the coordinates of a sound source object in a virtual space, based on a first operation by a user; a generation means 104 that generates arriving sound information 148 corresponding to arriving sound that virtually arrives at observation coordinates in the virtual space, based on the sound source coordinates and outgoing sound information 144 corresponding to outgoing sound virtually emitted from the sound source object; and an audio output means 110a that outputs output sound to the user based on the arriving sound information 148.

[0075] <4. Effects of Information Processing Device 2> According to the information processing device 2, it is possible to improve the sound experienced by the user. As described with reference to FIGS. 2 to 4, the user can select the sound source coordinates p x The instrument object 142a is placed at the sound source coordinate p y The user can experience, through the avatar, which is the observation object, how the sound sounds when the singer object 142b is placed in the same place. In this way, if the sound that the user experiences is not what the user intended, the user can easily check the sound source coordinates p x and the sound source coordinates p of the singer object 142b yBy changing the settings, users can experience how sounds sound in different situations. In other words, users can recreate ideal acoustics in a virtual space.

[0076] The observation coordinates may also be determined based on a second operation by the user. The way a sound is heard varies depending on the combination of the sound source coordinates and the observation coordinates. This configuration allows the user to determine not only the sound source coordinates but also the observation coordinates, thereby improving the acoustics experienced by the user in the virtual space.

[0077] Furthermore, the change means 108 may change the outgoing voice information 144 based on a third operation by the user. The way a sound is heard varies depending on the type of sound being emitted, in addition to the combination of the sound source coordinates and the observation coordinates. With this configuration, the user can change not only the sound source coordinates and the observation coordinates, but also the type of sound being virtually emitted from the sound source object. This makes it possible to improve the sound that the user experiences in the virtual space.

[0078] Furthermore, the first operation may include reading placement information 152 regarding the placement of sound sources in real space, and the determining means 102 may determine the sound source coordinates based on the placement information 152. For example, by designing a virtual space that reproduces a real space, the user can virtually experience the real space. In such a case, by reflecting the placement of sound sources in the real space in the virtual space, the user can also virtually experience the sounds of the real space. According to the above configuration, the placement of sound sources in the real space can be easily reproduced in the virtual space, thereby more efficiently improving the sounds experienced by the user.

[0079] Furthermore, the sound source objects may include a first sound source object and a second sound source object, and the determining means 102 may determine first sound source coordinates, which are the coordinates of the first sound source object, and second sound source coordinates, which are the coordinates of the second sound source object, based on the first operation. In real space, multiple sound sources are located at different coordinates, and sounds emitted from each sound source reach the observer via different propagation paths. With the above configuration, it is possible to reproduce such an environment in real space in a virtual space. Furthermore, since the first sound source coordinates and the second sound source coordinates are determined based on the first operation of the user, the user can experience various sound source locations and the associated sounds in the virtual space.

[0080] Furthermore, the virtual space may be a reproduction of the space inside or around a feature that existed in the past. Compared to a virtual space that has never existed in reality and is simply created based on an image of the past, a virtual space that reproduces a feature that actually existed in the past can be perceived as more persuasive by the user. For example, the user's experience of listening to Rentaro Taki's piano in a virtual space that has never existed in reality and is simply created based on an image of the past is thought to be significantly different from that of listening to Rentaro Taki's piano in a virtual space that reproduces the building of the Tokyo Music School, where Rentaro Taki is said to have actually played the piano. In particular, the user can experience greater interest when at least a portion of the sound source object, the transmitted voice information 144, and the propagation information 146 correspond to a reproduced space from the past.

[0081] <5. Hardware Configuration> 5, an example of a hardware configuration in which the above-described information processing device 2 is realized by a computer 70 will be described. Note that the functions of each device can also be realized by dividing them into multiple devices.

[0082] As shown in FIG. 5, the computer 70 includes a processor 700 , a storage device 702 , an input I / F 704 , a data I / F 706 , a communication I / F 708 , and a display device 710 .

[0083] The processor 700 controls various processes in the computer 70 by executing programs stored in the storage device 702. For example, each functional unit included in the control unit 10 of the information processing device 2 can be realized by the processor 700 executing the programs stored in the storage device 702.

[0084] The storage device 702 is a storage medium such as a RAM (Random Access Memory), etc. The RAM temporarily stores the program code of the program executed by the processor 700 and data required when the program is executed.

[0085] The storage device 702 may also be a non-volatile storage medium such as a hard disk drive (HDD) or flash memory. The storage device 702 stores an operating system and various programs for implementing the above-described configurations. The storage medium storing the various programs may be a non-transitory computer-readable medium. The storage device 702 may also store tables that register various pieces of information and a database that manages the tables. Such programs and data are loaded into the storage device 702 as needed and referenced by the processor 700.

[0086] The input I / F 704 is a device for receiving input from a user. Specific examples of the input I / F 704 include a camera, a button, a microphone, a keyboard, a mouse, a touch panel, various sensors, and a wearable device. The input I / F 704 may be connected to the computer 70 via an interface such as a USB (Universal Serial Bus).

[0087] The data I / F 706 is a device for inputting data from outside the computer 70. A specific example of the data I / F 706 is a drive device for reading data stored in various storage media. The data I / F 706 may be provided outside the computer 70. In this case, the data I / F 706 is connected to the computer 70 via an interface such as a USB.

[0088] The communication I / F 708 is a device for performing data communication with devices external to the computer 70 via the communication network 4, either wired or wirelessly. The communication I / F 708 may be provided external to the computer 70. In this case, the communication I / F 708 is connected to the computer 70 via an interface such as a USB.

[0089] The display device 710 is a device for displaying various types of information. Specific examples of the display device 710 include a liquid crystal display, an organic EL (Electro-Luminescence) display, and a display of a wearable device. The display device 710 may be provided outside the computer 70. In this case, the display device 710 is connected to the computer 70 via, for example, a display cable. Furthermore, when a touch panel is used as the input I / F 704, the display device 710 can be configured as an integral part of the input I / F 704.

[0090] The components of the information processing device 2 described in the above embodiment are assumed to implement predetermined processing in cooperation with other hardware when a program stored in the storage device 702 is executed by the processor 700. In other words, these components are assumed to be software or firmware, as well as corresponding hardware, and in both of these concepts, they are also referred to as "functions," "means," "parts," "processing circuits," "units," or "modules," and can be interpreted as such.

[0091] <6. Variations> -Variation 1- In the above embodiment, the information processing device 2 is described as generating the transmission performance information 144a and the transmission singing information 144b by reading out the audio files stored in advance in the storage unit 14 (see S12 and S16), but is not limited to this. For example, the transmission audio information 144 may be generated and changed based on the recorded audio information 150.

[0092] A specific example will be described with reference to the flowchart of Fig. 6. In this example, a first user and a second user simultaneously access the virtual space using their respective terminal devices 3. The first user positions his or her avatar at the sound source coordinates p x The second user positions his / her avatar at the observation coordinate q as the singer object 142b located at z In other words, the second user can experience the feeling as if the avatar of the first user is singing in the virtual space.

[0093] The information processing device 2 calculates the sound source coordinates p x and based on the operation of the second user, the observation coordinates q z The determination is made (S100).

[0094] Next, the information processing device 2 reads the recorded voice information 150 generated by the singing of the first user (S102). The recorded voice information 150 is generated, for example, by the first user recording his / her own singing using the voice input device 4, which is a microphone.

[0095] Next, the information processing device 2 generates or modifies the transmitted singing information 144b based on the recorded voice information 150 (S104).

[0096] Next, the information processing device 2 calculates the sound source coordinates p x and the transmitted singing information 144b generated or modified based on the recorded voice information 150, the observation coordinate q yThe reached singing information 148b is generated (S106).

[0097] Next, the information processing device 2 outputs the output sound based on the reached singing information 148b (S108). Specifically, the information processing device 2 causes the output sound to be output to the terminal device 3 of the second user. This allows the second user to experience the sound recorded by the first user in S102 as if it were coming from the singer object 142b (i.e., the avatar of the first user).

[0098] Next, the information processing device 2 determines whether the recording by the first user has finished (S110). If the recording has finished (S110 YES), the information processing device 2 ends its operation. On the other hand, if the recording is continuing (S110 NO), the information processing device 2 continues to read the recorded voice information 150 generated by the singing of the first user (S102) and performs the subsequent processes.

[0099] In this example, the only sound source object is the avatar of the first user, but this is not limiting. For example, as described in the above embodiment, a musical instrument object 142a may be further placed as a sound source object, allowing the second user to experience both the singing of the avatar of the first user and the performance of the musical instrument object 142a.

[0100] In this example, the singing of the first user is experienced by the second user, but this is not limiting. For example, the first user may have his / her avatar act as the singer object 142b and also act as the observation object 142c that listens to his / her singing.

[0101] According to the configuration of this example, an acoustic experience that utilizes a hybrid of real space and virtual space can be realized. For example, a first user in this example can have a performance in real space reflected in the virtual space, and feel the reverberation and spaciousness of the sound that cannot be felt in the real space. In contrast, a second user can experience a realistic live performance by the first user in the virtual space.

[0102] -Variation 2- In the above embodiment, the description has been mainly focused on the sound experienced by the user, but the user may also be able to experience images (including videos) in the virtual space at the same time. Specifically, the information processing device 2 may output to the user an image corresponding to the field of view when the sound source object is viewed from the observation coordinates.

[0103] An example of an image output to the user will be described with reference to Fig. 7. The arrangement of various objects shown in Fig. 7 corresponds to the arrangement of various objects shown in Fig. 3. Observation coordinate q z The user's avatar, placed at x and the singer object 142b placed at the sound source coordinates p y The visual field of the user's avatar includes a musical instrument object 142a arranged in the center of the sound source object 142. In addition to the sound source object, the visual field of the user's avatar also includes a fixture object 142d and a fixture object 142e. This configuration allows the user to experience sound with a higher sense of realism.

[0104] -Variation 3- In the above embodiment, the acoustic propagation characteristics of the propagation information 146 are m (m=1 M) and the observation coordinate q n (m=1...N). However, in practice, it may be difficult to set (in advance) the acoustic propagation characteristics for all combinations of coordinates. For example, in the propagation information 146, the acoustic propagation characteristics f (x-1)z and acoustic propagation characteristics f (x+1)z is set, but the acoustic propagation characteristics f xzIt is possible that is not set. In such a case, the observation coordinate q z For example, the sound observed at the sound source coordinate p x The acoustic propagation characteristic f (x-1)z The result of applying the same x The acoustic propagation characteristic f (x+1)z It may be generated by averaging the result of applying

[0105] <7. Other embodiments> The following aspects are also included within the scope of the present invention.

[0106] [Appendix 1] A program according to one embodiment of the present invention causes a computer 70 to function as a determination means 102 that determines sound source coordinates, which are the coordinates of a sound source object in a virtual space, based on a first operation by a user; a generation means 104 that generates arriving sound information 148 corresponding to arriving sound that virtually arrives at observation coordinates in the virtual space, based on the sound source coordinates and outgoing sound information 144 corresponding to outgoing sound virtually emitted from the sound source object; and an audio output means 110a that outputs output sound to the user based on the arriving sound information 148.

[0107] [Appendix 2] In the program described in Supplementary Note 1, the observation coordinates may be determined based on a second operation by the user.

[0108] [Appendix 3] The program described in Supplementary Note 1 or Supplementary Note 2 may cause the computer 70 to further function as a change means 108 that changes the outgoing voice information 144 based on a third operation by the user.

[0109] [Appendix 4] In the program described in Appendix 3, the third operation may include reading recorded audio information 150 relating to recorded audio recorded in real space, and the modification means 108 may modify the outgoing audio information 144 based on the recorded audio information 150.

[0110] [Appendix 5] In the program described in any one of Supplementary Notes 1 to 4, the first operation may include reading placement information 152 regarding the placement of the sound source in real space, and the determination means 102 may determine the sound source coordinates based on the placement information 152.

[0111] [Appendix 6] In the program according to any one of Supplementary Note 1 to Supplementary Note 5, the virtual space may be a reproduction of the space inside or around a feature that existed in the past.

[0112] [Appendix 7] In the program described in any one of Supplementary Notes 1 to 6, the sound source object may include a first sound source object and a second sound source object, and the determination means 102 may determine first sound source coordinates, which are the coordinates of the first sound source object, and second sound source coordinates, which are the coordinates of the second sound source object, based on the first operation.

[0113] [Appendix 8] The program described in any one of Supplementary Notes 1 to 7 may further cause the computer 70 to function as an image output means 110b that outputs to the user an image corresponding to the field of view when the sound source object is viewed from the observation coordinates.

[0114] [Appendix 9] In the program according to any one of Supplementary Note 1 to Supplementary Note 8, the generating means 104 may generate the arriving sound information 148 by referring to propagation information 146 relating to acoustic propagation characteristics between any two coordinates in the virtual space.

[0115] [Appendix 10] A program according to another aspect of the present invention causes a computer 70 to function as: a receiving means 200 that receives from a user a first operation related to determining sound source coordinates, which are the coordinates of a sound source object in a virtual space; an acquiring means 202 that acquires arriving sound information 148 corresponding to arriving sound that virtually arrives at observation coordinates in the virtual space, the arriving sound information 148 being generated based on the sound source coordinates and outgoing sound information 144 corresponding to outgoing sound virtually emitted from the sound source object; and an audio output means 204a that outputs an output sound to the user based on the arriving sound information 148.

[0116] [Appendix 11] An information processing device according to another aspect of the present invention includes a determination unit that determines sound source coordinates, which are the coordinates of a sound source object in a virtual space, based on a first operation of a user; a generation unit that generates arriving sound information 148 corresponding to arriving sound that virtually arrives at observation coordinates in the virtual space, based on the sound source coordinates and outgoing sound information 144 corresponding to outgoing sound virtually emitted from the sound source object; and an audio output unit that outputs output sound to the user based on the arriving sound information 148. [Explanation of symbols]

[0117] 2...information processing device, 70...computer, 102...determining means, 104...generating means, 108...changing means, 110...output means, 110a...audio output means, 110b...image output means, 144...outgoing audio information, 146...propagation information, 148...arriving audio information, 150...recorded audio information, 152...location information, 200...receiving means, 202...acquiring means, 204...output means, 204a...audio output means, 204b...image output means

Claims

1. Computer, an acquisition means for acquiring recorded audio information relating to a recorded audio recording of a performance including musical playing and / or singing of a user in real space; a determination means for determining sound source coordinates, which are coordinates of a sound source object that realizes the performance by an avatar in a virtual space of the user, based on a first operation of the user; a generating means for generating arriving sound information corresponding to arriving sound that virtually arrives at observation coordinates including coordinates of the avatar in the virtual space, based on the sound source coordinates and outgoing sound information corresponding to outgoing sound virtually emitted from the sound source object, the outgoing voice information is generated in real time based on the recorded voice information in accordance with the progress of recording of the recorded voice information; generating the received voice information in real time in response to generation of the outgoing voice information; generating means; a voice output unit that outputs an output voice to the user who recorded the recorded voice information based on the received voice information; A program that functions as a

2. The program according to claim 1 , wherein the observation coordinates are determined based on a second operation of the user.

3. The computer further comprises: The program according to claim 1 , further comprising: a change unit that changes the outgoing voice information based on a third operation by the user.

4. the first operation includes reading location information regarding a location of a sound source in real space; The program according to claim 1 , wherein the determining means determines the sound source coordinates based on the location information.

5. The program according to claim 1 , wherein the virtual space is a reproduction of a space inside or around a feature that existed in the past.

6. the sound source objects include a first sound source object and a second sound source object; The program according to claim 1 , wherein the determining means determines first sound source coordinates that are coordinates of the first sound source object and second sound source coordinates that are coordinates of the second sound source object based on the first operation.

7. The computer further comprises:

2. The program according to claim 1, wherein the program functions as image output means for outputting to the user an image corresponding to a field of view when the sound source object is viewed from the observation coordinates.

8. The program according to claim 1 , wherein the generating means generates the sound arrival information by referring to propagation information relating to acoustic propagation characteristics between any two coordinates in the virtual space.

9. Computer, a first acquisition means for acquiring recorded audio information relating to a recorded audio recording of a performance including musical playing and / or singing of a user in a real space; a receiving means for receiving, from the user, a first operation related to determining sound source coordinates, which are coordinates of a sound source object that realizes the performance by an avatar of the user in a virtual space; a second acquisition means for acquiring arriving sound information corresponding to arriving sound virtually arriving at observation coordinates including coordinates of the avatar in the virtual space, the arriving sound information is generated based on the sound source coordinates and outgoing sound information corresponding to outgoing sound virtually emitted from the sound source object; the outgoing voice information is generated in real time based on the recorded voice information in accordance with the progress of recording of the recorded voice information; the arriving voice information is generated in real time in response to generation of the outgoing voice information; A second acquisition means; a voice output unit that outputs an output voice to the user who recorded the recorded voice information based on the received voice information; A program that functions as a

10. An acquisition unit that acquires recorded audio information regarding recorded audio that records a performance, including playing and / or singing, of a user in real space; a determination unit that determines sound source coordinates, which are coordinates of a sound source object that realizes the performance by an avatar in a virtual space of the user, based on a first operation of the user; a generating unit that generates arriving sound information corresponding to arriving sound that virtually arrives at observation coordinates including coordinates of the avatar in the virtual space, based on the sound source coordinates and outgoing sound information corresponding to outgoing sound virtually emitted from the sound source object, the outgoing voice information is generated in real time based on the recorded voice information in accordance with the progress of recording of the recorded voice information; generating the received voice information in real time in response to generation of the outgoing voice information; a generation unit; a voice output unit that outputs an output voice to the user who recorded the recorded voice information based on the received voice information; An information processing device comprising:

Images