Control apparatus, control method and program

Abstract

To enable generating a sound field based on a virtual sound source depending on a position of a listening point even when the listening point moves.SOLUTION: A control apparatus comprises: acquisition means that acquires a position of a listening point; first calculation means that calculates positions of one or more virtual sound sources for the position of the acquired listening point; second calculation means that calculates positions of a plurality of speakers which are arranged around the listening point for the position of the acquired listening point; generation means that generates an output signal for each speaker on the basis of sound source signals generated by the one or more virtual sound sources, and position of the virtual sound source for the position of the listening point and positions of the plurality of speakers for the position of the listening point; and reproduction means that reproduces a sound depending on the output signal in each speaker.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a control device, a control method, and a program. [Background technology]

[0002] There is a system that generates a sound field that immerses a listener at a listening point in a virtual world by rendering sounds emitted by a virtual sound source placed in the virtual world to multiple speakers placed around the listening point. Patent Document 1 discloses a technology that calculates a distribution ratio of a sound signal to be supplied to a speaker for each virtual sound source based on the positional relationship of each speaker with respect to the listening point and a virtual sound source that is in a fixed position with respect to the listening point, and distributes the signal to each speaker with the distribution ratio to generate a sound field. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-44261 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the real world, even if the listening point moves, the sound is always heard from the position of the sound source. We want the sound emitted by a virtual sound source to be heard from the position of the virtual sound source even if the listening point moves. However, when the listening point moves within the listening range surrounded by speakers, the direction of the virtual sound source as seen from the listening point changes, and the positional relationship between the listening point and the speakers also changes. Therefore, if the sound is generated under the same conditions as before the movement, it will become an unnatural sound field, hindering the sense of immersion in the virtual world.

[0005] In the technology of Patent Document 1, even if the speaker arrangement changes, it is possible to generate a constant sound field independent of the speaker arrangement by responding to the change. However, in Patent Document 1, the position of the virtual sound source relative to the listening point is fixed and does not change, and it is not assumed that the listening point will move in the first place. When the listening point moves, in the technology of Patent Document 1, the position of the virtual sound source as seen from the listening point does not change in the generation of the sound field, so it is not possible to realistically reproduce a sound field by a virtual sound source at a certain position according to the movement of the listening point.

[0006] Therefore, an object of the present invention is to make it possible to generate a sound field with a virtual sound source according to the position of the listening point even if the listening point moves. [Means for solving the problem]

[0007] The control device according to the present invention is characterized in having an acquisition means for acquiring a position of a listening point, a first calculation means for calculating the position of one or more virtual sound sources relative to the acquired position of the listening point, a second calculation means for calculating the positions of a plurality of speakers arranged around the listening point relative to the acquired position of the listening point, a generation means for generating an output signal to each speaker based on a sound source signal emitted by the one or more virtual sound sources, the position of the virtual sound source relative to the position of the listening point and the positions of the plurality of speakers relative to the position of the listening point, and a reproduction means for causing each speaker to reproduce a sound corresponding to the output signal. Effect of the Invention

[0008] According to the present invention, even if the listening point moves, a sound field can be generated by a virtual sound source according to the position of the listening point. [Brief description of the drawings]

[0009] [Figure 1] FIG. 13 is a diagram illustrating an example of movement of a listening point. [Diagram 2] FIG. 2 is a diagram illustrating an example of a functional configuration of a control device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a control device. [Figure 4] 10 is a diagram illustrating an example of a data structure of virtual sound source information. [Diagram 5] FIG. 4 is a diagram illustrating an example of a data configuration of speaker information. [Figure 6] 11 is a diagram illustrating an example of a data configuration of speaker group information. [Figure 7] 10 is a flowchart illustrating an example of a sound field generation process in the control device. [Figure 8] 13 is a flowchart illustrating an example of a calculation process of relative virtual sound source coordinates. [Figure 9] 13 is a flowchart illustrating an example of a sound source rendering process. [Figure 10] FIG. 2 is a diagram illustrating an example of a functional configuration of a control device. [Figure 11] 10 is a flowchart showing an example of a sound field / image generation process in the control device. [Figure 12] 13 is a flowchart illustrating an example of a sound source rendering process. [Figure 13] 10 is a diagram illustrating an example of a data structure of virtual sound source information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment described below does not limit the present invention, and not all of the combinations of features described in the present embodiment are necessarily essential configurations. Note that the same components will be described with the same reference numerals.

[0011] A virtual sound source and its sound image when the listening point moves will be described below. Fig. 1(a) to Fig. 1(c) are diagrams for explaining an example of the movement of the listening point, and show a schematic diagram of the relationship between the listening point, the virtual sound source, the sound image of the virtual sound source, and the speaker arrangement. In Fig. 1(a) to Fig. 1(c), 101 is a virtual sound source, 102 is a listening point, 103 is a sound image of the virtual sound source, 104, 105, and 106 are speakers, and 108 is a listening range. An image of the distribution of the signal of the virtual sound source 101 to the speakers is shown by arrow 107.

[0012] 1(a) shows an example of the positional relationship between the listening point 102, the virtual sound source, and the sound image of the virtual sound source when the listening point 102 is located at the center of the listening range 108. In this case, by appropriately distributing the signal of the virtual sound source 101 to the speakers 105 and 106 as shown by the arrow 107, it is possible to present the sound image 103 felt at the listening point 102 at the position where the virtual sound source 101 is located.

[0013] Suppose that listening point 102 moves within listening range 108 from the state shown in Fig. 1(a) to the state shown in Fig. 1(b). In this case, if the distribution of the signal of virtual sound source 101 to each speaker is left unchanged, that is, distributed in the same way as in the example shown in Fig. 1(a), the sound image 103 felt at listening point 102 will shift from the position of virtual sound source 101, as shown in Fig. 1(b).

[0014] In each embodiment described below, the distribution of the signal of the virtual sound source 101 to each speaker is appropriately changed according to the movement of the listening point 102 so that the sound image 103 felt at the listening point 102 coincides with the position of the virtual sound source 101. For example, as shown in FIG. 1(c), according to the movement of the listening point 102, the signal of the virtual sound source 101 is appropriately changed so as to be distributed to the speakers 104 and 105 as indicated by the arrow 107 so that the sound image 103 can be presented at the position of the virtual sound source 101 even if the listening point 102 moves. For convenience of explanation, an example in which the speakers are arranged two-dimensionally is shown in FIG. 1(a) to FIG. 1(c), but each embodiment described below is applicable not only to the case where the speakers are arranged two-dimensionally but also to the case where the speakers are arranged three-dimensionally.

[0015] (Embodiment 1) In the first embodiment, an example will be described in which a sound field is generated by appropriately distributing a sound source signal emitted by a virtual sound source placed at an arbitrary position in a virtual space to speakers according to the movement of a listening point. In this embodiment, for the sake of simplicity, it will be described assuming that the position of a sound source (virtual sound source) in a virtual space is known in advance and that a sound source signal emitted from the sound source can be obtained.

[0016] 2 is a block diagram showing an example of the functional configuration of the control device in embodiment 1. The control device in this embodiment has a listening point acquisition unit 201, a virtual sound source information database (virtual sound source information DB) 202, a first coordinate calculation unit 203, a speaker information database (speaker information DB) 204, and a second coordinate calculation unit 205. The control device in this embodiment also has a sound source rendering unit 206 and a sound reproduction unit 207.

[0017] The listening point acquisition unit 201 acquires the position of the listening point by analyzing a signal from a position sensor (not shown) for detecting the position. The position sensor is, for example, an infrared sensor that directly detects the position of the listener (listening point) by infrared rays, or a sensor that detects the position by short-range wireless communication by attaching a wireless tag to the target listening point. Note that the position sensor is not limited to these, and various sensors can be used within the scope of the gist of this disclosure.

[0018] The virtual sound source information DB202 stores virtual sound source information of each virtual sound source with respect to one or more virtual sound sources arranged in a virtual space. The virtual sound source information includes, for example, information such as virtual sound source coordinates indicating the position where the virtual sound source is arranged. The details of the virtual sound source information will be described later. The first coordinate calculation unit 203 calculates the relative coordinates (virtual sound source relative coordinates) of each virtual sound source with respect to the position of the listening point based on the position of the listening point acquired by the listening point acquisition unit 201 and the virtual sound source coordinates received from the virtual sound source information DB202.

[0019] The speaker information DB 204 stores speaker information for each of the multiple speakers that make up the speaker set 208. The speaker information includes, for example, information such as real-world speaker coordinates that indicate the positions at which the speakers are arranged. Details of the speaker information will be described later. The second coordinate calculation unit 205 calculates the relative coordinates of each speaker with respect to the position of the listening point (speaker relative coordinates) based on the position of the listening point acquired by the listening point acquisition unit 201 and the speaker coordinates received from the speaker information DB 204.

[0020] The sound source rendering unit 206 performs a process of distributing the sound source signal to appropriately selected speakers by performing a panning process on the sound source signal emitted by the virtual sound source based on the position of the virtual sound source relative to the position of the listening point and the position of the speaker relative to the position of the listening point. The sound source rendering unit 206 selects a speaker to which the sound source signal emitted by the virtual sound source is to be distributed from among the multiple speakers constituting the speaker set 208 based on the virtual sound source relative coordinates and the speaker relative coordinates, and performs a process of distributing the sound source signal to the selected speaker. The sound source rendering unit 206 performs this process of distributing the sound source signal for each virtual sound source, and generates an output signal for each speaker constituting the speaker set 208. The sound source rendering unit 206 is an example of a generating means.

[0021] The sound source rendering unit 206 includes a panning calculation unit 211 , a speaker set creation unit 212 , an inverse matrix calculation unit 213 , and a signal correction unit 214 . The panning calculation unit 211 distributes the input sound source signal of the virtual sound source to appropriately selected speakers, and generates an output signal for each speaker. In this embodiment, the panning calculation unit 211 performs panning calculation based on a known vector based amplitude panning (VBAP) method as an example. In the VBAP method, the panning calculation unit 211 first calculates the product of a direction vector of a direction from a listening point toward the position of a virtual sound source and an inverse matrix obtained from the inverse matrix calculation unit 213. Next, the panning calculation unit 211 determines a speaker set to which the input sound source signal of the virtual sound source is to be distributed and a gain for each speaker according to the calculation result, and generates an output signal for each speaker by accumulating the sound source signal of the virtual sound source based on the result.

[0022] The speaker group creation unit 212 creates adjacent speaker pairs or triplet pairs for all speakers based on the speaker relative coordinates. These speaker groups are also created according to the VBAP method. That is, the speaker group creation unit 212 creates speaker group pairs when the speakers are arranged two-dimensionally, i.e., on a plane, and creates speaker group triplets when the speakers are arranged three-dimensionally, i.e., three-dimensionally.

[0023] For example, when creating a pair of speaker pairs, adjacent speakers are connected by straight lines on the plane on which the speakers are arranged, forming a ring surrounding the listening point. The speaker pair creation unit 212 creates a speaker pair by pairing two speakers connected by straight lines. For example, when creating a pair of speaker pairs, the speaker pair creation unit 212 creates a speaker pair by pairing three speakers that are adjacent to each other when viewed from the listening point in a listening space surrounded by the speakers. At this time, the speaker pair creation unit 212 creates the speaker pair of three such that there is no overlap between the planes formed by each speaker pair, and the surface of the space surrounded by the speakers as viewed from the listening point is filled with all of the planes formed by each speaker pair.

[0024] The inverse matrix calculation unit 213 creates a matrix in which directional vectors from the listening point to each speaker are arranged for each speaker pair created by the speaker pair creation unit 212, and further calculates the inverse matrix. The signal correction unit 214 calculates the amount of correction for sound pressure and delay for the output signal to each speaker output from the panning calculation unit 211 based on the distance between the listening point and the speaker, and appropriately corrects the output signal to each speaker.

[0025] The sound reproducing unit 207 appropriately amplifies the output signal to each speaker output from the sound source rendering unit 206, and outputs it to each speaker constituting the speaker set 208. The speaker set 208 is composed of a plurality of speakers 210 arranged around the listening point 209 so as to surround the listening point 209. These speakers convert the output signal amplified by the sound reproducing unit 207 into sound, output and reproduce it, and generate a sound field suitable for the position of the listening point 209. In this embodiment, the listening point 209 can move freely around and move within a space (listening space) surrounded by the plurality of speakers 210 constituting the speaker set 208. Note that, although FIG. 2 shows an example in which a plurality of speakers are arranged three-dimensionally as the speaker set 208, the present invention is not limited thereto, and the speaker set 208 may be one in which a plurality of speakers are arranged two-dimensionally.

[0026] 3 is a block diagram showing an example of a hardware configuration of the control device in this embodiment. The control device in this embodiment has an input / output unit 301, a CPU 302, a RAM 303, an external storage unit 304, an operation unit 305, a display unit 306, a ROM 307, a communication IF unit 308, and a bus 309. The input / output unit 301, the CPU 302, the RAM 303, the external storage unit 304, the operation unit 305, the display unit 306, the ROM 307, and the communication IF unit 308 are connected to each other via the bus 309 so as to be able to communicate with each other.

[0027] The input / output unit 301 receives input of a position sensor signal and a sound source signal of a virtual sound source from the outside, and transmits them to other components via a bus 309 according to an instruction from the CPU 302 as appropriate. The input / output unit 301 also transmits the generated output signal to the speaker stored in the RAM 303 or the external storage unit 304 to the outside via the bus 309 according to an instruction from the CPU 302 as appropriate. The CPU (Central Processing Unit) 302 comprehensively controls each component of the control device. The CPU 302 controls other components by sending control signals via the bus 309 according to a program, and performs various calculations. In this embodiment, the CPU 302 executes each function described in FIG. 2 by executing processing according to a program stored in the ROM 307 or the external storage unit 304. The RAM (Random Access Memory) 303 temporarily stores a part of the program being executed, associated data, calculation results of the CPU 302, and the like. The CPU 302 loads necessary programs and data into the RAM 303 and executes the programs by reading and writing as required.

[0028] The external storage unit 304 stores the program body and data to be accumulated for a long period of time. For example, the functions of the virtual sound source information DB 202 and the speaker information DB 204 are realized by the external storage unit 304. The external storage unit 304 is, for example, an HDD (hard disk drive) or an SSD (solid state drive). The operation unit 305 accepts various instructions and operations from the user, converts them into control signals, and transmits them to the CPU 302 via the bus 309. The CPU 302 performs control of the running program and control instructions for other configurations according to the control signals. The display unit 306 displays the status of the running program and the output of the program to the user. The ROM (Read Only Memory) 307 stores fixed programs and fixed parameters, such as programs for starting and ending this hardware device and programs for controlling basic input and output. The communication IF unit 308 can input and output data to and from a communication network such as the Internet.

[0029] The various data configurations in this embodiment will be described below. The data structure of the virtual sound source information will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of the data structure of the virtual sound source information. As shown in FIG. 4, the virtual sound source information includes a virtual sound source ID 401, a sound source signal channel 402, a virtual sound source coordinate 403, and a virtual sound source relative coordinate 404. The virtual sound source ID 401 is an ID number for identifying this virtual sound source information, and is uniquely assigned to each virtual sound source information, i.e., each virtual sound source. The sound source signal channel 402 is the number of the channel of the sound source signal linked to this virtual sound source among the channels of the sound source signal input to the control device. The virtual sound source coordinate 403 is a coordinate indicating the position of the virtual sound source, and is indicated by a three-dimensional coordinate in the virtual world targeted for sound field generation. Note that, for convenience of explanation, in this embodiment, the position of the virtual sound source is fixed, and the virtual sound source coordinate 403 does not change at the same coordinate regardless of time. However, this is not limited to this, and can also be applied to a case where the position of the virtual sound source changes with time. The virtual sound source relative coordinates 404 are coordinates indicating the position of the virtual sound source as seen from the listening point, and are calculated in the sound field generation process and stored in this field.

[0030] The data structure of the speaker information will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the data structure of the speaker information. As shown in Fig. 5, the speaker information includes a speaker ID 501, speaker coordinates 502, and speaker relative coordinates 503. The speaker ID 501 is an ID number for identifying this speaker information, and is uniquely assigned to each piece of speaker information, i.e., each speaker. The speaker coordinates 502 are coordinates indicating the position of the speaker, and are shown in three-dimensional coordinates in the real world. The speaker relative coordinates 503 are coordinates indicating the position of the speaker as viewed from the listening point, and are calculated in the sound field generation process and stored in this field.

[0031] The data structure of the speaker group information will be described with reference to FIG. 6. FIG. 6 is a diagram showing an example of the data structure of the speaker group information. FIG. 6 shows an example of the speaker group information when the speaker arrangement is three-dimensional. As shown in FIG. 6, the speaker group information includes a speaker group ID 601, a number of dimensions 602, speaker IDs 603, 604, and 605, and a direction vector inverse matrix 606. The speaker group ID 601 is an ID number for identifying this speaker group information, and is uniquely assigned to each piece of speaker group information. The number of dimensions 602 stores information indicating whether the speaker arrangement is two-dimensional or three-dimensional. For example, the number of dimensions 602 stores a numerical value indicating whether the speaker arrangement is two-dimensional or three-dimensional, and indicates the number of speaker IDs stored with this numerical value. The speaker IDs 603 to 605 are speaker IDs of three speakers that constitute this speaker group. The direction vector inverse matrix 606 is an inverse matrix of a direction vector matrix created by the direction vectors when the three speakers constituting this speaker pair are viewed from the listening point, and is calculated by the inverse matrix calculation unit 213 and stored in this field. Note that not only the direction vector inverse matrix but also the direction vector matrix may be stored in the speaker pair information.

[0032] The sound field generation process by the control device in this embodiment will be described below. Fig. 7 is a flowchart showing an example of the sound field generation process by the control device in this embodiment. In step S701, the listening point acquisition unit 201 receives and analyzes a signal from a position sensor (not shown) to acquire the position of the listening point in the coordinate system of the real world. The listening point acquisition unit 201 outputs listening point coordinates indicating the acquired position of the listening point to the first coordinate calculation unit 203 and the second coordinate calculation unit 205.

[0033] In step S702, second coordinate calculation unit 205 acquires speaker information from speaker information DB 204, and calculates the relative coordinates of each speaker seen from the listening point (speaker relative coordinates) for each speaker constituting speaker set 208. Second coordinate calculation unit 205 calculates the speaker relative coordinates based on the listening point coordinates output from listening point acquisition unit 201 and the speaker coordinates stored in the speaker information acquired from speaker information DB 204. In this embodiment, the speaker relative coordinates are indicated by the coordinates of each speaker in a three-dimensional coordinate system that has the listening point position as the center (reference, origin) and the directions of each axis are the same as those of the coordinate system in the real world.

[0034] The speaker relative coordinates are calculated by simply subtracting the listening point coordinates from the speaker coordinates for each axis of the real-world coordinate system. For example, the speaker coordinates in the real-world coordinate system are (xsp, ysp, zsp), and the listening point coordinates in the real-world coordinate system are (xl, yl, zl). In this case, the relative coordinates (x'sp, y'sp, z'sp) of the speaker are (xsp-xl, ysp-yl, zsp-zl). The relative coordinates of each speaker calculated in this way are stored as speaker relative coordinates 503 in the speaker information, and are output to the sound source rendering unit 206 together with the speaker information.

[0035] In step S703, the first coordinate calculation unit 203 acquires virtual sound source information from the virtual sound source information DB 202, performs a calculation process of the virtual sound source relative coordinates, and calculates the relative coordinates of the virtual sound source seen from the listening point (virtual sound source relative coordinates). The first coordinate calculation unit 203 performs a calculation process of the virtual sound source relative coordinates for each virtual sound source arranged in the virtual world. The first coordinate calculation unit 203 calculates the virtual sound source relative coordinates based on the listening point coordinates in the coordinate system of the real world output from the listening point acquisition unit 201 and the virtual sound source coordinates in the coordinate system of the virtual world stored in the virtual sound source information acquired from the virtual sound source information DB 202. Here, the virtual sound source relative coordinates in this embodiment are coordinates in a three-dimensional coordinate system in which the position of the listening point is the center (reference, origin) and the directions of each axis are the same as those of the coordinate system of the real world, similar to the speaker relative coordinates. Details of the calculation process of the virtual sound source relative coordinates performed in step S703 will be described later with reference to FIG. 8. Then, the first coordinate calculation unit 203 outputs the virtual sound source information of each virtual sound source including the calculated virtual sound source relative coordinates to the sound source rendering unit 206.

[0036] In step S704, the sound source rendering unit 206 performs sound source rendering processing, and renders sound source signals of virtual sound sources for the multiple speakers that make up the speaker set 208 to generate a sound field. The sound source rendering unit 206 generates output signals for each speaker based on the speaker relative coordinates calculated in step S702 and the virtual sound source relative coordinates calculated in step S703, thereby generating a sound field. The sound source rendering unit 206 then outputs the generated output signals for each speaker to the sound reproducing unit 207. Details of the sound source rendering processing performed in step S704 will be described later with reference to FIG. 9.

[0037] In step S705, the sound reproducing unit 207 performs appropriate tuning and amplification on the output signals to each speaker output from the sound source rendering unit 206, and outputs an electrical signal to each speaker constituting the speaker set 208. Each speaker of the speaker set 208 converts the input electrical signal into sound and outputs it. As a result, a sound image of each virtual sound source felt at the listening point is generated at the correct position corresponding to the position where the virtual sound source is arranged. Therefore, by generating the sound image of one or more virtual sound sources at the correct position as seen from the listening point, a sound field appropriate for the position of the listening point at that time is generated.

[0038] In step S706, the control device determines whether or not to end the sound field generation process, based on the user's operation of the operation unit 305, etc. If the control device determines not to end the sound field generation process (NO in step S706), the sound field generation process continues, and the process returns to step S701 to acquire the position of the listening point, and the sound field generation process for the next listening point position is executed. In this way, the control device acquires the position of the listening point for each processing unit time in which a series of processes from steps S701 to S705 are performed, and generates a sound field by a virtual sound source according to the acquired position of the listening point. If the control device determines to end the sound field generation process (YES in step S706), the sound field generation process is terminated.

[0039] Fig. 8 is a flowchart showing an example of the calculation process of the virtual sound source relative coordinates in step S703 in Fig. 7. All the processes in the calculation process of the virtual sound source relative coordinates shown in Fig. 8 are executed in the first coordinate calculation unit 203.

[0040] In the flowchart shown in FIG. 8, the processes from step S801 to step S805 are executed as a loop process for all virtual sound sources to be placed. In step S801, the first coordinate calculation unit 203 selects one virtual sound source to be processed from one or more virtual sound sources placed in the virtual world. In step S802, the first coordinate calculation unit 203 searches the virtual sound source information DB 202 using the virtual sound source ID of the virtual sound source selected in step S801, and acquires the virtual sound source coordinates of the virtual sound source to be processed.

[0041] In step S803, the first coordinate calculation unit 203 converts the virtual sound source coordinates described in the virtual world coordinate system acquired in step S802 into coordinates in the real world coordinate system. The conversion from the virtual sound source coordinates in the virtual world coordinate system to the virtual sound source coordinates in the real world coordinate system can be performed, for example, by a general three-dimensional coordinate conversion calculation. For example, the virtual sound source coordinates in the virtual world coordinate system of the virtual sound source to be processed are (xs, ys, zs). When the virtual world coordinates are rotated in the order of γ around the Z axis, β around the Y axis, and α around the X axis, the directions of each axis match. The difference between the origins of each coordinate system at that time is shifted by x, y, and z on each axis in the real world coordinate system. In this case, the virtual sound source coordinates (xsa, ysa, zsa) in the real world coordinate system are calculated by the following formula.

[0042]

number

[0043] In step S804, the first coordinate calculation unit 203 calculates the relative coordinates of the virtual sound source seen from the listening point (virtual sound source relative coordinates). The first coordinate calculation unit 203 calculates the virtual sound source relative coordinates based on the virtual sound source coordinates in the real world coordinate system obtained in step S803 and the listening point coordinates in the real world coordinate system obtained in step S701 of FIG. 7. The virtual sound source relative coordinates can be obtained by simply subtracting the listening point coordinates from the virtual sound source coordinates for each axis of the real world coordinate system, similar to the speaker relative coordinates. For example, if the listening point coordinates in the real world coordinate system are (xl, yl, zl), the relative coordinates (x's, y's, z's) of the virtual sound source to be processed are (xsa-xl, ysa-yl, zsa-zl). The virtual sound source relative coordinates calculated in this way are stored as the virtual sound source relative coordinates 404 in the virtual sound source information of the virtual sound source to be processed.

[0044] In step S805, the first coordinate calculation unit 203 judges whether or not the loop process for all virtual sound sources has been completed. If the first coordinate calculation unit 203 judges that there is a virtual sound source that has not been processed (NO in step S805), the process returns to step S801 to select one virtual sound source from among the unprocessed virtual sound sources, and the loop process for the next virtual sound source is executed. On the other hand, if the first coordinate calculation unit 203 judges that the loop process for all virtual sound sources has been completed (YES in step S805), the process of step S806 is executed.

[0045] In step S806, the first coordinate calculation unit 203 outputs the virtual sound source information of all virtual sound sources including the virtual sound source relative coordinates stored in the RAM 103 to the sound source rendering unit 206. When the process of step S806 is completed, the calculation process of the virtual sound source relative coordinates is terminated, and the process returns to the sound field generation process shown in FIG.

[0046] Fig. 9 is a flowchart showing an example of the sound source rendering process in step S704 in Fig. 7. All the processes in the sound source rendering process shown in Fig. 9 are executed in the sound source rendering unit 206.

[0047] In step S901, the speaker set creation unit 212 of the sound source rendering unit 206 determines whether the speaker arrangement in the speaker set 208 is two-dimensional or three-dimensional. The speaker set creation unit 212 determines whether the speaker arrangement is two-dimensional or three-dimensional, for example, by using the speaker relative coordinates output from the second coordinate calculation unit 205 in step S702 of Fig. 7.

[0048] In step S902, the speaker group creation unit 212 creates speaker groups based on the determination made in step S901. If the speaker group creation unit 212 determines that the speaker arrangement is two-dimensional, it creates a pair of speaker groups, and if the speaker arrangement is three-dimensional, it creates a pair of speaker groups. For each of the speaker groups created in this manner, the speaker group creation unit 212 generates speaker group information such as that shown in FIG. 6 that stores the speaker IDs of each speaker that constitutes the speaker group. Then, the speaker group creation unit 212 outputs the speaker group information and the relative coordinates of each speaker to the inverse matrix calculation unit 213.

[0049] The processes from step S903 to step S906 are executed as a loop process for all speaker pairs created in step S902. The processes from step S903 to S906 are executed by the inverse matrix calculation unit 213 of the sound source rendering unit 206.

[0050] In step S903, the inverse matrix calculation unit 213 of the sound source rendering unit 206 selects one speaker pair to be processed from among all the speaker pairs created in step S902. The inverse matrix calculation unit 213 selects one speaker pair to be processed in ID order based on the speaker pair IDs stored in the speaker pair information, for example.

[0051] In step S904, the inverse matrix calculation unit 213 creates a direction vector matrix for the speaker pair selected in step S903. The inverse matrix calculation unit 213 first calculates, from the speaker relative coordinates of each speaker constituting the speaker pair, a unit vector from the listening point toward the speaker relative coordinates, i.e., a direction vector indicating the direction of the speaker as seen from the listening point. This can be obtained by multiplying each coordinate value of the speaker relative coordinates by the same magnification so that the magnitude of the vector becomes 1. For example, when the speakers are arranged in three dimensions, the direction vector v for a speaker whose speaker relative coordinates are (x, y, z) is calculated by the following formula.

[0052]

number

[0053] Next, the inverse matrix calculation unit 213 creates a direction vector matrix by arranging these direction vectors as rows. Therefore, in the case of a two-dimensional speaker arrangement, a direction vector matrix is ​​created as a 2×2 square matrix, and in the case of a three-dimensional speaker arrangement, a direction vector matrix is ​​created as a 3×3 square matrix.

[0054] In step S905, the inverse matrix calculation unit 213 calculates the inverse matrix of the direction vector matrix created in step S904. The obtained inverse matrix is ​​stored in the speaker pair information of the speaker pair to be processed.

[0055] In step S906, the inverse matrix calculation unit 213 determines whether or not the loop processing for all speaker sets has been completed. If the inverse matrix calculation unit 213 determines that there is a speaker set that has not been processed (NO in step S906), the process returns to step S903 to select one speaker set from the unprocessed speaker sets, and processing is performed for the next speaker set. On the other hand, if the inverse matrix calculation unit 213 determines that the loop processing for all speaker sets has been completed (YES in step S906), the speaker set information for all speaker sets is output to the panning calculation unit 211, and the process of step S907 is executed.

[0056] The processes from the next steps S907 to S916 are executed as a loop process for all the virtual sound source information output from the first coordinate calculation unit 203 in step S703 in Fig. 7, i.e., for all the virtual sound sources to be placed. The processes from steps S907 to S916 are executed in the panning calculation unit 211 of the sound source rendering unit 206.

[0057] In step S907, the panning calculation unit 211 of the sound source rendering unit 206 selects one piece of virtual sound source information to be processed from the one or more pieces of virtual sound source information output from the first coordinate calculation unit 203 in step S703 of FIG.

[0058] In step S908, the panning calculation unit 211 uses the virtual sound source relative coordinates stored in the virtual sound source information selected in step S907 to create a direction vector of the virtual sound source, which is a unit vector pointing from the listening point to the virtual sound source relative coordinates.

[0059] In step S909, the panning calculation unit 211 calculates the product of the direction vector of the virtual sound source created in step S908 and the inverse matrix of the direction vector matrix stored in all the speaker pair information. As a result of this calculation, if the speaker arrangement is two-dimensional, two values ​​are obtained, and if the speaker arrangement is three-dimensional, three values ​​are obtained. The following describes the case where the speaker arrangement is three-dimensional. Note that, even if the speaker arrangement is two-dimensional, the same processing can be performed, except that the number of target values ​​is reduced from three to two.

[0060] In step S910, the panning calculation unit 211 determines the speaker group that outputs the sound source signal and the gain for each speaker that constitutes the speaker group, based on the result obtained in step S909.

[0061] The panning calculation unit 211 determines the speaker pair that outputs the sound source signal, for example, as follows. First, the panning calculation unit 211 excludes speaker pairs that include negative values ​​in the value obtained as a result of calculation in step S909 from the selection candidates. Next, the panning calculation unit 211 calculates the difference between each of the three values ​​for each speaker pair that remains as a selection candidate, and selects the speaker pair with the largest difference. Through this selection, the pair of speakers that distributes the sound source signal associated with the virtual sound source to be processed is determined.

[0062] Furthermore, the panning calculation unit 211 determines the gain for each speaker, for example, as follows: The panning calculation unit 211 calculates the root mean square of the three values ​​calculated for the speaker group selected as the speaker group that outputs the sound source signal, that is, the square root of the sum of the squares. Since this is the total energy, the gain of the signal to be distributed to each speaker of the speaker group is determined by dividing the three values ​​by the root mean square.

[0063] In step S911, the panning calculation unit 211 calculates the distance between the listening point and the virtual sound source, using the virtual sound source relative coordinates stored in the virtual sound source information selected in step S907.

[0064] In step S912, the panning calculation unit 211 calculates the distance attenuation of sound pressure according to the distance between the listening point and the virtual sound source calculated in step S911, and adds the calculated distance attenuation to the gain for each speaker calculated in step S910. For example, assuming that the reference distance is ro [m], the distance between the listening point and the virtual sound source is r1 [m], the virtual space in which the sound field is generated is a free space, and the virtual sound source is a point sound source, the distance attenuation D can be calculated by the following formula. D=20log(ro / r1) [dB] …(Formula 3)

[0065] In step S913, the panning calculation unit 211 reads sound source signals for a processing unit time corresponding to the time for performing a series of processes in steps S701 to S705 in the sound field generation process of Fig. 7 into a specified area on the RAM 303. The sound source signals are signals of a channel specified by the sound source channel of the virtual sound source information to be processed selected in step S907. This sound source signal may be a signal stored in a file, and in this case, the previous file read position may be stored in the RAM 303, and the signal for the processing unit time may be read from that position. Also, sound source signals input in real time may be read for the processing unit time immediately after the time when they were previously read.

[0066] In step S914, the panning calculation unit 211 generates an output signal to be output to each speaker by individually superimposing the gain for each speaker calculated in step S912 on the sound source signal read in step S913.

[0067] In step S915, the panning calculation unit 211 adds the output signals for each speaker generated in step S914 to the output signal buffer for each speaker. By adding the output signals for each speaker, multiple sound source signals are assigned to each speaker. As a result, the sound source signals of multiple virtual sound sources are arranged in the correct direction as seen from the listening point and played back simultaneously, generating a sound field appropriate for the listening point.

[0068] In step S916, the panning calculation unit 211 judges whether or not the processing for all the virtual sound source information output from the first coordinate calculation unit 203 in step S703 in Fig. 7 is completed. If the panning calculation unit 211 judges that there is unprocessed virtual sound source information (NO in step S916), the process returns to step S907 to select one piece of virtual sound source information from the unprocessed virtual sound source information, and the processing is performed for the next virtual sound source information. On the other hand, if the panning calculation unit 211 judges that the processing for all the virtual sound source information is completed (YES in step S916), the process of step S917 is executed.

[0069] In step S917, the signal correction unit 214 of the sound source rendering unit 206 calculates the distance between each speaker and the listening point. The signal correction unit 214 calculates the distance between the speaker and the listening point, for example, by using the speaker relative coordinates output from the second coordinate calculation unit 205 in step S702 of FIG.

[0070] In step S918, the signal correction unit 214 calculates the amount of correction for the sound pressure and delay of the sound emitted from each speaker based on the distance between each speaker and the listening point calculated in step S917, and corrects the output signal to each speaker. This optimizes the output sound from each speaker at the listening point, and the sound image is reproduced in the correct position.

[0071] In step S919, the sound source rendering unit 206 outputs the output signals for each speaker stored in the output signal buffer for each speaker to the sound reproducing unit 207. When the process of step S919 ends, the sound source rendering process ends and the process returns to the sound field generation process shown in FIG.

[0072] As described above, by appropriately changing the distribution of the sound source signal of the virtual sound source to each speaker so that the sound image perceived at the listening point coincides with the position of the virtual sound source as the listening point moves, it is possible to present a sound image at the position of the virtual sound source even if the listening point moves.

[0073] According to this embodiment, the position of the listening point is acquired for each processing time unit, the position of the virtual sound source relative to the listening point position and the position of the speaker relative to the listening point position are calculated, and the sound source signal of the virtual sound source is allocated to each speaker accordingly. This makes it possible to generate a sound field by the virtual sound source according to the position of the listening point even if the listening point moves. Therefore, even if the listening point moves, the sound image of the virtual sound source can be presented at the correct position as seen from the listening point, as if the virtual sound source were there in reality.

[0074] (Embodiment 2) In the first embodiment, an example is described in which a sound field suitable for the position of a listening point is generated based on a virtual sound source arranged in a virtual space in a listening space surrounded by speakers. In the second embodiment, an example is described in which a three-dimensional image of a virtual space suitable for a listening point is generated and played back in addition to the generation of a sound field by a virtual sound source. Note that a description of the same configuration and processing as in the first embodiment is omitted.

[0075] The control device in this embodiment is assumed to operate in cooperation with a virtual viewpoint image generating device that generates an image (virtual viewpoint image) when a virtual camera (virtual camera) is placed at an arbitrary position in a virtual space in which a subject and a background are placed. The coordinates indicating the position of the virtual camera are input to the control device from outside.

[0076] Fig. 10 is a block diagram showing an example of the functional configuration of a control device in embodiment 2. In Fig. 10, components having the same functions as those shown in Fig. 2 are given the same reference numerals, and duplicated explanations are omitted. The control device in this embodiment has a listening point acquisition unit 201, a virtual sound source information DB 202, a first coordinate calculation unit 203, a speaker information DB 204, a second coordinate calculation unit 205, a sound source rendering unit 206, and a sound reproduction unit 207. The control device in this embodiment also has an operation unit 1001, a viewing space movement unit 1002, a video generation unit 1004, a video display unit 1005, and a sound source shielding calculation unit 1006.

[0077] The operation unit 1001 accepts user operations to freely move around the viewing space surrounded by speakers. The user operations input to the operation unit 1001 are converted into movement commands and transmitted to the viewing space movement unit 1002. The viewing space movement unit 1002 moves the viewing space within the virtual space based on the received movement commands, calculates the coordinates of the viewing space in the virtual space after the movement, and outputs the calculated coordinates to the video generation unit 1004.

[0078] The image generating unit 1004 moves the viewing space to the coordinates specified by the viewing space moving unit 1002, and generates a three-dimensional image of the virtual world seen from the position of the listening point acquired by the listening point acquiring unit 201, using the virtual world 3D model 1003. The virtual world 3D model 1003 includes 3D data and arrangement information of each component that configures the virtual world. The image generated by the image generating unit 1004 is output to the image display unit 1005. The image display unit 1005 performs color adjustment for display and division processing for displaying on multiple displays on the three-dimensional image of the virtual world generated by the image generating unit 1004, and outputs it to the multi-display 1007 for display.

[0079] The sound source occlusion calculation unit 1006 calculates a state in which a sound source object is occluded by an obstruction in the image of the virtual world generated by the image generation unit 1004, based on the virtual sound source information of each virtual sound source acquired from the virtual sound source information DB 202. The sound source occlusion calculation unit 1006 outputs the calculation result of the occlusion state of the sound source object by the obstruction to the sound source rendering unit 206.

[0080] The multi-display 1007 is composed of a plurality of LED displays, a projection display, and the like. The multi-display 1007 displays the image of the virtual world output from the image display unit 1005 inside the viewing space. This allows the system of the present disclosure to visually immerse the listener at the listening point 209 in the virtual world. Note that FIG. 10 illustrates a case where five displays are installed with each speaker 210 constituting the speaker set 208 hidden, and the state where the speakers 210 are hidden is represented by dotted lines. It is assumed that the sound from each speaker 210 passes through the display, or that each speaker 210 is embedded in the display and the sound from each speaker 210 is emitted from the display surface.

[0081] In this embodiment, an example of the hardware configuration of the control device is omitted since it is the same as that of the first embodiment.

[0082] The process performed by the control device in this embodiment will now be described with reference to a flowchart of FIG 11, which shows an example of the sound field and image generation process performed by the control device in this embodiment. In step S1101, the viewing space moving unit 1002 performs a process of moving the viewing space in the virtual space in accordance with a user's operation instruction to the operation unit 1001. The viewing space moving unit 1002 changes the coordinates and orientation in the virtual space of a reference point determined in the viewing space in accordance with a user's operation instruction to the operation unit 1001. This type of processing is generally performed in the field of games using 3DCG (three-dimensional computer graphics) and is well known, so a description thereof will be omitted. The viewing space moving unit 1002 outputs the coordinates and orientation of the changed viewing space to the video generation unit 1004 and the first coordinate calculation unit 203.

[0083] The process of step S1102 is similar to the process of step S701 in FIG. 7 in the first embodiment, and therefore a description thereof will be omitted. The processes of steps S1103 to S1106 and steps S1107 to S1109 are executed in parallel, except that the process of step S1108 must be completed before the process of step S1105 can be executed. The processes in steps S1103 and S1104 are similar to the processes in steps S702 and S703 in FIG. 7 in the first embodiment, respectively, and therefore will not be described.

[0084] In step S1105, the sound source rendering unit 206 performs a sound source rendering process, and renders the sound source signals of the virtual sound sources for the multiple speakers that make up the speaker set 208 to generate a sound field. The sound source rendering unit 206 generates output signals for each speaker based on the speaker relative coordinates calculated in step S1103 and the virtual sound source relative coordinates calculated in step S1104. At this time, the sound source rendering unit 206 renders the sound source signals of each virtual sound source, taking into account the occlusion situation for each virtual sound source calculated in step S1108. In this way, output signals for each speaker that makes up the sound field at the listening point are generated, and the generated output signals for each speaker are output to the sound reproduction unit 207. Details of the sound source rendering process performed in step S1105 will be described later with reference to FIG. 12. The process of step S1106 is similar to the process of step S705 in FIG. 7 in the first embodiment, and therefore a description thereof will be omitted.

[0085] In step S1107, the image generating unit 1004 generates an image of the virtual world using the virtual world 3D model 1003. The image generating unit 1004 calculates listening point coordinates in the virtual space based on the coordinates and orientation of the viewing space output from the viewing space moving unit 1002 in step S1101 and the position of the listening point acquired by the listening point acquiring unit 201 in step S1102. Then, the image generating unit 1004 generates an image of the virtual world to be displayed on the multi-display 1007 using the virtual world 3D model 1003 based on the listening point coordinates in the virtual space obtained by calculation. Such processing is generally performed in the field of image generation using 3DCG and is well known, so a description thereof will be omitted.

[0086] In step S1108, the sound source shielding calculation unit 1006 acquires virtual sound source information of each virtual sound source from the virtual sound source information DB 202, and calculates the shielding state of the virtual sound source object in the image of the virtual world generated in step S1107. Based on the virtual sound source information of each virtual sound source, the sound source shielding calculation unit 1006 determines whether or not there is another object shielding the sound source in the virtual world image generated in step S1107 for each virtual sound source object. If there is another object shielding the virtual sound source, the sound source shielding calculation unit 1006 analyzes and calculates what the shielding object is and how much the sound source is shielded. For example, the sound source shielding calculation unit 1006 calculates an analysis result such as a shielding object of a certain virtual sound source being a tree, and 40% of the sound source being hidden. Then, the sound source shielding calculation unit 1006 outputs the obtained result to the sound source rendering unit 206.

[0087] In step S1109, the image display unit 1005 appropriately outputs and displays the image of the virtual world generated in step S1107 on the multi-display 1007. As a result, the image of the virtual world seen from the listening point is displayed on the multi-display 1007, allowing the listener at the listening point 209 to be visually immersed in the virtual world.

[0088] Fig. 12 is a flowchart showing an example of the sound source rendering process in step S1105 of Fig. 11. All the processes in the sound source rendering process shown in Fig. 12 are executed in the sound source rendering unit 206.

[0089] The processing in steps S1201 to S1213 is similar to the processing in steps S901 to S913 in FIG. 9 in the first embodiment, respectively, and therefore description thereof will be omitted. In step S1214, the sound source rendering unit 206 processes the sound source signal of the virtual sound source read in step S1213, based on the occlusion state of the virtual sound source to be processed calculated in step S1108 of Fig. 11. For example, when the virtual sound source is occluded by a tree, an acoustic characteristic filter reflecting the acoustic characteristics due to occlusion by the tree is applied to the sound source signal, and when the occlusion rate is 40%, the sound pressure is processed to 60%. The processing in steps S1215 to S1220 is similar to the processing in steps S914 to S919 in FIG. 9 in the first embodiment, respectively, and therefore a description thereof will be omitted.

[0090] In the above example, the three-dimensional image of the generated virtual world is displayed on a multi-display 1007, but this is not limited to this and may be displayed, for example, on an HMD (head-mounted display) that can be worn by the user.

[0091] According to this embodiment, even if the listening point moves, the sound image of the virtual sound source is presented at the correct position as seen from the listening point, together with the image of the virtual space as seen from the listening point, as in the first embodiment. Furthermore, when an object that is the sound source is occluded by another object in the image of the virtual space, sound processing is performed taking into account the occlusion situation. Therefore, the sound image of the virtual space is presented as it appears in reality, further enhancing the sense of immersion in the virtual world at the listening point.

[0092] In addition, in the figures of each embodiment described above, an example in which the speakers are arranged three-dimensionally is shown, but the present invention is not limited to this, and the speakers may be arranged two-dimensionally. In addition, in each embodiment described above, the position of the listening point is acquired for each processing unit time, and the position of the virtual sound source relative to the listening point position and the position of the speaker relative to the listening point position are calculated. However, the present invention is not limited to this, and when it is detected that the listening point position has moved, the position of the virtual sound source relative to the listening point position and the position of the speaker relative to the listening point position may be recalculated, and the positions of the speakers relative to the listening point position may be used for the rest of the calculations.

[0093] (Other embodiments) In the above-mentioned embodiment, the position where the virtual sound source is placed (coordinates of the virtual sound source) is fixed, but it is also applicable to the case where the virtual sound source moves. For example, the coordinates of the virtual sound source with time added to the virtual sound source information of the virtual sound source may be stored, and the coordinates of the virtual sound source at that time may be acquired. An example of the data configuration of the virtual sound source information in this case is shown in FIG. 13. As shown in FIG. 13, the virtual sound source information in this case includes a virtual sound source ID 1301, a sound source signal channel 1302, a start time code 1303, an end time code 1304, a total number of frames 1305, and a sound source coordinate history 1306.

[0094] The virtual sound source ID 1301 is an ID number for identifying this virtual sound source information, and is uniquely assigned to each piece of virtual sound source information, i.e., each virtual sound source. The sound source signal channel 1302 is the number of the channel of the sound source signal linked to this virtual sound source among the channels of the sound source signal input to the control device. The start time code 1303 indicates the beginning time code of the sound source coordinate history stored in this virtual sound source information. The end time code 1304 indicates the end time code of the sound source coordinate history stored in this virtual sound source information. The total number of frames n 1305 indicates the number of elements of the sound source coordinate history. The sound source coordinate history 1306 stores pairs 1307-1, 1307-2, ..., 1307-n of time codes and sound source coordinates for the total number of frames. By configuring the virtual sound source information in this way, it becomes possible to obtain the virtual sound source coordinates in the time code.

[0095] In the above embodiment, the panning process is based on the VBAP method, but similar processes can be performed using other panning algorithms. In the above embodiment, the virtual sound source coordinates are obtained from the virtual sound source information DB 202, but a system that obtains the position of an actual object that serves as a virtual sound source in real time may be combined, and the coordinates obtained from this system may be used as the virtual sound source coordinates.

[0096] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0097] It should be noted that the above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0098] The disclosure of the present embodiment includes the following configurations, methods, etc. (Configuration 1) An acquisition means for acquiring the position of a listening point; a first calculation means for calculating the positions of one or more virtual sound sources relative to the obtained positions of the listening points; a second calculation means for calculating positions of a plurality of speakers arranged around the listening point relative to the acquired position of the listening point; A generating means for generating an output signal to each speaker based on a sound source signal emitted by the one or more virtual sound sources, a position of the virtual sound source relative to the position of the listening point, and positions of the multiple speakers relative to the position of the listening point; a reproduction means for causing each speaker to reproduce a sound corresponding to the output signal. (Configuration 2) The control device according to configuration 1, wherein the generation means distributes the sound source signal to the speakers and generates output signals for each speaker based on a position of a virtual sound source relative to the position of the listening point and positions of a plurality of speakers relative to the position of the listening point. (Configuration 3) The control device according to configuration 2, characterized in that the generation means selects a pair of speakers to which the sound source signal is distributed from among the plurality of speakers based on a position of a virtual sound source relative to the position of the listening point and positions of the plurality of speakers relative to the position of the listening point, and distributes the sound source signal to the selected pair of speakers. (Configuration 4) The generating means includes: A speaker pair creating means for creating a pair of speakers that are adjacent to each other when viewed from the acquired position of the listening point; an inverse matrix calculation means for calculating an inverse matrix for each of the sets of speakers using a vector of a direction of each speaker as viewed from the position of the listening point; The control device according to any one of configurations 1 to 3, further comprising a panning calculation means for determining a set of speakers to which the sound source signal is allocated based on the inverse matrix and a position of a virtual sound source relative to the position of the listening point, and generating an output signal for each speaker. (Configuration 5) The control device according to any one of configurations 1 to 4, wherein the generating means has a signal correcting means for correcting the output signal output to each speaker in accordance with the distance between the listening point and the speaker. (Configuration 6) 6. The control device according to configuration 5, wherein the signal correction means performs correction relating to at least one of sound pressure and delay. (Configuration 7) an image generating means for generating a three-dimensional image of a virtual space seen from the position of the listening point; The control device according to any one of configurations 1 to 6, further comprising: an image display means for displaying a three-dimensional image of the virtual space generated by the image generation means inside the space in which the plurality of speakers are arranged. (Configuration 8) 8. The control device according to configuration 7, further comprising space movement means for moving a space in which the plurality of speakers are arranged within the virtual space. (Configuration 9) The control device according to configuration 7 or 8, wherein the generation means adjusts the output signal to each speaker depending on a state of obstruction of the virtual sound source by an obstruction object in the three-dimensional image of the virtual space generated by the image generation means. (Configuration 10) 10. The control device according to any one of configurations 1 to 9, comprising the plurality of speakers. (Method 1) A control method performed by a control device, An acquisition step for acquiring the positions of the listening points; a first calculation step of calculating the positions of one or more virtual sound sources relative to the obtained positions of the listening points; a second calculation step of calculating positions of a plurality of speakers arranged around the listening point relative to the obtained position of the listening point; a generating step of generating an output signal for each speaker based on a sound source signal emitted by the one or more virtual sound sources, a position of the virtual sound source relative to the position of the listening point, and a position of a plurality of speakers relative to the position of the listening point; and a reproduction step of reproducing, from each speaker, a sound corresponding to the output signal. (Program 1) an acquisition step for acquiring the location of a listening point; a first calculation step of calculating the positions of one or more virtual sound sources relative to the obtained positions of the listening points; a second calculation step of calculating positions of a number of speakers arranged around the listening point relative to the obtained position of the listening point; a generating step of generating output signals for each speaker based on a sound source signal emitted by the one or more virtual sound sources, a position of the virtual sound source relative to the position of the listening point, and positions of a plurality of speakers relative to the position of the listening point; A reproduction step of reproducing, from each speaker, a sound corresponding to the output signal. [Explanation of symbols]

[0099] 201: Listening point acquisition unit 202: Virtual sound source information DB 203: First coordinate calculation unit 204: Speaker information DB 205: Second coordinate calculation unit 206: Sound source rendering unit 207: Sound reproduction unit 208: Speaker set 209: Listening point 210: Speaker 211: Panning calculation unit 212: Speaker set creation unit 213: Inverse matrix calculation unit 214: Signal correction unit 1001: Operation unit 1002: Viewing space movement unit 1003: Virtual world 3D model 1004: Image generation unit 1005: Image display unit 1006: Sound source shielding calculation unit

Claims

1. acquisition means for acquiring the position of the listening point; a first calculation means for calculating the position of one or more virtual sound sources relative to the obtained position of the listening point; a second calculation means for calculating positions of a plurality of speakers arranged around the listening point relative to the acquired position of the listening point; a generating means for generating output signals to each of the plurality of speakers based on a sound source signal emitted by the one or more virtual sound sources, a position of the virtual sound source relative to the position of the listening point, and positions of the plurality of speakers relative to the position of the listening point; a reproduction means for reproducing the output signal through the plurality of speakers.

2. 2. The control device according to claim 1, wherein the generating means generates the output signal based on the position of a virtual sound source relative to the position of the listening point and the positions of a plurality of speakers relative to the position of the listening point.

3. 3. The control device according to claim 2, wherein the generation means selects a pair of speakers to which the sound source signal is to be distributed from the plurality of speakers based on the position of a virtual sound source relative to the position of the listening point and the positions of the plurality of speakers relative to the position of the listening point, and distributes the sound source signal to the selected pair of speakers.

4. The generating means a pair creating means for creating a pair of speakers each consisting of adjacent speakers as viewed from the position of the listening point; an inverse matrix calculation means for calculating an inverse matrix for each of the speaker sets using a vector of the direction of each of the plurality of speakers as seen from the position of the listening point; 2. The control device according to claim 1, further comprising a panning calculation means for determining a set of speakers to which the sound source signal is to be allocated based on the inverse matrix and the position of a virtual sound source relative to the position of the listening point, and for generating an output signal to each of the plurality of speakers.

5. 2. The control device according to claim 1, wherein the generating means includes a signal correcting means for correcting the output signals output to the plurality of speakers in accordance with the distance between the listening point and the speakers.

6. 6. The control device according to claim 5, wherein the signal correcting means performs correction relating to at least one of sound pressure and delay.

7. an image generating means for generating a three-dimensional image of the virtual space seen from the position of the listening point; 2. The control device according to claim 1, further comprising: image display means for displaying a three-dimensional image of the virtual space generated by the image generation means inside the space in which the plurality of speakers are arranged.

8. 8. The control device according to claim 7, further comprising space movement means for moving the space in which the plurality of speakers are arranged within the virtual space.

9. The control device according to claim 7, characterized in that the generation means adjusts the output signals to the plurality of speakers depending on the obstruction state of the virtual sound source by an obstruction object in the three-dimensional image of the virtual space generated by the image generation means.

10. The first calculation means converts the coordinates of each of the one or more virtual sound sources described in a coordinate system of the virtual world into coordinates in a coordinate system of the real world, and calculates the relative position of each of the one or more virtual sound sources with respect to the position of the listening point using the converted coordinates.

2. The control device according to claim 1.

11. The control device described in Claim 1, characterized in that the positions of the one or more virtual sound sources relative to the position of the listening point, and the positions of the multiple speakers relative to the position of the listening point, are expressed as coordinates in a three-dimensional coordinate system whose origin is the position of the listening point and whose axis directions coincide with the axis directions of a real-world coordinate system.

12. The device further comprises a detection means for detecting movement of the listening point, the second calculation means recalculates the positions of the one or more virtual sound sources and the positions of the plurality of speakers relative to the position of the listening point detected by the detection means; The generating means regenerates the output signal based on the recalculated positions of the one or more virtual sound sources relative to the listening point and the positions of the plurality of speakers relative to the listening point.

2. The control device according to claim 1.

13. An acquisition step of acquiring the position of a listening point; a first calculation step of calculating the positions of one or more virtual sound sources relative to the obtained positions of the listening points; a second calculation step of calculating positions of a plurality of speakers arranged around the listening point relative to the acquired position of the listening point; a generating step of generating output signals for each of the plurality of speakers based on a sound source signal emitted by the one or more virtual sound sources, a position of the virtual sound source relative to the position of the listening point, and positions of the plurality of speakers relative to the position of the listening point; and a reproduction step of reproducing the output signal through the plurality of speakers.

14. an acquisition step for acquiring the location of the listening point; a first calculation step of calculating the positions of one or more virtual sound sources relative to the obtained positions of the listening points; a second calculation step of calculating positions of a plurality of speakers arranged around the listening point relative to the acquired position of the listening point; a generating step of generating output signals for each of the plurality of speakers based on sound source signals emitted by the one or more virtual sound sources, positions of the virtual sound sources relative to the position of the listening point, and positions of the plurality of speakers relative to the position of the listening point; a reproduction step of reproducing the output signal through the plurality of speakers.

Images