Information processing device, control method for information processing device, and program

Abstract

This invention provides an information processing device, a control method for the information processing device, and a program that enable appropriate correction even when the sound pressure level or sound quality of the sound data acquired to generate a virtual sound source used for virtual viewpoint sound changes. [Solution] The information processing device 100 that generates a virtual sound source used to generate virtual viewpoint sound includes: a data correction unit 122 which is a correction means for correcting multiple highly directional sound data generated based on sound data acquired by multiple microphones; a similarity calculation unit 124 which is a first calculation means for calculating the similarity between the multiple corrected sound data obtained by the correction means; a data selection unit 125 which is a selection means for selecting corrected sound data to be used to generate a virtual sound source from the multiple corrected sound data based on the calculated similarity between the corrected sound data; and a data synthesis unit 126 which is a synthesis means for generating a virtual sound source by synthesizing the selected corrected sound data.

Description

[Technical Field]

[0001] This disclosure relates to an information processing device, a control method for the information processing device, and a program. [Background technology]

[0002] To enhance the sense of realism when viewing virtual viewpoint images from a user-specified virtual viewpoint, there is a technology that generates an audio signal corresponding to the virtual viewpoint based on the audio signal captured in the shooting environment, and plays it back in sync with the virtual viewpoint image as virtual viewpoint audio. In sports competitions such as baseball and soccer, the virtual viewpoint image is an image from a virtual viewpoint that has entered the field, and virtual viewpoint audio requires the generation of audio data that matches the viewpoint. For example, when playing back sounds of competition such as the sound of a baseball being hit or caught, or the sound of a soccer shot, it is desirable that the sounds of competition be heard loudly when the sounds of competition are played back together with an image of a virtual viewpoint close to the location where the sounds of competition are generated. However, since it is difficult to place microphones inside the field in many sports competitions, it is common to place microphones from outside the field and facing inward to capture the sounds of competition. Therefore, if the distance from the microphone to the location where the sounds of competition are generated is large, the sounds of competition may be greatly attenuated and drowned out by the sound of the crowd.

[0003] Patent Document 1 discloses a technique for correcting the delay amount and sound pressure level of acoustic data based on the positional relationship between the target sound source and each microphone. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-53804 [Overview of the project] [Problems that the invention aims to solve]

[0005] However, there was a challenge in appropriately correcting changes in the sound pressure level and sound quality of acoustic data due to factors other than distance attenuation. [Means for solving the problem]

[0006] The information processing device according to this disclosure is an information processing device for generating a virtual sound source used for generating virtual viewpoint sound, and is characterized by comprising: correction means for correcting a plurality of highly directional sound data generated based on sound data acquired by a plurality of microphones; a first calculation means for calculating the similarity between the plurality of corrected sound data obtained by the correction means; selection means for selecting the corrected sound data to be used for generating the virtual sound source from the plurality of corrected sound data based on the calculated similarity between the corrected sound data; and synthesis means for synthesizing the selected corrected sound data to generate the virtual sound source. [Effects of the Invention]

[0007] Even if the sound pressure level or sound quality of the acquired audio data changes, appropriate correction can be performed. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the functional configuration of an information processing device. [Figure 2] This figure shows an example of the hardware configuration of an information processing device. [Figure 3] This is a flowchart showing an example of processing performed by an information processing device. [Figure 4] This is a diagram illustrating the channel configuration of the A format. [Figure 5] This diagram illustrates highly directional processing. [Figure 6] This diagram illustrates examples of combinations used in calculating cross-correlation and the results of those combinations. [Modes for carrying out the invention]

[0009] The following description is based on the drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0010] This embodiment will be described using keirin (bicycle racing) as an example of a sound collection target for generating virtual viewpoint sound. In keirin, for example, the sounds to be collected include the sounds emitted when the racing cyclist is running, the cheers from the stands, and the sound of the bell striking to indicate the progress of the race. It should be noted that the application of the technology disclosed herein is not limited to keirin, but may also be to other sports competitions, music concerts, theatrical performances, etc.

[0011] When generating virtual viewpoint sound, sound sources placed in the virtual space are referred to as virtual sound sources, and 3D models of subjects and virtual sound sources are placed as objects in the virtual space. The position where the virtual sound source is placed is also called the sound source position VP, and the sound source position VP may be obtained by identifying and tracking each player, etc. from 3D models or texture images, or it may be obtained by an external sensor.

[0012] This embodiment describes a method for generating a virtual sound source from acoustic data, including sports sounds and ambient sounds, acquired by multiple microphones, based on the positional relationship between each microphone and each object. As an example of a microphone, a first-order ambisonic microphone is used, and will be simply referred to as an ambisonic microphone below. The position where the microphones are placed will also be referred to as the microphone position MP.

[0013] Figure 1 is a block diagram showing an example of the functional configuration of the information processing device 100 in this embodiment. The information processing device 100 includes a storage unit 110, a virtual sound source generation unit 120, a virtual viewpoint image generation unit 130, and a virtual viewpoint sound generation unit 140.

[0014] The storage unit 110 stores virtual viewpoint content configuration data, including image data constituting the virtual viewpoint image, sound data constituting the virtual viewpoint sound, microphone position MP, and sound source position VP. Each virtual viewpoint content configuration data stored in the storage unit 110 is associated with time information, and each data can be retrieved at a given time by specifying each data and the corresponding ID and time information. The image data stored in the storage unit 110 is acquired, for example, by multiple cameras (imaging devices). The sound data stored in the storage unit 110 is acquired, for example, by multiple microphones (sound pickup devices). In the ambisonics method, the sound data as it is picked up is called A format, and in primary ambisonics, A format consists of 4 channels of sound data. In this embodiment, since multiple ambisonic microphones are used, A format, i.e., 4 channels of sound data, is stored for each ambisonic microphone. The A format sound data picked up by this microphone is also called sound data A.

[0015] The virtual viewpoint image generation unit 130 generates a virtual viewpoint image based on a virtual viewpoint acquired from an external source. Specifically, the virtual viewpoint image generation unit 130 acquires image data for generating a virtual viewpoint image from the storage unit 110 based on a virtual viewpoint acquired from an external source. Then, the virtual viewpoint image generation unit 130 generates a virtual viewpoint image based on the acquired image data and outputs the generated virtual viewpoint image. The virtual viewpoint image generation unit 130 is an example of an image generation means.

[0016] The virtual viewpoint sound generation unit 140 generates virtual viewpoint sound corresponding to a virtual viewpoint acquired from an external source. Specifically, the virtual viewpoint sound generation unit 140 acquires a virtual sound source and sound source position VP from the storage unit 110 based on the virtual viewpoint acquired from an external source. Then, the virtual viewpoint sound generation unit 140 generates virtual viewpoint sound based on the acquired virtual sound source and sound source position VP, and outputs the generated virtual viewpoint sound. For the processing used to generate virtual viewpoint sound, known surround processing techniques such as VBAP (Vector Base Amplitude Panning) may be used. Alternatively, binaural processing based on head-related transfer functions (HRTF) may be used, or a combination of these may be used. The virtual viewpoint sound generation unit 140 is an example of sound generation means.

[0017] The virtual sound source generation unit 120 acquires acoustic data A, microphone position MP, and sound source position VP stored in the storage unit 110, generates a virtual sound source based on the acquired data, and outputs the generated virtual sound source to the storage unit 110. The virtual sound source generation unit 120 includes a high directivity processing unit 121, a data correction unit 122, a power calculation unit 123, a similarity calculation unit 124, a data selection unit 125, and a data synthesis unit 126.

[0018] The high-directivity processing unit 121 acquires acoustic data A, microphone position MP, and sound source position VP from the storage unit 110, and generates highly directional acoustic data (high-directivity acoustic data) based on the acquired data. Hereafter, the high-directivity acoustic data will also be referred to as acoustic data D. The high-directivity processing unit 121 outputs the generated acoustic data D to the data correction unit 122. Acoustic data D is monaural acoustic data corresponding to the number of ambisonic microphones used to generate the virtual sound source. For example, if four ambisonic microphones are used to generate one virtual sound source, acoustic data D will consist of four monaural acoustic data.

[0019] The data correction unit 122 acquires acoustic data D from the high-directivity processing unit 121, acquires microphone position MP and sound source position VP from the storage unit 110, and generates corrected acoustic data based on the acquired data. Hereafter, the corrected acoustic data will also be referred to as acoustic data R. The data correction unit 122 outputs the generated acoustic data R to the power calculation unit 123, the similarity calculation unit 124, and the data synthesis unit 126. Acoustic data R is obtained by correcting the waveform of acoustic data D, and its data structure is the same as that of high-directivity acoustic data. The data correction unit 122 is an example of a correction means.

[0020] The power calculation unit 123 acquires acoustic data R from the data correction unit 122 and calculates the power PWR of each acoustic data R (corrected acoustic data) based on the acquired acoustic data R. Power PWR is, for example, the time average of the squared amplitude value of the acoustic data. The power calculation unit 123 outputs the calculated power PWR of each acoustic data R to the data selection unit 125. The power calculation unit 123 is an example of a second calculation means.

[0021] The similarity calculation unit 124 acquires acoustic data R from the data correction unit 122 and calculates the similarity between each acoustic data R (between corrected acoustic data) based on the acquired acoustic data R. The similarity calculation unit 124 outputs the calculated similarity to the data selection unit 125. The similarity calculation unit 124 is an example of the first calculation means.

[0022] The data selection unit 125 obtains the power PWR of each sound data R from the power calculation unit 123, obtains the similarity from the similarity calculation unit 124, and selects the sound data R to be used for generating a virtual sound source from among multiple sound data R based on the acquired data. In other words, the data selection unit 125 selects the sound data R to be synthesized when generating a virtual sound source based on the power PWR of each acquired sound data R and the similarity between each sound data R. The data selection unit 125 outputs the selection results regarding the sound data R to the data synthesis unit 126. The data selection unit 125 is an example of a selection means.

[0023] The data synthesis unit 126 acquires acoustic data R from the data correction unit 122, acquires selection results for the acoustic data R from the data selection unit 125, and synthesizes the acoustic data based on each acquired data. The data synthesis unit 126 synthesizes the acoustic data R selected from the acquired acoustic data R to be used for generating a virtual sound source. The data synthesis unit 126 outputs the synthesized acoustic data as a virtual sound source V to the storage unit 110. The data synthesis unit 126 is an example of a synthesis means.

[0024] Figure 2 is a block diagram showing an example of the hardware configuration of the information processing device in this embodiment. The information processing device in this embodiment includes a CPU 211, ROM 212, RAM 213, auxiliary storage device 214, image display unit 215, sound output unit 216, operation unit 217, communication interface 218, and bus 219. The CPU 211, ROM 212, RAM 213, auxiliary storage device 214, image display unit 215, sound output unit 216, operation unit 217, and communication interface 218 are connected to each other via the bus 219 so that they can communicate with one another.

[0025] The CPU (Central Processing Unit) 211 controls the entire information processing device using computer programs and data stored in the ROM 212 and RAM 213, thereby realizing each function of the information processing device 100 shown in Figure 1. Note that the information processing device may have one or more dedicated hardware components separate from the CPU 211, and at least a portion of the processing performed by the CPU 211 may be executed by the dedicated hardware. Examples of dedicated hardware include ASICs (Application-Specific Integrated Circuits).

[0026] ROM (Read Only Memory) 212 stores programs and other data that do not require modification. RAM (Random Access Memory) 213 temporarily stores programs and data supplied from auxiliary storage devices 214, as well as data supplied from external sources via the communication interface 218. Auxiliary storage devices 214 are composed of, for example, hard disk drives (HDDs) or solid-state drives (SSDs), and store various types of data such as image data, sound data, metadata, and configuration information.

[0027] The image display unit 215 is composed of, for example, a liquid crystal display or LEDs, and displays a GUI (Graphical User Interface) for the user to operate the system. The sound output unit 216 is composed of, for example, a D / A converter, volume amplifier, speaker or headphones, and outputs the generated virtual viewpoint sound. The operation unit 217 is composed of, for example, a keyboard, mouse, joystick, touch panel, etc., and receives various instructions from the user and inputs them to the CPU 211. The CPU 211 operates as a display control unit that controls the image display unit 215, and as an operation control unit that controls the operation unit 217.

[0028] The communication interface (I / F) 218 ​​is used for communication with external devices. For example, if the information processing device is connected to an external device by a wired connection, the communication cable is connected to the communication interface 218. Also, for example, if the information processing device has a function to communicate wirelessly with an external device, the communication interface 218 is equipped with an antenna. The bus 219 connects each part and transmits information.

[0029] The operation of the information processing device 100 in this embodiment will be described with reference to Figure 3. Figure 3 is a flowchart showing an example of processing by the information processing device 100 in this embodiment. Figure 3 shows an example of virtual sound source generation processing performed by the virtual sound source generation unit 120. The processing shown in Figure 3 is started, for example, when the information processing device 100 receives an instruction to start processing related to recording data necessary for the virtual sound source generation unit 120 to generate a virtual sound source. The instruction to start processing may be made by user operation via the operation unit 217 of the information processing device 100, or the instruction may be input from another device. Note that the timing for starting the execution of the processing shown in Figure 3 is not limited to the timing described above.

[0030] The process shown in Figure 3 is achieved by the CPU 211 loading a program stored in ROM 212 or the like into RAM 213 and executing it. Note that at least a portion of the process shown in Figure 3 may be executed by one or more dedicated hardware components separate from the CPU 211.

[0031] In this embodiment, the information processing device 100 (virtual sound source generation unit 120) repeats the processing steps S301 to S313 for each virtual sound source to be generated, and by repeating this n times, n virtual sound sources can be generated. For example, if the target of generating virtual sound sources is running sounds, the number of virtual sound sources is the number of running sounds, and if there are 9 runners, then n=9.

[0032] In step S301, the high-directivity processing unit 121 determines an ID indicating the virtual sound source V to be generated, i.e., a virtual sound source ID. The virtual sound source ID is different for each virtual sound source, and a single virtual sound source can be uniquely identified by the virtual sound source ID. In this embodiment, as an example, the virtual sound source ID is set to an integer value starting from 0, and the virtual sound source V and sound source position VP corresponding to virtual sound source ID = i are respectively determined by the virtual sound source V i and sound source position VP i Let's assume that.

[0033] In step S302, the high-directional processing unit 121 controls the virtual sound source Vi Ambisonic microphone used in the generation processing, that is, the virtual sound source V i generation microphone M is determined. For example, the high directivity processing unit 121 determines the virtual sound source V i for the target subject position to generate (the virtual sound source V i sound source position VP i ) is a microphone whose distance is smaller than a predetermined threshold as the virtual sound source V i generation microphone M. Also, for example, the high directivity processing unit 121 may select a predetermined number of microphones starting from the one with the smallest distance to the virtual sound source V i sound source position VP i as the virtual sound source V i generation microphone M. Also, for example, the high directivity processing unit 121 may select, from among the microphones whose distance to the virtual sound source V i sound source position VP i is smaller than a predetermined threshold, a predetermined number of microphones starting from the one with the smallest distance as the virtual sound source V i generation microphone M. In the present embodiment, as an example, the ID indicating the virtual sound source V i generation microphone M used in the virtual sound source V i generation processing, that is, the used microphone ID is an integer value starting from 0, and the Ambisonic microphone corresponding to microphone ID=j is referred to as microphone M j .

[0034] The processing of the next step S303 is processing that is repeatedly executed for the number of virtual sound source V i generation microphones M used in the virtual sound source V i generation processing. In step S303, the high directivity processing unit 121 acquires, from the storage unit 110, the audio data A j corresponding to the microphone M j and the microphone position MP j , and the sound source position VP i , and generates high directivity audio data j (audio data D jThe high-directional processing unit 121 generates the high-directional processing for each microphone M based on the acquired data, and obtains acoustic data D for each microphone M. The high-directional processing unit 121 then outputs the obtained acoustic data D to the data correction unit 122.

[0035] In this embodiment, the highly directional processing refers to the conversion from A format to B format using the known first-order ambisonics method, and the directional enhancement processing towards the sound source direction for the B format. In this embodiment, the four channels of A format are each L f , R f , L b , R b Figure 4 shows the positional relationship with the four microphone capsules that make up the ambisonic microphone. With respect to the front 401 of the ambisonic microphone, the upper left front 402 is L f , right front lower 403 is R f , the left rear lower 405 is L b , 404 is R b The B format refers to an acoustic signal composed of four channels: an omnidirectional component W, a front-to-back component X, a left-to-right component Y, and a up-to-down component Z. Conversion from the A format to the B format can be performed using the following equation (1).

[0036]

number

[0037] Next, the direction enhancement processing for the B format will be explained with reference to Figures 5(a) and 5(b). The sound source 501 shown in Figures 5(a) and 5(b) is used to generate the virtual sound source V. i And, the Ambisonic Microphone 502 is Microphone M j And so on. Also, Mike M j The microphone position is MP j and sound source location VP i Let φ be the azimuth angle 511 and θ be the elevation angle 512. Here, the sound field rotated by φ around the z-axis relative to the B format can be obtained by the following equation (2).

[0038]

number

[0039] The sound field obtained by rotating the sound field obtained by equation (2) by θ around the y-axis can be obtained by the following equation (3).

[0040]

number

[0041] By applying equations (2) and (3) to the B format, Microphone M j The virtual sound source V is located in front of it, i.e., in the positive direction of X''. i The sound fields W”, X”, Y”, and Z”, where the virtual sound source V is located, are obtained. Furthermore, (W”, + X”) gives the frontal component of the sound field, i.e., the virtual sound source V i The components can be extracted as described above. The acoustic data extracted in this manner is then called acoustic data D. j This process is repeated for Ambisonic Microphone 503, Ambisonic Microphone 504, and Ambisonic Microphone 505, thereby creating a virtual sound source V. i Acoustic data D is obtained for each corresponding ambisonic microphone.

[0042] The next step, S304, involves processing the virtual sound source V i Virtual sound source V used in the generation process i This process is repeated for each generation of microphone M. In step S304, the data correction unit 122 controls the microphone M j Microphone position corresponding to MP j And, sound source position VP i The data is acquired from the storage unit 110, and based on the acquired data, acoustic data D j Correct and correct the acoustic data j (acoustic data R) j The data correction unit 122 generates the acquired sound source position VP. i and microphone position MP jBased on the positional relationship, acoustic data D j The sound pressure level and delay time are corrected, and the acoustic data R j The output is sent to the power calculation unit 123 and the similarity calculation unit 124. Outdoors, the sound source position VP i The distance d0 from the source is defined as the reference distance, the sound pressure level at the reference distance d0 is defined as L0, and the sound source position is defined as VP. i The distance attenuation, where L is the sound pressure level at a distance d from the source, can be calculated using the following equation (4).

[0043]

number

[0044] Here, if we set the reference distance d0 to 1m and the attenuation (L-L0) to ΔL, the attenuation ΔL can be calculated by the following equation (5). Acoustic data D j In contrast, the sound pressure level can be corrected using the calculated attenuation amount ΔL.

number

[0045] Furthermore, regarding time delay, for example, the sound source position VP i The sound emitted from the microphone was located at the MP j The time it takes to reach the sound source position VP i From microphone position MP j It can be calculated based on the distance to and the speed of sound. The data correction unit 122 uses the calculated time as the delay time and the acoustic data D j By shifting it in the time direction, the time delay is corrected. In this way, the acoustic data D j Corrected acoustic data R j You can obtain this.

[0046] The next step, S305, is a process that is repeated for each corrected acoustic data (acoustic data R). In step S305, the power calculation unit 123 calculates the acoustic data R j Based on the acoustic data R jPower PWR j The calculated power PWR j The output is sent to the data sorting unit 125. Here, the power PWR for acoustic data R, which consists of N samples (N is a natural number), is calculated by the following equation (6). In equation (6), R[n] is the amplitude value of each sample of the acoustic data R.

[0047]

number

[0048] The next steps, S306-S307, are processes that are repeatedly executed for each combination of corrected acoustic data (acoustic data R). In step S306, the similarity calculation unit 124 determines the combination of acoustic data R to be used for calculating similarity. For example, virtual sound source V i Acoustic data R for generating j If we have four acoustic data sets R0, R1, R2, and R3, the combinations of acoustic data R for calculating similarity will be as shown in Figure 6(a). Here, C is a value that indicates the similarity between the acoustic data R sets, for example, C 01 This indicates the similarity between acoustic data R0 and acoustic data R1.

[0049] In step S307, the similarity calculation unit 124 calculates the similarity between the acoustic data R based on the combination of acoustic data R determined in step S306, and outputs the calculated similarity C to the data selection unit 125. In this embodiment, as an example, cross-correlation is used as the similarity C. Cross-correlation is expressed as a value between (-1) and 1, where 1 indicates a correlation, 0 indicates no correlation, and (-1) indicates a negative correlation. For example, the cross-correlation C between acoustic data R1 and acoustic data R2, which are composed of N samples. xy This can be calculated using the following equation (7).

[0050]

number

[0051] Here, m is the lag, and by changing m, the correlation with respect to the temporal shift of the waveform can be calculated. Let x=R0 and y=R1, and the maximum value of the calculation result by equation (7) is the cross-correlation C. 01 The same calculation is performed for other combinations of acoustic data R0, R1, R2, and R3 to calculate the cross-correlation C between each acoustic data R. Figure 6(b) shows an example of each combination of acoustic data R and the calculation result of the cross-correlation C. Note that the processing in steps S306 and S307 described above is performed on the virtual sound source V i Acoustic data R for generating j The example given was when there are four virtual sound sources V i Acoustic data R for generating j The number is the number determined in step S302.

[0052] Next, in step S308, the similarity calculation unit 124 calculates the average value of the cross-correlation C corresponding to each acoustic data R, and outputs the calculated value as the similarity to the data selection unit 125. For example, in the examples shown in Figures 6(a) and 6(b), there are 6 combinations that include acoustic data R0, and the arithmetic mean of the cross-correlation of each combination is taken as the similarity C0 corresponding to acoustic data R0. Similarly, the similarities C1, C2, and C3 corresponding to acoustic data R1, R2, and R3 are calculated respectively, and the calculated similarities C0 to C3 are output as the similarity C to the data selection unit 125. In the example shown in Figure 6(b), the similarities C are C0=0.71, C1=0.74, C2=0.74, and C3=0.46, and it can be seen that the value of C3 is significantly smaller than the similarities of the other Cs. In this case, it is possible that the sound quality in the acoustic data has changed due to the influence of the transmission system from the sound source to microphone M3. Factors that could affect the transmission system include, for example, the influence of structures installed immediately before the race. However, even when the direct cause is unknown, this embodiment allows for detection in the form of a decrease in similarity.

[0053] The next steps, S309 to S312, are processes that are repeated for each corrected acoustic data (acoustic data R). In step S309, the data selection unit 125 selects the acoustic data R calculated in step S305. j Power PWR j Determine whether it is below the threshold. Power PWR j If the data sorting unit 125 determines that the power PWR is below the threshold (YES in S309), the data sorting unit 125 sorts the power PWR j Corresponding acoustic data R j It is determined that it will not be used to generate the virtual sound source, and the process in step S312 is executed. Meanwhile, the power PWR j If the data selection unit 125 determines that the threshold is exceeded (NO in S309), the process in step S310 is executed. As illustrated in the explanation of the process in step S308, if there is a structure between the sound source and the microphone M3, the power PWR3 may become significantly smaller due to its influence, and this step can detect such acoustic data R.

[0054] In step S310, the data selection unit 125 selects the acoustic data R calculated in step S307. j Corresponding similarity (average cross-correlation) C j Determine whether it is below the threshold. Similarity C j If the data sorting unit 125 determines that the similarity is below the threshold (YES in S310), the data sorting unit 125 determines the similarity C j Corresponding acoustic data R j It is determined that it will not be used to generate the virtual sound source, and the process in step S312 is executed. On the other hand, similarity C j If the data sorting unit 125 determines that the threshold is exceeded (NO in S310), the process in step S311 is executed.

[0055] In step S311, the data sorting unit 125 sorts the acoustic data R j The acoustic data selection flag j is set to TRUE, and the acoustic data selection flag j is output to the data synthesis unit 126. In other words, the data selection unit 125 sets the power PWRj exceeds the threshold, and the similarity C j exceeds the threshold, the acoustic data R j has its acoustic data selection flag j set to TRUE, and is selected as the acoustic data R to be used for generating a virtual sound source. The acoustic data selection flag j is a flag indicating whether the acoustic data R j is to be used for generating a virtual sound source. For the acoustic data R j that is used for generating a virtual sound source, the acoustic data selection flag j is set to TRUE, and for the acoustic data R j that is not used for generating a virtual sound source, the acoustic data selection flag j is set to FALSE.

[0056] In step S312, the data screening unit 125 sets the acoustic data selection flag j of the acoustic data R j to FALSE, and outputs the acoustic data selection flag j to the data synthesis unit 126. That is, the data screening unit 125 sets the acoustic data selection flag j of the acoustic data R j that satisfies at least one of the conditions that the power PWR j is equal to or less than the threshold, or the similarity C j is equal to or less than the threshold, to FALSE, so that the acoustic data R is not used for generating a virtual sound source.

[0057] In step S313, the data synthesis unit 126 refers to the acquired acoustic data selection flag of each acoustic data R, and synthesizes the acoustic data R j whose acoustic data selection flag is TRUE, and outputs the synthesized acoustic data R j to the storage unit 110 as a virtual sound source V i . For example, the data synthesis unit 126 calculates the arithmetic mean of the acoustic data R j whose acoustic data selection flag is TRUE, and outputs the result to the storage unit 110 as a virtual sound source V i .

[0058] In this embodiment, the information processing device does not use corrected acoustic data with power or similarity below a threshold for generating virtual sound sources, and instead generates virtual sound sources using corrected acoustic data with power and similarity above a threshold. According to this embodiment, even if a portion of the acquired acoustic data is attenuated by factors other than distance attenuation, and the sound pressure level and sound quality of the acoustic data change, it is possible to generate a virtual sound source with appropriate sound pressure level and sound quality using appropriate acoustic data.

[0059] In the embodiments described above, corrected acoustic data with power and similarity exceeding a threshold are selected as acoustic data to be used for generating virtual sound sources, but the invention is not limited to this. For example, selection may be performed based only on the similarity corresponding to the corrected acoustic data, and corrected acoustic data with similarity exceeding a threshold may be selected as acoustic data to be used for generating virtual sound sources. Alternatively, selection may be performed based only on the power of the corrected acoustic data, and corrected acoustic data with power exceeding a threshold may be selected as acoustic data to be used for generating virtual sound sources.

[0060] (Other embodiments) This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0061] Furthermore, the embodiments described above are merely examples of concrete implementations in carrying out this disclosure, and the technical scope of the invention should not be interpreted as being limited by them. In other words, this disclosure can be implemented in various ways without departing from its technical concept or its main features.

[0062] The disclosure of this embodiment includes the following configurations and methods, etc. (Composition 1) An information processing device for generating virtual sound sources used in generating virtual viewpoint sound, A correction means for correcting multiple highly directional acoustic data generated based on acoustic data acquired by multiple microphones, A first calculation means for calculating the similarity between a plurality of corrected acoustic data obtained by the correction means, A selection means for selecting the corrected sound data to be used to generate the virtual sound source from the plurality of corrected sound data based on the similarity between the calculated corrected sound data, An information processing apparatus characterized by having a synthesis means for synthesizing the selected corrected acoustic data to generate the virtual sound source. (Configuration 2) The information processing apparatus according to configuration 1, characterized in that the first calculation means calculates the similarity based on the cross-correlation between the corrected acoustic data. (Composition 3) The information processing apparatus according to configuration 1 or 2, characterized in that the selection means selects the corrected sound data whose similarity to the corrected sound data exceeds a threshold as corrected sound data to be used for generating the virtual sound source. (Composition 4) The system includes a second calculation means for calculating the power of the plurality of corrected acoustic data obtained by the correction means, The information processing apparatus according to any one of configurations 1 to 3, characterized in that the selection means selects the corrected sound data to be used for generating the virtual sound source based on the calculated similarity between the corrected sound data and the power of the corrected sound data. (Composition 5) The information processing device according to configuration 4, characterized in that the second calculation means calculates the power of the corrected acoustic data by averaging the squared value of the amplitude value of the corrected acoustic data over time. (Composition 6) The information processing apparatus according to configuration 4 or 5, characterized in that the selection means selects the corrected sound data whose similarity to the corrected sound data exceeds a threshold and whose power exceeds a threshold as corrected sound data to be used for generating the virtual sound source. (Composition 7) The information processing apparatus according to any one of configurations 1 to 6, characterized in that the correction means corrects the sound pressure level of the highly directional acoustic data. (Composition 8) The information processing apparatus according to any one of configurations 1 to 7, characterized in that the correction means corrects the delay time of the highly directional acoustic data. (Composition 9) The information processing apparatus according to configuration 7 or 8, characterized in that the correction means corrects the highly directional acoustic data based on the distance from the position of the sound source corresponding to the virtual sound source to the position of the microphone. (Composition 10) The information processing apparatus according to any one of configurations 1 to 9, characterized in that the synthesis means generates the virtual sound source by arithmetic mean averaging the selected corrected sound data. (Composition 11) An information processing device according to any one of configurations 1 to 10, characterized by having sound generation means for generating virtual viewpoint sound corresponding to a virtual viewpoint based on the virtual sound source. (Composition 12) An information processing device according to any one of configurations 1 to 11, characterized by having an image generation means for generating a virtual viewpoint image that can be seen from a virtual viewpoint based on image data. (Method 1) A control method for an information processing device that generates a virtual sound source used for generating virtual viewpoint sound, A correction process that corrects multiple highly directional acoustic data generated based on acoustic data acquired by multiple microphones, A first calculation step for calculating the similarity between a plurality of corrected acoustic data obtained in the correction step, A selection step of selecting the corrected sound data to be used to generate the virtual sound source from the plurality of corrected sound data based on the calculated similarity between the corrected sound data, A control method for an information processing device, characterized by comprising a synthesis step of synthesizing selected corrected acoustic data to generate the virtual sound source. (Program 1) A program to cause a computer to function as one of the information processing devices described in any one of the configurations 1 to 12. [Explanation of Symbols]

[0063] 100: Information processing unit 110: Storage unit 120: Virtual sound source generation unit 121: High directivity processing unit 122: Data correction unit 123: Power calculation unit 124: Similarity calculation unit 125: Data selection unit 126: Data synthesis unit

Claims

1. An information processing device for generating virtual sound sources used in generating virtual viewpoint sound, A correction means for correcting multiple highly directional acoustic data generated based on acoustic data acquired by multiple microphones, A first calculation means for calculating the similarity between a plurality of corrected acoustic data obtained by the correction means, A selection means for selecting the corrected sound data to be used to generate the virtual sound source from the plurality of corrected sound data based on the similarity between the calculated corrected sound data, An information processing apparatus characterized by having a synthesis means for synthesizing the selected corrected acoustic data to generate the virtual sound source.

2. The information processing apparatus according to claim 1, characterized in that the first calculation means calculates the similarity based on the cross-correlation between the corrected acoustic data.

3. The information processing apparatus according to claim 1, characterized in that the selection means selects the corrected sound data whose similarity to the corrected sound data exceeds a threshold as corrected sound data to be used for generating the virtual sound source.

4. The system includes a second calculation means for calculating the power of the plurality of corrected acoustic data obtained by the correction means, The information processing apparatus according to claim 1, wherein the selection means selects the corrected sound data to be used for generating the virtual sound source based on the calculated similarity between the corrected sound data and the power of the corrected sound data.

5. The information processing apparatus according to claim 4, characterized in that the second calculation means calculates the power of the corrected acoustic data by averaging the squared value of the amplitude value of the corrected acoustic data over time.

6. The information processing apparatus according to claim 4, characterized in that the selection means selects the corrected sound data for use in generating the virtual sound source if the similarity to the corrected sound data exceeds a threshold and the power of the corrected sound data exceeds a threshold.

7. The information processing apparatus according to claim 1, characterized in that the correction means corrects the sound pressure level of the highly directional acoustic data.

8. The information processing apparatus according to claim 1, characterized in that the correction means corrects the delay time of the highly directional acoustic data.

9. The information processing apparatus according to claim 7, characterized in that the correction means corrects the highly directional acoustic data based on the distance from the position of the sound source corresponding to the virtual sound source to the position of the microphone.

10. The information processing apparatus according to claim 1, characterized in that the synthesis means generates the virtual sound source by arithmetic mean averaging the selected corrected sound data.

11. The information processing apparatus according to claim 1, characterized by having sound generation means for generating virtual viewpoint sound corresponding to a virtual viewpoint based on the virtual sound source.

12. The information processing apparatus according to claim 1, characterized by having an image generation means for generating a virtual viewpoint image that can be seen from a virtual viewpoint based on image data.

13. A control method for an information processing device that generates a virtual sound source used for generating virtual viewpoint sound, A correction process that corrects multiple highly directional acoustic data generated based on acoustic data acquired by multiple microphones, A first calculation step for calculating the similarity between a plurality of corrected acoustic data obtained in the correction step, A selection step of selecting the corrected sound data to be used to generate the virtual sound source from the plurality of corrected sound data based on the calculated similarity between the corrected sound data, A control method for an information processing device, characterized by comprising a synthesis step of synthesizing selected corrected acoustic data to generate the virtual sound source.

14. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 12.

Images