Information processing device, control method for information processing device, and program

Abstract

Reduces noise in virtual view audio. [Solution] The information processing device includes: position detection means for detecting the position of an object within a predetermined area; selection means for selecting a microphone group to be processed from among a plurality of microphone groups for collecting sound within a predetermined area, the microphone group whose positional relationship with the object satisfies predetermined conditions; noise reduction means for performing noise reduction processing on acoustic data in accordance with changes in the selected microphone group to be processed; a first generation means for generating highly directional acoustic data from sound collection signals obtained from a plurality of microphones for each microphone group to be processed; and a second generation means for generating sound for playback based on a plurality of acoustic data including the generated highly directional acoustic data.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for an information processing apparatus, and a program.

Background Art

[0002] Regarding the viewing of a virtual viewpoint image representing the appearance of a predetermined area from an arbitrarily specified viewpoint, in order to improve the sense of presence, generation of virtual viewpoint sound corresponding to the viewpoint is required. For example, when reproducing a competitive sound such as the sound of a ball being kicked together with a virtual viewpoint image from a viewpoint as if approaching a player or a ball in a field such as soccer or rugby, it is desirable that the competitive sound be heard loudly. However, since it is difficult to install a microphone inside the field, the microphone is installed outside the field and directed inward to collect sound. Therefore, when the distance from the microphone to the generation position of the competitive sound is long, the competitive sound may be drowned out by cheers and the competitive sound may not be collected.

[0003] Patent Document 1 discloses a technique for appropriately selecting a microphone group for acquiring a plurality of sound collection signals used in a process of generating highly directional acoustic data using the plurality of sound collection signals collected by a plurality of microphones according to the position of an object.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technology disclosed in Patent Document 1, the selected microphone group to be processed changes in accordance with the change in the position of the object being sound-collected. When the microphone group to be processed changes, virtual viewpoint sound is generated by splicing together highly directional acoustic data from different microphone groups to be processed, which can result in discontinuities in the waveform at the connection points. As a result, the virtual viewpoint sound generated based on highly directional acoustic data containing discontinuities also contains discontinuities, which may be perceived as noise by the viewer. This disclosure aims to reduce noise in virtual viewpoint sound. [Means for solving the problem]

[0006] The information processing apparatus according to the present invention is characterized by comprising: position detection means for detecting the position of an object within a predetermined area; selection means for selecting a microphone group to be processed from among a plurality of microphone groups for collecting sound within the predetermined area, the microphone group whose positional relationship with the object satisfies predetermined conditions; noise reduction means for performing noise reduction processing on acoustic data in accordance with changes in the selected microphone group to be processed; a first generation means for generating highly directional acoustic data from sound collection signals obtained from a plurality of microphones for each microphone group to be processed; and a second generation means for generating sound for playback based on a plurality of acoustic data including the generated highly directional acoustic data. [Effects of the Invention]

[0007] This can reduce noise in virtual view audio. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the functional configuration of the information processing device in Embodiment 1. [Figure 2] This figure illustrates an example of the data structure of a table in Embodiment 1. [Figure 3A] This is a flowchart showing an example of processing in the information processing device according to Embodiment 1. [Figure 3B]This flowchart shows an example of the process for updating the microphone group selection flag in Embodiment 1. [Figure 4] This figure illustrates an example of microphone group arrangement and selection in Embodiment 1. [Figure 5] This figure illustrates an example of the data structure of a table in Embodiment 2. [Figure 6A] This is a flowchart showing an example of processing by the information processing device in Embodiment 2. [Figure 6B] This flowchart shows an example of the process for updating the microphone group selection flag in Embodiment 2. [Figure 7] This figure illustrates an example of microphone group arrangement and selection in Embodiment 2. [Figure 8] This figure shows an example of the hardware configuration of an information processing device. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. Note that the following embodiments do not limit the invention as defined in the claims. While multiple features are described in the embodiments, not all of these features are necessarily essential to the invention, and the features may be combined in any way. Furthermore, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0010] In the following explanation, the process of generating highly directional acoustic data (highly directional acoustic data) using multiple sound pickup signals from multiple microphones will also be referred to as highly directional acoustic generation processing. Furthermore, the set of microphones used to acquire the multiple sound pickup signals used in highly directional acoustic generation processing will also be referred to as a microphone group. For example, virtual viewpoint audio corresponding to a virtual viewpoint image is generated using highly directional acoustic data generated based on the sound pickup signals of the microphone group being processed.

[0011] <Embodiment 1> In Embodiment 1, the information processing device performs noise reduction processing in response to changes in the microphone group selected as the target for applying the highly directional sound generation processing. As an example, noise reduction processing is performed in response to changes between the microphone group selected for a processing frame and the microphone group selected for the frame immediately preceding it. This reduces noise caused by waveform discontinuities resulting from the use of highly directional sound data based on different microphone groups. In selecting microphone groups, microphone groups whose positional relationship with a subject (object within a predetermined area) within a predetermined shooting area satisfies predetermined conditions are selected as the microphone groups to be processed. In Embodiment 1, the predetermined condition is that the distance between the subject and the microphone group falls below a threshold. In this embodiment, the subject refers to an object or person that is the focus of attention within the shooting area, such as a ball or a player holding the ball.

[0012] Figure 1 is a block diagram showing an example of the functional configuration of the information processing device 100 in Embodiment 1. The information processing device 100 includes a position detection unit 101, a distance calculation unit 102, a speed detection unit 103, a microphone group selection unit 104, a change detection unit 105, a sound pickup signal acquisition unit 106, a first sound generation unit 107, a noise reduction processing unit 108, and a second sound generation unit 109. The information processing device 100 also includes an acoustic data storage unit 110 and a table storage unit 120.

[0013] The position detection unit 101 reads, for example, image data or video data used to generate a virtual viewpoint image, and detects an object (hereinafter also referred to as the subject) and its position within a predetermined area based on that data. In this embodiment, the virtual viewpoint image is an image generated based on multiple images obtained by photographing the shooting area from different directions, and is an image corresponding to the position and orientation of a specified virtual viewpoint. Any known detection method can be applied to detect the subject from the image. In this embodiment, the position of the subject is detected based on image data or video data, but this is not limited to this. For example, the position of a specific object measured by a position measuring device such as GPS may be detected as the position of the subject.

[0014] The distance calculation unit 102 calculates the distance between the position of the subject detected by the position detection unit 101 and each microphone group. The velocity detection unit 103 detects the velocity of the subject based on the time change of the position of the subject detected by the position detection unit 101. The microphone group selection unit 104 selects the microphone group to be processed for highly directional sound generation based on the distance calculated by the distance calculation unit 102. The change detection unit 105 detects whether or not there has been a change in the microphone group to be processed selected by the microphone group selection unit 104 in each processing frame.

[0015] The sound pickup signal acquisition unit 106 acquires sound pickup signals from all microphones registered in the microphone master table 130 and stores them as acoustic data in the acoustic data storage unit 110. The first acoustic generation unit 107 performs a highly directional acoustic generation process for the microphone group selected by the microphone group selection unit 104. The highly directional acoustic generation process takes multiple sound pickup signals from multiple microphones belonging to the same microphone group as input and generates highly directional acoustic data from these multiple sound pickup signals through signal processing. For the highly directional acoustic generation process, known signal processing techniques that ensure sensitivity in the target direction while reducing sensitivity in directions other than the target direction, such as beamforming, may be applied.

[0016] The noise reduction processing unit 108 performs noise reduction processing on the acoustic data in accordance with the change in the microphone group to be processed detected by the change detection unit 105. The noise reduction processing executed by the noise reduction processing unit 108 is, for example, processing such as fade-in and fade-out. By the noise reduction processing by this noise reduction processing unit 108, the noise caused by the discontinuity of the acoustic waveform is reduced. The second acoustic generation unit 109 generates virtual viewpoint sound using the sound collection signal acquired by the sound collection signal acquisition unit 106 and the high-directional acoustic data generated by the first acoustic generation unit 107. In the present embodiment, the virtual viewpoint sound becomes the sound for reproduction.

[0017] The acoustic data storage unit 110 stores the acoustic data generated by the sound collection signal acquisition unit 106, the first acoustic generation unit 107, and the noise reduction processing unit 108. The table storage unit 120 stores various types of information used in the processing in the information processing apparatus 100. For example, a microphone master table 130, a microphone group table 140, and a subject information table 150 are stored in the table storage unit 120.

[0018] The microphone master table 130 stores information regarding the microphones used for sound collection. The installed microphones are managed by the microphone master table 130. An example of the microphone master table 130 is shown in FIG. 2(A). As shown in FIG. 2(A), the microphone master table 130 stores a microphone ID 201 for uniquely identifying the installed microphone and a microphone group ID 202 for identifying the microphone group to which the microphone belongs. The high-directional acoustic generation processing is realized by performing signal processing using a plurality of sound collection signals collected by a plurality of microphones belonging to the same microphone group.

[0019] The microphone group table 140 stores information about the microphone group to which a microphone belongs. Microphone groups are managed by the microphone group table 140. An example of the microphone group table 140 is shown in Figure 2(B). As shown in Figure 2(B), the microphone group table 140 stores a microphone group ID 211 for uniquely identifying a microphone group, microphone positions 212 belonging to the microphone group, and a microphone group selection flag 213. The microphone position 212 is indicated by the X, Y, and Z coordinates of the microphone position belonging to the microphone group. In this embodiment, regardless of the number of microphones included in the microphone group, the position of the centroid of the microphones included in the microphone group is used as the microphone position. The microphone group selection flag 213 is a flag for determining whether or not a microphone group has been selected as a processing target for high-directional sound generation processing. In this embodiment, the flag is set to TRUE if the microphone group has been selected as a processing target for high-directional sound generation processing, and to FALSE if it has not been selected, and the flag for the current processing frame and the frame immediately preceding the current frame is stored.

[0020] The subject information table 150 stores information about the detected subject. Information about objects within a predetermined area detected as a subject (subject information) is managed by the subject information table 150. An example of the subject information table 150 is shown in Figure 2(C). The subject information table 150 includes a table that stores the subject ID 221, subject position 222, and subject speed 223 for uniquely identifying the subject, and a table that stores the microphone group ID 224 and distance 225. The table that stores the microphone group ID 224 and distance 225 is provided for each subject, and there are as many tables as there are subjects. The number of subjects and subject IDs may be set in advance, or when a new subject is detected by the position detection unit 101, a subject ID may be assigned and a row added to the table. The subject position 222 is indicated by the X, Y, and Z coordinates of the subject position, and stores the subject position in the current processing frame and the frame immediately preceding the current frame, as detected by the position detection unit 101. In this embodiment, the information of the centroid of the subject is used as the subject position. In this embodiment, the subject position in the current processing frame and the frame immediately preceding the current frame is stored, but this is not limited to this. For example, the subject position in frames beyond the current processing frame may be acquired and stored in advance. The subject velocity 223 stores the X, Y, and Z components of the subject velocity calculated by the velocity detection unit 103 based on the subject position information in the current frame and the frame immediately preceding the current frame. The microphone group ID 224 matches the microphone group ID 211 in the microphone group table 140. The distance 225 is the distance between the subject and each microphone group.

[0021] Next, the operation of the information processing device 100 will be described with reference to Figures 3A, 3B, and 4. Figures 3A and 3B are flowcharts illustrating an example of processing in Embodiment 1, from performing highly directional sound generation processing using sound signals picked up by microphones to generating virtual viewpoint sound. Figure 4 is a diagram illustrating an example of microphone group arrangement and an example of selecting the microphone group to be processed in Embodiment 1.

[0022] In Figure 3A, the processing in steps S301 to S316 is a loop process executed for each processing frame. Furthermore, the processing in steps S302 to S314 is a loop process executed for each microphone group, and the processing is performed for all microphone groups. That is, the processing in steps S302 to S314 is repeated for all microphone groups registered in the microphone group table 140. Figure 4 shows an example of microphone group placement, where microphone groups 431 to 444 are arranged around the outer perimeter of field 400. In the example shown in Figure 4, the processing in steps S302 to S314 is executed for microphone groups 431 to 444, respectively.

[0023] The processing in steps S303 to S305 is performed for all microphones belonging to the microphone group. In step S304, the sound pickup signal acquisition unit 106 acquires sound pickup signals for all microphones registered in the microphone master table 130 and stores them in the acoustic data storage unit 110. As a result of this process, the sound pickup signals acquired by all microphones belonging to the target microphone group in the loop processing of steps S302 to S314 are stored in the acoustic data storage unit 110. Note that any configuration is acceptable as long as the sound pickup signals are held in the storage area, for example, by storing the sound pickup signals as files.

[0024] In step S306, the process of updating the microphone group selection flag 213 in the microphone group table 140 is executed. Figure 3B is a flowchart showing an example of the microphone group selection flag update process in Embodiment 1 that is executed in step S306.

[0025] In Figure 3B, in step S321, the position detection unit 101 detects an object (subject) and its position within a predetermined area based on image data or motion data used to generate a virtual viewpoint image, and updates the position information 222 in the subject information table 150. For example, as shown in Figure 4, the position of subject 412 is detected. In this embodiment, the position of the subject is the centroid of the detected subject area, but it is not limited to this. For example, any position within the subject area (for example, the position closest to the microphone group) may be used.

[0026] In step S322, the distance calculation unit 102 calculates the distance between the position of the subject detected by the position detection unit 101 in step S321 and the target microphone group in the loop processing, and updates the distance 225 in the subject information table 150. The position of the microphone group is stored in the microphone position 212 of the microphone group table 140.

[0027] In step S323, the velocity detection unit 103 detects the velocity of the subject using the subject's position detected by the position detection unit 101, and updates the subject's velocity information 223 stored in the subject information table 150. In this embodiment, the velocity is calculated based on the subject's position in the processing frame and the previous frame, but is not limited to this. For example, it may be calculated as an average velocity or estimated velocity from the subject's position in multiple past frames.

[0028] In step S324, the microphone group selection unit 104 determines whether the distance between the microphone group and the subject position calculated in step S322 is less than a predetermined threshold. The microphone group selection unit 104 selects the microphone group whose calculated distance is less than the threshold as the microphone group to be processed. Therefore, if the microphone group selection unit 104 determines that the calculated distance is less than the threshold (YES in S324), the process in step S325 is executed. If the microphone group selection unit 104 determines that the calculated distance is not less than the threshold, i.e., that the calculated distance is greater than or equal to the threshold (NO in S324), the process in step S326 is executed.

[0029] In step S325, the microphone group selection unit 104 sets the microphone group selection flag 213 corresponding to the current frame in the microphone group table 140 to TRUE. In step S326, the microphone group selection unit 104 sets the microphone group selection flag 213 corresponding to the current frame in the microphone group table 140 to FALSE. In steps S325 and S326, the microphone group selection unit 104 updates the flag that was already stored as the microphone group selection flag 213 corresponding to the current frame, by storing it as the microphone group selection flag 213 corresponding to the previous frame.

[0030] In this embodiment, the threshold used in the determination process in step S324 is predetermined to be a single value, but is not limited to this. For example, different values ​​may be set for each processing frame according to the subject velocity calculated in step S323. Alternatively, the information processing device 100 may be separately equipped with a subject orientation detection unit (not shown) for detecting the orientation of a subject, and different thresholds may be set based on the angle from the subject orientation. Figure 4 shows the subject 412 in a certain processing frame N and the range 422 where the distance from the subject is within a predetermined threshold. Note that Figure 4 is shown in two dimensions for convenience, so the range 422 is shown as a circle, but the actual structure is three-dimensional, so the range 422 is approximately bounded by a sphere. Similarly, Figure 4 shows the subject 411 in the previous processing frame (N-1) and the range 421 indicating the threshold distance from the subject. In the example shown in Figure 4, processing frame N indicates that microphone groups 436, 437, and 438 have been selected as the microphone groups to be processed for highly directional sound generation processing. Furthermore, processing frame (N-1) indicates that microphone groups 435, 436, and 437 have been selected as the microphone groups to be processed for high-directional sound generation. In Figure 4, the microphone groups are shown as shapes for convenience, but in this embodiment, the parameters used to calculate the distance between the subject and the microphone group are the X, Y, and Z coordinates of their respective centroids. Therefore, in Figure 4, if the center point of the shape of each microphone group is included in the ranges 421 and 422, it is determined that the microphone group will be selected as the microphone group to be processed for high-directional sound generation.

[0031] The above describes the process of updating the microphone group selection flag, which is performed in step S306.

[0032] Returning to Figure 3A, in step S307, the microphone group selection unit 104 determines whether the microphone group selection flag in the current processing frame, which was updated in step S306, is TRUE or FALSE. If the microphone group selection unit 104 determines that the microphone group selection flag in the current frame is TRUE (YES in S307), the process in step S308 is executed. On the other hand, if the microphone group selection unit 104 determines that the microphone group selection flag in the current frame is FALSE (NO in S307), the process in step S311 is executed.

[0033] In step S308, the first sound generation unit 107 performs highly directional sound generation processing using the sound pickup signals from microphones belonging to the microphone group to be processed. The first sound generation unit 107 stores the highly directional sound data obtained as a result of the highly directional sound generation processing in the sound data storage unit 110. As with step S304, any configuration is acceptable as long as the highly directional sound data is held in the storage area, for example, by storing the highly directional sound data as a file.

[0034] In step S309, the change detection unit 105 determines whether the microphone group selection flag for the previous frame, held by the microphone group table 140, is TRUE or FALSE. If the change detection unit 105 determines that the microphone group selection flag for the previous frame is TRUE (YES in S309), the processing in step S310 is skipped and the processing in step S314 is executed. If the change detection unit 105 determines that the microphone group selection flag for the previous frame is FALSE (NO in S309), the processing in step S310 is executed. In other words, if the change detection unit 105 determines that a microphone group that was not the target of processing in the previous frame has become the target of processing in the current frame, the processing in step S310 is executed. In this embodiment, information on past frames is only retained for the previous frame, but information on any number of past frames may be retained. It may also be determined whether a predetermined number of frames have passed since a microphone group that was selected in a past frame was no longer selected. Similarly, it may be determined whether a predetermined number of frames have passed since a microphone group that was not selected in a past frame was selected.

[0035] In step S310, the noise reduction processing unit 108 applies a fade-in process to the highly directional acoustic data generated in step S308. Fade-in processing is a process that continuously increases the intensity of the sound over time. Fade-in processing maintains the continuity of the sound and reduces noise. In this embodiment, the fade-in processing is performed to be completed in the current frame, but it is not limited to this. For example, the fade-in processing may be set to be completed in any number of frames according to the subject speed stored in the subject information table 150.

[0036] In step S311, the change detection unit 105 determines, similar to step S309, whether the microphone group selection flag for the previous frame held in the microphone group table 140 is TRUE or FALSE. If the change detection unit 105 determines that the microphone group selection flag for the previous frame is TRUE (YES in S311), the process in step S312 is executed. That is, if the change detection unit 105 determines that a microphone group that was the target of processing in the previous frame is no longer the target of processing in the current frame, the process in step S312 is executed. If the change detection unit 105 determines that the microphone group selection flag for the previous frame is FALSE (NO in S311), the processes in steps S312 to S313 are skipped and the process in step S314 is executed.

[0037] In step S312, the first sound generation unit 107 performs highly directional sound generation processing using the sound pickup signals of microphones belonging to the microphone group to be processed, in the same manner as in step S308. The first sound generation unit 107 stores the highly directional sound data obtained as a result of the highly directional sound generation processing in the sound data storage unit 110.

[0038] In step S313, the noise reduction processing unit 108 applies a fade-out process to the highly directional acoustic data generated in step S312. The fade-out process is a process that continuously reduces the intensity of the sound over time. The fade-out process maintains the continuity of the sound and reduces noise. In this embodiment, as in step S310, the fade-out process is performed to be completed in the current frame, but this is not limited to this. For example, the fade-out process may be set to be completed in any number of frames according to the subject speed stored in the subject information table 150.

[0039] In step S315, the second sound generation unit 109 generates virtual viewpoint sound using the sound data stored in the sound data storage unit 110. For example, virtual viewpoint sound may be generated by synthesizing all the sound pickup signals and highly directional sound data stored in the sound data storage unit 110, or virtual viewpoint sound may be generated by synthesizing only the highly directional sound data without using the sound pickup signals.

[0040] In this embodiment, the noise reduction processing unit 108 applies a fade-in or fade-out process to the highly directional acoustic data generated in step S308, but is not limited to this. For example, when the second acoustic generation unit 109 generates virtual viewpoint acoustics, it may apply a fade-in or fade-out process to the sound pickup signals picked up from the microphones included in each microphone group, based on the microphone group selection flag in the microphone group table 140.

[0041] Furthermore, in this embodiment, highly directional sound generation processing is performed for all microphone groups determined to be below a threshold distance from the subject's position, but this is not limited to this. For example, the number of microphone groups to be processed by highly directional sound generation processing may be limited. For example, a predetermined number of microphone groups may be selected in order of shortest distance from the subject's position and subjected to highly directional sound generation processing. Alternatively, if none of the microphone groups are below the threshold distance, a predetermined number of microphone groups may be selected in order of shortest distance from the subject's position and subjected to highly directional sound generation processing.

[0042] Furthermore, although this embodiment describes the subject as one, there may be multiple subjects. When multiple subjects are detected and processed, by performing steps S302 to S314 for each subject (steps S303 to S305 are common to all subjects and therefore do not need to be performed), it becomes possible to generate virtual viewpoint sound based on the highly directional acoustic data generated for each subject.

[0043] In Embodiment 1, noise reduction processing is performed on the acoustic data in accordance with the change in the microphone group being processed between the processing frame and the frame immediately preceding it. This reduces noise caused by waveform discontinuities that occur when using highly directional acoustic data based on different microphone groups being processed. Therefore, it is possible to reduce noise in the virtual viewpoint acoustics generated as playback acoustics, thereby realizing virtual viewpoint acoustics with reduced auditory discomfort.

[0044] <Embodiment 2> In Embodiment 1 described above, the distance between the microphone group and the subject in the processing frame and the frame immediately preceding the processing frame is used to determine which microphone group will undergo high-directional sound generation processing. In Embodiment 2, two thresholds, a first threshold and a second threshold (first threshold < second threshold), are set in each processing frame, and the decision of whether or not to select a microphone group for high-directional sound generation processing is made by comparing the distance between the microphone group and the subject with each threshold. In other words, unlike Embodiment 1, in Embodiment 2, two types of microphone groups to be processed are selected within the same processing frame as microphone groups whose positional relationship with the subject satisfies predetermined conditions. Note that in this embodiment, two thresholds are set, but three or more may be used.

[0045] The functional configuration of the information processing device 100 in Embodiment 2 is the same as that of the information processing device 100 in Embodiment 1 shown in Figure 1, so its explanation will be omitted. In Embodiment 2, microphone groups are managed by a microphone group table 140, as shown in Figure 5 as an example. Figure 5 is a diagram showing an example of the microphone group table 140 in Embodiment 2. As shown in Figure 5, the microphone group table 140 stores a microphone group ID 501 for uniquely identifying a microphone group, microphone positions 502 belonging to the microphone group, and a microphone group selection flag 503. The microphone group ID 501 and microphone position 502 are the same as the microphone group ID 211 and microphone position 212 shown in Figure 2(B). The microphone group selection flag 503 is a flag for determining whether or not a microphone group has been selected as a processing target for high-directional sound generation processing. Unlike Embodiment 1, the microphone group selection flag 503 stores flags corresponding to the first threshold and the second threshold, respectively. The microphone master table 130 and subject information table 150 in Embodiment 2 are the same as those in Embodiment 1.

[0046] Figures 6A and 6B are flowcharts illustrating an example of the processing in Embodiment 2, from the generation of highly directional sound using the sound signal picked up by the microphone to the generation of virtual viewpoint sound. Figure 7 illustrates an example of microphone group arrangement and an example of selecting the microphone group to be processed in Embodiment 2.

[0047] Steps S601-S605, S608, S610, and S612-S614 in Figure 6A are the same processes as steps S301-S305, S308, S312, and S314-S316 in Figure 3A, respectively. Similarly, steps S621-S623 in Figure 6B are the same processes as steps S321-S323 in Figure 3B, respectively. Therefore, their explanations are omitted.

[0048] In step S606, the process of updating the microphone group selection flags corresponding to the first and second thresholds in the microphone group table 140 is executed. Figure 6B is a flowchart showing an example of the microphone group selection flag update process in Embodiment 2 that is executed in step S606.

[0049] In Figure 6B, in step S624, the microphone group selection unit 104 determines whether the distance between the microphone group and the subject position calculated in step S622 is less than a first threshold. The microphone group selection unit 104 selects the microphone group whose calculated distance is less than the first threshold as the microphone group to be processed. Therefore, if the microphone group selection unit 104 determines that the calculated distance is less than the first threshold (YES in S624), the process in step S625 is executed. If the microphone group selection unit 104 determines that the calculated distance is not less than the first threshold, i.e., that the calculated distance is greater than or equal to the first threshold (NO in S624), the process in step S626 is executed.

[0050] In step S625, the microphone group selection unit 104 sets both the flags corresponding to the first threshold and the second threshold of the microphone group selection flag 503 in the microphone group table 140 to TRUE.

[0051] In step S626, the microphone group selection unit 104 determines whether the distance between the microphone group and the subject position calculated in step S622 is less than the second threshold. The value of the second threshold is greater than the value of the first threshold. The microphone group selection unit 104 selects the microphone group whose calculated distance is less than the second threshold as the microphone group to be processed. Therefore, if the microphone group selection unit 104 determines that the calculated distance is less than the second threshold, that is, the calculated distance is greater than or equal to the first threshold but less than the second threshold (YES in S626), the process in step S627 is executed. If the microphone group selection unit 104 determines that the calculated distance is not less than the second threshold, that is, the calculated distance is greater than or equal to the second threshold (NO in S626), the process in step S628 is executed.

[0052] In step S627, the microphone group selection unit 104 sets the flag corresponding to the first threshold of the microphone group selection flag 503 in the microphone group table 140 to FALSE and the flag corresponding to the second threshold to TRUE. In step S628, the microphone group selection unit 104 sets both the flags corresponding to the first threshold and the second threshold of the microphone group selection flag 503 in the microphone group table 140 to FALSE.

[0053] Similar to Embodiment 1, the thresholds used in the determination process in steps S624 and S626 in this embodiment are predetermined, but are not limited to this. For example, different values ​​may be set for each processing frame according to the subject speed calculated in step S623. Figure 7 shows the range 721 and the range 722 in which the distance from the subject 711 to the subject is within the first threshold and the distance from the subject to the second threshold, respectively, in a given processing frame. In the example shown in Figure 7, microphone groups 736, 737, and 738 are selected as the microphone groups to be processed, corresponding to the first threshold. In addition, as the microphone groups corresponding to the second threshold, microphone groups 736, 737, and 738, as well as microphone group 735, are selected as the microphone groups to be processed. Similar to the example shown in Figure 4, in Figure 7, it is determined that a microphone group is selected as a microphone group to be processed if the center point of the shape of each microphone group is included in ranges 721 and 722.

[0054] The above describes the processing details of the microphone group selection flag update process performed in step S606.

[0055] Returning to Figure 6A, in step S607, the microphone group selection unit 104 determines whether the microphone group selection flag corresponding to the first threshold updated in step S606 is TRUE or FALSE. If the microphone group selection unit 104 determines that the microphone group selection flag corresponding to the first threshold is TRUE (YES in S607), the process in step S608 is executed. On the other hand, if the microphone group selection unit 104 determines that the microphone group selection flag corresponding to the first threshold is FALSE (NO in S607), the process in step S609 is executed.

[0056] In step S609, the microphone group selection unit 104 determines whether the microphone group selection flag corresponding to the second threshold updated in step S606 is TRUE or FALSE. If the microphone group selection unit 104 determines that the microphone group selection flag corresponding to the second threshold is TRUE (YES in S609), the process in step S610 is executed. On the other hand, if the microphone group selection unit 104 determines that the microphone group selection flag corresponding to the second threshold is FALSE (NO in S609), the processes in steps S610 to S611 are skipped and the process in step S612 is executed.

[0057] In step S611, the noise reduction processing unit 108 applies a gain adjustment process to the highly directional acoustic data generated in step S610. The gain adjustment process is performed by multiplying the amplitude of the highly directional acoustic by a gain coefficient G. The gain coefficient G is calculated, for example, using the following formula, where T1 is the value of the first threshold, T2 is the value of the second threshold, and D is the distance calculated in S622. G = ((T2 - D) / (T2 - T1)) / 100 In other words, the amplitude of microphone groups located between the first threshold value and the second threshold value is set to 100% and adjusted proportionally to the distance.

[0058] In this embodiment, the gain adjustment process is performed as described above, but it is not limited to this. For example, in the example above, the adjustment is proportional to the distance, but the gain coefficient may also be determined by applying an exponential function.

[0059] According to Embodiment 2, noise can be reduced by setting two thresholds in each processing frame and performing noise reduction processing within a single processing frame according to the difference in the microphone groups selected by each threshold. Therefore, it is possible to reduce the noise in the virtual viewpoint sound generated as playback sound, and to realize virtual viewpoint sound with reduced auditory discomfort.

[0060] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0061] For example, the information processing device 100 in Embodiment 1 and Embodiment 2 can be implemented by a computer as shown in Figure 8, and its CPU 801 performs the operations in Embodiment 1 and Embodiment 2. Figure 8 is a block diagram showing examples of computer hardware configurations applicable to the information processing device according to each embodiment.

[0062] The CPU 801 controls the entire computer using computer programs and data stored in the RAM 802 and ROM 803, and also executes the processes described above, which are performed by the information processing devices according to each embodiment described above. In other words, the CPU 801 functions as the functional unit shown in Figure 1. The RAM 802 has an area for temporarily storing computer programs and data loaded from the external storage device 806, data acquired from the outside via the I / F (interface) 807, etc. Furthermore, the RAM 802 has a work area used by the CPU 801 when executing various processes. That is, the RAM 802 can store various data such as microphone information and microphone parameters. It can also manage acoustic data. The ROM 803 stores computer setting data, boot programs, etc.

[0063] The operation unit 804 is composed of, for example, a keyboard and mouse, and allows the computer user to input various instructions to the CPU 801 through operation. The output unit 805 is composed of, for example, a liquid crystal display, and displays the processing results by the CPU 801. Note that the operation unit 804 and the output unit 805 are not necessarily required, and input and output may be performed via the I / F 807. The external storage device 806 is a large-capacity information storage device, such as a hard disk drive. The external storage device 806 stores the OS (operating system) and computer programs that enable the CPU 801 to implement the functions of each part shown in Figure 1. Furthermore, the external storage device 806 may also store each sound-collected signal to be processed. Acoustic data can also be stored in the external storage device 806. The computer programs and data stored in the external storage device 806 are loaded into the RAM 802 as appropriate according to the control of the CPU 801 and become targets for processing by the CPU 801. I / F807 can be connected to networks such as LANs and the Internet, as well as other devices such as projection and display devices. This computer can acquire and transmit various types of information via I / F807. 808 is a bus that connects the aforementioned functional units in a communication-enabled manner.

[0064] It should be noted that the embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0065] This embodiment includes the following configurations and methods. (Composition 1) A position detection means for detecting the position of an object within a predetermined area, A selection means for selecting a microphone group to be processed from among a plurality of microphone groups for capturing sound within the predetermined area, the microphone group whose positional relationship with the object satisfies predetermined conditions, Noise reduction means that performs noise reduction processing on acoustic data in accordance with changes in the selected microphone group to be processed, For each group of microphones to be processed, a first generation means generates highly directional acoustic data from sound pickup signals obtained from multiple microphones, An information processing apparatus characterized by having a second generation means for generating sound for reproduction based on a plurality of acoustic data including the generated highly directional acoustic data. (Configuration 2) It has a change detection means for detecting changes in the selected microphone group to be processed, The information processing apparatus according to configuration 1, characterized in that the noise reduction means performs noise reduction processing in accordance with the change in the microphone group to be processed detected by the change detection means. (Composition 3) The information processing apparatus according to configuration 1 or 2, characterized in that the selection means selects the microphone group whose distance from the object to the microphone group is less than a threshold as the microphone group to be processed. (Composition 4) The noise reduction means is If the microphone group that was to be processed becomes a microphone group that is not to be processed, a fade-out process is performed on the audio data of that microphone group. The information processing device according to configuration 3, characterized in that when a microphone group that is not subject to processing becomes a microphone group subject to processing, a fade-in process is performed on the acoustic data of that microphone group. (Composition 5) The system has a velocity detection means for detecting the velocity of the aforementioned object, The information processing apparatus according to configuration 3 or 4, characterized in that the threshold is set based on the velocity of the detected object. (Composition 6) The selection means selects the microphone group to be processed in which the distance between the object and the microphone group is less than a second threshold greater than a first threshold, The information processing apparatus according to configuration 1 or 2, characterized in that the noise reduction means performs gain adjustment on the acoustic data of the microphone group whose distance from the object is greater than or equal to the first threshold and less than the second threshold. (Composition 7) The information processing apparatus according to configuration 6, characterized in that the noise reduction means adjusts the gain with a gain coefficient corresponding to the distance to the object. (Composition 8) The system has a velocity detection means for detecting the velocity of the aforementioned object, The information processing apparatus according to configuration 6 or 7, characterized in that the first threshold and the second threshold are set based on the velocity of the detected object. (Composition 9) The system has a velocity detection means for detecting the velocity of the aforementioned object, The information processing apparatus according to any one of configurations 1 to 8, characterized in that the noise reduction means applies different processing depending on the speed of the detected object. (Composition 10) The information processing apparatus according to any one of configurations 1 to 9, characterized in that the noise reduction means applies different processing depending on the plurality of microphone groups to be processed selected by the selection means. (Method 1) A position detection step for detecting the position of an object within a predetermined area, A selection step of selecting a microphone group to be processed from among a plurality of microphone groups for capturing sound within the predetermined area, the microphone group whose positional relationship with the object satisfies predetermined conditions, A noise reduction step which performs noise reduction processing on acoustic data in accordance with changes in the selected microphone group to be processed, For each microphone group to be processed, a first generation step is performed to generate highly directional acoustic data from sound pickup signals obtained from multiple microphones, A control method for an information processing device, characterized by comprising a second generation step of generating sound for reproduction based on a plurality of acoustic data including the generated highly directional acoustic data. (Program 1) A position detection step that detects the position of an object within a predetermined area, A selection step of selecting a microphone group to be processed from among a plurality of microphone groups for capturing sound within the predetermined area, the microphone group whose positional relationship with the object satisfies predetermined conditions, A noise reduction step which performs noise reduction processing on acoustic data in accordance with changes in the selected microphone group to be processed, For each microphone group to be processed, a first generation step is performed to generate highly directional acoustic data from sound pickup signals obtained from multiple microphones, A program for causing a computer to perform a second generation step of generating sound for playback based on a plurality of acoustic data including the generated highly directional acoustic data. [Explanation of Symbols]

[0066] 101: Position detection unit 102: Distance calculation unit 103: Speed ​​detection unit 104: Microphone group selection unit 105: Change detection unit 106: Sound pickup signal acquisition unit 107: First sound generation unit 108: Noise reduction processing unit 109: Second sound generation unit 110: Sound data storage unit 120: Table storage unit 130: Microphone master table 140: Microphone group table 150: Subject information table

Claims

1. A position detection means for detecting the position of an object within a predetermined area, A selection means for selecting a microphone group to be processed from among a plurality of microphone groups for capturing sound within the predetermined area, the microphone group whose positional relationship with the object satisfies predetermined conditions, Noise reduction means that performs noise reduction processing on acoustic data in accordance with changes in the selected microphone group to be processed, For each group of microphones to be processed, a first generation means generates highly directional acoustic data from sound pickup signals obtained from multiple microphones, An information processing apparatus characterized by having a second generation means for generating sound for reproduction based on a plurality of acoustic data including the generated highly directional acoustic data.

2. It has a change detection means for detecting changes in the selected microphone group to be processed, The information processing apparatus according to claim 1, characterized in that the noise reduction means performs noise reduction processing in accordance with the change in the microphone group to be processed detected by the change detection means.

3. The information processing apparatus according to claim 1, characterized in that the selection means selects the microphone group whose distance from the object to the microphone group is less than a threshold as the microphone group to be processed.

4. The noise reduction means is If the microphone group that was to be processed becomes a microphone group that is not to be processed, a fade-out process is performed on the audio data of that microphone group. The information processing apparatus according to claim 3, characterized in that if a microphone group that is not subject to processing becomes a microphone group subject to processing, a fade-in process is performed on the acoustic data of that microphone group.

5. The system has a velocity detection means for detecting the velocity of the aforementioned object, The information processing apparatus according to claim 3, characterized in that the threshold is set based on the velocity of the detected object.

6. The selection means selects the microphone group to be processed in which the distance between the object and the microphone group is greater than a first threshold and less than a second threshold. The information processing apparatus according to claim 1, characterized in that the noise reduction means performs gain adjustment on the acoustic data of the microphone group whose distance from the object is greater than or equal to the first threshold and less than the second threshold.

7. The information processing apparatus according to claim 6, characterized in that the noise reduction means adjusts the gain with a gain coefficient corresponding to the distance to the object.

8. The system has a velocity detection means for detecting the velocity of the aforementioned object, The information processing apparatus according to claim 6, characterized in that the first threshold and the second threshold are set based on the velocity of the detected object.

9. The system has a velocity detection means for detecting the velocity of the aforementioned object, The information processing apparatus according to claim 1, characterized in that the noise reduction means applies different processing depending on the speed of the detected object.

10. The information processing apparatus according to claim 1, characterized in that the noise reduction means applies different processing depending on the plurality of microphone groups selected by the selection means.

11. A position detection step for detecting the position of an object within a predetermined area, A selection step of selecting a microphone group to be processed from among a plurality of microphone groups for capturing sound within the predetermined area, the microphone group whose positional relationship with the object satisfies predetermined conditions, A noise reduction step which performs noise reduction processing on acoustic data in accordance with changes in the selected microphone group to be processed, For each microphone group to be processed, a first generation step is performed to generate highly directional acoustic data from sound pickup signals obtained from multiple microphones, A control method for an information processing device, characterized by comprising a second generation step of generating sound for reproduction based on a plurality of acoustic data including the generated highly directional acoustic data.

12. A position detection step that detects the position of an object within a predetermined area, A selection step of selecting a microphone group to be processed from among a plurality of microphone groups for capturing sound within the predetermined area, the microphone group whose positional relationship with the object satisfies predetermined conditions, A noise reduction step which performs noise reduction processing on acoustic data in accordance with changes in the selected microphone group to be processed, For each microphone group to be processed, a first generation step is performed to generate highly directional acoustic data from sound pickup signals obtained from multiple microphones, A program for causing a computer to perform a second generation step of generating sound for playback based on a plurality of acoustic data including the generated highly directional acoustic data.

Images