SOUND PROCESSING METHOD AND SOUND PROCESSING APPARATUS

Abstract

To provide a sound image when a desired speaker is arranged in a space where the speaker is not arranged to the user without actually arranging the speaker.SOLUTION: A sound processing method acquires an image of an acoustic space, sets a plane in the acoustic space from the image, sets a virtual speaker in the image of the acoustic space, calculates the sound pressure distribution with respect to the plane from the characteristics of the virtual speaker, and overlays the sound pressure distribution on the plane.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] An embodiment of the present invention relates to a technique for confirming a sound image in a space. [Background technology]

[0002] The audio control device of Patent Document 1 is installed in a sound field where the volume level is to be measured. The audio control device detects and stores the output volume level of the speaker.

[0003] The audio control device graphs the output volume level at a designated listening position in the sound field. The audio control device displays the graph on a two-dimensional screen. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-17880 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the audio control device described in Patent Document 1 detects the output level of a speaker by installing an actual speaker in a space. Therefore, a conventional device such as the audio control device described in Patent Document 1 cannot output a sound image such as a sound pressure distribution when a desired speaker is installed in a space such as a room where no speaker is installed.

[0006] Therefore, an object of the present invention is to provide a user with a sound image that would be produced if desired speakers were placed in a space where no speakers have been placed, without the user actually placing the speakers. [Means for solving the problem]

[0007] The sound processing method involves obtaining an image of an acoustic space, setting a plane in the acoustic space from the image, setting a virtual speaker within the image of the acoustic space, calculating a sound pressure distribution on the plane from characteristics of the virtual speaker, and overlaying the sound pressure distribution on the plane. Effect of the Invention

[0008] The sound processing method can provide a user with a sound image that would be produced if desired speakers were placed in a space where no speakers are placed, without the user actually placing the speakers. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a hardware configuration of a sound processing device. [Diagram 2] FIG. 2 is a functional block diagram showing the configuration of the sound processing device according to the first embodiment. [Diagram 3] FIG. 3(A) is a diagram showing an example of an image before the virtual speaker and the sound pressure distribution are synthesized, and FIG. 3(B) is a diagram showing an example of an image after the virtual speaker and the sound pressure distribution are synthesized. [Figure 4] FIG. 4 is a schematic plan view of a space in which virtual speakers are set. [Diagram 5] FIG. 5 is a diagram showing an example of a manner in which sound pressure distribution is calculated for a plurality of planes. [Figure 6] FIG. 6 is a flowchart showing a method for synthesizing virtual speakers and sound pressure distribution into an image of a space. [Figure 7] FIG. 7 is a functional block diagram showing the configuration of the sound processing device according to the second embodiment. [Figure 8] FIG. 8 is a functional block diagram showing a configuration of a reproduction part of an anticipated listening sound of the sound processing device according to the embodiment. [Figure 9] FIG. 9 is a flow chart showing a method for generating an expected listening sound. [Figure 10] FIG. 10 is a flowchart showing a method of calculating the propagation characteristics shown in FIG. [Figure 11]FIG. 11 is a flowchart showing a method of generating the sound signal shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] [Hardware configuration of sound processing device] Fig. 1 is a diagram showing an example of a hardware configuration of a sound processing device. As shown in Fig. 1, the sound processing device 10 includes a calculation device 70 such as a computer as a hardware configuration. The calculation device 70 includes a CPU 71, a memory 72, an interface 731, a video output port 732, a sound output port 733, and a bus 700. The bus 700 connects the CPU 71, the memory 72, the interface 731, the video output port 732, and the sound output port 733.

[0011] The memory 72 stores various programs, data, etc., including programs for operating each part of the sound processing device. The CPU 71 executes the various programs stored in the memory 72 to realize the sound processing device. The memory 72 is not limited to storing various programs and data, and an external storage device connected via the interface 731 or a server connected to a network may store various programs and data. In this case, the CPU 71 reads out various programs and data from the server, etc., via the interface 731.

[0012] The interface 731 executes data communication with the operation input device 74 and the camera 75. The interface 731 also executes communication control with not only the operation input device 74 and the camera 75 but also various external devices and external networks connected to the arithmetic device 70.

[0013] The operation input device 74 is, for example, a touch panel. The operation input device 74 outputs instruction data and the like corresponding to the accepted operation to the CPU 71 via the interface 731. The instruction data is, for example, an instruction to start a composite image of sound pressure distribution, an instruction for the placement position of a virtual speaker, etc. The camera 75 outputs captured image data to the CPU 71 via the interface 731.

[0014] The video output port 732 is connected to the display device 76. The arithmetic unit 70, i.e., the sound processing device, outputs image data including sound pressure distribution (synthetic image data described below) to the display device 76 via the video output port 732. Note that when image data including sound pressure distribution is not output, the video output port 732 and the display device 76 can be omitted.

[0015] The sound output port 733 is connected to a sound emitting device 77. The sound emitting device 77 is, for example, a headphone speaker, a stationary stereo speaker, or the like. The calculation device 70, i.e., the sound processing device, outputs a sound (anticipated listening sound) calculated as being emitted from a virtual speaker to a desired position, to the sound emitting device 77. When the anticipated listening sound is not output, the sound output port 733 and the sound emitting device 77 can be omitted.

[0016] [Embodiment 1] (Functional configuration of sound processing device) Fig. 2 is a functional block diagram showing a configuration of a sound processing device according to embodiment 1. Fig. 3(A) is a diagram showing an example of an image before synthesis of a virtual speaker and a sound pressure distribution, and Fig. 3(B) is a diagram showing an example of an image after synthesis of the virtual speaker and the sound pressure distribution.

[0017] As shown in FIG. 2, the sound processing device 10 includes an image acquisition unit 11, a plane setting unit 12, a virtual speaker setting unit 13, a sound pressure distribution calculation unit 14, a synthetic image generation unit 15, and a speaker characteristic storage unit 20.

[0018] The image acquisition unit 11 acquires an image of the space (the "acoustic space" of the present invention) for which sound pressure distribution is to be calculated from the above-mentioned camera 75 or the like (see FIG. 3(A)). For example, the image acquisition unit 11 acquires an image 90P of a room 91 for which sound pressure distribution is to be calculated. The image acquisition unit 11 outputs the image 90P to the plane setting unit 12.

[0019] The plane setting unit 12 sets a plane for calculating sound pressure distribution from the image 90P. For example, the plane setting unit 12 sets a floor surface 911 in the room 91 from the image 90P. Specifically, the plane setting unit 12 can set a plane for calculating sound pressure distribution, specifically the floor surface 911 as a specific example, by various methods described later. At this time, the plane setting unit 12 sets xy coordinates on the floor surface 911, for example. The plane setting unit 12 outputs the xy coordinates of the floor surface 911 to the composite image generation unit 15 together with the image.

[0020] The virtual speaker setting unit 13 sets the position and type of the virtual speaker. The position of the virtual speaker is set by specifying the arrangement position of the virtual speaker in the displayed image. The arrangement position is specified, for example, by an operation input from the operation input device 74.

[0021] The virtual speaker setting unit 13 sets the position of the virtual speaker 92, for example, by xy coordinates set on the floor surface 911. The front direction of the virtual speaker 92 can be set by designation using the operation input device 74, for example.

[0022] The type of virtual speaker 92 is set by, for example, displaying an image (window) for selecting the type of virtual speaker on the image and specifying the type. The type is specified by, for example, an operation input from the operation input device 74.

[0023] The virtual speaker setting unit 13 outputs the position P92 of the virtual speaker and the type of the virtual speaker to the sound pressure distribution calculation unit .

[0024] The speaker characteristic storage unit 20 stores speaker characteristics for each type of virtual speaker. The speaker characteristics are configured by frequency characteristics over the entire circumference (360 degrees) around the speaker. More specifically, the entire circumference (360 degrees) is resolved at a predetermined resolution to set a plurality of angles. Then, a frequency characteristic is set for each of the plurality of angles. The speaker characteristics are configured by the frequency characteristics set for each of the plurality of angles. The speaker characteristics may be WAV data (time waveform) indicating the frequency characteristics, or may be a frequency spectrum.

[0025] The sound pressure distribution calculation unit 14 acquires the speaker characteristics corresponding to the virtual speaker from the speaker characteristics storage unit 20 based on the type of the virtual speaker.

[0026] Using the speaker characteristics and the virtual speaker position P92, the sound pressure distribution calculation unit 14 calculates the sound pressure at a plurality of positions on the floor surface 911. The sound pressure is the sound pressure of a direct sound.

[0027] FIG. 4 is a schematic plan view of a space in which a virtual speaker is set. More specifically, the sound pressure distribution calculation unit 14 sets a sound pressure calculation position Pp on the floor surface 911. The sound pressure distribution calculation unit 14 calculates a distance DIS between the sound pressure calculation position Pp and the virtual speaker position P92. The sound pressure distribution calculation unit 14 also calculates an azimuth angle θ of the sound pressure calculation position Pp with the virtual speaker position P92 as a reference point. For example, as shown in FIG. 4, the sound pressure distribution calculation unit 14 sets the azimuth angle θ=0 in the front direction of the virtual speaker, as an angle proceeding counterclockwise when the floor surface 911 is viewed from the front. Note that the method of setting the azimuth angle θ is not limited to this. Also, here, the reference point is set at the center of the virtual speaker 92 when viewed in a plane. However, the position of the reference point with respect to the virtual speaker 92 differs depending on the shape of the speaker that is a model of the virtual speaker 92. For example, the reference point may be set in front of the virtual speaker 92, or may be set in back of the virtual speaker 92. Furthermore, if the speaker that is the model of the virtual speaker 92 is a type that reflects sound on the ceiling 913 or the wall 912, the position of this reflection point becomes the position of the reference point. Note that the setting of the reference point is not limited to these, and can be set appropriately depending on the model of the virtual speaker 92, the speaker shape, etc.

[0028] The sound pressure distribution calculation unit 14 extracts the component of the azimuth angle θ in the speaker characteristics from the azimuth angle θ. This allows the sound pressure distribution calculation unit 14 to calculate the sound pressure (reference sound pressure) in the direction of the azimuth angle θ and at a distance of 1 m from the virtual speaker.

[0029] The sound pressure distribution calculation unit 14 performs a correction in the distance direction for the reference sound pressure using the distance DIS. More specifically, the sound pressure distribution calculation unit 14 performs a correction using a sound ray method such that the sound pressure attenuates inversely proportional to the square of the distance. That is, the sound pressure distribution calculation unit 14 performs a correction using k (sound pressure at distance DIS) / (DIS) 2 Here, k is a coefficient, which is basically set to "1", but can be adjusted according to temperature and humidity. This allows the sound pressure distribution calculation unit 14 to calculate the sound pressure taking into account the effect of sound energy absorption by the air.

[0030] The sound pressure distribution calculation unit 14 performs such sound pressure calculation processing for a plurality of positions set on the floor surface 911. For example, the sound pressure distribution calculation unit 14 performs sound pressure calculation processing for the entire floor surface 911 or within a desired range on the floor surface 911. Note that the desired range can be specified by, for example, an operation input via the operation input device 74.

[0031] The sound pressure distribution calculation unit 14 outputs the sound pressures calculated at the multiple positions, that is, the sound pressure distribution, to the synthetic image generation unit 15. At this time, the sound pressure distribution calculation unit 14 outputs the sound pressure distribution in association with xy coordinates.

[0032] 3B, the synthetic image generating unit 15 synthesizes the virtual speaker 92 and the sound pressure distribution 93 with the image 90P to generate a synthetic image 90. At this time, the synthetic image generating unit 15 synthesizes the image 90P by matching the x and y coordinates associated with the image 90P, the x and y coordinates associated with the virtual speaker 92, and the x and y coordinates associated with the sound pressure distribution 93.

[0033] As a result, as shown in FIG. 3(B), the synthetic image 90 is an image in which the virtual speaker 92 is placed at a specified position in the image 90P of the space (image of the room 91), and a corresponding sound pressure distribution 93 is superimposed on the floor surface 911. As a result, even if the real speaker is not actually placed, the user can install the virtual speaker 92 corresponding to the real speaker and visually grasp the sound pressure of the space. Here, as shown by hatching in FIG. 3(B), the sound pressure distribution 93 is an image whose display mode changes depending on the magnitude of the sound pressure. As a result, the user can visually grasp the sound pressure of the space when the virtual speaker 92 is installed more easily.

[0034] In the above description, the sound pressure distribution is not limited to the surface of the floor 911, i.e., the plane at the position of 0 in the z direction perpendicular to the x-y coordinate plane, but is also calculated for a plane at a predetermined height (z coordinate value other than 0). That is, the plane for calculating the sound pressure can be set to a plane at a predetermined height from the floor 911. In this case, the sound pressure distribution calculation unit 14 may perform distance correction taking into account the position on the z axis.

[0035] In the above description, the sound processing device 10 calculates a sound pressure distribution for one plane. However, the sound processing device 10 can also calculate a sound pressure distribution for each of a plurality of planes.

[0036] Fig. 5 is a diagram showing an example of a manner in which sound pressure distribution is calculated for a plurality of planes. As shown in Fig. 5, a concert hall or the like may have a plurality of floors, such as a first floor, a second floor, and a third floor. In this case, for example, as shown in Fig. 5, the sound processing device 10 sets the first floor as a first floor surface 911F, the second floor as a second floor surface 911S, and the third floor as a third floor surface 911T. Then, the sound processing device 10 calculates sound pressure distribution for each of the first floor surface 911F, the second floor surface 911S, and the third floor surface 911T using the above-mentioned method.

[0037] Also, the above-described configuration shows an aspect in which one virtual speaker 92 is arranged. However, the number of virtual speakers 92 is not limited to one, and may be multiple. In this case, the sound pressure distribution calculation unit 14 can calculate the sound pressure at each position by adding up the sound pressures from the multiple virtual speakers 92 in the frequency domain.

[0038] The above-described sound processing device 10 has been described as calculating the sound pressure distribution of direct sound. However, the sound processing device 10 can calculate the sound pressure distribution including indirect sound by using the setting parameters of the early reflection sound and reverberation sound of the space (room 91).

[0039] (Sound processing method (method of generating synthetic images)) In the above description, the method of generating a synthetic image, more specifically, the process of superimposing the virtual speaker 92 and the sound pressure distribution on the image of the space (room 91), is realized by a plurality of functional units. However, when a synthetic image is generated by executing a program in the arithmetic device 70 such as a computer as described above, it is sufficient to generally execute the process shown in the flowchart of FIG. 6. FIG. 6 is a flowchart showing a method of synthesizing the virtual speaker and the sound pressure distribution on the image of the space. Note that the specific contents of each process have been described above, so only a general flow will be described here.

[0040] The arithmetic device 70 acquires an image of the space from the camera 75 or the like (S11). The arithmetic device 70 sets a plane for the image of the space (S12). The arithmetic device 70 sets a virtual speaker 92 (S13).

[0041] The arithmetic device 70 calculates a sound pressure distribution from the positions of the virtual speakers 92 and the types (speaker characteristics) of the virtual speakers 92 (S14). The arithmetic device 70 synthesizes the virtual speakers 92 and the sound pressure distribution with an image of the space to generate a synthesized image (S15).

[0042] (Specific method of setting the plane (floor surface 911 as an example) for calculating sound pressure distribution) (A) The plane setting unit 12 labels each object shown in the image 90P as a floor, a sidewall, and a ceiling based on the characteristics of each object. Then, the plane setting unit 12 sets the surface of the object labeled as a floor as the floor surface 911. At this time, the plane setting unit 12 may set the surface of an object connected to the object labeled as a sidewall and the surface of an object facing the object labeled as a ceiling as the floor surface 911.

[0043] (B) The plane setting unit 12 assumes that the shape of the floor surface 911 is rectangular, and extracts an approximately rectangular shape extending in the horizontal and depth directions of the image 90P from the image 90P. The plane setting unit 12 sets this approximately rectangular plane on the floor surface 911.

[0044] (C) The user is provided with a movement detection device. The movement detection device detects the movement of the user's device. The user moves along the outer periphery (side) of the floor surface 911. The movement detection device detects the movement of the user's device and outputs the detection result to the plane setting unit 12. The plane setting unit 12 sets the floor surface 911 from the detection result. Note that the movement detection device may not detect the continuous movement of the user's device but may detect the position of the user's device. In this case, the user causes the movement detection device to detect the position at a vertex (corner) of the floor surface 911. The plane setting unit 12 sets, for example, an area surrounded by each specified position on the floor surface 911.

[0045] (D) The operation input device 74 accepts the specification of a position on the image 90P. The user specifies the position of a vertex (corner) of the floor surface 911 using the operation input device 74. The plane setting unit 12 sets, for example, an area surrounded by the specified positions on the floor surface 911. Alternatively, the operation input device 74 accepts the specification of an area. The user specifies an area on the floor surface 911 using the operation input device 74. The plane setting unit 12 sets the specified area on the floor surface 911.

[0046] The detection of the floor surface 911 can also be performed, for example, by using a known image analysis that extracts and uses a characteristic portion or the like of the floor surface 911. In this case, the user can set the floor surface 911 without separately specifying the floor surface 911.

[0047] [Embodiment 2] (Functional configuration of sound processing device) Fig. 7 is a functional block diagram showing the configuration of a sound processing device according to embodiment 2. Fig. 8 is a functional block diagram showing the configuration of a reproduction part of an anticipated listening sound of the sound processing device according to the embodiment.

[0048] 7, the sound processing device 10A differs from the above-described sound processing device 10 in that it further includes a listening position setting unit 31, a propagation characteristics calculation unit 32, a sound signal generation unit 33, and an original sound data storage unit 40. The other configuration of the sound processing device 10A is the same as that of the sound processing device 10, and a description of similar parts will be omitted.

[0049] The listening position setting unit 31 sets the listening position, for example, by designating the listening position on an image from the operation input device 74. The listening position setting unit 31 sets the listening position using x and y coordinates associated with space. The listening position setting unit 31 outputs the listening position to the propagation characteristics calculation unit 32.

[0050] As shown in FIG. 8, the propagation characteristic calculation unit 32 includes a positional relationship calculation unit 321, a speaker characteristic acquisition unit 322, an IFFT processing unit 323, and a characteristic adjustment unit 324.

[0051] Positional relationship calculation unit 321 calculates the distance to the listening position based on the position of virtual speaker 92, and the azimuth of the listening position, from the position of virtual speaker 92 and the listening position. Positional relationship calculation unit 321 calculates the distance and azimuth of the listening position in the same way as in the calculation of the sound pressure distribution described above. Positional relationship calculation unit 321 outputs the azimuth to speaker characteristic acquisition unit 322. Positional relationship calculation unit 321 outputs the distance to characteristic adjustment unit 324.

[0052] The speaker characteristic acquisition unit 322 acquires the speaker characteristics of the set virtual speaker 92 from the speaker characteristic storage unit 20. The speaker characteristic acquisition unit 322 acquires the calculated azimuth angle component (desired azimuth angle component) of the acquired speaker characteristics as a frequency waveform (frequency spectrum). The speaker characteristic acquisition unit 322 outputs the frequency waveform of the desired azimuth angle component of the speaker characteristics to the IFFT processing unit 323.

[0053] The IFFT processing unit 323 generates impulse response data for the azimuth angle of the listening position by performing an inverse Fourier transform on the frequency spectrum of the desired azimuth angle component of the speaker characteristics. The IFFT processing unit 323 outputs the impulse response data to the characteristic adjustment unit 324.

[0054] The characteristic adjustment unit 324 corrects the impulse response data for distance attenuation using the distance of the listening position. The characteristic adjustment unit 324 corrects the impulse response data for distance attenuation using the ray of sound method, similar to the above-mentioned sound pressure distribution calculation unit 14.

[0055] This allows the propagation characteristics calculation unit 32 to calculate impulse response data (impulse response data of the listening position) set according to the type of virtual speaker 92 and the listening position relative to position P92 of the virtual speaker 92. The propagation characteristics calculation unit 32 outputs the impulse response data of the listening position to the sound signal generation unit 33. This impulse response data corresponds to the "sound propagation characteristics" of the present invention.

[0056] As shown in FIG. 8, the sound signal generation unit 33 includes an FFT processing unit 331 , an original sound data acquisition unit 332 , an FFT processing unit 333 , a convolution calculation unit 334 , an IFFT processing unit 335 , and an output sound adjustment unit 336 .

[0057] The FFT processing unit 331 generates a frequency waveform (frequency spectrum) of the impulse response data at the listening position by performing a Fourier transform on the impulse response data at the listening position output from the characteristic adjustment unit 324 of the propagation characteristic calculation unit 32. The FFT processing unit 331 outputs the frequency waveform of the impulse response data at the listening position to the convolution calculation unit 334.

[0058] The original sound data acquisition unit 332 acquires original sound data to be audible from the original sound data storage unit 40. More specifically, the original sound data acquisition unit 332 accepts a selection of original sound data to be audible, for example, by designation using the operation input device 74. The original sound data acquisition unit 332 acquires the selected original sound data from the original sound data storage unit 40. Note that the original sound data storage unit 40 stores the original sound data as time waveform data such as WAV data, for example, and the original sound data acquisition unit 332 acquires this time waveform original sound data. The original sound data acquisition unit 332 outputs the original sound data to the FFT processing unit 333.

[0059] The FFT processing unit 333 generates original sound data of a frequency waveform by performing a Fourier transform on the original sound data of a time waveform. The FFT processing unit 333 outputs the original sound data of a frequency waveform to the convolution calculation unit 334. At this time, the FFT processing unit 333 may multiply the original sound data by a window function and perform FFT in a desired short time.

[0060] The convolution calculation unit 334 performs a convolution calculation on the impulse response data at the listening position and the original sound data to generate predicted listening sound data of a frequency waveform, and outputs the predicted listening sound data to the IFFT processing unit 335.

[0061] The IFFT processing unit 335 performs an inverse Fourier transform on the expected listening sound data in the frequency waveform to generate an expected listening sound signal in the time waveform. The IFFT processing unit 335 outputs the expected listening sound signal in the time waveform to the output sound adjustment unit 336.

[0062] This enables the sound signal generator 33 to generate an anticipated listening sound signal according to the listening position in space and the position P92 and type of the virtual speaker 92.

[0063] The output sound adjustment unit 336 generates an expected listening sound signal SoL for the L channel and an expected listening sound signal SoR for the R channel from the expected listening sound signal. The output sound adjustment unit 336 outputs the expected listening sound signal SoL for the L channel to the L channel speaker 82L and outputs the expected listening sound signal SoR for the R channel to the R channel speaker 82R. The L channel speaker 82L emits the expected listening sound signal SoL for the L channel, and the R channel speaker 82R emits the expected listening sound signal SoR for the R channel.

[0064] This allows the user to virtually hear sound from a desired speaker placed at a desired position in a space, as if he or she were actually in that location, even if the desired speaker is not actually placed there.

[0065] Furthermore, the output sound adjustment unit 336 adjusts the level balance between the predicted listening sound signal SoL for the L channel and the predicted listening sound signal SoR for the R channel using the azimuth angle θ from the positional relationship calculation unit 321, i.e., the direction of the listening position relative to the virtual speaker 92. This allows the user to hear the predicted listening sound from the virtual speaker 92 placed at a desired position in the space with a more realistic feel.

[0066] In the above example of the sound processing device 10A, the number of virtual speakers 92 is one, but the number of virtual speakers 92 may be multiple. In this case, the sound processing device 10A may synthesize and output predicted listening sound signals from the multiple virtual speakers 92.

[0067] Moreover, the above-mentioned sound processing device 10A uses stereo speakers to output a stereo type expected listening sound. However, a configuration is adopted in which a plurality of speakers are connected to the sound processing device, and the sound processing device can realize sound image localization by adjusting the phase and amplitude of the expected listening sound signal output to each of the plurality of speakers. This allows the user to hear the expected listening sound from the virtual speaker 92 placed at a desired position in the space with a more realistic feeling.

[0068] Furthermore, the above-described sound processing device 10A sets the listening position by designation using the operation input device 74. However, by providing the camera 75 with a position detection function, the sound processing device 10A can set the position of the camera 75, i.e., the image capturing position, as the listening position. This allows the user to set the position of the camera 75, i.e., the user's position, as the listening position without having to take the trouble of setting the listening position. Therefore, the user can easily hear the expected listening sound at the image capturing position.

[0069] Furthermore, the sound processing device 10A outputs a synthetic image and emits an expected listening sound. However, if only the expected listening sound is to be emitted, the sound processing device 10A can omit the image acquisition unit 11, the plane setting unit 12, the sound pressure distribution calculation unit 14, and the synthetic image generation unit 15.

[0070] (Sound processing method (method of generating predicted listening sound)) In the above description, the method for generating an expected listening sound is realized by a plurality of functional units. However, when an expected listening sound is generated by executing a program in a calculation device 70 such as a computer as described above, the processes shown in the flowcharts of Figs. 9, 10, and 11 may be executed in outline. Fig. 9 is a flowchart showing a method for generating an expected listening sound. Fig. 10 is a flowchart showing a method for calculating the propagation characteristics shown in Fig. 9. Fig. 11 is a flowchart showing a method for generating a sound signal shown in Fig. 9. Note that the specific contents of each process have been described above, so only a general flow will be described here.

[0071] The arithmetic device 70 acquires the listening position and calculates the propagation characteristics (impulse response data of the listening position) corresponding to the listening position (S21). More specifically, as shown in Fig. 10, the arithmetic device 70 calculates the positional relationship between the position P92 of the virtual speaker 92 and the listening position (S31). The arithmetic device 70 acquires the speaker characteristics (S32). The arithmetic device 70 generates impulse response data for the listening position using the positional relationship between the position P92 of the virtual speaker 92 and the listening position and the speaker characteristics (S33).

[0072] The arithmetic device 70 generates an expected listening sound by using the propagation characteristics of the listening position and the original sound data to be audible (S22). More specifically, as shown in Fig. 11, the arithmetic device 70 acquires the original sound data to be audible (S41). The arithmetic device 70 performs a convolution operation on the propagation characteristics, i.e., the impulse response data for the listening position, and the original sound data (S42). The arithmetic device 70 adjusts the LR balance according to the listening position (S43).

[0073] By performing such processing, a user can virtually hear sound from a desired speaker placed at a desired position in a space, as if he or she were actually in that location, even if the desired speaker is not actually placed there.

[0074] Furthermore, the user can hear the listening sound from the virtual speaker 92 in the form of stereo sound with adjusted LR balance. This allows the user to hear the listening sound from the virtual speaker 92 placed at a desired position in the space with a more realistic feel.

[0075] Note that, if sound is reproduced through a headphone speaker instead of the process of step S43 described above, more accurate sound image localization becomes possible. This further improves the reproducibility of the listening sound from the virtual speaker 92 placed at a desired position in the space. Therefore, the user can hear the listening sound from the virtual speaker 92 with an even more realistic feeling.

[0076] The description of the present embodiment is illustrative in all respects and is not restrictive. The scope of the present invention is indicated by the claims, not by the above-described embodiments. Furthermore, the scope of the present invention is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0077] 10, 10A: Sound processing device 11: Image acquisition section 12: Plane setting section 13: Virtual speaker setting section 14: Sound pressure distribution calculation section 15: Composite image generation unit 20: Speaker characteristic memory section 31: Listening position setting unit 32: Propagation characteristic calculation unit 33: Sound signal generator 40: Original sound data storage unit 70: Arithmetic device 71: CPU 72: Memory 74: Operation input device 75: Camera 76: Display device 77: Sound emitting device 82L: Left channel speaker 82R:R channel speaker 90: Composite image 90P:Image 91: Room 92: Virtual speaker 93: Sound pressure distribution 321: Positional relationship calculation unit 322: Speaker characteristic acquisition unit 323: IFFT processing section 324:Characteristics adjustment section 331: FFT processing section 332: Original sound data acquisition unit 333: FFT processing section 334: Convolution operation unit 335: IFFT processing section 336: Output sound adjustment section 700: Bus 731: Interface 732:Video output port 733: Sound output port 911:Floor 911F: 1st floor 911S: Second Floor 911T: The third floor SoL: Expected listening sound signal for the left channel SoR: Predicted listening sound signal for R channel

Claims

1. Acquire an image of the acoustic space, A plane is set in the acoustic space from the image; Setting a virtual speaker within the image of the acoustic space; Calculating a sound pressure distribution on the plane from the characteristics of the virtual speaker; The sound pressure distribution is superimposed on the plane, A listening position is set within the image of the acoustic space, Calculating a propagation characteristic of a sound emitted from the virtual speaker to the listening position in response to the sound pressure distribution; generating sound data at the listening position using the sound propagation characteristics and original sound data to be listened to; Detecting an acquisition position of the image, and setting the acquisition position of the image to the listening position; Sound processing methods.

2. Reproducing the sound data to emit sound; The sound processing method according to claim 1 .

3. A plurality of the planes are set in the acoustic space. The sound processing method according to claim 1 or 2.

4. A plurality of the virtual speakers are set in the acoustic space, The sound pressure distribution is generated by synthesizing sound pressure distributions for each of the plurality of virtual speakers. The sound processing method according to any one of claims 1 to 3.

5. The sound pressure distribution is a sound pressure distribution of a direct sound from the virtual speaker. The sound processing method according to any one of claims 1 to 4.

6. analyzing features of the image to detect the plane; The sound processing method according to any one of claims 1 to 5.

7. an image acquisition unit for acquiring an image of an acoustic space; a plane setting unit that sets a plane in the acoustic space from the image; a virtual speaker setting unit that sets a virtual speaker within the image of the acoustic space; a sound pressure distribution calculation unit that calculates a sound pressure distribution on the plane from the characteristics of the virtual speaker; a synthetic image generating unit that generates a synthetic image by superimposing the sound pressure distribution on the plane; a listening position setting unit that sets a listening position within the image of the acoustic space; a propagation characteristic calculation unit that calculates a propagation characteristic of a sound emitted from the virtual speaker to the listening position in response to the sound pressure distribution; a sound signal generating unit that generates sound data at the listening position using the sound propagation characteristics and original sound data to be listened to; Equipped with the listening position setting unit detects the image acquisition position and sets the image acquisition position to the listening position; Sound processing device.

Images