Method for obtaining position of sound source

Abstract

The invention relates to a method for obtaining the position of a sound source relative to a dedicated reference point. A first sound signal and a plurality of second sound signals synchronized in time are recorded. The position may be obtained by applying the estimated filter to a correlation signal obtained by correlating the first sound signal with at least one of the plurality of second sound signals in the frequency domain. Two timing values are derived in the at least one filtered correlation signal, the two timing values exceeding a dedicated threshold in the time domain. The distance between the dedicated reference point and the sound source is then calculated based on the first timing value and the second timing value that are each obtained.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Danish patent application DKPA202270280 filed on May 31, 2022, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The invention relates to a method for obtaining the position of a sound source relative to a dedicated reference point. The invention also relates to a computer system and a non-transitory computer-readable storage medium. The invention also relates to a recording device. Background Art

[0004] Sound field or spatial audio systems and formats, such as ambisonics or Dolby Atmos, provide coded sound information related to a given sound scene. In this way, position information can be assigned to sound sources in the sound scene. These techniques are already known from certain computer games, where recorded sounds carry position information of game objects, but can also be used to capture events live, for example large orchestras or sporting events. The possible areas of application are therefore very wide and range from the immersive effects mentioned above (e.g. giving a feeling of being part of a sporting event) to virtual reality or augmented reality experiences.

[0005] In many cases, recording sound for such applications using spatial audio microphones is a challenge in itself. While these methods are useful for capturing live sound field information from a specific point in space, they also have some technical limitations as they are based on beamforming technology and are generally considered expensive. For example, the sound quality of people who are far away from the microphone may be degraded. In more noisy or reverberant situations, or if more than one person is speaking, it is difficult to identify and isolate individual sound sources for the purpose of equalization or other processing techniques.

[0006] At the same time, audio content creators are also aware of the need for high-quality audio, including the use of spatial audio information to improve the quality of recordings or add additional sound effects to increase the listener's immersion. Therefore, a lower-cost solution is needed to achieve the benefits and advantages of high-level spatial audio microphones. The solution should preferably work independently of hardware, allowing flexible use in different scenarios. Summary of the invention

[0007] The present disclosure and the principles proposed therein provide a method, a computer system, and a recording device to achieve the several benefits and advantages mentioned above.

[0008] The inventors have discovered a method that provides accurate determination of the location of a sound source relative to a dedicated reference point in terms of distance and angle. The proposed method is largely independent of the hardware used and is scalable to different quality levels. However, with certain dedicated hardware, the functionality and resolution of the method are greatly improved. Furthermore, the method allows for both offline and real-time processing. Thus, the proposed method can be included in a variety of applications, including but not limited to sound capture and processing for podcasts, movies, live broadcasts or other events, audio and teleconferencing, virtual reality, video game applications, and the like.

[0009] In one aspect, the inventors propose a method for determining the position of a sound source relative to a dedicated reference point. In this regard, the expression "position" does include the distance from the sound source to the dedicated reference point, the angle based on one or two axes passing through the reference point, or a combination thereof.

[0010] The method obtains a first sound signal recorded with a microphone at a sound source or at a known location of the sound source. Similarly, a plurality of second sound signals are recorded at known locations relative to a dedicated reference point. The reference point can be defined, for example, by dedicated hardware with a plurality of microphones. The first sound signal and the plurality of second sound signals are synchronized in time.

[0011] Typically, it is assumed that the first sound signal is recorded near the sound source, which means that due to the proximity between the microphone and the sound source, the sound emitted by the sound source is recorded at a higher level than reflections, reverberation and background noise, and means that the distance is relatively low compared to the distance between the sound source and the dedicated reference point. However, the term "at the sound source" should not be understood as a very limited meaning. On the contrary, the statement should include and allow a certain distance between the actual sound source and the microphone. In other words, the position of the microphone relative to the actual sound source is well known. Similarly, the multiple second sound signals are recorded at different locations, for which the distance and angle to the reference point are known. Time synchronization is important for the method proposed in the subsequent steps. In some cases, this time synchronization can be achieved by providing a common time base for any recorded sound signal. In some other cases, the recorded sound signal can be used to provide a time base, for example, by timely associating a dedicated start signal recorded and included in the first sound signal and the multiple second sound signals.

[0012] A generalized cross correlation or modified cross correlation (e.g., phase shift) between the first sound signal and at least one of the plurality of second sound signals is then calculated on a time frame-by-time basis to obtain at least one generalized cross correlation or phase shift signal for each frame of the recorded sound signal. The length of the frame is typically adjustable and can be adjusted during the estimation (e.g., when there is an indication that the sound source is moving).

[0013] The generalized cross correlation or phase shift signal is then used to estimate the distance between the sound source and the dedicated reference point.The distance estimation is performed by estimating a time delay between the first sound signal and at least one of the plurality of second sound signals using at least one phase shift signal.

[0014] The angle between the sound source and the dedicated reference point is estimated by estimating the time delay between each pair of the plurality of second sound signals using a weighted least mean square, wherein the weighted least mean square depends on the obtained phase shift signal between the first sound signal and the pair of the plurality of second sound signals.

[0015] In some aspects, the calculation of the phase transform is performed by correlating the first sound signal with at least one of the plurality of second sound signals in the frequency domain to obtain at least one correlation signal. Thereafter, the power spectrum is normalized and the at least one correlation signal is transformed back to the time domain. Discrete Fourier Transform, DFT and Inverse DFT may be used, and in some cases, in particular Short Time Fourier Transform, STFT and Inverse STFT or ISTFT may be used.

[0016] In the case of using a short-time Fourier transform STFT, the STFT can be performed on the first sound signal and at least one of the multiple second sound signals to obtain a corresponding spectrum. Then, a cross spectrum on each spectrum is obtained, and a spectral mask filter is applied to the obtained cross spectrum. After application, an inverse short-time Fourier transform (ISTFT) is performed to obtain at least one phase transformation signal. The above-mentioned masking filter can be estimated based on the signal-to-noise ratio in each frequency bin (frequency bin) of the first sound signal. For example, a quantile filter (particularly a median filter) can be used to smooth the power spectrum of each time slice of the power spectrum derived from the first sound signal. In response to the previous time slice, the noise is estimated for each time slice. Then, depending on whether the signal-to-noise ratio exceeds a predetermined threshold, the filter parameter is set to 1 or 0.

[0017] In some other aspects, a filter for operating the signal-to-noise ratio in each frequency bin of the first sound signal may be estimated by using a residual signal from a denoising process as a noise estimate, wherein the denoising process may optionally be based on machine learning.

[0018] In some aspects, time resolution can be increased by interpolation to improve distance and angle accuracy. One method is to upsample the first sound signal and at least one of the multiple second sound signals before correlating them in the digital domain. Another method is cubic interpolation. This can be achieved by upsampling them before discrete or short-time Fourier transforms, or alternatively, using a higher sampling frequency for IDFT and ISTFT, respectively, to transform them from the frequency domain back to the time domain. Therefore, in some cases, the frequency domain signal can be transformed back to the time domain at a higher transformation frequency than the transformation frequency used for the first sound signal and at least one of the multiple second sound signals to the digital domain.

[0019] In some cases, a corresponding phase shift signal may be calculated between the first sound signal and each of the plurality of second sound signals. This will then allow the distance from the sound source to each position of the second sound signal to be estimated, enabling further statistics to be performed to improve accuracy.

[0020] Some further aspects relate to the step of estimating the time delay. To this end, it is suggested to search for a maximum value in the at least one phase shift signal, or to detect a first amplitude value above a given threshold and to search for the maximum value within a specified window centered around the first amplitude value. This specified window may be appropriate in case of potential crosstalk between different microphones recording the various first sound signals, or if the microphone recording the sound signals is far away from the actual sound source where sound reflections are present.

[0021] In other words, a specified window centered on the first amplitude value provides a solution to suppress the recorded reflections from the sound signal, thereby reducing the risk of estimating distances or angles with false positive results. The length of the specified window can be set to a limit inversely proportional to the signal bandwidth estimated from the highest frequency component of the first sound signal. Another limit can be within the range of expected early reflections depending on the distance between the recording location of the first sound signal and the location of the one or more sound sources. In some cases, the length of the specified window can be proportional to the maximum flight time between the locations of the plurality of second sound signals.

[0022] In some further instances, the dedicated reference point is substantially centered between the recording locations of the plurality of second sound signals, and wherein the estimation of the distance between the sound source and the dedicated reference point uses an average of a set of time delays between each of at least one of the plurality of second sound signals and the first sound signal.

[0023] Some other aspects relate to the estimation of angles (including azimuth and elevation). If the amplitude value of the obtained phase shift signal is above a given threshold and within a specified window centered on the first amplitude value, the weighted least mean square is based on the phase shift signal between the obtained first sound signal and the plurality of second sound signals.

[0024] The method proposed at present allows estimating not only the angle and distance between a sound source and a reference point, but also the angle and distance between two microphones (for example, two microphones worn on some speakers are spaced apart). Both microphones record sound signals from the sound source, which are depicted as a first sound signal (recorded at the first microphone) and another first sound signal (recorded at the position of the second microphone). In this case, a phase shift can be calculated between the first sound signal and another first sound signal recorded at or associated with one or more sound sources to obtain another phase shift signal. Then, the time delay between the first sound signal and the other first sound signal can be estimated by using this other phase shift signal, thereby calculating the distance between the position of the microphone (recording the first sound signal) and the position associated with the recording of the other first sound signal.

[0025] The proposed aspect provides a simple tool to calculate the distance between the positions associated with two or more first sound signals. This not only helps to estimate the possible crosstalk between the two or more microphones (recording the first sound signals) (including classifying the sound signals as source signals or crosstalk based on whether the time delay is negative or positive), but also provides information about the relative distances between the microphones, which can be used for post-processing to perform position estimation. Therefore, the method can be used to obtain information about the sound source, which is far away from the position where the two (or more) first sound signals were recorded.

[0026] Another aspect involves post-processing and specific movement of the sound source during processing. In the case of a stationary sound source, the distance may not change during different frames (except that it may change due to the estimation). However, if the sound source moves slowly, its distance and angle will change over time. Such a sound source may be difficult to identify because a moving sound source will affect the STFT through Doppler shift. In addition, estimation noise can be identified as a moving sound source for two or more sound sources located at different locations (or vice versa).

[0027] To accommodate this observation, one aspect proposes applying a noise reduction filter to the estimated distance and / or estimated angle. Additionally or alternatively, a Kalman filter can be applied to the estimated distance and / or estimated angle and simplified results thereof to predict possible motions, respectively. In some cases, this filtering is achieved by applying a gradient or divergence to the estimated distance and / or estimated angle.

[0028] It has been found that a specific arrangement of microphones recording the second sound signals may be suitable. Possible reflections or errors can be identified more easily and can cancel each other out. Thus, the corresponding positions of a pair of multiple second sound signals can be located on a virtual line passing through a dedicated reference point and at the same distance from the dedicated reference point.

[0029] It is useful to position the microphone recording the second sound signal at a dedicated location. For example, the plurality of second sound signals may include four audio sound signals, wherein two of the four sound signals are recorded at a maximum spatial distance of a few centimeters. This distance is typically small enough to avoid accidentally recording direct sound and reflected sound from the same sound source at the same time, while being large enough to provide sufficient differences when cross-correlating the second sound signal with the first sound signal without resorting to excessive upsampling.

[0030] The speed at which sound travels through matter depends on the temperature of the matter. For accurate measurement, the air temperature is measured, particularly in the vicinity of the plurality of second sound sources. This measurement can be repeated periodically to compensate for temperature changes during the recording session. Distance and angle can then be estimated in response to the derived air temperature, which changes the speed of sound in the air.

[0031] In some further examples, a computer system is provided, comprising one or more processors and a memory. The memory is coupled to the one or more processors and comprises instructions. When the instructions are executed by the one or more processors, the instructions cause the one or more processors to perform the above-mentioned method and its various steps. Similarly, a non-transitory computer-readable storage medium can be provided, which comprises computer-executable instructions for performing the method according to any of the preceding claims.

[0032] On the other hand, it is related to a recording device, which includes a rectangular parallelepiped shape with a bottom surface, a top surface and four sides. The recording device is suitable for placing the bottom on any substantially flat surface (e.g., a floor, a table, etc.). The dive height can be slightly greater than its width or depth. In particular, the width and depth are similar or equal. The recording device also includes an accessible user interface on the top surface. The user interface may include one or more buttons, displays, switches, etc., to provide information to the user, and enable the user to interact with the device to implement its functions. In this regard, the recording device may include a processor suitable for reading the user's commands and acting accordingly. In addition, the processor is configured to process one or more sound signals in some cases at least in part using the aspects of the principles proposed herein.

[0033] The recording device further comprises a plurality of microphones, in particular omnidirectional microphones, wherein pairs of microphones are arranged on each of the respective sides, the first microphone of the pair being arranged at the top of the respective side and the second microphone of the pair being arranged at the bottom of the respective side.

[0034] The distance between the first microphone and the second microphone of each pair of microphones is not set equal to the distance between the first microphones of adjacent sides. In other words, two adjacent microphones are spaced the same distance from each other.

[0035] In some aspects, the distance from the first microphone to the top surface is greater than the distance from the second microphone to the bottom surface. In some other aspects, the outer dimensions of the recording device can be slightly larger than the dimensions of the two opposing microphones, that is, the microphones are slightly displaced and arranged inside the recording device. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Further aspects and embodiments according to the proposed principles will become apparent from the various embodiments and examples described in detail in conjunction with the accompanying drawings, in which

[0037] Figure 1 An embodiment of the proposed method is shown and several process steps for determining the location of a sound source are shown;

[0038] Figure 2 The steps of applying a spectral mask to obtain a frequency-weighted phase transform of a filtered correlation signal are shown;

[0039] Figure 3A It is a schematic diagram of a recording environment with several microphones, which is used to record more complex sound field scenes;

[0040] Figure 3B An embodiment of a sound field microphone implementing some aspects of the proposed principles is shown;

[0041] Figure 4 A process flow of a method according to some aspects of the proposed principles is shown. DETAILED DESCRIPTION

[0042] The following embodiments and examples disclose different aspects and combinations thereof according to the proposed principles. The embodiments and examples are not always drawn to scale. Similarly, different elements may be shown in enlarged or reduced sizes to emphasize various aspects. It goes without saying that the various aspects of the embodiments and examples shown in the figures can be combined with each other, which is not inconsistent with the principles of the present invention and need not be elaborated. Some aspects show regular structures or forms. It should be noted that in practice, slight differences and deviations from the ideal form may occur, but this does not violate the idea of ​​the present invention.

[0043] In addition, the various figures and aspects are not necessarily shown in the correct size, and the ratios between the various elements are not necessarily completely correct. Some aspects are highlighted by enlarging the display. However, terms such as "above", "on...", "below", "below...", "larger", "smaller", etc. are correctly expressed relative to the elements in the figures. Therefore, the relationship between these elements can be inferred from these figures.

[0044] Figure 3A An application using the method according to the proposed principles is shown. The scenario corresponds to a typical recording session, where multiple sound signals are recorded to obtain the sound field of the scenario. Although this example uses a natural person's voice recording, it can be appreciated that the present method and principles disclosed herein are not limited to voice processing or finding the position of a natural person. Instead, it can be used to locate any dedicated sound source relative to a reference point.

[0045] The current scene contains two sound sources depicted as P1 and P2, which in this embodiment are two people having a conversation in an at least partially enclosed space. Each person holds a microphone M1 and M2, which are close to their respective bodies. Alternatively, microphones M1 and M2 are mounted on their respective chests or bodies. Therefore, microphones M1 and M2 can be associated with the positions of their respective sound sources. Multiple second microphones M3 and M4 are located at position B1. Position B1 is also defined as a reference point. Therefore, people P1 and P2 are located at a certain distance and angle from reference point B1 and are separated from each other. Wall W is located on one side to produce reflections when each sound source P1 and P2 speaks.

[0046] The microphones of M1, M2, M3 and M4 are time-synchronized with each other, that is, in this case, a common time base is used to record the sound. When recording a conversation, microphone M1 records the voice of person P1 and also records the voice of person P2 with a certain delay. Likewise, microphones M3 and M4 record the voices of persons P1 and P2 with some delay due to the speed of sound and the distance of person P1 from reference point B1. Depending on the distance, the delay is different, but in any case, the direct way from the sound source to one of the microphones of M3 and M4 is called direct sound.

[0047] Assuming now that there is only a single sound source P1, one can simply use this direct sound to calculate the distance, i.e., use the direct sound to calculate the distance to the reference point B1, which is achieved by measuring the time delay between the sound signal recorded by microphone M1 and one of the microphones M3 or M4 multiplied by the speed of sound.

[0048] Since the speed of sound depends on the temperature, a temperature sensor T1 is located near the microphones M3 and M4 to measure the air temperature and thus correct for the effects of temperature changes. The scenario mentioned above is very simple and does not fit the real-world scenario. This time, the wall W will reflect parts of the speech, which will be recorded by microphone M1 at relatively low values, but will also be recorded by microphones M3 and M4 after some delay, which may have relatively high levels. Microphone M4 will also record the speech. Depending on the scenario, the reflected sound speech is superimposed on the ongoing speech. Due to possible constructive interference or other effects, it may happen that the recording of the indirect reflected sound includes a higher level than the direct sound. In more complex cases, a second sound source also provides a sound signal at the same time, resulting in the superposition of several different sound signals, some of which come from the sound sources P1 and P2 and some of which are reflections on the walls.

[0049] The present application aims at processing the recorded signals in such a way that the positions of the individual sound sources relative to a reference point can be identified and localized.

[0050] Another application for solving the problem of associating certain position information with a sound source exists in virtual reality (VR) applications. Such applications typically include a 360° stereo video signal with several objects in a virtual environment, some of which are associated with objects corresponding to sounds.

[0051] These objects (visual and auditory) are presented to the user through, for example, binocular headphones and stereo headphones, respectively. The binocular headphones are capable of tracking the position and orientation of the user's head (e.g., using an IMU / accelerometer), so that the video and audio played to the headphones and earphones, respectively, can be adjusted accordingly to maintain the illusion of virtual reality. For example, at a given moment, only a portion of the 360° video signal is displayed to the user, which corresponds to the user's current field of view in the virtual environment. When the user moves or rotates their head, the portion of the 360° signal displayed to the user changes to reflect how the movement will change the user's field of view in the virtual world. Similarly, when the user moves, the sound emitted from different positions in the virtual scene can be subjected to adaptive filtering of the left and right headphone channels to simulate the frequency-related phase and amplitude changes in the sound caused by the spatial offset between the ear and the human head and the upper body scattering in real life.

[0052] Some virtual reality works consist entirely of computer-generated images and separately pre-recorded or synthesized sounds. However, it is becoming increasingly popular to make "live action" VR recordings using a camera capable of recording a 360° field of view and several microphones capturing the sound field. The recorded sounds from the microphones are then processed using a method according to the proposed principles and aligned with the video signal to produce a VR recording that can be played through headphones and earphones as described above.

[0053] Another application for solving the problem of associating some position information with a sound source exists in Next Generation Audio (NGA) applications. Such applications typically include audio objects with metadata such as position.

[0054] These objects (visual and audio) are presented to the user through, for example, head-mounted stereo headphones with binaural rendering. As binocular headphones, such headphones are able to track the orientation of the user's head (e.g., using an IMU / accelerometer) so that the audio played by the headphones can be adjusted accordingly to maintain the illusion of immersion in the audio. For example, as the user moves or rotates their head, sounds emanating from different locations in the virtual scene, or scenes recorded using the innovation, can be subjected to adaptive filtering of the left and right headphone channels to simulate frequency-dependent phase and amplitude changes in the sound that occur in real life due to the spatial offset between the ears and the human head and upper body scattering.

[0055] Return to reference Figure 3A , which shows an embodiment of a sound recording device according to some aspects of the present invention, which is suitable for recording multiple sound signals to be used in the proposed method. Specifically, the sound recording device is a surround sound microphone designed for multiple-input multiple-output (MIMO) beamforming, which targets directivities corresponding to spherical harmonic basis functions.

[0056] The sound recording device is formed as a cuboid, because this shape with certain dimensions is suitable for recording sound files. In addition, the cuboid shape allows to have a display and a user interface on the top of the recording device, so that the bottom can be placed on a suitable surface and still be operated in an easy way. Screws on the bottom enable the device to be placed on a stand.

[0057] The eight microphones in the sound recording device are arranged in an octahedral configuration (i.e., the centers of the octahedral faces). The beamforming (so-called ambisonics B-format conversion) consists of a weighted sum, which depends on the spherical harmonic basis functions and the microphone configuration, and a set of filters that are used to adapt the beamformed signal to the scattering of the recording device in order to achieve a flat frequency response. For wavelengths longer than the physical dimensions of a cube, the acoustic scattering can be approximated as a hard sphere. Therefore, at lower frequencies, the filters can be adapted and simplified to this approximation.

[0058] The surface of the recording device introduces scattering which includes the effect of preventing destructive interference when a signal having the same wavelength as the device is recorded by two microphones on opposite sides.

[0059] therefore, Figure 3B The recording device includes 8 omnidirectional microphones, and four microphones 1A, 1B, 1C, 1D are located in the upper part, and one microphone is set on each side. Similarly, four microphones 2A, 2B, 2C and 2D are arranged in the lower part, and one microphone n is set on each side. There are no microphones at the bottom or top, so the cuboid can be placed on a surface, leaving space for the user interface.

[0060] By arranging the microphone tubes in their respective grooves, the microphone tubes are slightly moved towards the center. The distance d between two adjacent microphones (e.g., the distance between 1A and 1B or 2A and 2B) is equal. In other words, adjacent microphones are placed equidistant from each other.

[0061] equation

[0062]

[0063] According to the Shannon criterion, the upper limit of the spatial aliasing frequency f is defined by the distance d between any pair of adjacent microphones. lim Above this frequency, grating lobes will start to appear in the directivity of the beamformed signal. For wavelengths longer than the dimensions of the recording device, the acoustic scattering is approximately the same as for a hard sphere. The distance d is set to provide an upper frequency limit of 3 kHz to 4 kHz and allows the acoustic scattering from the recording device to the sphere to be approximated to the spatial aliasing frequency.

[0064] Reference now Figure 1 , Figure 1 The various blocks of the method according to the proposed principle are shown. For simplicity, the Figure 3A and Figure 3BThe above scenario is used to explain the method. This method is applicable to the post - processing of pre - recorded sound signals, but is also applicable to real - time sound signals (e.g., during audio conferences, live events, etc.). The method begins with providing one or more first sound signals and a plurality of second sound signals in boxes BM1 and BM2 respectively. The recorded sound signals preferably have the same digital resolution, which includes the same sampling frequency (e.g., 14 - bit at 96 kHz). In the case of using different resolutions or sampling frequencies, it is recommended to resample the various sound signals to obtain signals with the same resolution and sampling frequency.

[0065] The upper part of the picture includes elements 3’, R1, 30A and 31, and is related to identifying possible crosstalk between two or more first sound signals, which are sound signals recorded by microphones and whose positions are to be determined. As mentioned before, both the reflected sound and the direct sound are recorded by two microphones in box BM1. To determine which of the two or more microphones is actually located at the corresponding sound source, the signals recorded by the two microphones will be processed, filtered, and cross - correlated to obtain the time difference in the cross - correlation.

[0066] For this purpose, frequency - weighted generalized cross - correlation or phase transformation 3’ is used to process the two signals. In the first step, each first signal is transformed into the frequency domain using STFT to obtain a time - spectrum. A spectral mask filter is derived from the spectrum by first generating a smoothed power spectrum S(l,k), where l is the sound signal from the microphone and k is the corresponding frame of the sound signal. For each frequency bin, a first - order filter estimates the noise n(l,k) in the current frame based on the previous frame. The total noise n(l,k) is given by:

[0067] n(l,k)=(1 - α)log(S(l,k))+(n(l,k - 1)) α

[0068] The different α depends on S(l,k)<log(n(l,k - 1)). Thus, when the SNR is higher than a certain threshold, the filter mask is 1, otherwise it is 0. As a result, different filter masks are associated with each of the two first signals respectively. In the next step, a cross - spectrum is generated by cross - correlating pairs of the first signals and normalizing the result of the cross - correlation. Then, the corresponding estimated filter is applied to the normalized cross - spectrum, and an inverse STFT is performed to obtain the filtered correlation signal, see reference numeral R1. At this point, it should be noted that for the cross - spectrum R xy , the filter F x (for signal x) should be used; for the cross - spectrum R yx , the filter F y(for signal y). The filtered correlation signal is then used to estimate the signed time difference or delay of the direct sound in the two microphones used to record the first sound signal. The sign depicted in box 31 (i.e., dt>0 or dt<0) provides information about which microphone is closer to the actual sound source. Therefore, the microphone (and the sound signal) is associated with the corresponding sound source and the corresponding filter mask.

[0069] If the association of the sound signal with the corresponding sound source is defined, that is, if only one first signal is recorded, the above steps can be omitted. Referring back to Figure 3, the boxes 3, R2 to 35 shown in the lower part describe the various steps of estimating the distance and angle to the reference point. Box BM3 contains a plurality of second sound signals recorded by one or more second microphones, and the positions of the one or more second second microphones are fixed relative to the reference point. The second sound signal is recorded by a recording device, and for this embodiment, a total of 8 second sound signals are provided. The position of each second microphone is slightly different so that the angle can be obtained later, but the position of each second microphone is also close enough so that the influence of reflections from walls, etc., can be determined and filtered. With regard to the method currently proposed, it is not necessary to fully evaluate all the second sound signals to obtain the position. On the contrary, it has been found that the second sound signals recorded by the four microphones closest to the sound source are sufficient to be fully evaluated. In order to obtain information about which sound signal is closest, various options can be used.

[0070] On the one hand, since all microphones are synchronized in time, the second sound signal can be used and the correlation of the second sound signals between the opposing microphone pairs can be estimated to obtain the microphone that recorded the sound at the earliest time. Alternatively, the arrival of the sound at the corresponding microphone can be evaluated using blocks 3 and 30B explained further below.

[0071] The process is now similar to the processing described for the two or more first sound signals. However, in box 3, the first sound signal (the first sound signal for which the distance and angle will be determined) is now cross-correlated with at least one of the eight second sound signals. Box 3 may be performed for each second sound signal to provide a total of eight filtered cross-correlated signals, see for example reference R2. Alternatively, as described above, the first sound signal is correlated with the four second sound signals recorded by the microphone closest to the sound source.

[0072] Figure 2The frequency weighted phase transform FW-PHAT in an exemplary embodiment is shown. The two input signals are transformed into the frequency domain using SFTF and then the cross spectrum is derived from it. After normalizing the spectrum, the previously estimated filter is applied, which in this case is the spectral mask filter associated with the first sound signal. The result is then converted back to the time domain using the inverse SFTF.

[0073] If the signals in the frequency-weighted PHAT are uncorrelated, the time delays in blocks 30B and 30A are estimated by first identifying the maximum value that the peak would have. To this end, the noise variance is given by the noise maximum resulting from [sigma=mean(mask) / framesize] and sqrt(sigma*2*ln(framesize). The first value in the frequency-weighted PHAT that exceeds this maximum value (possibly including a scale for some headroom) is then searched for, and the search is refined to a local maximum close to this first value. The location of this maximum corresponds to the time of flight of the direct sound (n_max / sampling frequency). The distance is then given by the time of flight multiplied by the speed of sound, taking into account the temperature dependence of the speed of sound (assumed to be 20°C when the ambient temperature is not measured). The process in block 30B is repeated for each cross-spectrum. In block 32, the various results are further processed by using the average of the set of time of flight estimates. The distance is then subtracted from the estimate in block 33.

[0074] In some aspects, the distance (radius) to the sound source can be measured using the arrival time difference between the point microphone and the center of the array, which can be based on the vector m i and a l The average value of the arrival time difference between i is the position of the first microphone recording the first sound signal, a l are the positions of a total number L (L=8) of microphone elements l in the recording device. The time difference may be obtained from a previous evaluation of the correlation signal. Alternatively, it may be based on a time difference obtained from an evaluation of the sum of the correlation signals.

[0075] To obtain the angle between the sound source and the reference point, blocks R3, 30C and 34 to 36 are used. To avoid any influence of room reflections, a window function is used to truncate the FW-PHAT result of the first filtered correlation signal in block R2. As shown in block R2, the window function includes a width that depends on the distance between the second microphones. Since the second microphones recording the second sound signal are slightly spaced apart, the estimated distance between the sound source and the corresponding second microphone may also vary. The width of the window function used to truncate the first filtered correlation signal is substantially proportional to the maximum flight time between the second microphones.

[0076] The now truncated set of filtered correlation signals can be upsampled to provide finer temporal resolution, thereby producing a more accurate angle estimate.

[0077] Since the sound source is located at x i , angle estimation based on position The time difference of sound arrival between two adjacent microphones assessment, Given by:

[0078]

[0079] This means that if the source is far away from the array compared to the array baseline (plane wave assumption), the angle of arrival (both azimuth and elevation) can be estimated by trilateration. You can directly Alternatively, PH can be used mi The cross-correlation between pairs. When the cross-correlation is used, the upsampling of the correlation signal can be replaced by interpolation (e.g., upsampling of the cross-correlation) for finer temporal resolution. The interpolation will be performed on a smaller data set than the upsampling before the cross-correlation, making the processing more efficient.

[0080] and Therefore, under the plane wave assumption, the timing difference forms the observation vector for each frame Its weight is:

[0081]

[0082] From the matrix, the linear equation can be obtained using the matrix above and the proportional relationship that produces the vector A, where:

[0083]

[0084] The array forms a tetrahedron with linearly independent columns. However, other forms such as octahedral forms can also be used, as they also provide perpendicular Cartesian coordinate axes. This will ultimately be provided by the following formula The proportional relationship between and A:

[0085]

[0086] Location of the sound source It can be expressed in spherical coordinates [r, θ, φ], where Since the estimate is noisy and the variance of the noise is different for each observation, It can be estimated by weighted least mean square:

[0087]

[0088] Where W is a component with Each of these components is related to the mass observed by PHAT, where This function can be used to obtain the l and a k The quality function can be expressed as:

[0089]

[0090] Therefore, it is estimated by the following formula

[0091]

[0092] Wherein the second term represents a threshold for detecting a time delay during which the first sound signal is correlated with the corresponding sound signal. M and N are values ​​giving a ratio M / N used during PHAT to indicate that M of the N frequency bins contain information.

[0093] Figure 4 A flow chart of a method for determining distance and angle according to the proposed principle is shown. The method is suitable for real-time processing as well as for offline processing, where several previously recorded sound signals forming a sound field are processed.

[0094] The method comprises obtaining in step S1 a first sound signal recorded at a sound source, for which the distance and angle to a reference point must be determined. In step S2, a plurality of second sound signals are recorded in the vicinity of the reference point or at least at a known location or position of the reference point. The first sound signal and the plurality of second sound signals are synchronized in time. Such time synchronization can be achieved by referencing all sound signals to a common time base during the recording session.

[0095] Then, optionally, various signals are preprocessed in step S3. For example, the recorded sound signal can be denoised or equalized to improve the results of subsequent processing steps. However, care should be taken not to disrupt the timing of the signal. In some cases, it is also useful to apply a method of preserving the phase information of the recorded signal during the preprocessing step S3. In addition, STFT is performed on the first sound signal and each second sound signal. In step S3', the correlation between the second sound signal pairs is evaluated, wherein the pair of sound signals corresponds to the signals recorded by two opposing microphones. The correlation will determine the subset of microphones (at least four) closest to the sound source, because these microphones will first record their respective sound signals. These second sound signals are marked for later use in step S5.

[0096] In this example, there is only a single first sound signal associated with a single sound source. The first sound signal is processed by estimating a filter (particularly a spectral mask filter) in step S4. The filter acts on the signal-to-noise ratio in each frequency of the first sound signal in the time domain. The resulting spectral mask contains a set of "1"s and "0"s for each frequency bin.

[0097] In step S5, the first sound signal is correlated in the frequency domain with each of the labeled second sound signals in the plurality of second sound signals previously identified in step S3' and at least one correlation signal is obtained. The cross-correlation may be normalized before applying the filter estimated in step S4 to obtain one or more filtered correlation signals.

[0098] Until then, the steps to determine distance or angle are similar.

[0099] Now continue to determine the distance between the reference point and the sound source and steps S6 to S8. Step S6 includes obtaining a first timing value in the at least one filtered correlation signal that exceeds a dedicated threshold in the time domain. Then, in step S7, a second timing value corresponding to the threshold in the at least one filtered correlation signal is obtained based on the first timing value. Both steps S6 and S7 can use the previously described search for the maximum value in the PHAT signal (i.e., the filtered correlation signal). In step S8, the distance between the dedicated reference point and the sound source is based on the first timing value and the second timing value obtained respectively. However, one can also consider the temperature of the air. In the case of a pre-recorded signal, this information is stored and used in S9 to compensate for the temperature effects that affect the speed of sound.

[0100] Step S9 is performed to derive and estimate the angle of the sound source from the reference point. More precisely, the angle between the sound source and the dedicated reference point can be determined by estimating the time delay between each pair of multiple second sound signals using a weighted least mean square. The weighted least mean square depends on the phase shift signal between the obtained first sound signal and the previously obtained second sound signal subset. To this end, step S9 is performed several times. If the source is far away from the array relative to the array baseline, the arrival angle (azimuth and elevation) can be estimated by trilateral measurement, which is the usual case. The time difference between the first sound signal and the different subsets of second sound signals can be represented by the correlation result represented by PHAT. Alternatively, cross-correlation can be used before step S10.

[0101] In any case, the timing difference set (indicating the distance between each of the first microphone and the 4 second microphones closest to the source in the recorder) is processed to form an observation vector. This time difference is proportional to the distance between two adjacent microphones to some extent. Under the plane wave assumption, given the 8 microphone recording device given in this article, a vector with 12 components can be formed. Therefore, any subset of vectors between microphone pairs in the octahedron forms a vertical Cartesian coordinate axis. The sound source at a certain position is closest to 4 of the microphones.

[0102] In step S10, an additional aspect relates to the processing of sound signals that move over time. For example, if there is more than one first microphone, an active speaker detection algorithm can be used to identify the current active speaker and the first microphone associated with it. For moving sound signals, a dynamic model and Kalman filtering can be used to estimate the location of the sound source at different times. The Kalman filter tracks the estimated state of the system and the variance or uncertainty of the estimate. The estimate is updated using a state transition model and measurements.

Claims

1. A method for obtaining the position of a sound source relative to a dedicated reference point, comprising the following steps: - obtaining a first sound signal recorded with a microphone at or in association with one or more sound sources; - obtaining a plurality of second sound signals, each of which is recorded at a known position relative to the dedicated reference point; wherein the first sound signal and the plurality of second sound signals are synchronized in time; -For the first sound signal: - calculating a frequency-weighted cross-correlation between at least one of the plurality of second sound signals and the first sound signal to obtain at least one frequency-weighted cross-correlation signal; - estimating a time delay between at least one of the plurality of second sound signals and the first sound signal by using at least one frequency weighted cross-correlation signal to estimate a distance between the sound source and the dedicated reference point; - estimating the angle between the sound source and the dedicated reference point by estimating the time delay between each pair of the plurality of second sound signals using a weighted least mean square, whereby the weighted least mean square depends on an obtained frequency-weighted cross-correlation signal between a pair of the plurality of second sound signals and the first sound signal.

2. The method according to claim 1, wherein: The calculation of the frequency-weighted cross-correlation comprises the following steps: - correlating at least one of the plurality of second sound signals with the first sound signal in the frequency domain to obtain at least one correlation signal; - modifying the power spectrum by frequency weighting to obtain at least one frequency weighted correlation signal; - transforming said at least one correlation signal into the time domain.

3. The method according to any one of claims 1 or 2, wherein: A respective frequency-weighted cross-correlation signal is calculated between each of the plurality of second sound signals and the first sound signal.

4. The method according to any one of claims 2 or 3, wherein: The step of transforming the at least one correlation signal into the time domain comprises - performing the transformation at a higher transformation frequency than the transformation frequency of the step of transforming at least one of the first sound signal and the plurality of second sound signals into the digital domain.

5. The method according to any one of claims 2 to 4, wherein: The step of associating the first sound signal comprises the following steps: - up-sampling the first sound signal and at least one of the plurality of second sound signals before correlating the first sound signal and at least one of the plurality of second sound signals in the digital domain.

6. The method according to any one of the preceding claims, further comprising - calculating a phase shift between said first sound signal and another first sound signal recorded at or associated with one or more sound sources to obtain another phase shift signal; - estimating a distance between the position of the microphone and a position associated with a recording of the further first sound signal by estimating a time delay between the first sound signal and the further first sound signal using the further phase shifted signal.

7. A method according to any one of the preceding claims, wherein: The step of calculating the frequency weighted cross-correlation, in particular the phase shift, comprises the following steps: - performing a short-time Fourier transform (STFT) on at least one of the first sound signal and the plurality of second sound signals to obtain a corresponding frequency spectrum; - Obtain cross-spectrum on each spectrum graph; - applying a spectral mask filter to the obtained complex cross-spectrum; - performing an inverse short time Fourier transform ISTFT to obtain at least one phase transformed signal.

8. A method according to any one of the preceding claims, wherein: The step of calculating the frequency weighted cross-correlation, in particular the phase shift, comprises: - estimating a filter acting on the signal-to-noise ratio in each frequency bin of said first sound signal, comprising the steps of: - applying a quantile filter, in particular a median filter, for smoothing the power spectrum of each time slice (k) of the power spectrum derived from the one or more first recorded sound signals; - estimating the noise of each time slice (k) in response to the previous time slice; - for a given frequency, evaluating whether the signal-to-noise ratio exceeds a predetermined threshold, and setting a filter parameter for said frequency to 1 or 0 in response to said evaluation.

9. The method according to claim 8, wherein: Said estimating a filter acting on a signal-to-noise ratio in each frequency bin of said first sound signal comprises applying a residual signal from a denoising process as a noise estimate, wherein said denoising process can optionally be based on machine learning.

10. A method according to any one of the preceding claims, wherein: The steps to estimate time delay include - searching for a maximum value, in particular a phase shift signal, in said at least one frequency-weighted cross-correlation; or - detecting a first amplitude value above a given threshold and searching for a maximum value within a specified window centered around said first amplitude value.

11. The method according to claim 10, wherein: The search for a maximum, in particular a phase shifted signal, in the at least one frequency weighted cross correlation depends on an estimate of the time delay in nearby time frames.

12. The method according to claim 10, wherein: The window length of the specified window depends on at least one of the following: - inversely proportional to a signal bandwidth estimated from the highest frequency component of said first sound signal; - expected early reflections, the expected early reflections depending on the distance between the recording location of the first sound signal and the location of the one or more sound sources; - proportional to a maximum flight time between the positions of the plurality of second sound signals.

13. A method according to any one of the preceding claims, wherein: The dedicated reference point is located substantially at the center between the recording positions of the plurality of second sound signals, and wherein estimating the distance between the sound source and the dedicated reference point comprises one of the following steps: - obtaining an average value of a set of time delays between each of at least one of the plurality of second sound signals and the first sound signal, - obtaining a time delay between a signal formed by a sum of at least two of the plurality of second sound signals and the first sound signal.

14. A method according to any one of the preceding claims, wherein: The weighted least squares depends on one of the following: - said weighted least mean square depends on the obtained frequency-weighted cross-correlation, in particular on the phase shift signal between said first sound signal and a pair of said plurality of second sound signals, if the obtained amplitude value of the phase shift signal is above a given threshold and within a specified window centered around said first amplitude value; - an arrival time difference of direct sound of the plurality of frequency-weighted cross-correlation signals, wherein the plurality of frequency-weighted cross-correlation signals are given by the first sound signal and the plurality of second sound signals.

15. The method according to any one of the preceding claims, further comprising: - applying a noise reduction filter to the estimated distance and / or the estimated angle; or - applying a Kalman filter to the estimated distance and / or the estimated angle; - Applying a gradient or divergence to the estimated distance and / or the estimated angle.

16. A method according to any one of the preceding claims, wherein: Respective positions of a pair of the plurality of second sound signals are located on a virtual line passing through the dedicated reference point and are at the same distance from the dedicated reference point.

17. A method according to any one of the preceding claims, wherein: The plurality of second sound signals includes at least four audio sound signals, wherein two of the four sound signals are recorded at a maximum spatial distance of 15 cm.

18. The method according to any one of the preceding claims, further comprising: - obtaining air temperature information, in particular, air temperature information near the plurality of second sound sources; and - estimating said distance and / or angle in response to the obtained air temperature information.

19. A computer system comprising: - one or more processors; - a memory coupled to the one or more processors and comprising instructions which, when executed by the one or more processors, cause the one or more processors to perform a method according to any one of the preceding claims.

20. A non-transitory computer-readable storage medium comprising computer-executable instructions for performing the method according to any one of the preceding claims.

21. A recording device comprising: - a cuboid shape having a bottom and a top and four side faces, the recording device being suitable for being placed with the bottom on the surface; - a user interface accessible on said top surface; - a plurality of microphones, in particular omnidirectional microphones, wherein a pair of microphones is arranged on each of the respective sides, a first microphone of the pair of microphones being arranged at the top of the respective side and a second microphone of the pair of microphones being arranged at the bottom of the respective side; - wherein the distance between the first microphone and the second microphone of each pair of microphones is equal to the distance between the first microphones of adjacent sides.

22. The recording device according to claim 20, wherein A distance from the first microphone to the top surface is greater than a distance from the second microphone to the bottom surface.

Images