System and method for generating spatial audio with unified reverberation in real-time communication

Abstract

The present application provides a computer-implemented method for generating spatial audio with uniform reverberation in real-time communication sessions, executed by a real-time communication software application running on an electronic communication device. The method comprises: removing reverberation from a recorded speech signal of a far-end participant by a dereverberation method; generating a direct sound part by filtering the output signal through a head-related transfer function of a target direction; generating a reverberation part by convolving the output signal with a uniform room impulse response or an artificial reverberator; and combining the direct sound and the reverberation to generate a spatialized speech signal. If the speaker and the listener are located in two virtual conference rooms, the reverberation of the two rooms is joint; at this time, the output signals from the two rooms are convolved with the joint RIR to generate the reverberation part.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Patent Application No. 17,859 / 181, filed on July 7, 2022. Technical Field

[0003] This invention relates to noise suppression technology in voice communication, specifically to a system and method for generating spatial audio with uniform reverberation in real-time communication, and more specifically to a system and method for generating spatial audio with uniform reverberation in real-time virtual reality communication. Background Technology

[0004] Technological advancements have made real-time audio communication and video and / or messaging communication (RTC) possible over the internet. In RTC, all participants can interact instantly or with negligible transmission latency. One important application of RTC is teleconferencing, such as audio or video conferencing. In teleconferencing, participants often use headsets (or earpieces). Binaural stereo technology enables the spatialization of speech signals received by near-end participants. As used herein, two participants in the same physical environment (e.g., the same room) are referred to as each other's near-end participants; otherwise, they are referred to as each other's far-end participants. Furthermore, a participant's own voice is also referred to as that participant's near-end speech. In other words, a participant is also their own near-end participant.

[0005] Spatialization allows participants to virtually position remote participants in any way they wish. A traditional solution for creating virtual sound is to filter the received monophonic speech signal using a pair of head-related transfer functions (HRTFs), which define the direction-dependent transfer function between the sound source and the listener's eardrum. Furthermore, reverberation (e.g., early reflections and late reverberation) is added to binaural sound to enhance the perceived realism and externalization of the virtual sound.

[0006] Virtual Reality (VR) communication is a form of Real-Time Communication (RTC). In a VR scenario, participants are in the same virtual conference room. Therefore, the reverberation of the speech signal should be uniform (i.e., consistent and homogeneous). Otherwise, the immersive experience would be diminished. However, achieving this effect may not be easy because participants are in different physical environments, and the speech signal captured by the microphone already contains the reverberation of different rooms. Furthermore, because the speaker's voice includes the reverberation of their room, the speaker cannot perceive their own voice as if they were speaking in a virtual conference room. If the speaker is wearing headphones, the headphone shell blocks the high-frequency components of their own voice from traveling through the air. Therefore, the low-frequency components of the sound are amplified at the eardrum, a phenomenon known as the occlusion effect. The speaker's own voice sounds unnatural. Therefore, the high-frequency components of the sound should not be affected by the headphone shell.

[0007] Therefore, a new real-time communication system and method are needed to generate and provide uniform reverberation in virtual reality communications. This real-time communication system and method creates and enhances an immersive experience for listeners and speakers in the same or different virtual conference rooms. Summary of the Invention

[0008] This invention provides a computer-implemented method for generating spatial audio with uniform reverberation in a real-time communication session, based on various embodiments. The method is executed by a real-time communication software application in an electronic communication device and includes: acquiring a first audio signal stream from a first electronic communication device; removing reverberation from the first audio signal stream to generate a first dry signal stream; filtering the first dry signal stream using a head-correlation transfer function to generate a first direct sound component for binaural sound for a listener; acquiring a second audio signal stream from a second electronic communication device; removing reverberation from the second audio signal stream to generate a second dry signal stream; filtering the second dry signal stream using a head-correlation transfer function to generate a second direct sound component for binaural sound for a listener; and adding the first and second dry signal streams to generate a summed dry signal stream. The summed dry signal streams are convolved with a set of room impulse responses to generate the reverberation component of the binaural sound. The first and second direct sound components of the binaural sound are added together to generate the first summed direct sound component of the binaural sound for the left ear. The first and second direct sound components of the binaural sound are added together to generate the second summed direct sound component of the binaural sound for the right ear. The first summed direct sound component of the binaural sound for the left ear is added to the reverberation component of the binaural sound to generate the first audio signal for the listener's left ear. The second summed direct sound component of the binaural sound for the right ear is added to the reverberation component of the binaural sound to generate the second audio signal for the listener's right ear.

[0009] In one implementation, a real-time communication session refers to a real-time virtual reality communication session.

[0010] In one embodiment, the set of room impulse responses includes a pair of room impulse responses.

[0011] In one implementation, each room impulse response in the group of room impulse responses is obtained by measurement from a reference conference room or is artificially synthesized.

[0012] In one implementation, the step of removing reverberation from the first audio signal stream to generate the first dry signal stream employs a dereverberation model.

[0013] In one implementation, the step of removing reverberation from the second audio signal stream to generate the second dry signal stream also employs a dereverberation model.

[0014] This invention provides a computer-implemented method for generating spatial audio with uniform reverberation in a real-time communication session. The method is executed by a real-time communication software application in an electronic communication device, comprising: acquiring a first audio signal stream from a first electronic communication device; removing reverberation from the first audio signal stream to generate a first dry signal stream; filtering the first dry signal stream using a head-correlation transfer function to generate a first direct sound component for the left ear (binaural sound) for a listener; acquiring a second audio signal stream from a second electronic communication device; removing reverberation from the second audio signal stream to generate a second dry signal stream; filtering the second dry signal stream using a head-correlation transfer function to generate a second direct sound component for the right ear (binaural sound) for a listener; generating a first early reflection for a first distal participant corresponding to the first electronic communication device; generating a second early reflection for a second distal participant corresponding to the second electronic communication device; adding the first and second dry signal streams to generate a summed dry signal stream; convolving the summed dry signal stream with a set of room impulse responses to generate a reverberation component for the binaural sound; and combining the first direct sound component with the second... The first direct sound component is generated by adding the two direct sound components together to produce the first summed direct sound component for the left ear; the first and second direct sound components are added together to produce the second summed direct sound component for the right ear; the first and second early reflections are added together to generate the first summed early reflection for the listener's left ear; the first and second early reflections are added together to generate the second summed early reflection for the listener's right ear; the first summed direct sound component for the left ear is added to the first summed early reflection for the left ear to generate the second summed early reflection for the left ear. The first audio signal for the left ear is generated by adding the first set of binaural sounds with early reflections; the second set of binaural sounds with early reflections for the right ear is added to the second set of binaural sounds with early reflections for the right ear to generate a second set of binaural sounds with early reflections for the right ear; the first set of binaural sounds with early reflections for the left ear is added to the reverberation of the binaural sounds to generate a first audio signal for the left ear; the second set of binaural sounds with early reflections for the left ear is added to the reverberation of the binaural sounds to generate a second audio signal for the right ear.

[0015] In one implementation, a real-time communication session refers to a real-time virtual reality communication session.

[0016] In one embodiment, the set of room impulse responses includes a pair of room impulse responses.

[0017] In one implementation, each room impulse response in the group of room impulse responses is obtained by measurement from a reference conference room or is artificially synthesized.

[0018] In one implementation, the step of removing reverberation from the first audio signal stream to generate the first dry signal stream employs a dereverberation model.

[0019] In one implementation, the step of removing reverberation from the second audio signal stream to generate the second dry signal stream also employs a dereverberation model.

[0020] Furthermore, this invention also provides a computer-implemented method for generating spatial audio with uniform reverberation for a headphone user in a real-time communication session. The method is executed by a real-time communication software application in an electronic communication device and includes: acquiring an audio signal stream of the user's speech from the audio input interface of the electronic communication device; removing reverberation from the audio signal stream to generate a dry signal stream; convolving the dry signal stream with a set of room impulse responses to generate a reverberant portion of the binaural sound; filtering the reverberant portion of the binaural sound using a high-pass filter to form a high-pass filtered signal; filtering the high-pass filtered signal using an oral-ear transfer function to form an oral-ear transfer function filtered signal; filtering the oral (left) ear transfer function filtered signal using a first set of headphone compensation filters to generate a first audio signal for the user's left ear; and filtering the oral (right) ear transfer function filtered signal using a second set of headphone compensation filters to generate a second audio signal for the user's right ear.

[0021] In one implementation, a real-time communication session refers to a real-time virtual reality communication session.

[0022] In one embodiment, the set of room impulse responses includes a pair of room impulse responses.

[0023] In one implementation, each room impulse response in the group of room impulse responses is obtained by measurement from a reference conference room or is artificially synthesized.

[0024] In one implementation, the step of removing reverberation from the first audio signal stream to generate the first dry signal stream employs a dereverberation model. Attached Figure Description

[0025] This patent or application document contains at least one color drawing. The Patent Office will, upon request and upon payment of the relevant fees, provide a copy of this patent or application disclosure with color drawings.

[0026] The technical features of the invention will be specifically pointed out in the claims, and the invention, its structure, and its method of use can also be better understood by referring to the following description and the accompanying drawings, which form a part of the description. All the accompanying drawings also constitute a part of the invention, wherein the same reference numerals denote the same parts:

[0027] Figure 1This is a simplified block diagram of a virtual conference room in real-time communication, drawn according to an embodiment of the present invention.

[0028] Figure 2 is a flowchart of a method for processing wet audio signals to generate dry audio signals according to an embodiment of the present invention.

[0029] Figure 2B This is a flowchart illustrating a method for processing wet audio signals to generate dry audio signals according to an embodiment of the present invention.

[0030] Figure 3 This is a flowchart illustrating a method for generating spatial sound with uniform reverberation from remote participants in real-time communication, as described in an embodiment of the present invention.

[0031] Figure 4 This is a flowchart illustrating a method for generating spatial audio with uniform reverberation and early reflections in real-time communication, as described in an embodiment of the present invention.

[0032] Figure 5 This is a flowchart illustrating a method for generating spatial sound with uniform reverberation in real-time communication, as described in an embodiment of the present invention.

[0033] Figure 6 This is a flowchart illustrating a method for generating spatial sound with uniform reverberation when a listener uses headphones in real-time communication, according to an embodiment of the present invention.

[0034] Figure 7 This is a simplified block diagram of an electronic communication device according to an embodiment of the present invention.

[0035] Figure 8 This is a simplified block diagram of two real-time communication virtual conference rooms drawn according to an embodiment of the present invention.

[0036] Those skilled in the art will understand that, for the sake of clarity and simplicity, the various components in the accompanying drawings are not necessarily drawn to scale. The dimensions of some components in the drawings may be enlarged relative to others to aid in understanding the invention. Furthermore, the specific order of certain elements, parts, components, modules, steps, operations, events, and / or processes described or illustrated herein may not be necessary in practical application. Those skilled in the art will understand that, for the sake of clarity and simplicity, useful and / or necessary elements that are well-known and readily understood in existing feasible embodiments may not be described herein, in order to clearly present various embodiments of the invention. Detailed Implementation

[0037] A novel real-time (RT) spatial audio rendering system can provide uniform reverberation for participants in real-time virtual reality communication sessions over the Internet. Figure 1 An exemplary virtual conference room, designated by reference numeral 100, is shown. This virtual conference room 100 (e.g., a virtual reality conference room) includes three participants 102, 104, and 106, located in three separate physical rooms 112, 114, and 116, respectively. Participants 102 and 104 are remote participants of participant 106. In one embodiment, participant 106 wears headphones 118 (or headsets, such as virtual reality headsets); in another embodiment, participant 106 uses an audio output device, such as a speaker system operatively coupled to electronic communication device 126.

[0038] Participants 102, 104, and 106 respectively operate (or otherwise use) their respective electronic communication devices 122, 124, and 126 to participate in a real-time communication session between them via a network (e.g., the Internet). In one embodiment, the session refers to a real-time virtual reality communication session. The electronic communication device may be a smartphone, tablet, laptop, desktop computer, etc. Figure 7 Further explanation will be given regarding electronic communication equipment (e.g., electronic communication equipment 122).

[0039] Please refer to Figure 7 , Figure 7 This is a schematic block diagram of an electronic communication device (e.g., electronic communication device 122). Electronic communication device 122 includes a processor 702 (e.g., a central processing unit), a number of memories 712 operably coupled to the processor 702, an audio input interface 714 (e.g., a microphone) operably coupled to the processor 702, an audio output interface 716 (e.g., a speaker) operably coupled to the processor 702, a video input interface 718 (e.g., a camera) operably coupled to the processor 702, a video output interface 720 (e.g., a video controller and display screen) operably coupled to the processor 702, and a network interface 722 operably coupled to the processor 702. In one embodiment, electronic communication device 122 also includes an operating system 704 and a dedicated real-time communication software application 706 suitable for execution by the processor 702. The real-time communication software application 706 is programmed using one or more computer programming languages ​​(e.g., C, C++, C#, Java, etc.) and includes components and modules for generating spatial sound with uniform reverberation.

[0040] Reverberation provides a sense of immersion, similar to a virtual room. However, excessive reverberation can reduce sound clarity, making it unsuitable for certain situations, such as virtual meetings with multiple participants over the internet. Traditional dereverberation methods (i.e., methods to eliminate reverberation), such as the Weighted Prediction Error (WPE) method, require multiple audio channels as input. However, in RTC scenarios, many devices only support mono audio acquisition. For mono audio dereverberation, one traditional solution is to use machine learning (ML)-based dereverberation methods, which can perform mono speech dereverberation better than other traditional methods. ML-based dereverberation methods require paired datasets of dry speech signals (i.e., reverberant-free speech signals) and wet speech signals (i.e., reverberant speech signals) along with a noise dataset as training data to train the dereverberation model. During training, noise is typically added to the wet speech signal as input to the dereverberation model, while the dry speech signal is used as the output, thereby enhancing the performance of the dereverberation model in noisy environments. After training, this dereverberation model learns to recover the dry speech signal from the wet speech signal, even when the speech is noisy. The principle behind this dereverberation model is based on, for example, convolutional neural networks (CNNs), recurrent neural networks (RNNs), attention-based neural networks (ANNs), or combinations of these networks.

[0041] Figure 2A and 2B De-reverberation methods applying traditional ML-based dereverberation models are described respectively; these dereverberation methods are respectively based on... Figure 2A Method 200 and Figure 2BMethod 250 is briefly described below. De-reverberation methods 200 and 250 are performed by a dedicated real-time communication software application 706. In step 202, the dedicated real-time communication software application 706 predicts the spectrum of the dry speech signal. In step 204, the dedicated real-time communication software application 706 preprocesses the wet speech signal from step 202 using a Short-Time Fourier Transform (STFT). It converts the wet speech signal from step 202 into a spectrum, which serves as the input signal to the trained de-reverberation model in step 206. The de-reverberation model in step 206 outputs the spectrum of the dry signal. In step 208, post-processing is performed using an inverse STFT to obtain the waveform of the dry speech signal from step 210. Figure 2B The method 250 shown can also be used to directly predict the dry speech signal using a dedicated real-time communication software application 706. In this case, the dereverberation model trained in step 206 is also referred to herein as an end-to-end model. The end-to-end model does not require the preprocessing in step 204 or the postprocessing in step 208.

[0042] This invention provides a system and method for generating spatial sound with uniform reverberation in a VR scene from, for example, remote participants (e.g., users 102, 104 to user 106) and near participants. User 106 listens to the audio using headphones 118. Figure 3 , 4 Sections 5 and 6 will further elaborate on this process performed by the real-time communication software application 706.

[0043] First turn Figure 3 , Figure 3 A novel real-time communication software application 706 is described, denoted by reference numeral 300, for generating spatial sound with uniform reverberation from remote participants (such as users 102, 104 and devices 122, 123). Method 300 also describes the data flow between various steps. Electronic communication devices 122, 124 respectively capture the speech of users 102, 104 and transmit it to the electronic communication device 126 of participant 106 via a network connection (e.g., the Internet). The audio data of the user's speech is transmitted in the form of an audio signal. One signal stream represents audio data within a time period (e.g., one millisecond or two milliseconds). Electronic communication device 126 outputs the processed audio signal with uniform reverberation to an audio output device (e.g., a speaker system).

[0044] In step 302, a dedicated real-time communication software application 706 acquires a first audio signal stream received from a first user's electronic device (e.g., electronic communication device 122). In step 304, the real-time communication software application 706 removes reverberation from the first audio signal stream to generate a first dry signal stream. In one embodiment, method 200 is performed at step 304, or method 250 may also be performed at step 304. In step 306, the real-time communication software application 706 filters the first dry signal stream in the desired direction using a head correlation transfer function (HRTF) to generate a first direct sound portion of binaural sound for the listener (e.g., user 106). The desired direction refers to the direction relative to user 106 in the virtual conference room 116. The HRTF does not contain any room information and can be taken from any relevant database, such as the CIPIC database, MIT database, IRCAM database, or measured separately. In this invention, binaural sound (also referred to herein as binaural signal) includes audio signals from the left ear and audio signals from the right ear.

[0045] In step 352, a dedicated real-time communication software application 706 acquires a second audio signal stream received from a second user's electronic device (e.g., electronic communication device 124). In step 354, the real-time communication software application 706 removes reverberation from the second audio signal stream to generate a second dry signal stream. In one embodiment, method 200 is performed at step 354. Alternatively, method 250 may be performed at step 354. In step 356, the real-time communication software application 706 filters the second dry signal stream in the desired direction using HRTF to generate a second direct-sound portion of binaural sound for the listener (e.g., user 106).

[0046] In step 332, the real-time communication software application 706 adds the first and second dry signal streams to generate a summed dry signal stream. In RTC, audio signals are typically processed in frames, and summation is performed by adding the first and second frames of the dry signal streams (e.g., a 10-millisecond frame at a sampling rate of 48 kHz, which generates 480 samples). In step 334, the dedicated real-time communication software application 706 convolves the summed dry signal stream with a set (i.e., one or more) of room impulse responses (RIRs) to generate a reverberant portion of the binaural sound for the listener (e.g., user 106). In one embodiment, in step 334, the room impulse response comprises a pair of RIRs without direct sound, which are processed to generate the reverberant portion of the binaural sound. The RIRs contain room-related auditory information, which can be obtained by measurement or synthesized artificially. Importantly, the room-related auditory information contained in the pair of RIRs used should match the information provided by the virtual room. RIRs can be pre-configured and stored in a server computer system (e.g., a cloud server system). For example, three types of meeting rooms can be defined based on size: small, medium, and large meetings. Therefore, three sets of RIRs for these three types of meeting rooms can be pre-configured and stored in the server computer system. These three sets of RIRs can be obtained through measurement (in a real room of the same size) or through artificial synthesis. In one implementation, the corresponding RIR is loaded from the server computer system and used according to the selected virtual meeting room. In step 334, the real-time communication software application 706 loads the stored RIR from the server according to the selected virtual meeting room. The direct sound to reverberation energy ratio is selected based on the virtual distance between the remote participant and the listener. Figure 1 In the embodiment shown, the remote participants are participants 102 and 104, and the listener is participant 106.

[0047] In step 308, the real-time communication software application 706 adds the first and second direct-sound components of the binaural sound from the left ear to generate the first summed direct-sound component of the binaural sound from the left ear. In other words, it adds the left-ear binaural sound from the first and second direct-sound components. In step 358, the real-time communication software application 706 adds the first and second direct-sound components of the binaural sound from the right ear to generate the second summed direct-sound component of the binaural sound from the right ear. The signals received by the left and right ears are different. One reason is the use of HRTF, a function that simulates the acoustic transmission path from the sound source to the listener's two ears. For example, if the sound source is from the right, the signal sound pressure level in the right ear is higher than that in the left ear, and the signal takes longer to travel to the left ear than to the right ear. In step 310, the real-time communication software application 706 adds the direct sound portion of the first summed binaural sound to the reverberation portion of the generated binaural sound to generate a first audio signal for the left ear of a listener (e.g., user 106). In step 360, the real-time communication software application 706 adds the direct sound portion of the second summed binaural sound to the right ear to generate a second audio signal for the right ear of a listener (e.g., user 106). The first audio signal generated in step 310 and the second audio signal generated in step 360 are played through an audio output device (e.g., a speaker system operatively coupled to the electronic communication device 126) or otherwise output.

[0048] Sound reflection density increases over time. Early reflections are observed within milliseconds of the arrival of the direct sound, including several discrete reflections from walls, ceilings, and floors. The temporal structure of early reflections depends heavily on the geometry of the room and the listener's position relative to the sound source. These factors play a significant role in sound source localization, rendering, and speech intelligibility. Late reverberation, consisting of high-density reflections, contributes to the listener's sense of immersion. The point in time between early reflections and late reverberation is called the mixing time. To produce more realistic reverberation, early reflections and late reverberation should be generated separately.

[0049] Figure 4 A method for generating uniform reverberation in real-time communication is shown, which incorporates processing of early reflections. (Reference) Figure 4 , Figure 4 A novel real-time communication software application 706 is shown, illustrating a method for generating spatial sound with uniform reverberation from remote participants (e.g., users 102, 104 and electronic communication devices 122, 123) in a VR scene that takes into account early reflections. This method is denoted by reference numeral 400. Method 400 also demonstrates the data flow between the various steps.

[0050] In step 402, a dedicated real-time communication software application 706 acquires a first audio signal stream from the first user's electronic device (e.g., electronic communication device 122). In step 404, the real-time communication software application 706 removes reverberation from the first audio signal stream to generate a first dry signal stream. In one embodiment, method 200 may be performed in step 404, or method 250 may be performed in step 404. In step 406, the real-time communication software application 706 filters the first dry signal stream in the desired direction using HRTF to generate a first direct sound portion of binaural sound for the listener (e.g., user 106). The desired direction refers to the direction relative to user 106 in the virtual conference room 116.

[0051] In step 452, a dedicated real-time communication software application 706 acquires a second audio signal stream from a second user's electronic device (e.g., electronic communication device 124). In step 454, the real-time communication software application 706 removes reverberation from the second audio signal stream to generate a second dry signal stream. In one embodiment, method 200 may be performed in step 454, or method 250 may be performed in step 454. The real-time communication software application 706 filters the second dry signal stream in the desired direction using HRTF to generate a second direct-sound portion of binaural sound for the listener (e.g., user 106).

[0052] In step 432, the real-time communication software application 706 adds the first and second dry signal streams to generate a summed dry signal stream. In step 434, the dedicated real-time communication software application 706 convolves the summed dry signal stream with a set of RIRs to generate the reverberant portion of the binaural sound for user 106. In one embodiment, in step 434, the set of RIRs includes a pair of RIRs without a direct sound portion to generate the reverberant portion of the binaural sound. The RIRs can be obtained by measuring a reference conference room or by artificial synthesis. The RIRs can be pre-configured and stored on a server computer system (e.g., a cloud server system). In step 434, the real-time communication software application 706 loads the stored RIRs from the server according to the selected virtual conference room. The energy ratio of direct sound to reverberation is selected based on the virtual distance between the remote participant and the listener. In the illustration of method 400, the remote participant refers to participants 102 and 104, and the listener refers to participant 106.

[0053] In step 408, the real-time communication software application 706 generates a first early reflection for the first remote participant 102. In step 458, the real-time communication software application 706 generates a second early reflection for the first remote participant 104. The generation of the early reflection requires a room model (e.g., the geometry of a virtual meeting room 100). In one embodiment, the room model refers to a three-dimensional model of room 100. In other embodiments, the room model simply refers to the dimensions of the room.

[0054] In step 434, the post-reverberation component is obtained through convolution operations. Alternatively, an artificial reverberator (e.g., a Schroeder reverberator, a feedback delay network, a scattering delay network, etc.) can be used to synthesize the post-reverberation component. The mixing time between the early reflections and the post-reverberation depends on the geometry or reverberation time of the virtual room 100. Different formulas can be used to calculate the mixing time. In one embodiment, the mixing time can be derived from the following formula:

[0055]

[0056] Where V is the volume of the virtual meeting room and Tmix is ​​the mixing time.

[0057] In step 410, the real-time communication software application 706 adds the first and second direct sound components of the binaural sound of the listener (e.g., user 106) to generate the first summed direct sound component of the binaural sound for the left ear of the listener (e.g., user 106). In step 460, the real-time communication software application 706 adds the first and second direct sound components of the binaural sound to generate the second summed direct sound component of the right ear of the listener (e.g., user 106).

[0058] In step 412, the real-time communication software application 706 adds the first early reflection of the first remote participant 102 and the second early reflection of the second remote participant 104 in the virtual meeting room 100 to generate a first summed early reflection for the left ear of the listener (e.g., user 106). In step 462, the real-time communication software application 706 adds the first early reflection of the first remote participant 102 and the second early reflection of the second remote participant 104 in the virtual meeting room 100 to generate a second summed early reflection for the right ear of the listener (e.g., user 106).

[0059] In step 414, the real-time communication software application 706 adds the first summed direct sound portion of the binaural sound in the left ear to the first summed early reflection, generating a first summed direct sound portion of binaural sound with early reflection for the left ear of the listener (e.g., user 106). In step 464, the real-time communication software application 706 adds the second summed direct sound portion of the binaural sound in the right ear to the second summed early reflection, generating a second summed direct sound portion of binaural sound with early reflection for the right ear of the listener (e.g., user 106).

[0060] In step 416, the real-time communication software application 706 adds the direct sound portion of the summed binaural sounds with early reflections to the reverberation portion of the binaural sounds to generate a first audio signal for the left ear of a listener (e.g., user 106). Because the perceived reverberation of the speech signals from users 102 and 104 is the same as that of user 106, the binaural reverberation is uniform. In step 466, the real-time communication software application 706 adds the direct sound portion of the summed binaural sounds with early reflections to the reverberation portion of the binaural sounds to generate a second audio signal for the right ear of a listener (e.g., user 106). The first audio signal generated in step 416 and the second audio signal generated in step 466 are then played through an audio output device (e.g., a stereo speaker system) or otherwise output.

[0061] In addition, there is a method for generating spatial sound signals with uniform reverberation, in which reverberation is added directly to the dry speech signal after a déreverberation process. In this method, the dry signal is filtered using a uniform RIR (with the direct sound component) or an artificial reverberator, and then the reverberation signal is spatialized using an HRTF filter. This process will... Figure 5 Further details are provided below. (See reference.) Figure 5 , Figure 5 A method for generating spatial sound with uniform reverberation is demonstrated via a real-time communication software application 706, denoted by reference numeral 500.

[0062] In step 502, real-time communication software application 706 acquires a first audio signal stream from a first user's electronic device (e.g., electronic communication device 122). In step 504, real-time communication software application 706 removes reverberation from the first audio signal stream to generate a first dry signal stream. In one embodiment, method 200 may be performed in step 504, or method 250 may be performed in step 504. In step 506, dedicated real-time communication software application 706 convolves the first dry signal stream with a set of RIRs (e.g., a pair of RIRs) to generate a first reverberant portion of binaural sound. In step 508, real-time communication software application 706 filters the first reverberant portion of binaural sound to generate a first direct sound portion of binaural sound for the listener (e.g., user 106).

[0063] In step 552, the real-time communication software application 706 acquires a second audio signal stream from the second user's electronic device (e.g., electronic communication device 124). In step 554, the real-time communication software application 706 removes reverberation from the second audio signal stream to generate a second dry signal stream. In one embodiment, method 200 is performed in step 554, or method 250 may also be performed in step 554. In step 556, the dedicated real-time communication software application 706 convolves the second dry signal stream with a set of RIRs (e.g., a pair of RIRs) to generate a second reverberant portion of the binaural sound. In step 558, the real-time communication software application 706 filters the second reverberant portion of the binaural sound to generate a second direct sound portion of the binaural sound for the listener (e.g., user 106).

[0064] In step 510, the real-time communication software application 706 adds the first and second direct sound components of the binaural sound from the listener's (e.g., user 106) left ear. In step 560, the real-time communication software application 706 adds the first and second direct sound components of the binaural sound from the listener's (e.g., user 106) right ear. The output signals of steps 508 and 558 are "binaural signals," i.e., left ear signal and right ear signal. Figure 5 In step 508 or 558, the left arrow outputs the left ear signal, and the right arrow outputs the right ear signal. Therefore, the input to step 510 is the left ear signal from steps 508 and 558, and the input to step 560 is the right ear signal from steps 508 and 558. The audio signals from steps 510 and 560 are then played through an audio output device (e.g., a speaker system) or output in other ways.

[0065] In one implementation, user 106 is equipped with headphones 118 while participating in a virtual reality conference room 100. When user 106 speaks using headphones 118, the user's voice travels through the air and is blocked by the outer shell of headphones 118. The outer shell of headphones 118 has a low-pass filtering effect. The system and method provided in this invention simulate the high-frequency components of the speaker's own voice, filtered by the headphone shell and transmitted through the air. For details on optimizing the immersive experience of listener 106 while wearing headphones 118, see [link to documentation]. Figure 6 .

[0066] refer to Figure 6 , Figure 6 A method, denoted by reference numeral 600, is illustrated for a novel real-time communication software application 706 to generate spatial sound with uniform reverberation in a VR scene when a listener (e.g., user 106) participates in a virtual reality conference room using headphones 118. Method 600 also demonstrates the data flow between the various steps.

[0067] In step 602, a dedicated real-time communication software application 706 acquires the audio signal stream of user 106 speaking. In step 604, the real-time communication software application 706 removes reverberation from the audio signal stream to generate a dry signal stream. In one embodiment, method 200 is performed at step 604, or method 250 may also be performed at step 604. In step 606, the real-time communication software application 706 convolves the dry signal stream with a set of RIRs (e.g., a pair of RIRs) to generate the reverberant portion of the binaural sound. In the convolution operation, the energy ratio of direct sound to reverberation needs to be selected based on the distance between the mouth and the microphone. In step 608, the real-time communication software application 706 uses a high-pass filter to filter the reverberant portion of the binaural sound. Since the housing of the earphone 118 significantly attenuates high-frequency components but not low-frequency components, in one embodiment, the audio signal may be filtered by a high-pass filter with a cutoff frequency of approximately 1 kHz. In step 610, the real-time communication software application 706 uses METF to filter the high-pass filtered signal. In one implementation, METF is measured using a head model (e.g., a head-to-torso simulator).

[0068] In step 612, the real-time communication software application 706 further filters the signal, which has already been filtered by the mouth-to-ear transfer function, using a first set of headphone compensation filters to compensate for the influence of the headphones and generate a first audio signal for the listener's (e.g., user 106) left ear. In step 614, the real-time communication software application 706 further filters the signal, which has already been filtered by the mouth-to-ear transfer function, using a second set of headphone compensation filters to compensate for the influence of the headphones and generate a second audio signal for the listener's (e.g., user 106) right ear. The first audio signal generated in step 612 and the second audio signal generated in step 614 are then transmitted to the headphone 118.

[0069] The delay in the signal processing chain should match the actual transmission delay of the speech signal from mouth to ear. The ideal delay from mouth to ear can be determined by the initial delay of the METF (i.e., the linear phase component). In signal processing, the initial delay of the METF should be removed to allow time for other signal processing units. Microphone placement is particularly important. The distance between the mouth and microphone should be as short as possible and no greater than the distance between the mouth and ear. Processing time depends on computing power, microphone placement, and the order of different filters. Certain steps (e.g., dereverberation and HpCF filtering) can be pre-defined and stored to minimize processing time in real-time communication. If the delay caused by signal processing is less than the required delay from mouth to ear, additional delay time can be added.

[0070] refer to Figure 8 , Figure 8 The real-time communication between two different virtual meeting rooms is shown. The second virtual meeting room is designated 800. Participant 802 uses electronic communication device 822 and is located in physical room 812. In this invention, room 100 and room 800 are referred to as a set of combined rooms.

[0071] The system and method provided by this invention can simultaneously enhance the immersive experience for both listeners and speakers in different virtual meeting rooms. Reverberation generated by the microphone in the recorded audio signal is removed to generate a dry signal, which is used to simulate the direct sound and reverberation components. The main difference lies in the use of RIR to generate reverberant sound in single and multiple virtual meeting room scenarios. In a single virtual meeting room scenario, only the reverberation of one virtual meeting room needs to be considered. Conversely, in a multiple virtual meeting room scenario, the reverberation of two different virtual rooms needs to be combined to simulate the effect of a speaker in one room and a listener in another.

[0072] In RIR, the envelope of the reverberation component is usually obtained by approximation using an exponential function:

[0073]

[0074] Where γ is the weight, T is the slope decay rate, and h env henv(t) is an exponential function, where t is time. The value of T depends on the room's reverberation time. A simple way to generate the reverberation component in RIR is to multiply henv(t) by white noise. This operation can be performed in different frequency bands.

[0075] If there are two combined rooms, the envelope of the reverberation component can be considered as a linear combination of the two reverberation envelopes in the two rooms:

[0076]

[0077] Where A and B refer to room A and room B respectively, h env (t) is an exponential function, γ0, γA, and γB are the weights to be fitted, and T A T is the reverberation decay rate of room A. B t is the reverberation decay rate of room B, and t is time. Artificial reverberation devices (e.g., grouped feedback delay networks) can also be used to simulate the reverberation of the combined room. Early reflections can be simulated using methods such as image source methods and ray tracing, depending on the listener's position, speaker location, and the geometry of the combined room.

[0078] Based on the above description, it is obvious that many other modifications and variations are possible with respect to the present invention. Therefore, please note that within the scope of the appended claims, the present invention may be implemented in ways different from those specifically described above.

[0079] This specification is provided for better illustration and explanation, and is not intended to be exclusive or to limit the invention to the specific forms described above. The foregoing description is intended to better explain the principles of the invention and their practical application, enabling those skilled in the art to better utilize the invention to implement various embodiments and make various modifications for the intended suitable uses. It should be understood that the words “a” or “an” as used herein include both singular and plural forms. Conversely, where appropriate, the case of multiple elements mentioned herein should also include their singular forms.

[0080] The scope of this invention is not limited to the contents of the above description, but is defined by the claims. Furthermore, although the claims may seem narrow, it should be understood that the scope of this invention is much broader than that defined by the claims. We will file broader claims in one or more applications claiming priority to this application. Any content disclosed in the above description and drawings that is not included within the scope of the claims is not disclosed herein, and we reserve the right to file one or more patent applications regarding such content in the future.

Claims

1. A computer-implemented method for generating spatial audio with uniform reverberation in a real-time communication session, wherein, The method is executed by a real-time communication software application in an electronic communication device, and the method includes: 1) Obtain a first audio signal stream from a first electronic communication device; 2) Remove reverberation from the first audio signal stream to generate a first dry signal stream; 3) The first dry signal stream is filtered by the head-related transfer function to generate the first direct sound part of the binaural sound for the listener; 4) Obtain the second audio signal stream from the second electronic communication device; 5) Remove the reverberation from the second audio signal stream to generate a second dry signal stream; 6) The second dry signal stream is filtered by the head-related transfer function to generate a second direct sound component of binaural sound for the listener; 7) Add the first dry signal stream and the second dry signal stream to generate a summed dry signal stream; 8) Convolve the summed dry signal stream with a set of room impulse responses to generate the reverberation component of the binaural sound; 9) Add the first direct sound portion and the second direct sound portion of the binaural sound to generate the first direct sound portion of the binaural sound in the left ear; 10) Add the first direct sound portion and the second direct sound portion of the binaural sound to generate the second direct sound portion of the binaural sound in the right ear; 11) Add the direct sound portion of the first summed binaural sound of the left ear to the reverberation portion of the binaural sound to generate a first audio signal for the listener's left ear; and 12) Add the direct sound portion of the second summed binaural sound of the right ear to the reverberation portion of the binaural sound to generate a second audio signal for the listener's right ear.

2. The method according to claim 1, wherein, The real-time communication session refers to a real-time virtual reality communication session.

3. The method according to claim 1, wherein, The set of room impulse responses includes a pair of room impulse responses.

4. The method according to claim 3, wherein, Each room impulse response in the set of room impulse responses was measured from a reference conference room or synthesized artificially.

5. The method according to claim 1, wherein, The step of removing reverberation from the first audio signal stream to generate the first dry signal stream employs a dereverberation model.

6. The method according to claim 5, wherein, The step of removing reverberation from the second audio signal stream to generate the second dry signal stream employs the aforementioned dereverberation model.

7. A computer-implemented method for generating spatial audio with uniform reverberation in a real-time communication session, wherein, The method is executed by a real-time communication software application in an electronic communication device, and the method includes: 1) Obtain a first audio signal stream from a first electronic communication device; 2) Remove reverberation from the first audio signal stream to generate a first dry signal stream; 3) The first dry signal stream is filtered using the head-related transfer function to generate the first direct sound part of the left ear binaural sound for the listener; 4) Obtain the second audio signal stream from the second electronic communication device; 5) Remove the reverberation from the second audio signal stream to generate a second dry signal stream; 6) The second dry signal stream is filtered by the head-related transfer function to generate a second direct sound component of the right ear binaural sound for the listener; 7) Generate a first early reflection for the first remote participant corresponding to the first electronic communication device; 8) Generate a second early reflection for the second remote participant corresponding to the second electronic communication device; 9) Add the first dry signal stream and the second dry signal stream together to generate the added dry signal stream; 10) Convolve the summed dry signal stream with a set of room impulse responses to generate the reverberation component of the binaural sound; 11) Add the first direct sound portion and the second direct sound portion to generate the first summed direct sound portion of the left ear's two-ear sound; 12) Add the first direct sound portion and the second direct sound portion to generate the second summed direct sound portion of the right ear's two-ear sound; 13) Add the first early reflection and the second early reflection to generate a first summed early reflection for the listener's left ear; 14) Add the first early reflection and the second early reflection to generate a second summed early reflection for the listener's right ear; 15) Add the first summed direct sound portion of the binaural sound of the left ear to the first summed early reflection of the left ear to generate a first summed direct sound portion of the binaural sound with early reflection for the left ear. 16) Add the second summed direct sound portion of the binaural sound of the right ear to the second summed early reflection of the right ear to generate a second summed direct sound portion of the binaural sound with early reflection for the right ear. 17) Adding the direct sound portion of the summed binaural sounds with early reflections to the reverberant portion of the binaural sounds to generate the first audio signal for the left ear; and 18) The direct sound portion of the summed binaural sound with early reflections is added to the reverberation portion of the binaural sound to generate the second audio signal for the right ear.

8. The method according to claim 7, wherein, The real-time communication session refers to a real-time virtual reality communication session.

9. The method according to claim 7, wherein, The set of room impulse responses includes a pair of room impulse responses.

10. The method according to claim 9, wherein, Each room impulse response in the set of room impulse responses is derived from measurements of a reference conference room or is artificially synthesized.

11. The method according to claim 7, wherein, The step of removing reverberation from the first audio signal stream to generate the first dry signal stream employs a dereverberation model.

12. The method according to claim 11, wherein, The step of removing reverberation from the second audio signal stream to generate the second dry signal stream employs the aforementioned dereverberation model.

13. A computer-implemented method for generating spatial audio with uniform reverberation for a user wearing headphones in a real-time communication session, wherein, The method is executed by a real-time communication software application in an electronic communication device, and the method includes: 1) Obtain the audio signal stream of the user's speech from the audio input interface of the electronic communication device; 2) Remove reverberation from the audio signal stream to generate a dry signal stream; 3) Convolve the dry signal stream with a set of room impulse responses to generate the reverberation component of the binaural sound; 4) The reverberation portion of the binaural sound is filtered by a high-pass filter to form a high-pass filtered signal; 5) The high-pass filter signal is filtered by the mouth-ear transfer function to form the mouth-ear transfer function filtered signal; 6) The mouth-to-ear transfer function filtered signal is filtered using a first set of headphone compensation filters to generate a first audio signal for the user's left ear; and 7) The mouth-to-ear transfer function filter signal is filtered by the second set of headphone compensation filters to generate a second audio signal for the user's right ear.

14. The method according to claim 13, wherein, The real-time communication session refers to a real-time virtual reality communication session.

15. The method according to claim 13, wherein, The set of room impulse responses includes a pair of room impulse responses.

16. The method according to claim 15, wherein, Each room impulse response in the set of room impulse responses was measured by a reference conference room or synthesized artificially.

17. The method according to claim 13, wherein, The step of removing reverberation from the audio signal stream to generate a dry signal stream employs a dereverberation model.

Images