Synchronization of audio stream exchanges during an audio conference

Abstract

Synchronization of audio stream exchanges during an audio conference The invention relates to a method for synchronizing audio data during an audio conference between a plurality of participants via client devices, the method being implemented by a first client device of the conference sharing the same common space as a second client device and in which the following steps are executed: - exchange (E23, E29) of timestamp information between the client device and a server device connected with the client devices of the common space; - calculation (E24) of a clock drift of the client device with respect to the server device from the information exchanged;- during the capture of audio data via a microphone on the client device, processing (E25) of the captured audio packets by resampling the packets according to the calculated clock drift value and transmission (E26) of the resampled packets to the server device with packet timestamp information. The invention also relates to a method of processing audio data by a server device. The invention relates to a client device and a server device implementing the respective synchronization and processing methods. Abstract figure: Figure 2;

Description

Title of the invention: Synchronization of audio stream exchanges during an audio conference Technical field

[0001] The present invention relates to the field of audio or video conferencing and more particularly to improving audio quality when several people participating in the conference share the same environment. Prior art

[0002] Conferencing systems (audio conferencing and video conferencing) allow for virtual meetings between remote people. Due to the Covid crisis and the associated lockdowns, this use has become widespread and today regular teleworking has made this tool inevitable.

[0003] These systems can be presented in stand-alone form to equip so-called dedicated meeting rooms, or in software form on a PC or smartphone via applications such as Teams®, Zoom® or Skype® to name but a few.

[0004] These applications are installed on communication devices that include software modules called conference clients that control this type of application. The devices can connect to each other directly (in "peer-to-peer" mode), or use a conference bridge, which allows the signals from the participants' microphones to be aggregated and an individualized audio mix to be sent back to them (the sum of all the microphones except their own).

[0005] In the specific case of a so-called "hybrid" meeting, that is to say a meeting involving at least one meeting room and one or more remote participants (or even one or more other remote meeting rooms), some of the participants (the co-participants) are co-located in the same space (the meeting room).

[0006] Each of the co-participants can bring their PC and / or smartphone into the room to connect individually to the virtual meeting in order to follow the visual elements of the meeting (video, presentation, etc.). However, it is necessary that, with the exception of one of the co-participants, whom we will call the main co-participant, all co-participants deactivate their microphone and speaker on their respective device.

[0007] Indeed: - if one of the co-participants (which we will call secondary co-participant) leaves his microphone activated, the remote participants will receive the audio signals from 2 microphones (primary and secondary co-participant) at different times because the systems Conference systems do not guarantee synchronization of sound sources. This results in an intolerable echo sensation on the side of remote participants. It should be noted that if the microphone signals of the conference clients were synchronized with an accuracy of less than approximately 50ms, the echo sensation would become tolerable but still audible, and almost inaudible below 20ms. However, the main co-participant's loudspeaker will emit the sound picked up by the microphone of the secondary co-participant, thus creating an intolerable echo in the meeting room (a participant in a meeting room will be heard live, then through the main co-participant's loudspeaker with a delay of around a few hundred ms).

[0008] - if one of the secondary co-participants leaves his loudspeaker activated, all the co Participants will hear the audio signals of the remote participants through both speakers (primary and secondary co-participant) at different times. This also results in an intolerable echo sensation on the co-participants' side.

[0009] Since current conference systems do not guarantee the synchronization of microphone or loudspeaker sound signals, the sound in the meeting room is therefore generally only picked up by the microphone of the main co-participant, and the sound from the remote participants is only reproduced by the loudspeaker of the main co-participant. As a result, the co-participants who are far from the main co-participant:

[0010] - will be difficult to understand by remote participants, due to attenuation acoustics linked to their distance from the microphone, but also due to the reverberation of the room which reduces the intelligibility indices

[0011] - will have difficulty understanding remote participants, due to attenuation acoustics linked to their distance from the loudspeaker, but also due to the reverberation of the room which reduces the intelligibility indices.

[0012] US patent document 11425258 attempts to solve the echo problem by dedicating one of the devices in the same space, or the conference bridge, to advanced synchronization processing of the data it receives from other devices in the common space before transmitting a mixed stream to other remote participants.

[0013] This type of system focuses the processing load on a single device, which can be problematic for the equipment in question which may not be designed to absorb such a processing load, especially if the number of participants is large.

[0014] On the other hand, the synchronization implemented in this document is performed on data stored in the memory of a common device, it is therefore based on a correlation analysis of the received and stored audio signals in order to obtain the time differences between the signals, independently of the latencies due to the acoustic propagation distances. This type of analysis is expensive in terms of processing load and lacks reliability since it relies on audio data that may be noisy or unstable.

[0015] There is therefore a need to improve state-of-the-art methods in this type of situation. Statement of the invention

[0016] The invention improves the state of the art.

[0017] To this end, the invention relates to a method for synchronizing audio data during an audio conference between a plurality of participants via client devices, the method being implemented by a first client device of the conference sharing the same common space as a second client device and in which the following steps are executed: - exchange of timestamp information between the client device and a server device connected with the client devices in the common area; - calculation of a clock drift of the client device compared to the server device from the information exchanged; - when capturing audio data via a microphone of the client device, processing the captured audio packets by resampling the packets according to the calculated clock drift value and transmitting the resampled packets to the server device with packet timestamp information.

[0018] The invention thus makes it possible to overcome the problems of intelligibility of the interventions of participants in an audio conference which may appear when several participants in the conference are in the same room and only use a single microphone. The synchronization method as described makes it possible to use the microphones of the client devices, as close as possible to each of the participants. This method distributes the processing load on the client equipment rather than concentrating it on a single equipment and provides reliable synchronization of the audio signals, since it is carried out on time stamp information which is reliable data.

[0019] In a particular embodiment, the method further comprises the following preliminary steps: - activation of a conference application; - configured the audio settings of the conferencing application to disconnect an internal sound card driving at least the microphone of the client device.

[0020] These preliminary steps ensure that the audio data captured by the microphone of the client device is not transmitted directly to the remote participants of the conference. This audio data is synchronized by the method described above before being sent to the server device which, after processing, will send it to the remote participants directly or via a conference bridge.

[0021] In an additional embodiment, when receiving audio data of the server device, the method implements a step of converting the received data by resampling according to the calculated clock drift value and a step of transmitting the resampled packets to at least one loudspeaker of the client device.

[0022] Thus, the audio data is reproduced using the speakers of the client devices, in a synchronized manner and as close as possible to the participants using these devices, so they are more understandable.

[0023] In a particular embodiment, the exchange of timestamp information is carried out by steps of regularly sending a data packet and collecting data on the time of sending and receiving the data packet, over a time window.

[0024] These exchanges of time stamp information make it possible to measure the network propagation time between a client device and a server device and thus obtain the information allowing the transformation of the time from one device to another, these exchanges being carried out regularly, they make it possible to have good precision of the synchronization data over time to adapt the synchronization of the audio data to a possible change in the network conditions.

[0025] In a particular embodiment, the calculation of the clock drift between the client device and the server device is carried out from the time data collected over a sliding time window and according to a linear regression function applied to the timestamp data collected during the returns of the data packet from the server device to the client device.

[0026] This calculation method makes it possible to obtain the clock drift between the devices precisely, and thus makes it possible to adjust the clock offset values ​​less frequently.

[0027] In a particular embodiment, a linear regression function is also applied to the timestamp data collected during the trips of the data packet from the client device to the server device.

[0028] This makes it possible to verify that the calculation on the return data is consistent with the calculation carried out for the outbound data. Otherwise, filtering of the problematic measurement or correction of the drift value may be applied.

[0029] In an alternative embodiment, the linear regression function is randomly applied to the timestamp data collected during the returns or outbound journeys of the data packet.

[0030] This variant makes it possible to overcome the differences which could exist between the outgoing data and the return data, for example network conditions which would not be identical or sending and receiving conditions which would be different.

[0031] In one embodiment, the timestamp information of the resampled audio packets is adapted to the clock of the server device before being sent to the server device.

[0032] This timestamp data then allows the server device to create a mixed stream with the different resampled data received from the different client devices, without having to perform additional synchronization.

[0033] The invention also relates to a method for processing audio data during an audio conference between a plurality of participants via client devices, the method being implemented by a server device connected with at least one client device sharing the same common space and in which the following steps are executed: - exchange of timestamp information between a client device in the common area and the server device; - when receiving audio data packets with associated timestamp information, from the client devices in the common space:

[0034] - storing the packets in a temporary memory; - time alignment of stored packets, based on timestamp information; - mixing the aligned packets to obtain a mixed audio stream; - sending the mixed audio stream to remote participants of the audio conference.

[0035] The method implemented by a server device receiving data synchronized with its own clock makes it possible to mix the different data to create a stream to be transmitted to the remote participants directly or via a conference bridge. This stream, once played by the different speakers of the remote participants, does not cause the echo phenomena which can appear when the received data is not synchronized.

[0036] According to one embodiment, the method further comprises the following preliminary steps: - activation of a conference application; - configuring the audio settings of the conference application so that the application's input audio data comes from a processing module producing the mixed audio stream.

[0037] Thus, the audio signals transmitted by the conference application to the remote participants or to a conference bridge do not come directly from the audio data captured by the microphones of the devices in the common space but come from a processing module implementing the method for processing audio data received from the client devices in the common space. The captured audio signals are thus routed to this processing module in order to obtain a synchronized audio stream to transmit.

[0038] The invention relates to a client device sharing the same common space as a second client device during an audio conference between a plurality of participants, the client device comprising at least one microphone controlled by an internal sound card, and at least one client audio module controlled by a microprocessor and capable of implementing a method for synchronizing audio data, the method comprising the following steps: - exchange of timestamp information between the client device and a server device connected with the client devices in the common area; - calculation of a clock drift of the client device compared to the server device from the information exchanged; - when capturing audio data via the at least one microphone of the client device, processing the captured audio packets by resampling the packets according to the calculated clock drift value and transmitting the resampled packets to the server device with packet timestamp information.

[0039] In one embodiment, the device further comprises a conference client module capable of activating a conference application, the internal sound card being configured as being disconnected from the conference client module for the conference application.

[0040] The client device has the same advantages as the synchronization method it implements.

[0041] The invention also relates to a server device connected with at least one client device sharing the same common space during an audio conference between a plurality of participants, the server device comprising at least one audio server module controlled by a microprocessor and capable of implementing a method for processing audio data, the method comprising the following steps: - exchange of timestamp information between a client device in the common area and the server device; - when receiving audio data packets with associated timestamp information from client devices in the common area: - storing packets in a temporary memory; - time alignment of stored packets, based on timestamp information; - mixing the aligned packets to obtain a mixed audio stream; - sending the mixed audio stream to remote participants of the audio conference.

[0042] According to one embodiment, the server device comprises a client module of conference capable of activating a conference application and at least one virtual sound card configured as an input for the audio data of the conference client module of the audio conference, the virtual sound card receiving the mixed audio stream from the server audio module.

[0043] The server device has the same advantages as the processing method it implements.

[0044] The invention relates to a terminal comprising a client device and / or a server device as described.

[0045] The invention relates to a computer program comprising instructions for implementing the synchronization or data processing methods as described above, when they are executed by a processor.

[0046] Finally, the invention relates to a storage medium, readable by a processor, storing a computer program comprising instructions for executing the synchronization method and / or the data processing method described above. Brief description of the drawings

[0047] Other characteristics and advantages of the invention will appear more clearly on reading the following description of particular embodiments, given as simple illustrative and non-limiting examples, and the appended drawings, among which:

[0048] [Fig. 1a] illustrates a conference system comprising a conference bridge and a plurality of client devices, at least two of which are grouped in a common space and implementing a synchronization method according to one embodiment of the invention, one of the client devices also playing the role of a server device implementing a data processing method according to one embodiment of the invention;

[0049] [Fig. 1b] illustrates a conference system comprising a conference bridge and a plurality of client devices, at least two of which are grouped in a common space and implementing a synchronization method according to one embodiment of the invention, a server device also being grouped in the common space and implementing a data processing method according to one embodiment of the invention;

[0050] [Fig.2] illustrates in flowchart form the main steps of a synchronization method implemented in a client device and a data processing method implemented in a server device, according to one embodiment of the invention;

[0051] [Fig.3] illustrates the step of exchanging timestamp information between a client device and a server device implemented in the synchronization method and the data processing method according to the invention;

[0052] [Fig.4] illustrates a representation of the data recorded in a temporary memory of the server device during a data processing method according to the invention; and

[0053] [Fig.5] illustrates examples of structural embodiments of a client device or a server device according to an embodiment of the invention. Description of the embodiments

[0054] [Fig. 1a] describes a conference system according to one embodiment of the invention. This conference system comprises a conference bridge (PC) and a plurality of client devices. The remote client devices DD1 and DD2 are remote devices represented as being alone in their respective spaces E1, E2. The client devices DC1 and DC2 are represented as belonging to the same common space (SR) such as a meeting room. These devices DC1 and DC2 belong to two participants represented as U1 and U2 who are therefore present in this common space.

[0055] Obviously, such a conference can be provided between participants of several common spaces each grouping several participants. In this case, in each common space, the methods according to the invention described below are implemented.

[0056] Each of the client devices (DD1, DD2, DC1 and DC2) comprises at least one microphone (Mi) and at least one loudspeaker (HP) controlled by an internal sound card (ISC for "Internal sound card" in English) which is responsible for capturing the sounds of the common space by the microphone and for reproducing the sounds of the remote parties on the loudspeaker. The devices can also comprise a conference client software module (CC or PCC) controlling a conference application such as, for example, a web videoconferencing application.

[0057] The remote client devices DD1 and DD2 operate in a conventional manner, that is to say that the internal sound card controlling the microphone and the loudspeaker is configured as an input and output audio parameter in the conference application controlled by the conference client (CC) software module. This conference client retrieves the sounds captured from the ISC sound card and transmits them to the PC conference bridge. Conversely, the conference client retrieves the audio data from the conference bridge and transmits them to the sound card so that they are played back on the loudspeaker. In an alternative embodiment, the conference system does not include a conference bridge, the conference client devices then communicating directly with each other.

[0058] The data exchange arrows shown in Figures 1a and 1b represent, for those illustrated in solid lines, the exchange of audio signals through a communication network, in dotted lines, the exchange of data other than audio through a communication network, for example video and those illustrated in dotted lines, the transfer of acoustic signals through the common space.

[0059] In the common space SR illustrated in [Fig. 1a], two client devices are represented. Obviously, this common space can include more than two client devices. In [Fig. 1a], a client device DC1 is considered a main client device because it is this device that hosts the conference client responsible for audio communication with the conference bridge or with the remote devices for the remote participants of the common space. It is also this device that hosts the device, which will subsequently be called the server device, intended to exchange audio data with the conference client.

[0060] This device has a particular function compared to another client device in this common space. Indeed, this server device comprises an audio module also called an audio server module (AS) which implements a processing method according to an embodiment of the invention and as described later in [Fig.2].

[0061] For this server device, a virtual sound card (VSC) is configured as input and output audio parameters of the conference application driven by the main conference client module (PCC). This virtual sound card is a software module that allows audio signals to be routed from one application to another within the same device. In this case, the virtual sound card receives the processed audio data from the server audio module and transmits it to the main conference client module (PCC) for sending to the conference bridge (PC).

[0062] According to another embodiment, the server device can make the mixed audio stream available to the conference client via the use of a shared memory space, such as, for example, a Linux 'pipe', or even IP communication. It should be noted that such an embodiment requires that the conference client offers a specific programming interface, adapted to the server device.

[0063] The audio server module (AS) receives audio data from its own audio client module (AC) which retrieves the data captured by the microphone via the internal sound card (ISC).

[0064] This server audio module of the server device DC1 also retrieves the synchronized audio data from the different client audio modules of the client devices of the common space, in the illustration of [Fig.la], the client device DC2.

[0065] The audio data was obtained according to a synchronization method described with reference to [Fig.2] and implemented by the various client audio modules of the client devices in the common space. This synchronized audio data is in the form of data packets resampled at the clock frequency of the device server, with packet timestamp information, the packet timestamp information being adapted to the server device's clock.

[0066] Upon receiving these audio data packets, the server audio module of the server device stores these packets in a temporary memory with the associated timestamp information. It implements a step of time alignment of the received packets according to the associated timestamp information to obtain a mixed audio stream that it sends to the main conference client module via the virtual sound card. The data thus processed is then transmitted to the PC conference bridge which transmits the audio data to the other remote client devices DD1 and DD2.

[0067] In this embodiment, this server device is a device such as a communication terminal or a computer of a participant. In this case, the server device is also a client device.

[0068] In one embodiment, the client device DC2 illustrated in [Fig. 1a] comprises, in addition to its internal sound card (ISC), and a conference client module (CC) as previously described, a virtual sound card (VSC) which is configured as input and / or output audio parameters of the conference application driven by the conference client module (CC).

[0069] Indeed, when configuring the input and output audio parameters of the client device's conference application, several choices are possible as indicated by the illustrated connector, the choice here being between the internal sound card (ISC) and the virtual sound card (VSC). Unlike the server device, this virtual sound card does not receive audio data from an audio module so that the conference client module does not transmit audio data to the conference bridge. Similarly, if the conference client module receives audio data from the conference bridge, it transmits it to the virtual sound card which is configured as the output audio parameter, but since this virtual sound card is not connected to the audio client (AC) module, the data will not be played by the speaker.

[0070] This embodiment allows for compatibility with any type of commercially available conference application.

[0071] In another embodiment, the conferencing application may provide a configuration mode in which the audio outputs and inputs are disconnected.

[0072] In another embodiment, the conference application may provide a configuration mode in which the audio output is connected to the internal sound card to transmit directly to the speaker, the audio input is connected to the virtual sound card to be powered by the server audio module, itself powered by the client devices.

[0073] In another embodiment, the conferencing application may provide a configuration mode in which the audio input is connected to the internal sound card for directly acquire the signal from the internal microphone, and the audio output is connected to the virtual sound card to feed the server audio, which will itself feed the client devices.

[0074] In another embodiment, the client device DC2 may not include a conference client module or a virtual sound card. In this case, the user of this client device is not connected to a conference application but participates in the conference via the microphone of his device and a client audio (AC) module.

[0075] The DC2 client device includes an audio client (AC) software module which retrieves the data captured by the microphone via the internal sound card (ISC) and in return sends the data received from the audio server module to the speaker, via the internal sound card.

[0076] This client audio module (AC) implements a synchronization method as described with reference to [Fig.2]. The resulting data is then transmitted to the server audio module (AS) of the server device (DC1) via a communication network, for example a local network. This local network may be, for example, a wireless network of the Wi-Fi or Bluetooth type or a wired network of the Ethernet type.

[0077] This audio server module (AS) as described previously, implements a data processing method as described with reference to [Fig.2].

[0078] When the server device receives an audio stream from the PC conference bridge via its main conference client (PCC) module, it transmits this stream to the client devices in its common area via its virtual sound card and its server audio module. The client audio modules of the client devices implement the synchronization process before sending the resulting audio data to their respective internal sound card for this data to be played on their respective loudspeaker. Thus, the data is played in a synchronized manner on the client devices in the common area, as close as possible to the user of the device.

[0079] Thus, the microphones and loudspeakers of the client devices in the common area are used to capture and reproduce the sound, as close as possible to the user of the device. The captured sound is thus more intelligible than if it had been captured by a device further away from the user. However, this captured sound is not transmitted directly by the client device but is transmitted, after time-stamping adapted to the server clock by the client device, to the server device which is responsible for temporally aligning and mixing the audio data that it receives and which are thus synchronized.

[0080] The processing load of the server device is reduced since it does not have to perform specific synchronization processing, such as a correlation analysis between audio signals for example. Similarly, in the embodiment described here, the audio server device also does not have to perform the resampling processing which guarantees fine synchronization over time, this processing being taken into supported by the client device. Remote participants, users of devices DD1 and DD2, will therefore hear participants U1 and U2 more clearly while avoiding an echo phenomenon from disturbing them.

[0081] It should be noted that all the audio streams mentioned can be mono, but also stereo or multichannel.

[0082] [Fig. 1b] represents an alternative embodiment where a device (DD) independent of the client devices of the participants is dedicated to implementing the data processing method as described with reference to [Fig.2]. This data processing method is implemented by a server audio module (AS) of the dedicated processing device (DD). This server audio module, as described previously, retrieves the synchronized audio data from the different client audio modules of the client devices in the common area. In one embodiment, this device may be, for example, a terminal, a computer or an octopus, present in the meeting room.

[0083] This dedicated device may also include a sound card (ISC) controlling at least one microphone and one loudspeaker and a client audio module (AC) which recovers the data captured by the microphone via its internal sound card (ISC) and in return, sends the received data to the loudspeaker via its internal sound card.

[0084] This dedicated device acts as a server device, in the sense that it receives the synchronized audio data from the other client devices in the common space, in the form of data packets resampled at the clock frequency of this server device, with time stamp information of the data packets, the time stamp information being adapted to the clock of the server device.

[0085] When receiving these audio data packets, the audio server module (AS) of this dedicated device stores these packets in a temporary memory with the associated timestamp information. It implements a step of time alignment of the received packets according to the associated timestamp information to obtain a mixed audio stream that it sends to the main conference client module which is located on a main client device, represented here in DC1. This sending is carried out via a communication bus such as for example an external sound card (ESC) of this client device. This is carried out for example by a wired connection between the two devices, for example a connection via a USB type socket.

[0086] For this primary client device, the configuration of input and output audio data is done by choosing this external sound card instead of choosing its internal sound card (ISC).

[0087] In an exemplary embodiment, the server device (DD) may serve as a wireless network access point (e.g., WiFi) for client devices in the common area to communicate with each other.

[0088] [Fig. 2] illustrates the main steps of a synchronization method implemented in a client device (DC) as represented in DC2 in [Fig. 1a] and of a data processing method implemented in a server device (DS) as represented in DC1 in [Fig. 1a], according to an embodiment of the invention.

[0089] Optionally, the client devices can activate, in a prior step E21, a conference application, to participate in a conference between several participants, such as a video conference or an audio conference.

[0090] The server device (DS) implements a preliminary step of activating a conference application in E27.

[0091] For this, in a conventional manner, the software application installed on the device is opened. A configuration of the audio inputs and outputs to be applied for this conference is then carried out in E22, in the case where the conference application has been activated for the client device and in E28 for the server device. This configuration is carried out on the interface of the conference application, for example, by the user of the audio conference.

[0092] For the client device, configuring the audio parameters of the conference application involves disconnecting an internal sound card controlling at least the microphone of the client device. In one embodiment, this disconnection is performed by configuring a virtual sound card, in place of the internal sound card, this virtual sound card not receiving any audio data, for this client device.

[0093] In another embodiment where the conference application proposes as input / output configuration a “none” mode, i.e. without input and output audio data, then this mode can be selected.

[0094] It is also possible to configure the audio input and output of the application differently and to apply, for example, the configuration described above only to the input data.

[0095] For the server device, the configuration of the audio parameters of the conference application is performed so that the input audio data of the application comes from a processing module producing the mixed audio stream.

[0096] In one embodiment, this configuration can be done by connecting to the conference module controlling the conference application, a virtual sound card in place of the internal sound card of the server device. This virtual sound card receives, for this server device, the audio data processed in E30 detailed below, before sending to the conference bridge or to the remote devices directly, in E34 via the conference client module.

[0097] Similarly, it is possible to configure the audio input and output of the application differently and to apply, for example, the configuration described above only to the input data. In an alternative embodiment, the server device may make the mixed audio stream available to the conference client via the use of a shared memory space. In this case, the conference client offers a specific programming interface, adapted to the server device so that such configuration of input and output data is possible.

[0098] Steps E23 and E29 represent the exchanges of timestamp information between a client device and a server device of the common space.

[0099] This exchange of information allows the client device to know both the clock offset and a clock drift that may exist between the client device and the server device.

[0100] [Fig.3] illustrates this step of exchanging timestamp information between the client devices and the server device of a common space. These exchanges can be carried out for example according to the NTP protocol (For “Network Time Protocol” in English) as described for example in the technical specification RFC-5905 “Network Time Protocol Version 4” of June 2010.

[0101] Other network synchronization protocols can of course be used here, such as, for example, the PTP protocol (for “Precision Time Protocol” in English).

[0102] Initially at time Tcs, the client device sends a message M, in the form of a data packet, which can be called synchronization data, to query the server device on the current time, it sends this message in which it fills in a first field with the current time Tcs indicated by its local clock;

[0103] At Tmr, when the server device receives the message M, it immediately completes a second field of the message M with the current time TMR indicated by its local clock;

[0104] At Tms, when the server device sends its response message, it completes a third field with the current time TMS indicated by its local clock;

[0105] At TCR, when the client device receives the response message, it immediately notes the TCR time of reception indicated by its local clock.

[0106] The client device can then calculate in step E24 illustrated in [Fig.2], the round trip delays “delta” of these two messages which makes it possible to evaluate the network latency as well as the “theta” difference between its local clock and that of the server device.

[0107] Here we estimate that the forward delay and the return delay (delta) are the same, that is to say that the network conditions remain unchanged during this message exchange.

[0108] The calculation performed also takes into account a possible “alpha” drift of the clock of the client device compared to that of the server device.

[0109] Indeed, two independently operating devices have their own clock. A clock is defined as a monotonic function equal to a time which increases at the rate determined by the clock frequency. It can originate at the instant the device is started, or at an arbitrarily defined instant such as, for example, January 1, 1970 for NTP.

[0110] The clocks of two devices are necessarily different and three parameters are defined:

[0111] - clock offset (theta): time difference at the origin between two clocks;

[0112] - clock drift (alpha): frequency ratio between two clocks;

[0113] - clock deviation: variation of the drift over time, or derivative second of the clock relative to time.

[0114] A classical modeling of a clock neglects the clock deviation, mainly caused by the construction of the component called quartz, but also by temperature changes. Thus, in a server / client network context, as here between a server device and a client device, the clock of the client device Te is expressed as a function of the clock of the server device TM according to an equation such as Tc = f(TM) = alpha *TM + theta where alpha represents the clock drift of the client with respect to that of the server device, and theta represents the offset of the client clock.

[0115] Thus, in the present case, at step E24 illustrated in [Fig.2], the following equations are used to estimate the delta delay:

[0116] Tcs + delta = alpha* TMR + theta

[0117] TCR - delta = alpha* TMS + theta

[0118] Which gives the value of delta according to the equation:

[0119] delta = ((alpha* TMR - Tcs) + (TCR - alpha* TMS)) / 2

[0120] This value representing the offset between the clocks of the server device and that of the client device is a time which can be expressed in seconds or milliseconds.

[0121] This exchange of timestamp information (E23, E29) is carried out by steps of regularly sending a data packet (message M) and collecting timestamp data for sending and receiving this packet. In one embodiment, a data packet is sent every second. Another sending frequency can of course be provided.

[0122] The information is collected over a sliding time window which may be for example one hour. To be able to obtain the value of the “alpha” clock drift, a linear regression (for example a least squares function) is carried out taking into account the TMS, TCR timestamp values ​​collected in this time window, i.e. the timestamp values ​​collected during the returns of the data packet from the server device to the client device.

[0123] We thus obtain a value of alpha representing the clock drift, that is to say the frequency ratio between the clock of the server device and the clock of the client device.

[0124] When starting a client audio application implementing the syn method timing, the time window is empty or has not reached the predefined size. In this case, the drift value is set to 1 and a minimum delay is respected before calculating a new drift value with the data collected in this minimum time interval. This interval can be for example 7s.

[0125] Little by little, the size of the window of the history of the collected data reaches a predetermined value, for example 30 min or 1h. Only the data on this sliding window is then kept, the previous data is erased from the memory of the device. Indeed, a modification of the network configuration could for example lead to a modification of the delay or drift values.

[0126] In a particular embodiment, a linear regression function is also applied to the data going from the data packet from the client device to the server device. Thus, if a large difference in the alpha value is observed with this new calculation, it will be possible to either filter the problematic measurement or apply a correction to the alpha value. In the latter case, an average between the two values ​​could, for example, be taken.

[0127] In an alternative embodiment, the linear regression function is randomly applied either to the return or outbound data of the data packet. This alternative makes it possible to overcome differences that could exist between the outbound data and the return data, for example network conditions that are not identical or sending and receiving conditions that are different.

[0128] The obtained drift value is a dimensionless value equal to the ratio of the clock frequencies of the server device and the client device.

[0129] This value is then used to adapt, at E25, the data captured by the microphone via the ISC sound card of the client device before transferring them at E26 to the server device or to adapt the data received from the server device before broadcasting to the speaker of the client device via its ISC sound card.

[0130] For this, in E25, the data captured by the microphone of the client device are processed by resampling according to the value of the calculated clock drift.

[0131] Knowing the value of the clock frequency of the FM server device, for each audio packet, a resampling module is implemented to adapt the sampling frequency (Fc) of the client device to that of the server device in the following way: Fc = FM * alpha.

[0132] The instant Tccapture of the capture of an audio data packet by the client device is converted into timestamp data adapted to the clock of the server device using the inverse function of f defined above, as follows: TMcapture=f'(Tc capture^ i.e. here TMcapture =( Tccapture - theta) / alpha.

[0133] We thus obtain data packets resampled at the frequency of the server device clock Fc with capture times adapted to the device clock server.

[0134] These resampled data packets are transmitted to the server device with a timestamp information of this packet according to the clock of the server device.

[0135] Similarly, when receiving audio data by the server device which itself received it from the conference bridge, the data packets are resampled in E25 according to the calculated clock drift value and transmitted to the speaker of the client device via its ISC sound card, so that the audio data is played.

[0136] Thus, the broadcasting of audio streams on the different loudspeakers of the client devices in the same meeting room does not generate echo phenomena since there is no longer any lag or drift in time of the sounds played. The restitution is also carried out as close as possible to the users of these client devices for a better understanding of the audio stream.

[0137] If the clock drift information were not shared, this would inevitably result, at the time when the clock offset had to be updated due to the time drift, in either a temporal overlap or silence between 2 successive audio packets from the same client when temporally aligning the received packets.

[0138] If resampling were not used to give audio packets the adequate duration in the server's clock, the inevitable result would be either too much data accumulation in the server if the server's clock frequency is lower than that of the client, or data starvation in the server if the server's clock frequency is higher than that of the client, resulting in both cases in audible audio artifacts ("clicks").

[0139] On the server device side, an audio data processing method is implemented.

[0140] In the same way as described for the client device, an exchange of timestamp information between a client device of the common space and the server device is carried out by the steps E23 and E29 illustrated in [Fig.2].

[0141] As described previously, the client device transmits in E26 data packets resampled at the clock frequency of the server device, with timestamp information, the timestamp information being adapted to the clock of the server device.

[0142] When receiving these audio data packets at E30, the server device stores these packets at E31 in a temporary FIFO (for First In First Out) type memory with the associated timestamp information.

[0143] Then in E32, a step of time alignment of the received packets is carried out according to the associated timestamp information. A representation of these data packets aligned in the FIFO type temporary memory is illustrated in [Fig.4].

[0144] This [Fig.4] shows that the data packets (illustrated by consecutive blocks AP) received from the different client devices (DCi, DC2, ..., DCk) are recorded in memory with the associated timestamp information. Thus, the data packets received from DCi were stored in a FIFO memory dedicated to DCi with the associated timestamp information Ti>Mi+n,..., Ti>Mi+3,Ti>Mi+2, Ti>Mi+1 to Ti^ .

[0145] The same applies to data packets received from client devices DC2 to DC K with the corresponding timestamp information.

[0146] A time alignment according to step E32 of [Fig.2] makes it possible to shift the data blocks so that at a time T, the time stamp information of the blocks recorded in the different FIFOs correspond.

[0147] This shift is illustrated in [Fig.4]. The time adjustment is performed by aligning all the data in the FIFOs so that a vertical axis corresponds to the same time T.

[0148] In step E33 of [Fig.2], the server device mixes the different data packets thus aligned at a time T to be able to constitute a mixed audio stream (Stream-T) to be transmitted to the conference bridge at E34. The mixing step can be a simple summation of all the aligned signals, or a summation with an arbitrary weighting or resulting from an analysis. An analysis of the volume variations between different participants makes it possible, for example, to define a compensation gain to be applied as a weighting to the low volume signals.

[0149] In the case where there are no data packets at this time T for one of the client devices, then a silence data can be inserted.

[0150] When receiving audio streams from the conference bridge (PC) at E30, the server device sends the audio packets to the client devices of the common space at E26 so that they can process them as explained previously in step E25.

[0151] [Fig.5] illustrates a structural embodiment of a client device or a server device (100) according to an embodiment of the invention.

[0152] Such devices can be included in a communication terminal such as a smartphone, electronic tablet or a personal computer, or even a dedicated device such as an audio conference octopus.

[0153] The client device comprises a processing circuit typically including: - a MEM memory for storing instruction data of a computer program within the meaning of the invention as well as various applications, an audio or video conferencing application being able, for example, to be part of these applications; - an INT interface capable of receiving user instructions such as, for example, audio parameter configuration information and of displaying various data, such as, for example, video images from a video-type conference. conference ; - an ISC sound card capable of recovering audio data captured by a Mi microphone and sending audio data to an HP speaker for playback;

[0154] - an AC client audio software module capable of recovering the data captured by the mi microphone via the internal ISC sound card. This client audio module can also send the received data to the speaker via the internal sound card. This AC client audio module implements a synchronization method as described with reference to [Fig.2]

[0155] - a COM communication interface capable of sending or receiving data, including audio data, to or from another device, such as a server device via a network R;

[0156] - a processor PROC for executing the computer program instructions that stores the MEM memory, in particular the instructions for implementing the synchronization method as described with reference to [Fig.2]; the processor being capable of controlling the different modules described.

[0157] The client device may optionally include a CC conference client software module capable of controlling a conference application such as, for example, a web videoconferencing application.

[0158] In this case, it may include a virtual sound card VSC configured as the input and / or output audio module of the conference application driven by the CC conference client module. This virtual sound card is a software module that allows audio signals to be routed from one application to another within the device. In the case of a client device, the virtual sound card does not receive any audio data. It can be used to disconnect the ISC sound card from the conference client module.

[0159] Similarly, the server device, in one embodiment, comprises a processing circuit typically including: - a MEM memory for storing instruction data of a computer program within the meaning of the invention as well as various applications, an audio or video conferencing application being able, for example, to be part of these applications; - an INT interface capable of receiving user instructions such as, for example, audio parameter configuration information and of displaying various data, for example, video images from a videoconference type conference; - an ISC sound card capable of recovering audio data captured by a Mi microphone and sending it to an AC client audio module and conversely receiving audio data from the client audio module and sending it to an HP speaker for playback.

[0160] - a CC conference client software module, which is also called in the case of the server device, main conference client, capable of driving a conference application such as, for example, a videoconferencing web application and sending audio streams to other remote devices or to a conference bridge via the COM communication interface.

[0161] - an AC client audio software module capable of recovering the data captured by the mi microphone via the internal ISC sound card and send the received data to the speaker via the internal sound card.

[0162] - an AS server audio module which implements a processing method according to a embodiment of the invention and as described with reference to [Fig.2].

[0163] The server audio module AS receives the audio data from its own client audio module AC and also retrieves the synchronized audio data from the different client audio modules of the client devices in a common space, via the communication interface COM to implement the processing method.

[0164] - a VSC virtual sound card configured as the input and output audio module of the conferencing application driven by the CC Conferencing Client Module. This virtual sound card is a software module that allows audio signals to be routed from one application to another within the device. In the case of a server device, the virtual sound card receives the processed audio data from the server audio module and transmits it to the CC Conferencing Client Module; the virtual sound card can also send the audio data from the CC Conferencing Client Module to the server audio module.

[0165] - a COM communication interface capable of sending or receiving data, including audio data, to or from other devices, such as client devices or a conference bridge via a network R; - a processor PROC for executing the computer program instructions stored in the memory MEM, in particular the instructions for implementing the data processing method as described with reference to [Fig.2]; the processor being capable of controlling the various modules described above.

[0166] In another embodiment, the server device may not include a CC conference client software module, nor a VSC virtual card. This server device may, however, have a USB type connection port for communicating audio data to another client device. This type of server device is, for example, a dedicated device for audio conferences, such as, for example, an audio conference octopus.

[0167] Of course, this [Fig.5] illustrates an example of a structural embodiment of a device (client or server) within the meaning of the invention. Figures 1 to 4 commented above describe in detail functional embodiments of these devices.

Claims

Claims

1. A method for synchronizing audio data during an audio conference between a plurality of participants via client devices, the method being implemented by a first client device of the conference sharing the same common space as a second client device and in which the following steps are performed: - exchange of timestamp information (E23, E29) between the client device and a server device connected with the client devices of the common space; - calculation (E24) of a clock drift of the client device with respect to the server device from the exchanged information; - when capturing audio data via a microphone of the client device, processing (E25) the captured audio packets by resampling the packets according to the value of the calculated clock drift and transmitting (E26) the resampled packets to the server device with packet timestamp information.

2. Method according to claim 1, further comprising the following preliminary steps: - activation of a conference application (E21); - configuration (E22) of the audio parameters of the conference application to disconnect an internal sound card controlling at least the microphone of the client device.

3. Method according to one of claims 1 or 2, wherein when receiving audio data from the server device, the method implements a step of converting the received data by resampling according to the calculated clock drift value and a step of transmitting the resampled packets to at least one speaker of the client device.

4. Method according to one of claims 1 or 2, in which the exchange of timestamp information is carried out by steps of regularly sending a data packet and collecting data on the time of sending and receiving the data packet, over a time window.

5. The method of claim 4, wherein the calculation of the clock drift between the client device and the server device is performed from the time data collected over a sliding time window and according to a linear regression function applied to the timestamp data collected during the returns of the data packet. from the server device to the client device.

6. The method of claim 5, wherein a linear regression function is also applied to the timestamp data collected during the data packet's trips from the client device to the server device.

7. The method of claim 5, wherein the linear regression function is randomly applied to either the timestamp data collected on the returns or the outbound journeys of the data packet.

8. A method according to any preceding claim, wherein the timestamp information of the resampled audio packets is adapted to the clock of the server device before being sent to the server device.

9. Method for processing audio data during an audio conference between a plurality of participants via client devices, the method being implemented by a server device connected with at least one client device sharing the same common space and in which the following steps are executed: - exchange (E23, E29) of timestamp information between a client device of the common space and the server device; - upon reception (E30) of audio data packets with associated timestamp information, from the client devices of the common space: - storage (E31) of the packets in a temporary memory; - temporal alignment (E32) of the stored packets, according to the timestamp information; - mixing (E33) of the aligned packets to obtain a mixed audio stream; - sending (E34) of the mixed audio stream to remote participants of the audio conference.

10. Method according to claim 9, further comprising the following preliminary steps: - activation of a conference application (E27); - configuration (E28) of the audio parameters of the conference application so that the input audio data of the application comes from a processing module producing the mixed audio stream.

11. Client device sharing the same common space as a second client device during an audio conference between a plurality of participants, the client device comprising at least one microphone (Mi) controlled by an internal sound card, and at least one client audio module (AC) controlled by a microprocessor and capable of implementing a method for synchronizing audio data, the method comprising the following steps: - exchanging timestamp information between the client device and a server device connected with the client devices in the common space; - calculating a clock drift of the client device relative to the server device from the exchanged information; - when capturing audio data via the at least one microphone of the client device, processing the captured audio packets by resampling the packets according to the value of the calculated clock drift and transmitting the resampled packets to the server device with packet timestamp information.

12. The client device of claim 11, further comprising a conference client module (CC) adapted to activate a conference application, the internal sound card being configured to be disconnected from the conference client module for the conference application.

13. Server device connected with at least one client device sharing the same common space during an audio conference between a plurality of participants, the server device comprising at least one audio server module (AS) controlled by a microprocessor and capable of implementing a method for processing audio data, the method comprising the following steps: - exchange of timestamp information between a client device of the common space and the server device; - upon receipt of audio data packets with associated timestamp information, from the client devices of the common space: - storage of the packets in a temporary memory; - time alignment of the stored packets, according to the timestamp information; - mixing of the aligned packets to obtain a mixed audio stream; - sending of the mixed audio stream to remote participants of the audio conference.

14. Server device according to claim 13, comprising a conference client module (CC) capable of activating a conference application and at least one virtual sound card configured as an input for the audio data of the conference client module of the audio conference, the card virtual sound receiving the mixed audio stream from the server audio module.

15. Communication terminal comprising a client device according to one of claims 11 to 12 and / or a server device according to one of claims 13 to 14.

16. Storage medium, readable by a processor, storing a computer program comprising instructions for executing the synchronization method according to one of claims 1 to 8 and / or the data processing method according to one of claims 9 to 10.

Images