Method for communicating voice data and related devices

Abstract

The audio data communication method described herein uses a thread pool to separate the generation of audio data by the audio source from the transmission of audio data through the network cloud by the transmitting network hardware abstraction layer. The thread pool separates the generation of audio data from the transmission of audio data, thereby preventing network latency from affecting the generation of audio data. The method separates the reception of audio data from the network cloud by pulling audio data through at least a portion of the receiving side, thereby delivering the correct amount of audio data to the audio device at the correct time. The data requirements of the audio device control the flow of audio data through the receiving side, not the reception of audio data from the network cloud.

Description

[Technical Field]

[0001] [Cross-reference to Related Application] This application claims the priority and benefit of U.S. Provisional Patent Application No. 63 / 534,061 filed on August 22, 2023, the entire content of which is incorporated herein by reference.

[0002] The present disclosure relates to network communication of data, and more particularly, to a method and related apparatus for communicating audio data from an audio source to an audio device via a network cloud so that the audio data can be used in real time at the audio device. [Background Art]

[0003] In some cases, it is desirable to transmit digital audio data from an audio source to an audio device via a network cloud, and for the audio device to use the audio data in real time. For example, the audio device may broadcast the audio data in real time in a human-recognizable form (e.g., it may generate human-recognizable sounds). However, when audio data is received from an audio source via a network cloud, certain problems arise when an audio output device broadcasts the audio data in real time. For example, unstable events on the transmitting side may cause unstable transmission of audio from the audio source to the network cloud. Another example is that if the timing between receiving audio data from the network cloud and the audio device's use of that audio data is unstable, the audio device's operation may become unstable. For example, the sound may become unstable (e.g., indistinguishable). In such systems, delays that vary depending on network quality may occur by feeding data into a jitter buffer based on received network events and dealing with packet loss or packet bursts. Network buffer overruns may result in previously canceled echoes returning. While this example describes the broadcasting of audio data from an audio device, various other uses of audio data by audio devices may also require stable communication of digital audio data from the audio source to the audio device via a network cloud. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] Therefore, there is a need for improved methods, related equipment, and configurations for communicating voice data from a voice source to a voice device via a network cloud, and for the voice device to utilize the voice data in real time. [Means for solving the problem]

[0005] The methods described herein, as well as related apparatus and configurations, can overcome these requirements and other drawbacks. Further improvements and advantages will become apparent to those skilled in the art upon examination of this disclosure.

[0006] The method for communicating audio data described herein includes, in various embodiments, the steps of: generating audio data representing an audio signal from an audio source included in an audio hardware abstraction layer; and generating an event by filling an audio buffer included in the audio hardware abstraction layer with audio data from the audio source. The method may also include the step of triggering a thread in a thread pool with the event, where the thread transfers the audio data from the audio hardware abstraction layer to the transmitting network hardware abstraction layer, and the thread pool separates the audio hardware abstraction layer from the transmitting network hardware abstraction layer. The method may also include the step of sending the audio data as a data packet from the transmitting network hardware abstraction layer to a network cloud.

[0007] The method for communicating voice data described herein includes, in various embodiments, the steps of: receiving data packets from a network cloud by a receiving network hardware abstraction layer; and pushing data packets from the receiving network hardware abstraction layer to a voice output hardware abstraction layer. In various embodiments, the method includes the steps of: pulling voice data from the voice output hardware abstraction layer by a voice device, thereby generating a synchronization event; and using the synchronization event to control the pull of voice data through the voice output hardware abstraction module, thereby making the voice data available in a state that can be pulled by a voice device as requested by the voice device.

[0008] This specification also describes an associated voice communication system that implements at least some steps of a method for communicating voice data. At least parts of the method for communicating voice data and the associated voice communication system may be implemented by a computer, including hardware and associated operable software. This specification also describes configurations in the form of a non-volatile computer-readable medium, including computer-readable instructions. When an instruction is executed, one or more computers function as at least part of the voice communication system or cause at least some of the method steps of the method for communicating voice data to be implemented. At least parts of the method for communicating voice data, the voice communication system, and the associated configurations may be implemented in various embodiments in a cloud or distributed computing environment.

[0009] This abstract provides a basic understanding of some aspects of the methods and apparatus described herein, as a prelude to the detailed description that follows. Therefore, this abstract is not intended to identify or define the main elements of the methods, apparatus, and associated configurations described herein. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram of an exemplary implementation example of the voice communication system according to the present invention. [Figure 2A] Figure 1 is a schematic diagram of a partial example of an implementation of the voice communication system. [Figure 2B] Figure 1 is a schematic diagram of a partial example of an implementation of the voice communication system. [Figure 3] Figure 1 is a Rummler-Brache diagram illustrating the exemplary operation of an exemplary implementation of a voice communication system. [Figure 4A] Figure 1 is a Rummler-Brache diagram illustrating the exemplary operation of the transmitting side in an exemplary implementation of a voice communication system. [Figure 4B] Figure 1 is a Rummler-Brache diagram illustrating the exemplary operation of the receiving side in an exemplary implementation of a voice communication system.

[0011] The figures are illustrative only, and the implementation examples shown therein have been selected for the sake of clarity. For example, the components of the various systems shown in the figures may be selected for illustrative purposes and may be grouped in various ways in the figures for the sake of clarity. In other implementation examples, the system may include various other components, and the components may be grouped in various other ways. The method steps of the various methods shown in the figures may be performed in other orders, for example, and may be divided or subdivided in various ways in other implementation examples. Information flows and processing flows in the figures attached to this specification are indicated by arrows, and it should be noted that additional information flows may occur between the various components and additional processing flows may occur in various other implementation examples. In each figure, the same reference numerals indicate the same or similar elements. Relative terms used herein, such as “generally,” “about,” and “substantially,” may refer to engineering tolerances, manufacturing tolerances, or scientific tolerances such as ±0.1%, ±1%, ±2.5%, ±5%, or other such tolerances, as recognized by a person skilled in the art who has studied this disclosure. The figures and the systems, methods, and configurations shown therein should not be constrained unless expressly stated otherwise. [Modes for carrying out the invention]

[0012] This specification describes a method for communicating voice data from a voice source to a voice device via a network cloud, as well as related voice communication systems and configurations. In various embodiments, the voice source is located on the transmitting side of the network cloud, and the voice device is located on the receiving side of the network cloud.

[0013] In various embodiments, the method separates the generation of audio data by the audio source from the transmission of the audio data via the network cloud at the transmitting end. This separation may eliminate unstable events at the transmitting end that could cause instability when transmitting audio data from the audio source to the network cloud and then to the audio device. For example, this separation may prevent network cloud transmission delays from interfering with the generation of audio data by the audio source. That is, for example, this separation prevents network cloud delays from propagating back to the audio source.

[0014] The method controls the flow of data through at least a portion of the receiver using a synchronization event generated by the pull of audio data by an audio device and a timing event indicating the rate at which audio data is received from a network cloud, in various embodiments. Therefore, the data flow of audio data through at least a portion of the receiver is controlled not by the rate at which data is received from the network cloud, but by the rate at which audio data is pulled by the audio device (e.g., the rate is controlled by the audio device). This results in a stable flow of audio data from at least a portion of the receiver to the audio device. Thus, the method can solve problems caused by unstable timing between the reception of audio data from the network cloud and the use of audio data by the audio device. For example, by precisely controlling the rate at which audio data flows through the receiver, processing of audio data, such as echo cancellation, at the receiver can be facilitated, and the stability of audio data use by the audio device can be maintained. The method described herein enables the handling of echo reappearances when the audio source is in the same room as the audio device. The method described herein provides a constant delay in live events, independent of network fluctuations.

[0015] In certain embodiments, one computer may be configured to implement at least a portion of the method and associated apparatus and configuration at the transmitting end, and another computer may be configured to implement at least a portion of the method and associated apparatus and configuration at the receiving end. In other embodiments, both the transmitting and receiving ends may be implemented by the same computer, for example, with respect to voice over Internet Protocol (VoIP). This specification also describes configurations in the form of a non-volatile computer-readable medium containing computer-readable instructions. When an instruction is executed, one or more computers function as at least part of the apparatus or cause at least some of the method steps of the method to be implemented.

[0016] Software including computer-readable instructions used herein may be in the form of high-level code such as C or Java, or in the form of machine code. In some embodiments, the software may run on a single computer. In other embodiments, two or more computers may communicate with each other over a network, and the software may be configured in various ways. This allows parts of the software to be distributed operable across two or more computers and executed by two or more computers. Although the methods described herein are generally described as being implemented by software, in various embodiments they may be implemented in hardware or in combination of hardware and software. As will be recognized by those skilled in the art who have studied this disclosure, the methods, apparatus, and associated configurations described herein may be implemented in a distributed computing environment in which a particular task is performed by processors and data storage distributed within a network cloud. A nominal representation of data may be the data itself, or it may be a pointer, description, or other data that may be used to generate the data.

[0017] As used herein, a processor is an electronic circuit that performs operations on data, such as arithmetic operations, logical operations, and input / output (I / O) operations. As used herein, a computer includes one or more processors and, in various embodiments, may include memory, a display, a microphone, a speaker, a mouse, a keyboard, a storage device, an input / output device, a network interface, and the like. A computer may include, for example, a single-processor or multi-processor computer, a minicomputer, a mainframe computer, and personal computers, portable computing devices, mobile devices, cell phones, tablets, watches, and other processor-based devices, which are readily recognizable to those skilled in the art who have considered this disclosure. As used herein, a computer may, in various embodiments, include one or more processors or processes distributed within a network cloud.

[0018] The terms network cloud, cloud, or network as used herein include, for example, the Internet, local area networks (LANs), cellular networks (e.g., 4G, 5G), text messaging networks (e.g., MMS or SMS networks), wide area networks (WANs), and combinations thereof. Data may be communicated through the network cloud by various wired and wireless technologies and combinations thereof. The network cloud may include, for example, processors, data storage devices, input / output devices, computers, servers, routers, amplifiers, wireless transmitters, wireless receivers, optical devices, virtualization resources, etc., which are readily recognizable to those skilled in the art who have considered this disclosure.

[0019] In various embodiments, audio data is a binary representation of sound, such as human voice, musical instrument, or other vibrations that propagate as such acoustic waves. In certain embodiments, audio data may include data packets configured for network transmission. Data packets may include both control information and payloads, configured in various ways, as will be readily apparent to those skilled in the art who have studied this disclosure.

[0020] Figure 1 shows an exemplary voice communication system 10, which includes a voice source 20 that communicates with a voice device 120 via a network cloud 99 in order to communicate voice data 15 from the voice source 20 to the voice device 120 via a network cloud 99. The voice source is configured to generate voice data 15. Thus, in some implementations, the voice source 20 may include a microphone configured to capture sound in analog or digital format. In such implementations, the voice source 20 digitizes the sound, thereby forming the voice data 15. Thus, for example, the voice data 15 may include digital representations of sounds captured simultaneously by the voice source 20, such as spoken language and music. The voice data 15 may also include digital representations of physiologically generated sounds, such as heartbeats, respiration, and digestive sounds such as peristaltic contractions. In various implementations, the transmitted voice data 15 may generally represent a complete acoustic spectrum from low to high (ultrasonic) frequencies and may be transmitted losslessly so that no information is lost due to compression between the voice source 20 and the voice device 120. The voice device 120 may then analyze the voice data 15 to detect medical conditions, including, for example, cardiac, digestive, and cognitive pathologies. The voice device 120 may use various artificial intelligence techniques (e.g., large-scale language models) to detect medical conditions using the voice data 15. As a further example, the voice data 15 may include natural sounds for detecting natural phenomena and machine-related sounds for detecting mechanical defects.

[0021] In other implementations, the audio data 15 may include a digital representation of sound captured at an earlier point in time, stored using a digital storage medium in a digital format such as the mp4 format, for example, and may thereafter be input via the audio source 20. As another example, the audio data 15 may include a digital representation of sound stored in analog format, for example, phonograph records or magnetic tape recordings. The audio data 15 may be from a concurrent time point, an earlier time point, or both a concurrent time point and an earlier time point, therefore, real-time in the context of the voice communication system 10 generally refers to the use of the audio data 15 by the audio device 120 concurrently with the communication of the audio data 15 from the audio source 20.

[0022] The audio device 120 may generally be any device that utilizes audio data that requires accurate delivery of the reconstructed audio data 15. For example, in some implementations, the audio device 120 may broadcast the audio data 15 in a form perceptible to humans. In such implementations, the audio device 120 may include amplifiers, speakers, and the like, as well as associated operable software. For such audio devices, the audio data is delivered accurately so that the broadcast reproduces the audio data 15 generated by the audio source 20, and is free from distortion and noise, and is otherwise clear.

[0023] In other implementations, the audio device 120 may utilize the audio data 15 by further processing the audio data 15. For example, the audio device 120 may utilize the audio data when performing speech recognition, transcription, language translation, or audio correction. The audio device 120 may be implemented at least in part using artificial intelligence (AI), and may be implemented at least in part in a cloud. In various implementations, the audio device 120 may be virtual, physical, or both virtual and physical.

[0024] Figure 2A shows the transmitter 11 of an exemplary voice communication system 10, which includes a voice hardware abstraction layer 40, a thread pool 50, and a transmitter network hardware abstraction layer 60, working together to communicate voice data 15 from a voice source 20 to a network cloud 99. In this implementation example, the transmitter 11 of the exemplary voice communication system 10, which includes the voice hardware abstraction layer 40, the thread pool 50, and the transmitter network hardware abstraction layer 60, constitutes part of a computer 97.

[0025] As shown in the figure, the audio hardware abstraction layer 40 includes an audio source 20 and an audio buffer 30. Here, the audio buffer 30 works in cooperation with the audio source 20 to store audio data 15 from the audio source 20 during its generation. In this implementation example, the audio hardware abstraction layer 40 includes a software layer and a hardware layer that work together operably. The software layer may be operable to support and control the audio source 20 and the audio buffer 30. The software layer may also provide an application programming interface (API) for accessing the audio source 20, the audio buffer 30, and the audio data 15. In various implementation examples, the software layer may be formed as part of the operating system, firmware, or a combination thereof of the computer 97. The hardware layer may include hardware that supports the audio source 20 and the audio buffer 30. The hardware layer may also include hardware that supports the software layer. The hardware layer may include a processor, an A / D converter for digitizing analog sound captured by a microphone, and memory, etc., as will be readily recognizable to those skilled in the art who have studied this disclosure. In this implementation example, a portion of the hardware of computer 97 may constitute the hardware layer. In this implementation example, the transmitter 11 is shown to be implemented by computer 97, but in various other implementation examples, at least a portion of the transmitter 11 may be implemented in the cloud, in a distributed computing environment, or in other ways.

[0026] As shown in Figure 2A, the transmitting network hardware abstraction layer 60 includes a network interface 80 and a network buffer 70. The network buffer 70 works with the network interface 80 to store voice data 15 for communication with the network cloud 99. In this implementation example, the transmitting network hardware abstraction layer 60 includes a software layer and a hardware layer that work together operably. The hardware layer includes the network interface 80, which includes hardware that implements network communication. The hardware layer may also include other hardware that supports the network interface 80 and the software layer.

[0027] The software layer may be capable of operating to support and control the hardware layer, including the network interface 80 and the network buffer 70. In this example implementation, the software layer may be formed as part of the operating system, firmware, or a combination thereof of the computer 97.

[0028] The transmitting network hardware abstraction layer 60, including hardware and software layers, may be configured to implement, for example, Ethernet (IEEE 802.3), wireless (IEEE 802.11), Bluetooth, 5G, or other wired or wireless communication with the network cloud 99, as readily apparent to a person skilled in the art who has considered this disclosure. The transmitting network hardware abstraction layer 60 may include an API for communicating voice data 15 with a network interface 80 including a network buffer 70, and for controlling data communication with the network cloud 99. The voice data 15 may be communicated with the network cloud 99 in the form of data packets by the transmitting network hardware abstraction layer 60.

[0029] As shown in Figure 2A, the audio hardware abstraction layer 40 communicates audio data 15 with the transmitting network hardware abstraction layer 60 via a thread pool 50. The thread pool 50 maintains multiple threads waiting for tasks to be allocated for concurrent execution. The number of threads in the thread pool 50 may depend on the number of processors. For example, four processors are required to support a thread pool such as thread pool 50, which has four threads. As an example, Apple's iPhone 12 uses the Apple A14 Bionic system-on-a-chip, which has six processors (cores), and may implement a thread pool with six threads, such as thread pool 50. In certain implementation examples, it has been observed that two threads are generally sufficient.

[0030] Figure 2B shows the receiver 13 of an exemplary voice communication system 10, which includes a receiver-side network hardware abstraction layer 95, an audio output hardware abstraction layer 101, and an audio device 120, and which cooperate to receive audio data 15 from a network cloud 99 and communicate the received audio data 15 to the audio device 120. In various implementations, the receiver 13 of the exemplary voice communication system 10, which includes the receiver-side network hardware abstraction layer 95, the audio output hardware abstraction layer 101, and the audio device 120, may constitute part of a computer, such as a computer 97. In various implementations, at least part of the receiver 13 may be implemented in the cloud, in hardware, in a distributed computing environment, or in other ways.

[0031] As shown in Figure 2A, the receiving network hardware abstraction layer 95 includes a network interface 90 that communicates with the network cloud 99. In this implementation example, the receiving network hardware abstraction layer 95 includes a software layer and a hardware layer that cooperate operably with each other. The hardware layer includes the network interface 90, which contains hardware for implementing network communication. The hardware layer may also include other hardware that supports the network interface 90 and the software layer. The software layer may be operable to support and control the hardware layer, which includes the network interface 90. The receiving network hardware abstraction layer 95 is configured to implement wired or wireless communication with the network cloud 99. The receiving network hardware abstraction layer 95 may also include an API for controlling data communication with the network cloud 99. The receiving network hardware abstraction layer 95 may include a buffer that can operate with the network interface 90. For example, the receiving network hardware abstraction layer 95, which includes a hardware layer and a software layer, may be configured to implement Ethernet (IEEE 802.3), wireless (IEEE 802.11), Bluetooth, 5G, or other wired or wireless communication with the network cloud 99, as can be easily recognized by a person skilled in the art who has considered this disclosure.

[0032] As shown in Figure 2A, the audio output hardware abstraction layer 101 includes an audio data management module 104, a first audio processing module 108, and an audio interface 116. These may be configured as a software layer and a hardware layer that work together operably. The software layer may be operable to support and control the audio data management module 104, the first audio processing module 108, the second audio processing module 112, and the audio interface 116. The software layer of the audio output hardware abstraction layer 101 may provide an application programming interface (API) for accessing the audio data management module 104, the first audio processing module 108, the second audio processing module 112, and the audio interface 116. The audio output hardware abstraction layer 101 may include buffers that can work with the audio data management module 104, the first audio processing module 108, the second audio processing module 112, and the audio interface 116. The software layer of the audio output hardware abstraction layer 101 may, in various implementations, be formed as part of the operating system, firmware, or a combination thereof of a computer such as computer 97. The hardware layer of the audio output hardware abstraction layer 101 includes hardware that supports the audio data management module 104, the first audio processing module 108, and the audio interface 116. The hardware layer may also include hardware that supports the software layer. Parts of a computer such as computer 97 may form at least part of the hardware layer. In various implementations, at least part of the hardware layer may be configured in the cloud.

[0033] During operation, the exemplary voice communication system 10 communicates voice data 15 from a voice source 20 at the transmitting side 11 to a voice device 120 at the receiving side 13 via a network cloud 99, according to the exemplary method 300 shown in Figure 3. As shown in Figure 3, the method 300 begins in step 301. In step 305, the voice source 20, which is included in the voice hardware abstraction layer 40 at the transmitting side 11 of the voice communication system 10, generates the voice data 15.

[0034] In step 310, the voice data 15 is transferred from the voice hardware abstraction layer 40 to the transmitting network hardware abstraction layer 60 using threads in the thread pool.

[0035] In step 315, the voice data 15 is transmitted via the network cloud 99 by the transmitting network hardware abstraction layer 60. The voice data 15 may also be transmitted in the form of data packets.

[0036] In step 320, the voice data 15 is received from the network cloud 99 by the receiving network hardware abstraction layer 95. The voice data 15 may also be received from the network cloud 99 as a data packet.

[0037] In step 325, the audio data 15 is pushed from the receiving network hardware abstraction layer 95 to the audio output hardware abstraction layer 101.

[0038] In step 330, the audio data 15 is pulled by the audio device 120 through the audio output hardware abstraction layer 101. This ensures that the audio device 120 receives the audio data exactly as requested by the audio device 120. Therefore, the audio device 120 controls the flow of the audio data 15 through the audio output hardware abstraction layer 101, and does not control the reception of the audio data 15 by the receiving network hardware abstraction layer 95.

[0039] In step 335, the audio device 120 pulls audio data 15 from the audio output hardware abstraction layer 101.

[0040] In step 340, the audio device uses the audio data 15. Method 300 ends in step 361.

[0041] The transmitting side 11 of the voice communication system 10 may implement the exemplary method 400 shown in Figure 4A.

[0042] As shown in Figure 4A, method 400 begins in step 401. In step 405, the audio data 15 is generated by the audio source 20.

[0043] Next, in step 410, the audio buffer 30 is filled with audio data 15 while the audio source 20 is generating the audio data 15.

[0044] In step 415, event 54 is triggered when the audio buffer 30 is filled.

[0045] In step 420, event 54 starts a thread in thread pool 50.

[0046] In step 425, the thread transfers the audio data 15 from the audio buffer 30 of the audio hardware abstraction layer 40 to the transmitting network hardware abstraction layer 60, which includes the network interface 80 and the network buffer 70. By using threads from the thread pool 50, performance is improved, and runtime delays caused by frequently creating and destroying threads for short-lived tasks, such as transferring the audio data 15 from the audio hardware abstraction layer 40 to the transmitting network hardware abstraction layer 60, can be avoided.

[0047] The audio buffer size of the audio buffer 30 (e.g., 512, 1024, etc.) may be defined using the audio hardware abstraction layer 40 based on the audio time samples (in ms) and sample rate. Changing the buffer size may disrupt the timing of event 54. In certain implementation examples, the first event 54 may take longer to occur than subsequent events 54. This is because the audio buffer 30 must be filled to trigger the first event 54, while subsequent events 54 may occur earlier than the first event 54 because the audio buffer is already partially filled. The audio communication system 10 and method 400 may employ optimized acquisition using an optimized buffer size. Event 54 may be triggered when the audio buffer 30 is filled, thereby acquiring audio data 15 from the audio buffer 30 when the audio buffer 30 is full, preventing overflow. In some implementation examples, the audio buffer size of the audio buffer 30 is fixed. Therefore, the size cannot be adjusted using the audio hardware abstraction layer 40.

[0048] In step 430, the voice data 15 is transmitted as a data packet via the network cloud 99 by the transmitting network hardware abstraction layer 60. Method 400 ends in step 441.

[0049] In this implementation example, the thread pool 50 implements a software design that achieves execution concurrency. For example, the audio source 20 may sample audio at a fixed rate. This generates audio data 15 at a fixed rate and fills the audio buffer 30 at a fixed rate. Even though the audio data 15 is generated at a fixed rate, the transmitter 11 cannot force the transmission of the audio data 15 from the transmitter network hardware abstraction layer 60 to the network cloud 99. To reconcile the reception of the audio data 15 by the transmitter network hardware abstraction layer 60 with the transmission of the audio data 15 as a data packet via the network cloud 99 by the transmitter network hardware abstraction layer 60, the thread pool 50 separates the audio hardware abstraction layer 40 from the transmitter network hardware abstraction layer 60. Therefore, in this implementation example, the generation of audio data 15, which is generally at a fixed rate, is separated from the transmission of audio data 15 via the network cloud 99, which may vary. Therefore, the generation of the audio data 15 is not hindered by fluctuations in the rate at which the audio data 15 is transmitted over the network cloud 99 (for example, the audio source 20 does not need to wait for the network cloud 99).

[0050] Note that thread pools, such as thread pool 50, are typically used to isolate the receiving side (not the transmitting side) of a network, such as the receiving side 13 of the network cloud 99. In the transmitting side 11 of the voice communication system 10, a thread in thread pool 50 becomes available whenever voice data 15 becomes available in the voice hardware abstraction layer 40. While the current thread is attempting to send the voice data 15 to the network cloud 99 via the transmitting network hardware abstraction layer 60, the next thread waits to receive data from the voice hardware abstraction layer 40. Due to delays in the network cloud 99, multiple threads may be required. In this implementation example, the transmission of voice data 15 from the transmitting network hardware abstraction layer 60 to the network cloud 99 does not block the transmitting side 11. The voice hardware abstraction layer 40 does not need to wait for the transmission of voice data 15 via the network cloud 99. Therefore, there is no voice buffer overflow from the voice buffer 30, and consequently, no loss of voice data 15.

[0051] The receiving side 13 of the voice communication system 10 may implement the exemplary method 500 shown in Figure 4B. The exemplary method 500 begins in step 501. In step 505, the receiving side network hardware abstraction layer 95 listens on the network cloud 99 for data packets of voice data 15.

[0052] In step 510, the receiving network hardware abstraction layer 95 receives data packets of voice data 15 from the network cloud 99.

[0053] In step 515, a timing event 114 is generated. The timing event 114 indicates the timing of the reception of data packets of voice data 15 from the network cloud 99 by the receiving network hardware abstraction layer 95, and the timing of the resulting push of data packets of voice data 15 from the receiving network hardware abstraction layer 95 to the voice output hardware abstraction layer 101.

[0054] In step 520, the voice data 15 is pushed from the receiving network hardware abstraction layer 95 to the voice data management module 104. A data producer as described herein is a module (process) that generates or owns data. A data consumer is a module (process) that requires data generated or owned by a data producer. Push means that the data producer controls the communication of data to the data consumer. In step 515, the receiving network hardware abstraction layer 95, which is the data producer, controls the communication of data to the voice data management module 104, which is the data consumer. For example, the receiving network hardware abstraction layer 95 may push the voice data 15 to the voice data management module 104 as it is generated by being received from the network cloud 99.

[0055] In step 525, the voice data management module 104 manages the data packets, for example, by reconfiguring data packets, correcting data packets, supplementing data packets, dropping data packets (if too late), and / or arranging data packets in the correct order. Timing events 114 may be used in the management of data packets. Fluctuations in timing events 114 may be observed indicating the stability of the input stream of data packets from the network cloud 99.

[0056] In step 530, the first voice processing module 108 pulls voice data 15 from the voice data management module 104. Pulling means that the data consumer controls the communication of data from the data producer. For example, the data consumer may send a data request to the data producer to initiate the communication of data. The data request may occur at either regular or irregular time intervals. For example, in step 530, the first voice processing module 108, which is the data consumer, pulls voice data 15 from the voice data management module 104, which is the data producer. Therefore, the first voice processing module 108, not the voice data management module 104, controls the communication of voice data 15 from the voice data management module 104.

[0057] In step 535, the first audio processing module 108 processes the audio data 15.

[0058] In step 540, the second audio processing module 112 pulls audio data 15 from the first audio processing module 108.

[0059] In step 545, the second audio processing module 112 processes the audio data 15.

[0060] In step 550, the voice interface 116 pulls voice data 15 from the second voice processing module 112 in response to a request from the voice device 120.

[0061] In step 555, the audio device 120 pulls audio data 15 from the audio interface 116.

[0062] In step 560, a synchronization event 118 is generated when the audio device 120 pulls audio data 15 from the audio interface 116. The synchronization event 118 indicates the timing when the audio data 15 is being pulled from the audio interface 116 by the audio device 120. Note that in various implementation examples, the audio data 15 is generally pulled from the audio interface 116 at a constant rate, so the synchronization event 118 is generated at a constant rate.

[0063] In step 565, the pull of audio data 15 through the audio output hardware abstraction layer is controlled using timing events 114 and synchronization events 118. For example, using synchronization events 118, the audio data management module 104 can compare synchronization events 118 (e.g., the timing of the pull of audio data 15 from the audio interface 116 by the audio device 120) with timing events 114 (e.g., the timing of the reception of data packets of audio data 15 from the network cloud 99 by the receiving network hardware abstraction layer 95). The audio data management module 104 may use synchronization events 118 and timing events 114 to manage the buffer size and perform data packet loss concealment. Data packet loss concealment includes, for example, extending data packets to make the buffer last longer until more data packets become available.

[0064] The flow of audio data 15 through the audio output hardware abstraction layer 101 is controlled using timing events 114 and synchronization events 118 to provide the audio device 120 with the audio data 15 requested by the audio device 120. Thus, the audio data 15 may be variably received from the network cloud 99 by the receiving network hardware abstraction layer 95 and, correspondingly, variably pushed from the receiving network hardware abstraction layer 95 to the audio output hardware abstraction layer 101. The audio output hardware abstraction layer 101 then pulls the audio data 15 internally using timing events 114 and synchronization events 118, thereby delivering the audio data 15, reconstructed at a substantially constant rate, to the audio device 120 as requested by the audio device 120.

[0065] For example, by using the synchronization event 118, the voice data management module 104 can compare the synchronization event 118 (e.g., the timing of the voice device 120 pulling voice data 15 from the voice interface 116) with the timing event 114 (e.g., the timing of the receiving network hardware abstraction layer 95 receiving data packets of voice data 15 from the network cloud 99). The voice data management module 104 may use the synchronization event 118 and the timing event 114 to manage the buffer size and perform data packet loss concealment. Data packet loss concealment includes stretching data packets to make the buffer last longer until more data packets are available.

[0066] Therefore, the pull of audio data 15 by the audio device 120 controls the flow of audio data 15 through the receiving side 13, rather than the reception of audio data 15 from the network cloud 99. The stability of processing of audio data 15 by the audio device 120 may be maintained by precisely timing the pull of audio data 15 by the audio device 120. For example, the pull of audio data 15 at regular time intervals by the audio device 120, which is configured as a speaker system for generating audible sound, may maintain the stability of broadcasting of audio data 15 by the audio device 120.

[0067] The exemplary method 500 ends in step 571.

[0068] The exemplary receiver 13 of the exemplary voice communication system 10 includes, for illustrative purposes, a first voice processing module 108 and a second voice processing module 112. Note that in various implementations, the receiver 13 may include one or more voice processing modules, such as the first voice processing module 108 and the second voice processing module 112. In various implementations, voice processing modules such as the first voice processing module 108 and the second voice processing module 112 may be configured to implement, for example, echo cancellation, low-pass filtering, high-pass filtering, noise reduction, gain control, resampling, jitter control, and other sound quality improvements. The voice processing modules may include heuristics that detect network fluctuations in advance and compensate for these network fluctuations.

[0069] For example, if the first audio processing module 108 is configured to implement echo cancellation, an echo signal (the same as the signal but at a lower volume) follows the signal. The first audio processing module determines the delay between the signal and the echo signal and uses this delay to cancel the echo signal. In order to accurately cancel the echo signal, the timing between the signal and the echo signal must be kept constant. Therefore, the audio data 15 is pulled from the audio data management module 104 at a constant rate by the first audio processing module 108. In this example, the pull of audio data 15 from the audio interface 116 by the audio device 120 generates a synchronization event 118. The synchronization event 118 is communicated from the audio interface 116 to the audio data management module 104, and then the first audio processing module 108 keeps the pull of audio data 15 from the audio data management module 104 constant. This ensures that the timing between the signal and the echo signal is kept constant.

[0070] The above description, along with the figures, discloses and illustrates various exemplary implementations. These implementations are not intended to limit the scope of protection, but rather to aid in understanding the context of the language used herein and in the claims. The abstract is provided solely to satisfy the requirements of 37C.FR1.72(b). Therefore, the abstract is not intended to identify or define the main elements of the methods, systems, and configurations disclosed herein. By examining this disclosure and the exemplary implementations herein, those skilled in the art will readily recognize that various changes, modifications, and variations can be made without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A method for communicating voice data, A step of generating audio data that represents an audio signal using an audio source included in the audio hardware abstraction layer, The steps include generating an event by filling the audio buffer included in the audio hardware abstraction layer with the audio data from the audio source, The steps include: triggering a thread in the thread pool in response to the event, wherein the thread transfers the audio data from the audio hardware abstraction layer to the transmitting network hardware abstraction layer, and the thread pool separates the audio hardware abstraction layer from the transmitting network hardware abstraction layer; The steps include: transmitting the voice data as a data packet from the transmitting network hardware abstraction layer to the network cloud; The steps include receiving the data packets from the network cloud using a receiving network hardware abstraction layer, The steps include pushing the data packet from the receiving network hardware abstraction layer to the audio output hardware abstraction layer, A step of pulling the data packets from the audio output hardware abstraction layer by an audio device, thereby generating a synchronization event; A step of controlling the pull of the data packet through the audio output hardware abstraction module using the synchronization event, thereby making the data packet available for pull by the audio device. Methods that include...

2. A method for transmitting audio data, The steps include generating audio data using audio sources included in the audio hardware abstraction layer, The steps include: filling the audio buffer, which is further included in the aforementioned audio hardware abstraction layer, with the audio source; The steps include generating an event by filling the aforementioned audio buffer, The steps include transferring the audio data from the audio hardware abstraction layer to the transmitting network hardware abstraction layer using a thread from the thread pool that was triggered by the event, The steps include transmitting the voice data from the transmitting network hardware abstraction layer to the network cloud, Includes, The thread pool separates the voice hardware abstraction layer from the transmitting network hardware abstraction layer, thereby separating the generation of voice data from the transmission of voice data. method.

3. A method for receiving audio data, The receiving network hardware abstraction layer receives voice data representing a voice signal as a data packet from the network cloud, and The steps include pushing the audio data from the receiving network hardware abstraction layer to the audio output hardware abstraction layer, A step of pulling the audio data from the audio output hardware abstraction layer by an audio device, thereby generating a synchronization event; The steps include controlling the pull of the data packets through the audio output hardware abstraction module using the aforementioned synchronization event, Includes, The step of pulling the data packets from the audio output hardware abstraction module by the audio device controls the flow of the data packets through the receiving side, rather than the receiving side receiving the data packets from the network cloud. method.

Images