System and method for realizing robust audio transmission by fusing RTC and CDN

Abstract

The invention discloses an audio transmission method for a user voice segment. The method includes obtaining, by an encoder, a user's speech segment. The method also includes obtaining reference samples associated with the user's speech segment from one or more servers in communication with the encoder. The method further includes establishing a timestamp for the user speech segment associated with the reference sample, and encoding, by an encoder, an audio data packet including the user speech segment and the timestamp associated with the user speech segment. The method further includes transmitting the audio data packet in real time to a decoder, during decoding, fusing the user's speech segment with the reference sample according to a timestamp associated with the user's speech segment.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of audio processing, and in particular to optimizing audio transmission techniques. BACKGROUND

[0002] People can communicate online through various communication channels and multiple media types. For example, real-time communication (RTC) can be conducted through audio and / or video conferencing or streaming, or a simple telephone voice call. The audio and / or video communication can be (or can include) voice, sound (e.g., a singing segment), visual content, or a combination thereof. Such RTC can include one or more sending users sending audio and / or video to one or more receiving users. For example, a concert can be live-streamed to many viewers. As another example, a sending user can live-stream a singing of a song (e.g., karaoke) to viewers, which can include the sending user’s singing and its accompaniment, etc.

[0003] In RTC, users can desire to improve the quality of the transmitted audio. For example, users can desire to reduce or eliminate buffering, sound packet loss, jitter, or a combination thereof caused by unstable network conditions. SUMMARY

[0004] In one aspect, a method of audio transmission of a user’s voice segment is disclosed. The method includes receiving, by an encoder, the user’s voice segment, and obtaining, from one or more servers in communication with the encoder, a reference sample associated with the user’s voice segment. The method further includes creating a timestamp of the reference sample associated with the user’s voice segment, and encoding, by the encoder, an audio packet including the user’s voice segment and the timestamp associated with the user’s voice segment. The method further includes transmitting, in real-time, the audio packet to a decoder, and during decoding, fusing the user’s voice segment in the audio packet with the reference sample based on the timestamp associated with the user’s voice segment.

[0005] On the other hand, the present invention discloses a non-transitory computer-readable storage medium configured to store a computer program for audio transmission of a user's voice segment. The computer program includes instructions executable by a processor. The instructions executable by the processor include receiving the user's voice segment through an encoder. The instructions executable by the processor also include obtaining reference samples associated with the user's voice segment from one or more servers in communication with the encoder. The instructions executable by the processor include creating a timestamp of the reference sample associated with the user's voice segment, and encoding an audio data packet including the user's voice segment and the timestamp associated with the user's voice segment through the encoder. The instructions executable by the processor include transmitting the audio data packet to a decoder in real time, and during decoding, fusing the user's voice segment in the audio data packet with the reference sample based on the timestamp associated with the user's voice segment.

[0006] In another aspect, the present invention discloses a method for audio transmission of a user's voice segment. The method includes obtaining a reference sample associated with the voice segment from one or more servers via a content delivery network. The method also includes receiving an audio data packet from an encoder in real time, the audio data packet including the user's voice segment and a timestamp associated with the user's voice segment that is a time offset relative to the reference sample. Furthermore, the method includes fusing the user's voice segment with the reference sample based on the timestamp associated with the user's voice segment after receiving the audio data packet from the encoder. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The following detailed description will be described with reference to the accompanying drawings, which will facilitate a better understanding of the present invention. It should be noted that, in accordance with conventional practice, the various parts in the drawings are not drawn to scale, but rather the sizes of the various parts are arbitrarily enlarged or reduced in the drawings for the sake of clarity.

[0008] Figure 1 is an example diagram of a media transmission system.

[0009] Figure 2 is an example diagram of a real-time audio communication system.

[0010] Figure 3 is an example diagram of a real-time audio communication system for audio recording.

[0011] Figure 4 is a diagram of another example of a real-time audio communication system for audio recording.

[0012] Figure 5 This is an example flow chart of a real-time audio communication technology in one embodiment of the present invention.

[0013] Figure 6is an example flowchart of real-time audio communication techniques in another embodiment of the invention. DETAILED DESCRIPTION

[0014] An audio communication system includes a sending end (i.e., a sending device) and a receiving end (i.e., a receiving device). The sending end can perform operations such as audio capture, audio conversion (e.g., converting an analog audio signal to a digital signal), audio encoding, and audio transmission. For example, the sending end can be a client device that captures audio in real-time and transmits (e.g., via streaming) the audio to one or more receiving ends. As another example, the sending end can also be a streaming server that streams real-time audio or pre-recorded audio to one or more receiving ends. The receiving end performs operations such as audio decoding, audio decompression, audio conversion (e.g., converting from a digital format to an original analog audio signal), and transmitting the audio to one or more playback devices (e.g., earphones, speakers, etc.). Thus, as described above, the sending end can be or can include an encoder and the receiving end can be or can include a decoder. Moreover, the sending end and the receiving end can communicate over a network. That is, encoded audio data can be transmitted from the sending end to the receiving end over the network. For example, the audio data can be transmitted from the sending end to the receiving end via multiple servers of the network.

[0015] The captured audio can be any type of sound wave captured by the sending end (e.g., a microphone of the sending end). For example, the captured audio can be speech (e.g., a speech segment) of a user of the sending end. The captured speech (e.g., the speech segment) can be the user speaking or singing. However, the invention is not limited to only capturing speech of a user. For example, the audio captured or otherwise obtained by the sending end can also be live stream audio data or pre-recorded audio data such as a music file, an audiobook, a presentation, etc., or a combination thereof. Additionally, it should be noted that while audio communication is described herein, video communication scenarios are also contemplated. That is, the audio transmitted from the sending end to the receiving end can be transmitted in conjunction with video data (e.g., a video conference and / or a video stream that can include audio content).

[0016] Various techniques can be used to encode and decode audio. For example, analog-to-digital conversion (ADC) can be used to encode / decode audio data, in which a continuous analog audio signal can be converted into discrete digital samples. In such encoding / decoding, a snapshot of the continuous analog audio signal can be taken periodically and assigned a digital value, such that the converted digital audio data can be converted back into a continuous analog signal for the receiving end to play audio. In addition, one or more compression algorithms (e.g., lossless and / or lossy compression, such as Free Lossless Audio Codec (FLAC), MP3, etc.) can be used to compress the audio data so that the receiving end can decompress the audio data and play the audio. Furthermore, when the audio data is transmitted in a digital format, the digital audio data can be divided into segments (e.g., packets) for transmission over a network. In such a case, each packet can contain a portion of the audio data as well as additional information for synchronization and / or error correction (e.g., error correction information to avoid packet loss, jitter, etc.).

[0017] The above-described conventional techniques of audio transmission can be used to encode, transmit, and decode audio data in real-time communication (RTC) over a network. For example, in a live karaoke application, a singer can sing into a microphone of a device (i.e., a sending end), such that the device can capture the singing into audio data, encode the audio data, transmit the audio data to an audience (e.g., one or more users of a receiving end), and play the audio data so that the audience can hear the singer’s performance in real time. In such a case, the audio data sent to the audience can also include a relevant accompaniment to the singing, such that the audio data played for the audience at the receiving end includes both the singing and the relevant accompaniment.

[0018] However, based on the above scenario, network conditions can significantly affect the quality of audio playback at the receiving end in RTC applications. For example, transmitting (e.g., streaming) the singer’s performance and the associated accompaniment can consume a large amount of bandwidth. In addition, actual network conditions can also vary, which can result in audio packet loss, jitter (e.g., caused by communication lag during RTC transmission), corrupted audio data, etc. Similarly, conventional packet loss concealment (PLC) algorithms employed by the receiving end to address some of the above issues are typically designed for speech audio data rather than singing / music audio data, whereby speech audio data consumes significantly less bandwidth, produces lower packet loss, and / or has less jitter.

[0019] According to the embodiments of the present disclosure, in addition to improving the listening experience of the receiving end, the network bandwidth consumption of the network and the processing power consumption of the receiving and sending devices can also be reduced. The singing audio data can be separated from the related accompaniment audio data during audio transmission. Therefore, the singing audio data can be transmitted from the sending end to the receiving end via RTC transmission, while the related accompaniment audio data can be transmitted in a manner other than RTC transmission or the receiving end can access the data in other ways. Therefore, the pressure on the network bandwidth can be significantly reduced, thereby reducing the packet loss and / or jitter that may be caused by fluctuations in network conditions. Therefore, by providing higher quality audio output, the final listening experience of the receiving end can be significantly improved. It should also be noted that the embodiments of the present disclosure can be used for any type of audio transmission and are not particularly limited to singing / music audio data.

[0020] First, let's look at an example of the hardware and software architecture for implementing a real-time audio communication system to provide a more detailed understanding of the specific embodiments. It should be noted that the present invention is not limited to real-time audio communication systems, and the real-time audio communication systems described herein are for illustrative purposes only, as they typically put pressure on network bandwidth consumption. Therefore, the present invention can be implemented using any audio and / or video communication system.

[0021] Figure 1 FIG. 1 is a diagram illustrating an example system for media transmission, represented by system 100, including transmission of real-time wide-angle audio data. Figure 1 As shown, system 100 may include multiple devices and networks, such as device 102 , device 104 , device 106 , and network 108 .

[0022] The device can be implemented by any configuration of one or more computers, such as a microcomputer, a mainframe computer, a supercomputer, a general-purpose computer, a special-purpose computer, an integrated computer, a database computer, a remote server computer, a personal computer, a laptop computer, a tablet computer, a mobile phone, a personal data assistant (PDA), a wearable computing device, or a computing service provider (e.g., a network host or a cloud service provider). In some embodiments, the device can be implemented in the form of a plurality of computer groups that are located in different geographical locations and can communicate with each other (e.g., via a network). Although certain operations can be shared by multiple computers, in some embodiments, different computers can be assigned to different operations. In some embodiments, the system 100 can be implemented using a general-purpose computer / processor with a computer program that executes any corresponding technology, algorithm, and / or instruction described herein when running. In addition, a special-purpose computer / processor comprising dedicated hardware can also be used to execute any method, algorithm, or instruction described herein.

[0023] Device 102 may include an internal configuration of hardware, including a processor 110 and a memory 112. Processor 110 may be any type of device capable of manipulating or processing information. In some implementations, processor 110 may include a central processing unit (e.g., a central processing unit, i.e., a CPU). In some implementations, processor 110 may include a graphics processor (e.g., a graphics processing unit, i.e., a GPU). Although the examples herein may be implemented with a single processor as shown, the use of multiple processors may embody advantages in speed and efficiency. For example, processor 110 may be distributed across multiple machines or devices (each machine or device having one or more processors), which may be directly coupled or connected via a network (e.g., a local area network).

[0024] The memory 112 may include any transient or non-transient storage device capable of storing code (such as instructions) and data that can be accessed by the processor (via a bus). The memory 112 described herein may be random access memory (RAM), read-only memory (ROM), optical / magnetic disk, hard disk, solid-state drive, flash drive, secure digital (SD) memory card, memory stick, compact flash (CF) card, or any combination of any suitable type of storage device. In some implementations, the memory 112 may be distributed across multiple machines or devices, such as network-based storage or cloud-based storage. The memory 112 may store data (not shown), an operating system (not shown), and one or more applications (not shown). The data may include any data for processing (for example, an audio stream, a wide-angle video stream, or a multimedia stream). At least one application may include a program that allows the processor 110 to execute instructions to generate control signals, which can be used to implement the technical functions described below. For example, when used as a transmitter and / or receiver, the application may include a program for executing Figure 5 and Figure 6 Instructions related to the technology described.

[0025] In some embodiments, in addition to the processor 110 and the memory 112, the device 102 may also include an auxiliary (e.g., external) storage device (not shown). The auxiliary storage device may be a storage device in the form of any suitable non-transitory computer-readable medium, such as a memory card, a hard drive, a solid-state drive, a flash drive, or an optical drive. Furthermore, the auxiliary storage device may be a component of the device 102 or a shared device accessible via a network. In some embodiments, the application programs in the memory 112 may be stored in whole or in part in the auxiliary storage device and loaded into the memory 112 for processing as needed.

[0026] Device 102 may include input / output (I / O) devices. For example, device 102 may include I / O device 114. I / O device 114 may be implemented in various ways. For example, it may be a display adapted for device 102 for displaying images of graphical data. I / O device 114 may be any device capable of transmitting visual, auditory, or tactile signals to a user, such as a display, a touch-sensitive device (e.g., a touch screen), a speaker, a headset, a light-emitting diode (LED) indicator light, or a vibration motor. I / O device 114 may also be any type of input device that requires or does not require user intervention, such as a keyboard, a numeric keypad, a mouse, a trackball, a microphone, a touch-sensitive device (e.g., a touch screen), a sensor, or a gesture-sensing input device.

[0027] I / O devices 114 may also include communication devices for transmitting signals and / or data. For example, I / O devices 114 may include wired devices for transmitting signals (e.g., audio signals) or data (e.g., audio data) from device 102 to another device. For another example, I / O devices 114 may include wireless transmitters or receivers using compatible protocols to transmit signals from device 102 to another device or receive signals from another device to device 102.

[0028] Device 102 may include a communication device 116 for communicating with another device. Communication may be performed via a network 108. Network 108 may be any combination of one or more communication networks of any suitable type, including but not limited to networks using Bluetooth communication, infrared communication, near field connection (NFC), a wireless network, a wired network, a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a cellular data network, or the Internet. Communication device 116 may be implemented in various ways, such as by a repeater / transceiver device, a modem, a router, a gateway, a circuit, a chip, a wired network adapter, a wireless network adapter, a Bluetooth adapter, an infrared adapter, an NFC adapter, a cellular network chip, or any combination of any suitable type of device that is adapted to device 102 to provide functionality for communicating with network 108.

[0029] Device 104, similar to device 102, may include a processor 118, a memory 120, an I / O device 122, and a communication device 124. Components 118-124 of device 104 may be implemented similarly to components 110-116 of device 102. Additionally, device 106 may include a processor 126, a memory 128, an I / O device 130, and a communication device 132. Components 126-132 of device 106 may be implemented similarly to corresponding elements 110-116 of device 102 and corresponding elements 118-124 of device 104.

[0030] Device 102, device 104, and device 106 can each be a receiving device (i.e., a receiving end) or a transmitting device (i.e., a transmitting end), for example, and can function as a receiving or transmitting device at different times during a real-time communication session. A receiving end can perform decoding operations, such as the decoding of an audio stream as described herein. Thus, a receiving end can also be referred to as a decoding device, which can include a decoder or simply be a decoder. A transmitting end can also be referred to as an encoding device, which can include an encoder or simply be an encoder. Devices 102, 104, and 106 can communicate with each other via network 108.

[0031] Figure 2 is an example diagram of a real-time audio communication system 200. Specifically, Figure 2 The example shown in has shown the real-time audio communication system for " karaoke television " (KTV).Certainly, this system also can realize other real-time audio communication.

[0032] like Figure 2 As shown, system 200 may include multiple singers and multiple audience members communicating via various networks. For example, system 200 may include a lead singer 202, a chorus 204, and an audience 206 communicating via network 208. For ease of illustration, lead singer 202 may use device 102 or a portion of device 102, chorus 204 may use device 104 or a portion of device 104, and the audience may use device 106 or a portion of device 106. Based on the above arrangement, lead singer 202 and chorus 204 may sing along with pre-recorded music in real time. For example, lead singer 202 and chorus 204 may sing along with pre-recorded music according to lyrics displayed on the displays of device 102 and device 104, respectively. The singing and pre-recorded music may then be transmitted to audience 206 in real time for listening and / or viewing, for example, via I / O devices 130 (e.g., speakers and / or display) of device 106.

[0033] To better implement the above-mentioned real-time streaming, the lead singer 202 (such as the device 102) can communicate with an application programming interface (API) 210 or can execute an application programming interface (API) 210 to merge the lead singer 202's performance with music stored in the music library. For example, the device 102 can include the API 210, and the API 210 can include a set of rules or protocols stored in the memory 112 and executed by the processor 110. The user can execute these rules or protocols by interacting with the device 102 (such as via the I / O device 114).

[0034] For example, the lead singer 202 can interact with the KTV application of the device 102 through the I / O device 114 to select a song to sing along with, as indicated by the song request 218. Based on the song request 218, the device 102 prompts the API 210 to execute appropriate rules or protocols to send a song request 222 to the music library 212. It should be noted that the music library 212 can be stored locally on the device 102 (e.g., in the memory 112) or the music library 212 can be stored externally and accessed by the device 102, such as on one or more servers. When the song request 222 is sent to the music library 212, a music download 224 or a music stream can be initiated to transfer the desired song from the music library 212 to the device 102 through the API 210. Finally, the lead singer 202 can be ready to start singing along with the selected song using the device 102.

[0035] By analogy, the chorus 204 can interact with the KTV application of the device 104 through the I / O device 122 to select the same song selected by the lead singer 202. To enable the selection of the same song, the lead singer 202 (e.g., the device 102) can share a token with the chorus 204 (e.g., the device 104) through the token sharing 207 to ensure that both the lead singer 202 and the chorus have the right to select the same song at the same time. Based on the token sharing 207, the chorus can submit a song request 226 similar to the song request 218.

[0036] Based on the song request 226, the device 104 invokes the API 214 (which can be similar to the API 210) to execute appropriate rules or protocols to send a song request 230 to the music library 216. It should be noted that the music library 216 can be stored locally on the device 104 (e.g., in the memory 120) or the music library 216 can be stored externally and accessed by the device 104, such as on one or more servers. If the music library 216 is stored externally, the music library 212 and the music library 216 can be a single music library that is accessible by both the device 102 and the device 104.

[0037] When the song request 230 is sent to the music library 216, a music download 232 or a music stream can be initiated to send the selected song from the music library 216 to the device 104 through the API 214. By this point, the chorus 204 is also ready to sing the selected song with the lead singer using the device 104. That is, the chorus 204 and the lead singer 202 can sing the selected song at the same time and in real-time streaming. It should also be noted that the chorus 204 can not be needed, in which case the lead singer 202 can complete the above steps to sing alone.

[0038] As mentioned above, the singing of the lead singer 202 and the chorus 204 can be transmitted to the audience 206 in real time through the network 208. The network 208 is similar to the network 108 described above. In order to transmit the singing and music to the audience 206 in real time, the lead singer 202 (e.g., from the device 102) transmits a lead singer stream 236 to the audience 206 (e.g., to the device 106) through the API 210, which can include the singing of the lead singer 202 and the related accompaniment.

[0039] Similarly, a chorus stream 238 including the singing of the chorus 204 and the related accompaniment thereof can be transmitted from the chorus 204 (e.g., from the device 104) to the audience 206 (e.g., to the device 106) via the API 214. The lead singer stream 236 and the chorus stream 238 can be transmitted to the audience 206 through the network 208.

[0040] In addition, in some cases, a background music (BGM) stream 234 can also be transmitted from the device 102 (e.g., from the lead singer 202) and / or the device 104 (e.g., the chorus 204) to the device 106 (e.g., the audience 206) for providing background music when the lead singer 202 and the chorus 204 are not actively performing real-time singing. Thus, based on the above, multiple participants on multiple devices can communicate through the network 208 to participate in a KTV stream.

[0041] Figure 3 An example of a real-time audio communication system for audio recording is shown, denoted as system 300. The system 300 can be implemented by a sending and / or receiving device such as the device 102, the device 104, and the device 106. That is, the system 300 can be a part of the system 100 in Figure 1 . The system 300 can be configured for real-time audio communication, referring to the KTV described in Figure 1 . Of course, the system 300 can be used for any type of real-time audio communication. Figure 2

[0042] As shown in Figure 3 , a user 302 can record his voice (e.g., capture his speaking and / or singing as a voice clip) through a recording device 304 (e.g., a microphone). For example, the user 302 can use the device 102 (the recording device 304 can be the I / O device 114 of the device 102) to record the voice of the user 302. After the voice of the user 302 is recorded, the voice (e.g., the voice clip) can be processed as audio data, e.g., one or more audio data packets, by the system 300, which includes all or part of the content of the recorded voice (e.g., the voice clip) of the user 302.

[0043] ​For example, audio data obtained from user 302 can be processed by audio processing module (APM) 306 first. APM 306 can be (or can include) hardware and / or software configured to manage, enhance, analyze, process audio data, or a combination thereof. That is, APM 306 can change (e.g., improve) the quality and / or one or more characteristics of audio data through one or more processing stages within APM 306. For example, APM 306 can reduce background audio noise, can reduce or eliminate unwanted echoes, can compress and / or decompress audio data, can adjust the frequency balance in the underlying audio signals of audio data (e.g., equalization (EQ) module), can perform dynamic range compression (DRC of audio data) to control the difference between loud and soft parts of the underlying audio signals of audio data, can correct for pitch variations, can apply various audio effects (e.g., reverb, chorus, distortion, other environmental elements, etc.), can perform speech-to-text recognition (e.g., provide text with audio transmission to a receiving end), etc. For example, in a KTV, user 302 can use recording device 304 to record singing while the user 302 is singing along with music. In this case, the music being played can also be recorded by recording device 304 when user 302 is singing. APM 306 can filter out the music at this time so that only the singing of user 302 is retained.

[0044] After audio data is processed by APM 306, it can be transmitted to mixer 308. Mixer 308 can fuse multiple audio sources together to create a final mixed audio that is finally transmitted to an audience. For example, mixer 308 can be (or can include) one or more modules that can fuse audio data (e.g., singing) from user 302 with relevant accompaniment 310 to create a final mixed audio that contains the singing and the relevant accompaniment 310. For example, user 302 can sing along with a selected song, the singing can be recorded and filtered, and then the singing can be mixed and aligned with the accompaniment to be transmitted to an audience. After mixer 308 creates the final mixed audio, encoder 312 encodes the audio data for transmission to receiving end 316 over network 314. The encoding of encoder 312 can be any type of encoding (e.g., encoding techniques described above) that forms the final mixed audio for transmission.

[0045] Network 314 is similar to network 108 in that it establishes communication between encoder 312 and receiving end 316. For example, encoder 312, mixer 308, APM 306, recording device 304, or a combination thereof can be part of device 102, and receiving end 316 can be device 104 and / or device 106.

[0046] Further, it should be noted that the APM 306 and / or the mixer 308 can be part of the encoder 312 or separate from the encoder 312. Further, in some configurations, the system 300 can also not include the APM 306. For example, the system 300 can include or be part of a neural network that can provide the functionality of the APM 306. For example, the encoder 312 can be a neural network encoder configured to provide audio processing functionality similar to the APM 306 described above.

[0047] When the receiving end 316 receives the encoded audio data, the decoder 318 can decode the audio data using any of the decoding techniques described above. The decoder 318 can be part of the receiving end 316 (e.g., part of the device 104 and / or the device 106). When the decoder 318 decodes the audio data, the decoded audio data (e.g., the final mixed audio before encoding) can be played through a speaker or headphone or other playback device 320 so that the audience 322 can hear the final mixed audio. In the example above, the audience 322 can hear the final mixed audio that includes the singing of the user 302 and the accompaniment 310 associated with the singing.

[0048] Figure 4 Another example of a real-time audio communication system for audio recording is shown, represented by system 400. The system 400 can be implemented by a sending end and / or a receiving end (e.g., the device 102, the device 104, and the device 106 of Figure 1 ). That is, the system 400 can be part of the system 100 of Figure 1 . The system 400 can be configured for real-time audio communication, such as KTV as described in Figure 2 and Figure 3 . See Figure 2 and Figure 3 . Of course, the system 400 can be implemented for any type of real-time audio communication.

[0049] Compared to the system 300, the system 400 can provide another way for transmitting audio data. As described below, the system 400 provides a way to transmit music or other audio data separately from the recorded audio, for relieving network bandwidth pressure.

[0050] To further illustrate the above improvements, user 402 records their voice (e.g., captures their speaking and / or singing as voice clips) using recording device 404. Recording device 404 is similar to recording device 304. For example, user 402 may use device 102, and recording device 404 may be I / O device 114 of device 102, to record the voice of user 402. After the voice of user 402 is recorded, the voice can be processed by system 400 as audio data, e.g., one or more audio data packets containing all or part of the voice of user 402.

[0051] For example, the audio data obtained from the user 402 may first be processed by the audio processing module (APM) 406. The APM 406 is similar to Figure 3 That is, APM 406 can change (e.g., improve) the quality and / or one or more characteristics of the audio data through one or more processing stages within APM 406. For example, APM 406 can filter out background noise (e.g., background music) from the audio recorded by recording device 404.

[0052] After the audio data is processed by APM 406, it can be transferred to mixer 408, which is similar to Figure 3 308 in the audio mixer. Mixer 308 mixes the audio recorded by user 302 with music 310, but mixer 408 may not perform a similar mix for the audio recorded by user 402. That is, mixer 408 can perform similar operations as mixer 308, but without mixing the audio recorded by user 402 with the associated accompaniment. For example, mixer 408 can combine one or more different audio signals to prepare the transmission of audio data, these different audio signals can be derived from the audio recorded by user 402 or created at APM 406. Thus, mixer 408 can create a transitional audio mix.

[0053] After the mixer 408 creates the transition mixed audio, the encoder 410 can encode the audio data and send it to the receiving end 418 via the real-time network (RTN) 412. The encoding of the encoder 410 can be any type of encoding, such as the encoding techniques described above, for preparing the transition mixed audio for transmission.

[0054] RTN 412, similar to network 314 and network 108, establishes communication between encoder 410 and receiver 418. For example, encoder 410, mixer 408, APM 406, recording device 404, or a combination thereof may be part of device 102, and receiver 418 may be device 104 and / or device 106. In this case, the interim mixed audio may be sent from encoder 410 to receiver 418 in real time via RTN 412.

[0055] In addition to receiving the encoded audio data (e.g., the encoded transition mix audio), the receiver 418 may also receive the associated accompaniment 414 via a content delivery network (CDN) 416. The CDN 416 can be any network separate from the RTN, allowing the associated accompaniment 414 to be transmitted or otherwise accessed by the receiver 418 independently of the audio data being transmitted from the encoder 410. In other words, the receiver 418 can receive the accompaniment 414 associated with the audio recorded by the user 402 separately from the audio recorded by the user 402. In this case, the associated accompaniment 414 does not need to be transmitted via the RTN 412, but can instead be sent to the receiver 418 via the CDN 416. In other words, the associated accompaniment 414 can be sent to the receiver 418 or accessed by the receiver 418 in a manner that does not consume the bandwidth of the RTN 412. As a result, the system 400 can reduce the overall bandwidth utilization of the RTN 412, thereby reducing audio packet loss, jitter, or other types of losses incurred when the receiver 418 receives the audio data.

[0056] It should be noted that the associated accompaniment 414 can be transmitted to the receiving end 418 before the encoder 410 transmits the audio data (e.g., recorded audio), can be transmitted during the encoder 410 transmitting the audio data, or can be transmitted to the receiving end 418 after the encoder 410 transmits the audio data. For example, the associated accompaniment 414 can be transmitted to the receiving end 418 via the CDN 416 (e.g., from one or more servers that may contain a music library) before the encoder 410 transmits the audio data. In this case, the associated accompaniment 414 can be stored locally on the receiving end 418 and accessed locally when needed.

[0057] For example, the system 400 (e.g., the receiving end 418) may include a decision module 420, which may be or may include hardware and / or software for managing, enhancing, analyzing, or a combination of the audio data (shown as a solid line) and the associated accompaniment 414 (shown as a dashed line) received from the encoder 410. The decision module 420 may include one or more processes and / or one or more rules for evaluating the audio data and the associated accompaniment 414 from the encoder 410.

[0058] For example, the audio recorded by recording device 404 can be associated with a timestamp. This association can be performed at APM 406, mixer 408, or encoder 410. In this case, the recorded audio, or audio data containing all or a portion of the recorded audio, is associated with a corresponding timestamp to indicate the time point of each portion of the recorded speech. In this case, the recorded audio and timestamps can be encoded by encoder 410 and transmitted to receiver 418 via RTN 412. Thus, decision module 420 can align the recorded audio with the associated accompaniment based on the timestamp of the recorded speech.

[0059] For example, the timestamp associated with the recorded audio may indicate a relative position offset from the beginning of the associated accompaniment 414 (e.g., from the beginning of the song being sung by user 402). Thus, decision module 420 may align the associated accompaniment 414 with the recorded audio based on the timestamp associated with the recorded audio, such that the recorded audio (e.g., singing) is aligned with the corresponding portion of the accompaniment. Similarly, the associated accompaniment 414 may also include one or more timestamps, such that the timestamps of the associated accompaniment 414 are compared and / or aligned with the timestamps of the recorded audio (e.g., singing). Thus, from a temporal perspective, the singing voice and the accompaniment may be accurately aligned.

[0060] It should also be noted that in addition to aligning the associated accompaniment 414 with the co-recorded audio, the decision module 420 may also perform other operations. For example, the decision module 420 may also check for integrity issues with the associated accompaniment 414 and / or the voice, such as data loss in an audio packet. In this case, the decision module 420 may determine whether the data received from the encoder 410 and the associated accompaniment 414 should be further processed (e.g., mixed and / or decoded) or whether additional steps are required to address or correct integrity issues (e.g., fill in the missing audio packet data). For example, if the decision module 420 determines that an audio packet has been lost, the decision module 420 may replace the lost audio packet with a locally stored audio packet, which may be part of the associated accompaniment 414.

[0061] After the decision module 420 evaluates the audio data, the decoder 422 can use any of the above-mentioned decoding techniques to decode the audio data. The decoder 422 can be part of the receiving end 418 (e.g., a part of the device 104 and / or the device 106). In addition, the decoder 422 can also be (or include) a mixer (such as the mixer 408), thereby, the decoder 422 can fuse the relevant accompaniment 414 with the recorded audio (e.g., the voice data sent from the encoder 410) to create a final mixed audio. The mixing operation is performed based on the timestamps as described above.

[0062] When the decoder 422 decodes and / or mixes the audio data, the decoded audio data (e.g., the final mixed audio including the recorded audio and the associated accompaniment 414) can be played through a playback device 424 (e.g., a speaker or a headphone) so that a listener 426 can hear the final mixed audio. In the example above, the listener 426 can hear the final mixed audio including the user’s 402 singing and the accompaniment 414 associated with the song.

[0063] Figure 5 FIG. 5 illustrates an example flowchart of techniques for real-time audio communication in accordance with an embodiment of the present disclosure. The techniques 500 can be implemented by a sending device (e.g., the device 102 of FIG. 1) and / or a receiving device (e.g., the devices 104 and 106 of FIG. 1). The techniques 500 can be implemented as software modules stored in the memory 112, the memory 120, and / or the memory 128 of the devices 102, 104, and 106, respectively, as instructions and / or data executable by the processors 110, 118, and / or 126 of the devices 102, 104, and 106, respectively. As another example, the techniques 500 can be implemented in hardware as a dedicated chip in which instructions executable by the dedicated chip are stored. Figure 1 Figure 1 Figure 1 Figure 1

[0064] The techniques 500 can be performed by the sending device and / or the receiving device at each time step. For example, if the sending device transmits audio data for playback at a rate of one audio data packet every 33 milliseconds, the techniques 500 can be performed approximately every 33 milliseconds by the sending device and / or the receiving device. Similarly, the techniques 500 can be performed by the sending device and / or the receiving device based on a defined time period. For example, the techniques 500 can be performed by the sending device and / or the receiving device once every X time steps, where X can be defined as a number of time steps. Portions of the techniques 500 performed by the sending device can be communicated to the receiving device over a network (e.g., the network 108). Similarly, portions of the techniques 500 performed by the receiving device can be communicated to the sending device over a network (e.g., the network 108).

[0065] As described above, the sending device can be configured to record an audio signal, such as a user’s voice. The audio recording can be accomplished at 502 using a recording device (e.g., a microphone), which can be similar to the recording device 404 of FIG. 4. Figure 4 As described above, the recorded audio can be a user’s singing associated with a particular song or music.

[0066] After the audio is recorded at 502, the audio data can be processed at 504 by an audio processing module (APM). The APM can be similar to the APM 406 of FIG. 4. Figure 4 ​​​​APM 406. That is, the APM can process the audio data at 504 to remove background noise or otherwise improve the quality of the recorded singing. Further, a timestamp can be associated with the audio data at 504 so that the recorded singing can be properly aligned with the associated accompaniment.

[0067] After processing the audio data at 504, the audio data (e.g., the recorded singing / voice clip) can be encoded at 506 using any of the encoding techniques described above. After encoding at 506, the audio data (e.g., the recorded singing / voice clip) can be transmitted to a receiving end at 508. In this case, the sending end / encoder can be the device 102 and the receiving end can be the device 104 and / or the device 106. The encoded audio data can be transmitted over a network, such as the real-time network (RTN) 412 in Figure 4 . Further, it should be noted that other steps can also be performed prior to encoding the audio data at 506. For example, the audio data can also be processed by a mixer similar to the mixer 408 in Figure 4 .

[0068] Prior to, during, or after transmitting the audio data at 508, the associated accompaniment (e.g., song) can be sent at 510. The associated accompaniment can also be sent to the receiving end so that both the associated accompaniment and the audio data from the encoder can be accessed by the receiving end. However, as described above, the associated accompaniment can be delivered to the receiving end over a separate network connection (e.g., over the content delivery network (CDN) 416) separate from the transmission of the audio data to reduce the bandwidth required on the RTN 412 used to deliver the recorded singing / audio.

[0069] After transmitting the audio data from the encoder (e.g., the sending end) and transmitting the associated accompaniment, the receiving end can process the audio data and the associated accompaniment at 512. For example, the receiving end can include a decision module (similar to the decision module 420 in Figure 4 . In this case, the receiving end can align the recorded audio / singing with the associated accompaniment, such as based on the timestamp, so that the recorded audio / singing and the associated accompaniment can be properly aligned for eventual playback of the audio to a viewer. However, the processing at 512 is not limited to any one process and can include any manipulation and / or alteration of the audio data and / or the associated accompaniment.

[0070] After processing at 512, the audio data (e.g., recorded audio / singing) and the associated accompaniment can be decoded using any of the above-mentioned decoding techniques at 514. The decoding at 514 can be completed by a decoder (similar to decoder 422). For example, the decoding at 514 can also include mixing the associated accompaniment with the recorded audio / singing to create a final mixed audio. After decoding is completed at 514, the audio data and the associated accompaniment (e.g., final mixed audio) can be played using one or more playback devices (e.g., microphones or headphones) at 516. Therefore, based on technology 500, the associated accompaniment and the recorded singing can be transmitted separately to the receiving end, but can be aligned to provide the audience with correct and improved audio for playback, while also reducing the pressure on the real-time network (RTN) used to transmit the recorded singing.

[0071] Figure 6 600 is a flowchart of another embodiment of the present invention for a method of real-time audio communication, for example, a method for real-time audio transmission of user voice segments. Figure 1 device 102) and / or a receiving end (e.g. Figure 1 The technique 600 may be implemented as a device stored in Figure 1 The software modules in the memory 112, the memory 120 and / or the memory 128 are respectively operable by Figure 1 Instructions and / or data executed by processor 110, processor 118, and / or processor 126. As another example, technique 600 may be implemented in hardware as a dedicated chip storing instructions executable by the dedicated chip.

[0072] At 602, a user's voice segment may be obtained by an encoder (e.g., a transmitting end). For example, the encoder may include a recording device, such as a microphone, for recording the user's voice (e.g., the user's singing). At 604, one or more servers in communication with the encoder obtain a reference sample associated with the user's voice segment. The reference sample may be an accompaniment associated with the user's singing recorded at 602. The reference sample may be, for example, Figure 4 414. Additionally, at 604, the reference samples may be stored locally on a decoder (eg, a receiving end) in communication with the encoder.

[0073] Additionally, at 606, an associated timestamp is created for the user's speech obtained at 602, where the timestamp is established relative to a reference sample. After the timestamp is created at 606, encoding is completed at 608. The encoding may be performed by an encoder (e.g., a transmitter), i.e., encoding an audio data packet containing the user's speech segment (e.g., a singing recording) and its associated timestamp.

[0074] The audio data packet is then transmitted to a decoder (e.g., a receiving end) in real time at 610. The user's voice segment (e.g., a singing recording) in the audio data packet can be fused with a reference sample (e.g., a related accompaniment) during decoding, and the fusion operation is based on the time stamp associated with the user's voice segment.

[0075] As described above, those skilled in the art should understand that all or part of the contents described herein can be implemented using a general-purpose computer or processor with a computer program, which can execute any corresponding techniques, algorithms and / or instructions described herein when executed.

[0076] The computing devices described herein, as well as the algorithms, methods, and instructions stored thereon and / or executed thereby, may be implemented using hardware, software, or any combination thereof. Hardware may include computers, intellectual property (IP) cores, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors, or any other suitable circuitry. In the claims, the term "processor" should be understood to include any of the foregoing, including any combination of one or more thereof.

[0077] The various functions described herein can be described using functional components and various processing operations. The processes and sequences described herein can be executed individually or in any combination. Functional modules can be implemented using any number of hardware and / or software components capable of performing specific functions. For example, the content described herein can utilize various integrated circuit components, such as memory elements, processing elements, logic elements, lookup tables, etc., which are capable of performing various functions under the control of one or more microprocessors or other control devices. Similarly, for content implemented through software programming or software programs, any programming or scripting language, such as C, C++, Java, or assembler, can be used, and any combination of data structures, objects, processes, routines, or other programming elements can be used to execute various algorithms. Each function can be implemented by executing algorithms on one or more processors. Furthermore, the functions described herein can utilize any number of conventional techniques for electronic configuration, signal processing and / or control, data processing, etc. The terms "mechanism" and "element" are used extensively herein, but do not necessarily imply limitation to mechanical or physical implementation, but can include software routines adapted for processor operation.

[0078] Embodiments or partial embodiments of the present invention may take the form of a computer program product that can be accessed through a computer-usable medium or a computer-readable medium. A computer-usable medium or a computer-readable medium may be any suitable device that specifically contains, stores, transmits, or transmits a program or data structure for use by or connected to any processor. The medium may be in the form of an electronic, magnetic, optical, electromagnetic, or semiconductor device, etc., and may also include other applicable media. The above-mentioned computer-usable medium or computer-readable medium may be referred to as a non-transient memory or medium, and may include RAM or other volatile memory or storage devices that may change over time. Unless otherwise specifically stated herein, the device memory described herein is not necessarily physically provided in the device, but can be accessed remotely by the device and does not have to be adjacent to other physically provided memories in the device.

[0079] One or more functions performed in embodiments of the present invention may be implemented as machine-readable instructions in code for operating one or more hardware combinations. Computational code may be implemented as one or more modules, which may be used to perform one or more of the combined functions as computational tools, with input and output data being transferred between each module and one or more other modules during operation of the methods and systems of the present invention.

[0080] The terms "signal" and "data" are used interchangeably herein. Furthermore, not all components of a computing device are necessarily implemented in the same manner. Information, data, and signals can be represented using a variety of different technologies and methods. For example, any data, instructions, commands, information, signals, bits, symbols, and chips mentioned herein can be represented by one or more combinations of voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or the like.

[0081] This disclosure uses the word "exemplary" to mean an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily the best way to implement the disclosure. The word "exemplary" is used to present concepts in a concrete way. Furthermore, the phrases "one aspect" or "an aspect" appear throughout this disclosure, but they do not necessarily refer to the same embodiment or functionality.

[0082] As used herein, the word "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, "X includes / contains A or B" is intended to mean any natural inclusive permutation, unless otherwise specified or the context clearly indicates otherwise. In other words, "X includes A or B" can mean any of the following: X includes A; X includes B; X includes A and B. Similarly, "X includes either A and B" means "X includes A or B." As used herein, the word "and / or" is intended to mean "and" or an inclusive "or." That is, "X includes A, B, and / or C" is intended to mean that X can include any combination of A, B, and C, unless otherwise specified or the context clearly indicates otherwise. In other words, if X includes A, X includes B, X includes C, X includes A and B, X includes B and C, X includes A and C, or X includes all of A, B, and C, then each or more of the above satisfies the description of "X includes A, B, and / or C." By analogy, “X includes at least one of A, B, and C” is equivalent to “X includes A, B, and / or C.”

[0083] The term "if" or "if" used herein may mean "when", "when..." or "assuming" depending on the context.

[0084] In the content of the present invention (especially the claims), "a", "an", "said" and "the" and similar demonstrative pronouns should be understood to include one and more singular and plural forms. In addition, the description of the numerical range in this article is only a simple way of description, which is intended to indicate each individual numerical value included in the range, and each individual value is incorporated into the range, which is equivalent to being listed separately in this article. Finally, the steps of all methods described herein can be performed in any suitable order, unless otherwise specified herein or clearly contradicted by the context. The use of examples or exemplary language (e.g., "such as") provided herein is intended to better illustrate the present invention and is not intended to limit the scope of the invention unless otherwise specified.

[0085] Various headings and subheadings are used herein to list enumerated items. These are included to enhance readability and simplify the process of finding and citing materials. These headings and subheadings are not intended to, and should not be used to, affect the interpretation of the claims or to limit the scope of the claims in any way. The specific embodiments shown and described herein are illustrative examples of the present invention and are not intended to limit the scope of the invention in any way.

[0086] All references (including publications, patent applications, and patents) cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and set forth in its entirety.

[0087] Although the present invention has been described in conjunction with certain embodiments and implementations, it should be understood that the present invention is not limited to the implementations disclosed herein, and the disclosure of the present invention is intended to cover various variations and equivalents within the scope of the claims, which should be given the broadest interpretation to cover all such variations and equivalents permitted by law.

Claims

1. A method for transmitting audio of a user's voice clip, comprising: receiving a user's voice clip via an encoder; obtaining, from one or more servers in communication with the encoder, reference samples associated with the user's speech segments; Creating a timestamp associated with the user's speech segment, wherein the timestamp refers to a time offset relative to the reference sample; encoding an audio data packet by an encoder, wherein the audio data packet includes the user's voice segment and the timestamp associated with the user's voice segment; as well as The audio data packet is transmitted to a decoder in real time, and during decoding, the user's voice segment in the audio data packet is fused with the reference sample based on the time stamp associated with the user's voice segment.

2. The method according to claim 1, further comprising: The reference samples associated with the user's speech segment are transmitted from one or more servers to a decoder via a content delivery network.

3. The method according to claim 2, wherein: Before transmitting the audio data packet to the decoder, the reference sample is transmitted to the decoder through a content delivery network, and the audio data packet is transmitted to the decoder in real time. The method of claim 3 , wherein the reference samples are also transmitted to an encoder. 5 . The method according to claim 1 , wherein the reference sample is an accompaniment data packet, and the user's voice segment is the user's singing data associated with the accompaniment data packet.

6. The method according to claim 1, wherein The step of fusing the user's voice segment with the reference sample according to the timestamp associated with the user's voice segment includes: The user's speech segment is aligned with the reference sample according to the timestamp that is time-offset relative to the beginning of the reference sample.

7. The method according to claim 1, wherein The step of fusing the user's voice segment with the reference sample according to the timestamp associated with the user's voice segment includes: The timestamp associated with the user's voice segment and the timestamp associated with the reference sample are aligned, wherein the value of the timestamp associated with the user's voice segment is equal to the value of the timestamp associated with the reference sample.

8. The method according to claim 1, further comprising: The user's voice clip is filtered to remove background audio obtained by the encoder.

9. A system for audio transmission of a user's speech segment, comprising: non-transient memory; as well as A processor configured to execute instructions stored in the non-transitory memory to perform the operations of the method of claim 1.

10. The system according to claim 9, wherein: The processor is further configured to execute instructions stored in the non-transitory memory for the following operations: The reference samples associated with the user's speech segment are transmitted from one or more servers to a decoder via a content delivery network.

11. The system of claim 10, wherein the reference samples are transmitted to the decoder via a content delivery network before the audio data packet is transmitted to the decoder, and the audio data packet is configured to be transmitted to the decoder via real-time communication.

12. The system according to claim 9, wherein: The step of fusing the user's voice segment with the reference sample based on the timestamp associated with the user's voice segment includes: The user's speech segment is aligned with the reference sample according to the timestamp that is time-offset relative to the start moment of the reference sample.

13. The system according to claim 9, wherein: The step of fusing the user's voice segment with the reference sample according to the timestamp associated with the user's voice segment includes: The timestamp associated with the voice segment and the timestamp associated with the reference sample are aligned, wherein the value of the timestamp associated with the user's voice segment is equal to the value of the timestamp associated with the reference sample.

14. The system according to claim 9, wherein: The processor is further configured to execute instructions stored in the non-transitory memory to: The user's voice clip is filtered to remove background audio obtained by the encoder.

15. A non-transitory computer-readable storage medium configured to store a computer program for audio transmission of a user voice segment, the computer program comprising instructions executable by a processor to perform the operations of the method of claim 1.

16. A method for transmitting audio of a user's voice segment, comprising: obtaining reference samples associated with the speech segment from one or more servers via a content delivery network; receiving an audio data packet from an encoder in real time, the audio data packet including a user's voice segment and a timestamp associated with the user's voice segment, the timestamp being a time offset relative to the reference sample; as well as After receiving the audio data packet from the encoder, the user's voice segment is fused with the reference sample according to the timestamp associated with the user's voice segment.

17. The method according to claim 16, wherein The user's voice segment is acquired by the encoder in real time.

18. The method according to claim 16, wherein The reference samples are also transmitted from the one or more servers to the encoder.

19. The method according to claim 16, wherein The step of fusing the user's voice segment with the reference sample according to the timestamp associated with the user's voice segment includes: The user's speech segment is aligned with the reference sample according to the timestamp that is time-offset relative to the start moment of the reference sample.

20. The method according to claim 16, wherein fusing the user's voice segment with the reference sample according to the timestamp associated with the user's voice segment comprises: The timestamp associated with the user's speech segment is aligned with the timestamp associated with the reference sample.

21. The method of claim 16, further comprising: Determining whether the audio data packet transmitted by the encoder is lost; as well as If it is determined that the audio data packet transmitted by the encoder is lost, the audio data packet is replaced with a substitute audio packet contained in the reference sample.

Images