Low-delay network audio transmission method and system

Abstract

The invention discloses a low-delay network audio transmission method and system, and relates to the technical field of audio transmission. The low-delay network audio transmission system comprises an audio acquisition module, an audio coding module, a network transmission module, an audio decoding module and an audio playing module; according to the method, an Opus coding algorithm is used for coding and dividing the audio frames into audio frames, and time domains, frequency domains and statistical characteristic parameters of the audio frames are extracted and then bundled into data packets for transmission; the receiving end performs integrity verification according to the characteristic parameters, during integrity verification, the characteristic parameters of the receiving end and the sending end are compared, such as an amplitude mean value, a variance, a frequency spectrum mean value, a variance and a peak amplitude, if the difference is within a threshold value or is compared with the threshold value through specific verification value calculation, the transmission is complete, and otherwise, the transmission is lost or damaged; the method and the system can reduce time delay, guarantee data accuracy, improve system operation efficiency and optimize coding to adapt to different network environments.

Description

Technical Field

[0001] The present invention relates to the technical field of audio transmission, and in particular to a low-latency network audio transmission method and system. Background Art

[0002] With the rapid development of Internet technology, network audio applications are becoming more and more widespread, such as online voice calls, live webcasts, remote conferences, etc. However, there are many challenges in the audio transmission process.

[0003] In scenarios with high real-time requirements, such as online game voice communication and remote real-time command, traditional audio transmission methods often have large delays. The network environment is complex and changeable. During the audio data transmission process, data packets may be lost or damaged due to network fluctuations, signal interference, etc. Once this happens, the audio signal recovered by the receiving end will have problems such as noise, freeze, or even completely unable to play, seriously affecting the audio quality.

[0004] Some existing audio transmission technologies have tried to reduce latency and ensure audio quality, but the results are limited. Some technologies reduce latency by optimizing network transmission protocols, but they cannot fundamentally solve the problem of time consumption in the audio encoding and decoding process; in terms of dealing with audio data loss or damage, some simple verification methods have low accuracy and cannot effectively detect slight data changes, resulting in quality problems in the restored audio.

[0005] Therefore, there is an urgent need for a network audio transmission method and system that can effectively reduce latency and ensure audio quality. Summary of the invention

[0006] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a low-latency network audio transmission method and system.

[0007] A low-latency network audio transmission method comprises the following steps:

[0008] Step 1: Audio Collection

[0009] Collect audio signals;

[0010] Step 2: Audio encoding

[0011] The digital audio signal is encoded using the Opus encoding algorithm, and a number of audio frames are obtained after the digital audio signal is segmented. Then, for each audio frame, the corresponding characteristic parameters are extracted and analyzed, and the several audio frames are respectively combined with the characteristic parameters to be bundled into several data packets;

[0012] Step 3: Network transmission

[0013] Transmit the data packet;

[0014] Step 4: Integrity Check

[0015] Perform integrity check on the transmitted data packets according to the characteristic parameters;

[0016] Step 5: Audio decoding

[0017] Adopt Opus decoding algorithm to restore all received data packets after integrity check into audio signals;

[0018] Step 6: Audio playback

[0019] Play the decoded audio signal;

[0020] During the acquisition process, it samples the audio signal at a fixed sampling frequency and converts the audio signal into a digital audio signal.

[0021] As a further solution of the present invention, the characteristic parameters corresponding to the audio frame include time domain characteristics, frequency domain characteristics and statistical characteristics;

[0022] As a further solution of the present invention, the audio frame is a segment of a specific duration into which a continuous audio signal is divided when the audio signal is digitally processed and analyzed, and each frame contains a specified number of audio sampling points;

[0023] As a further solution of the present invention, each data packet contains a specified amount of audio data and packet header information, and the packet header information includes a data packet sequence number, a timestamp and characteristic parameters, and the data packet sequence number is marked sequentially by the timestamp and in accordance with the time trend.

[0024] As a further solution of the present invention, the extraction and analysis method of the time domain features is as follows:

[0025] Select an audio frame;

[0026] First, obtain the amplitude value of each audio sampling point in the audio frame;

[0027] Next, the average value of the amplitude values ​​at all audio sampling points in the audio frame is calculated and recorded as the amplitude mean;

[0028] Then, combined with the amplitude mean, the variance of the amplitude values ​​at all audio sampling points in the audio frame is calculated and recorded as the amplitude variance.

[0029] As a further solution of the present invention, the extraction and analysis method of frequency domain features is as follows:

[0030] Select an audio frame;

[0031] First, perform a fast Fourier transform on the audio frame and obtain the frequency spectrum of the audio frame;

[0032] Next, the amplitude values ​​of the audio frame at multiple frequency points are obtained;

[0033] Then, the average value of the amplitude values ​​at multiple frequency points in the audio frame is calculated and recorded as the spectrum mean;

[0034] Afterwards, the variance of the amplitude values ​​at multiple frequency points in the audio frame is calculated in combination with the spectrum mean, and is recorded as the spectrum variance.

[0035] As a further solution of the present invention, the statistical features are extracted and analyzed as follows:

[0036] Select an audio frame;

[0037] First, obtain the amplitude value of each audio sampling point in the audio frame;

[0038] Then, among the amplitude values ​​of all the audio sampling points in the audio frame, a maximum amplitude value is selected and recorded as the peak amplitude.

[0039] As a further solution of the present invention, the integrity verification method is as follows:

[0040] Compare the characteristic parameters of the received encoded audio data with the characteristic parameters of the original audio data sent by the sender:

[0041] If the feature difference between the two is within a certain threshold range, the audio transmission is considered to be complete; otherwise, the audio transmission is considered to be lost or damaged.

[0042] As a further solution of the present invention, the specific comparison method is:

[0043] Extract the amplitude mean and amplitude variance in the time domain features, the spectrum mean and spectrum variance in the frequency domain features, and the peak amplitude in the statistical features;

[0044] The mean amplitude value in the selected time domain features;

[0045] The amplitude mean of the original audio data at the sending end is set to M1, the amplitude mean of the encoded audio data at the receiving end is set to M2, and the amplitude mean difference threshold is set to CM;

[0046] If |M1-M2|≤CM, the amplitude mean value check is passed, otherwise it means the amplitude mean value check fails;

[0047] According to the comparison method of amplitude mean, the amplitude variance in time domain features, the spectrum mean and spectrum variance in frequency domain features, and the peak amplitude in statistical features are compared and verified;

[0048] In the data packet, the amplitude mean and amplitude variance in the time domain features, the spectrum mean and spectrum variance in the frequency domain features, and the peak amplitude in the statistical features are compared;

[0049] If at least one of the check results fails, it is considered that the audio transmission is lost or damaged.

[0050] As a further solution of the present invention, the integrity verification method is also as follows:

[0051] Extract the characteristic parameters of the received encoded audio data and the characteristic parameters of the original audio data sent by the sender:

[0052] Then, the amplitude mean, amplitude variance, spectrum mean, spectrum variance, and peak amplitude of the audio frame corresponding to the transmitting end are marked as M1, V1, FM1, FV1, and P1, respectively, and the amplitude mean, amplitude variance, spectrum mean, spectrum variance, and peak amplitude of the audio frame corresponding to the receiving end are marked as M2, V2, FM2, FV2, and P2, respectively;

[0053] Then through:

[0054] Calculate the check value G;

[0055] Then the check value G is compared with the preset check threshold Gy:

[0056] If G>Gy, it is determined that the transmitted audio signal is complete;

[0057] If G≤Gy, it is determined that the transmitted audio signal is lost or damaged.

[0058] A low-latency network audio transmission system, the system is used to perform a low-latency network audio transmission method, the system comprising:

[0059] Audio acquisition module: used to collect audio signals and convert them into audio digital audio signals;

[0060] Audio encoding module: used to encode the collected audio digital audio signal;

[0061] Network transmission module: used to transmit the encoded audio data to the receiving end through the network;

[0062] Audio decoding module: used to decode the received encoded audio data and restore it to the original audio digital audio signal;

[0063] Audio playback module: used to convert the decoded audio digital audio signal into an analog signal and play it out.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] Low latency: After the audio signal is quickly collected and converted into a digital signal, Opus encoding is immediately performed to split it into audio frames and bundled into data packets for transmission based on characteristic parameters, which reduces the waiting time for signal processing. At the same time, the receiving end can quickly perform integrity verification and decoding and playback based on the characteristic parameters, greatly reducing the delay in the audio transmission process, making the audio transmission more real-time, and suitable for scenarios with high real-time requirements, such as online meetings, real-time voice communications, etc.

[0066] Data accuracy: The time domain characteristics, frequency domain characteristics and statistical characteristics of the audio frame are used for integrity verification. Whether it is by directly comparing the characteristic parameters of the receiving end and the sending end, or by calculating the verification value and comparing it with the threshold, it can accurately determine whether the audio data is complete during the transmission process. Once loss or damage occurs, it can be discovered and measures can be taken in time, effectively ensuring the accuracy of the audio data, allowing users to receive accurate audio information.

[0067] System efficiency: The audio acquisition module, encoding module, transmission module, decoding module and playback module work together to form a complete and efficient audio transmission system. Each module has a clear division of labor and cooperates closely with each other. There is a complete processing flow from the audio acquisition source to the final playback, which makes the entire audio transmission system run smoothly and efficiently, and improves the overall performance and stability of the system.

[0068] Coding advantages: When encoding digital audio signals, the Opus coding algorithm can effectively reduce the amount of data, reduce network transmission pressure, increase transmission speed, and ensure audio quality. In the case of limited network bandwidth, high-quality audio transmission can still be achieved, and it can adapt to audio transmission requirements in different network environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A schematic diagram of a process of a low-latency network audio transmission method of the present invention;

[0070] Figure 2 The system block diagram of a low-latency network audio transmission system of the present invention. DETAILED DESCRIPTION

[0071] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] Embodiment 1

[0073] See also Figure 1 As shown, the present invention is a low-latency network audio transmission method, comprising the following steps:

[0074] Step 1: Audio Collection

[0075] Collect audio signals;

[0076] During the acquisition process, it samples the audio signal at a fixed sampling frequency and converts the audio signal into a digital audio signal;

[0077] In this embodiment, the sampling frequency can be set according to actual needs, generally 8kHz, 16kHz, 44.1kHz or 48kHz;

[0078] Step 2: Audio encoding

[0079] Encode digital audio signals;

[0080] During encoding:

[0081] First, the digital audio signal is divided into several audio frames using the Opus encoding algorithm;

[0082] Among them, an audio frame is to divide a continuous audio signal into segments of specific duration when digitally processing and analyzing the audio signal, and each frame contains a specified number of audio sampling points;

[0083] Then, for each audio frame, the corresponding feature parameters are extracted and analyzed, and the feature parameters include time domain features, frequency domain features and statistical features;

[0084] Then, several audio frames are bundled into several data packets in combination with characteristic parameters;

[0085] Each data packet contains a specified amount of audio data and packet header information, and the packet header information includes a data packet sequence number, a timestamp, and characteristic parameters, and the data packet sequence number is marked in sequence according to the time trend through the timestamp;

[0086] Herein, an audio frame is selected as an example;

[0087] The extraction and analysis method of time domain features is as follows:

[0088] First, obtain the amplitude value of each audio sampling point in the audio frame and mark it as X i , i=1, 2, ... n, n represents the number of audio sampling points in the audio frame;

[0089] Next, through Calculate the amplitude mean M of all audio sampling points in the audio frame;

[0090] Subsequently, through Calculate the amplitude variance V of all audio sampling points in the audio frame;

[0091] The extraction and analysis method of frequency domain features is as follows:

[0092] First, perform a fast Fourier transform on the audio frame and obtain the frequency spectrum of the audio frame;

[0093] Next, obtain the amplitude values ​​of the audio frame at multiple frequency points and mark them as X j , j = 1, 2, ... m, m represents the number of each frequency point in the audio frame;

[0094] Subsequently, through Calculate the frequency spectrum mean FM of multiple frequency points in the audio frame;

[0095] Afterwards, through Calculate the frequency spectrum variance FV of multiple frequency points in the audio frame;

[0096] The statistical features are extracted and analyzed as follows:

[0097] First, obtain the amplitude value X of each audio sampling point in the audio frame i ;

[0098] pass Calculate the peak amplitude P of all audio sampling points in the audio frame;

[0099] Step 3: Network transmission

[0100] Transmit the data packet;

[0101] Step 4: Integrity check

[0102] Perform integrity check on the transmitted data packets according to the characteristic parameters;

[0103] The integrity check method is as follows:

[0104] Compare the characteristic parameters of the received encoded audio data with the characteristic parameters of the original audio data sent by the sender:

[0105] If the difference between the two features is within a certain threshold, the audio transmission is considered to be complete; otherwise, the audio transmission is considered to be lost or damaged;

[0106] The specific comparison method is:

[0107] Extract the amplitude mean M and amplitude variance V in the time domain features, the spectrum mean FM and spectrum variance FV in the frequency domain features, and the peak amplitude P in the statistical features;

[0108] Take the amplitude mean M in the time domain feature as an example;

[0109] The amplitude mean of the original audio data at the sending end is set to M1, the amplitude mean of the encoded audio data at the receiving end is set to M2, and the amplitude mean difference threshold is set to CM;

[0110] If |M1-M2|≤CM, the amplitude mean value check is passed, otherwise it means the amplitude mean value check fails;

[0111] According to the comparison method of amplitude mean, the amplitude variance V in the time domain feature, the spectrum mean FM and spectrum variance FV in the frequency domain feature, and the peak amplitude P in the statistical feature are compared and checked;

[0112] In the data packet, the amplitude mean M and amplitude variance V in the time domain features, the spectrum mean FM and spectrum variance FV in the frequency domain features, and the peak amplitude P in the statistical features are compared;

[0113] If at least one of the verification results fails, it is considered that the audio transmission is lost or damaged;

[0114] Step 5: Audio decoding

[0115] Adopt Opus decoding algorithm to restore all received data packets after integrity check into audio signals;

[0116] Step 6: Audio playback

[0117] Play the decoded audio signal.

[0118] This embodiment performs sampling at a fixed sampling frequency during the audio acquisition process, and the sampling frequency can be set according to actual needs. It is flexible and can adapt to different audio processing scenarios. The Opus encoding algorithm is used to divide the digital audio signal into audio frames, and the time domain features, frequency domain features and statistical features are extracted, which can fully describe the characteristics of the audio signal and provide richer information for subsequent processing and transmission. The data packet contains a specified amount of audio data and packet header information, which is convenient for management and identification during the transmission process, and also provides a basis for integrity verification. By performing integrity verification on the transmitted data packet, the loss or damage during the audio transmission process can be discovered in time to ensure the quality of audio transmission. The Opus decoding algorithm is used to restore the received integrity-verified data packet into an audio signal, ensuring the accuracy and efficiency of decoding.

[0119] Embodiment 2

[0120] As the second embodiment of the present invention, when the present application is implemented, compared with the first embodiment, the technical solution of the present embodiment is different from that of the first embodiment only in that,

[0121] The integrity check method is as follows:

[0122] Extract the characteristic parameters of the received encoded audio data and the characteristic parameters of the original audio data sent by the sender:

[0123] Then, the amplitude mean, amplitude variance, spectrum mean, spectrum variance, and peak amplitude of the audio frame corresponding to the transmitting end are marked as M1, V1, FM1, FV1, and P1, respectively, and the amplitude mean, amplitude variance, spectrum mean, spectrum variance, and peak amplitude of the audio frame corresponding to the receiving end are marked as M2, V2, FM2, FV2, and P2, respectively;

[0124] Then through:

[0125] Calculate the check value G;

[0126] Then the check value G is compared with the preset check threshold Gy:

[0127] If G>Gy, it is determined that the transmitted audio signal is complete;

[0128] If G≤Gy, it is determined that the transmitted audio signal is lost or damaged;

[0129] This embodiment adds a new integrity check method based on the first embodiment. By calculating the check value and comparing it with the preset check threshold, the integrity of the audio transmission can be judged more accurately; this new check method can complement the check method of the first embodiment and improve the reliability of the audio transmission integrity check.

[0130] Embodiment 3

[0131] As the third embodiment of the present invention, when the present application is implemented, compared with the first and second embodiments, the technical solution of this embodiment is to combine the solutions of the first and second embodiments, and the difference between the technical solution of this embodiment and the first and second embodiments is only in this embodiment;

[0132] This embodiment combines the solutions of embodiment one and embodiment two, integrates the advantages of the two solutions, and provides a more comprehensive and reliable low-latency network audio transmission method and system; through the combination of multiple integrity verification methods, problems in the audio transmission process can be more effectively detected, and the quality and stability of audio transmission can be further improved; it provides users with better audio services and meets the audio transmission needs of different users in different scenarios.

[0133] See also Figure 2 As shown, the present invention also provides a low-latency network audio transmission system, which is used to perform a low-latency network audio transmission method, and the system includes:

[0134] Audio acquisition module: used to collect audio signals and convert them into audio digital audio signals;

[0135] Audio encoding module: used to encode the collected audio digital audio signal to reduce the amount of data and facilitate network transmission;

[0136] Network transmission module: used to transmit the encoded audio data to the receiving end through the network;

[0137] Audio decoding module: used to decode the received encoded audio data and restore it to the original audio digital audio signal;

[0138] Audio playback module: used to convert the decoded audio digital audio signal into an analog signal and play it out.

[0139] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0140] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A low-latency network audio transmission method, characterized in that: The following steps are involved: The first step is to collect audio signals and convert them into digital audio signals; The second step is to encode the digital audio signal using the Opus encoding algorithm, and obtain a number of audio frames after the digital audio signal is segmented. Then, for each audio frame, the corresponding characteristic parameters are extracted and analyzed, and the audio frames are respectively combined with the characteristic parameters to be bundled into a number of data packets, and then the data packets are transmitted; The third step is to perform integrity check on the transmitted data packets according to the characteristic parameters, and then use the Opus decoding algorithm to restore the data packets that pass the integrity check into audio signals, and then play the decoded audio signals.

2. A low-latency network audio transmission method according to claim 1, characterized in that: in, The characteristic parameters corresponding to the audio frame include time domain features, frequency domain features and statistical features.

3. A low-latency network audio transmission method according to claim 2, characterized in that: The extraction and analysis method of time domain features is as follows: Select an audio frame; First, obtain the amplitude value of each audio sampling point in the audio frame; Next, the average value of the amplitude values ​​at all audio sampling points in the audio frame is calculated and recorded as the amplitude mean; Then, combined with the amplitude mean, the variance of the amplitude values ​​at all audio sampling points in the audio frame is calculated and recorded as the amplitude variance.

4. A low-latency network audio transmission method according to claim 3, characterized in that: The extraction and analysis method of frequency domain features is as follows: Select an audio frame; First, perform a fast Fourier transform on the audio frame and obtain the frequency spectrum of the audio frame; Next, the amplitude values ​​of the audio frame at multiple frequency points are obtained; Then, the average value of the amplitude values ​​at multiple frequency points in the audio frame is calculated and recorded as the spectrum mean; Afterwards, the variance of the amplitude values ​​at multiple frequency points in the audio frame is calculated in combination with the spectrum mean, and is recorded as the spectrum variance.

5. A low-latency network audio transmission method according to claim 4, characterized in that: The statistical features are extracted and analyzed as follows: Select an audio frame; First, obtain the amplitude value of each audio sampling point in the audio frame; Then, among the amplitude values ​​of all the audio sampling points in the audio frame, a maximum amplitude value is selected and recorded as the peak amplitude.

6. A low-latency network audio transmission method according to claim 5, characterized in that: Each data packet contains a specified amount of audio data and packet header information, and the packet header information includes a data packet sequence number, a timestamp, and characteristic parameters, and the data packet sequence number is marked in sequence by the timestamp and in accordance with the time trend.

7. A low-latency network audio transmission method according to claim 6, characterized in that: The integrity check method is as follows: Compare the characteristic parameters of the received encoded audio data with the characteristic parameters of the original audio data sent by the sender: If the difference between the two characteristics is within a certain threshold range, it means that the audio transmission is complete; otherwise, it means that the audio transmission is lost or damaged.

8. A low-latency network audio transmission method according to claim 7, characterized in that: The specific comparison method is: Extract the amplitude mean and amplitude variance in the time domain features, the spectrum mean and spectrum variance in the frequency domain features, and the peak amplitude in the statistical features; The mean amplitude value in the selected time domain features; The amplitude mean of the original audio data at the sending end is set to M1, the amplitude mean of the encoded audio data at the receiving end is set to M2, and the amplitude mean difference threshold is set to CM; If |M1-M2|≤CM, the amplitude mean value check is passed, otherwise it means the amplitude mean value check fails; According to the comparison method of amplitude mean, the amplitude variance in time domain features, the spectrum mean and spectrum variance in frequency domain features, and the peak amplitude in statistical features are compared and verified; In the data packet, the amplitude mean and amplitude variance in the time domain features, the spectrum mean and spectrum variance in the frequency domain features, and the peak amplitude in the statistical features are compared; If at least one of the check results fails, it is considered that the audio transmission is lost or damaged.

9. A low-latency network audio transmission method according to claim 6, characterized in that: The integrity check method is also as follows: Extract the characteristic parameters of the received encoded audio data and the characteristic parameters of the original audio data sent by the sender: Then, the amplitude mean, amplitude variance, spectrum mean, spectrum variance, and peak amplitude of the audio frame corresponding to the transmitting end are marked as M1, V1, FM1, FV1, and P1, respectively, and the amplitude mean, amplitude variance, spectrum mean, spectrum variance, and peak amplitude of the audio frame corresponding to the receiving end are marked as M2, V2, FM2, FV2, and P2, respectively; Then through: Calculate the check value G; Then the check value G is compared with the preset check threshold Gy: If G>Gy, it is determined that the transmitted audio signal is complete; If G≤Gy, it is determined that the transmitted audio signal is lost or damaged.

10. A low-latency network audio transmission system, the method being used to execute a low-latency network audio transmission method according to any one of claims 1 to 9, characterized in that: The system includes: Audio acquisition module: used to collect audio signals and convert them into audio digital audio signals; Audio encoding module: used to encode the collected audio digital audio signal; Network transmission module: used to transmit the encoded audio data to the receiving end through the network; Audio decoding module: used to decode the received encoded audio data and restore it to the original audio digital audio signal; Audio playback module: used to convert the decoded audio digital audio signal into an analog signal and play it out.

Images