Data transmission method, system, device, computer readable storage medium and equipment

Abstract

The present application provides a data transmission method, a data transmission system, a data transmission device, a computer-readable storage medium and an electronic device; it relates to the field of computer technology, and the data transmission method includes: obtaining the loudness corresponding to the target audio data packet; calculating the target error correction code of the target audio data packet according to the loudness corresponding to the target audio data packet; transmitting the target audio data packet and the target error correction code to the receiving terminal. It can be seen that the implementation of the technical solution of the present application can improve the bandwidth utilization rate and improve the problem of wasting network resources.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data transmission method, a data transmission system, a data transmission device, a computer-readable storage medium, and an electronic device. Background Art

[0002] During the transmission of audio data, packet loss usually occurs due to reasons such as unstable transmission network. In order to solve the packet loss problem, forward error correction technology is generally used to calculate the error correction code of the transmitted audio data, so that the receiver can use the error correction code to perform data error correction on the audio data, thereby ensuring that the receiver can obtain the complete audio data transmitted. However, during the real-time audio data transmission process, the following situation usually exists: the audio data in the audio data packet (for example, slight ambient sound) may not be perceived by the human ear after being decoded and output. If the error correction code is also calculated for such data in the above manner, it is easy to cause the problem of low bandwidth utilization during the data transmission process.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the present application is to provide a data transmission method, a data transmission system, a data transmission device, a computer-readable storage medium and an electronic device, which can improve bandwidth utilization and improve the problem of waste of network resources.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to one aspect of the present application, a data transmission method is provided, comprising:

[0007] Get the loudness corresponding to the target audio data packet;

[0008] Calculate a target error correction code for the target audio data packet according to the loudness corresponding to the target audio data packet;

[0009] The target audio data packet and the target error correction code are transmitted to the receiving terminal.

[0010] In an exemplary embodiment of the present application, before obtaining the loudness corresponding to the target audio data packet, the method further includes:

[0011] A target audio data packet whose audio characteristics meet preset conditions is screened from the received multiple audio data packets.

[0012] In an exemplary embodiment of the present application, the audio data packet includes the loudness corresponding to the audio data packet, the audio bit stream and the audio features corresponding to the audio bit stream, and the audio features corresponding to the audio bit stream include the energy distribution corresponding to the audio bit stream and the energy amplitude corresponding to each frequency point in the audio bit stream.

[0013] In an exemplary embodiment of the present application, before screening a target audio data packet whose audio features meet a preset condition from the received multiple audio data packets, the method further includes:

[0014] Obtain error correction codes corresponding to multiple audio data packets respectively;

[0015] Perform packet loss detection on multiple audio data packets according to the error correction code to obtain packet loss rates corresponding to the multiple audio data packets respectively;

[0016] The packet loss rate is fed back to the sending terminal and the errors of multiple audio data packets are corrected according to the error correction code.

[0017] In an exemplary embodiment of the present application, the received multiple audio data packets are sent by a sending terminal;

[0018] The specific method for the sending terminal to send multiple audio data packets is as follows:

[0019] The sending terminal collects the audio signal and extracts the features of the audio signal to obtain the audio features;

[0020] The sending terminal encodes the audio signal to obtain an audio code stream;

[0021] The sending terminal packages the audio code stream and audio features into an audio data packet and sends it to the server.

[0022] In an exemplary embodiment of the present application, the preset condition includes a preset energy amplitude and / or a preset signal-to-noise ratio, and screening a target audio data packet whose audio characteristics meet the preset condition from the received multiple audio data packets includes:

[0023] If it is detected that there is at least one energy amplitude greater than a preset energy amplitude in the audio features of the audio code stream, the audio data packet to which the audio code stream belongs is determined as the target audio data packet; and / or,

[0024] If it is detected that at least one signal-to-noise ratio greater than a preset signal-to-noise ratio exists in the audio features of the audio code stream, the audio data packet to which the audio code stream belongs is determined as the target audio data packet.

[0025] In an exemplary embodiment of the present application, the method further includes:

[0026] The sending terminal performs frame processing on the audio code stream according to a preset duration to obtain multiple audio frames;

[0027] The sending terminal processes the multiple audio frames respectively through a preset window function to obtain multiple reference frames;

[0028] The transmitting terminal calculates the power spectra corresponding to the multiple reference frames respectively;

[0029] The sending terminal calculates the loudness corresponding to the audio data packet based on the power spectrum.

[0030] In an exemplary embodiment of the present application, the preset window function is a Hanning window function, a Hamming window function, a Blackman window function, a Kaiser window function, a triangular window function or a rectangular window function.

[0031] In an exemplary embodiment of the present application, the sending terminal calculates the loudness corresponding to the audio data packet according to the power spectrum, including:

[0032] The transmitting terminal calculates the frequency point loudness of each frequency point in the power spectrum according to the energy amplitude of each frequency point in the power spectrum;

[0033] The sending terminal calculates the loudness weight of each frequency point in the power spectrum according to the frequency point loudness;

[0034] The transmitting terminal calculates the weighted sum of the energy amplitude of each frequency point in the power spectrum and the loudness weight of each frequency point in the power spectrum as the loudness value of the reference frame corresponding to the power spectrum;

[0035] The transmitting terminal determines the sum of the loudness values ​​of the multiple reference frames as the loudness corresponding to the audio data packet.

[0036] In an exemplary embodiment of the present application, calculating a target error correction code of a target audio data packet according to the loudness corresponding to the target audio data packet includes:

[0037] Determine the reference redundancy according to the packet loss rate fed back by the receiving terminal; wherein the packet loss rate corresponds to the historical unit time closest to the sending time of the target audio data packet;

[0038] Calculate the target redundancy corresponding to the target audio data packet according to the reference redundancy and the loudness corresponding to the target audio data packet;

[0039] A target error correction code for a target audio data packet is calculated according to the target redundancy.

[0040] In an exemplary embodiment of the present application, calculating a target redundancy corresponding to a target audio data packet according to a reference redundancy and a loudness corresponding to a target audio data packet includes:

[0041] The loudness corresponding to the target audio data packet is used as the input of a preset function expression to calculate the output value;

[0042] The product of the output value and the reference redundancy is determined as the target redundancy of the target audio data packet.

[0043] In an exemplary embodiment of the present application, transmitting a target audio data packet and a target error correction code to a receiving terminal includes:

[0044] The target audio data packet and the target error correction code are packaged as a data packet to be transmitted and transmitted to the receiving terminal, so that after decoding the data packet to be transmitted, the receiving terminal detects the packet loss corresponding to the target audio data packet according to the target error correction code in the decoding result and corrects the target audio data packet according to the packet loss situation.

[0045] In an exemplary embodiment of the present application, after decoding the data packet to be transmitted, the receiving terminal detects the packet loss corresponding to the target audio data packet according to the target error correction code in the decoding result and corrects the target audio data packet according to the packet loss situation in a specific manner:

[0046] The receiving terminal decodes the data packet to be transmitted to obtain the audio code stream and the target error correction code in the target audio data packet;

[0047] The receiving terminal detects the missing code of the audio code stream in the target audio data packet according to the target error correction code;

[0048] The receiving terminal restores the lost code according to the target error correction code to achieve error correction for the target audio data packet.

[0049] In an exemplary embodiment of the present application, after the receiving terminal restores the lost code according to the target error correction code to correct the error of the target audio data packet, the method further includes:

[0050] The receiving terminal mixes all error-corrected target audio data packets to obtain a mixed signal and plays it.

[0051] According to one aspect of the present application, a data transmission system is provided, including a sender terminal, a server, and a receiver terminal, wherein:

[0052] The sending terminal is used to send a target audio data packet to the server, where the target audio data packet includes a loudness corresponding to the target audio data packet;

[0053] The server is used to receive a target audio data packet and obtain the loudness corresponding to the target audio data packet; calculate a target error correction code for the target audio data packet according to the loudness corresponding to the target audio data packet; and transmit the target audio data packet and the target error correction code to a receiving terminal;

[0054] The receiving terminal is used to detect the packet loss situation corresponding to the target audio data packet according to the target error correction code.

[0055] According to one aspect of the present application, there is provided a data transmission device, including:

[0056] A loudness acquisition unit, used to acquire the loudness corresponding to the target audio data packet;

[0057] an error correction code calculation unit, used to calculate a target error correction code for a target audio data packet according to a loudness corresponding to the target audio data packet;

[0058] The data sending unit is used to transmit the target audio data packet and the target error correction code to the receiving terminal.

[0059] In an exemplary embodiment of the present application, the above-mentioned device further includes:

[0060] The data screening unit is used to screen a target audio data packet whose audio characteristics meet a preset condition from the received multiple audio data packets before the loudness obtaining unit obtains the loudness corresponding to the target audio data packet.

[0061] In an exemplary embodiment of the present application, the audio data packet includes the loudness corresponding to the audio data packet, the audio bit stream and the audio features corresponding to the audio bit stream, and the audio features corresponding to the audio bit stream include the energy distribution corresponding to the audio bit stream and the energy amplitude corresponding to each frequency point in the audio bit stream.

[0062] In an exemplary embodiment of the present application, the above-mentioned device further includes:

[0063] An error correction code acquisition unit, used for acquiring error correction codes corresponding to the plurality of audio data packets respectively before the data screening unit screens the target audio data packets whose audio features meet the preset conditions from the received plurality of audio data packets;

[0064] A packet loss detection unit, configured to perform packet loss detection on a plurality of audio data packets according to an error correction code, and obtain packet loss rates corresponding to the plurality of audio data packets respectively;

[0065] The data error correction unit is used to feed back the packet loss rate to the sending terminal and perform error correction on multiple audio data packets according to the error correction code.

[0066] In an exemplary embodiment of the present application, the received multiple audio data packets are sent by a sending terminal;

[0067] The specific method for the sending terminal to send multiple audio data packets is as follows:

[0068] The sending terminal collects the audio signal and extracts the features of the audio signal to obtain the audio features;

[0069] The sending terminal encodes the audio signal to obtain an audio code stream;

[0070] The sending terminal packages the audio code stream and audio features into an audio data packet and sends it to the server.

[0071] In an exemplary embodiment of the present application, the preset condition includes a preset energy amplitude and / or a preset signal-to-noise ratio, and the data screening unit screens a target audio data packet whose audio characteristics meet the preset condition from the received multiple audio data packets, including:

[0072] If it is detected that there is at least one energy amplitude greater than a preset energy amplitude in the audio features of the audio code stream, the audio data packet to which the audio code stream belongs is determined as the target audio data packet; and / or,

[0073] If it is detected that at least one signal-to-noise ratio greater than a preset signal-to-noise ratio exists in the audio features of the audio code stream, the audio data packet to which the audio code stream belongs is determined as the target audio data packet.

[0074] In an exemplary embodiment of the present application, it further includes:

[0075] The sending terminal performs frame processing on the audio code stream according to a preset duration to obtain multiple audio frames;

[0076] The sending terminal processes the multiple audio frames respectively through a preset window function to obtain multiple reference frames;

[0077] The transmitting terminal calculates the power spectra corresponding to the multiple reference frames respectively;

[0078] The sending terminal calculates the loudness corresponding to the audio data packet based on the power spectrum.

[0079] In an exemplary embodiment of the present application, the preset window function is a Hanning window function, a Hamming window function, a Blackman window function, a Kaiser window function, a triangular window function or a rectangular window function.

[0080] In an exemplary embodiment of the present application, the sending terminal calculates the loudness corresponding to the audio data packet according to the power spectrum, including:

[0081] The transmitting terminal calculates the frequency point loudness of each frequency point in the power spectrum according to the energy amplitude of each frequency point in the power spectrum;

[0082] The sending terminal calculates the loudness weight of each frequency point in the power spectrum according to the frequency point loudness;

[0083] The transmitting terminal calculates the weighted sum of the energy amplitude of each frequency point in the power spectrum and the loudness weight of each frequency point in the power spectrum as the loudness value of the reference frame corresponding to the power spectrum;

[0084] The transmitting terminal determines the sum of the loudness values ​​of the multiple reference frames as the loudness corresponding to the audio data packet.

[0085] In an exemplary embodiment of the present application, the error correction code calculation unit calculates a target error correction code of a target audio data packet according to the loudness corresponding to the target audio data packet, including:

[0086] Determine the reference redundancy according to the packet loss rate fed back by the receiving terminal; wherein the packet loss rate corresponds to the historical unit time closest to the sending time of the target audio data packet;

[0087] Calculate the target redundancy corresponding to the target audio data packet according to the reference redundancy and the loudness corresponding to the target audio data packet;

[0088] A target error correction code for a target audio data packet is calculated according to the target redundancy.

[0089] In an exemplary embodiment of the present application, the error correction code calculation unit calculates the target redundancy corresponding to the target audio data packet according to the reference redundancy and the loudness corresponding to the target audio data packet, including:

[0090] The loudness corresponding to the target audio data packet is used as the input of a preset function expression to calculate the output value;

[0091] The product of the output value and the reference redundancy is determined as the target redundancy of the target audio data packet.

[0092] In an exemplary embodiment of the present application, a data sending unit transmits a target audio data packet and a target error correction code to a receiving terminal, including:

[0093] The target audio data packet and the target error correction code are packaged as a data packet to be transmitted and transmitted to the receiving terminal, so that after decoding the data packet to be transmitted, the receiving terminal detects the packet loss corresponding to the target audio data packet according to the target error correction code in the decoding result and corrects the target audio data packet according to the packet loss situation.

[0094] In an exemplary embodiment of the present application, after decoding the data packet to be transmitted, the receiving terminal detects the packet loss corresponding to the target audio data packet according to the target error correction code in the decoding result and corrects the target audio data packet according to the packet loss situation in a specific manner:

[0095] The receiving terminal decodes the data packet to be transmitted to obtain the audio code stream and the target error correction code in the target audio data packet;

[0096] The receiving terminal detects the missing code of the audio code stream in the target audio data packet according to the target error correction code;

[0097] The receiving terminal restores the lost code according to the target error correction code to achieve error correction for the target audio data packet.

[0098] In an exemplary embodiment of the present application, after the receiving terminal restores the lost code according to the target error correction code to correct the error of the target audio data packet, the method further includes:

[0099] The receiving terminal mixes all the error-corrected target audio data packets to obtain a mixed signal and plays it.

[0100] According to one aspect of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned methods by executing the executable instructions.

[0101] According to one aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any one of the methods described above is implemented.

[0102] The exemplary embodiments of the present application may have some or all of the following beneficial effects:

[0103] In the data transmission method provided in an example implementation of the present application, the loudness corresponding to the target audio data packet can be obtained; the target error correction code of the target audio data packet is calculated according to the loudness corresponding to the target audio data packet; and the target audio data packet and the target error correction code are transmitted to the receiving terminal. According to the above scheme description, on the one hand, the present application can calculate the corresponding target error correction code according to the loudness corresponding to the target audio data packet, rather than calculating the error correction code for all audio data packets, so as to improve the bandwidth utilization; on the other hand, compared with the prior art of calculating the error correction codes of all audio data packets and transmitting all audio data packets and their corresponding error correction codes, it can realize the transmission of valid data, thereby improving the problem of wasting network resources.

[0104] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0106] Figure 1 A schematic diagram showing an exemplary system architecture of a data transmission method and a data transmission device to which the embodiments of the present application can be applied;

[0107] Figure 2A schematic diagram showing the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown;

[0108] Figure 3 The following schematically shows an architecture diagram of a data transmission method based on an embodiment of the present application;

[0109] Figure 4 A flowchart of a data transmission method according to an embodiment of the present application is schematically shown;

[0110] Figure 5 The figure schematically shows an acoustic equal loudness curve diagram according to an embodiment of the present application;

[0111] Figure 6 Schematically shows a loudness weight curve diagram according to an embodiment of the present application;

[0112] Figure 7 The structure of an FEC packet according to an embodiment of the present application is schematically shown;

[0113] Figure 8 The structure of the FEC header according to an embodiment of the present application is schematically shown;

[0114] Fig. 9 The structure block diagram of a data transmission system according to an embodiment of the present application is schematically shown;

[0115] Fig.10 The structure block diagram of a data transmission system according to another embodiment of the present application is schematically shown;

[0116] Fig.11 The structure block diagram of a data transmission system according to another embodiment of the present application is schematically shown;

[0117] Fig.12 The structural block diagram of a data transmission device according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0118] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present application will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical solutions of the present application may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present application.

[0119] In addition, the accompanying drawings are only schematic illustrations of the present application and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0120] Figure 1 A schematic diagram of a system architecture of an exemplary application environment in which a data transmission method and a data transmission device according to an embodiment of the present application can be applied is shown.

[0121] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with display screens, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The numbers of terminal devices, networks and servers are merely illustrative.

[0122] Specifically, the data transmission method provided in the embodiment of the present application is generally executed by the server 105, and accordingly, the data transmission device is generally arranged in the server 105. However, it is easy for those skilled in the art to understand that the data transmission method provided in the embodiment of the present application can also be executed by the terminal device 101, 102 or 103, and accordingly, the data transmission device can also be arranged in the terminal device 101, 102 or 103, and this is not particularly limited in the present exemplary embodiment. For example, in an exemplary embodiment, the server 105 can obtain the loudness corresponding to the target audio data packet; calculate the target error correction code of the target audio data packet according to the loudness corresponding to the target audio data packet; and transmit the target audio data packet and the target error correction code to the receiving terminal. According to the implementation requirements, there can be any number of terminal devices, networks and servers.

[0123] Figure 2 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0124] It should be noted that Figure 2 The computer system 200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0125] like Figure 2 As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage part 208 into a random access memory (RAM) 203. Various programs and data required for system operation are also stored in the RAM 203. The CPU 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0126] The following components are connected to the I / O interface 205: an input section 206 including a keyboard, a mouse, etc.; an output section 207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the I / O interface 205 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed, so that a computer program read therefrom is installed into the storage section 208 as needed.

[0127] In particular, according to an embodiment of the present application, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 209, and / or installed from a removable medium 211. When the computer program is executed by the central processing unit (CPU) 201, various functions defined in the method and apparatus of the present application are executed.

[0128] Generally speaking, it is difficult to avoid packet loss during audio transmission. The reasons for packet loss are usually: wireless channel interference of WIFI or mobile network, router congestion during peak hours, insufficient performance of mobile devices, etc. When an audio data packet takes too long to be transmitted on the network, that is, it cannot be transmitted in time when it needs to be played, it will be judged as packet loss even if it is received later. The existing packet loss processing method is generally: the receiving end performs packet loss detection on the data transmitted by the sending end when it receives it. If there is packet loss, it will feedback to the sending end to make it resend the data until it detects that there is no packet loss in the received data.

[0129] See also Figure 3 , Figure 3 The following schematically shows an architecture diagram of a data transmission method based on an embodiment of the present application. Figure 3 The architecture diagram shown includes: a sending terminal 310, a server 320 and a receiving terminal 330; wherein the sending terminal 310 includes a feature extraction module 311, a speech encoding module 312 and a forward encoding module 313, the server 320 includes a packet loss detection module 321, an audio routing module 322 and a forward encoding module 323, and the receiving terminal 330 includes a packet loss detection module 331, a speech decoding module 332, a mixing module 333 and a playback module 334.

[0130] Specifically, the feature extraction module 311 can collect audio signals, perform feature extraction on the collected audio signals, obtain audio features, and send the collected audio signals to the speech coding module 312. The speech coding module 312 can encode the audio signals to obtain corresponding audio code streams, and send the audio code streams to the forward coding module 323. The forward coding module 323 can package the matching audio code streams and audio features into audio data packets, calculate the error correction code of the audio data packets according to the packet loss rate in the previous unit time fed back by the packet loss detection module 321, and package and send them to the server 320, so that the packet loss detection module 321 in the server 320 performs packet loss detection on the received audio data packets and corrects the audio data packets according to the error correction code when there is packet loss in the audio data packets, and sends the corrected audio data packets to the audio routing module 322.

[0131] Furthermore, the audio routing module 322 can filter the audio data packets in the data group according to the audio features corresponding to each data packet in the data group to obtain the target audio data packet; wherein the maximum energy amplitude corresponding to the target audio data packet is greater than that of other audio data packets in the data group, or the energy spectrum distribution corresponding to the target audio data packet is more similar to the preset energy spectrum distribution than that of other audio data packets in the data group. Furthermore, the audio routing module 322 can send the target audio data packet to the forward encoding module 323, so that the forward encoding module 323 calculates the error correction code of the target audio data packet according to the packet loss rate in the previous unit time fed back by the packet loss detection module 331 and packages it to the receiving terminal 330. Furthermore, the packet loss detection module 331 in the receiving terminal 330 performs packet loss detection on the received target audio data packet and corrects it according to the error correction code when there is packet loss in the target audio data packet, and sends the corrected target audio data packet to the speech decoding module 332.

[0132] Furthermore, the speech decoding module 332 can decode the audio code stream in each target audio data packet into an audio signal, and send the decoded audio signal to the mixing module 333, so that the mixing module 333 performs mixing processing on the audio signal. Furthermore, the mixing module 333 can send the mixing processing result to the playing module 334, so that the playing module 334 plays the mixing processing result.

[0133] In modern communication systems, error control methods usually include the following types: Automatic Repeat Request (ARQ), Forward Error Correction (FEC), Hybrid Error Correction (HEC) and Information Repeat Request (IRQ). ARQ is an error detection mechanism that ensures data reliability through feedback response. The decoder at the receiving end uses the error detection code sent by the sending end to determine whether there is an error in the transmission, and informs the sending end of the error detection result through the feedback channel. If the receiving end receives the data correctly, it sends a confirmation message ACK to the sending end. Otherwise, the sending end considers that the data transmission has failed and resends the message to the receiving end until it receives the confirmation information from the receiving end and then continues to send new data. The ARQ method does not have error correction capability and can only detect errors but not correct them. Its advantage is that the decoding equipment is simple. When the redundancy is constant, the error detection capability of the code is much higher than the error correction capability, and a lower bit error rate can be obtained. However, sending ACK will increase network overhead and affect the transmission rate. The continuity and real-time performance of the transmitted information are also poor, and it is not suitable for the transmission of video streams. FEC is an error correction mechanism that recovers lost data packets by sending redundant information. The sender sends an error correction code with error correction capability, and the receiver decodes it according to the received data and the coding rules used to correct errors that occur during the transmission process. The FEC method avoids the retransmission of information, shortens the time required to recover erroneous data, and has a small transmission delay. Therefore, compared with the ARQ method, FEC is more suitable for processing information loss in real-time communication and can greatly reduce the packet loss rate. However, due to the addition of redundant packets, the bandwidth utilization is inevitably reduced and network congestion may also be caused.

[0134] Furthermore, when FEC is applied to data transmission for multi-person conferences, since users usually have a weaker perception of audio signals with low loudness, the applicant has thought of using audio signals with low loudness that are not easily perceived by the user's hearing as screening conditions for audio data packets. Specifically, if the server can not only screen the audio data packets according to the energy of the audio signal, but also perform secondary screening on the screened audio data packets according to the loudness of the audio signal, and then calculate the error correction code for the secondary screened audio data packets and transmit the data packets containing the secondary screened audio data packets and the error correction code to the receiving terminal, it can improve bandwidth utilization and improve the problem of waste of network resources.

[0135] Based on the above problems, this exemplary embodiment provides a data transmission method. The data transmission method can be applied to the above server 105, and can also be applied to one or more of the above terminal devices 101, 102, 103, which is not particularly limited in this exemplary embodiment. Figure 4 As shown, the data transmission method may include the following steps S410 to S430:

[0136] Step S410: obtaining the loudness corresponding to the target audio data packet.

[0137] Step S420: Calculate a target error correction code of the target audio data packet according to the loudness corresponding to the target audio data packet.

[0138] Step S430: Transmit the target audio data packet and the target error correction code to the receiving terminal.

[0139] Step S410 and step S430 may be executed by a server. For example, the server may be a routing server for screening path signals.

[0140] Implementation Figure 1 The method shown can calculate the corresponding target error correction code according to the loudness corresponding to the target audio data packet, rather than calculating the error correction code for all audio data packets, so as to improve the bandwidth utilization. In addition, compared with the prior art of calculating the error correction codes of all audio data packets and transmitting all audio data packets and their corresponding error correction codes, it can realize the transmission of valid data, thereby improving the problem of wasting network resources.

[0141] Next, the above steps of this exemplary embodiment are described in more detail.

[0142] In step S410, the loudness corresponding to the target audio data packet is obtained.

[0143] In the embodiment of the present application, optionally, before obtaining the loudness corresponding to the target audio data packet, the method further includes: screening the target audio data packet whose audio features meet preset conditions from the received multiple audio data packets.

[0144] The audio data packet includes the loudness corresponding to the audio data packet, the audio code stream, and the audio features corresponding to the audio code stream. The audio features corresponding to the audio code stream may include the energy distribution corresponding to the audio code stream and the energy amplitude corresponding to each frequency point in the audio code stream. In addition, the number of target audio data packets is one or more, which is not limited in the embodiments of the present application.

[0145] It can be seen that by implementing this optional embodiment, the received audio data packets can be screened by audio features to reduce the amount of forwarded data, thereby reducing the loss of network resources and improving data transmission efficiency.

[0146] In the embodiment of the present application, optionally, before screening the target audio data packet whose audio features meet the preset conditions from the received multiple audio data packets, the method further includes:

[0147] Obtain error correction codes corresponding to multiple audio data packets respectively; perform packet loss detection on the multiple audio data packets according to the error correction codes to obtain packet loss rates corresponding to the multiple audio data packets respectively; feed back the packet loss rates to the sending terminal and perform error correction on the multiple audio data packets according to the error correction codes.

[0148] Specifically, the number of the sender terminal can be one or more, which is not limited by the embodiment of the present application. When the embodiment of the present application is applied to a multi-person conference, the conference member terminal in the multi-person conference can be used as a sender terminal and also as a receiver terminal. In addition, multiple audio data packets can belong to the same data packet, and the sender terminal can send one data packet or multiple data packets at a time, each data packet includes one or more audio data packets, and the multiple audio data packets correspond to different audio paths.

[0149] As an optional implementation, the method of obtaining the error correction codes corresponding to the multiple audio data packets can be specifically: obtaining the error correction codes corresponding to the multiple audio data packets according to the encoding method of Reed-solomon codes (RS encoding). Specifically including: constructing polynomials corresponding to the audio data packets according to Galois Field (GF), determining the check equation for representing the audio data packet according to the polynomial, and solving the check equation to obtain the error correction code corresponding to the audio data packet.

[0150] As an optional implementation, packet loss detection is performed on multiple audio data packets according to the error correction code to obtain the packet loss rates corresponding to the multiple audio data packets. Specifically, the continuity of each codeword in the audio code stream of the multiple audio data packets is detected according to the error correction code. If two codewords without continuity are detected, the missing codeword between the two codewords without continuity is determined as the loss code of the corresponding audio data packet.

[0151] As an optional implementation, the method of feeding back the packet loss rate to the sending terminal and correcting multiple audio data packets according to the error correction code can be specifically: feeding back the packet loss rate within a preset unit time (e.g., 10 seconds) to the sending terminal and correcting multiple audio data packets according to the error correction codes corresponding to the multiple audio data packets.

[0152] It can be seen that the implementation of this optional embodiment can provide a reference condition for the sending terminal to calculate the redundancy by transmitting the packet loss rate to the sending terminal. In this way, the redundancy calculated by the sending terminal each time can be related to the packet loss rate in the previous preset unit time, thereby improving the adaptability of the present application.

[0153] In the embodiment of the present application, optionally, the received multiple audio data packets are sent by a sending terminal;

[0154] Among them, the sending terminal sends multiple audio data packets in the following specific manner: the sending terminal collects audio signals and extracts features of the audio signals to obtain audio features; encodes the audio signals to obtain audio code streams; packages the audio code streams and audio features into audio data packets and sends them to the server.

[0155] It should be noted that the audio signal collected by the sender terminal can be an analog signal. The audio code stream is used to represent the data flow used by the audio signal per unit time. The audio code stream = sampling rate * number of bits * channel, such as 44100 * 16 * 2 = 1.41 Mbit / sec.

[0156] As an optional implementation, the audio feature corresponding to the audio signal may include at least one of a zero-crossing rate, a short-time energy, a short-time autocorrelation function, and a short-time average amplitude difference, which is not limited in the embodiment of the present application.

[0157] When the audio feature corresponding to the audio signal includes a zero-crossing rate, the sending terminal collects the audio signal and extracts the feature of the audio signal. The specific method for obtaining the audio feature can be: the sending terminal collects the audio signal and extracts the feature according to The zero-crossing rate of the audio signal is calculated as the audio feature of the audio signal, where N is the frame length of the audio signal and n is the number of frames of the audio signal. The calculated zero-crossing rate can be used to characterize the number of times each frame of the audio signal passes through the zero value. The above zero-crossing rate can be used to determine the unvoiced and voiced sounds in the audio signal, which is beneficial for the server to screen multiple audio data packets received.

[0158] When the audio feature corresponding to the audio signal includes short-time energy, the sending terminal collects the audio signal and extracts the feature of the audio signal. The specific method for obtaining the audio feature can be: the sending terminal collects the audio signal and extracts the feature according to the short-time energy. Detect the nth frame audio signal x in the collected audio signal n (m) short-time energy E n, the short-time energy corresponding to each frame is obtained as the audio feature of the audio signal; wherein N is the frame length of the audio signal, and n is a positive integer. Generally speaking, the energy of human voice is greater than the energy of noise. The calculated short-time energy corresponding to each frame can be used to distinguish between human voice and noise in the audio signal, which is beneficial for the server to filter out noise from multiple received audio data packets according to the energy corresponding to each frame, thereby improving the processing efficiency of audio data.

[0159] When the audio feature corresponding to the audio signal includes a short-time autocorrelation function, the sending terminal collects the audio signal and extracts the feature of the audio signal. The specific method for obtaining the audio feature can be: the sending terminal collects the audio signal and extracts the feature according to The short-time autocorrelation function of the audio signal is calculated as the audio feature of the audio signal; wherein N is the frame length of the audio signal, n is a positive integer, w is used to represent the window function, and w'(m) is used to represent the audio frame after windowing. The calculated short-time autocorrelation function can be used to measure the similarity of the signal's own time waveform, which is beneficial for the server to detect similar characteristics of the audio according to the short-time autocorrelation function, thereby improving the processing efficiency of the audio data.

[0160] When the audio feature corresponding to the audio signal includes the short-time average amplitude difference, the sending terminal collects the audio signal and extracts the feature of the audio signal. The specific method for obtaining the audio feature can be: the sending terminal collects the audio signal and extracts the feature according to the short-time average amplitude difference. The short-time average amplitude difference of the audio signal is calculated as the audio feature of the audio signal; wherein k=0, 1, ..., N-1, and the calculated short-time average amplitude difference can be used to measure the change of the audio amplitude.

[0161] In addition, the audio features corresponding to the audio signal may also include spectrogram, short-time power spectral density, spectral entropy, fundamental frequency, resonance peak and other features, which are not limited in the embodiments of the present application.

[0162] It can be seen that by implementing this optional embodiment, the sending terminal can extract features of the audio signal, which can help the server select the path signal based on the result of the feature extraction, thereby ensuring the real-time audio output of the multi-person conference while also ensuring the effect of the audio output.

[0163] In the embodiment of the present application, optionally, the preset condition includes a preset energy amplitude and / or a preset signal-to-noise ratio, and screening a target audio data packet whose audio features meet the preset condition from the received multiple audio data packets includes:

[0164] If it is detected that there is at least one energy amplitude greater than a preset energy amplitude in the audio features of the audio code stream, the audio data packet to which the audio code stream belongs is determined as the target audio data packet; and / or,

[0165] If it is detected that at least one signal-to-noise ratio greater than a preset signal-to-noise ratio exists in the audio features of the audio code stream, the audio data packet to which the audio code stream belongs is determined as the target audio data packet.

[0166] Specifically, each audio frame in the audio code stream corresponds to an energy amplitude, which is used to represent the short-term energy corresponding to the audio frame. The signal-to-noise ratio is the ratio of the average power of the audio signal to the average power of the noise, that is, signal-to-noise ratio (dB) = 10*log10(S / N) (dB).

[0167] It can be seen that by implementing this optional embodiment, the audio data packets can be screened by energy amplitude or signal-to-noise ratio to streamline the audio data packets that need to be transmitted and improve data transmission efficiency.

[0168] In the embodiment of the present application, optionally, the above method further includes:

[0169] The sending terminal performs frame processing on the audio code stream according to a preset time length to obtain multiple audio frames; processes the multiple audio frames respectively through a preset window function to obtain multiple reference frames; calculates power spectra corresponding to the multiple reference frames respectively; and calculates the loudness corresponding to the audio data packet according to the power spectrum.

[0170] Specifically, the durations corresponding to multiple audio frames are consistent (e.g., 10ms or 20ms), and the durations corresponding to multiple reference frames and audio frames are consistent. In addition, since the Fast Fourier Transform (FFT) can only transform time domain data of a limited length, it is necessary to truncate the time domain signal. Even for a periodic signal, if the time length of the truncated periodic signal is not an integer multiple of the period, the truncated signal will be prone to leakage. Therefore, the present application applies a preset window function that can make the audio signal in the time domain meet the periodicity requirements of the Fourier transform to reduce signal leakage. The preset window function is a Hanning window function, a Hamming window function, a Blackman window function, a Kaiser window function, a triangular window function, or a rectangular window function.

[0171] As an optional implementation, a plurality of audio frames may be processed separately through a preset window function to obtain a plurality of reference frames by: determining time domain expressions corresponding to the plurality of audio frames respectively; performing dot multiplication of the time domain expressions corresponding to the plurality of audio frames respectively with the preset window function to obtain reference frames corresponding to the plurality of audio frames respectively.

[0172] As an optional implementation manner, a method of calculating power spectra corresponding to the multiple reference frames may be: performing fast Fourier transform on the multiple reference frames to determine the power spectra corresponding to the multiple reference frames.

[0173] It can be seen that by implementing this optional embodiment, the loudness of the target audio data packet can be calculated, and the loudness can be used as a condition for calculating redundancy, so that the redundancy used to calculate the error correction code is positively correlated with the loudness of the audio, thereby improving the error correction accuracy of the receiving terminal for audio with higher loudness, thereby improving the audio signal output quality of the receiving terminal.

[0174] In the embodiment of the present application, optionally, the sending terminal calculates the loudness corresponding to the audio data packet according to the power spectrum, including:

[0175] The sending terminal calculates the frequency loudness of each frequency point in the power spectrum according to the energy amplitude of each frequency point in the power spectrum; calculates the loudness weight of each frequency point in the power spectrum according to the frequency loudness; calculates the weighted sum between the energy amplitude of each frequency point in the power spectrum and the loudness weight of each frequency point in the power spectrum as the loudness value of the reference frame corresponding to the power spectrum; and determines the sum of the loudness values ​​of multiple reference frames as the loudness corresponding to the audio data packet.

[0176] Specifically, the frequency point is the number of a fixed frequency, which can be used as the only representation of a fixed frequency. In addition, loudness is a subjective feeling used to describe the size of a sound, and its unit is sone. A 1000Hz pure tone with a sound pressure level of 40dB has a loudness of 1 sone; a 2-sone sound is twice as loud as a 40-sound level sound; and a 4-sone sound is four times as loud as a 40-sound level sound. In other words, when the sound pressure level increases by 10dB, the loudness doubles, and the human ear's perception of loudness changes with the sound pressure level. Please refer to Figure 5 , Figure 5 The figure schematically shows an acoustic equal loudness curve diagram according to an embodiment of the present application. Figure 5 The acoustic equal loudness curve shown in the figure shows curves of various loudness levels (i.e., 100phon, 80phon, 60phon, 40phon, 20phon, threshold thrshold phon), with the horizontal axis representing the sound wave frequency (Frequency) and the vertical axis representing the sound pressure level (Sound Pressure Level). Figure 5 As shown in the acoustic equal loudness curve, the equal loudness curve is a curve that describes the relationship between the sound pressure level and the sound wave frequency under equal loudness conditions. In the low and medium frequencies (below 1kHz), the lower the frequency, the greater the sound pressure intensity (energy) required for equal loudness. In other words, the greater the sound energy, the more consistent the human ear's hearing experience. In the high and medium frequencies (above 1kHz), frequencies in different frequency bands correspond to different acoustic hearing perception characteristics. See Figure 5 It can be seen that if loudness is used as a condition for calculating redundancy, so that the redundancy used to calculate the error correction code is positively correlated with the loudness of the audio, the error correction accuracy of the receiving terminal for audio with higher loudness can be improved, thereby improving the audio signal output quality of the receiving terminal.

[0177] As an optional implementation manner, a method for calculating the frequency loudness of each frequency point in the power spectrum according to the energy amplitude of each frequency point in the power spectrum may specifically be: determining the absolute value P(i, j) of the energy amplitude of each frequency point i in the power spectrum, where j=0 to K-1, and K is the total number of frequency points.

[0178] As an optional implementation, the method of calculating the loudness weight of each frequency point in the power spectrum according to the frequency point loudness may be specifically: the loudness weight cof(freq) of each frequency point freq in the power spectrum is calculated based on the following formula; wherein,

[0179]

[0180]

[0181]

[0182] Loudness of freq = 4.2 + afy * (dB-cfy) / (1 + bfy * (dB-cfy));

[0183]

[0184] See the expression for cof(freq) Figure 6 , Figure 6 The figure schematically shows a loudness weight curve diagram according to an embodiment of the present application. Figure 6 In the curve graph shown, the horizontal axis is used to represent frequency, and the vertical axis is used to represent loudness weight. When the loudness of a frequency point is known, it can be obtained by Figure 6 The curve diagram shown determines the weight corresponding to the frequency point.

[0185] As an optional implementation manner, a weighted sum between the energy amplitude of each frequency point in the power spectrum and the loudness weight of each frequency point in the power spectrum is calculated as the loudness value of the reference frame corresponding to the power spectrum. Specifically, the method of calculating the weighted sum between the energy amplitude of each frequency point in the power spectrum and the loudness weight of each frequency point in the power spectrum as the loudness value EP(i) of the reference frame corresponding to the power spectrum may be: wherein i is the frame number, and k is the frequency number.

[0186] It can be seen that by implementing this optional embodiment, the loudness corresponding to the target audio data packet can be determined according to the loudness corresponding to each frequency point as a condition for calculating redundancy, thereby improving the error correction accuracy of the receiving terminal for audio with higher loudness and improving the audio signal output quality of the receiving terminal.

[0187] In step S420, a target error correction code of the target audio data packet is calculated according to the loudness corresponding to the target audio data packet.

[0188] In an embodiment of the present application, optionally, calculating a target error correction code for a target audio data packet according to a loudness corresponding to a target audio data packet includes: determining a reference redundancy according to a packet loss rate fed back by a receiving terminal; wherein the packet loss rate corresponds to a historical unit time closest to a sending time of the target audio data packet; calculating a target redundancy corresponding to the target audio data packet according to the reference redundancy and the loudness corresponding to the target audio data packet; and calculating a target error correction code for the target audio data packet according to the target redundancy.

[0189] Specifically, since the target audio data packet can be one or more, when the number of target audio data packets is multiple, multiple target audio data packets can correspond to one reference error correction code or multiple reference error correction codes. For example, if one reference error correction code can be used to correct 5 target audio data packets, then 10 target audio data packets can correspond to 2 reference error correction codes, that is, every 5 target audio data packets correspond to one reference error correction code. In addition, it should be noted that the reference error correction code can be transmitted in the form of a redundant packet. In addition, the redundancy is inversely proportional to the code rate, that is, the smaller the code rate, the greater the redundancy, and the code rate can be understood as coding efficiency = number of information code elements k / code length n.

[0190] As an optional implementation, before determining the reference redundancy according to the packet loss rate fed back by the receiving terminal, the method may further include: the receiving terminal calculates the packet loss rate according to all audio data packets received within a unit time and feeds the packet loss rate back to the server.

[0191] As an optional implementation, the method of determining the reference redundancy based on the packet loss rate fed back by the receiving terminal can be specifically: determining the target preset interval in which the packet loss rate fed back by the receiving terminal is located from multiple preset intervals; selecting the reference redundancy corresponding to the target preset interval. For example, the target preset interval [20% to 30%) in which the packet loss rate of 23% fed back by the receiving terminal is located can be determined from multiple preset intervals such as [0 to 20%), [20% to 30%), [30% to 40%), and [40% to 50%). Since the redundancy corresponding to the target preset interval [20% to 30%) is 25%, 25% can be selected as the reference redundancy.

[0192] As an optional implementation, the method of calculating the target error correction code corresponding to the target audio data packet according to the target redundancy can be specifically as follows: determine the audio code stream in the target audio data packet; group the code words in the audio code stream according to each group of x bits, and input the grouping results into the encoder in sequence, so that the encoder generates each group of corresponding code words of length z according to the target redundancy; wherein the code words of length z include the code words in the audio code stream of x bits and the target error correction code of y bits for error correction, z=x+y, and the target redundancy corresponds to y. In addition, the code words of length z can be RS coding, wherein RS coding is linear cyclic coding.

[0193] As another optional implementation, the number of target audio data packets is k, k is a positive integer, and the method of calculating the target error correction code corresponding to the target audio data packet according to the target redundancy can be specifically: compressing the nth audio data packet by using the target redundancy and the data compression algorithm; determining the obtained compression result as the target error correction code of the (n+1)th audio data packet; wherein the (n+1)th audio data packet is the above-mentioned target audio data packet, the compression result matches the target redundancy, and n is a positive integer. That is to say, when the nth audio data packet is lost, the nth audio data packet can be repaired by the target error correction code in the (n+1)th audio data packet.

[0194] It can be seen that by implementing this optional embodiment, the target redundancy of the target error correction code can be determined according to the historical packet loss rate per unit time, that is, the target redundancy of the target error correction code can be adjusted in real time according to the network conditions, which can effectively utilize the bandwidth and achieve effective control of the packet loss rate.

[0195] In the embodiment of the present application, optionally, the target redundancy corresponding to the target audio data packet is calculated according to the reference redundancy and the loudness corresponding to the target audio data packet, including: taking the loudness corresponding to the target audio data packet as the input of a preset function expression to calculate the output value; and determining the product of the output value and the reference redundancy as the target redundancy of the target audio data packet.

[0196] Specifically, the target redundancy is greater than the reference redundancy.

[0197] As an optional implementation, the method of using the loudness corresponding to the target audio data packet as the input of the preset function expression to calculate the output value may be: using the loudness EP(i) corresponding to the target audio data packet as the input of the preset function expression f(EP(i)) to calculate the output value. In addition, it should be noted that the preset function expression f(EP(i)) may be a monotonically increasing function, and the output value range of the preset function expression f(EP(i)) is [0,1], so that the target redundancy can be increased as the loudness corresponding to the target audio data packet increases, that is, the target redundancy is positively correlated with the loudness corresponding to the target audio data packet.

[0198] Further optionally, the method of determining the product of the output value and the reference redundancy as the target redundancy of the target audio data packet can be specifically: determining the product f(EP(i))*red_org(i) of the output value of f(EP(i)) and the reference redundancy red_org(i) as the target redundancy red'(i) of the target audio data packet.

[0199] It can be seen that by implementing this optional embodiment, the redundancy (i.e., the target redundancy) can be re-determined in combination with the loudness corresponding to the target audio data packet, and the positive correlation between the loudness corresponding to the target audio data packet and the target redundancy can be improved, thereby making the redundancy corresponding to the audio signal with higher loudness higher, which is beneficial for the receiving terminal to restore the target audio data packet more accurately.

[0200] In step S430, the target audio data packet and the target error correction code are transmitted to the receiving terminal.

[0201] Specifically, the packet loss situation includes a packet loss rate (eg, 50%, 0, 100%, etc.) corresponding to the target audio data packet.

[0202] In an embodiment of the present application, optionally, the target audio data packet and the target error correction code are transmitted to the receiving terminal, including: packaging the target audio data packet and the target error correction code into a data packet to be transmitted and transmitting them to the receiving terminal, so that the receiving terminal detects the packet loss situation corresponding to the target audio data packet according to the target error correction code in the decoding result after decoding the data packet to be transmitted, and corrects the target audio data packet according to the packet loss situation.

[0203] Specifically, the data packet to be transmitted may be a forward error correction (FEC) packet, wherein the FEC packet structure 700 may refer to Figure 7 , Figure 7 The structure diagram of the FEC packet according to an embodiment of the present application is schematically shown. Figure 7As shown, the FEC packet structure 700 may be composed of an RTP header 701, an FEC header 702, and an FEC payload 703; wherein both the FEC header 702 and the FEC payload 703 may serve as RTP payloads.

[0204] Furthermore, for Figure 7 For further explanation of the structure of FEC header 702 in Figure 8 , Figure 8 The structure diagram of the FEC header according to an embodiment of the present application is schematically shown. Specifically, the FEC header structure 800 is 12 bytes, including an SN cardinality field, a length recovery field, an E field, a PT recovery field, a mask field, and a TS recovery field; wherein the length recovery field is used to determine the length of the audio data packet to be corrected; the E field is used to indicate whether there is an extension part in the audio data packet; the PT recovery field is used to recover the payload type of the lost data packet, and the value of the PT recovery field can be obtained by performing an XOR operation on the values ​​of the payload type fields of multiple target audio data packets; the size of the Mask field is 24 bits, and the bit positions are 0 to 23 from low to high. If the i-th bit is set to 1, the target audio data packet with the sequence number N+i is associated with the FEC packet, wherein N is the value of the SN field; the value of the SN cardinality field is the minimum value of the sequence number of the target audio data packet related to the FEC packet. The value of the TS recovery field is the result obtained by XORing the timestamp of the target audio data packet related to the FEC packet, and the TS recovery field can be used to recover the timestamp of the lost data packet.

[0205] It can be seen that by implementing this optional embodiment, the error correction accuracy of the receiving terminal can be improved by transmitting the target error correction code related to the loudness of the target audio data packet.

[0206] In an embodiment of the present application, optionally, after decoding the data packet to be transmitted, the receiving terminal detects the packet loss situation corresponding to the target audio data packet according to the target error correction code in the decoding result and corrects the target audio data packet according to the packet loss situation. Specifically, the receiving terminal decodes the data packet to be transmitted to obtain the audio code stream and the target error correction code in the target audio data packet; detects the lost code of the audio code stream in the target audio data packet according to the target error correction code; and restores the lost code according to the target error correction code to achieve error correction for the target audio data packet.

[0207] As an optional implementation, the receiving terminal detects the missing code of the audio code stream in the target audio data packet according to the target error correction code in a specific manner: the receiving terminal detects the continuity of each code word in the audio code stream according to the target error correction code, and if two code words without continuity are detected, the missing code word between the two code words without continuity is determined as the missing code. The missing code word may be one or more, which is not limited in the embodiment of the present application.

[0208] As an optional implementation, the receiving terminal can also calculate the packet loss rate per unit time and feed back the packet loss rate to the server after restoring the lost code according to the target error correction code, so that the server can perform redundancy calculation in the next data transmission process according to the fed-back packet loss rate.

[0209] It can be seen that by implementing this optional embodiment, the receiving terminal can correct the received target audio data packet according to the target error correction code obtained by decoding. Since the target error correction code is positively correlated with the loudness of the target audio data packet, the higher the loudness of the audio data packet, the greater the probability that the audio data is correctly restored. Among them, the higher the loudness of the audio data, the greater the probability that the human voice is included. This can ensure that the loud audio signal output to the user is clear and complete, thereby improving the user experience.

[0210] In an embodiment of the present application, optionally, after the receiving terminal restores the lost code according to the target error correction code to achieve error correction for the target audio data packet, the above method also includes: the receiving terminal mixes all the corrected target audio data packets to obtain a mixed signal and plays it.

[0211] As an optional implementation, before the receiving terminal mixes all the target audio data packets after error correction, the following steps may also be included: the receiving terminal normalizes the format of the target audio data packets after error correction, so that the format of the target audio data packets after error correction can be guaranteed to be unified; then, the target audio data packets after error correction are converted to a preset sampling rate (e.g., 16 kHz, 32 kHz, 44.1 kHz, 48 kHz, etc.); then, the bit depth (Bit-Depth) or sampling format (Sample Format) of the target audio data packets after error correction is detected for consistency. If the bit depth is inconsistent or the sampling format is inconsistent, the target audio data packets after error correction are normalized accordingly so that the number of bits carrying the audio data of each sampling point is the same; then, the channels (e.g., mono or dual channels) of the target audio data packets after error correction are detected to be consistent. If not, a prompt message indicating that the channels are inconsistent is output; if consistent, the operation of mixing all the target audio data packets after error correction is performed. In addition, optionally, before the receiving terminal performs mixing processing on all error-corrected target audio data packets, the receiving terminal may also perform echo cancellation, noise suppression, silence detection and other processing on the error-corrected target audio data packets, which is not limited in the embodiments of the present application.

[0212] As an optional implementation, the receiving terminal mixes all the target audio data packets after error correction to obtain a mixed signal. The receiving terminal may adjust the volume of each target audio data packet after error correction according to the energy amplitudes corresponding to all the target audio data packets after error correction, and mix the results of the equalization adjustment to obtain a mixed signal.

[0213] As an optional implementation, after the receiving terminal mixes all error-corrected target audio data packets to obtain a mixed signal, the following steps may also be included: the receiving terminal performs overflow detection on the mixed signal, and if overflow exists, performs overflow processing / smoothing processing on the overflowed sampling points, and then plays the processing result.

[0214] It can be seen that by implementing this optional embodiment, the receiving terminal can mix and output the target audio data packet after correcting the error of the target audio data packet. The output result retains the important audio content in the meeting and discards the audio content that the human ear has poor perception ability. It can improve the message forwarding efficiency of multi-person meetings and ensure the real-time performance of the output audio signal, thereby improving the user experience.

[0215] Furthermore, in this exemplary embodiment, a data transmission system 900 is also provided. Fig. 9 , including a sender terminal 901, a server 902 and a receiver terminal 903, wherein:

[0216] The sending terminal 901 is used to send a target audio data packet to the server 902, where the target audio data packet includes the loudness corresponding to the target audio data packet;

[0217] The server 902 is used to receive a target audio data packet and obtain the loudness corresponding to the target audio data packet; calculate the target error correction code of the target audio data packet according to the loudness corresponding to the target audio data packet; and transmit the target audio data packet and the target error correction code to the receiving terminal 903;

[0218] The receiving terminal 903 is used to detect the packet loss situation corresponding to the target audio data packet according to the target error correction code.

[0219] It can be seen that implementation Fig. 9 The system shown can calculate the corresponding target error correction code according to the loudness corresponding to the target audio data packet, rather than calculating the error correction code for all audio data packets, so as to improve the bandwidth utilization. In addition, compared with the prior art of calculating the error correction codes of all audio data packets and transmitting all audio data packets and their corresponding error correction codes, it can realize the transmission of valid data, thereby improving the problem of wasting network resources.

[0220] Specifically, see Fig.10 . Fig.10The structure block diagram of a data transmission system 1000 according to another embodiment of the present application is schematically shown. Fig.10 As shown, Fig.10 The structural block diagram shown includes: a sending terminal 1010, a server 1020 and a receiving terminal 1030; wherein the sending terminal 1010 includes a feature extraction module 1011, a speech encoding module 1012 and a forward encoding module 1013, the server 1020 includes a packet loss detection module 1021, an audio routing module 1022, a forward encoding module 1023 and a perception analysis module 1024, and the receiving terminal 1030 includes a packet loss detection module 1031, a speech decoding module 1032, a mixing module 1033 and a playback module 1034.

[0221] Specifically, the feature extraction module 1011 can collect audio signals, perform feature extraction on the collected audio signals, obtain audio features, and send the collected audio signals to the speech coding module 1012. The speech coding module 1012 can encode the audio signals to obtain corresponding audio code streams, and send the audio code streams to the forward coding module 1023. The forward coding module 1023 can package the matched audio code streams and audio features into an audio data packet, wherein the audio data packet includes the loudness, audio code streams, and audio features corresponding to the audio data packet; the error correction code of the audio data packet is calculated according to the packet loss rate in the previous unit time fed back by the packet loss detection module 1021, and the audio data packet is packaged and sent to the server 1020, so that the packet loss detection module 1021 in the server 1020 performs packet loss detection on the received audio data packet, and when there is packet loss in the audio data packet, the audio data packet is corrected according to the error correction code, and the corrected audio data packet is sent to the audio routing module 1022.

[0222] Regarding the previous content, the audio data packet may not include loudness. After the forward encoding module 1013 in the sending terminal 1010 sends the audio data packet to the perceptual analysis module 1024 of the server 1020, the perceptual analysis module 1024 can calculate the loudness corresponding to the audio signal based on the audio features in the audio data packet.

[0223] Furthermore, the audio routing module 1022 can filter the audio data packets in the data group according to the audio features corresponding to each data packet in the data group to obtain the target audio data packet; wherein the maximum energy amplitude corresponding to the target audio data packet is greater than that of other audio data packets in the data group, or the energy spectrum distribution corresponding to the target audio data packet is more similar to the preset energy spectrum distribution than that of other audio data packets in the data group. Furthermore, the audio routing module 1022 can send the target audio data packet to the perception analysis module 1024, so that the perception analysis module 1024 filters the target audio data packet according to the loudness corresponding to the target audio data packet and sends the filtered target audio data packet to the forward encoding module 1023. The forward encoding module 1023 calculates the error correction code of the filtered target audio data packet according to the packet loss rate in the previous unit time fed back by the packet loss detection module 1031 and packages it and sends it to the receiving terminal 1030. Furthermore, the packet loss detection module 1031 in the receiving terminal 1030 performs packet loss detection on the received target audio data packet and, when packet loss occurs in the target audio data packet, corrects the error according to the error correction code and sends the corrected target audio data packet to the speech decoding module 1032; wherein the number of target audio data packets may be one or more.

[0224] Furthermore, the speech decoding module 1032 can decode the audio code stream in each target audio data packet into an audio signal, and send the decoded audio signal to the mixing module 1033, so that the mixing module 1033 performs mixing processing on the audio signal. Furthermore, the mixing module 1033 can send the mixing processing result to the playing module 1034, so that the playing module 1034 plays the mixing processing result.

[0225] It can be seen that implementation Fig.10 The system shown can calculate the corresponding target error correction code according to the loudness corresponding to the target audio data packet, rather than calculating the error correction code for all audio data packets, so as to improve the bandwidth utilization. In addition, compared with the prior art of calculating the error correction codes of all audio data packets and transmitting all audio data packets and their corresponding error correction codes, it can realize the transmission of valid data, thereby improving the problem of wasting network resources.

[0226] Further, in this example embodiment, optionally, the number of the above-mentioned sending terminal and receiving terminal can be multiple, and the above-mentioned server can be a server cluster. Fig.11 , Fig.11 Schematically shows a structural block diagram of a data transmission system 1100 according to another embodiment of the present application. Fig.11As shown, it includes a sender terminal 1111, a sender terminal 1112, ..., a sender terminal 111n, a server cluster 1130, a receiver terminal 1121, a receiver terminal 1122, ..., a receiver terminal 112n; wherein n is a positive integer and is greater than or equal to 3.

[0227] See also Fig.11 It can be seen that the server cluster 1130 in the present application can receive data packets sent from at least one of the sender terminal 1111, the sender terminal 1112, ..., and the sender terminal 111n. Each data packet may include one or more audio data packets. If the audio data packet meets the preset conditions, the audio data packet can be determined as a target audio data packet by the server cluster 1130, and the error correction code of the target audio data packet is calculated according to the loudness of the target audio data packet, and the target audio data packet and the error correction code are packaged as a data packet to be transmitted and forwarded to the receiving terminal 1121, the receiving terminal 1122, ..., and the receiving terminal 112n, so that the receiving terminal 1121, the receiving terminal 1122, ..., and the receiving terminal 112n can correct the target audio data packet with packet loss according to the error correction code.

[0228] It can be seen that implementation Fig.11 The system shown can calculate the corresponding target error correction code according to the loudness corresponding to the target audio data packet, rather than calculating the error correction code for all audio data packets, so as to improve the bandwidth utilization. In addition, compared with the prior art of calculating the error correction codes of all audio data packets and transmitting all audio data packets and their corresponding error correction codes, it can realize the transmission of valid data, thereby improving the problem of wasting network resources.

[0229] Furthermore, in this exemplary embodiment, a data transmission device is also provided. Fig.12 As shown, the data transmission device 1200 may include:

[0230] A loudness acquisition unit 1201 is used to acquire the loudness corresponding to the target audio data packet;

[0231] The error correction code calculation unit 1202 is used to calculate the target error correction code of the target audio data packet according to the loudness corresponding to the target audio data packet;

[0232] The data sending unit 1203 is used to transmit the target audio data packet and the target error correction code to the receiving terminal.

[0233] It can be seen that implementation Fig.12The device shown can calculate the corresponding target error correction code according to the loudness corresponding to the target audio data packet, rather than calculating the error correction code for all audio data packets, so as to improve the bandwidth utilization. In addition, compared with the prior art of calculating the error correction codes of all audio data packets and transmitting all audio data packets and their corresponding error correction codes, it can realize the transmission of valid data, thereby improving the problem of wasting network resources.

[0234] In an exemplary embodiment of the present application, the above-mentioned device further includes a data screening unit (not shown):

[0235] The data screening unit is configured to screen a target audio data packet whose audio features meet a preset condition from among the received multiple audio data packets before the loudness obtaining unit 1201 obtains the loudness corresponding to the target audio data packet.

[0236] The audio data packet includes the loudness corresponding to the audio data packet, the audio bit stream and the audio features corresponding to the audio bit stream. The audio features corresponding to the audio bit stream include the energy distribution corresponding to the audio bit stream and the energy amplitude corresponding to each frequency point in the audio bit stream.

[0237] It can be seen that by implementing this optional embodiment, the received audio data packets can be screened by audio features to reduce the amount of forwarded data, thereby reducing the loss of network resources and improving data transmission efficiency.

[0238] In an exemplary embodiment of the present application, the above-mentioned device further includes an error correction code acquisition unit (not shown), a packet loss detection unit (not shown) and a data error correction unit (not shown):

[0239] An error correction code acquisition unit, used for acquiring error correction codes corresponding to the plurality of audio data packets respectively before the data screening unit screens the target audio data packets whose audio features meet the preset conditions from the received plurality of audio data packets;

[0240] A packet loss detection unit, configured to perform packet loss detection on a plurality of audio data packets according to an error correction code, and obtain packet loss rates corresponding to the plurality of audio data packets respectively;

[0241] The data error correction unit is used to feed back the packet loss rate to the sending terminal and perform error correction on multiple audio data packets according to the error correction code.

[0242] It can be seen that the implementation of this optional embodiment can provide a reference condition for the sending terminal to calculate the redundancy by transmitting the packet loss rate to the sending terminal. In this way, the redundancy calculated by the sending terminal each time can be related to the packet loss rate in the previous preset unit time, thereby improving the adaptability of the present application.

[0243] In an exemplary embodiment of the present application, the received multiple audio data packets are sent by a sending terminal;

[0244] The specific method for the sending terminal to send multiple audio data packets is as follows:

[0245] The sending terminal collects the audio signal and extracts the features of the audio signal to obtain the audio features;

[0246] The sending terminal encodes the audio signal to obtain an audio code stream;

[0247] The sending terminal packages the audio code stream and audio features into an audio data packet and sends it to the server.

[0248] It can be seen that by implementing this optional embodiment, the sending terminal can extract features of the audio signal, which can help the server select the path signal based on the result of the feature extraction, thereby ensuring the real-time audio output of the multi-person conference while also ensuring the effect of the audio output.

[0249] In an exemplary embodiment of the present application, the preset condition includes a preset energy amplitude and / or a preset signal-to-noise ratio, and the data screening unit screens a target audio data packet whose audio characteristics meet the preset condition from the received multiple audio data packets, including:

[0250] If it is detected that there is at least one energy amplitude greater than a preset energy amplitude in the audio features of the audio code stream, the audio data packet to which the audio code stream belongs is determined as the target audio data packet; and / or,

[0251] If it is detected that at least one signal-to-noise ratio greater than a preset signal-to-noise ratio exists in the audio features of the audio code stream, the audio data packet to which the audio code stream belongs is determined as the target audio data packet.

[0252] It can be seen that by implementing this optional embodiment, the audio data packets can be screened by energy amplitude or signal-to-noise ratio to streamline the audio data packets that need to be transmitted and improve data transmission efficiency.

[0253] In an exemplary embodiment of the present application, it further includes:

[0254] The sending terminal performs frame processing on the audio code stream according to a preset duration to obtain multiple audio frames;

[0255] The sending terminal processes the multiple audio frames respectively through a preset window function to obtain multiple reference frames;

[0256] The transmitting terminal calculates the power spectra corresponding to the multiple reference frames respectively;

[0257] The sending terminal calculates the loudness corresponding to the audio data packet based on the power spectrum.

[0258] Among them, the preset window function is a Hanning window function, a Hamming window function, a Blackman window function, a Kaiser window function, a triangular window function or a rectangular window function.

[0259] It can be seen that by implementing this optional embodiment, the loudness of the target audio data packet can be calculated, and the loudness can be used as a condition for calculating redundancy, so that the redundancy used to calculate the error correction code is positively correlated with the loudness of the audio, thereby improving the error correction accuracy of the receiving terminal for audio with higher loudness, thereby improving the audio signal output quality of the receiving terminal.

[0260] In an exemplary embodiment of the present application, the sending terminal calculates the loudness corresponding to the audio data packet according to the power spectrum, including:

[0261] The transmitting terminal calculates the frequency point loudness of each frequency point in the power spectrum according to the energy amplitude of each frequency point in the power spectrum;

[0262] The sending terminal calculates the loudness weight of each frequency point in the power spectrum according to the frequency point loudness;

[0263] The transmitting terminal calculates the weighted sum of the energy amplitude of each frequency point in the power spectrum and the loudness weight of each frequency point in the power spectrum as the loudness value of the reference frame corresponding to the power spectrum;

[0264] The transmitting terminal determines the sum of the loudness values ​​of the multiple reference frames as the loudness corresponding to the audio data packet.

[0265] It can be seen that by implementing this optional embodiment, the loudness corresponding to the target audio data packet can be determined according to the loudness corresponding to each frequency point as a condition for calculating redundancy, thereby improving the error correction accuracy of the receiving terminal for audio with higher loudness and improving the audio signal output quality of the receiving terminal.

[0266] In an exemplary embodiment of the present application, the error correction code calculation unit 1202 calculates a target error correction code of a target audio data packet according to the loudness corresponding to the target audio data packet, including:

[0267] Determine the reference redundancy according to the packet loss rate fed back by the receiving terminal; wherein the packet loss rate corresponds to the historical unit time closest to the sending time of the target audio data packet;

[0268] Calculate the target redundancy corresponding to the target audio data packet according to the reference redundancy and the loudness corresponding to the target audio data packet;

[0269] A target error correction code for a target audio data packet is calculated according to the target redundancy.

[0270] It can be seen that by implementing this optional embodiment, the target redundancy of the target error correction code can be determined according to the historical packet loss rate per unit time, that is, the target redundancy of the target error correction code can be adjusted in real time according to the network conditions, which can effectively utilize the bandwidth and achieve effective control of the packet loss rate.

[0271] In an exemplary embodiment of the present application, the error correction code calculation unit 1202 calculates the target redundancy corresponding to the target audio data packet according to the reference redundancy and the loudness corresponding to the target audio data packet, including:

[0272] The loudness corresponding to the target audio data packet is used as the input of a preset function expression to calculate the output value;

[0273] The product of the output value and the reference redundancy is determined as the target redundancy of the target audio data packet.

[0274] It can be seen that by implementing this optional embodiment, the redundancy (i.e., the target redundancy) can be re-determined in combination with the loudness corresponding to the target audio data packet, and the positive correlation between the loudness corresponding to the target audio data packet and the target redundancy can be improved, thereby making the redundancy corresponding to the audio signal with higher loudness higher, which is beneficial for the receiving terminal to restore the target audio data packet more accurately.

[0275] In an exemplary embodiment of the present application, the data sending unit 1203 transmits a target audio data packet and a target error correction code to a receiving terminal, including:

[0276] The target audio data packet and the target error correction code are packaged as a data packet to be transmitted and transmitted to the receiving terminal, so that after decoding the data packet to be transmitted, the receiving terminal detects the packet loss corresponding to the target audio data packet according to the target error correction code in the decoding result and corrects the target audio data packet according to the packet loss situation.

[0277] It can be seen that by implementing this optional embodiment, the error correction accuracy of the receiving terminal can be improved by transmitting the target error correction code related to the loudness of the target audio data packet.

[0278] In an exemplary embodiment of the present application, after decoding the data packet to be transmitted, the receiving terminal detects the packet loss corresponding to the target audio data packet according to the target error correction code in the decoding result and corrects the target audio data packet according to the packet loss situation in a specific manner:

[0279] The receiving terminal decodes the data packet to be transmitted to obtain the audio code stream and the target error correction code in the target audio data packet;

[0280] The receiving terminal detects the missing code of the audio code stream in the target audio data packet according to the target error correction code;

[0281] The receiving terminal restores the lost code according to the target error correction code to achieve error correction for the target audio data packet.

[0282] It can be seen that by implementing this optional embodiment, the receiving terminal can correct the received target audio data packet according to the target error correction code obtained by decoding. Since the target error correction code is positively correlated with the loudness of the target audio data packet, the higher the loudness of the audio data packet, the greater the probability that the audio data is correctly restored. Among them, the higher the loudness of the audio data, the greater the probability that the human voice is included. This can ensure that the loud audio signal output to the user is clear and complete, thereby improving the user experience.

[0283] In an exemplary embodiment of the present application, after the receiving terminal restores the lost code according to the target error correction code to correct the error of the target audio data packet, the method further includes:

[0284] The receiving terminal mixes all error-corrected target audio data packets to obtain a mixed signal and plays it.

[0285] It can be seen that by implementing this optional embodiment, the receiving terminal can mix and output the target audio data packet after correcting the error of the target audio data packet. The output result retains the important audio content in the meeting and discards the audio content that the human ear has poor perception ability. It can improve the message forwarding efficiency of multi-person meetings and ensure the real-time performance of the output audio signal, thereby improving the user experience.

[0286] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0287] Since the various functional modules of the data transmission device of the example embodiment of the present application correspond to the steps of the example embodiment of the above-mentioned data transmission method, for details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the above-mentioned data transmission method of the present application.

[0288] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0289] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0290] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0291] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0292] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0293] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data transmission method, characterized in that: include: Obtaining the loudness corresponding to the target audio data packet; wherein the loudness corresponding to the audio data packet is calculated in the following manner: performing frame processing on the audio code stream in the audio data packet according to a preset time length to obtain a plurality of audio frames; processing the plurality of audio frames respectively by a preset window function to obtain a plurality of reference frames; calculating power spectra corresponding to the plurality of reference frames respectively; calculating the frequency point loudness of each frequency point in the power spectrum according to the energy amplitude of each frequency point in the power spectrum; calculating the loudness weight of each frequency point in the power spectrum according to the frequency point loudness; calculating the weighted sum between the energy amplitude of each frequency point in the power spectrum and the loudness weight of each frequency point in the power spectrum as the loudness value of the reference frame corresponding to the power spectrum; and determining the sum of the loudness values ​​of the plurality of reference frames as the loudness corresponding to the audio data packet; Determining a target preset interval in which the packet loss rate fed back by the receiving terminal is located from a plurality of preset intervals; wherein the packet loss rate corresponds to a historical unit time closest to the sending time of the target audio data packet; Select the reference redundancy corresponding to the target preset interval; The loudness corresponding to the target audio data packet is used as an input of a preset function expression to calculate an output value; wherein the preset function expression is a monotonically increasing function, and the output value range of the preset function expression is [0, 1]; Determine the product of the output value and the reference redundancy as the target redundancy of the target audio data packet; Calculating a target error correction code for the target audio data packet according to the target redundancy, wherein the target error correction code is positively correlated with the loudness corresponding to the target audio data packet; The target audio data packet and the target error correction code are transmitted to a receiving terminal.

2. The method according to claim 1, characterized in that Before obtaining the loudness corresponding to the target audio data packet, the method further includes: A target audio data packet whose audio characteristics meet preset conditions is screened from the received multiple audio data packets.

3. The method according to claim 2, characterized in that The audio data packet includes the loudness corresponding to the audio data packet, an audio code stream and audio features corresponding to the audio code stream. The audio features corresponding to the audio code stream include energy distribution corresponding to the audio code stream and energy amplitude corresponding to each frequency point in the audio code stream.

4. The method according to claim 2, characterized in that: Before screening target audio data packets whose audio features meet preset conditions from the received multiple audio data packets, the method further includes: Obtaining error correction codes corresponding to the multiple audio data packets respectively; Performing packet loss detection on the multiple audio data packets according to the error correction code to obtain packet loss rates corresponding to the multiple audio data packets respectively; The packet loss rate is fed back to the sending terminal and errors of the multiple audio data packets are corrected according to the error correction code.

5. The method according to claim 3, characterized in that: The received multiple audio data packets are sent by a sending terminal; The sending terminal sends the multiple audio data packets in the following manner: The sending terminal collects an audio signal and extracts features from the audio signal to obtain the audio features; The sending terminal encodes the audio signal to obtain the audio code stream; The sending terminal packages the audio code stream and the audio features into the audio data packet and sends the audio data packet to the server.

6. The method according to claim 3, characterized in that The preset condition includes a preset energy amplitude and / or a preset signal-to-noise ratio, and selecting a target audio data packet whose audio characteristics meet the preset condition from the received multiple audio data packets includes: If it is detected that there is at least one energy amplitude greater than the preset energy amplitude in the audio features of the audio code stream, the audio data packet to which the audio code stream belongs is determined as the target audio data packet; and / or, If it is detected that at least one signal-to-noise ratio greater than the preset signal-to-noise ratio exists in the audio features of the audio code stream, the audio data packet to which the audio code stream belongs is determined as the target audio data packet.

7. The method according to claim 3, characterized in that The preset window function is a Hanning window function, a Hamming window function, a Blackman window function, a Kaiser window function, a triangular window function or a rectangular window function.

8. The method according to any one of claims 1 to 7, characterized in that: Transmitting the target audio data packet and the target error correction code to a receiving terminal includes: The target audio data packet and the target error correction code are packaged as a data packet to be transmitted and transmitted to a receiving terminal, so that after decoding the data packet to be transmitted, the receiving terminal detects the packet loss situation corresponding to the target audio data packet according to the target error correction code in the decoding result and corrects the target audio data packet according to the packet loss situation.

9. The method according to claim 8, characterized in that The receiving terminal detects the packet loss situation corresponding to the target audio data packet according to the target error correction code in the decoding result after decoding the data packet to be transmitted, and performs error correction on the target audio data packet according to the packet loss situation in a specific manner: The receiving terminal decodes the data packet to be transmitted to obtain the audio code stream and the target error correction code in the target audio data packet; The receiving terminal detects the missing code of the audio code stream in the target audio data packet according to the target error correction code; The receiving terminal restores the lost code according to the target error correction code to achieve error correction for the target audio data packet.

10. The method according to claim 9, characterized in that After the receiving terminal restores the lost code according to the target error correction code to correct the error of the target audio data packet, the method further includes: The receiving terminal mixes all error-corrected target audio data packets to obtain a mixed signal and plays it.

11. A data transmission system, characterized in that: It includes a sender terminal, a server and a receiver terminal, wherein: The sending terminal is used to send a target audio data packet to the server, wherein the target audio data packet includes a loudness corresponding to the target audio data packet; wherein the loudness corresponding to the audio data packet is calculated in the following manner: performing frame processing on the audio code stream in the audio data packet according to a preset time length to obtain a plurality of audio frames; processing the plurality of audio frames respectively by a preset window function to obtain a plurality of reference frames; calculating power spectra corresponding to the plurality of reference frames respectively; calculating the frequency point loudness of each frequency point in the power spectrum according to the energy amplitude of each frequency point in the power spectrum; calculating the loudness weight of each frequency point in the power spectrum according to the frequency point loudness; calculating a weighted sum between the energy amplitude of each frequency point in the power spectrum and the loudness weight of each frequency point in the power spectrum as the loudness value of the reference frame corresponding to the power spectrum; and determining the sum of the loudness values ​​of the plurality of reference frames as the loudness corresponding to the audio data packet; The server is used to receive the target audio data packet and obtain the loudness corresponding to the target audio data packet; determine the target preset interval in which the packet loss rate fed back by the receiving terminal is located from multiple preset intervals; wherein the packet loss rate corresponds to the historical unit time closest to the sending time of the target audio data packet; select the reference redundancy corresponding to the target preset interval; use the loudness corresponding to the target audio data packet as the input of a preset function expression to calculate the output value; wherein the preset function expression is a monotonically increasing function, and the output value range of the preset function expression is [0,1]; determine the product of the output value and the reference redundancy as the target redundancy of the target audio data packet; calculate the target error correction code of the target audio data packet according to the target redundancy, and the target error correction code is positively correlated with the loudness corresponding to the target audio data packet; transmit the target audio data packet and the target error correction code to the receiving terminal; The receiving terminal is used to detect the packet loss situation corresponding to the target audio data packet according to the target error correction code.

12. A data transmission device, characterized in that: include: a loudness acquisition unit, configured to acquire the loudness corresponding to the target audio data packet; wherein the loudness corresponding to the audio data packet is calculated by: performing frame processing on the audio code stream in the audio data packet according to a preset time length to obtain a plurality of audio frames; processing the plurality of audio frames respectively by a preset window function to obtain a plurality of reference frames; calculating power spectra corresponding to the plurality of reference frames respectively; calculating the frequency point loudness of each frequency point in the power spectrum according to the energy amplitude of each frequency point in the power spectrum; calculating the loudness weight of each frequency point in the power spectrum according to the frequency point loudness; calculating the weighted sum between the energy amplitude of each frequency point in the power spectrum and the loudness weight of each frequency point in the power spectrum as the loudness value of the reference frame corresponding to the power spectrum; and determining the sum of the loudness values ​​of the plurality of reference frames as the loudness corresponding to the audio data packet; An error correction code calculation unit is used to determine a target preset interval in which a packet loss rate fed back by a receiving terminal is located from a plurality of preset intervals; wherein the packet loss rate corresponds to a historical unit time closest to the sending time of the target audio data packet; select a reference redundancy corresponding to the target preset interval; use the loudness corresponding to the target audio data packet as an input of a preset function expression to calculate an output value; wherein the preset function expression is a monotonically increasing function, and the output value range of the preset function expression is [0,1]; determine the product of the output value and the reference redundancy as the target redundancy of the target audio data packet; calculate a target error correction code for the target audio data packet according to the target redundancy, and the target error correction code is positively correlated with the loudness corresponding to the target audio data packet; The data sending unit is used to transmit the target audio data packet and the target error correction code to the receiving terminal, so that the receiving terminal detects the packet loss situation corresponding to the target audio data packet according to the target error correction code.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1-10 by executing the executable instructions.

Images