Audio buffer transmission control method, electronic equipment and storage medium

Abstract

The invention discloses an audio buffer transmission control method, electronic equipment and a storage medium. The method comprises the following steps: temporarily storing a received audio data packet into a receiving buffer area; according to a carrying interval, the audio data packets in the receiving buffer area are carried to a playing buffer area in sequence, and the carrying interval represents a time interval when two adjacent audio data packets are carried to the playing buffer area from the receiving buffer area; based on the number of current remaining packets in the receiving buffer area, adjusting playing influence parameters about the audio data packets in the playing buffer area, the number of the current remaining packets representing the number of the remaining audio data packets in the current receiving buffer area, and the number of the current remaining packets representing the number of the remaining audio data packets in the receiving buffer area; the playing influence parameter comprises at least one of the following parameters: a carrying interval and a playing speed of the audio data packet in the playing buffer area. According to the scheme, the playing quality of the audio data can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of audio transmission, and in particular to an audio buffer transmission control method, an electronic device, and a storage medium. Background Art

[0002] In the prior art, the instability of the transmission channel causes the time when the audio data is received by the audio receiving end to be uneven during the transmission of the audio data, which will cause the audio data to be played back. For example, in network intercom, there is a certain delay in the transmission of the audio data packet sent by the audio transmitting end through the network. Because the network state is constantly changing, the time when different audio data packets sent arrive at the audio receiving end is uneven. If the audio receiving end does not process it at this time and adopts the simplest strategy (play if there is audio data, and do not play if there is no audio data), it will cause the audio data packet to be played back more seriously. Or the audio receiving end adopts a caching strategy (such as caching 3s of audio data packets, which will cause some unnecessary communication delays. Therefore, there is an urgent need for a new audio data transmission method in the market to solve the above problems. Summary of the invention

[0003] The present application at least provides an audio buffer transmission control method, an electronic device, and a storage medium, which can improve the playback quality of audio data.

[0004] The first aspect of the present application provides an audio buffer transmission control method, the method comprising: temporarily storing received audio data packets in a receiving buffer; sequentially transferring the audio data packets in the receiving buffer to the playing buffer according to a transfer interval, wherein the transfer interval represents the time interval between two adjacent audio data packets being transferred from the receiving buffer to the playing buffer; and adjusting the playback influencing parameters of the audio data packets in the playing buffer based on the current number of remaining packets in the receiving buffer, wherein the current number of remaining packets represents the number of audio data packets remaining in the current receiving buffer, and the playback influencing parameters include at least one of the following: the transfer interval and the playback speed of the audio data packets in the playing buffer.

[0005] Among them, based on the current remaining number of packets in the receiving buffer, the playback influencing parameters of the audio data packets in the playback buffer are adjusted, including: using the current remaining number of packets and the current total number of transfers to determine the current average remaining number of packets, wherein the current total number of transfers represents the total number of audio data packets that have been transferred during this audio reception process; comparing the current average remaining number of packets with the current remaining number of packets to obtain a comparison result; based on the comparison result, adjusting the playback influencing parameters of the audio data packets in the playback buffer.

[0006] Among them, the current average number of remaining packages and the current total number of transfers are used to determine the current average number of remaining packages, including: obtaining the historical total number of remaining packages by multiplying the previous average number of remaining packages by the previous total number of transfers; obtaining the ratio of the sum of the historical total number of remaining packages and the current number of remaining packages to the current total number of transfers as the current average number of remaining packages.

[0007] Wherein, based on the comparison result, adjusting the playback influencing parameters of the audio data packets in the playback buffer includes at least one of the following steps: in response to the comparison result that the current number of remaining packets is greater than the current average number of remaining packets of a first multiple, adjusting the playback influencing parameters to speed up the playback of the audio data packets in the playback buffer; in response to the comparison result that the current number of remaining packets is less than the current average number of remaining packets of a second multiple, adjusting the playback influencing parameters to slow down the playback of the audio data packets in the playback buffer, the first multiple being greater than the second multiple; in response to the comparison result that the current number of remaining packets is between the current average number of remaining packets of the second multiple and the current average number of remaining packets of the first multiple, not adjusting the playback influencing parameters.

[0008] Among them, the playback influencing parameters include the transport interval and the playback speed of the audio data packets in the playback buffer; adjusting the playback influencing parameters to speed up the playback of the audio data packets in the playback buffer includes: determining the playback adjustment multiple, and based on the playback adjustment multiple, reducing the transport interval and increasing the playback speed; adjusting the playback influencing parameters to slow down the playback of the audio data packets in the playback buffer includes: obtaining the playback adjustment ratio, and based on the playback adjustment ratio, increasing the transport interval and reducing the playback speed.

[0009] Wherein, determining the playback adjustment multiple includes: obtaining the ratio between the number of interruption packets per unit time and the preset catch-up time as the playback adjustment multiple, wherein the interruption packet is an audio data packet sent by the audio transmitter during the network interruption period; and / or the playback adjustment multiple is greater than 1; based on the playback adjustment multiple, reducing the transport interval and increasing the playback speed, including: taking the product of the transport interval and the reciprocal of the playback adjustment multiple as the adjusted transport interval, and taking the product of the playback speed and the playback adjustment multiple as the adjusted playback speed; and / or obtaining the playback adjustment ratio, including: obtaining the ratio of the transport interval and the playback speed before adjustment as the playback adjustment ratio; and / or the playback adjustment ratio is less than 1; based on the playback adjustment ratio, increasing the transport interval and reducing the playback speed, including: the ratio between the increase value of the transport interval and the decrease value of the playback speed is the playback adjustment ratio.

[0010] Before the audio data packets in the receiving buffer are sequentially transferred to the playing buffer according to the transfer interval, the method includes: receiving audio acquisition data from the audio transmitting end; and determining an initial value of the transfer interval based on the audio acquisition data.

[0011] The audio acquisition data includes: audio data packet capacity, sampling rate and bit number. Based on the audio acquisition data, the initial value of the transport interval is determined, including: using the ratio between the audio data packet capacity and the bit number to obtain the number of sampling points of the audio data packet; using the ratio between the number of sampling points and the sampling rate to obtain the initial value of the transport interval.

[0012] Among them, before the audio data packets in the receiving buffer are sequentially transferred to the playing buffer according to the transfer interval, it includes: obtaining the receiving time of the received audio data packets; based on the receiving time, determining the transfer delay of this audio reception, wherein the transfer delay represents the waiting time for transferring the first audio data packet in the receiving buffer to the playing buffer after receiving the first audio data packet in this audio reception process; the method also includes: obtaining the previous minimum number of remaining packets; selecting the smaller value between the previous minimum number of remaining packets and the current number of remaining packets as the current minimum number of remaining packets; in response to the current minimum number of remaining packets being greater than a preset adjustment value, adjusting the transfer delay, and using the adjusted transfer delay as the transfer delay for the next audio reception process.

[0013] In which, in response to the current minimum number of remaining packets being greater than a preset adjustment value, the initial value of the transport delay is adjusted, including: obtaining the adjusted transport delay by multiplying the current minimum number of remaining packets, the transport interval and a preset coefficient, wherein the preset coefficient is adjusted based on the current average number of remaining packets, and the current average number of remaining packets represents the number of audio data packets remaining in the receiving buffer at each current transport in this audio receiving process; and / or, using the adjusted transport delay as the transport delay for the next audio receiving process is performed when the adjusted transport delay does not meet a change condition; after adjusting the transport delay in response to the current minimum number of remaining packets being greater than the preset adjustment value, it also includes: in response to the adjusted transport delay meeting a change condition, using the sum of the adjusted transport delay and the transport interval as the transport delay for the next audio receiving process, wherein the change condition at least includes that the adjusted transport delay is less than the preset delay value and that the receiving buffer does not receive the audio data packet within the preset receiving time in the current audio receiving process.

[0014] A second aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the audio buffer transmission control method in the first aspect.

[0015] A third aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, and when the program instructions are executed by a processor, the audio buffer transmission control method in the first aspect is implemented.

[0016] In the above scheme, during the audio reception process, after the audio receiving end receives the audio data packet sent by the audio sending end, it temporarily stores the audio data packet in the receiving buffer, and sequentially transfers the audio data packets in the receiving buffer to the playing buffer according to the transfer interval, and then dynamically adjusts the playback influencing parameters of the audio data packets in the playing buffer according to the current number of remaining packets in the receiving buffer, so as to reduce the occurrence of jamming, blank space, etc. in the audio data during the playback process, thereby improving the quality of audio playback.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0019] Figure 1 This is a structural diagram of an embodiment of an audio transmission scenario of the present application;

[0020] Figure 2 It is a flowchart of an embodiment of the audio buffer transmission control method of the present application;

[0021] Figure 3 It is a flowchart of an embodiment of a method for adjusting the transport delay of the present application;

[0022] Figure 4 It is a flowchart of another embodiment of the audio buffer transmission control method of the present application;

[0023] Figure 5 It is a schematic diagram of the framework of an embodiment of the audio buffer transmission control device of the present application;

[0024] Figure 6 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0025] Figure 7 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0026] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0027] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0028] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0029] The present application is applied to the field of audio transmission to solve the problem of audio playback jamming at the audio receiving end. Specifically, in the network intercom scenario, due to the instability of network transmission, the time for transmitting audio data packets to the audio receiving end through fixed-interval sampling points is uneven, resulting in a more serious jamming problem when the audio receiving end plays. At this time, there are two conventional playback strategies for the decoding and playback module of the audio receiving end: a. Regardless of whether there is data accumulation, only process audio data according to the set frequency; b. If there is no data accumulation, play as much audio data as received, and if there is data accumulation, increase the playback speed.

[0030] The above two strategies have the following problems respectively: a. As the playback time increases, the playback delay will gradually increase. Even if the network status is restored later, the delay cannot be restored; b. The playback speed is uneven, and there are frequent playback blanks and fast-forwarding.

[0031] This application sets a two-level buffer strategy consisting of a receiving buffer and a playing buffer at the audio receiving end to solve the above problems. For details, please refer to Figure 1 The audio transmitting end 110 uses an audio collecting device to collect audio to obtain audio data, encodes the audio data to obtain an audio data packet, and transmits the audio data packet to the audio receiving end 120 via network transmission. The audio receiving end 120 receives the audio data packet, temporarily stores the audio data packet in a secondary buffer structure, sends the audio data packet to the decoding module at a uniform speed through the secondary buffer structure for decoding, and plays the decoded audio data packet using a speaker, thereby completing the final decoding and playback. The following will specifically describe the processing and transmission process of the audio data packet in the secondary buffer structure.

[0032] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of the audio buffer transmission control method of the present application. Specifically, it may include the following steps:

[0033] Step S210: temporarily storing the received audio data packet in a receiving buffer.

[0034] In some embodiments, the audio data of the network transmission between the audio transmitting terminal 110 and the audio receiving terminal 120 has unevenness in time, but the general mechanism can ensure that the data can be completely sent from the audio transmitting terminal 110 to the audio receiving terminal 120 and the relative order of sending can also be guaranteed, for example, TCP (Transmission Control Protocol, transmission control protocol) mechanism can be adopted. Therefore, in the present embodiment, the timing of the audio data packet arriving at the receiving buffer is correct, but the time interval is unstable. And reduce the phenomenon of uneven audio data packet playback speed, the audio receiving terminal 120 temporarily stores the received audio data packet in the receiving buffer, and is not directly sent to the decoding and playing module. After the audio data packet reaches a certain number in the receiving buffer, the video and audio schoolbag is uniformly transmitted to the playing buffer.

[0035] In some embodiments, the audio transmitting end 110 obtains audio data through an audio acquisition device, encodes the audio data to obtain an audio data packet, and transmits the audio data packet to the audio receiving end 120 through a network transmission. In this embodiment, in order to process data at the audio receiving end 120, the audio transmitting end 110 of the audio data is not subject to too many restrictions. Therefore, after the audio transmitting end 110 collects the audio data, it will send the collected audio data to the audio receiving end 120 in a timely manner, and the audio receiving end 120 sends the audio data packet from the receiving buffer to the playing buffer according to the transport interval. In order to avoid the situation where the transport interval is large and the audio acquisition data of the audio transmitting end 110 is different from each other, resulting in blank spaces, etc.; or, the transport interval is small and the audio acquisition data of the audio transmitting end 110 is small, resulting in serious accumulation of audio data packets in the receiving buffer and long playback delay. Among them, the audio acquisition data represents the data related to the frequency of the audio data collected by the audio transmitting end 110. Therefore, in this embodiment, the transport interval of the audio receiving end 120 needs to be associated with the audio acquisition data of the audio transmitting end 110. Specifically, please refer to step S211 to step S212.

[0036] Step S211: receiving audio acquisition data from the audio sending end.

[0037] The audio acquisition data may include: audio data packet capacity, sampling rate and number of bits.

[0038] In some embodiments, the audio transmitting end 110 may write the audio acquisition data in the header byte position of the audio data packet when sending the audio data packet to the audio receiving end 120. Specifically, to avoid wasting resources, the audio acquisition data may be written in the header byte of the first or first few audio data packets sent by the audio transmitting end 110, and the audio receiving end 120 may parse the first audio data packet after receiving it, thereby obtaining the audio acquisition data. In addition, in addition to the above-mentioned header byte position, the audio acquisition data may also be written in other fixed positions of the audio data packet, which is not specifically limited herein.

[0039] In other embodiments, when the audio transmitting end 110 establishes a connection with the audio receiving end 120 , the audio transmitting end 110 sends the audio collection data to the audio receiving end 120 via a transmission protocol between the audio transmitting end 110 and the audio receiving end 120 .

[0040] Step S212: Determine the initial value of the transport interval based on the audio collection data.

[0041] In some embodiments, the ratio between the audio data packet capacity and the number of bits can be used to obtain the number of sampling points of the audio data packet. The ratio between the number of sampling points and the sampling rate can then be used to obtain the initial value of the transport interval. Specifically, the following formula can be used:

[0042]

[0043] Where m represents the capacity of the audio data packet, w represents the number of bits occupied by each sampling point, k represents the number of sampling points in the audio data packet, and f s represents the sampling rate, Indicates the transport interval T Step The initial value of .

[0044] In addition, before moving the audio data packet from the receiving buffer to the playing buffer, it is also necessary to determine how many audio data packets the receiving buffer must receive before moving the first audio data packet to the playing buffer. Specifically, the receiving time of the received audio data packet can be obtained; based on the receiving time, the transport delay of this audio reception is determined, wherein the transport delay represents the waiting time for moving the first audio data packet in the receiving buffer to the playing buffer after receiving the first audio data packet in this audio reception process. For example, set the receiving buffer to move the first audio data packet to the playing buffer after receiving 3 audio data packets. First, obtain the receiving times t1, t2, and t3 of the first three audio data packets, and use the receiving times to determine the transport delay T Delay =t3-t1, wherein the number of received audio data packets may be 4, 5, 6, etc., in addition to 3, and is not specifically limited here. For another example, the transport delay T DelaySet to the default value. When the receiving buffer receives the first audio data packet, the receiving time of the first audio data packet is determined to be t1. At t1+T Delay After the time, the first audio data packet is transferred to the play buffer. It is understandable that the transfer delay can be determined in other ways besides the above two ways, which are not specifically limited here.

[0045] In some embodiments, after receiving the audio data packet, the audio receiving end 120 may convert the data format of the received audio data packet so that the converted data format is a data format that can be played by the audio receiving end 120. In addition, the audio data packet received by the audio receiving end 120 may also be pre-processed by audio signal processing and feature extraction, which is not specifically limited here.

[0046] Step S220: The audio data packets in the receiving buffer are sequentially transferred to the playing buffer according to the transfer interval, wherein the transfer interval represents the time interval between two adjacent audio data packets being transferred from the receiving buffer to the playing buffer.

[0047] In some embodiments, the audio data packets temporarily stored in the receiving buffer are arranged according to the sending order of the audio transmitting end 110, so the receiving buffer also moves the audio data packets to the playing buffer in this order. Further, after the receiving buffer moves the first audio data packet to the playing buffer, the audio receiving end 120 moves the audio data packets temporarily stored in the receiving buffer to the playing buffer in order according to the moving interval. Among them, the way of moving the audio data packets from the receiving buffer to the playing buffer can be wired transmission, wireless transmission, etc., which is not specifically limited here.

[0048] Step S230: Based on the current number of remaining packets in the receiving buffer, adjust the playback influencing parameters of the audio data packets in the playback buffer, wherein the current number of remaining packets represents the number of audio data packets remaining in the current receiving buffer, and the playback influencing parameters include at least one of the following: a transport interval and a playback speed of the audio data packets in the playback buffer.

[0049] In some embodiments, in the process of transferring audio data packets from the receiving buffer to the playing buffer, the network transmission between the audio transmitting terminal 110 and the audio receiving terminal 120 is not stable, which will cause the number of audio data packets temporarily stored in the receiving buffer to be too many or empty. Therefore, in order to adapt to the network transmission status between the audio transmitting terminal 110 and the audio receiving terminal 120, the transfer interval can be dynamically adjusted. For example, the upper and lower receiving thresholds can be set in the audio receiving terminal 120. When the current number of remaining packets in the receiving buffer is greater than or equal to the upper threshold, the transfer interval is reduced by multiples, which can be reduced to 1 / 2 of the original transfer interval. etc.; when the current number of remaining packets in the receiving buffer is less than or equal to the lower threshold, the transfer interval is increased by multiples, which can be reduced to 2 times, 3 times, 4 times, etc. of the original transfer interval. Among them, the upper threshold and / or the lower threshold can be the same as the number of audio data packets set when triggering the transfer of the first audio data packet to the playback buffer. For another example, a mapping relationship between network transmission and the transfer interval can be established, and the transfer interval can be dynamically adjusted based on the mapping relationship. It can be understood that in addition to the above two methods, the method for adjusting the transfer interval can also be other methods, which are not specifically limited here.

[0050] In other embodiments, when the current number of remaining packets in the receiving buffer is too small, the network transmission speed is at a slower stage in the entire transmission process, and there is a greater risk of disconnection (or blank playback); when the current number of remaining packets in the receiving buffer is too large, the network transmission speed is at a faster stage in the entire transmission process, and there is almost no risk of disconnection. In addition, if the disconnection is caused by network fluctuations, when the network is restored, the audio receiving end 120 will receive the audio data packet sent by the audio transmitting end 110 during the disconnection, and the receiving buffer will continue to be at a higher level, resulting in an increase in playback delay. Therefore, it is necessary to control the playback speed of the decoding and playback module to further avoid disconnection. Specifically, refer to steps S231 to S233.

[0051] Step S231: Determine the current average number of remaining packets using the current number of remaining packets and the current total number of transfers, wherein the current total number of transfers represents the total number of audio data packets that have been transferred during the current audio reception process.

[0052] In some embodiments, when the audio data packets in the receiving buffer are transferred to the playing buffer, the audio receiving end 120 will record the current number of remaining packets and the current total number of transfers in the receiving buffer, and use the current number of remaining packets and the current total number of transfers to obtain the current initial average number of remaining packets, and use the current initial average number of remaining packets and all previous average numbers of remaining packets to obtain the current average number of remaining packets. Specifically, refer to the following formula:

[0053]

[0054] Among them, P n is the current number of remaining packages, n is the current total number of transports, Average` n is the current initial average number of remaining packets, Average n is the current average number of remaining packets, Average1, Average2, …, Average n-1 is the average number of remaining packages from all previous ones.

[0055] In other embodiments, the product of the previous average number of remaining packages and the previous total number of transports can be used to obtain the historical total number of remaining packages. The ratio of the sum of the historical total number of remaining packages and the current number of remaining packages to the current total number of transports is obtained as the current average number of remaining packages. Specifically, refer to the following formula:

[0056] Average0=0

[0057]

[0058] Furthermore, when calculating the Average n After that, you can select Average1, Average2, …, Average n Select the median as the current average number of remaining packages.

[0059] It is understandable that the method for obtaining the current average number of remaining packets may be other methods besides the above method, which is not specifically limited here.

[0060] Step S232: Compare the current average number of remaining packets with the current number of remaining packets to obtain a comparison result.

[0061] Step S233: Based on the comparison result, adjusting the playback influencing parameters of the audio data packets in the playback buffer.

[0062] In some embodiments, when the comparison result is that the current number of remaining packets is greater than the current average number of remaining packets of the first multiple, the playback impact parameter can be adjusted to speed up the playback of the audio data packets in the playback buffer. n ≥2Average n At this time, it is considered that the receiving buffer is sending audio data packets that are piled up, and the playback influencing parameter should be adjusted to speed up the playback of the audio data packets in the playback buffer. It can be understood that the first multiple can be 0.5, 0.8, 3, 5, etc., which is not specifically limited here.

[0063] Specifically, the playback adjustment multiple can be determined, and based on the playback adjustment multiple, the transfer interval is reduced and the playback speed is increased. First, the ratio between the number of interruption packets per unit time and the preset catch-up time is obtained as the playback adjustment multiple, wherein the interruption packet is an audio data packet sent by the audio transmitter 110 during the network interruption, and the playback adjustment multiple is greater than 1. Then, the product of the transfer interval and the reciprocal of the playback adjustment multiple is used as the adjusted transfer interval, and the product of the playback speed and the playback adjustment multiple is used as the adjusted playback speed. For example, because the network status is poor and causes a 1s interruption, it is hoped that the playback will catch up with the progress of the audio transmitter 110 as soon as possible through 3 seconds of fast playback, that is, it takes 3 seconds to complete the sending of 4s frames. Therefore, the playback adjustment multiple is The adjusted playback speed is The original playback speed, the adjusted transfer interval is

[0064] In some embodiments, when the comparison result is that the current number of remaining packets is less than the second multiple of the current average number of remaining packets, the playback impact parameter may also be adjusted to slow down the playback of the audio data packets in the playback buffer, and the first multiple is greater than the second multiple. For example, the second multiple is When the network status is poor, the playback impact parameter should be adjusted to slow down the playback of the audio data packets in the playback buffer. It can be understood that the second multiple can be 1.2, etc., are not specifically limited here.

[0065] Specifically, a playback adjustment ratio can be obtained, and based on the playback adjustment ratio, the transport interval is increased and the playback speed is decreased. First, the ratio of the transport interval and the playback speed before adjustment is obtained as the playback adjustment ratio, wherein the playback adjustment ratio is less than 1. Then, the ratio between the increase value of the transport interval and the decrease value of the playback speed is always controlled to be the playback adjustment ratio. In addition, the playback adjustment ratio can be set in advance, for example, to 5%, that is, the ratio of the increase value of the transport interval and the decrease value of the playback speed is 5%. During the adjustment process, the ratio remains unchanged, but the increase value of the transport interval and the decrease value of the playback speed change.

[0066] In some embodiments, when the comparison result is that the current number of remaining packets is between the second multiple of the current average number of remaining packets and the first multiple of the current average number of remaining packets, it is considered to be a normal fluctuation, and the playback influencing parameter is not adjusted.

[0067] See also Figure 3 , Figure 3 This is a flow chart of an embodiment of the method for adjusting the transport delay of the present application. After completing an audio reception, preparations can be made for the next audio reception based on the situation of the current audio reception. Specifically, the following steps are included:

[0068] Step S310: Obtain the last minimum number of remaining packets.

[0069] In some embodiments, after completing the audio reception this time, the cache situation and the handling situation in the secondary buffer structure in this audio reception can be used as a reference for the next audio reception, so that the next audio reception and audio playback process are smoother. Specifically, the minimum number of remaining packets in the receiving buffer during this audio reception process can be obtained, and the adjustment direction of the handling delay of the next audio reception process is determined based on the minimum number of remaining packets. However, before completing the audio reception this time, after the audio transmitting terminal 110 stops sending audio data packets to the audio receiving terminal 120, the number of packets remaining in the receiving buffer during the handling process is constantly decreasing until the number of packets in the receiving buffer is 0. If the number of remaining packets in this process is taken into account, the minimum number of remaining packets is ultimately 0. Therefore, after the audio transmitting terminal 110 stops sending audio data packets to the audio receiving terminal 120, the minimum number of remaining packets recorded is not of practical reference significance. Therefore, the "last minimum number of remaining packets" at this time is the minimum number of remaining packets recorded before the audio transmitting terminal 110 stops sending audio data packets to the audio receiving terminal 120.

[0070] Specifically, at the beginning of this audio reception, the initial minimum number of remaining packets Min0 can be set to the preset maximum value. After the first transfer occurs, the smaller value is selected from the first current number of remaining packets P1 and Min0 recorded as Min1. Repeat the above steps to get the previous minimum number of remaining packets Min n-1 .

[0071] Step S320: Select the smaller value between the last minimum number of remaining packages and the current number of remaining packages as the current minimum number of remaining packages.

[0072] In some embodiments, after the audio transmitting terminal 110 stops sending audio data packets to the audio receiving terminal 120, the current number of remaining packets gradually decreases until it reaches 0. Therefore, after the audio transmitting terminal 110 stops sending audio data packets to the audio receiving terminal 120, the current number of remaining packets recorded has no practical reference significance. At the same time, the "last minimum number of remaining packets" is the minimum number of remaining packets recorded before the audio transmitting terminal 110 stops sending audio data packets to the audio receiving terminal 120. Therefore, the "current number of remaining packets" at this time is the current number of remaining packets recorded when the audio transmitting terminal 110 stops sending audio data packets to the audio receiving terminal 120. Then select the smaller value between the last minimum number of remaining packets and the current number of remaining packets as the current minimum number of remaining packets:

[0073] Min n =min(Min n-1 ,P n ) (6).

[0074] In other embodiments, the minimum "current remaining number of packets" can be selected from all the recorded "current remaining number of packets" before the audio transmitting end 110 stops sending audio data packets to the audio receiving end 120 as the current remaining number of packets at this time: n =min(P1,P2,…,P n ). Then use Formula 5 to select the smaller value between the last minimum number of remaining packages and the current number of remaining packages as the current minimum number of remaining packages.

[0075] It is understandable that, in addition to the above two methods, other methods may be used to obtain the current number of remaining packets recorded when the audio transmitting end 110 stops sending audio data packets to the audio receiving end 120, which is not specifically limited here.

[0076] Step S330: In response to the current minimum number of remaining packets being greater than a preset adjustment value, the transport delay is adjusted, and the adjusted transport delay is used as the transport delay for the next audio receiving process.

[0077] In some embodiments, when the current minimum number of remaining packets is greater than a preset adjustment value, the adjusted transport delay can be obtained by multiplying the current minimum number of remaining packets, the transport interval, and a preset coefficient, wherein the preset coefficient is adjusted based on the current average number of remaining packets, and the current average number of remaining packets represents the number of audio data packets remaining in the receiving buffer at each transport during the current audio reception process. Specifically, refer to the following formula:

[0078]

[0079] in, is the transport delay of the next audio reception process, q is a preset coefficient, q∈(0,1).

[0080] The adjusted transport delay is used as the transport delay of the next audio reception process when the adjusted transport delay does not meet the change condition. After the transport delay is adjusted, in response to the adjusted transport delay meeting the change condition, a delay of T Step There is still a situation where no audio data packet arrives in the receiving buffer. At this time, the audio data packet in the playing buffer has also been played, and a blank playback will occur. Therefore, the sum of the adjusted transport delay and the transport interval can be used as the transport delay of the next audio receiving process to avoid frequent and short blank playback, wherein the change condition includes at least the adjusted transport delay being less than the preset delay value and the receiving buffer not receiving the audio data packet within the preset receiving time during the current audio receiving process.

[0081] In some other embodiments, the preset adjustment value is set to 0.n >0, it indicates that the transport interval in this audio reception process is too large, resulting in too many audio data packets temporarily stored in the receiving buffer, and may further cause accumulation in the receiving buffer. Therefore, based on this time as a reference, the current transport delay is adjusted to serve as the transport delay for the next audio reception process. Specifically, the preset delay increase value is added to the current transport delay. For example, the preset delay increase value can be 0.1 seconds, then the transport delay for the next audio reception process is:

[0082]

[0083] Similarly, when the current minimum number of remaining packets for the next time is greater than the preset adjustment value, the preset delay increase value can be added to the transport delay of the next audio reception process to obtain the adjusted transport delay, and the adjusted transport delay is used as the transport delay of the next audio reception process. The preset delay increase value is not specifically limited here.

[0084] It is understandable that, in addition to the above two methods, other methods may be used to adjust the transport delay, which are not specifically limited here.

[0085] See also Figure 4 , Figure 4 It is a flowchart of another embodiment of the audio buffer transmission control method of the present application.

[0086] Step S410: temporarily storing the received audio data packet in a receiving buffer.

[0087] This step is the same as the above step S210 and will not be described again here.

[0088] Step S420: In response to the audio data packets in the receiving buffer satisfying the transfer condition, the audio data packets in the receiving buffer are sequentially transferred to the playing buffer according to the transfer interval.

[0089] In some embodiments, the transport condition may be that the number of audio data packets stored in the receiving buffer is greater than or equal to the preset transport quantity. Specifically, after the audio transmitting terminal 110 establishes a connection with the audio receiving terminal 120, the audio transmitting terminal 110 sends an audio data packet to the audio receiving terminal 120, and the audio receiving terminal 120 temporarily stores the received audio data packet in the receiving buffer. When the number of audio data packets in the receiving buffer reaches the preset transport quantity, the audio data packets are transported to the playing buffer, and the subsequent transport process is transported according to the transport interval. If the end signal sent by the audio transmitting terminal 110 is received later, the audio receiving terminal 120 may not transport the audio data packets to the playing buffer according to whether the number of audio data packets in the receiving buffer meets the transport condition. Wherein, the end signal indicates the end of this audio receiving process.

[0090] In addition, the transfer condition may be that the total capacity of the audio data packets stored in the receiving buffer is within a preset transfer range of the receiving buffer. For example, the preset transfer range may be 1 / 4 of the total capacity of the receiving buffer. It is understandable that the transport conditions may be other situations and are not specifically limited here.

[0091] Step S430: Based on the current number of remaining packets in the receiving buffer, adjusting the playback influencing parameters of the audio data packets in the playback buffer.

[0092] This step is the same as the above step S230 and will not be described again here.

[0093] In the audio receiving process of this application, after the audio receiving end receives the audio data packet sent by the audio sending end, the audio data packet is temporarily stored in the receiving buffer, and the audio data packets in the receiving buffer are sequentially transferred to the playing buffer according to the transfer interval, and then the playback influencing parameters of the audio data packets in the playing buffer are dynamically adjusted according to the current number of remaining packets in the receiving buffer, so that the occurrence of jamming, blank space, etc. in the audio data during the playback process can be reduced, thereby improving the quality of audio playback.

[0094] This application can adaptively adjust the transmission delay according to different network environments and changing network environments to improve the audio playback quality under low latency. At the same time, it can also dynamically adjust the playback speed according to the number of audio data packets accumulated in the receiving buffer to ensure the real-time nature of the intercom and the output audio quality.

[0095] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0096] See also Figure 5 , Figure 5 It is a schematic diagram of the framework of an embodiment of the audio buffer transmission control device of the present application. The audio buffer transmission control device 500 includes a storage module 510, a transport module 520 and an adjustment module 530. The storage module 510 executes to temporarily store the received audio data packets in the receiving buffer. The transport module 520 executes to sequentially transport the audio data packets in the receiving buffer to the playing buffer according to the transport interval, wherein the transport interval represents the time interval between two adjacent audio data packets being transported from the receiving buffer to the playing buffer. The adjustment module 530 executes to adjust the playback influencing parameters of the audio data packets in the playing buffer based on the current number of remaining packets in the receiving buffer, wherein the current number of remaining packets represents the number of audio data packets remaining in the current receiving buffer, and the playback influencing parameters include at least one of the following: the transport interval and the playback speed of the audio data packets in the playing buffer.

[0097] In some embodiments, the adjustment module 530 executes adjustment of the playback impact parameters of the audio data packets in the playback buffer based on the current remaining number of packets in the receiving buffer, including: determining the current average remaining number of packets using the current remaining number of packets and the current total number of transfers, wherein the current total number of transfers represents the total number of audio data packets that have been transferred during this audio reception process; comparing the current average remaining number of packets with the current remaining number of packets to obtain a comparison result; and adjusting the playback impact parameters of the audio data packets in the playback buffer based on the comparison result.

[0098] In some embodiments, the adjustment module 530 uses the current number of remaining packages and the current total number of transfers to determine the current average number of remaining packages, including: using the product of the previous average number of remaining packages and the previous total number of transfers to obtain the historical total number of remaining packages; obtaining the ratio of the sum of the historical total number of remaining packages and the current number of remaining packages to the current total number of transfers as the current average number of remaining packages.

[0099] In some embodiments, the adjustment module 530 performs adjustment of playback impact parameters regarding audio data packets in the playback buffer based on the comparison result, including at least one of the following steps: in response to the comparison result that the current number of remaining packets is greater than a first multiple of the current average number of remaining packets, adjusting the playback impact parameters to speed up the playback of the audio data packets in the playback buffer; in response to the comparison result that the current number of remaining packets is less than a second multiple of the current average number of remaining packets, adjusting the playback impact parameters to slow down the playback of the audio data packets in the playback buffer, the first multiple being greater than the second multiple; in response to the comparison result that the current number of remaining packets is between the second multiple of the current average number of remaining packets and the first multiple of the current average number of remaining packets, not adjusting the playback impact parameters.

[0100] In some embodiments, the adjustment module 530 executes playback influencing parameters including a transfer interval and a playback speed of audio data packets in a playback buffer; adjusts the playback influencing parameters to speed up the playback of audio data packets in the playback buffer, including: determining a playback adjustment multiple, and based on the playback adjustment multiple, reducing the transfer interval and increasing the playback speed; adjusts the playback influencing parameters to slow down the playback of audio data packets in the playback buffer, including: obtaining a playback adjustment ratio, and based on the playback adjustment ratio, increasing the transfer interval and reducing the playback speed.

[0101] In some embodiments, the adjustment module 530 performs determination of the playback adjustment multiple, including: obtaining the ratio between the number of interruption packets per unit time and the preset catch-up time as the playback adjustment multiple, wherein the interruption packet is an audio data packet sent by the audio transmitter during the network interruption period; and / or the playback adjustment multiple is greater than 1; based on the playback adjustment multiple, reducing the transport interval and increasing the playback speed, including: taking the product of the transport interval and the reciprocal of the playback adjustment multiple as the adjusted transport interval, and taking the product of the playback speed and the playback adjustment multiple as the adjusted playback speed; and / or obtaining the playback adjustment ratio, including: obtaining the ratio of the transport interval and the playback speed before adjustment as the playback adjustment ratio; and / or the playback adjustment ratio is less than 1; based on the playback adjustment ratio, increasing the transport interval and reducing the playback speed, including: the ratio between the increase value of the transport interval and the decrease value of the playback speed is the playback adjustment ratio.

[0102] In some embodiments, before the transfer module 520 sequentially transfers the audio data packets in the receiving buffer to the playing buffer according to the transfer interval, the following steps are performed: receiving audio acquisition data from the audio transmitter; and determining an initial value of the transfer interval based on the audio acquisition data.

[0103] In some embodiments, the transport module 520 executes audio acquisition data including: audio data packet capacity, sampling rate and number of bits, and determines the initial value of the transport interval based on the audio acquisition data, including: using the ratio between the audio data packet capacity and the number of bits to obtain the number of sampling points of the audio data packet; using the ratio between the number of sampling points and the sampling rate to obtain the initial value of the transport interval.

[0104] In some embodiments, the storage module 510 executes before sequentially transferring the audio data packets in the receiving buffer to the playback buffer according to the transfer interval, including: obtaining the receiving time of the received audio data packet; determining the transfer delay of this audio reception based on the receiving time, wherein the transfer delay represents the waiting time for transferring the first audio data packet in the receiving buffer to the playback buffer after receiving the first audio data packet in this audio reception process; the method also includes: obtaining the previous minimum number of remaining packets; selecting the smaller value between the previous minimum number of remaining packets and the current number of remaining packets as the current minimum number of remaining packets; in response to the current minimum number of remaining packets being greater than a preset adjustment value, adjusting the transfer delay, and using the adjusted transfer delay as the transfer delay for the next audio reception process.

[0105] In some embodiments, the storage module 510 performs, in response to the current minimum number of remaining packets being greater than a preset adjustment value, adjusting the initial value of the transport delay, including: obtaining the adjusted transport delay by multiplying the current minimum number of remaining packets, the transport interval and a preset coefficient, wherein the preset coefficient is adjusted based on the current average number of remaining packets, and the current average number of remaining packets represents the number of audio data packets remaining in the receiving buffer at each current transport in the current audio receiving process; and / or, using the adjusted transport delay as the transport delay for the next audio receiving process is performed when the adjusted transport delay does not meet a change condition; after adjusting the transport delay in response to the current minimum number of remaining packets being greater than the preset adjustment value, it also includes: in response to the adjusted transport delay meeting a change condition, using the sum of the adjusted transport delay and the transport interval as the transport delay for the next audio receiving process, wherein the change condition includes at least that the adjusted transport delay is less than the preset delay value and that the receiving buffer does not receive the audio data packet within the preset receiving time in the current audio receiving process.

[0106] See also Figure 6 , Figure 6 : is a schematic diagram of a framework of an embodiment of an electronic device 60 of the present application. The electronic device 60 includes a memory 61 and a processor 62 coupled to each other, and the processor 62 is used to execute program instructions stored in the memory 61 to implement the steps of any of the above-mentioned audio buffer transmission control method embodiments, or to implement the steps of any of the above-mentioned transport delay adjustment method embodiments. In a specific implementation scenario, the electronic device 60 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 60 may also include mobile devices such as laptops and tablet computers, which are not limited here.

[0107] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps of any of the above-mentioned audio buffer transmission control method embodiments, or to implement the steps of any of the above-mentioned transport delay adjustment method embodiments. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0108] See also Figure 7 , Figure 7 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 70 of the present application. The computer-readable storage medium 70 stores program instructions 701 that can be executed by a processor, and the program instructions 701 are used to implement the steps of any of the above-mentioned audio buffer transmission control method embodiments, or to implement the steps of any of the above-mentioned transport delay adjustment method embodiments.

[0109] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0110] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0111] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0112] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0113] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. An audio buffer transmission control method, characterized in that: include: Temporarily store the received audio data packets in the receiving buffer; The audio data packets in the receiving buffer are sequentially transferred to the playing buffer according to a transfer interval, wherein the transfer interval represents a time interval between two adjacent audio data packets being transferred from the receiving buffer to the playing buffer; Based on the current number of remaining packets in the receiving buffer, the playback influencing parameters of the audio data packets in the playback buffer are adjusted, wherein the current number of remaining packets represents the number of audio data packets currently remaining in the receiving buffer, and the playback influencing parameters include at least one of the following: the transport interval and the playback speed of the audio data packets in the playback buffer.

2. The method according to claim 1, characterized in that The adjusting, based on the current number of remaining packets in the receiving buffer, the playback impact parameter of the audio data packets in the playback buffer comprises: Determine the current average number of remaining packets using the current number of remaining packets and the current total number of transfers, wherein the current total number of transfers represents the total number of audio data packets that have been transferred during the current audio reception process; Comparing the current average number of remaining packets with the current number of remaining packets to obtain a comparison result; Based on the comparison result, the playback influencing parameters of the audio data packets in the playback buffer are adjusted.

3. The method according to claim 2, characterized in that The method of determining the current average number of remaining packages by using the current number of remaining packages and the current total number of transports includes: The total number of remaining packages in history is obtained by multiplying the last average number of remaining packages by the last total number of transports. The ratio of the sum of the total number of remaining packages in history and the current number of remaining packages to the current total number of transport times is obtained as the current average number of remaining packages.

4. The method according to claim 2, characterized in that: The step of adjusting the playback influencing parameters of the audio data packets in the playback buffer based on the comparison result comprises at least one of the following steps: In response to the comparison result being that the current number of remaining packets is greater than the current average number of remaining packets of the first multiple, adjusting the playback influencing parameter to speed up the playback of the audio data packets in the playback buffer; In response to the comparison result being that the current number of remaining packets is less than a second multiple of the current average number of remaining packets, adjusting the playback impact parameter to slow down the playback of the audio data packets in the playback buffer, the first multiple being greater than the second multiple; In response to the comparison result that the current number of remaining packets is between the second multiple of the current average number of remaining packets and the first multiple of the current average number of remaining packets, the playback influencing parameter is not adjusted.

5. The method according to claim 4, characterized in that The playback influencing parameters include the transport interval and the playback speed of the audio data packets in the playback buffer; The adjusting the playback influencing parameter to speed up the playback of the audio data packets in the playback buffer includes: Determining a playback adjustment multiple, and based on the playback adjustment multiple, reducing the transport interval and increasing the playback speed; The adjusting the playback influencing parameter to slow down the playback of the audio data packets in the playback buffer includes: A playback adjustment ratio is obtained, and based on the playback adjustment ratio, the transport interval is increased and the playback speed is decreased.

6. The method according to claim 5, characterized in that The determining of the playback adjustment multiple includes: Obtaining a ratio between the number of interruption packets per unit time and a preset catch-up time as the playback adjustment factor, wherein the interruption packet is an audio data packet sent by the audio sending end during the network interruption period; And / or, the playback adjustment multiple is greater than 1; and based on the playback adjustment multiple, reducing the transport interval and increasing the playback speed includes: The product of the transport interval and the reciprocal of the playback adjustment multiple is used as the adjusted transport interval, and the product of the playback speed and the playback adjustment multiple is used as the adjusted playback speed; And / or, obtaining the playback adjustment ratio includes: acquiring a ratio of the transport interval to the playback speed before adjustment as the playback adjustment ratio; And / or, the playback adjustment ratio is less than 1; and based on the playback adjustment ratio, increasing the transport interval and reducing the playback speed comprises: The ratio between the increase value of the transport interval and the decrease value of the play speed is the play adjustment ratio.

7. The method according to claim 1, characterized in that Before sequentially transferring the audio data packets in the receiving buffer to the playing buffer according to the transfer interval, the method includes: Receive audio acquisition data from the audio transmitter; An initial value of the transport interval is determined based on the audio collection data.

8. The method according to claim 7, characterized in that The audio collection data includes: audio data packet capacity, sampling rate and bit number. The determining the initial value of the transport interval based on the audio collection data includes: Using the ratio between the capacity of the audio data packet and the number of bits, the number of sampling points of the audio data packet is obtained; The initial value of the transport interval is obtained by using the ratio between the number of sampling points and the sampling rate.

9. The method according to claim 1, characterized in that: Before sequentially transferring the audio data packets in the receiving buffer to the playing buffer according to the transfer interval, the method includes: Obtaining the receiving time of the received audio data packet; Based on the receiving time, determining a transport delay of the current audio reception, wherein the transport delay represents a waiting time for transporting the first audio data packet in the receiving buffer to the playing buffer after receiving the first audio data packet in the current audio reception process; The method further comprises: Get the last minimum number of remaining packages; Select the smaller value between the previous minimum number of remaining packages and the current number of remaining packages as the current minimum number of remaining packages; In response to the current minimum number of remaining packets being greater than a preset adjustment value, the transport delay is adjusted, and the adjusted transport delay is used as the transport delay for the next audio receiving process.

10. The method according to claim 9, characterized in that In response to the current minimum number of remaining packages being greater than a preset adjustment value, adjusting the initial value of the transport delay includes: The adjusted transport delay is obtained by multiplying the current minimum number of remaining packets, the transport interval and a preset coefficient, wherein the preset coefficient is adjusted based on the current average number of remaining packets, and the current average number of remaining packets represents the number of audio data packets remaining in the receiving buffer at each current transport during the current audio reception process; And / or, the using the adjusted transport delay as the transport delay of the next audio receiving process is performed when the adjusted transport delay does not meet the change condition; after the transport delay is adjusted in response to the current minimum number of remaining packets being greater than a preset adjustment value, the method further includes: In response to the adjusted transport delay satisfying a change condition, the sum of the adjusted transport delay and the transport interval is used as the transport delay for the next audio receiving process, wherein the change condition includes at least that the adjusted transport delay is less than a preset delay value and that the receiving buffer does not receive the audio data packet within a preset receiving time during the current audio receiving process.

11. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the audio buffer transmission control method according to any one of claims 1 to 10.

12. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the audio buffer transmission control method according to any one of claims 1 to 10 is implemented.

Images