Audio processing method, electronic device and storage medium in real-time call

Abstract

The present application discloses a method, electronic device, and storage medium for processing audio in a real-time call. The method includes: in response to the completion of initialization of a network receiving module and a sound card, discarding the audio buffered in the network receiving module; controlling the network receiving module to receive the user's audio and transmit the user's audio to an audio buffer; obtaining a preset length of the audio sample when the sound card periodically reads the audio buffer; and adaptively adjusting the length of the audio buffer based on the relationship between the preset length and the length of the audio sample in the current audio buffer, so that the sound card reads the audio sample from the audio buffer with the preset length. In this way, the present application can reduce the audio delay of real-time calls and improve the audio quality of real-time calls.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to an audio processing method, electronic device, and storage medium in a real-time call. Background Art

[0002] With the widespread adoption of wireless and mobile networks, real-time voice calls over these networks have become commonplace. However, due to network transmission delays and fluctuations, audio transmission is prone to delays and poor audio quality, impacting the user experience. Summary of the Invention

[0003] The main technical problem solved by this application is to provide an audio processing method, electronic device and storage medium in real-time calls, which can discard early audio in the network receiving module during the initialization phase and adjust the length of the audio buffer according to the preset length of the audio sample read by the sound card and the length of the audio sample in the audio buffer.

[0004] To solve the above technical problems, the first aspect of the present application provides an audio processing method in a real-time call, the method comprising: in response to the completion of initialization of a network receiving module and a sound card, discarding the audio cached in the network receiving module; controlling the network receiving module to receive the user's audio and transmitting the user's audio to an audio buffer; obtaining a preset length of the sound card when periodically reading audio samples in the audio buffer; adaptively adjusting the length of the audio buffer based on the relationship between the preset length and the length of the audio samples currently in the audio buffer, so that the sound card reads the audio samples from the audio buffer with the preset length.

[0005] The step of adaptively adjusting the length of the audio buffer according to the size relationship between the preset length and the length of the audio samples currently in the audio buffer comprises: determining whether the audio sample length is less than the preset length; if so, increasing a length increment for the audio buffer and controlling the sound card to play silent data equal to the preset length within the current cycle; otherwise, further determining whether the audio sample length is greater than the sum of the length increment and the preset length; if so, removing some audio samples with earlier time nodes in the audio buffer so that the length of the audio samples in the audio buffer is the sum of the length increment and the preset length; otherwise, controlling the sound card to read audio of the preset length from the audio buffer.

[0006] The step of controlling the sound card to play the silent data having the same length as the preset length in the current cycle includes: when controlling the sound card to play the silent data having the same length as the preset length in the current cycle, accelerating the playing of the silent data.

[0007] The initial value of the length increment is 0, and the step of increasing the length increment of the audio buffer includes increasing the length increment equal to the preset length to the audio buffer.

[0008] Among them, after the step of removing some audio samples with earlier time nodes in the audio buffer so that the length of the audio samples in the audio buffer is the sum of the length increment and the preset length, it also includes: when the audio sample length is greater than the sum of the length increment and the preset length and the length increment is greater than 0, subtracting the preset length from the length increment to obtain a new length increment and updating the new length increment for the audio buffer.

[0009] The length increment is less than or equal to a first threshold value times the preset length.

[0010] The audio buffer stores audio samples according to a first-in-first-out principle, and the step of removing some audio samples with earlier time nodes in the audio buffer includes: removing from the audio buffer the audio sample whose length is the difference obtained by subtracting the sum of the length increment and the preset length from the audio sample length.

[0011] The method further includes: reading parameter information of the sound card, setting a preset length for reading the audio sample for the sound card according to the parameter information, and configuring a sampling rate of the sound card.

[0012] In order to solve the above technical problems, the second aspect of the present application provides an electronic device, which includes a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the audio processing method in real-time calls of the above first aspect.

[0013] In order to solve the above technical problems, the third aspect of the present application provides a computer storage medium, on which program data is stored. When the program data is executed by a processor, the audio processing method in real-time calls of the above first aspect is implemented.

[0014] The beneficial effect of the present application is that after the network receiving module and the sound card are initialized, the audio cached in the network receiving module during the initialization phase is discarded to avoid audio delays caused by early audio cached during the initialization phase. The length of the audio buffer is adjusted according to the preset length of the audio samples read by the sound card and the length of the audio samples in the audio buffer, so that the sound card can stably read audio samples from the audio buffer with the preset length, thereby reducing the audio delay of real-time calls and improving the quality of the audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0016] Figure 1 This is a flowchart of an embodiment of a method for processing audio in a real-time call provided by the present application;

[0017] Figure 2 This is a flowchart of another embodiment of the method for processing audio in a real-time call provided by the present application;

[0018] Figure 3 This is a schematic structural diagram of an embodiment of an electronic device provided by the present application;

[0019] Figure 4 It is a structural diagram of an embodiment of a computer storage medium provided by this application. DETAILED DESCRIPTION

[0020] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0021] The terms "system" and "network" are often used interchangeably in this document. The term "and / or" is simply a description of an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " generally indicates that the related objects are in an "or" relationship. Furthermore, "multiple" in this document means two or more than two.

[0022] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a method for processing audio in a real-time call provided by the present application, the method comprising:

[0023] Step S101: in response to the network receiving module and the sound card being initialized, the audio buffered in the network receiving module is discarded.

[0024] Specifically, when the user chooses to accept the remote real-time voice call invitation, the sound card and network receiving module are initialized immediately. The sound card includes a sound card control module, a playback device and a recording device. When initializing the sound card, the sound card control module needs to be initialized. The initialization of the sound card control module takes longer than the network receiving module. The initialization process of the sound card control module takes about 50-200 milliseconds. During the initialization process, the network receiving module will receive the audio data sent by the other end, read out the early audio cached in the network protocol stack of the network receiving module during the initialization process and discard it, so as to avoid the delay caused by the initialization process from existing in the audio playback process, thereby avoiding the delay caused by the early audio received during the initialization process to the real-time call audio transmission.

[0025] Step S102: Control the network receiving module to receive the user's audio and transmit the user's audio to the audio buffer.

[0026] Specifically, after the early audio in the network receiving module is cleared, the network receiving module is controlled to continue receiving the user's audio, and then transmit the user's audio to the audio buffer.

[0027] Specifically, the network receiving module receives the user's audio and then copies the user's audio to the corresponding audio buffer. During the stage of playing the peer user's audio, the audio data in the network receiving module is copied to the audio playing buffer.

[0028] Step S103: obtaining a preset length of the audio sample periodically read from the audio buffer by the sound card.

[0029] Specifically, the playback device in the sound card periodically reads audio samples from the audio buffer for playback. The length of the audio samples read by the sound card each time can be preset, either manually or by a default value set by the mobile terminal based on the sound card's parameters. The preset length of the audio samples read from the audio buffer by the sound card is obtained and saved.

[0030] Step S104: Adaptively adjusting the length of the audio buffer according to the relationship between the preset length and the length of the audio samples in the current audio buffer, so that the sound card reads audio samples from the audio buffer with the preset length.

[0031] Specifically, when the length of the audio sample in the audio buffer is less than the preset length, the audio buffer is in an underload state. The reason for the underload of the audio buffer is probably due to the volatility of the network transmission. The current network transmission is sometimes fast and sometimes slow. Therefore, the next time the user's audio is transmitted to the audio buffer, it is likely to be a longer data packet, thereby increasing the length of the audio buffer to cope with the user's audio with a longer data length that will be transmitted from the network receiving module.

[0032] Furthermore, since the length of the audio samples in the current audio buffer is insufficient for the sound card to play a complete audio of the preset length, the sound card is controlled to directly play a segment of silent data this time, and wait for the audio samples in the audio buffer to increase. When the audio samples are read next time, the audio samples of the preset length are read from the audio buffer for playback, so that the audio playback is more coherent.

[0033] Furthermore, when the audio buffer has been underloaded multiple times, causing the length of the audio buffer to increase multiple times to more than 3 times the preset length, it is determined whether the length of the audio samples in the current audio buffer is greater than or equal to 3 times the preset length. If so, it means that the current network volatility is large and the delay is already long. For the listener, the audio quality is poor, it sounds intermittent, and some voices are unclear. In this case, some audio samples that are greater than 3 times the preset length and at an earlier time point are deleted, and only audio samples with 3 times the preset length are retained, so that the audio samples in the audio buffer will not accumulate too much and cause excessive delay. For the deleted audio samples, although the user will not be able to hear the corresponding audio, when the audio quality is poor, the listener will not be able to hear all the audio clearly even if all audio samples are played, and the audio delay will be longer. If the deleted audio sample length only includes audio below 200 milliseconds, it will not be obvious to the listener. In addition, the sound card device still reads audio samples of the preset length from the audio buffer for playback.

[0034] The audio processing method for real-time calls provided in this embodiment discards the audio cached in the network receiving module during the initialization phase after the network receiving module and the sound card are initialized to avoid audio delay caused by early audio cached during the initialization phase. The length of the audio buffer is adjusted based on the preset length of audio samples read by the sound card and the length of audio samples in the audio buffer so that the sound card can stably read audio samples from the audio buffer at the preset length, thereby reducing the audio delay of real-time calls and improving the audio quality.

[0035] See also Figure 2 , Figure 2 1 is a flow chart of another embodiment of the method for processing audio in a real-time call provided by the present application, the method comprising:

[0036] Step S201: In response to the network receiving module and the sound card being initialized, the audio buffered in the network receiving module is discarded.

[0037] Specifically, the early audio data cached during the initialization process is extracted from the network protocol stack and discarded, so that the early audio will not be received in the audio buffer and the sound card will not play the early audio to the listener. Therefore, after the network receiving module is initialized, the audio cached in the network receiving module during the initialization phase is discarded, which can reduce the audio delay caused by the network itself.

[0038] Step S202: Control the network receiving module to receive the user's audio and transmit the user's audio to the audio buffer.

[0039] Specifically, after the early audio in the network receiving module is cleared, the network receiving module waits for the user's audio to be transmitted to the network receiving module. After receiving the user's audio, the network receiving module transmits the user's audio to the audio buffer.

[0040] Optionally, the audio processing method in the real-time call further includes: reading parameter information of a sound card, setting a preset length of audio samples to be read for the sound card according to the parameter information, and configuring a sampling rate of the sound card.

[0041] Specifically, after obtaining parameter information such as the sound card model, sampling bit number, sampling frequency, etc., the control system of the mobile terminal can automatically set the optimal sampling rate and the preset length for reading audio samples according to the parameter information. In addition, the user can also modify the default values ​​of the mobile terminal and manually set the sampling rate and preset length.

[0042] Step S203: obtaining a preset length of the audio sample periodically read from the audio buffer by the sound card.

[0043] Specifically, a preset length of an audio sample read by a sound card from an audio buffer is obtained, and the preset length is saved.

[0044] Step S204: Determine whether the audio sample length is less than a preset length.

[0045] Specifically, when the length of the audio samples in the audio buffer is less than the preset length, step S205 is executed; when the length of the audio samples in the audio buffer is greater than or equal to the preset length, step S206 is executed.

[0046] Step S205: increasing the length increment of the audio buffer, and controlling the sound card to play silent data equal to the preset length in the current cycle.

[0047] Specifically, when the length of the audio samples in the audio buffer is less than a preset length, the audio buffer is in an underloaded state. Due to the volatility of wireless or mobile networks, the length of the audio data received by the network receiving module also fluctuates. When the user's audio is subsequently transmitted to the audio buffer, it is likely to be a longer data packet. Therefore, it is necessary to increase the length of the audio buffer to improve the audio buffer's ability to accommodate audio. The single increase in the length of the audio buffer can be a preset length. In addition, if the sound card cannot read an audio sample of the preset length within the current cycle, the sound card is controlled not to read audio samples from the audio buffer within the current cycle, and directly play silent data of the preset length.

[0048] Furthermore, when the sound card is controlled to play silent data of the same length as the preset length in the current cycle, the silent data is played faster, thereby reducing the audio delay caused by the increase in silent data, and also making it difficult for the listener to notice the increase in silent data.

[0049] Furthermore, an initial value of the length increment is 0, and a single length increment can be set to a preset length. Furthermore, the step of increasing the length increment of the audio buffer includes: increasing the length increment of the audio buffer by a length equal to the preset length. The length of the single length increment is equal to the preset length, so that the length increment of the audio buffer has a multiple relationship associated with the preset length.

[0050] Optionally, the length increment is less than or equal to a preset length times a first threshold. If the audio buffer experiences multiple underloads and the length increment is increased for the audio buffer each time an underload occurs, the total length increment of the audio buffer will accumulate continuously, causing the length of the audio buffer to be too long. When the network fluctuates severely, if the length increment is not restricted, the length of the audio buffer will be too long, ultimately leading to serious delays in the audio samples in the audio buffer.

[0051] In a specific application scenario, the first threshold is 3. When the audio buffer has increased by a length increment of 3 times the preset length and encounters an underload situation, the length increment of the audio buffer will no longer be increased, and the sound card will only be controlled to play silent data equal to the preset length in the current cycle.

[0052] Step S206: Determine whether the audio sample length is greater than the sum of the length increment and the preset length.

[0053] Specifically, when the audio sample length is greater than the sum of the length increment and the preset length, steps S207-S208 are executed; when the audio sample length is less than or equal to the sum of the length increment and the preset length, step S208 is executed.

[0054] Step S207: remove some audio samples with earlier time nodes in the audio buffer, so that the length of the audio samples in the audio buffer is the sum of the length increment and the preset length.

[0055] Specifically, when the length of the audio samples in the audio buffer is already long, it means that the audio delay time is relatively long at this time. Some audio samples with earlier time nodes in the audio buffer are removed, and only audio samples with time points close to the current time point and a length that is the sum of the length increment and the preset length are retained.

[0056] In one application, the audio buffer stores audio samples according to the first-in-first-out principle. When removing audio samples from the audio buffer, the length is the difference obtained by subtracting the sum of the length increment and the preset length from the audio sample length, and the audio sample that enters the audio buffer first is removed from the audio buffer.

[0057] Specifically, assuming that the length of the audio sample in the current audio buffer is length, the length increment is delta, and the preset length is play_length, when length>play_length+delta, the audio sample with the time point farthest from the current time point and the length of length-(play_length+delta) is removed from the current audio buffer, so that the audio sample closest to the current time point and the length of play_length+delta remains in the audio buffer, thereby reducing the delay caused by the excessive length of the audio samples cached in the audio buffer.

[0058] Furthermore, after the step of removing some audio samples with earlier time nodes in the audio buffer so that the length of the audio samples in the audio buffer is the sum of the length increment and the preset length, it also includes: when the audio sample length is greater than the sum of the length increment and the preset length and the length increment is greater than 0, subtracting the preset length from the length increment to obtain a new length increment and updating the new length increment for the audio buffer.

[0059] Specifically, when the length of the audio sample is greater than the sum of the length increment and the preset length, it means that the audio buffer is no longer underloaded. Then, when the current length increment is greater than 0, the current length increment is subtracted from the preset length to obtain a new length increment value, and the new length increment is updated to the audio buffer, so that the length increment of the audio buffer is gradually reduced, thereby reducing the length of the audio samples cached in the audio buffer, and thereby reducing the impact on audio transmission delay caused by the excessive length of audio samples in the audio buffer.

[0060] Step S208: Control the sound card to read audio of a preset length from the audio buffer.

[0061] Specifically, when the length of the audio samples in the audio buffer has been adjusted accordingly, the sound card is controlled to read audio of a preset length from the audio buffer. Assuming that the length of the audio samples in the current audio buffer is length, the length increment is delta, and the preset length is play_length, after adjusting the length of the audio samples, play_length≤length≤play_length+delta, the sound card is controlled to read audio of play_length from the audio buffer for playback.

[0062] The audio processing method for real-time calls provided in this embodiment discards the early audio cached in the network receiving module protocol stack during the initialization phase, and when the length of the audio sample in the audio buffer is less than the preset length of the audio sample read by the sound card, increases the length of the audio buffer to increase the length of the subsequent audio buffer that can receive user audio, thereby reducing the impact of network volatility; when the audio sample length is greater than the preset length and the length increment of the first threshold times, removes part of the audio from the audio buffer to reduce audio delay and improve the user's call experience.

[0063] See also Figure 3 , Figure 3 This is a structural diagram of an embodiment of an electronic device provided in the present application. The electronic device 30 includes a memory 301 and a processor 302 coupled to each other, wherein the memory 301 stores program data (not shown in the figure), and the processor 302 calls the program data to implement the audio processing method in real-time calls in any of the above embodiments. For an explanation of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0064] See also Figure 4 , Figure 4 This is a structural diagram of an embodiment of a computer storage medium provided in the present application. The computer storage medium 40 stores program data 400. When the program data 400 is executed by the processor, the audio processing method in real-time calls in any of the above embodiments is implemented. For an explanation of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0065] It should be noted that the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this embodiment.

[0066] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0067] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0068] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for processing audio in a real-time call, characterized in that: The method comprises: In response to completion of initialization of the network receiving module and the sound card, discarding the audio buffered in the network receiving module; Controlling the network receiving module to receive the user's audio and transmit the user's audio to the audio buffer; Obtaining a preset length of the audio sample in the audio buffer when the sound card periodically reads the audio sample; Adaptively adjusting the length of the audio buffer according to the relationship between the preset length and the length of the audio samples currently in the audio buffer so that the sound card reads the audio samples from the audio buffer with the preset length; specifically comprising: determining whether the audio sample length is less than the preset length; if so, increasing a length increment for the audio buffer and controlling the sound card to play silent data equal to the preset length in a current cycle, wherein a single length increment of the audio buffer is the preset length, and a total length increment in the audio buffer is less than or equal to a first threshold value times the preset length; when the audio buffer has increased by a length increment of the first threshold value times the preset length, no further length increment is increased for the audio buffer, and the sound card is controlled to play silent data equal to the preset length in the current cycle; otherwise, further determining whether the audio sample length is greater than the sum of the length increment and the preset length; if so, removing some audio samples with earlier time nodes in the audio buffer so that the length of the audio samples in the audio buffer is the sum of the length increment and the preset length; otherwise, controlling the sound card to read audio of the preset length from the audio buffer.

2. The method according to claim 1, characterized in that The step of controlling the sound card to play silent data having a length equal to the preset length in the current cycle includes: When the sound card is controlled to play the silent data having the same length as the preset length in the current cycle, the silent data is played faster.

3. The method according to claim 1, characterized in that After the step of removing some audio samples with earlier time nodes in the audio buffer so that the length of the audio samples in the audio buffer is the sum of the length increment and the preset length, the method further includes: When the length of the audio sample is greater than the sum of the length increment and the preset length and the length increment is greater than 0, the preset length is subtracted from the length increment to obtain a new length increment and the new length increment is updated for the audio buffer.

4. The method according to claim 1, wherein The audio buffer stores audio samples according to a first-in-first-out principle, and the step of removing some audio samples with earlier time nodes in the audio buffer includes: An audio sample whose length is the difference obtained by subtracting the sum of the length increment and the preset length from the audio sample length and which enters the audio buffer first is removed from the audio buffer.

5. The method according to claim 1, wherein Also includes: Parameter information of the sound card is read, and a preset length of the audio sample to be read is set for the sound card according to the parameter information, and a sampling rate of the sound card is configured.

6. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 5.

7. A computer storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images