Audio processing method, device, electronic device and storage medium

By filtering and selecting audio RTP packets in real-time voice conferencing system, the client's problem of handling pressure and environmental noise floor is solved, and audio quality improvement and bandwidth optimization are achieved.

CN114827101BActive Publication Date: 2024-11-22JD DIGITS HAIYI INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210387295.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-13
Publication Date
2024-11-22
Estimated Expiration
2042-04-13

AI Technical Summary

Technical Problem

In real-time voice conference scenarios, the server forwards all audio in the room without distinction, resulting in increased client processing pressure and problems such as ambient noise floor and unnecessary bandwidth transmission.

Method used

By obtaining and receiving real-time transmission protocol (RTP) packets transmitted between clients and other servers in the target server, filtering out the ambient noise floor, selecting real speaking audio, and sending it to the client, reducing unnecessary bandwidth transmission.

Benefits of technology

Improves audio quality, reduces client processing pressure and bandwidth usage, and reduces resource consumption.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114827101B_ABST
    Figure CN114827101B_ABST
Patent Text Reader

Abstract

The present disclosure provides an audio processing method, device, electronic device and storage medium, which relate to the field of data processing technology. The scheme is: obtaining at least one first audio real-time transport protocol RTP packet sent by a client belonging to a target server; receiving at least one second audio RTP packet sent by the remaining servers in the distributed network to which the target server belongs; selecting at least one candidate audio RTP packet from the first audio RTP packet and the second audio RTP packet; and sending at least one candidate audio RTP packet to the client. In the present disclosure, the audio RTP packet is filtered and selected, the environmental background noise is filtered out, and the real speaking audio is selected and sent to the client, which improves the audio quality and reduces unnecessary bandwidth transmission.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to an audio processing method, device, electronic device and storage medium. Background Art

[0002] In a real-time voice conference scenario, the client in the room will send the generated audio to the server and receive the audio transmitted by other clients in the room from the server. In related technologies, the server will indiscriminately forward all audio in the room to the client, increasing the processing pressure on the client. Summary of the invention

[0003] The present disclosure provides an audio processing method, device, electronic device and storage medium.

[0004] According to one aspect of the present disclosure, there is provided an audio processing method, comprising:

[0005] Acquire at least one first audio real-time transport protocol RTP packet sent by a client belonging to a target server;

[0006] Receiving at least one second audio RTP packet sent by remaining servers in the distributed network to which the target server belongs;

[0007] Selecting at least one candidate audio RTP packet from the first audio RTP packet and the second audio RTP packet;

[0008] Send at least one candidate audio RTP packet to the client.

[0009] In the disclosed embodiment, the audio RTP packets are filtered and selected to filter out the ambient background noise, and the real speech audio is selected to be sent to the client, thereby improving the audio quality and reducing unnecessary bandwidth transmission.

[0010] According to another aspect of the present disclosure, there is provided an audio processing device, comprising:

[0011] An acquisition module, used for acquiring at least one first audio real-time transport protocol RTP packet sent by a client belonging to a target server;

[0012] A receiving module, used for receiving at least one second audio RTP packet sent by the remaining servers in the distributed network to which the target server belongs;

[0013] A selection module, used for selecting at least one candidate audio RTP packet from the first audio RTP packet and the second audio RTP packet;

[0014] The sending module is used to send at least one candidate audio RTP packet to the client.

[0015] According to another aspect of the present disclosure, there is provided an electronic device, including a memory and a processor;

[0016] The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the audio processing method of the embodiment of the first aspect of the present disclosure.

[0017] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the audio processing method of the embodiment of the first aspect of the present disclosure is implemented.

[0018] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the audio processing method of the embodiment of the first aspect of the present disclosure is implemented.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0021] Figure 2 is a schematic diagram of a real-time voice communication system based on the real-time transport protocol RTP;

[0022] Figure 3 is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0023] Figure 4 is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0024] Figure 5 is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0025] Figure 6 is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0026] Figure 7 is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0027] Figure 8 is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0028] Fig. 9 is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0029] Fig.10 is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0030] Fig.11 is a structural diagram of an audio processing device according to an embodiment of the present disclosure;

[0031] Fig.12 is a structural diagram of an audio processing device according to an embodiment of the present disclosure;

[0032] Fig.13 The block diagram is a block diagram of an electronic device used to implement the audio processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0034] The audio processing method, device, electronic device and storage medium of the present disclosure are described below with reference to the accompanying drawings.

[0035] Figure 1 is a flowchart of an audio processing method according to an embodiment of the present disclosure, the method is applied to a target server, such as Figure 1 As shown, the method comprises the following steps:

[0036] S101, obtaining at least one first audio real-time transport protocol RTP packet sent by a client belonging to a target server.

[0037] Real-time Transport Protocol (RTP) is a network transmission protocol commonly used for streaming media-related communications and entertainment, including telephone, video conferencing, television, and network-based push-to-talk services. In the disclosed embodiment, a distributed network based on RTP is constructed, including multiple servers, which can transmit audio to each other. Each server corresponds to multiple clients, and the clients and corresponding servers can transmit audio to each other. Figure 2 As shown, the generated audio data is transmitted via the audio RTP packet.

[0038] In the distributed network constructed according to the embodiment of the present disclosure, there are multiple servers for collecting, forwarding, encoding and other processing of transmitted audio RTP packets. The target server is one of the multiple servers. The audio processing method of the present disclosure is explained below using the target server as an example.

[0039] The target server may correspond to one or more clients. During audio transmission, each client collects the generated audio signal and sends one audio RTP packet to the corresponding server. Accordingly, the target server obtains the audio RTP packets sent by all its own clients as the first audio RTP packets. The number of first audio RTP packets is the same as the number of corresponding clients, which is at least one.

[0040] Optionally, the client may be a mobile client, a computer client, etc.

[0041] S102, receiving at least one second audio RTP packet sent by remaining servers in the distributed network to which the target server belongs.

[0042] Each server in the distributed network obtains the first audio RTP packet sent by its own client, and can send the obtained first audio RTP packet to the remaining servers according to the situation. Correspondingly, the target server can receive the audio RTP packet sent by the remaining servers in the distributed network as the second audio RTP packet. It should be noted that there is at least one second audio RTP packet.

[0043] Optionally, the target server filters the acquired first audio RTP packet, and sends the filtered first audio RTP packet to the remaining servers.

[0044] S103: Select at least one candidate audio RTP packet from the first audio RTP packet and the second audio RTP packet.

[0045] In order to reduce unnecessary bandwidth transmission and save resources, the first audio RTP packet and the second audio RTP packet may be screened, and some audio RTP packets may be selected as candidate audio RTP packets.

[0046] Optionally, an audio RTP packet with a high server priority is selected as a candidate audio RTP packet.

[0047] Optionally, an audio RTP packet with an earlier arrival time is selected as a candidate audio RTP packet.

[0048] Optionally, an audio RTP packet with a loud volume is selected as a candidate audio RTP packet.

[0049] It should be noted that at least one candidate audio RTP packet is selected.

[0050] S104: Send at least one candidate audio RTP packet to the client.

[0051] The candidate audio RTP packets are further processed to generate target audio RTP packets, and the target audio RTP packets are sent to the client for playback.

[0052] In the disclosed embodiment, the target server obtains at least one first audio real-time transport protocol RTP packet sent by a client belonging to the target server, receives at least one second audio RTP packet sent by the remaining servers in the distributed network to which the target server belongs, selects at least one candidate audio RTP packet from the first audio RTP packet and the second audio RTP packet, and sends the at least one candidate audio RTP packet to the client. In the disclosed embodiment, the audio RTP packet is filtered and selected, the ambient background noise is filtered out, and the real speaking audio is selected and sent to the client, thereby improving the audio quality and reducing unnecessary bandwidth transmission.

[0053] Figure 3 is a flowchart of an audio processing method according to an embodiment of the present disclosure, based on the above embodiment, further combined with Figure 3 , explaining a process of selecting at least one candidate audio RTP packet from a first audio RTP packet and a second audio RTP packet, comprising the following steps:

[0054] S301, obtaining sound intensity characterization values ​​of the first audio RTP packet and the second audio RTP packet at each sampling time within a set period, wherein the sound intensity characterization value characterizes the size of the sampled sound.

[0055] In order to filter out the environmental noise in the audio RTP packet, the audio RTP packet with a larger volume can be selected from the first audio RTP packet and the second audio RTP packet according to the volume as the candidate audio RTP packet.

[0056] The target server includes an RTP sampler, which can periodically sample the received first audio RTP packet and the second audio RTP packet to obtain the sound intensity representation value of the audio RTP packet at this time. The sound intensity representation value is an extended identifier contained in the audio RTP packet, which can represent the size of the sampled sound, and the value range is [0,127].

[0057] The sampling period of the RTP sampler can be adjusted as needed, and optionally can be set to 100ms.

[0058] S302: Select a candidate audio RTP packet from the first audio RTP packet and the second audio RTP packet based on the sound intensity characterization value.

[0059] Based on the sound intensity characterization value, a preset number of audio RTP packets are selected from the first audio RTP packet and the second audio RTP packet as candidate audio RTP packets.

[0060] In some implementations, the target server identifies whether there is a first candidate audio RTP packet from a subordinate client of the remaining servers in the candidate audio RTP packet, that is, the candidate audio RTP packet screened out from the second audio RTP packet is used as the first candidate audio RTP packet. The first candidate audio RTP packet is removed from the candidate audio RTP packet to obtain the remaining second candidate audio RTP packet. The remaining second candidate audio RTP packet is sent to the remaining servers. At this time, for any remaining server, the second candidate audio RTP packet sent by the target server is the second audio RTP packet received by the remaining server.

[0061] In the disclosed embodiment, the sound intensity characterization values ​​of the first audio RTP packet and the second audio RTP packet at each sampling time within a set period are obtained, wherein the sound intensity characterization value characterizes the size of the sampled sound, and based on the sound intensity characterization value, a candidate audio RTP packet is selected from the first audio RTP packet and the second audio RTP packet. In the disclosed embodiment, a fixed number of audio RTP packets are selected as candidate audio RTP packets based on the sound intensity characterization value, thereby filtering out most of the ambient background noise, improving audio quality, and reducing unnecessary bandwidth transmission.

[0062] Figure 4 is a flowchart of an audio processing method according to an embodiment of the present disclosure, based on the above embodiment, further combined with Figure 4 , a process of selecting a candidate audio RTP packet from a first audio RTP packet and a second audio RTP packet based on a sound intensity characterization value is explained, comprising the following steps:

[0063] S401: For any audio RTP packet of the first audio RTP packet and the second audio RTP packet, an average sound intensity representation value of any audio RTP packet is obtained according to the sound intensity representation value of any audio RTP packet at each sampling time.

[0064] Get the sound intensity representation value L of any audio RTP packet at each sampling time i , i=1,2,...,K, the average sound intensity characterization value AL of the audio RTP packet is calculated based on the size and number of the sound intensity characterization values.

[0065]

[0066] S402: Select a preset number of audio RTP packets from the first audio RTP packet and the second audio RTP packet as candidate audio RTP packets based on the average sound intensity characterization value.

[0067] In some implementations, the sound intensity characterization value is negatively correlated with the sound volume, and the first audio RTP packet and the second audio RTP packet are sorted from small to large according to the average sound intensity characterization value. A preset number of audio RTP packets are selected in order from the first audio RTP packet and the second audio RTP packet as candidate audio RTP packets.

[0068] In some other implementations, the sound intensity characterization value is positively correlated with the sound volume, and the first audio RTP packet and the second audio RTP packet are sorted from large to small according to the average sound intensity characterization value. A preset number of audio RTP packets are selected in order from the first audio RTP packet and the second audio RTP packet as candidate audio RTP packets.

[0069] The number of candidate audio RTP packets may be determined according to the total number of the first audio RTP packets and the second audio RTP packets, and may be adjusted according to actual conditions.

[0070] In the embodiment of the present disclosure, for any audio RTP packet in the first audio RTP packet and the second audio RTP packet, the average sound intensity characterization value of any audio RTP packet is obtained according to the sound intensity characterization value of any audio RTP packet at each sampling time, and a preset number of audio RTP packets are selected from the first audio RTP packet and the second audio RTP packet based on the average sound intensity characterization value as candidate audio RTP packets. In the embodiment of the present disclosure, the candidate audio RTP packets are selected based on the average sound intensity characterization value, thereby avoiding the misscreening phenomenon caused by the low volume of the real speech audio RTP packet at certain sampling times, and improving the accuracy of the selection of the candidate audio RTP packets.

[0071] Figure 5 is a flowchart of an audio processing method according to an embodiment of the present disclosure, based on the above embodiment, further combined with Figure 5 , a process of sending at least one candidate audio RTP packet to a client for mixing is explained, including the following steps:

[0072] S501: For any candidate audio RTP packet, extract attribute information of any candidate audio RTP packet, and map the attribute information to a corresponding subscription channel to generate a target audio RTP packet of the corresponding subscription channel.

[0073] Extract attribute information of any candidate audio RTP packet, the attribute information including synchronization source identifier, RTP payload type and packet sequence number.

[0074] The synchronization source identifier is used to mark the source client of a certain audio RTP packet. The synchronization source identifier can be used to find out which client generates the audio RTP packet.

[0075] The RTP payload type is used to determine whether the transmission is audio or video.

[0076] The RTP packet sequence number is the sequence number identifier of the audio RTP packet. The packet sequence number can be used to determine whether there is packet loss during the transmission of the audio RTP packet.

[0077] The target server assigns the synchronization source identifier and the load type based on the configuration information of the subscription channel corresponding to any candidate audio RTP packet. The subscription channel is pre-created when the server creates a room. After the client joins the room, it subscribes to the N subscription channels of the target server.

[0078] It should be noted that the number of subscription channels is the same as the number of selected candidate audio RTP packets.

[0079] The target server obtains the offset between the first packet sequence number of the candidate audio RTP packet mapped for the first time and the current second packet sequence number of the subscription channel, and for subsequent candidate audio RTP packets, adds the first packet sequence number of the subsequent candidate audio RTP packets to the offset to obtain the second packet sequence number corresponding to the subsequent candidate audio RTP packets.

[0080] After mapping the attribute information to the corresponding subscription channel, a target audio RTP packet of the corresponding subscription channel is generated.

[0081] S502: Send the target audio RTP packet to the client.

[0082] The server sends the target audio RTP packet to the client for playback.

[0083] In the disclosed embodiment, for any candidate audio RTP packet, the attribute information of any candidate audio RTP packet is extracted, the attribute information is mapped to the corresponding subscription channel to generate a target audio RTP packet of the corresponding subscription channel, and the target audio RTP packet is sent to the client. In the disclosed embodiment, the target audio RTP packet sent to the client is generated through RTP packet mapping.

[0084] Figure 6 is a flowchart of an audio processing method according to an embodiment of the present disclosure, based on the above embodiment, further combined with Figure 6 , explaining the process of generating the target audio RTP packet, and also including the following steps:

[0085] S601: In response to the presence of audio RTP packets originating from the client itself, the audio RTP packets originating from the client itself are removed.

[0086] In order to further save bandwidth resources and prevent the received audio from containing echo, the audio RTP packets from the client itself need to be removed. Optionally, the audio RTP packets can be removed before mapping or after mapping.

[0087] In some implementations, after obtaining at least one candidate audio RTP packet, it is possible to first identify whether there is a target candidate audio RTP packet originating from the client itself in the at least one candidate audio RTP packet. If there is a target candidate audio RTP packet in the candidate audio RTP packet, the target candidate audio RTP packet can be eliminated in order to save resources. Further, the remaining candidate audio RTP packets can be mapped into the target audio RTP packets.

[0088] In other implementations, after obtaining at least one candidate audio RTP packet, the candidate audio RTP packet can be mapped first, and then it is identified whether there is a target audio RTP packet originating from the client itself in the mapped target audio RTP packet. If there is a target audio RTP packet originating from the client itself, the target audio RTP packet originating from the client itself can be eliminated in order to save resources.

[0089] S602: Send the remaining audio RTP packets to the client.

[0090] For any client, the server obtains the target audio RTP packet after removing the audio RTP packet from the client itself, and sends it to the corresponding client for playback.

[0091] In the disclosed embodiment, in response to the presence of audio RTP packets originating from the client itself, the audio RTP packets originating from the client itself are removed, and the remaining audio RTP packets are sent to the client. In the disclosed embodiment, the audio RTP packets sent by the client itself are removed, thereby avoiding the phenomenon of the echo of the client's own voice in the received audio.

[0092] Figure 7 is a flowchart of an audio processing method according to an embodiment of the present disclosure, the method is applied to a client, such as Figure 7 As shown, the method comprises the following steps:

[0093] S701, sending at least one first audio RTP packet to the target server to which it belongs.

[0094] Any client collects audio data in the current environment, generates a first audio RTP packet, and sends it to the target server to which it belongs.

[0095] S702, receiving at least one candidate audio RTP packet sent by the target server, wherein the candidate audio RTP packet is selected by the target server from the first audio RTP packet and the second audio RTP packet from the remaining servers, and the target server and the remaining servers belong to the same distributed network.

[0096] It should be noted that in order to avoid the echo of the client's own voice in the audio received by the client, the target server needs to remove the audio RTP packets from the client itself, that is, the candidate audio RTP packets do not include the first audio RTP packet collected by the client itself.

[0097] In the disclosed embodiment, the client sends at least one first audio RTP packet to the target server to which it belongs, and receives at least one candidate audio RTP packet sent by the target server, wherein the candidate audio RTP packet is selected by the target server from the first audio RTP packet and the second audio RTP packet from the remaining servers, and the target server and the remaining servers belong to the same distributed network. The target audio RTP packet received by the client in the disclosed embodiment is screened, which greatly reduces the environmental background noise in the received audio and improves the audio quality.

[0098] Figure 8 is a flowchart of an audio processing method according to an embodiment of the present disclosure, which, based on the above embodiment, further includes:

[0099] S801, subscribing to a preset number of subscription channels from a target server.

[0100] When creating a room, the target server pre-creates N virtual audio streams as subscription channels. After the client joins the room, it subscribes to the N subscription channels of the target server. It should be noted that the number of subscription channels N is the number of candidate audio RTP packets selected from the first audio RTP packet and the second audio RTP packet.

[0101] S802, receiving a target audio RTP packet transmitted by each subscription channel from the subscription channel, wherein the attribute information of the target audio RTP packet is obtained by the target server mapping the attribute information of the candidate audio RTP packet to the corresponding subscription channel.

[0102] The target server obtains the attribute information of the candidate audio RTP packet and maps the attribute information to the corresponding subscription channel to generate the target audio RTP packet of the corresponding subscription channel. Accordingly, the client can receive the target audio RTP packet transmitted by each subscription channel from the subscription channel.

[0103] For the specific implementation of mapping the attribute information to the corresponding subscription channel, please refer to the relevant introduction in each embodiment of the present disclosure, which will not be repeated here.

[0104] In the disclosed embodiment, the client subscribes to a preset number of subscription channels from the target server, and receives the target audio RTP packet transmitted by each subscription channel from the subscription channel, wherein the attribute information of the target audio RTP packet is obtained by the target server mapping the attribute information of the candidate audio RTP packet to the corresponding subscription channel. In the disclosed embodiment, through the subscription channel, the audio RTP packets of multiple real streams can be dynamically mapped to the audio RTP packets of fixed N virtual streams.

[0105] Fig. 9 This is a flow chart of an audio processing method according to an embodiment of the present disclosure. Based on the above embodiment, before sending at least one first audio RTP packet to the target server to which it belongs, it also includes:

[0106] S901, collecting audio at each sampling time within a set period, and generating a sound representation value of the audio based on the sound volume of the collected audio.

[0107] Audio is collected at each sampling moment within a set period, the decibel value of the audio at the sampling moment is obtained, and a sound characterization value of the audio is generated based on the decibel value.

[0108] Among them, the sound intensity representation value and the decibel value can be negatively correlated or positively correlated.

[0109] S902: Generate a first audio RTP packet based on the audio collected at each sampling time and the sound representation value corresponding to the audio.

[0110] An audio RTP packet is generated based on the collected audio, an extended header of the audio RTP packet is generated based on the sound characterization value, and a first audio RTP packet is generated.

[0111] In the embodiment of the present disclosure, audio is collected at each sampling time within a set period, and a sound characterization value of the audio is generated based on the sound volume of the collected audio, and a first audio RTP packet is generated based on the audio collected at each sampling time and the sound characterization value corresponding to the audio. In the embodiment of the present disclosure, the first audio RTP packet is generated based on the sound characterization value, so that the server can easily obtain the sampled sound volume of the audio RTP packet by identifying the sound characterization value.

[0112] Fig.10 is a flowchart of an audio processing method according to an embodiment of the present disclosure, such as Fig.10 As shown, based on the audio processing method provided by the present disclosure, the process of audio processing in an actual application scenario includes the following steps:

[0113] Fig.10 The distributed network shown includes server 1 and server 2. Taking these two servers as examples, the process of audio processing is explained.

[0114] During audio transmission, the client corresponding to server 1 sends the client audio to server 1 as the first audio RTP packet, and at the same time, server 2 sends the client audio corresponding to server 2 to server 1 as the second audio RTP packet. Among them, the second audio RTP packet sent by server 2 to server 1 is the audio filtered by the selector.

[0115] Accordingly, the client corresponding to server 2 sends the client audio to server 2 as the first audio RTP packet, and at the same time, server 1 sends the client audio corresponding to server 1 to server 2 as the second audio RTP packet. The second audio RTP packet sent by server 1 to server 2 is the audio filtered by the selector.

[0116] The server may include an RTP sampler, a selector, and a packet mapper to process received audio RTP packets.

[0117] The RTP sampler obtains the sound intensity characterization value of the first audio RTP packet and the second audio RTP packet at each sampling time within a set period, wherein the sound intensity characterization value characterizes the size of the sampled sound.

[0118] The selector selects a candidate audio RTP packet from the first audio RTP packet and the second audio RTP packet based on the sound intensity characterization value. If the candidate audio RTP packet includes an audio RTP packet transmitted from a local client of the server, the audio RTP packet is sent to another server.

[0119] The packet mapping extracts the attribute information of any candidate audio RTP packet, and maps the attribute information to the corresponding subscription channel to generate a target audio RTP packet of the corresponding subscription channel.

[0120] In response to the existence of target audio RTP packets originating from the client itself, the target audio RTP packets originating from the client itself are removed, and the remaining target audio RTP packets are sent to the client for mixing processing.

[0121] For the specific implementation of this embodiment, please refer to the relevant introduction in each embodiment of the present disclosure, which will not be repeated here.

[0122] In the disclosed embodiment, the audio RTP packets are filtered and selected to filter out the ambient background noise, and the real speech audio is selected to be sent to the client, thereby improving the audio quality and reducing unnecessary bandwidth transmission.

[0123] Fig.11 is a structural diagram of an audio processing device according to an embodiment of the present disclosure, such as Fig.11 As shown, the audio processing device 1100 includes:

[0124] The acquisition module 1110 is used to acquire at least one first audio real-time transport protocol RTP packet sent by a client belonging to a target server;

[0125] The receiving module 1120 is used to receive at least one second audio RTP packet sent by the remaining servers in the distributed network to which the target server belongs;

[0126] A selection module 1130 is used to select at least one candidate audio RTP packet from the first audio RTP packet and the second audio RTP packet;

[0127] The sending module 1140 is used to send at least one candidate audio RTP packet to the client.

[0128] It should be noted that the above explanation of the audio processing method embodiment is also applicable to the audio processing device of this embodiment, and will not be repeated here.

[0129] In the disclosed embodiment, the audio RTP packets are filtered and selected to filter out the ambient background noise, and the real speech audio is selected to be sent to the client, thereby improving the audio quality and reducing unnecessary bandwidth transmission.

[0130] Furthermore, in a possible implementation of the embodiment of the present disclosure, the selection module 1130 is also used to: obtain the sound intensity characterization value of the first audio RTP packet and the second audio RTP packet at each sampling time within a set period, wherein the sound intensity characterization value characterizes the size of the sampled sound; and select a candidate audio RTP packet from the first audio RTP packet and the second audio RTP packet based on the sound intensity characterization value.

[0131] Furthermore, in a possible implementation of the embodiment of the present disclosure, the selection module 1130 is also used to: for any audio RTP packet in the first audio RTP packet and the second audio RTP packet, obtain an average sound intensity characterization value of any audio RTP packet according to the sound intensity characterization value of any audio RTP packet at each sampling moment; based on the average sound intensity characterization value, select a preset number of audio RTP packets from the first audio RTP packet and the second audio RTP packet as candidate audio RTP packets.

[0132] Furthermore, in a possible implementation of the embodiment of the present disclosure, the selection module 1130 is also used to: in response to the sound intensity characterization value being negatively correlated with the sound volume, sort the first audio RTP packet and the second audio RTP packet from small to large according to the average sound intensity characterization value; and select a preset number of audio RTP packets from the first audio RTP packet and the second audio RTP packet in sequence as candidate audio RTP packets.

[0133] Furthermore, in a possible implementation of the embodiment of the present disclosure, the selection module 1130 is also used to: in response to the sound intensity characterization value being positively correlated with the sound volume, sort the first audio RTP packet and the second audio RTP packet from large to small according to the average sound intensity characterization value; and select a preset number of audio RTP packets from the first audio RTP packet and the second audio RTP packet in sequence as candidate audio RTP packets.

[0134] Furthermore, in a possible implementation of the embodiment of the present disclosure, the sending module 1140 is also used to: extract attribute information of any candidate audio RTP packet, map the attribute information to the corresponding subscription channel to generate a target audio RTP packet of the corresponding subscription channel; and send the target audio RTP packet to the client.

[0135] Furthermore, in a possible implementation of the embodiment of the present disclosure, the attribute information includes a synchronization source identifier, a load type and a packet number of RTP, and the sending module 1140 is also used to: assign a synchronization source identifier and a load type based on the configuration information of the subscription channel corresponding to any candidate audio RTP packet; obtain the offset between the first packet number of the candidate audio RTP packet mapped for the first time and the current second packet number of the subscription channel; for subsequent candidate audio RTP packets, add the first packet number of the subsequent candidate audio RTP packets to the offset to obtain the second packet number corresponding to the subsequent candidate audio RTP packet.

[0136] Furthermore, in a possible implementation of the embodiment of the present disclosure, the sending module 1140 is also used to: identify whether there is a target candidate audio RTP packet originating from the client itself in at least one candidate audio RTP packet; in response to the existence of the target candidate audio RTP packet, eliminate the target candidate audio RTP packet; map the remaining candidate audio RTP packets into the target audio RTP packets and send them to the client.

[0137] Furthermore, in a possible implementation of the embodiment of the present disclosure, the sending module 1140 is also used to: identify whether there are target audio RTP packets originating from the client itself in the target audio RTP packets; in response to the existence of target audio RTP packets originating from the client itself, remove the target audio RTP packets originating from the client itself; and send the remaining target audio RTP packets to the client.

[0138] Furthermore, in a possible implementation of the embodiment of the present disclosure, the acquisition module 1110 is further used to: identify whether there is a first candidate audio RTP packet from a client subordinate to the remaining server in the candidate audio RTP packet; remove the first candidate audio RTP packet from the candidate audio RTP packet to obtain the remaining second candidate audio RTP packet; and send the remaining second candidate audio RTP packet to the remaining server.

[0139] Fig.12 is a structural diagram of an audio processing device according to an embodiment of the present disclosure, such as Fig.12 As shown, the audio processing device 1200 includes:

[0140] The sending module 1210 is used to send at least one first audio RTP packet to the target server to which it belongs;

[0141] The receiving module 1220 is used to receive at least one candidate audio RTP packet sent by the target server, wherein the candidate audio RTP packet is selected by the target server from the first audio RTP packet and the second audio RTP packet from the remaining servers, and the target server and the remaining servers belong to the same distributed network.

[0142] It should be noted that the above explanation of the audio processing method embodiment is also applicable to the audio processing device of this embodiment, and will not be repeated here.

[0143] In the embodiment of the present disclosure, the target audio RTP packet received by the client is screened, which greatly reduces the environmental background noise in the received audio and improves the audio quality.

[0144] Furthermore, in a possible implementation manner of the embodiment of the present disclosure, at least one candidate audio RTP packet does not include the first audio RTP packet collected by the client itself.

[0145] Furthermore, in a possible implementation of the embodiment of the present disclosure, the sending module 1210 is also used to: collect audio at each sampling moment within a set period, and generate a sound characterization value of the audio based on the sound volume of the collected audio; and generate a first audio RTP packet based on the audio collected at each sampling moment and the sound characterization value corresponding to the audio.

[0146] Furthermore, in a possible implementation of the embodiment of the present disclosure, the sending module 1210 is further configured to: subscribe to a preset number of subscription channels from the target server.

[0147] Furthermore, in a possible implementation of the embodiment of the present disclosure, the receiving module 1220 is also used to: receive the target audio RTP packet transmitted by each subscription channel from the subscription channel, wherein the attribute information of the target audio RTP packet is obtained by the target server mapping the attribute information of the candidate audio RTP packet to the corresponding subscription channel.

[0148] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0149] Fig.13 A schematic block diagram of an example electronic device 130 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0150] like Fig.13 As shown, it includes a memory 131, a processor 132, and a computer program stored in the memory 131 and executable on the processor 132. When the processor 132 executes the program, the aforementioned audio processing method is implemented.

[0151] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0153] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0154] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0155] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0156] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0157] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An audio processing method, characterized in that: Executed by the target server, including the following steps: Acquire at least one first audio real-time transport protocol packet sent by a client belonging to the target server, the target server corresponds to one or more clients, during audio transmission, each of the clients collects the generated audio signal and sends one audio real-time transport protocol packet to the corresponding server, the target server acquires the audio real-time transport protocol packets sent by all the clients belonging to it as the first audio real-time transport protocol packets, and the number of the first audio real-time transport protocol packets is the same as the number of the corresponding clients; Receiving at least one second audio real-time transport protocol packet sent by remaining servers in the distributed network to which the target server belongs; Selecting at least one candidate audio real-time transport protocol packet from the first audio real-time transport protocol packet and the second audio real-time transport protocol packet; Sending the at least one candidate audio real-time transport protocol packet to the client; Wherein, receiving at least one second audio real-time transport protocol packet sent by the remaining servers in the distributed network to which the target server belongs, comprises: obtaining a first audio real-time transport protocol packet sent by a client belonging to each server in the distributed network, the target server screening the obtained first audio real-time transport protocol packet, and sending the screened first audio real-time transport protocol packet to the remaining servers, the target server receiving the audio real-time transport protocol packet sent by the remaining servers in the distributed network as the second audio real-time transport protocol packet, and the second audio real-time transport protocol packet has at least one path; The sending the at least one candidate audio real-time transport protocol packet to the client comprises: For any candidate audio real-time transport protocol packet, extract attribute information of any candidate audio real-time transport protocol packet, map the attribute information to the corresponding subscription channel to generate a target audio real-time transport protocol packet of the corresponding subscription channel, the attribute information including: synchronization source identifier, payload type and packet sequence number; The target audio real-time transport protocol packet is sent to the client.

2. The method according to claim 1, characterized in that The step of selecting at least one candidate audio real-time transport protocol packet from the first audio real-time transport protocol packet and the second audio real-time transport protocol packet comprises: Acquire the sound intensity representation value of the first audio real-time transport protocol packet and the second audio real-time transport protocol packet at each sampling time within a set period, wherein the sound intensity representation value represents the size of the sampled sound; Based on the sound intensity representation value, the candidate audio real-time transport protocol packet is selected from the first audio real-time transport protocol packet and the second audio real-time transport protocol packet.

3. The method according to claim 2, characterized in that The selecting the candidate audio real-time transport protocol packet from the first audio real-time transport protocol packet and the second audio real-time transport protocol packet based on the sound intensity representation value comprises: For any one of the first audio real-time transport protocol packet and the second audio real-time transport protocol packet, according to the sound intensity representation value of the any one audio real-time transport protocol packet at each sampling time, obtaining an average sound intensity representation value of the any one audio real-time transport protocol packet; Based on the average sound intensity characterization value, a preset number of audio real-time transport protocol packets are selected from the first audio real-time transport protocol packet and the second audio real-time transport protocol packet as the candidate audio real-time transport protocol packets.

4. The method according to claim 3, characterized in that The selecting, based on the average sound intensity characterization value, a preset number of audio real-time transport protocol packets from the first audio real-time transport protocol packet and the second audio real-time transport protocol packet as the candidate audio real-time transport protocol packets comprises: In response to the sound intensity characterization value being negatively correlated with the sound volume, the first audio real-time transport protocol packet and the second audio real-time transport protocol packet are sorted from small to large according to the average sound intensity characterization value; The preset number of audio real-time transport protocol packets are selected in sequence from the first audio real-time transport protocol packets and the second audio real-time transport protocol packets as the candidate audio real-time transport protocol packets.

5. The method according to claim 3, characterized in that: The selecting, based on the average sound intensity characterization value, a preset number of audio real-time transport protocol packets from the first audio real-time transport protocol packet and the second audio real-time transport protocol packet as the candidate audio real-time transport protocol packets comprises: In response to the sound intensity characterization value being positively correlated with the sound volume, the first audio real-time transport protocol packet and the second audio real-time transport protocol packet are sorted from large to small according to the average sound intensity characterization value; The preset number of audio real-time transport protocol packets are selected in sequence from the first audio real-time transport protocol packets and the second audio real-time transport protocol packets as the candidate audio real-time transport protocol packets.

6. The method according to claim 1, characterized in that The attribute information includes a synchronization source identifier, a load type of a real-time transport protocol, and a packet sequence number, and mapping the attribute information to a corresponding subscription channel to generate a target audio real-time transport protocol packet of the corresponding subscription channel includes: Assigning the synchronization source identifier and the load type based on the configuration information of the subscription channel corresponding to any candidate audio real-time transport protocol packet; Obtaining an offset between a first packet sequence number of the candidate audio real-time transport protocol packet mapped for the first time and a current second packet sequence number of the subscription channel; For the subsequent candidate audio real-time transport protocol packet, the first packet sequence number of the subsequent candidate audio real-time transport protocol packet is added to the offset to obtain the second packet sequence number corresponding to the subsequent candidate audio real-time transport protocol packet.

7. The method according to claim 1, characterized in that The method further comprises: Identify whether there is a target candidate audio real-time transport protocol packet from the client itself in the at least one candidate audio real-time transport protocol packet; In response to the existence of the target candidate audio real-time transport protocol packet, removing the target candidate audio real-time transport protocol packet; The remaining candidate audio real-time transport protocol packets are mapped into the target audio real-time transport protocol packets and sent to the client.

8. The method according to claim 1, characterized in that The step of sending the target audio real-time transport protocol packet to the client comprises: Identify whether there is a target audio real-time transport protocol packet from the client itself in the target audio real-time transport protocol packet; In response to the existence of a target audio real-time transport protocol packet originating from the client itself, removing the target audio real-time transport protocol packet originating from the client itself; The remaining target audio real-time transport protocol packets are sent to the client.

9. The method according to any one of claims 1 to 5, characterized in that: After selecting at least one candidate audio real-time transport protocol packet from the first audio real-time transport protocol packet and the second audio real-time transport protocol packet, the method further includes: Identify whether there is a first candidate audio real-time transport protocol packet from the client under the remaining server in the candidate audio real-time transport protocol packet; Eliminate the first candidate audio real-time transport protocol packet from the candidate audio real-time transport protocol packets to obtain the remaining second candidate audio real-time transport protocol packets; The remaining second candidate audio real-time transport protocol packets are sent to the remaining servers.

10. An audio processing method, characterized in that: Executed by the client, including the following steps: Sending at least one first audio real-time transport protocol packet to the target server to which it belongs, the target server corresponding to one or more clients, during audio transmission, each of the clients collects the generated audio signal and sends one audio real-time transport protocol packet to the corresponding server, the target server obtains the audio real-time transport protocol packets sent by all the clients belonging to it as the first audio real-time transport protocol packets, and the number of the first audio real-time transport protocol packets is the same as the number of the corresponding clients; receiving at least one candidate audio real-time transport protocol packet sent by the target server, wherein the candidate audio real-time transport protocol packet is selected by the target server from the first audio real-time transport protocol packet and the second audio real-time transport protocol packet from the remaining servers, and the target server and the remaining servers belong to the same distributed network; The second audio real-time transport protocol packet from the remaining servers includes: the client sends the first audio real-time transport protocol packet to each server in the distributed network, the target server screens the obtained first audio real-time transport protocol packet, and sends the screened first audio real-time transport protocol packet to the remaining servers, the target server receives the audio real-time transport protocol packet sent by the remaining servers in the distributed network as the second audio real-time transport protocol packet, and the second audio real-time transport protocol packet has at least one path; The receiving of at least one candidate audio real-time transport protocol packet sent by the target server comprises: For any candidate audio real-time transport protocol packet, extract attribute information of any candidate audio real-time transport protocol packet, map the attribute information to the corresponding subscription channel to generate a target audio real-time transport protocol packet of the corresponding subscription channel, the attribute information including: synchronization source identifier, payload type and packet sequence number; The target audio real-time transport protocol packet is sent to the client.

11. The method according to claim 10, characterized in that The at least one candidate audio real-time transport protocol packet does not include the first audio real-time transport protocol packet collected by the client itself.

12. The method according to claim 10 or 11, characterized in that: Before sending at least one first audio real-time transport protocol packet to the target server to which it belongs, the method further includes: Performing audio collection at each sampling time within a set period, and generating a sound characterization value of the audio based on the sound volume of the collected audio; The first audio real-time transport protocol packet is generated based on the audio collected at each sampling time and the sound characterization value corresponding to the audio.

13. The method according to claim 10 or 11, characterized in that: Before sending at least one first audio real-time transport protocol packet to the target server to which it belongs, the method further includes: Subscribe to a preset number of subscription channels from the target server.

14. The method according to claim 13, characterized in that The method further comprises: The target audio real-time transport protocol packet transmitted by each subscription channel is received from the subscription channel, wherein the attribute information of the target audio real-time transport protocol packet is obtained by the target server mapping the attribute information of the candidate audio real-time transport protocol packet to the corresponding subscription channel.

15. An audio processing device, characterized in that: include: an acquisition module, configured to acquire at least one first audio real-time transport protocol packet sent by a client belonging to a target server, wherein the target server corresponds to one or more clients, and during audio transmission, each of the clients collects the generated audio signal and sends one audio real-time transport protocol packet to the corresponding server, and the target server acquires the audio real-time transport protocol packets sent by all the clients belonging to the target server as the first audio real-time transport protocol packets, and the number of the first audio real-time transport protocol packets is the same as the number of the corresponding clients; A receiving module, used for receiving at least one second audio real-time transport protocol packet sent by the remaining servers in the distributed network to which the target server belongs; A selection module, configured to select at least one candidate audio real-time transport protocol packet from the first audio real-time transport protocol packet and the second audio real-time transport protocol packet; A sending module, used to send the at least one candidate audio real-time transport protocol packet to the client; The receiving module is further used to obtain a first audio real-time transport protocol packet sent by a client of each server in the distributed network, the target server filters the obtained first audio real-time transport protocol packet, and sends the filtered first audio real-time transport protocol packet to the remaining servers, and the target server receives the audio real-time transport protocol packet sent by the remaining servers in the distributed network as a second audio real-time transport protocol packet, and the second audio real-time transport protocol packet has at least one path; The sending the at least one candidate audio real-time transport protocol packet to the client comprises: For any candidate audio real-time transport protocol packet, extract attribute information of any candidate audio real-time transport protocol packet, map the attribute information to the corresponding subscription channel to generate a target audio real-time transport protocol packet of the corresponding subscription channel, the attribute information including: synchronization source identifier, payload type and packet sequence number; The target audio real-time transport protocol packet is sent to the client.

16. An audio processing device, characterized in that: include: A sending module, used for sending at least one first audio real-time transport protocol packet to a target server to which it belongs, wherein the target server corresponds to one or more clients, and during audio transmission, each of the clients collects the generated audio signal and sends one audio real-time transport protocol packet to the corresponding server, and the target server obtains the audio real-time transport protocol packets sent by all the clients belonging to it as the first audio real-time transport protocol packets, and the number of the first audio real-time transport protocol packets is the same as the number of the corresponding clients; a receiving module, configured to receive at least one candidate audio real-time transport protocol packet sent by the target server, wherein the candidate audio real-time transport protocol packet is selected by the target server from the first audio real-time transport protocol packet and the second audio real-time transport protocol packet from the remaining servers, and the target server and the remaining servers belong to the same distributed network; The second audio real-time transport protocol packet from the remaining servers includes: the client sends the first audio real-time transport protocol packet to each server in the distributed network, the target server screens the obtained first audio real-time transport protocol packet, and sends the screened first audio real-time transport protocol packet to the remaining servers, the target server receives the audio real-time transport protocol packet sent by the remaining servers in the distributed network as the second audio real-time transport protocol packet, and the second audio real-time transport protocol packet has at least one path; The receiving of at least one candidate audio real-time transport protocol packet sent by the target server comprises: For any candidate audio real-time transport protocol packet, extract attribute information of any candidate audio real-time transport protocol packet, map the attribute information to the corresponding subscription channel to generate a target audio real-time transport protocol packet of the corresponding subscription channel, the attribute information including: synchronization source identifier, payload type and packet sequence number; The target audio real-time transport protocol packet is sent to the client.

17. An electronic device, characterized in that: Including memory and processor; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the method according to any one of claims 1 to 14.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

19. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 14.