Audio pushing system, method and device, medium and product

Abstract

The invention discloses an audio pushing system, method and device, a medium and a product. The system comprises an audio receiving module used for determining a plurality of heterogeneous terminals corresponding to an audio pushing request and audio data to be pushed; the media agent module creates a virtual terminal for the heterogeneous terminal, allocates a first port and a second port, and sends the first port to the heterogeneous terminal based on the virtual terminal, so that the heterogeneous terminal is connected with the virtual terminal based on the first port, and negotiates a to-be-used audio coding type and a terminal sampling rate with the virtual terminal; the audio pushing module is connected with the media agent module according to the second port, and based on the audio attribute information, the terminal sampling rate and the to-be-used audio coding type, to-be-played audio data of the to-be-pushed audio data under the terminal sampling rate and an encoder instance corresponding to the to-be-used audio coding type are determined; and the media agent module sends the to-be-pushed data packet to the heterogeneous terminal, so that the heterogeneous terminal plays the to-be-played audio data based on the encoder instance.

Description

Technical Field

[0001] The present invention relates to the field of computer processing technology, and in particular to an audio push system, method, device, medium and product. Background Art

[0002] With the rapid development of the communications and coal mining industries, the number of mining terminals supporting audio playback is increasing. These terminals typically support audio playback using corresponding audio encoding types. Before audio playback can be performed on a terminal, it is usually necessary to push the corresponding audio to the terminal so that the terminal can play the pushed audio.

[0003] Currently, traditional mining audio push systems typically use a CAN bus-based audio transmission method. This method converts the audio encoding type into an audio signal suitable for transmission on the CAN bus. The converted audio signal is then sent to a mining terminal device connected to the bus that supports the audio encoding type. The mining terminal device then decodes the audio signal and restores the original audio encoding type for playback.

[0004] This bus-based serial audio push method suffers from low audio push efficiency. Furthermore, due to bus bandwidth limitations, it typically only supports push of audio of a single encoding type, failing to meet the need for parallel push of audio of multiple encoding types. Summary of the Invention

[0005] The present invention provides an audio push system, method, device, medium and product to realize the parallel push of audio data of different encoding types to heterogeneous terminals that support multiple audio encoding types, while ensuring that the heterogeneous terminals can correctly play the audio data, thereby improving the efficiency and stability of audio push.

[0006] According to one aspect of the present invention, an audio push system is provided, which includes: an audio receiving module, a media proxy module and an audio push module; wherein,

[0007] The audio receiving module is configured to, upon receiving an audio push request, determine a list of terminals to be pushed and audio data to be pushed corresponding to the audio push request, and send the list to the media proxy module; wherein the list of terminals to be pushed includes multiple heterogeneous terminals;

[0008] The media proxy module is configured to create a virtual terminal for each heterogeneous terminal and allocate a first port and a second port associated with a session between the heterogeneous terminal and the virtual terminal;

[0009] The media proxy module is configured to send the first port to the heterogeneous terminal based on the virtual terminal, so that the heterogeneous terminal is connected to the virtual terminal based on the first port, negotiate with the virtual terminal to determine an audio coding type to be used from a plurality of audio coding types to be selected supported by the heterogeneous terminal, negotiate a terminal sampling rate from a plurality of sampling rates supported by the heterogeneous terminal, and provide feedback on the audio coding type to be used and the terminal sampling rate;

[0010] The media proxy module is configured to send the second port, the audio attribute information of the audio data to be pushed, and the terminal sampling rate and the audio encoding type to be used fed back by the heterogeneous terminal to the audio push module;

[0011] The audio push module is configured to connect to the media proxy module via the second port, determine a data packet to be pushed based on the audio attribute information, the terminal sampling rate, and the audio encoding type to be used, and send the data packet to the media proxy module; wherein the data packet to be pushed includes the audio data to be played at the terminal sampling rate and an encoder instance corresponding to the audio encoding type to be used;

[0012] The media proxy module is configured to send the received data packet to be pushed to the heterogeneous terminal, so that the heterogeneous terminal plays the audio data to be played based on the encoder instance.

[0013] According to another aspect of the present invention, there is provided an audio push method, the method comprising:

[0014] When the audio receiving module receives an audio push request, it determines a list of terminals to be pushed and audio data to be pushed corresponding to the audio push request, and sends them to the media proxy module; wherein the list of terminals to be pushed includes multiple heterogeneous terminals;

[0015] Creating a virtual terminal for each heterogeneous terminal based on the media proxy module, and allocating a first port and a second port associated with a session between the heterogeneous terminal and the virtual terminal;

[0016] The media proxy module sends the first port to the heterogeneous terminal based on the virtual terminal, so that the heterogeneous terminal is connected to the virtual terminal based on the first port, negotiates with the virtual terminal to determine an audio coding type to be used from multiple audio coding types to be selected supported by the heterogeneous terminal, negotiates a terminal sampling rate from multiple sampling rates supported by the heterogeneous terminal, and feeds back the audio coding type to be used and the terminal sampling rate;

[0017] Sending the second port, the audio attribute information of the audio data to be pushed, and the terminal sampling rate and the audio encoding type to be used fed back by the heterogeneous terminal to the audio push module based on the media proxy module;

[0018] Based on the audio push module being connected to the media proxy module via the second port, determining a data packet to be pushed based on the audio attribute information, the terminal sampling rate, and the audio encoding type to be used, and sending the data packet to the media proxy module; wherein the data packet to be pushed includes the audio data to be played at the terminal sampling rate and an encoder instance corresponding to the audio encoding type to be used;

[0019] The received data packet to be pushed is sent to the heterogeneous terminal based on the media proxy module, so that the heterogeneous terminal plays the audio data to be played based on the encoder instance.

[0020] According to another aspect of the present invention, an electronic device is provided, comprising:

[0021] at least one processor; and a memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the audio push method described in any embodiment of the present invention.

[0023] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the audio push method according to any embodiment of the present invention when executed.

[0024] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the audio push method according to any embodiment of the present invention.

[0025] The technical solution of the embodiment of the present invention is as follows: when an audio push request is received, the audio receiving module determines multiple heterogeneous terminals and audio data to be pushed corresponding to the audio push request; the media proxy module creates a virtual terminal for the heterogeneous terminal, and allocates a first port and a second port associated with the session between the heterogeneous terminal and the virtual terminal; the media proxy module sends the first port to the heterogeneous terminal based on the virtual terminal, so that the heterogeneous terminal is connected to the virtual terminal based on the first port, and negotiates with the virtual terminal the audio encoding type to be used from a plurality of audio encoding types to be selected supported by the heterogeneous terminal, and negotiates the terminal sampling rate from a plurality of sampling rates supported by the heterogeneous terminal; the media proxy module sends the second port, audio attribute information of the audio data to be pushed, and the terminal sampling rate and audio encoding type to be used fed back by the heterogeneous terminal to the audio push module; the audio push module is connected to the media proxy module according to the second port, and based on the audio attribute The method comprises the following steps: a) receiving a data packet to be pushed and receiving a data packet to be played at the terminal sampling rate, and b) receiving a data packet to be pushed; and c) receiving a data packet to be pushed; and d ... Furthermore, the media proxy module sends the audio data to be pushed at the terminal sampling rate and the encoder instance corresponding to the audio encoding type to be used to the heterogeneous terminal, so that the heterogeneous terminal plays the audio data to be played based on the encoder instance, thereby realizing the parallel push of audio data of different encoding types to heterogeneous terminals that support multiple audio encoding types, while ensuring that the heterogeneous terminals can play the audio data correctly, thereby improving the efficiency and stability of audio push.

[0026] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A schematic diagram of the structure of an audio push system provided by an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of the structure of the audio push module provided in an embodiment of the present invention;

[0030] Figure 3 A schematic structural diagram of an audio receiving module provided in an embodiment of the present invention;

[0031] Figure 4 This is a structural diagram of an audio push system provided according to Embodiment 2 of the present invention;

[0032] Figure 5 is a timing diagram for illustrating audio push based on an audio push system according to the second embodiment of the present invention;

[0033] Figure 6 is a schematic diagram of an audio push method provided according to a second embodiment of the present invention;

[0034] Figure 7 This is a flowchart of an audio push method provided according to Embodiment 3 of the present invention;

[0035] Figure 8 The present invention is a schematic diagram of an electronic device that implements the audio push method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0037] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] Example 1

[0039] Figure 1 This is a structural diagram of an audio push system provided by an embodiment of the present invention. This embodiment is applicable to scenarios where audio of different encoding types is pushed to heterogeneous terminals that support multiple audio encoding types. Figure 1 The audio push system provided in this embodiment includes: an audio receiving module 110, a media proxy module 120, and an audio push module 130. The structure of the audio push system of this embodiment is described in detail below.

[0040] The audio receiving module 110 is used to determine the terminal list to be pushed and the audio data to be pushed corresponding to the audio push request upon receiving the audio push request, and send them to the media proxy module 120; the terminal list to be pushed includes multiple heterogeneous terminals;

[0041] The media proxy module 120 is configured to create a virtual terminal for each heterogeneous terminal and allocate a first port and a second port associated with a session between the heterogeneous terminal and the virtual terminal;

[0042] The media proxy module 120 is configured to send the first port to the heterogeneous terminal based on the virtual terminal, so that the heterogeneous terminal is connected to the virtual terminal based on the first port, negotiate with the virtual terminal to determine an audio coding type to be used from multiple audio coding types to be selected supported by the heterogeneous terminal, negotiate a terminal sampling rate from multiple sampling rates supported by the heterogeneous terminal, and provide feedback on the audio coding type to be used and the terminal sampling rate;

[0043] The media proxy module 120 is configured to send the second port, the audio attribute information of the audio data to be pushed, and the terminal sampling rate and the audio coding type to be used fed back by the heterogeneous terminal to the audio push module 130;

[0044] The audio push module 130 is configured to connect to the media proxy module 120 via the second port, determine a data packet to be pushed based on the audio attribute information, the terminal sampling rate, and the audio encoding type to be used, and send the data packet to the media proxy module 120; the data packet to be pushed includes the audio data to be played at the terminal sampling rate and the encoder instance corresponding to the audio encoding type to be used;

[0045] The media proxy module 120 is configured to send the received data packets to be pushed to the heterogeneous terminal, so that the heterogeneous terminal plays the audio data to be played based on the encoder instance.

[0046] The audio push request may refer to an instruction or signal to push corresponding audio data to a specified terminal device. The specific audio data to be pushed is referred to as the audio data to be pushed. This audio data can include pre-recorded voice files, system-generated warning voices, voice content generated using text-to-speech technology, real-time audio data, and so on. The terminal devices to which the audio data is pushed are heterogeneous terminals. The list of terminals to be pushed refers to the collection of all terminal devices specified in the audio push request that need to receive the audio data, i.e., a collection of heterogeneous terminals. Heterogeneous terminals refer to terminal devices that differ in hardware architecture, operating system, software environment, audio playback type, and other aspects. For example, these terminals may be various types of communication terminals currently used in the coal mining industry. A virtual terminal is a logical communication entity that simulates an actual physical terminal device. A virtual terminal can handle communication tasks associated with an actual terminal device, such as audio data encoding, decoding, and transmission. The first port is a logical network interface used to establish a communication connection between a heterogeneous terminal and a virtual terminal, thereby distinguishing between different virtual terminals and heterogeneous terminals. The second port is used to establish a communication connection between the media proxy module 120 and the audio push module 130. The audio encoding type to be selected refers to the audio encoding format supported for playback by the heterogeneous terminal. For example, the audio encoding type to be selected includes but is not limited to PCMU, PCMA, AMR, AMR-WB, etc. The terminal sampling rate refers to the audio sampling rate supported by the heterogeneous terminal, that is, the number of audio samples collected per second. For example, the sampling rate can be 8kHz, 16kHz, etc. The audio encoding type to be used refers to the audio encoding format determined during the negotiation process between the heterogeneous terminal and the virtual terminal. Audio attribute information may refer to the characteristics of the audio data to be pushed itself. The encoder instance refers to a specific audio encoder object, which is used to convert the audio data to be pushed from the current encoding type into an audio encoding format suitable for playback by the heterogeneous terminal.

[0047] In this embodiment, an audio push request can be triggered in a variety of ways, such as by a dispatch operator's action at the dispatch desk; automatically generated by the system's detection of an abnormality; or triggered by a preset timed task, pushing the corresponding audio data to a designated terminal device. The audio push request can be transmitted over the network in the form of a data packet, containing a list of terminals to be pushed, the audio data to be pushed, etc. Alternatively, the audio push request can be expressed in a format such as HTTP (Hypertext Transfer Protocol), HTTPS (Secure Hypertext Transfer Protocol), WebSocket (a full-duplex communication protocol), MQTT (Message Transfer Protocol), FTP (File Transfer Protocol), or SFTP (Secure File Transfer Protocol), carrying a list of terminals to be pushed, the audio data to be pushed, or a link to the audio data. This terminal list lists the identifiers (e.g., device IDs) of all terminal devices that need to receive audio. This terminal list can be manually entered or automatically generated based on preset rules, such as based on the device's type, location, or group, without limitation. When the audio receiving module 110 receives an audio push request, it can parse the audio push request, obtain a list of terminals to be pushed, and obtain the audio data to be pushed from a storage location or extract the audio data to be pushed carried in the request. Then, the audio receiving module 110 sends the list of terminals to be pushed and the audio data to be pushed to the media proxy module 120. Optionally, the heterogeneous terminals in the list of terminals to be pushed include at least fixed-line phones, mobile terminals, broadcast systems, smart mining lamps, smart wearable devices, and vehicle-mounted devices. For example, in a coal mine scenario, fixed-line phones can be installed in fixed locations such as a dispatching room and a wellhead, and broadcast systems can be installed in locations such as tunnels and working faces. The dispatching operator can communicate with the mobile terminals or smart mining lamps of underground workers through fixed-line phones or broadcast systems. Alternatively, the broadcast system can broadcast to fixed-line phones, mobile terminals, or smart mining lamps, and the data for the call or broadcast is audio data.

[0048] Furthermore, the media proxy module 120 can create a corresponding virtual terminal for each heterogeneous terminal based on the characteristics of the heterogeneous terminal (such as supported audio codecs, communication interfaces, bandwidth limitations, etc.) through the SIP (Session Initialization Protocol) and SDP (Session Description Protocol) protocols. Furthermore, a media proxy port pair is allocated for the heterogeneous terminal's session. A communication connection is established between the heterogeneous terminal and the virtual terminal based on the first port in the media proxy port pair, ensuring that audio data can be correctly transmitted between the heterogeneous terminal and the virtual terminal. A communication connection is then established between the media proxy module 120 and the audio push module 130 based on the second port. For example, assuming that a fixed-line heterogeneous terminal supports audio codecs A and B, and a smart miner's lamp heterogeneous terminal supports audio codecs B and C, and audio data needs to be pushed to both the fixed-line heterogeneous terminal and the smart miner's lamp heterogeneous terminal, a virtual terminal supporting audio codecs A and B is created for the fixed-line heterogeneous terminal and a media proxy port pair is allocated. Create a virtual terminal that supports audio coding types B and C for the intelligent mining lamp heterogeneous terminal and allocate a media proxy port pair.

[0049] Furthermore, the virtual terminal can send the first port and multiple audio coding types supported by the heterogeneous terminal to the heterogeneous terminal. Alternatively, the virtual terminal can include the first port in the virtual terminal SIP call request, so that the heterogeneous terminal receives the first port in the SIP call request. The heterogeneous terminal establishes a connection between its own port and the first port, thereby achieving a communication connection with the virtual terminal. Furthermore, the heterogeneous terminal can negotiate with the virtual terminal to select an audio coding type to be used from the multiple audio coding types to be selected through a negotiation algorithm (such as priority matching, optimal performance, etc.). Alternatively, after establishing a connection between the heterogeneous terminal and the virtual terminal, the two parties can exchange their supported audio coding types to be selected and select an audio coding type to be selected that is supported by both parties as the audio coding type to be used. Furthermore, the heterogeneous terminal can feedback the audio coding type to be used and the terminal sampling rate of the heterogeneous terminal to the virtual terminal in the media proxy module 120. This allows the system to determine the audio data supported by the heterogeneous terminal based on the audio coding type to be used and the terminal sampling rate of the heterogeneous terminal, ensuring that the audio data can be correctly played on the heterogeneous terminal. Furthermore, the media proxy module 120 receives the terminal sampling rate and the audio codec type to be used fed back by the heterogeneous terminal, and obtains audio attribute information of the audio data to be pushed from the metadata of the audio data to be pushed or an audio database. Optionally, the audio attribute information includes, but is not limited to, the file name, file size, file identifier, file content, sampling rate, sampling bit depth, number of channels, duration, audio payload type, etc. The media proxy module 120 sends the second port, the audio attribute information, and the terminal sampling rate and the audio codec type to be used fed back by the heterogeneous terminal to the audio push module 130.

[0050] The audio push module 130 can establish a connection between its own port and a second port, thereby achieving a communication connection with the media proxy module 120. Furthermore, the audio push module 130 can determine whether the sampling rate of the audio data to be pushed is consistent with the terminal sampling rate. If the sampling rate of the audio data to be pushed is inconsistent with the terminal sampling rate, it can perform sampling rate conversion on the audio data to be pushed to obtain the audio data to be played at the terminal sampling rate. At the same time, a corresponding encoder instance can be created based on the audio encoding type to be used. After the audio push module 130 determines the data packet to be pushed based on the audio data to be played and the encoder instance, it can send the data packet to be pushed to the media proxy module 120. The media proxy module 120 can push the data packet to the heterogeneous terminal. After the heterogeneous terminal receives the data packet to be pushed, it can use the encoder instance to encode audio data of any encoding type into audio data of the audio encoding type to be used, which is suitable for playback by the heterogeneous terminal. This enables the parallel push of audio data of different encoding types to multiple different heterogeneous terminals through different virtual terminals.

[0051] Exemplarily, the audio receiving module 110 interacts with the media proxy module 120, requesting the media proxy module 120 to allocate a media proxy port pair (including a first port and a second port) for the session of each heterogeneous terminal for constructing a subsequent terminal audio stream transmission channel. Each virtual terminal creates a unique call request (including the first port) for all heterogeneous terminals in parallel. After the heterogeneous terminal successfully connects to the first port of the virtual terminal based on its own port, the feedback response message includes the terminal sampling rate and the audio encoding type to be used fed back by the heterogeneous terminal). The virtual terminal processes the response message of the heterogeneous terminal connected to it, and dynamically extracts the audio transmission port, the audio encoding type to be used, and the terminal sampling rate of each heterogeneous terminal from the response message. The media proxy module 120 constructs the audio attribute information such as the audio encoding type to be used and the terminal sampling rate, the pre-allocated second port, and the unique hash value of the audio data to be pushed, which has been dynamically identified by each heterogeneous terminal, into a private push task signaling, and sends the push task signaling to the audio push module 130, notifying the audio push module 130 to generate the data packets to be pushed for each heterogeneous terminal.

[0052] The technical solution provided by this embodiment is to determine, through the audio receiving module, multiple heterogeneous terminals and audio data to be pushed corresponding to the audio push request when receiving the audio push request; the media proxy module creates a virtual terminal for the heterogeneous terminal, and allocates a first port and a second port associated with the session between the heterogeneous terminal and the virtual terminal; the media proxy module sends the first port to the heterogeneous terminal based on the virtual terminal, so that the heterogeneous terminal is connected to the virtual terminal based on the first port, and negotiates with the virtual terminal the audio encoding type to be used from the multiple audio encoding types to be selected supported by the heterogeneous terminal, and negotiates the terminal sampling rate from the multiple sampling rates supported by the heterogeneous terminal; the media proxy module sends the second port, the audio attribute information of the audio data to be pushed, and the terminal sampling rate and the audio encoding type to be used fed back by the heterogeneous terminal to the audio push module; the audio push module is connected to the media proxy module according to the second port, and based on the audio attribute The method comprises the following steps: a) receiving a data packet to be pushed and receiving a data packet to be played at the terminal sampling rate, and b) receiving a data packet to be pushed; and c) receiving a data packet to be pushed; and d ... Furthermore, the media proxy module sends the audio data to be pushed at the terminal sampling rate and the encoder instance corresponding to the audio encoding type to be used to the heterogeneous terminal, so that the heterogeneous terminal plays the audio data to be played based on the encoder instance, thereby realizing the parallel push of audio data of different encoding types to heterogeneous terminals that support multiple audio encoding types, while ensuring that the heterogeneous terminals can play the audio data correctly, thereby improving the efficiency and stability of audio push.

[0053] Based on the above embodiments, see Figure 2 , Figure 2 Schematic diagram of the structure of the audio push module provided by the embodiment of the present invention. Optionally, the audio push module 130 includes: an audio determination unit 1301, an instance creation unit 1302, an encapsulation unit 1303 and a push unit 1304; wherein,

[0054] The audio determination unit 1301 is configured to determine the audio data to be pushed at the terminal sampling rate based on the audio attribute information and the terminal sampling rate, and send the audio data to be played to the encapsulation unit 1303;

[0055] The instance creation unit 1302 is configured to create an encoder instance corresponding to the audio coding type to be used, and send the encoder instance to the encapsulation unit 1303;

[0056] The encapsulation unit 1303 is used to encapsulate the encoder instance and the audio data to be played to obtain a data packet to be pushed, and send the data packet to be pushed to the pushing unit 1304;

[0057] The pushing unit 1304 is configured to send the data packet to be pushed to the media proxy module 120 based on the second port.

[0058] In this embodiment, the audio determination unit 1301 can convert the audio data to be pushed into audio data to be played at the terminal sampling rate according to the terminal sampling rate, or extract the audio data to be pushed at the terminal sampling rate from a preset position according to the audio attribute information. The instance creation unit 1302 can call the encoder library or module to create a corresponding encoder instance according to the audio encoding type to be used (such as G.711, AMR, AMR-WB, etc.), so that the encoder instance can encode the audio data to be played into audio data of the audio encoding type to be used. Further, the encapsulation unit 1303 can encapsulate the encoder instance and the audio data to be played according to a preset protocol format (such as RTP protocol, i.e., real-time transport protocol) to obtain a data packet to be pushed, and send the data packet to be pushed to the push unit 1304. The push unit 1304 receives the data packet to be pushed sent by the encapsulation unit 1303 and sends the data packet to be pushed to the media proxy module 120 connected to its port (i.e., the port of the audio push module 130), so that the media proxy module 120 sends the data packet to be pushed to the heterogeneous terminal. The technical solution provided in this embodiment can ensure that audio data can be played correctly on different heterogeneous terminals by creating an encoder instance corresponding to the audio encoding type to be used. The encoder instance and the audio data to be played are encapsulated and processed to ensure that the audio data maintains integrity and consistency during transmission, reduce the amount of transmitted data, and increase the transmission speed.

[0059] Exemplarily, the audio push module 130 parses the push task signaling (including the second port, audio attribute information, terminal sampling rate, and audio encoding type to be used, etc.) sent by the media agent module 120, and dynamically allocates an idle thread to trigger the instance creation unit 1302 to create a corresponding encoder instance according to the audio encoding type to be used parsed in the push task signaling, thereby realizing the conversion of multiple encoding types such as PCMU, PCMA, AMR, and AMR-WB for heterogeneous terminals; the audio determination unit 1301 determines the audio data to be played from the audio cache pool according to the terminal sampling rate and audio attribute information parsed in the push task signaling; the encapsulation unit 1303 encapsulates the encoder instance and the audio data to be played into RTP voice stream to obtain the data packet to be pushed.

[0060] Based on the above embodiment, optionally, the audio determination unit 1301 is specifically used to search from the audio cache pool whether there is audio data to be pushed at the terminal sampling rate based on audio attribute information and the terminal sampling rate. If so, the found audio data to be played is sent to the encapsulation unit 1303.

[0061] The audio buffer pool is a storage area in the system for storing audio data to be used at different sampling rates.

[0062] In the present embodiment, can be based on audio attribute information, from audio buffer pool, search whether there is to be used audio data that the sampling rate of the audio data to be pushed is converted. If so, then can continue to search based on terminal sampling rate and have to be pushed audio data to be used under the terminal sampling rate, if so, then can use the audio data to be used found as audio data to be played. And then send the audio data to be played to encapsulation unit 1303, so that encapsulation unit 1303 encapsulates encoder instance and audio data to be played. This kind of benefit is that, can effectively improve audio data acquisition efficiency.

[0063] Based on the above embodiments, see Figure 3 , Figure 3 Schematic diagram of the structure of the audio receiving module provided by an embodiment of the present invention. Optionally, the audio receiving module 110 includes: a feature extraction unit 1101, a sampling rate conversion unit 1102 and an audio buffer unit 1103;

[0064] The feature extraction unit 1101 is used to extract features from the audio data to be pushed to obtain audio attribute information; the audio attribute information includes audio sampling rate and audio data identifier;

[0065] The sampling rate conversion unit 1102 is configured to convert the audio data to be pushed into audio data to be used at a plurality of preset sampling rates based on the audio sampling rate;

[0066] The audio cache unit 1103 is used to store the audio data to be used and the audio data identifiers of multiple preset sampling rates in the audio cache pool, so that the audio determination unit 1301 can determine the audio data to be pushed at the terminal sampling rate from the audio data to be used in the audio cache pool.

[0067] The audio data identifier is an identifier used to uniquely identify the audio data to be pushed. For example, the audio data identifier can be represented by a file name, hash value, or other unique identifier. The preset sampling rate refers to a set of predefined sampling rates used to convert audio data into a format suitable for playback on different heterogeneous terminals. The audio data to be used refers to the audio data after the sampling rate conversion.

[0068] It is understood that after converting the audio data to be pushed into audio data to be used at multiple preset sampling rates, the audio cache unit 1103 can store the audio data identifier as a key and the audio data to be used at the preset sampling rate as a value in the audio cache pool in the form of a key-value pair. This allows the audio determination unit 1301 to search the audio data to be used in the audio cache pool when determining the audio data to be pushed at the terminal sampling rate.

[0069] Exemplarily, when the audio receiving module 110 receives an audio push request, the feature extraction unit 1101 parses the task number, the list of terminals to be pushed, and the audio file information (including audio attribute information such as file name, file size, audio data identifier, and audio data to be pushed) from the audio push request; the feature extraction unit 1101 extracts audio attribute information such as sampling rate, sampling bit depth, number of channels, duration, and actual audio load type from the audio data to be pushed; the sampling rate conversion unit 1102 uniformly converts the audio data to be pushed into PCM intermediate format audio data with multiple sampling rates such as 8kHz and 16kHz, that is, the audio data to be used. The audio cache unit 1103 uniquely binds the audio data to be used and the extracted audio attribute information, and adds them to the key-value type audio cache pool (where the key is the audio data identifier).

[0070] For example, feature extraction is performed on the audio data to be pushed, the extracted audio sampling rate is 16kHz, and the audio data is identified as "audio_001". The preset sampling rates are 8kHz and 16kHz. The audio data to be pushed can be converted into 8kHz and 16kHz audio data to be used. The 8kHz and 16kHz audio data to be used are stored in the audio buffer pool, and the audio data identifier "audio_001" is used as the index of the 8kHz and 16kHz audio data to be used. This method can effectively improve the efficiency of determining the audio data of the corresponding sampling rate, and at the same time, it can ensure that the audio data can be played correctly on different heterogeneous terminals.

[0071] Based on the above embodiment, optionally, the media proxy module 120 is specifically configured to receive the data packet to be pushed based on the second port, and send the data packet to be pushed to the heterogeneous terminal based on the first port associated with the same session as the second port.

[0072] In this embodiment, the media proxy module 120 can receive a data packet to be pushed sent by the port of the audio push module 130 connected to the second port, and then send the data packet to be pushed to the first port associated with the same session as the second port, so that the virtual terminal can push the data packet to the port of the heterogeneous terminal connected to the first port, so that the heterogeneous terminal receives the data packet to be pushed.

[0073] Based on the above embodiment, optionally, the audio push module 130 further includes: a transmission attribute determination unit and a transmission mode adjustment unit; wherein,

[0074] A transmission attribute determination unit, configured to determine data transmission attribute information generated in the process of sending the data packet to be pushed to the heterogeneous terminal; wherein the data transmission attribute information includes at least one transmission parameter selected from the group consisting of packet loss rate, transmission delay attribute, and transmission fluctuation attribute;

[0075] The transmission mode adjustment unit is used to adjust the transmission mode based on the data transmission attribute information when some transmission parameters meet the preset adjustment conditions, so as to send the data packet to be pushed to the heterogeneous terminal based on the adjusted transmission mode.

[0076] Among them, data transmission attribute information may refer to various parameters generated during the audio data transmission process, which are used to reflect the quality and performance of data transmission. The packet loss rate refers to the proportion of data packets lost during network transmission to the total number of data packets sent. The transmission delay attribute refers to the time required for data to travel from the sender to the receiver. The higher the transmission delay attribute, the more delayed the audio transmission and the longer the transmission time. The transmission fluctuation attribute refers to the degree of variation in the transmission delay, that is, the jitter of the delay. The larger the transmission fluctuation attribute, the more unstable the audio transmission is, and the more likely it is to become stuck or discontinuous. The preset adjustment conditions are pre-configured conditions used to determine whether the transmission parameters are abnormal.

[0077] In this embodiment, while sending data packets to be pushed to a heterogeneous terminal, network monitoring tools and protocol analysis tools can be used to monitor transmission parameters such as packet loss rate, transmission delay, and transmission fluctuation in real time. For example, the packet loss rate can be calculated by counting the number of packets sent by the audio push module 130 (i.e., the sender) and the number of packets received by the heterogeneous terminal (i.e., the receiver). The number of lost packets is determined, thereby calculating the packet loss rate. The transmission delay is calculated based on the timestamp and current time in the packets received by the heterogeneous terminal. The transmission parameters are compared with preset adjustment conditions in real time or periodically. If any transmission parameter exceeds a preset threshold for the corresponding parameter in the preset adjustment conditions, an adjustment to the transmission mode can be triggered. For example, an adjustment to the transmission mode can be triggered when the packet loss rate exceeds 5%, the transmission delay exceeds 200ms, or the transmission fluctuation exceeds a certain range. These thresholds can be flexibly set to suit different application scenarios. For example, in real-time audio transmission, where latency requirements are high, the threshold for the transmission delay attribute can be set lower. In contrast, in audio file transmission scenarios, the tolerance for packet loss can be relatively high. Furthermore, the transmission mode of audio data can be dynamically adjusted based on all or part of the data transmission attribute information, or part of the transmission parameters that meet preset adjustment conditions, to optimize transmission quality and performance. Optionally, adjusting the transmission mode includes, but is not limited to, changing the encoding type, adjusting bandwidth, rerouting, adjusting audio encoding parameters, data fragmentation, etc. For example, when the packet loss rate is too high, a more robust encoding type can be switched, such as switching from AMR to G.711, which is also supported by heterogeneous terminals, or switching to a more reliable transmission protocol; when the transmission delay attribute is too high, the bandwidth can be increased to increase the transmission speed; when the transmission fluctuation attribute is too large, a more stable network path can be reselected; or, a packet retransmission mechanism can be enabled to reduce the impact of packet loss on audio quality; when the transmission delay attribute is high, the audio bit rate can be reduced to reduce the data volume, thereby reducing transmission delay; or, when the packet loss rate is high, the data packets to be pushed can be fragmented for transmission, so that smaller data packets can be transmitted quickly in the network, reducing transmission delay. Apply the adjusted transmission method to audio data transmission and continue sending the data packets to the heterogeneous terminal. The system can continue to monitor the transmission parameters to ensure that the adjusted transmission method has effectively optimized the transmission quality. If the transmission parameters still do not meet the preset adjustment conditions, the transmission method can be further adjusted until satisfactory transmission results are achieved.

[0078] For example, when sending data packets to be pushed, the audio push module 130 can continuously count information such as the number of packets sent, timestamps, and sequence numbers, and periodically generate SR (Sender Report) messages. It can also parse RR (Receiver Report) or NACK (Packet Loss Retransmission Request) messages fed back by heterogeneous terminals to extract transmission parameters such as packet loss rate, transmission delay attributes, and transmission jitter attributes. Based on the extracted transmission parameters, the transmission method, such as the sending frequency and number of retries, can be dynamically adjusted to reduce delays and lags, such as retransmitting lost packets through the RTX channel.

[0079] It should be noted that the heterogeneous terminal can record the arrival time and sequence number continuity of each data packet to be pushed, calculate the real-time packet loss rate (such as detecting packet loss through sequence number intervals), jitter attributes (packet arrival time variance) and delay attributes (based on the NTP timestamp difference in the SR message); generate an RR (Receiver Report) message based on the statistical results, which contains fields such as the maximum received sequence number, the number of packet losses, delay attributes, and jitter attributes. If the heterogeneous terminal detects continuous packet loss, it triggers a NACK message to explicitly request the retransmission of a specific packet sequence number. The RR (Receiver Report) or NACK (packet loss retransmission request) message is sent to the audio push module 130. Of course, the heterogeneous terminal can also send the recorded information to the audio push module 130 so that the audio push module 130 determines the transmission parameters based on this information.

[0080] In this solution, the audio push module 130 also constructs the RTCP protocol when encapsulating and sending the RTP voice stream, but only serves as the RTCP sender of the complete audio transmission link. The RTP-Proxy media proxy module 120 serves as the RTCP receiver of the complete audio transmission link (working in conjunction with the RTCP sender of the audio push module 130) and the RTCP sender (working in conjunction with the RTCP receiver of the target terminal). In this way, the RTP stream transmission status and transmission quality of the two independent transmission channels, "audio sending module -> RTP-Proxy media proxy module 120 side" and "RTP-Proxy media proxy module 120 other side -> target terminal", can be monitored.

[0081] In this way, by real-time detection of the transmission parameters of the data packets sent to each heterogeneous terminal and dynamic adjustment of the transmission mode, the transmission quality and performance of the audio data can be optimized, ensuring that the audio data can be played stably and with high quality on different heterogeneous terminals, reducing delays and freezes.

[0082] Example 2

[0083] As an alternative embodiment of the above embodiment, Figure 4FIG. 1 is a schematic diagram of the structure of the audio push system provided according to Embodiment 2 of the present invention. For details, please refer to the following specific content.

[0084] In this embodiment, the audio push system includes an external application layer, an audio receiving module, a media proxy module, an audio push module and a heterogeneous terminal group. Among them, the external application layer integrates multiple platform systems such as three-dimensional command and dispatch, AI intelligent outbound calling, etc. The audio receiving module includes a request parsing unit, a feature extraction unit, a sampling rate conversion unit and an audio cache unit; the media proxy module includes a virtual terminal call negotiation unit, a media data coordination unit and an audio real-time push notification unit. The audio push module includes an audio determination unit, an instance creation unit, an encapsulation unit and a push unit, a transmission attribute determination unit and a transmission mode adjustment unit. The heterogeneous terminal group includes fixed-line phones, mobile terminals, smart mining lamps, vehicle-mounted equipment, other terminal devices, etc. These terminals can support multiple audio coding types (such as PCMU, PCMA, AMR, AMR-WB, etc.).

[0085] See also Figure 5 , Figure 5 It can be represented as a timing diagram of audio push based on the audio push system. The external application layer sends the scheduling service message of audio push (i.e., audio push request) to the audio receiving module through the application layer protocol such as HTTP. The request parsing unit in the audio receiving module is used to parse the audio data to be pushed corresponding to the audio push request; the feature extraction unit extracts the audio attribute information of the audio data to be pushed; the sampling rate conversion unit converts the audio data to be pushed into audio data to be used at a plurality of preset sampling rates and caches it in the audio cache pool. The audio receiving module sends the audio data to be pushed and the list of terminals to be pushed to the media proxy module.

[0086] The media data coordination unit in the media proxy module allocates a media proxy port pair (including a first port and a second port) to each heterogeneous terminal session; the virtual terminal call negotiation unit creates a virtual terminal for the heterogeneous terminal, and sends the first port to the heterogeneous terminal based on the virtual terminal, so that the heterogeneous terminal is connected to the virtual terminal based on the first port. The heterogeneous terminal and the virtual terminal negotiate the audio encoding type to be used and the terminal sampling rate, and the heterogeneous terminal feeds back the audio encoding type to be used and the terminal sampling rate to the media proxy module. The audio real-time push notification unit in the media proxy module sends the second port, the audio attribute information of the audio data to be pushed, and the terminal sampling rate and the audio encoding type to be used fed back by the heterogeneous terminal to the audio push module. The media proxy module is also used to manage the communication between the internal private network address and the external public network.

[0087] The audio push module is used to connect to the media proxy module based on the second port. The audio push module dynamically allocates idle threads from the task pool to handle the real-time audio push tasks of each heterogeneous terminal; the audio determination unit in the audio push module determines the audio data to be pushed at the terminal sampling rate from the audio buffer pool; the instance creation unit creates an encoder instance corresponding to the audio encoding type to be used; the encapsulation unit encapsulates the encoder instance and the audio data to be played, determines the data packet to be pushed (such as the RTP media stream data packet), and sends the data packet to be pushed to the push unit; the push unit sends the data packet to be pushed to the media proxy module based on the second port. The media proxy module receives the data packet to be pushed based on the second port and sends the data packet to be pushed to the heterogeneous terminal based on the first port associated with the same session as the second port. The heterogeneous terminal plays the audio data to be played based on the encoder instance. The transmission attribute determination unit detects the transmission quality (i.e., transmission parameters) of the RTP media stream data packet sent to each terminal in real time; the transmission mode adjustment unit is used to adjust the transmission mode based on the transmission parameters to reduce delay and lag.

[0088] In order to make those skilled in the art further understand the technical solutions of the embodiments of the present invention, a specific application scenario example is given. Figure 6 The heterogeneous terminals include heterogeneous terminal 1, heterogeneous terminal 2, heterogeneous terminal 3, and heterogeneous terminal 4. When there are multiple heterogeneous terminals, audio data can be pushed to each of them in parallel based on the technical solution provided in this embodiment. The following takes the example of pushing audio data to heterogeneous terminal 1 as an example.

[0089] Audio data can be pushed by sending an audio push request from an external application layer via an application layer protocol such as HTTP. A feature extraction unit in the audio receiving module determines the list of terminals to be pushed and the audio data to be pushed from the audio push request. It then extracts audio attribute information from the audio data, including the file name, file size, audio data identifier, sampling rate, sampling bit depth, number of channels, duration, and actual audio payload type. A sampling rate conversion unit converts the audio data to be pushed into PCM intermediate format audio data at multiple sampling rates, such as 8kHz and 16kHz, to be used. An audio cache unit uniquely binds the audio data to be used and the extracted audio attribute information, adding them to a key-value audio cache pool. The audio receiving module sends the audio data to be pushed and the list of terminals to be pushed to the media proxy module. The media data coordination unit in the media proxy module allocates port 1 and port 1 to the session of heterogeneous terminal 1. A virtual terminal call negotiation unit creates a virtual terminal for the heterogeneous terminal. Virtual terminal VT1 sends a unique call request to heterogeneous terminal 1, entering port 1 in the call request. Port 1 of heterogeneous terminal 1 is connected to port 1. Heterogeneous Terminal 1 negotiates with the virtual terminal to determine the audio codec type T1_AudioCodec and the terminal sampling rate T1_Sample to be used. The heterogeneous terminal then sends these values back to the media proxy module. The real-time audio push notification unit in the media proxy module uses a call request to send the second port 1, audio attribute information, the terminal sampling rate, and the audio codec type to be used to the audio push module.

[0090] The audio push module dynamically allocates an idle thread to trigger the audio push task of the heterogeneous terminal T1, and allocates the audio push port T1, connecting the audio push port T1 to the second port 1. The instance creation unit of the audio push module creates a corresponding encoder instance according to the audio encoding type T1_AudioCodec, which is used for conversion of encoding types such as PCMU, PCMA, AMR or AMR-WB. The audio determination unit of the audio push module determines the audio data to be played based on the sampling rate T1_Sample and audio attribute information. The encapsulation unit of the audio push module combines the encoder instance and the audio data to be played to perform the RTP media stream encapsulation work and obtains data packet 1 (i.e., the data packet to be pushed). The audio push module pushes data packet 1 adapted to the encoding type of the heterogeneous terminal 1 to the second port 1 through the audio push port 1. The media proxy module forwards data packet 1 to port 1 of the heterogeneous terminal 1 through the first port 1 associated with the heterogeneous terminal 1 session, so that the heterogeneous terminal 1 plays the audio data.

[0091] Based on this, a complete audio stream transmission channel is formed for each heterogeneous terminal: audio push port => second port -> first port => terminal port, => indicating the direction of audio data flow. The audio stream in each transmission channel dynamically adapts to the respective terminal's encoding type, ultimately achieving parallel, real-time audio data push to multiple heterogeneous terminals.

[0092] The technical solution of this embodiment solves the compatibility issue of multiple audio encoding types for heterogeneous terminals by dynamically identifying the audio encoding types supported by heterogeneous terminals based on a virtual terminal call negotiation mechanism, creating multiple encoder instances, and pushing encapsulated data packets to heterogeneous terminals in real time. Furthermore, by pre-caching audio data at different sampling rates and using a pre-allocated task pool mechanism to dynamically allocate idle task threads to handle audio push tasks for each terminal, this supports parallel real-time push of audio data to a large number of heterogeneous terminals, effectively reducing transmission latency and ensuring the real-time performance of audio push to large-scale terminals.

[0093] Example 3

[0094] Figure 7 This is a flowchart of an audio push method provided according to the third embodiment of the present invention. This embodiment is applicable to the case of pushing audio to heterogeneous terminals that support multiple audio encoding types. The method can be executed by an audio push system. The audio push system can be implemented in the form of hardware and / or software. The audio push system can be configured in a computing device. The audio push system includes an audio receiving module, a media proxy module, and an audio push module. Figure 1 As shown, the method includes:

[0095] S210: When the audio receiving module receives an audio push request, it determines a terminal list to be pushed and audio data to be pushed corresponding to the audio push request, and sends the list to the media proxy module; the terminal list to be pushed includes multiple heterogeneous terminals.

[0096] S220: Create a virtual terminal for each heterogeneous terminal based on the media proxy module, and allocate a first port and a second port associated with a session between the heterogeneous terminal and the virtual terminal.

[0097] S230: Based on the media proxy module, the first port is sent to the heterogeneous terminal based on the virtual terminal, so that the heterogeneous terminal is connected to the virtual terminal based on the first port, and the audio coding type to be used is negotiated with the virtual terminal from a plurality of audio coding types to be selected supported by the heterogeneous terminal, and the terminal sampling rate is negotiated from a plurality of sampling rates supported by the heterogeneous terminal, and the audio coding type to be used and the terminal sampling rate are fed back.

[0098] S240: Based on the media proxy module, the second port, the audio attribute information of the audio data to be pushed, and the terminal sampling rate and the audio coding type to be used fed back by the heterogeneous terminal are sent to the audio push module.

[0099] S250. Based on the audio push module being connected to the media proxy module according to the second port, the audio push module determines the data packet to be pushed and sends it to the media proxy module based on the audio attribute information, the terminal sampling rate and the audio encoding type to be used; wherein the data packet to be pushed includes the audio data to be played at the terminal sampling rate of the audio data to be pushed and the encoder instance corresponding to the audio encoding type to be used.

[0100] S260: Send the received data packet to be pushed to the heterogeneous terminal based on the media proxy module, so that the heterogeneous terminal plays the audio data to be played based on the encoder instance.

[0101] On the basis of the above technical solution, optionally, the audio push module includes: an audio determination unit, an instance creation unit, an encapsulation unit and a push unit; wherein,

[0102] The audio determination unit is configured to determine, based on the audio attribute information and the terminal sampling rate, the audio data to be pushed, the audio data to be played at the terminal sampling rate, and send the audio data to be played to the encapsulation unit;

[0103] The instance creation unit is configured to create an encoder instance corresponding to the audio coding type to be used, and send the encoder instance to the encapsulation unit;

[0104] The encapsulation unit is configured to encapsulate the encoder instance and the audio data to be played to obtain a data packet to be pushed, and send the data packet to be pushed to the pushing unit;

[0105] The pushing unit is configured to send the data packet to be pushed to the media proxy module based on the second port.

[0106] On the basis of the above technical solution, optionally, the audio determination unit is specifically used to search from the audio cache pool whether there is audio data to be played at the terminal sampling rate of the audio data to be pushed based on the audio attribute information and the terminal sampling rate. If so, the found audio data to be played is sent to the encapsulation unit.

[0107] On the basis of the above technical solution, optionally, the audio receiving module includes: a feature extraction unit, a sampling rate conversion unit and an audio buffer unit;

[0108] The feature extraction unit is used to extract features from the audio data to be pushed to obtain audio attribute information; wherein the audio attribute information includes an audio sampling rate and an audio data identifier;

[0109] The sampling rate conversion unit is configured to convert the audio data to be pushed into audio data to be used at a plurality of preset sampling rates based on the audio sampling rate;

[0110] The audio cache unit is used to store the audio data to be used at the multiple preset sampling rates and the audio data identifiers in the audio cache pool, so that the audio determination unit determines the audio data to be pushed at the terminal sampling rate from the audio data to be used in the audio cache pool.

[0111] Based on the above technical solution, optionally, the media proxy module is specifically configured to receive the data packet to be pushed based on the second port, and send the data packet to be pushed to the heterogeneous terminal based on the first port associated with the same session as the second port.

[0112] On the basis of the above technical solution, optionally, the audio push module further includes: a transmission attribute determination unit and a transmission mode adjustment unit; wherein,

[0113] a transmission attribute determining unit, configured to determine data transmission attribute information generated in the process of sending the data packet to be pushed to the heterogeneous terminal; wherein the data transmission attribute information includes at least one transmission parameter selected from the group consisting of packet loss rate, transmission delay attribute, and transmission fluctuation attribute;

[0114] A transmission mode adjustment unit is used to adjust the transmission mode based on the data transmission attribute information when some of the transmission parameters reach a preset adjustment condition, so as to send the data packet to be pushed to the heterogeneous terminal based on the adjusted transmission mode.

[0115] On the basis of the above technical solution, optionally, the heterogeneous terminals include at least a landline telephone, a mobile terminal, a broadcasting system, an intelligent mining lamp and a vehicle-mounted device.

[0116] The technical solution provided by the embodiment of the present invention determines, through the audio receiving module, multiple heterogeneous terminals and audio data to be pushed corresponding to the audio push request when receiving the audio push request; the media proxy module creates a virtual terminal for the heterogeneous terminal, and allocates a first port and a second port associated with the session between the heterogeneous terminal and the virtual terminal; the media proxy module sends the first port to the heterogeneous terminal based on the virtual terminal, so that the heterogeneous terminal is connected to the virtual terminal based on the first port, and negotiates with the virtual terminal the audio encoding type to be used from the multiple audio encoding types to be selected supported by the heterogeneous terminal, and negotiates the terminal sampling rate from the multiple sampling rates supported by the heterogeneous terminal; the media proxy module sends the second port, audio attribute information of the audio data to be pushed, and the terminal sampling rate and audio encoding type to be used fed back by the heterogeneous terminal to the audio push module; the audio push module is connected to the media proxy module according to the second port, and based on the audio The method comprises the following steps: a) determining the audio data to be pushed at the terminal sampling rate and the encoder instance corresponding to the audio encoding type to be used; b) sending the received data packet to be pushed to the heterogeneous terminal so that the heterogeneous terminal plays the audio data to be played based on the encoder instance, thereby solving the problem of low audio push efficiency and inability to support parallel push of audio of multiple encoding types in the existing bus-based serial encoding audio push method. The method is implemented by creating a virtual terminal for each heterogeneous terminal, allocating a pair of ports, and then sending the first port to the heterogeneous terminal based on the virtual terminal so that the heterogeneous terminal connects to the virtual terminal based on the first port, and negotiates with the virtual terminal to determine the audio encoding type to be used from multiple audio encoding types to be selected supported by the heterogeneous terminal, and negotiates the terminal sampling rate from multiple sampling rates supported by the heterogeneous terminal. The audio push module connects to the media proxy module based on the second port, and determines the audio data to be pushed at the terminal sampling rate and the encoder instance corresponding to the audio encoding type to be used based on the audio attribute information, the terminal sampling rate and the audio encoding type to be used. Furthermore, the media proxy module sends the audio data to be pushed at the terminal sampling rate and the encoder instance corresponding to the audio encoding type to be used to the heterogeneous terminal, so that the heterogeneous terminal plays the audio data to be played based on the encoder instance, thereby realizing the parallel push of audio data of different encoding types to heterogeneous terminals that support multiple audio encoding types, while ensuring that the heterogeneous terminals can play the audio data correctly, thereby improving the efficiency and stability of audio push.

[0117] Example 4

[0118] Figure 8: is a structural diagram of an electronic device that implements the audio push method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0119] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory 12 and a random access memory 13, that is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory 12 or the computer program loaded from the storage unit 18 into the random access memory 13. The random access memory 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, the read-only memory 12, and the random access memory 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0120] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0121] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the audio push method.

[0122] In some embodiments, the audio push method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the read-only memory 12 and / or the communication unit 19. When the computer program is loaded into the random access memory 13 and executed by the processor 11, one or more steps of the audio push method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the audio push method in any other appropriate manner (for example, by means of firmware).

[0123] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0124] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0125] In the context of the present invention, computer-readable storage medium can be a tangible medium that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage medium can include but is not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage medium can be a machine-readable signal medium. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device or any suitable combination of the foregoing.

[0126] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0127] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0128] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0129] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication unit 19, or installed from the storage unit 18, or installed from the read-only memory 12. When the computer program is executed by the processor 11, the above-mentioned functions defined in the method of the embodiment of the present invention are performed.

[0130] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the audio push method provided in any embodiment of the present invention.

[0131] The computer program product may be implemented by writing computer program code for performing the operations of the present invention in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0133] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. An audio push system, characterized in that: The system includes: an audio receiving module, a media proxy module and an audio pushing module; wherein, The audio receiving module is configured to, upon receiving an audio push request, determine a list of terminals to be pushed and audio data to be pushed corresponding to the audio push request, and send the list to the media proxy module; wherein the list of terminals to be pushed includes multiple heterogeneous terminals; The media proxy module is configured to create a virtual terminal for each heterogeneous terminal and allocate a first port and a second port associated with a session between the heterogeneous terminal and the virtual terminal; The media proxy module is configured to send the first port to the heterogeneous terminal based on the virtual terminal, so that the heterogeneous terminal is connected to the virtual terminal based on the first port, negotiate with the virtual terminal to determine an audio coding type to be used from a plurality of audio coding types to be selected supported by the heterogeneous terminal, negotiate a terminal sampling rate from a plurality of sampling rates supported by the heterogeneous terminal, and provide feedback on the audio coding type to be used and the terminal sampling rate; The media proxy module is configured to send the second port, the audio attribute information of the audio data to be pushed, and the terminal sampling rate and the audio encoding type to be used fed back by the heterogeneous terminal to the audio push module; The audio push module is configured to connect to the media proxy module via the second port, determine a data packet to be pushed based on the audio attribute information, the terminal sampling rate, and the audio encoding type to be used, and send the data packet to the media proxy module; wherein the data packet to be pushed includes the audio data to be played at the terminal sampling rate and an encoder instance corresponding to the audio encoding type to be used; The media proxy module is configured to send the received data packet to be pushed to the heterogeneous terminal, so that the heterogeneous terminal plays the audio data to be played based on the encoder instance.

2. The system according to claim 1, wherein: The audio push module includes: an audio determination unit, an instance creation unit, an encapsulation unit and a push unit; wherein, The audio determination unit is configured to determine, based on the audio attribute information and the terminal sampling rate, the audio data to be pushed, the audio data to be played at the terminal sampling rate, and send the audio data to be played to the encapsulation unit; The instance creation unit is configured to create an encoder instance corresponding to the audio coding type to be used, and send the encoder instance to the encapsulation unit; The encapsulation unit is configured to encapsulate the encoder instance and the audio data to be played to obtain a data packet to be pushed, and send the data packet to be pushed to the pushing unit; The pushing unit is configured to send the data packet to be pushed to the media proxy module based on the second port.

3. The system according to claim 2, characterized in that The audio determination unit is specifically used to search from the audio cache pool whether there is audio data to be played at the terminal sampling rate of the audio data to be pushed based on the audio attribute information and the terminal sampling rate. If so, the found audio data to be played is sent to the encapsulation unit.

4. The system according to claim 1, wherein: The audio receiving module includes: a feature extraction unit, a sampling rate conversion unit and an audio buffer unit; The feature extraction unit is used to extract features from the audio data to be pushed to obtain audio attribute information; wherein the audio attribute information includes an audio sampling rate and an audio data identifier; The sampling rate conversion unit is configured to convert the audio data to be pushed into audio data to be used at a plurality of preset sampling rates based on the audio sampling rate; The audio cache unit is used to store the audio data to be used at the multiple preset sampling rates and the audio data identifiers in the audio cache pool, so that the audio determination unit determines the audio data to be pushed at the terminal sampling rate from the audio data to be used in the audio cache pool.

5. The system according to claim 1, wherein: The media proxy module is specifically configured to receive the data packet to be pushed based on the second port, and send the data packet to be pushed to the heterogeneous terminal based on the first port associated with the same session as the second port.

6. The system according to claim 1, wherein: The audio push module further includes: a transmission attribute determination unit and a transmission mode adjustment unit; wherein, a transmission attribute determining unit, configured to determine data transmission attribute information generated in the process of sending the data packet to be pushed to the heterogeneous terminal; wherein the data transmission attribute information includes at least one transmission parameter selected from the group consisting of packet loss rate, transmission delay attribute, and transmission fluctuation attribute; A transmission mode adjustment unit is used to adjust the transmission mode based on the data transmission attribute information when some of the transmission parameters reach a preset adjustment condition, so as to send the data packet to be pushed to the heterogeneous terminal based on the adjusted transmission mode.

7. The system according to claim 1, wherein: The heterogeneous terminals include at least fixed-line phones, mobile terminals, broadcast systems, intelligent mining lamps and vehicle-mounted equipment.

8. An audio push method, characterized in that: include: When the audio receiving module receives an audio push request, it determines a list of terminals to be pushed and audio data to be pushed corresponding to the audio push request, and sends them to the media proxy module; wherein the list of terminals to be pushed includes multiple heterogeneous terminals; Creating a virtual terminal for each heterogeneous terminal based on the media proxy module, and allocating a first port and a second port associated with a session between the heterogeneous terminal and the virtual terminal; The media proxy module sends the first port to the heterogeneous terminal based on the virtual terminal, so that the heterogeneous terminal is connected to the virtual terminal based on the first port, negotiates with the virtual terminal to determine an audio coding type to be used from multiple audio coding types to be selected supported by the heterogeneous terminal, negotiates a terminal sampling rate from multiple sampling rates supported by the heterogeneous terminal, and feeds back the audio coding type to be used and the terminal sampling rate; Sending the second port, the audio attribute information of the audio data to be pushed, and the terminal sampling rate and the audio encoding type to be used fed back by the heterogeneous terminal to the audio push module based on the media proxy module; Based on the audio push module being connected to the media proxy module via the second port, determining a data packet to be pushed based on the audio attribute information, the terminal sampling rate, and the audio encoding type to be used, and sending the data packet to the media proxy module; wherein the data packet to be pushed includes the audio data to be played at the terminal sampling rate and an encoder instance corresponding to the audio encoding type to be used; The received data packet to be pushed is sent to the heterogeneous terminal based on the media proxy module, so that the heterogeneous terminal plays the audio data to be played based on the encoder instance.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the audio push method according to claim 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the audio push method according to claim 8 when executed.

Images