Audio data transmission method, apparatus, device, storage medium, and program product

Abstract

The application discloses an audio data transmission method, device, equipment, storage medium and program product, which are applied to a first terminal. The method comprises the following steps: acquiring a UDP communication state and a packet loss rate between the first terminal and a media server; sending a first instruction to a talkback server, wherein the first instruction is used for instructing the talkback server to authorize corresponding speech right permissions of the first terminal in the case of receiving a first request; collecting audio data in the case of receiving the speech right permissions; and sending a second instruction to a cloud storage server in the case that the UDP communication state and the packet loss rate meet preset conditions, wherein the preset conditions comprise any one of the following: the packet loss rate is not less than a first preset threshold, and the UDP communication state is an abnormal state, the second instruction carries the audio data, and the second instruction is used for instructing the cloud storage server to store the audio data and grant access permissions of the download request to a second terminal in the case that the cloud storage server receives a download request sent by the second terminal.

Description

Technical Field

[0001] This invention belongs to the technical field of data transmission, and particularly relates to an audio data transmission method, apparatus, device, storage medium, and program product. Background Technology

[0002] Public network intercom is a technology that enables the real-time communication function of traditional walkie-talkies through the wireless network of mobile terminals, allowing users to quickly and conveniently conduct voice communication through smartphones or specially designed intercom devices.

[0003] In public network intercom systems, voice transmission is primarily conducted using half-duplex communication mode, ensuring that only one party can speak at any given time, effectively avoiding communication chaos. When a user wants to speak, they must first gain the right to speak. Once granted the right, their mobile terminal captures the voice signal and transmits it to the receiving end via the wireless network. Upon receiving the voice data, the receiving end decodes it and plays it back to the listener.

[0004] However, in environments with poor network connectivity, such as remote mountainous areas or tunnels where signal coverage is insufficient, voice data may be lost during transmission. Summary of the Invention

[0005] This invention provides an audio data transmission method, apparatus, device, storage medium, and program product that can effectively avoid data loss due to poor network conditions.

[0006] In a first aspect, embodiments of this application provide an audio data transmission method applied to a first terminal, the method comprising:

[0007] Obtain the UDP communication status and packet loss rate between the first terminal and the media server;

[0008] Send a first instruction to the intercom server. The first instruction carries a first request for speaking rights. The first instruction is used to instruct the intercom server to authorize the first terminal with the corresponding speaking rights upon receiving the first request.

[0009] Once permission to speak is granted, audio data is collected.

[0010] If the UDP communication status and packet loss rate meet preset conditions, a second instruction is sent to the cloud storage service client. The preset conditions include any one of the following: the packet loss rate is not less than a first preset threshold, the UDP communication status is abnormal, the second instruction carries audio data, and the second instruction is used to instruct the cloud storage service client to store the audio data and, if the cloud storage service client receives a download request sent by the second terminal, to grant the second terminal access to the download request.

[0011] In a second aspect, the embodiments of the present application also provide an audio data transmission method, applied to a media server, the method comprising:

[0012] receiving a third instruction sent by the first terminal, the third instruction being an instruction sent by the first terminal in a case where the first terminal determines that the UDP communication state is a normal state and the packet loss rate is less than a second preset threshold, the third instruction carrying audio data, the audio data being collected by the first terminal in a case where the first terminal sends the intercom server the first instruction carrying the request for the right to speak and receives the authorized right to speak;

[0013] storing the audio data to obtain stored audio data;

[0014] sending the stored audio data to the second terminal and a cloud storage server.

[0015] In a third aspect, the embodiments of the present application provide an audio data transmission device, applied to a first terminal, the device comprising:

[0016] an obtaining module, configured to obtain a UDP communication state and a packet loss rate between the first terminal and a media server;

[0017] a first sending module, configured to send a first instruction to an intercom server, the first instruction carrying a first request for a right to speak, the first instruction being used to instruct the intercom server to authorize the first terminal with the corresponding right to speak in a case where the intercom server receives the first request;

[0018] a collecting module, configured to collect audio data in a case where the right to speak is received;

[0019] a second sending module, configured to send a second instruction to a cloud storage server in a case where the UDP communication state and the packet loss rate meet preset conditions, the preset conditions including any of the following: the packet loss rate is not less than a first preset threshold, and the UDP communication state is an abnormal state, the second instruction carrying the audio data, the second instruction being used to instruct the cloud storage server to store the audio data and grant a second terminal a download request access right in a case where the cloud storage server receives a download request sent by the second terminal.

[0020] In a fourth aspect, the embodiments of the present application provide an audio data transmission device, applied to a media server, the device can comprise:

[0021] a receiving module, configured to receive a third instruction sent by a first terminal, the third instruction being an instruction sent by the first terminal in a case where the first terminal determines that a UDP communication state is a normal state and a packet loss rate is less than a second preset threshold, the third instruction carrying audio data, the audio data being collected by the first terminal in a case where the first terminal sends an intercom server a first instruction carrying a request for a right to speak and receives the authorized right to speak.

[0022] a storage module, configured to store the audio data, and obtain stored audio data;

[0023] a sixth sending module, configured to send the stored audio data to the second terminal and the cloud storage server.

[0024] In a fifth aspect, an embodiment of the present application provides an electronic device applied to the first terminal, the device comprising a processor and a memory storing computer program instructions;

[0025] The processor, when executing the computer program instructions, implements the audio data transmission method according to any one of the first aspect.

[0026] In a sixth aspect, an embodiment of the present application provides an electronic device applied to the media server, the device comprising a processor and a memory storing computer program instructions;

[0027] The processor, when executing the computer program instructions, implements the audio data transmission method according to the second aspect.

[0028] In a seventh aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing computer program instructions, and the computer program instructions, when executed by a processor, implement the audio data transmission method according to any one of the first aspect.

[0029] In an eighth aspect, an embodiment of the present application provides a computer program product, the program product comprising a computer program, and the computer program, when executed by a processor, implements the audio data transmission method according to any one of the first aspect and the second aspect.

[0030] The audio data transmission method, device, equipment, storage medium and program product provided by the embodiments of the present application can obtain the UDP communication state and the packet loss rate between the first terminal and the media server, and can understand the current network communication environment in real time. Further, when the current network connection state meets the preset condition, that is, the network connection state is poor, the audio data is collected, and then the collected audio data is uploaded to the cloud storage service for storage. In this way, the second terminal can directly obtain the audio data from the cloud storage server when the network connection state is good, which can effectively avoid the problem of data loss caused by poor network state in the audio data transmission process. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. For those skilled in the art, without paying creative labor, other drawings can also be obtained according to these drawings.

[0032] Figure 1 is a schematic diagram of an architecture of an example of an audio data transmission method application system provided by an embodiment of the present application;

[0033] Figure 2 is a flowchart of an audio data transmission method provided by an embodiment of the first aspect of the present application;

[0034] Figure 3 is a flowchart of a specific implementation of S201 provided by an embodiment of the first aspect of the present application;

[0035] Figure 4 is a flowchart of an audio data transmission method provided by an embodiment of the first aspect of the present application;

[0036] Figure 5 is a flowchart of an audio data transmission method provided by an embodiment of the second aspect of the present application;

[0037] Figure 6 is an example flowchart of an audio data transmission method provided by an embodiment of the present application in a first scenario;

[0038] Figure 7 is an example flowchart of an audio data transmission method provided by an embodiment of the present application in a second scenario;

[0039] Figure 8 is an example flowchart of an audio data transmission method provided by an embodiment of the present application in a third scenario;

[0040] Figure 9 is a structural schematic diagram of an audio data transmission apparatus provided by an embodiment of the third aspect of the present application;

[0041] Figure 10 is a structural schematic diagram of an audio data transmission apparatus provided by an embodiment of the fourth aspect of the present application;

[0042] Figure 11 is a structural schematic diagram of an electronic device provided by an embodiment of the fifth aspect of the present application. DETAILED DESCRIPTION

[0043] The features and exemplary embodiments of the various aspects of the present application will be described in detail below, in order to make the purposes, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are intended to explain the present application only, not to limit the present application. The present application can be implemented without some of these specific details by those skilled in the art. The following description of the embodiments is merely intended to provide a better understanding of the present application by showing examples of the present application.

[0044] It is to be noted that the relative terms such as first and second and the like are used herein solely to distinguish one entity or action from another, without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0045] To solve the problems in the prior art, an audio data transmission method, device, equipment, storage medium and program product are provided in the embodiments of the present application. The audio data transmission method mentioned in the embodiments of the present application can be related to a terminal, a signaling service, a talkback service, a user service, a media service, and a cloud storage service. For ease of understanding, the application system of the audio data transmission method in the embodiments of the present application is first introduced briefly. Figure 1 is an architecture diagram of an example of an application system of the audio data transmission method provided by an embodiment of the present application, as Figure 1 indicated, the application system can include a terminal 11, a signaling server 12, a talkback server 13, a media server 14, a cloud storage server 15, and a user server 16.

[0046] The terminal 11 can be a talkback machine, a mobile phone application, computer software, or any other device capable of sending and receiving instructions. The terminal 11 can serve as an interface for a user to interact with a talkback system. The user can perform operations such as logging in, establishing a connection, creating or joining a channel / temporary session, listening to talkback content, applying for or releasing a right to speak, etc. through the terminal 11. Meanwhile, the terminal 11 can also be responsible for voice collection, coding and decoding, playing, and collection of talkback-related data and monitoring of data packet transmission and reception.

[0047] The signaling server 12 is responsible for establishing and maintaining a WebSocket connection with the terminal 11 and ensuring reliable transmission of uplink and downlink signaling data. WebSocket is a network communication protocol that allows a client and a server to send and receive messages simultaneously. Therefore, through the WebSocket connection, the signaling server 12 can receive and forward signaling requests from the terminal 11 in real time, such as creation of a channel / temporary session, application for a right to speak, etc., to realize real-time communication between the terminal 11 and the talkback server 13 and the media server 14.

[0048] The talkback server 13 is used to process the talkback-related business logic, receive and process the temporary session or channel creation request and the talkback right application signaling data from the signaling server 12. Meanwhile, the talkback server 13 is also used to maintain the channel or temporary session member relationship, priority and other state information, to ensure the order and efficiency in the talkback process.

[0049] The media server 14 is responsible for receiving the request from the terminal 11, and is used to establish and maintain the real-time communication media channel, to ensure the real-time transmission of the audio data. Meanwhile, the media server 14 is also responsible for maintaining the state information of the talkback room, such as the room members and the voice transmission state. In addition, the media server 14 also has the ability to transmit, record and forward the talkback voice.

[0050] The cloud storage server 15 is responsible for storing the audio data files generated in the talkback process. Through the cloud storage server 15, the user can access and download these files at any time and any place, to facilitate the subsequent processing and analysis. Meanwhile, the cloud storage server 15 also provides the data backup capability, to ensure the safety and reliability of the data.

[0051] The user server 16 is responsible for managing the basic information of the user, including the account and device association information, and is used to provide the user login service, to ensure the authenticity and security of the user identity. In addition, the user server 16 can also be responsible for maintaining the online / offline state information of the user, to enable the other terminal 11 to accurately determine the current state of the user, including the offline state and the online state.

[0052] The audio data transmission method, device, equipment, storage medium and program product provided in the present application will be described below.

[0053] The first aspect of the present application provides an audio data transmission method, which is applied to a first terminal, that is, the audio data transmission method can be executed by the first terminal. Figure 2 is one of the flowcharts of the audio data transmission method provided by an embodiment of the first aspect of the present application. As shown in the figure, the audio data transmission method can include S201-S204. Figure 2 S201-S204.

[0054] S201: Obtain the UDP communication state and the packet loss rate between the first terminal and the media server;

[0055] S202: Send a first instruction to the talkback server, the first instruction carrying a first request for applying for the talkback right, and the first instruction being used to instruct the talkback server to authorize the first terminal with the corresponding talkback right in the case of receiving the first request;

[0056] S203: In the case of receiving the talkback right, collect the audio data;

[0057] S204: In the case that the UDP communication state and the packet loss rate meet preset conditions, a second instruction is sent to the cloud storage server, the preset conditions include any of the following: the packet loss rate is not less than a first preset threshold, the UDP communication state is an abnormal state, the second instruction carries audio data, and the second instruction is used to instruct the cloud storage server to store the audio data and, in the case that the cloud storage server receives a download request sent by the second terminal, grant the second terminal access permission of the download request.

[0058] By obtaining the UDP communication state and the packet loss rate between the first terminal and the media server, the embodiments of the present application can learn the current network communication environment in real time. Further, in the case that the current network connection state meets the preset conditions, i.e., the network connection state is poor, the audio data is collected, and then the collected audio data is uploaded to the cloud storage service for storage. In this way, the second terminal can directly obtain the audio data through the cloud storage server in the case that the network connection state is good, which can effectively avoid the problem of data loss caused by poor network state in the process of audio data retransmission.

[0059] In some embodiments, in S201, UDP (User Datagram Protocol) is a connectionless network protocol used to provide a data transmission service. The UDP communication state generally refers to the state of the UDP protocol in the data transmission process. The packet loss rate refers to the proportion of data packet loss in the network communication process. Packet loss can be caused by various reasons, including but not limited to network congestion, hardware failure, signal interference, etc.

[0060] In specific implementation, a network quality monitoring module can be integrated in the first terminal to determine the UDP communication state and the packet loss rate. The network quality monitoring module can be responsible for periodically sending UDP data packets to the media server for network quality detection. Specifically, the transmission and reception of UDP data packets between the first terminal and the media server can be recorded, and the transmission and reception of the collected UDP data packets can be used to regularly calculate the packet loss rate and the UDP communication state.

[0061] Figure 3 is a flowchart of a specific implementation of S201 provided by an embodiment of the first aspect of the present application. As shown in Figure 3 As a specific example, the foregoing S201 can include S2011-S2018.

[0062] S2011: A detection data packet is sent to the media server every preset first time period.

[0063] The detection data packet can include data type, sequence number, terminal address, terminal identification (account number, MAC), and time stamp. The data type is used to indicate that the detection data packet is used for network monitoring. The terminal will record the data locally after sending the UDP data packet. The first time period can be adjusted according to the actual network traffic demand. For example, if the size of the UDP transceiving data per second PS is 200 bytes, the total traffic consumption LC in 24 hours can be calculated as 16 megabytes (M).

[0064] If the first time period is set to 15 seconds, the network monitoring traffic consumption in 24 hours is about 1 megabyte (i.e., 200 byte*24*60*60 / 15).

[0065] Further, if the first time period is set to ST (unit: second), the calculation formula of the traffic consumption (LC) is: LC=(PS*24*60*60) / (1024*1024), ST=LC / SC (PS is in byte, LC and SC are in M).

[0066] S2012: For each detection data packet, if it is determined that a response data packet corresponding to the detection data packet and sent by the media server is not received within a preset second time period from the sending of the detection data packet, it is determined that the detection data packet is lost.

[0067] In some embodiments, when the media server receives the detection data packet, a corresponding response data packet is generated and sent to the first terminal. As a specific example, the content of the response data packet can include data type, request sequence number, response sequence number, terminal identification, response time stamp, and server address. If the first terminal receives the corresponding response data packet within a set timeout time, i.e., within the preset second time period from the sending, it is recorded that this network detection is successful, indicating that the data packet successfully returns between the first terminal and the media server, and the network communication state is good. If the first terminal fails to receive the response data packet within the timeout time, it is recorded that this network detection fails, i.e., a packet loss occurs. The recorded data can include UDP data packet sequence number, time, response success or failure flag, and response time consumption.

[0068] In order to ensure the responsiveness and reliability of the system, the average time consumption (AVT) and the maximum time consumption (MAT) of the detection data packet request response can be collected, and based on this, the second time period (RT) should satisfy the following conditions:

[0069] 2*MAT>RT, to ensure that the response time of the detection data packet has sufficient buffer to avoid timeout due to extreme conditions.

[0070] RT > MAT to ensure that the response time of the system is still acceptable in the worst case.

[0071] RT < ST to ensure the continuity and effectiveness of network monitoring.

[0072] S2013: Record the total number of sent detection packets and the total number of lost packets in a preset time period, which includes a plurality of preset first time periods.

[0073] In some embodiments, the preset time period (PT) is composed of a plurality of first time periods (ST), and the unit of PT is second, and the value is greater than ST. The packet loss rate of the detection packet is counted in the preset time period, which can reflect how many of the sent detection packets fail to successfully arrive at the destination within a certain time.

[0074] S2014: Determine the packet loss rate according to the total number of sent packets and the total number of lost packets.

[0075] In some embodiments, MR (packet loss rate) = MC (total number of failed detection packet responses in a time period) / SUC (total number of sent detection packets in a time period) * 100%.

[0076] By calculating the packet loss rate in a specific time period, the network quality can be effectively evaluated, which helps better understand the performance of the network.

[0077] S2015: Record the number of consecutive packet losses of the detection packet.

[0078] S2016: Determine whether the number of consecutive packet losses of the detection packet exceeds a preset number threshold.

[0079] S2017: In the case where it is determined that the number of consecutive packet losses of the detection packet exceeds the preset number threshold, determine that the UDP communication state is an abnormal state.

[0080] S2018: In the case where it is determined that the number of consecutive packet losses of the detection packet does not exceed the preset number threshold, determine that the UDP communication state is a normal state.

[0081] In some embodiments, when the detection packet fails to respond for a consecutive preset number threshold, the media server is considered abnormal, and the first terminal cannot successfully establish a UDP connection in the channel or the temporary session.

[0082] By continuously monitoring the transmission of detection packets and recording the number of consecutive lost detection packets in real time, the current network communication quality can be evaluated. When the number of consecutive packet losses reaches or exceeds a preset number threshold, it is considered that the current UDP communication state has become abnormal. The preset number threshold can be set based on factors such as the stability of the network environment, the demand for data transmission, and the user-acceptable delay or packet loss rate.

[0083] In some embodiments, in S202, the intercom server is used to manage the intercom system permissions. The first request included in the first instruction is that the first terminal requests to obtain the permission to speak when using the intercom system, i.e., the right to speak. In the scenario where multiple users or devices share an intercom system, applying for the right to speak can effectively ensure that only one user or device is speaking at any time, avoiding communication chaos. In addition, the first instruction is not only a request, but it also instructs the intercom server to authorize the first terminal with the corresponding right to speak after receiving this request.

[0084] In some embodiments, in S203, after the first terminal sends the request and obtains the authorization of the intercom server, the first terminal has the permission to speak in the intercom system. After obtaining the right to speak, the first terminal can start collecting audio data. That is, the user corresponding to the first terminal can start speaking, and the microphone or other audio collection device of the first terminal will capture these sounds and convert them into digital signals or data.

[0085] Since the right to speak in the intercom system is a limited resource, when a user no longer needs to speak, the right should be released so that other users can use it. In some embodiments, the foregoing method further includes: in the case where the audio data collection is completed, sending a fourth instruction to the intercom server, the fourth instruction carrying a second request to apply for canceling the authorization of the right to speak, and the fourth instruction being used to instruct the intercom server to cancel the corresponding right to speak of the first terminal in the case where the second request is received.

[0086] In a specific implementation, when a user finishes speaking or audio inputting, such as talking, singing, playing audio, etc. through the first terminal, the audio data collection process ends. After the audio data collection ends, in order to release the right to speak so that other users can use the intercom system to speak, the first terminal sends a fourth instruction signal or request to the intercom server. The fourth instruction contains a request that the first terminal hopes to cancel the right to speak obtained before. The fourth instruction also instructs the intercom server to cancel the right to speak granted to the first terminal before receiving the request. This timely and automatic right-to-speak release mechanism not only ensures the smoothness and orderliness of communication in the intercom system, but also improves the user experience, because when a user finishes speaking, other users can almost immediately have the opportunity to speak without long waiting or potential communication conflicts.

[0087] In some embodiments, in S204, by monitoring the UDP communication state and the packet loss rate, the network quality can be evaluated in real time. Once the network conditions meet the preset conditions, i.e., the network quality is poor, the system sends a second instruction to the cloud storage server, carrying the audio data and instructing the cloud storage server to store, which can effectively avoid the problem of audio quality degradation and call quality degradation caused by data transmission when the network condition is poor. When the second terminal requests to download, the cloud storage server will grant access rights according to the received instruction to ensure the security and accessibility of the data.

[0088] Figure 4 is a flowchart of the audio data transmission method provided by the first aspect of the present application.

[0089] S401: In the case where the UDP communication state is normal and the packet loss rate is less than a first preset threshold, it is determined whether the packet loss rate is less than a second preset threshold, which is less than the first preset threshold.

[0090] S402: In the case where the packet loss rate is less than the second preset threshold, a third instruction is sent to the media server, the third instruction carrying the audio data, the third instruction being used to instruct the media server to store the audio data, and the third instruction also being used to instruct the media server to send the stored audio data to the second terminal and the cloud storage server in the case where the stored audio data is received.

[0091] S403: In the case where the packet loss rate is not less than the second preset threshold, the audio data is stored and the stored audio data is sent to the cloud storage server and the second terminal.

[0092] In the embodiments of the present application, two different thresholds are set to more finely evaluate the network quality. The first preset threshold can represent a basic acceptable level of network quality, and the second preset threshold can represent a better network state. When the packet loss rate is less than the first preset threshold, the network state is considered normal but not necessarily optimal. At this time, it can be further determined whether the packet loss rate is less than the second preset threshold to determine the optimal recording address of the audio data.

[0093] The setting of the second preset threshold allows the system to select the media server as the recording address of the audio data when the network state is good, i.e., the packet loss rate is less than the second preset threshold, reduces the delay of data transmission, and improves the transmission efficiency. Further, the media server can also store the audio data and send the audio data to the second terminal and the cloud storage server. In the case that the network condition allows, the media server is used as the priority for recording the audio data, which can more effectively utilize the network resources.

[0094] Even in the case that the network state is not optimal, i.e., the packet loss rate is not less than the second preset threshold, the audio data can be recorded locally to reduce the possibility of audio data loss. By storing the audio data locally in the first terminal first, we can effectively reduce the possibility of audio data loss in the network transmission process. Subsequently, the stored audio data is sent to the cloud storage server and the second terminal to ensure data backup and sharing.

[0095] The second aspect of the present application provides an audio data transmission method applied to a media server, i.e., the audio data transmission method can be executed by the media server. Figure 5 is a flowchart of the audio data transmission method provided by an embodiment of the second aspect of the present application. As shown in Figure 5 the audio data transmission method can include S501-S503.

[0096] S501: receiving a third instruction sent by the first terminal, the third instruction being an instruction sent by the first terminal in the case that the first terminal determines that the UDP communication state is normal and the packet loss rate is less than the second preset threshold, the third instruction carrying audio data, the audio data being collected by the first terminal based on the first terminal sending a first instruction carrying an application right-to-speak permission to the intercom server and receiving an authorized right-to-speak permission;

[0097] S502: storing the audio data to obtain stored audio data;

[0098] S503: sending the stored audio data to the second terminal and the cloud storage server.

[0099] In this embodiment, the media server receives a third instruction from the first terminal. This third instruction is sent when the first terminal determines that the UDP communication status is normal and detects that the packet loss rate is less than a second preset threshold, indicating that the current network status is good enough to support stable transmission of audio data. After receiving the audio data, the media server performs a storage operation to ensure the integrity and reliability of the audio data, effectively preventing data loss or corruption during subsequent forwarding. After the audio data is successfully stored, the media server forwards the stored audio data to the second terminal and the cloud storage service provider. Sending audio data to the second terminal enables real-time intercom functionality between the first and second terminals, allowing real-time audio communication between them. Sending audio data to the cloud storage service provider is for backup and storage of the audio data for playback when needed. By selecting the media server as the recording and distribution center for audio data when the network status is good, data transmission latency can be reduced and transmission efficiency improved.

[0100] To facilitate understanding, the following example illustrates the interaction process between the first terminal, the second terminal, the intercom server, the cloud storage service provider, and the media server in the audio data transmission method. Figure 6 The flowchart illustrates an exemplary audio data transmission method provided in this application embodiment under a first scenario, where the packet loss rate is not less than a first preset threshold and / or the UDP communication state is abnormal. Figure 6 As shown, the audio data transmission method may include steps A1 to A14.

[0101] In step A1, the first terminal sends a request for the right to speak to the intercom server.

[0102] In step A2, after receiving the request for speaking rights, the intercom server responds to the first terminal's request by confirming and authorizing the first terminal with speaking rights.

[0103] In step A3, the first terminal collects the user's audio data.

[0104] In step A4, the first terminal uploads audio data to the cloud storage service provider.

[0105] In step A5, after receiving the audio data, the cloud storage service client stores the audio data and generates an audio file.

[0106] In step A6, the cloud storage service client sends audio file storage information to the first terminal.

[0107] In step A7, after receiving the audio file storage information, the first terminal sends the audio file storage information to the intercom server.

[0108] In step A8, the intercom server records after receiving the audio file storage information.

[0109] In step A9, the intercom server sends the audio file storage information to the second terminal.

[0110] In step A10, the second terminal sends an audio file download request to the cloud storage server.

[0111] In step A11, the cloud storage server sends the audio file to the second terminal.

[0112] In step A12, the second terminal plays the audio file.

[0113] In step A13, the first terminal sends a release of the right to speak notification to the intercom server.

[0114] In step A14, the intercom server sends the release of the right to speak notification to the second terminal.

[0115] The specific description in steps A1 to A14 above can refer to the related content in the above embodiments, which will not be repeated here.

[0116] Figure 7 The audio data transmission method provided by the embodiments of the present application is shown in the second scenario. The second scenario is that the UDP communication state is normal and the packet loss rate is less than a second preset threshold. As shown in Figure 7 The audio data transmission method can include steps B1 to B11.

[0117] In step B1, the first terminal sends a right to speak application to the intercom server.

[0118] In step B2, the intercom server confirms and authorizes the right to speak permission to the first terminal in response to the right to speak application of the first terminal after receiving the right to speak application.

[0119] In step B3, the first terminal collects audio data of the user.

[0120] In step B4, the first terminal uploads the audio data to the media server.

[0121] In step B5, the media server stores the audio data and generates an audio file after receiving the audio data.

[0122] In step B6, the media server uploads the audio file to the cloud storage server.

[0123] In step B7, the cloud storage server stores the audio file after receiving the audio file and sends audio file storage information to the media server.

[0124] In step B8, the media server sends the audio file to the second terminal after receiving the audio file storage information.

[0125] In step B9, the second terminal plays the audio file.

[0126] In step B10, the first terminal sends a speech right release notification to the talkback server.

[0127] In step B11, the talkback server sends a speech right release notification to the second terminal.

[0128] The specific description of steps B1 to B11 above can be referred to the related content in the above embodiments, which will not be repeated here.

[0129] Figure 8 The audio data transmission method provided by the embodiments of the present application is exemplarily shown in a flowchart in a third scenario, that is, the UDP communication state is normal and the packet loss rate is less than the first preset threshold but not less than the second preset threshold. As shown in Figure 8 The audio data transmission method can include steps C1 to C11.

[0130] In step C1, the first terminal sends a speech right application to the talkback server.

[0131] In step C2, the talkback server confirms and authorizes the speech right permission to the first terminal in response to the speech right application of the first terminal after receiving the speech right application.

[0132] In step C3, the first terminal collects audio data of the user.

[0133] In step C4, the first terminal stores the audio data and generates an audio file.

[0134] In step C5, the first terminal uploads the audio file to the cloud storage server after generating the audio data.

[0135] In step C6, the cloud storage server stores the audio file after receiving the audio file and sends audio file storage information to the media server.

[0136] In step C7, the first terminal sends the audio file to the talkback server.

[0137] In step C8, the talkback server records the audio file after receiving the audio file.

[0138] In step C9, the talkback server sends the audio file to the second terminal.

[0139] In step C10, the second terminal plays the audio file.

[0140] In step C11, the first terminal sends a notification to the intercom server to release the right to speak.

[0141] In step C12, the intercom server sends a notification to the second terminal to release the right to speak.

[0142] For detailed explanations of steps C1 to C12 above, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0143] A third aspect of this application provides an audio data transmission device applied to a first terminal. Figure 9 This is a schematic diagram of the structure of an audio data transmission apparatus 900 provided in an embodiment of the third aspect of this application. Figure 9 As shown, the audio data transmission device 900 may include an acquisition module 901, a first transmission module 902, an acquisition module 903, and a second transmission module 904.

[0144] The acquisition module 901 is used to acquire the UDP communication status and packet loss rate between the first terminal and the media server.

[0145] The first sending module 902 is used to send a first instruction to the intercom server. The first instruction carries a first request for speaking rights and permissions. The first instruction is used to instruct the intercom server to authorize the first terminal with the corresponding speaking rights and permissions upon receiving the first request.

[0146] The acquisition module 903 is used to acquire audio data when speech permission is received;

[0147] The second sending module 904 is used to send a second instruction to the cloud storage service terminal when the UDP communication status and packet loss rate meet preset conditions. The preset conditions include any one of the following: the packet loss rate is not less than a first preset threshold, the UDP communication status is abnormal, the second instruction carries audio data, and the second instruction is used to instruct the cloud storage service terminal to store the audio data and, when the cloud storage service terminal receives the download request sent by the second terminal, to grant the second terminal access to the download request.

[0148] In the embodiments of the present application, the UDP communication state and the packet loss rate between the first terminal and the media server are acquired by the acquisition module 901, so that the current communication environment can be learned in real time. Further, the first sending module 902 is configured to apply for the right to speak to the intercom server, and the collection module 903 collects audio data after the right to speak is acquired. Then, the second sending module 904 can record the audio data in the case that the current network connection state meets the preset condition, i.e., the network connection state is poor, and record and store the audio data to be sent in real time to the cloud storage service. In this way, the second terminal can acquire the first audio file in the case that the network connection state is better, and the problem of data loss caused by poor network state can be effectively avoided.

[0149] In some embodiments, the foregoing apparatus can further include:

[0150] The first determining module is configured to determine whether the packet loss rate is less than a second preset threshold in the case that the UDP communication state is normal and the packet loss rate is less than a first preset threshold, the second preset threshold being less than the first preset threshold.

[0151] The third sending module is configured to send a third instruction to the media server in the case that the packet loss rate is less than the second preset threshold, the third instruction carrying the audio data, the third instruction being used to instruct the media server to store the audio data, and the third instruction being further used to instruct the media server to send the stored audio data to the second terminal and the cloud storage server in the case that the stored audio data is received.

[0152] In some embodiments, the foregoing apparatus can further include:

[0153] The fourth sending module is configured to store the audio data and send the stored audio data to the cloud storage server and the second terminal in the case that the packet loss rate is not less than the second preset threshold.

[0154] In some embodiments, the foregoing acquisition module can include:

[0155] The first sending sub-module is configured to send a detection data packet to the media server every preset first time period.

[0156] The first determining sub-module is configured to determine that the detection data packet is lost in the case that it is determined that no response data packet corresponding to the detection data packet sent by the media server is received within a preset second time period from the sending of the detection data packet for each detection data packet.

[0157] The first recording sub-module is configured to record the total number of sent detection data packets and the total number of lost detection data packets within a preset time period, the preset time period including a plurality of preset first time periods.

[0158] The second determining sub-module is configured to determine the packet loss rate according to the total number of sending and the total number of packet loss.

[0159] In some embodiments, the foregoing obtaining module can include:

[0160] The second recording sub-module is configured to record the number of times of continuous packet loss of the detection data packet.

[0161] The third determining sub-module is configured to determine that the UDP communication state is an abnormal state when it is determined that the number of times of continuous packet loss of the detection data packet exceeds the preset number threshold.

[0162] The fourth determining sub-module is configured to determine that the UDP communication state is a normal state when it is determined that the number of times of continuous packet loss of the detection data packet does not exceed the preset number threshold.

[0163] In some embodiments, the foregoing apparatus can further include:

[0164] The fifth sending module is configured to send a fourth instruction to the intercom server when the audio data collection is completed, the fourth instruction carrying a second request for applying for cancellation of the speech right authorization, and the fourth instruction being used to instruct the intercom server to cancel the corresponding speech right permission of the first terminal when the second request is received.

[0165] The fourth aspect of the present application provides an audio data transmission apparatus applied to a media server. Figure 10 An audio data transmission apparatus 1000 provided by an embodiment of the fourth aspect of the present application is shown in a structural schematic diagram. As shown in the figure, the audio data transmission apparatus 1000 can include a receiving module 1001, a storage module 1002, and a sixth sending module 1003. Figure 10

[0166] The receiving module 1001 is configured to receive a third instruction sent by the first terminal, the third instruction being an instruction sent by the first terminal when it is determined that the UDP communication state is a normal state and the packet loss rate is less than a second preset threshold, the third instruction carrying audio data, and the audio data being obtained by the first terminal based on the audio data collected when the first terminal sends a first instruction carrying a request for speech right permission to the intercom server and receives the authorized speech right permission.

[0167] The storage module 1002 is configured to store the audio data to obtain stored audio data.

[0168] The sixth sending module 1003 is configured to send the stored audio data to the second terminal and the cloud storage server.

[0169] ​In the embodiment of the present application, the media server receives the third instruction from the first terminal through the receiving module 1001. The third instruction is sent when the first terminal determines that the UDP communication state is normal and detects that the packet loss rate is less than the second preset threshold, which represents that the current network state is good enough to support stable transmission of audio data. Subsequently, the storage module 1002 will perform storage operation after receiving the audio data to ensure the integrity and reliability of the audio data, effectively avoiding the loss or damage of data in the subsequent forwarding process. After that, the sixth sending module 1003 will forward the successfully stored audio data to the second terminal and the cloud storage server after the audio data is successfully stored. Sending the audio data to the second terminal is to realize the real-time talkback function of the first terminal and the second terminal, so that the first terminal and the second terminal can perform real-time audio communication. Sending the audio data to the cloud storage server is to backup and store the audio data for playback when needed. By selecting the media server as the recording and distribution center of the audio data when the network state is good, the delay of data transmission can be reduced and the transmission efficiency can be improved.

[0170] The fifth aspect of the present application provides an electronic device applied to a first terminal, Figure 11 The structure schematic diagram of the electronic device provided by an embodiment of the fifth aspect of the present application is shown. The electronic device 1100 includes a processor 1101 and a memory 1102 storing computer program instructions; the processor 1101 implements the audio data transmission method of any one of the preceding aspects when executing the computer program instructions.

[0171] Specifically, the processor 1101 can include a central processing unit (CPU), or a specific integrated circuit (Application Specific Integrated Circuit, ASIC), or can be configured to implement one or more integrated circuits of the embodiments of the present application.

[0172] The memory 1102 can include a mass storage for data or instructions. By way of example and not limitation, the memory 1102 can include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 1102 can include removable or non-removable (or fixed) media. Where appropriate, the memory 1102 can be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, the memory 1102 is a non-volatile solid-state memory.

[0173] The memory can include read-only memory (ROM), random-access memory (RAM), magnetic disk storage mediums, optical storage mediums, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Accordingly, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software that, when executed (by one or more processors), is operable to perform the operations described with reference to the method of transmitting audio data according to the first aspect of the present disclosure.

[0174] In one example, the electronic device 1100 can further include a communication interface 1103 and a bus 1110. Wherein, as shown, the processor 1101, the memory 1102, the communication interface 1103 are connected through the bus 1110 and complete the communication between each other. Figure 11

[0175] The communication interface 1103 is mainly used to realize the communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0176] The bus 1110 includes hardware, software or both to couple components of the electronic device 1100 to each other. By way of example, and not limitation, the bus 1110 can include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or combination of two or more of these. Where appropriate, the bus 1110 can include one or more buses. Although the present application describes and illustrates a particular bus, the present application contemplates any suitable bus or interconnect.

[0177] The sixth aspect of the present application provides an electronic device applied to a media server. The electronic device includes a processor and a memory storing computer program instructions; the processor implements the method of transmitting audio data according to the second aspect when executing the computer program instructions.

[0178] ​In particular, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software that, when executed (by one or more processors), is operable to perform the operations described with reference to the method of promoting a payment limit according to embodiments of the second aspect of the application. The processor executes the computer program by reading executable program code stored in the memory, to implement the audio data transmission method according to the second aspect of the application.

[0179] In some examples, the electronic device can further include a communication interface and a bus. The memory, the processor, and the communication interface are connected through the bus and complete communication with each other.

[0180] The connection relationship and specific implementation modes of the above-mentioned memory, processor, communication interface, and bus can be referred to the related content in the electronic device of the fifth aspect, and will not be described here again.

[0181] In addition, in combination with the audio data transmission method in the above-mentioned embodiments, the seventh aspect of the application can provide a computer storage medium for implementation. The computer storage medium has computer program instructions stored thereon; the computer program instructions are executed by the processor to implement any one of the audio data transmission methods in the above-mentioned embodiments.

[0182] The eighth aspect of the application can provide a computer program product, which includes a computer program. The computer program is executed by the processor to implement any one of the above-mentioned audio data transmission methods and achieve the same technical effects. To avoid repetition, details will not be described here.

[0183] It should be clear that each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the differences from other embodiments. For the device embodiments, the equipment embodiments, and the computer-readable storage medium embodiments, the relevant parts can be referred to the description of the method embodiments. The present application is not limited to the specific steps and structures described above and shown in the drawings. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps, after understanding the spirit of the present application. Moreover, for the sake of brevity, detailed descriptions of known methods and techniques are omitted here.

[0184] It is to be understood that the present application is not limited to particular configurations and processes described herein and as illustrated in the drawings. The detailed description set forth above and the appended drawings are intended as a description of particular configurations and processes of the present application and are not intended to represent the only configurations and processes in which the present application can be practiced. The detailed description set forth above and the appended drawings are intended as a description of particular configurations and processes of the present application and are not intended to represent the only configurations and processes in which the present application can be practiced. The term "exemplary" used throughout this description means "serving as an example, instance, or illustration," and should not necessarily be construed as preferred or advantageous over other configurations and processes. In addition, the term "but not limited to" as used herein is intended to indicate that the listed items are the only possible options available to one of ordinary skill in the art, but rather, the listed items are at least the preferred options among many possible options. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. Furthermore, as used herein, the common abbreviation "CPU" is intended to generally refer to any of the one or more processors, microprocessors, central processing units, digital signal processors, controllers, arithmetic logic units, and so on.

[0185] The functions shown in the structural block diagrams above can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, functional cards, and so on. When implemented in software, the elements of the present application are program or code segments that are used to perform the required tasks. The program or code segments can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. The "machine-readable medium" can include any medium that can store or transfer information. Examples of the machine-readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic medium, radio frequency (RF) links, and so on. The code segments can be downloaded via computer networks such as the Internet, intranet, and so on.

[0186] It is also to be understood that the example embodiments described herein are based on a series of steps or apparatuses to describe some methods or systems. However, the present application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in an order different from the embodiments, or several steps can be performed simultaneously.

[0187] The above detailed description includes references to the accompanying drawings, which form a part of this disclosure. The drawings show, by way of illustration, specific embodiments in which the disclosure can be practiced. In the drawings, the same reference numbers can be used to depict corresponding or like components. In addition, the words "example" and / or "exemplary" are used herein to mean serving as an example, instance, or illustration, and not necessarily as preferable or advantageous over other examples. Furthermore, the use of the term "or" in the detailed description and / or in the claims is used to mean "and / or" unless otherwise indicated. In addition, the article "a" is intended to include the particular noun that it precedes and any plural of that noun that might be meant. Also, the use of the term "at least one of" followed by a list of items means that the list is inclusive of at least one of the items, and that the list is exclusive of any other items not specifically listed. The term "at least one of" is not intended to mean "one or more of" unless specifically stated.

[0188] The above merely describes a specific implementation of the present application. Those skilled in the art can clearly understand the specific working processes of the system, modules and units described above for the convenience and brevity of description, and the corresponding processes in the foregoing method embodiments can be referred to, which will not be described herein again. It should be understood that the protection scope of the present application is not limited in this way, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed in the present application, and these modifications or replacements should be covered in the protection scope of the present application.

Claims

1. An audio data transmission method, applied to a first terminal, characterized in that, The method includes: Obtain the UDP communication status and packet loss rate between the first terminal and the media server; Send a first instruction to the intercom server. The first instruction carries a first request for speaking rights. The first instruction is used to instruct the intercom server to authorize the first terminal with the corresponding speaking rights upon receiving the first request. Upon receiving the aforementioned speech authorization, audio data is collected; When the UDP communication state and the packet loss rate meet preset conditions, a second instruction is sent to the cloud storage service terminal. The preset conditions include any one of the following: the packet loss rate is not less than a first preset threshold, the UDP communication state is an abnormal state, the second instruction carries audio data, and the second instruction is used to instruct the cloud storage service terminal to store the audio data and, when the cloud storage service terminal receives a download request sent by the second terminal, to grant the second terminal access to download the request. When the UDP communication status is normal and the packet loss rate is less than a first preset threshold, it is determined whether the packet loss rate is less than a second preset threshold, and the second preset threshold is less than the first preset threshold. If the packet loss rate is less than the second preset threshold, a third instruction is sent to the media server. The third instruction carries the audio data and is used to instruct the media server to store the audio data. The third instruction is also used to instruct the media server to send the stored audio data to the second terminal and the cloud storage service server after receiving the stored audio data. If the packet loss rate is not less than the second preset threshold, the audio data is stored and sent to the cloud storage service client and the second terminal.

2. The method according to claim 1, characterized in that, The process of obtaining the packet loss rate between the first terminal and the media server includes: At each preset first time interval, a detection data packet is sent to the media server; For each detection data packet, if no response data packet corresponding to the detection data packet is received from the media server within a preset second time period from the date the detection data packet was sent, it is determined that the detection data packet is lost. Record the total number of detected data packets sent and the total number of packets lost within a preset time period, wherein the preset time period includes multiple preset first time periods; The packet loss rate is determined based on the total number of packets sent and the total number of packets lost.

3. The method according to claim 2, characterized in that, The step of obtaining the UDP communication status between the first terminal and the media server includes: Record the number of consecutive packet losses that occurred in the detected data packets; If the number of consecutive packet losses of the detected data packet exceeds a preset threshold, the UDP communication state is determined to be abnormal. If the number of consecutive packet losses of the detected data packets does not exceed a preset threshold, the UDP communication state is determined to be normal.

4. The method according to claim 1, characterized in that, The method includes: When the audio data acquisition ends, a fourth instruction is sent to the intercom server. The fourth instruction carries a second request to cancel the voice authorization. The fourth instruction is used to instruct the intercom server to cancel the corresponding voice authorization of the first terminal upon receiving the second request.

5. An audio data transmission method, applied to a media server, characterized in that, The method includes: The system receives a third instruction sent by a first terminal, which is used to execute the audio data transmission method as described in any one of claims 1-4. The third instruction is an instruction sent by the first terminal to the first terminal when the UDP communication status is determined to be normal and the packet loss rate is less than a second preset threshold. The third instruction carries audio data, which is collected based on the first terminal sending a first instruction carrying the request for speaking rights to the intercom server and receiving the authorization for speaking rights. Store the audio data to obtain the stored audio data; The stored audio data is sent to the second terminal and the cloud storage service.

6. An audio data transmission device, applied to a first terminal, characterized in that, The device includes: The acquisition module is used to acquire the UDP communication status and packet loss rate between the first terminal and the media server; The first sending module is used to send a first instruction to the intercom server. The first instruction carries a first request for speaking rights and permissions. The first instruction is used to instruct the intercom server to authorize the first terminal with corresponding speaking rights and permissions upon receiving the first request. The acquisition module is used to acquire audio data upon receiving the aforementioned speech permission. The second sending module is used to send a second instruction to the cloud storage service terminal when the UDP communication state and the packet loss rate meet preset conditions. The preset conditions include any one of the following: the packet loss rate is not less than a first preset threshold, the UDP communication state is an abnormal state, the second instruction carries audio data, and the second instruction is used to instruct the cloud storage service terminal to store the audio data and grant the second terminal access to download the request when the cloud storage service terminal receives the download request sent by the second terminal. The first judgment module is used to determine whether the packet loss rate is less than a second preset threshold when the UDP communication state is normal and the packet loss rate is less than a first preset threshold. The second preset threshold is less than the first preset threshold. The third sending module is used to send a third instruction to the media server when the packet loss rate is less than the second preset threshold. The third instruction carries the audio data and is used to instruct the media server to store the audio data. The third instruction is also used to instruct the media server to send the stored audio data to the second terminal and the cloud storage service terminal after receiving the stored audio data. The fourth sending module is used to store audio data and send the stored audio data to the cloud storage service terminal and the second terminal when the packet loss rate is not less than the second preset threshold.

7. An audio data transmission device, applied to a media server, characterized in that, The device may include: A receiving module is used to receive a third instruction sent by a first terminal, the first terminal being used to execute the audio data transmission method as described in any one of claims 1-4; the third instruction is an instruction sent by the first terminal to the first terminal when it is determined that the UDP communication state is normal and the packet loss rate is less than a second preset threshold, the third instruction carries audio data, the audio data being collected based on the first terminal sending a first instruction carrying a request for speaking rights to the intercom server and receiving authorization for speaking rights; A storage module is used to store the audio data, thereby obtaining the stored audio data; The sixth sending module is used to send the stored audio data to the second terminal and the cloud storage service terminal.

8. An electronic device applied to a first terminal, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the audio data transmission method as described in any one of claims 1-4.

9. An electronic device used in a media server, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the audio data transmission method as described in claim 5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the audio data transmission method as described in any one of claims 1-5.

11. A computer program product, characterized in that, Includes a computer program, which, when executed, implements the audio data transmission method according to any one of claims 1-5.

Images