Real-time audio and video interaction method and device, computer device, storage medium and computer program product

Abstract

The application relates to a real-time audio-video interaction method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring an audio-video media stream, creating a peer-to-peer connection object; generating session description protocol request information and setting the information as a local media description, sending the information to a receiving end through a signaling channel, receiving and processing session description protocol response information fed back by the receiving end, completing media capability negotiation of both parties, exchanging network information; performing an address connectivity check process of an interactive connection establishment framework, cooperating with the receiving end to determine an optimal transmission path of the audio-video media stream transmission, performing a handshake process through a data packet transmission layer security protocol, and negotiating to generate an end-to-end encryption key; encapsulating and transmitting the encrypted audio-video media stream, acquiring network transmission state information based on a real-time transmission control protocol, and optimizing a communication transmission process in real time. The method can reduce delay and cost and provide a more secure decentralized network environment.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a real-time audio and video interaction method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the development of Internet communication technology, real-time audio and video interaction technology has gradually become a core supporting technology in many fields such as online education, remote collaboration, and live interaction. It can break spatial limitations and realize the instant transmission of audio and video information between users, meeting people's needs for efficient communication.

[0003] In traditional technologies, audio and video transmission mainly relies on a client / server architecture, where all audio and video data needs to be received, processed, and distributed through a central server.

[0004] However, the traditional client / server architecture has several prominent problems. First, server and bandwidth costs are high. As the number of users grows, data traffic increases exponentially. Service providers need to continuously invest huge sums of money to build and maintain larger server clusters, while also bearing the high bandwidth costs associated with massive data transmission, severely limiting the scale and economic viability of the service. Second, scalability is poor and there is a risk of single point of failure. The central server has limited processing power and bandwidth resources. When user concurrency surges, the server can easily become a performance bottleneck, leading to video stuttering, audio-visual asynchrony, or even service interruption. Once the central server fails, all user services relying on it will be affected, and normal audio and video communication will be impossible. Third, transmission latency is high. Data needs to be transmitted through a loop from the user end to the central server and then to the target user end. The processing and forwarding processes in the intermediate links introduce additional latency, making it difficult to meet the stringent requirements of millisecond-level low latency in real-time interactive scenarios. Furthermore, in the traditional model, the uplink bandwidth and computing resources of a large number of terminal devices are idle most of the time, resulting in a huge waste of resources and failing to fully utilize distributed edge network resources. Summary of the Invention

[0005] Therefore, it is necessary to provide a real-time audio and video interaction method, device, computer equipment, computer-readable storage medium, and computer program product that can reduce costs and increase efficiency in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a real-time audio and video interaction method. The method includes:

[0007] Call the media capture interface to obtain audio and video media streams, create peer connection objects, configure interactive connections for the peer connection objects, and establish a server list;

[0008] A session description protocol request message is generated and set as a local media description. It is sent to the receiving end through a signaling channel. The session description protocol response message fed back by the receiving end is received and processed to complete the media capability negotiation between the two parties and exchange network information.

[0009] The address connectivity check process of the interactive connection establishment framework is executed, and the optimal transmission path of the audio and video media stream is determined in cooperation with the receiving end. The handshake process is executed through the data packet transport layer security protocol to negotiate and generate an end-to-end encryption key.

[0010] The audio and video media streams encrypted with the end-to-end encryption key are encapsulated and transmitted. Network transmission status information is obtained based on the real-time transmission control protocol, and the communication transmission process is optimized in real time based on the network transmission status information.

[0011] In some embodiments of the method, the exchange of network information includes:

[0012] Collect locally generated first network connection candidate addresses and send them to the receiving end via the signaling channel;

[0013] Receive the second network connection candidate address sent by the receiving end, and add the second network connection candidate address to the local peer connection object.

[0014] In some embodiments of the method, the execution of the address connectivity check process of the interactive connection establishment framework, in cooperation with the receiving end, to determine the optimal transmission path for the audio and video media stream transmission includes:

[0015] The interactive connection establishment framework invokes the built-in inspection mechanism of the peer connection object, uses all the exchanged network connection candidate addresses to form an address pair, and sends a connectivity check test packet to the receiving end.

[0016] All feasible connection paths are filtered out using the test package, and the direct connection path with the lowest latency is selected as the optimal transmission path.

[0017] If all direct connection paths fail to be established, the system will fall back to using the network address translation relay tunnel server path as the optimal transmission path.

[0018] In some embodiments of the method, the step of performing a handshake process via the Packet Transport Layer Security (PaLS) protocol to negotiate and generate an end-to-end encryption key includes:

[0019] The handshake process is performed through the data transport layer security protocol to encrypt the data channel end-to-end and negotiate to generate an end-to-end encryption key.

[0020] The audio and video media streams are encrypted using the end-to-end encryption key via a secure real-time transmission protocol.

[0021] In some embodiments of the method, the step of optimizing the communication transmission process in real time based on the network transmission status information includes:

[0022] By combining adaptive bitrate adjustment, forward error correction, and packet loss concealment, the audio and video transmission parameters are dynamically adjusted based on the network transmission status information. The audio and video transmission parameters include at least one of bitrate, resolution, and frame rate.

[0023] In some embodiments of the method, the interactive connection establishment server list includes a network address translation passthrough server list and / or a network address translation relay passthrough server list.

[0024] According to a second aspect of the present disclosure, a real-time audio and video interaction device is provided. The device includes:

[0025] The media capture module is used to call the media capture interface to obtain audio and video media streams, create peer connection objects, configure interactive connections for the peer connection objects, and establish a server list.

[0026] The information exchange module is used to generate Session Description Protocol (STP) request information, set it as a local media description, send it to the receiving end through the signaling channel, receive and process the STP response information fed back by the receiving end, complete the media capability negotiation between the two parties, and exchange network information.

[0027] The secure connection establishment module is used to execute the address connectivity check process of the interactive connection establishment framework, cooperate with the receiving end to determine the optimal transmission path for the audio and video media stream transmission, execute the handshake process through the data packet transport layer security protocol, and negotiate to generate an end-to-end encryption key.

[0028] The stable transmission module is used to encapsulate and transmit the audio and video media streams encrypted by the end-to-end encryption key, obtain network transmission status information based on the real-time transmission control protocol, and optimize the communication transmission process in real time based on the network transmission status information.

[0029] According to a third aspect of the present disclosure, a computer device is provided. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0030] Call the media capture interface to obtain audio and video media streams, create peer connection objects, configure interactive connections for the peer connection objects, and establish a server list;

[0031] A session description protocol request message is generated and set as a local media description. It is sent to the receiving end through a signaling channel. The session description protocol response message fed back by the receiving end is received and processed to complete the media capability negotiation between the two parties and exchange network information.

[0032] The address connectivity check process of the interactive connection establishment framework is executed, and the optimal transmission path of the audio and video media stream is determined in cooperation with the receiving end. The handshake process is executed through the data packet transport layer security protocol to negotiate and generate an end-to-end encryption key.

[0033] The audio and video media streams encrypted with the end-to-end encryption key are encapsulated and transmitted. Network transmission status information is obtained based on the real-time transmission control protocol, and the communication transmission process is optimized in real time based on the network transmission status information.

[0034] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0035] Call the media capture interface to obtain audio and video media streams, create peer connection objects, configure interactive connections for the peer connection objects, and establish a server list;

[0036] A session description protocol request message is generated and set as a local media description. It is sent to the receiving end through a signaling channel. The session description protocol response message fed back by the receiving end is received and processed to complete the media capability negotiation between the two parties and exchange network information.

[0037] The address connectivity check process of the interactive connection establishment framework is executed, and the optimal transmission path of the audio and video media stream is determined in cooperation with the receiving end. The handshake process is executed through the data packet transport layer security protocol to negotiate and generate an end-to-end encryption key.

[0038] The audio and video media streams encrypted with the end-to-end encryption key are encapsulated and transmitted. Network transmission status information is obtained based on the real-time transmission control protocol, and the communication transmission process is optimized in real time based on the network transmission status information.

[0039] According to a fifth aspect of the present disclosure, a computer program product is provided. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0040] Call the media capture interface to obtain audio and video media streams, create peer connection objects, configure interactive connections for the peer connection objects, and establish a server list;

[0041] A session description protocol request message is generated and set as a local media description. It is sent to the receiving end through a signaling channel. The session description protocol response message fed back by the receiving end is received and processed to complete the media capability negotiation between the two parties and exchange network information.

[0042] The address connectivity check process of the interactive connection establishment framework is executed, and the optimal transmission path of the audio and video media stream is determined in cooperation with the receiving end. The handshake process is executed through the data packet transport layer security protocol to negotiate and generate an end-to-end encryption key.

[0043] The audio and video media streams encrypted with the end-to-end encryption key are encapsulated and transmitted. Network transmission status information is obtained based on the real-time transmission control protocol, and the communication transmission process is optimized in real time based on the network transmission status information.

[0044] The real-time audio and video interaction solution provided in this application, through a complete process design, enables direct and secure real-time audio and video interaction between the sending and receiving ends. This eliminates excessive reliance on a central server and significantly reduces the bandwidth and server construction and maintenance costs for service providers. Simultaneously, through a low-latency transmission path and dynamic optimization mechanism, it significantly improves the real-time performance and smoothness of audio and video communication, providing users with a natural and fluid interactive experience and meeting the real-time communication needs of scenarios such as online education and remote collaboration.

[0045] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0047] Figure 1 This is a flowchart illustrating a real-time audio and video interaction method according to an exemplary embodiment;

[0048] Figure 2 This is a schematic diagram illustrating a specific process of a real-time audio and video interaction method according to an exemplary embodiment;

[0049] Figure 3 This is a structural block diagram of a real-time audio and video interaction device according to an exemplary embodiment;

[0050] Figure 4 This is a diagram illustrating the internal structure of a computer device according to an exemplary embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure. The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded. For example, the use of terms such as "first," "second," etc., to denote names does not indicate any specific order.

[0053] In some embodiments provided in this disclosure, the execution of the real-time audio and video interaction method can be controlled by a unified controller or by multiple controllers. These controllers may include controllers on local terminals or controllers on remote servers. In some embodiments, the controllers on local terminals and the controllers on servers may work together to complete the real-time audio and video interaction control processing. The local terminal mentioned in this disclosure may include, but is not limited to, various robotic devices, in-vehicle devices, personal computers, laptops, smartphones, tablets, wearable devices, medical devices, VR (Virtual Reality) devices, etc. The server may also be a server, server cluster, distributed subsystem, cloud processing platform, server containing blockchain nodes, or a combination thereof. The controllers described in this disclosure may include various control units capable of implementing logic processing functions, including but not limited to CPU (Central Processing Unit), PLC (Programmable Logic Controller), ECU (Electronic Control Unit), MCU (Microcontroller Unit), FPGA (Field Programmable Gate Array), and CPLD (Complex Programmable Logic Device), as well as controllers composed of one or more logic function units, chips, etc.

[0054] In some embodiments of this disclosure, a real-time audio and video interaction method is provided, such as... Figure 1 As shown, when applied to the initiating end, it includes the following steps:

[0055] S20. Call the media capture interface to obtain audio and video media streams, create peer connection objects, and configure interactive connection to establish a server list for the peer connection objects.

[0056] The media capture interface is a functional interface in a terminal device used to acquire audio and video data. It can call upon the device's hardware, such as its camera and microphone, to collect audio and video information that meets communication requirements. In some examples, the `navigator.media.getUserMedia(constraints)` API (Application Programming Interface) can be used. This allows the application to request permission from the user to access the camera and microphone. The `constraints` parameter specifies requirements such as video resolution (width: 1280, height: 720), frame rate, and whether only audio is required. It outputs a `MediaStream` object containing one or more audio / video tracks (`MediaStreamTrack`).

[0057] Peer-to-peer connection objects are the core functional modules supporting real-time audio and video communication. They integrate multiple capabilities such as connection establishment, data transmission, and encryption / decryption, and are the key carriers for realizing point-to-point communication.

[0058] The interactive connection establishment server list is a set of servers configured to achieve network traversal, used to help terminal devices bypass network address translation and firewall restrictions and find a communicable path.

[0059] S22. Generate Session Description Protocol Request information and set it as a local media description. Send it to the receiving end through the signaling channel. Receive and process the Session Description Protocol Response information fed back by the receiving end to complete the media capability negotiation between the two parties and exchange network information.

[0060] The Session Description Protocol (STP) request information is an information carrier in which the sender transmits its own media capabilities to the receiver. It includes key parameters such as supported audio and video encoding methods and resolutions, providing a basis for negotiation for media interaction between the two parties.

[0061] Local media description is descriptive data used by terminal devices to record their own media capabilities and related configuration information during real-time audio and video interaction. It serves as the basis for negotiating media parameters between the device and the peer.

[0062] The signaling channel is a communication channel used to transmit session description protocol-related information and network connection information, enabling information exchange between the two devices based on network communication technology.

[0063] The Session Description Protocol (STP) response information is the response information generated by the receiver after receiving the STP request information sent by the initiator, based on its own media device capabilities and network configuration.

[0064] S24. Execute the address connectivity check process of the interactive connection establishment framework, cooperate with the receiving end to determine the optimal transmission path for the audio and video media stream transmission, execute the handshake process through the data packet transport layer security protocol, and negotiate to generate an end-to-end encryption key.

[0065] The Interactive Connection Establishment Framework is a solution that integrates multiple network traversal technologies. Through a series of checks and negotiation processes, it determines the optimal communication path between the two devices.

[0066] The Transport Layer Security (TLS) protocol is used to ensure the security of data transmission. It establishes a secure communication channel through a handshake process to prevent data from being stolen or tampered with during transmission.

[0067] An end-to-end encryption key is a proprietary key negotiated through an encryption protocol. It is held only by the two communicating parties and is used to encrypt transmitted audio and video data.

[0068] S26. Encapsulate and transmit the audio and video media stream encrypted with the end-to-end encryption key, obtain network transmission status information based on the real-time transmission control protocol, and optimize the communication transmission process in real time based on the network transmission status information.

[0069] Real-time transmission control protocol is a protocol that assists the real-time transmission protocol in its operation. It can collect information related to network transmission status and provide data support for optimizing the transmission process.

[0070] In some embodiments of this disclosure, a complete process design enables direct and secure real-time audio and video interaction between the sending and receiving ends, eliminating excessive reliance on a central server and significantly reducing the bandwidth and server construction and maintenance costs for service providers. Simultaneously, through low-latency transmission paths and dynamic optimization mechanisms, the real-time performance and smoothness of audio and video communication are significantly improved, providing users with a natural and fluid interactive experience and meeting the real-time communication needs of scenarios such as online education and remote collaboration.

[0071] In some embodiments of this disclosure, S20 includes:

[0072] The list of interactive connection establishment servers includes a list of network address translation tunneling servers and / or a list of network address translation relay tunneling servers.

[0073] In some implementations, the sending end first invokes the device's media capture interface to request access to the camera and microphone from the user, acquiring a media stream containing audio and video tracks. These tracks will serve as the core content for subsequent transmissions. Next, the sending end creates a peer connection object, which is fundamental to all subsequent communication operations. During creation, an interactive connection establishment server list is configured for the peer connection object. This list may include a network address translation (NAT) tunneling server and a NAT relay tunneling server. The former helps the device obtain public IP address information, while the latter provides relay services when direct connection fails. This configuration prepares the device for overcoming network restrictions and establishing a connection.

[0074] In some examples, the list of interactive connection establishment servers includes a list of NAT traversal servers and / or a list of NAT relay traversal servers. STUN (Session Traversal Utilities for NAT) is used in the following scenario: a client sends a request to a STUN server on the public internet, and the server informs the client of its public IP address and port. This allows the client to identify its location on the public internet and exchange this information with a signaling server. TURN (Traversal Using Relaysaround NAT) is used in the following scenario: for strict networks that STUN cannot penetrate, such as symmetric NAT, ICE (Interactive Connectivity Establishment) uses a TURN server as a relay. Both clients connect to the same TURN server, and all data is forwarded through it.

[0075] In some examples, a list of STUN servers (e.g., {urls:'stun:stun.l.xxx.com:19302'}) and / or a list of TURN servers (username and credentials required) can be provided by configuration.

[0076] In some embodiments of this disclosure, the interactive connection establishment server list is flexibly configured, which can include both network address translation (NAT) traversal servers and NAT relay traversal servers, adapting to different network environments and traversal requirements. This flexible configuration method enhances the network traversal capability of the technical solution, enabling it to cope with NAT and firewall restrictions in various scenarios such as home networks and enterprise networks, providing comprehensive server support for subsequent connection establishment, and further improving the connection success rate.

[0077] In some embodiments of this disclosure, S22 includes:

[0078] Collect locally generated first network connection candidate addresses and send them to the receiving end via the signaling channel;

[0079] Receive the second network connection candidate address sent by the receiving end, and add the second network connection candidate address to the local peer connection object.

[0080] In some implementations, the sending end generates a Session Description Protocol (STP) request message, which details the audio and video encoding methods, resolutions, and other media capability parameters supported by the sending end. This request message is then set as a local media description, clarifying its own communication capabilities. The sending end transmits this STP request message to the receiving end via a pre-established signaling channel. Upon receiving the request message, the receiving end generates a STP response message based on its own media capabilities, also setting it as a local media description and sending it back to the sending end via the signaling channel. The sending end receives and processes this response message, completing the negotiation of media capabilities between the two parties and determining the commonly supported communication parameters. During this process, the sending end collects various locally generated network connection candidate addresses, including local network addresses and public network addresses obtained through a network address translation (NAT) server, and sends them to the receiving end via the signaling channel. Simultaneously, it receives the network connection candidate addresses sent by the receiving end and adds these addresses to its local peer connection objects, completing the exchange of network information.

[0081] In some examples, refer to Figure 2 The initiator creates an Offer: It calls `RTCPeerConnection.createOffer()` to create an SDP (Session Description Protocol) Offer. This SDP describes the media codecs (such as VP8, H.264, Opus) supported by the initiator, the resolution, and temporary media channel information. It then calls `RTCPeerConnection.setLocalDescription(offer)` to set the Offer as a local description. Finally, it sends this SDP Offer to the receiver via the signaling channel.

[0082] The receiver processes the offer: After receiving the offer via signaling, the receiver calls `RTCPeerConnection.setRemoteDescription(offer)` to inform its engine that "the other party provides these capabilities." The receiver calls `RTCPeerConnection.createAnswer()` to create an SDP Answer, which is a response to the offer and includes its supported codecs, etc. The receiver then calls `RTCPeerConnection.setLocalDescription(answer)` and sends this SDP Answer back to the initiator via the signaling channel.

[0083] The initiator finally confirmed: After receiving the answer, the initiator called setRemoteDescription(answer). At this point, the media capability negotiation between the two parties was completed.

[0084] In other examples, concurrently with SDP switching, the RTCPeerConnection object begins collecting ICE candidates (ICECandidates). A candidate is a possible connection address, which can include: host candidates, reflection candidates, and relay candidates. Host candidates are the local network interface card (NIC) IP addresses; reflection candidates are public IP addresses and ports obtained through the STUN server; and relay candidates are relay addresses assigned through the TURN server. Event Listening: When a new ICE candidate is collected, the onicecandidate event is triggered. Signaling Exchange: In this event handler, the application sends the candidate information to the peer via the signaling channel. The peer adds the candidate using the RTCPeerConnection.addIceCandidate(candidate) method.

[0085] In some embodiments of this disclosure, by standardizing the specific methods of network information exchange, it is ensured that the sending and receiving ends can fully obtain each other's candidate network connection addresses, providing sufficient address resources for subsequent connectivity checks. This comprehensive address interaction method helps to increase the range of available connection paths, thereby improving the success rate of connection establishment and reducing connection failures caused by incomplete network address information.

[0086] In some embodiments of this disclosure, S24 includes:

[0087] The interactive connection establishment framework invokes the built-in inspection mechanism of the peer connection object, uses all the exchanged network connection candidate addresses to form an address pair, and sends a connectivity check test packet to the receiving end.

[0088] All feasible connection paths are filtered out using the test package, and the direct connection path with the lowest latency is selected as the optimal transmission path.

[0089] If all direct connection paths fail to be established, the system will fall back to using the network address translation relay tunnel server path as the optimal transmission path.

[0090] In some implementations, the sending end initiates an address connectivity check process within the interactive connection establishment framework. Utilizing the built-in check mechanism of the peer connection object, it combines all candidate network connection addresses exchanged by both parties to form multiple address pairs. The sending end sends connectivity check test packets to the receiving end, filtering out all feasible paths capable of communication based on the transmission status of these test packets. Among these feasible paths, the direct connection path with the lowest transmission latency is prioritized as the optimal transmission path to ensure low-latency communication. If all direct connection paths fail to establish a connection, the process falls back to the path using a network address translation relay to penetrate the server as the optimal transmission path, ensuring normal communication. After the path is determined, the sending end performs a handshake process using the Transport Layer Security (TLS) protocol, encrypting the data channel end-to-end. During this process, an end-to-end encryption key is negotiated and generated, providing security for subsequent audio and video data transmission.

[0091] In some examples, the ICE framework can use all collected candidate address pairs to perform a series of STUN connectivity checks (i.e., sending test packets as described above) between the two parties. These checks will find all feasible connection paths (there may be multiple), and ultimately select the best path (usually the path with the lowest latency, such as a direct connection between reflection candidates). If the direct connection fails, it falls back to using a TURN relay.

[0092] In some embodiments of this disclosure, a systematic address connectivity check process can accurately select the optimal transmission path. Prioritizing direct paths ensures low-latency communication, while the fallback mechanism of relay paths ensures communication continuity in complex network environments. This path selection method balances low latency and stability, effectively addressing connectivity challenges in different network environments and improving the adaptability and reliability of the technical solution.

[0093] In some embodiments of this disclosure, S24 includes:

[0094] The handshake process is performed through the data transport layer security protocol to encrypt the data channel end-to-end and negotiate to generate an end-to-end encryption key.

[0095] The audio and video media streams are encrypted using the end-to-end encryption key via a secure real-time transmission protocol.

[0096] In some implementations, the sending end uses a negotiated end-to-end encryption key to encrypt the audio and video media streams, then encapsulates the encrypted media streams and transmits them to the receiving end through a determined optimal transmission path.

[0097] In some examples, once ICE successfully establishes a transport layer connection, a DTLS (Datagram Transport Layer Security) handshake is immediately performed on top of it. The entire data channel can be encrypted using DTLS, and a key for media encryption can be negotiated. The DTLS-negotiated key can then be used to encrypt all audio and video media streams using the Secure Real-time Transport Protocol (SRTP). Therefore, all communication is end-to-end encrypted by default.

[0098] In some embodiments of this disclosure, a dual encryption mechanism is constructed through the handshake process of the Data Packet Transport Layer Security (DLS) protocol and the encryption application of the Secure Real-Time Transport Protocol (SSL), achieving end-to-end encryption of audio and video data. This encryption method ensures that even if data is intercepted during transmission, it cannot be decrypted, effectively preventing data eavesdropping and tampering, fully protecting user communication privacy, complying with relevant data privacy protection requirements, and is suitable for audio and video communication in sensitive fields such as healthcare and finance.

[0099] In some embodiments of this disclosure, S26 includes:

[0100] By combining adaptive bitrate adjustment, forward error correction, and packet loss concealment, the audio and video transmission parameters are dynamically adjusted based on the network transmission status information. The audio and video transmission parameters include at least one of bitrate, resolution, and frame rate.

[0101] In some implementations, during transmission, the sending end continuously acquires network transmission status information based on a real-time transmission control protocol, including data such as packet loss, transmission delay, and jitter. Based on this network transmission status information, and in conjunction with adaptive bitrate adjustment, forward error correction, and packet loss concealment techniques, audio and video transmission parameters, including bitrate, resolution, and frame rate, are dynamically adjusted. When the network condition is good, transmission quality is appropriately improved; when network fluctuations occur, parameters are adjusted promptly to avoid stuttering, ensuring the stability and smoothness of the entire communication transmission process.

[0102] In some examples, encrypted audio and video data is transmitted via SRTP packets. Accompanying RTCP (RTP Control Protocol, the control protocol accompanying the Real-Time Transport Protocol) packets are continuously exchanged between the two parties, reporting network statistics such as packet loss rate, jitter, and round-trip time.

[0103] In other examples, the congestion control algorithm inside RTCPeerConnection dynamically adjusts the video bitrate, resolution, or frame rate based on RTCP feedback to adapt to changing network bandwidth. For packet loss, forward error correction and packet loss concealment (such as NetEQ for audio) techniques are enabled to compensate for data loss and ensure smoothness. By listening to RTCPeerConnection events (such as onconnectionstatechange), the application can obtain the connection status (connected, disconnected, failed, closed) and provide appropriate prompts or reconnection logic.

[0104] In some embodiments of this disclosure, multiple anti-network fluctuation technologies are combined to dynamically adjust transmission parameters according to network transmission status, enabling flexible responses to issues such as network bandwidth changes, packet loss, and jitter. When network conditions are poor, parameter adjustments prevent audio and video stuttering and pixelation; when network conditions are good, transmission quality is improved, providing users with a clear and smooth audio and video experience, significantly enhancing the adaptability and robustness of the technical solution in complex network environments.

[0105] The real-time audio and video interaction methods disclosed herein, through a complete process design, achieve direct and secure real-time audio and video interaction between the sending and receiving ends, eliminating excessive reliance on central servers and significantly reducing the bandwidth and server construction and maintenance costs for service providers. Simultaneously, through low-latency transmission paths and dynamic optimization mechanisms, the real-time performance and smoothness of audio and video communication are significantly improved, providing users with a natural and fluid interactive experience and meeting the real-time communication needs of scenarios such as online education and remote collaboration.

[0106] It is understood that the various embodiments of the methods described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. Related details can be found in the descriptions of other method embodiments.

[0107] It should be understood that although the steps in the flowcharts shown in the accompanying drawings are displayed sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least a portion of the steps or stages of other steps.

[0108] Based on the description of the above-described real-time audio and video interaction method embodiments, this disclosure also provides a real-time audio and video interaction device for implementing the above-described real-time audio and video interaction method. The device may include a system (including a distributed system), software (application), module, component, controller, server, terminal, etc., using the method described in the embodiments of this specification, combined with necessary hardware implementation. Based on the same innovative concept, the devices in one or more embodiments provided by the embodiments of this disclosure are as described in the following embodiments. Since the implementation schemes and methods for solving the problem by the devices are similar, the implementation of specific devices in the embodiments of this specification can refer to the implementation of the foregoing method, and repeated details will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0109] Figure 3 This is a schematic block diagram illustrating a real-time audio and video interaction device according to an exemplary embodiment. The device can be the aforementioned terminal, a server, or a module, component, device, control unit, etc., integrated into the terminal. For details, please refer to... Figure 3 The device 100 may include: a media capture module 120, an information exchange module 140, a secure connection establishment module 160, and a stable transmission module 180. The media capture module 120 is used to call a media capture interface to acquire audio and video media streams, create peer connection objects, and configure an interactive connection establishment server list for the peer connection objects. The information exchange module 140 is used to generate Session Description Protocol (STP) request information, set it as a local media description, send it to the receiving end through a signaling channel, receive and process STP response information from the receiving end, complete media capability negotiation between the two parties, and exchange network information. The secure connection establishment module 160 is used to execute the address connectivity check process of the interactive connection establishment framework, cooperate with the receiving end to determine the optimal transmission path for the audio and video media stream, execute a handshake process through the Packet Transport Layer Security (PaLP-SS) protocol, and negotiate to generate an end-to-end encryption key. The stable transmission module 180 is used to encapsulate and transmit the audio and video media stream encrypted with the end-to-end encryption key, acquire network transmission status information based on the Real-Time Transmission Control Protocol (RTC), and optimize the communication transmission process in real time based on the network transmission status information.

[0110] In some embodiments of the device, the information exchange module 140 is further configured to collect locally generated first network connection candidate addresses and send them to the receiving end via the signaling channel; it is also configured to receive a second network connection candidate address sent by the receiving end and add the second network connection candidate address to the local peer connection object.

[0111] In some embodiments of the device, the secure connection establishment module 160 is further configured to invoke the built-in inspection mechanism of the peer connection object through the interactive connection establishment framework, use all exchanged network connection candidate addresses to form address pairs, and send a connectivity check test packet to the receiving end; it is also configured to filter out all feasible connection paths through the test packet, and prioritize the direct connection path with the lowest latency as the optimal transmission path; and it is also configured to fall back to using the network address translation relay traversal server path as the optimal transmission path if all direct connection path establishment fails.

[0112] In some embodiments of the device, the secure connection establishment module 160 is further configured to perform a handshake process through the data packet transport layer security protocol, perform end-to-end encryption on the data channel, and negotiate to generate an end-to-end encryption key; it is also configured to use the end-to-end encryption key to encrypt the audio and video media stream through a secure real-time transmission protocol.

[0113] In some embodiments of the device, the stable transmission module 180 combines adaptive bitrate adjustment, forward error correction, and packet loss concealment to dynamically adjust audio and video transmission parameters according to the network transmission status information. The audio and video transmission parameters include at least one of bitrate, resolution, and frame rate.

[0114] In some embodiments of the device, the interactive connection establishment server list includes a network address translation passthrough server list and / or a network address translation relay passthrough server list.

[0115] Each module in the aforementioned real-time audio and video interactive device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0116] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a real-time audio and video interaction method.

[0117] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0118] Based on the foregoing description of the relevant methods and apparatus embodiments, this disclosure also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the real-time audio and video interaction method described in any embodiment of this specification.

[0119] Based on the foregoing description of the relevant methods and apparatus embodiments, this disclosure also provides a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by the processor of a computer device, enables the computer device to implement the real-time audio and video interaction method as described in any embodiment of this disclosure.

[0120] Based on the foregoing description of the relevant methods and apparatus embodiments, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the real-time audio and video interaction method described in any embodiment of this specification.

[0121] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, hardware + program embodiments are relatively simple in description because they are fundamentally similar to method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0122] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0123] It should be noted that the apparatus, computer equipment, storage medium, and computer program products described above may also include other implementation methods according to the description of the method embodiments. Specific implementation methods can be found in the description of the relevant method embodiments. Furthermore, new embodiments formed by combinations of features from various methods, apparatuses, devices, and server embodiments still fall within the scope of this disclosure and will not be elaborated upon here.

[0124] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling and communication connections between the devices or units shown or described can be implemented through direct and / or indirect coupling / connection, through standard or custom interfaces or protocols, and can be implemented electrically, mechanically, or in other forms.

[0125] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0126] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A real-time audio and video interaction method, characterized in that, Applied to the initiating end, the method includes: Call the media capture interface to obtain audio and video media streams, create peer connection objects, configure interactive connections for the peer connection objects, and establish a server list; A session description protocol request message is generated and set as a local media description. It is sent to the receiving end through a signaling channel. The session description protocol response message fed back by the receiving end is received and processed to complete the media capability negotiation between the two parties and exchange network information. The address connectivity check process of the interactive connection establishment framework is executed, and the optimal transmission path of the audio and video media stream is determined in cooperation with the receiving end. The handshake process is executed through the data packet transport layer security protocol to negotiate and generate an end-to-end encryption key. The audio and video media streams encrypted with the end-to-end encryption key are encapsulated and transmitted. Network transmission status information is obtained based on the real-time transmission control protocol, and the communication transmission process is optimized in real time based on the network transmission status information.

2. The method according to claim 1, characterized in that, The exchanged network information includes: Collect locally generated first network connection candidate addresses and send them to the receiving end via the signaling channel; Receive the second network connection candidate address sent by the receiving end, and add the second network connection candidate address to the local peer connection object.

3. The method according to claim 1, characterized in that, The address connectivity check process of the interactive connection establishment framework, in cooperation with the receiving end, determines the optimal transmission path for the audio and video media stream transmission, including: The interactive connection establishment framework invokes the built-in inspection mechanism of the peer connection object, uses all the exchanged network connection candidate addresses to form an address pair, and sends a connectivity check test packet to the receiving end. All feasible connection paths are filtered out using the test package, and the direct connection path with the lowest latency is selected as the optimal transmission path. If all direct connection paths fail to be established, the system will fall back to using the network address translation relay tunnel server path as the optimal transmission path.

4. The method according to claim 1, characterized in that, The handshake process performed via the Data Packet Transport Layer Security (DTS) protocol to negotiate and generate an end-to-end encryption key includes: The handshake process is performed through the data transport layer security protocol to encrypt the data channel end-to-end and negotiate to generate an end-to-end encryption key. The audio and video media streams are encrypted using the end-to-end encryption key via a secure real-time transmission protocol.

5. The method according to claim 1, characterized in that, The real-time optimization of the communication transmission process based on the network transmission status information includes: By combining adaptive bitrate adjustment, forward error correction, and packet loss concealment, the audio and video transmission parameters are dynamically adjusted based on the network transmission status information. The audio and video transmission parameters include at least one of bitrate, resolution, and frame rate.

6. The method according to claim 1, characterized in that, The list of interactive connection establishment servers includes a list of network address translation (NAT) tunneling servers and / or a list of network address translation (NAT) relay tunneling servers.

7. A real-time audio and video interactive device, characterized in that, The device includes: The media capture module is used to call the media capture interface to obtain audio and video media streams, create peer connection objects, configure interactive connections for the peer connection objects, and establish a server list. The information exchange module is used to generate Session Description Protocol (STP) request information, set it as a local media description, send it to the receiving end through the signaling channel, receive and process the STP response information fed back by the receiving end, complete the media capability negotiation between the two parties, and exchange network information. The secure connection establishment module is used to execute the address connectivity check process of the interactive connection establishment framework, cooperate with the receiving end to determine the optimal transmission path for the audio and video media stream transmission, execute the handshake process through the data packet transport layer security protocol, and negotiate to generate an end-to-end encryption key. The stable transmission module is used to encapsulate and transmit the audio and video media streams encrypted by the end-to-end encryption key, obtain network transmission status information based on the real-time transmission control protocol, and optimize the communication transmission process in real time based on the network transmission status information.

8. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images