Audio and video call method and system, and storage medium

Abstract

The invention relates to an audio and video call method and system, and a storage medium. The method comprises the following steps: collecting audio and video data; compressing and encrypting the collected audio and video data; transmitting the compressed audio and video data between the clients; and a receiving end receives the data and decodes the data, wherein the decoded audio data are subjected to real-time audio and video rendering and then are respectively output to audio equipment and display equipment. According to the method, the WebRTC can be efficiently integrated in the Unity engine, and the stability, synchronism and safety of audio and video transmission are ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of real-time audio and video communication, and relates to a method and system for audio and video calls using the WebRtc protocol, in particular to a method and system for audio and video calls. Background Art

[0002] As the demand for real-time audio and video communications continues to grow, online conferences, multiplayer games and other application scenarios have put forward higher requirements for low-latency, high-quality audio and video transmission. However, existing audio and video communication technologies face many challenges when integrated with the Unity engine. Network fluctuations often cause audio and video transmission to be delayed, unstable or out of sync, which cannot meet the needs of scenarios with strong real-time and high interactivity.

[0003] As an open source technology, WebRTC provides peer-to-peer (P2P) communication and a variety of audio and video codecs (such as Opus, VP8, H.264), which can achieve efficient data transmission and compression. However, the ability to effectively integrate WebRTC in Unity is still insufficient, especially when the network fluctuates greatly or the bandwidth is limited, the quality, synchronization and transmission stability of audio and video are difficult to guarantee. In addition, in the existing technology, the high availability of servers, the encryption protection of audio and video transmission and the overall communication efficiency are still insufficient, which limits its application effect in multi-person real-time interactive scenarios.

[0004] Therefore, how to efficiently integrate WebRTC in the Unity engine to ensure the stability, synchronization and security of audio and video transmission has become a technical problem that needs to be solved urgently.

[0005] Therefore, in order to solve the above problems, the present invention provides a method, system and storage medium for audio and video calls. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a method, system and storage medium for audio and video calls, which can efficiently integrate WebRTC in the Unity engine to ensure the stability, synchronization and security of audio and video transmission.

[0007] The present invention solves the practical problem by adopting the following technical solutions:

[0008] A method for audio and video calling, comprising the following steps:

[0009] Integrate the WebRTC library in the Unity engine;

[0010] Collect audio and video data;

[0011] Compressing the collected audio and video data;

[0012] Transmit compressed audio and video data between clients;

[0013] The receiving end receives the data and decodes it:

[0014] The decoded audio data is rendered in real time and then output to the audio device and display device respectively.

[0015] Moreover, the specific method of integrating the WebRTC library in the Unity engine is:

[0016] Open Package Manager in the Unity project and choose to add the WebRTC package via Git URL;

[0017] Enter the Git address of the WebRTC library and complete the installation;

[0018] After installation, the WebRTC API is automatically loaded, allowing developers to call the WebRTC protocol in Unity for audio and video capture, compression, transmission, and rendering.

[0019] Moreover, the specific method of collecting audio and video data is:

[0020] Audio and video data are collected through microphones and camera devices, and the frame rate, resolution and sampling rate are dynamically adjusted. In the process of audio and video data collection, a buffer mechanism and timestamp synchronization strategy are used.

[0021] Moreover, the method for compressing and encrypting the collected audio and video data is:

[0022] Opus codec is used to compress audio data, VP8 or H.264 codec is used to compress video data, and the bit rate, resolution, and frame rate in the compression parameters are dynamically adjusted according to network conditions to ensure the best transmission quality. During data transmission, the SRTP secure real-time transport protocol is used to encrypt audio and video data.

[0023] Moreover, the specific method of transmitting the compressed and encrypted audio and video data between clients is:

[0024] Through WebRTC's P2P peer-to-peer mechanism, compressed audio and video data are transmitted directly between clients; and when the P2P connection cannot be established, multiple TURN servers are used for relaying, and automatically switching to backup servers when the server fails or is overloaded.

[0025] Moreover, the specific steps of receiving and decoding the data at the receiving end include:

[0026] At the receiving end of audio and video communication, the client receives audio and video data transmitted by the remote user through the WebRTC protocol, and then the receiving end decodes the audio and video data through the decoder. The received data is first decoded into the audio and video encapsulation format RTP real-time transport protocol through the network layer, and then handed over to the corresponding codec for decoding;

[0027] A buffering mechanism is used to handle network jitter, and a timestamp synchronization mechanism is used to keep audio and video data synchronized during playback.

[0028] Moreover, the specific method of performing real-time audio and video rendering on the decoded audio data and outputting them to the audio device and the display device respectively is:

[0029] In the Unity scene, the decoded audio data is played and the video frames are rendered to achieve smooth display of real-time audio and video calls.

[0030] A system for audio and video calls, comprising:

[0031] WebRTC library integration module, which is used to implement the WebRTC protocol in Unity and handle the real-time transmission, encoding, decoding and connection management of audio and video data;

[0032] Audio and video acquisition module, used to collect audio and video data;

[0033] Compression and encryption module, used to compress and encrypt the collected audio and video data;

[0034] The transmission module transmits the compressed audio and video data between clients;

[0035] Receiving module, the receiving end receives data and decodes it:

[0036] The rendering module performs real-time audio and video rendering on the decoded audio data and outputs them to the audio device and display device respectively.

[0037] Moreover, the audio and video acquisition module is also used for:

[0038] Audio and video data are collected through microphones and camera devices, and the frame rate, resolution and sampling rate are dynamically adjusted. In the process of audio and video data collection, a buffer mechanism and timestamp synchronization strategy are used.

[0039] A storage medium stores a computer program, which, when executed by a processor, implements the method for audio and video calls as claimed in any one of claims 1 to 7.

[0040] Advantages and beneficial effects of the present invention:

[0041] 1. The present invention develops a unified audio and video acquisition module by utilizing the cross-platform characteristics of Unity to support multi-device adaptation. Through adaptive acquisition strategies, acquisition parameters (such as resolution, frame rate, etc.) are dynamically adjusted to reduce delays, ensure the synchronization of audio and video, and ensure data consistency under multiple platforms. At the same time, efficient audio and video encoding and decoding algorithms (such as Opus and VP8) are used for compression, which can not only ensure audio and video quality, but also reduce bandwidth consumption. During the compression process, the end-to-end encryption function of the WebRTC protocol is combined to ensure the security of data during transmission and prevent data from being tampered with or stolen.

[0042] 2. The present invention optimizes real-time data transmission by integrating the WebRTC protocol and taking advantage of the Unity platform. Adopting an adaptive bandwidth control mechanism, the video quality is dynamically adjusted when the network fluctuates, delay is reduced, and the stability of data transmission is improved. At the same time, the collaborative work of audio and video data and graphics rendering is optimized to ensure that the audio and video streams are output synchronously with the rendering, and the phenomenon of audio and video being out of sync with graphics rendering is reduced in multi-person interactive scenarios. Transmitting audio and video streams in P2P mode reduces dependence on the server and improves the transmission efficiency of the system.

[0043] 3. The present invention uses the built-in decoding library of WebRTC, directly uses Opus, VP8 or H.264 codecs to decode audio and video streams, and uses WebRTC's optimized decoding algorithm to reduce delays and improve decoding efficiency. Combined with Unity's hardware acceleration support, low latency and high performance are ensured during the decoding process to avoid freezes or asynchronization problems. In addition, by utilizing Unity's powerful graphics rendering capabilities, multi-threaded rendering and resource scheduling optimization are used to reduce rendering delays and ensure synchronous output of audio and video streams. In multi-person real-time interactive scenarios, the rendering pipeline technology is optimized to ensure smoothness during the rendering process and eliminate screen freezes. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of audio and video data collection of the present invention;

[0045] Figure 2 The present invention is an audio and video data compression flow chart;

[0046] Figure 3 It is a flow chart of P2P connection and data transmission of the present invention;

[0047] Figure 4 It is a receiving buffer and data decoding flow chart of the present invention;

[0048] Figure 5 The flowchart of the audio and video data rendering of the present invention;

[0049] Figure 6 The figure is a complete flow chart of the audio and video call of the present invention. DETAILED DESCRIPTION

[0050] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0051] A method for audio and video calls, such as Figures 1 to 6 As shown, the following steps are included:

[0052] Step 1. Integrate the WebRTC library into the Unity engine.

[0053] The specific method of integrating the WebRTC library in the Unity engine is:

[0054] Open Package Manager in the Unity project and choose to add the WebRTC package via Git URL;

[0055] Enter the Git address of the WebRTC library

[0056] https: / / github.com / Unity-Technologies / com.unity.webrtc.git and complete the installation;

[0057] After installation, the WebRTC API is automatically loaded, allowing developers to call the WebRTC protocol in Unity for audio and video capture, compression, transmission, and rendering.

[0058] Step 2: Collect audio and video data;

[0059] The specific method for collecting audio and video data is:

[0060] Audio and video data are collected through microphones and camera devices, and the frame rate, resolution and sampling rate are dynamically adjusted to adapt to different network conditions. In the process of audio and video data collection, buffer mechanisms and timestamp synchronization strategies are used to ensure the continuity and synchronization of data during collection and transmission.

[0061] In this embodiment, the acquisition strategy of audio and video data is a crucial link, which affects the subsequent encoding, transmission and decoding effects. To adapt to different application scenarios, the system allows developers to flexibly configure audio and video acquisition parameters, including frame rate, resolution, sampling rate and buffer mechanism, and introduces audio and video synchronization strategy to ensure that the acquired data is synchronized during transmission and rendering.

[0062] The working principle of step 2 is:

[0063] 1. Audio collection strategy:

[0064] 1). Configuration of sampling rate and number of channels:

[0065] The sampling rate (such as 44.1kHz, 48kHz) and number of channels (mono, stereo) of audio acquisition can be configured according to the specific application scenario. Higher sampling rates are suitable for high-quality audio requirements, such as music or immersive scenarios, while lower sampling rates are suitable for real-time communication scenarios with limited bandwidth, such as voice calls.

[0066] int sampleRate = 44100; / / The sampling rate can be adjusted according to needs

[0067] int channels = 2; / / 1 represents mono, 2 represents stereo

[0068] AudioClip audioClip=Microphone.Start(null,true,10,sampleRate);

[0069] 2). Buffering mechanism for audio collection:

[0070] When collecting audio data in real time, the system uses a buffering mechanism to ensure the smoothness of audio data and prevent delays or fluctuations in the collection device from affecting the stable collection of audio. The buffer is used to temporarily store audio frames to ensure that stable audio data can still be output when the load of the collection device increases or fluctuates.

[0071] 3). Buffer size configuration:

[0072] Developers can configure the size of the audio buffer (such as 512 frames, 1024 frames) according to the application scenario. A larger buffer is suitable for scenarios with more processing delays, such as complex audio effect processing, while a smaller buffer is suitable for scenarios with higher real-time requirements.

[0073] int bufferSize = 1024; / / buffer size

[0074] float[]audioBuffer=new float[bufferSize]; / / buffer storage

[0075] audioClip.GetData(audioBuffer,0); / / Get audio frame data

[0076] 4). Collection part of audio and video synchronization mechanism:

[0077] To ensure the synchronization of audio and video in subsequent transmission and rendering, the audio collector will record the timestamp of each frame of audio and keep it consistent with the timestamp of video acquisition. The timestamp is used to ensure that the audio frame is transmitted and played synchronously with the corresponding video frame in subsequent processing.

[0078] float audioTimestamp = Time.time; / / Get the audio timestamp of the current frame

[0079] 2. Video acquisition strategy:

[0080] 1). Frame rate and resolution configuration:

[0081] The frame rate and resolution of the video can be flexibly configured according to device performance and bandwidth conditions. Lower frame rates (such as 15FPS) are suitable for scenarios with limited bandwidth or high real-time requirements, while higher frame rates (such as 30FPS, 60FPS) are suitable for high-quality scenarios such as games or HD video calls. Resolution support ranges from 640x480 to 1920x1080, which can be selected according to network conditions and device performance.

[0082] WebCamTexture webCamTexture=new WebCamTexture();

[0083] webCamTexture.requestedFPS = 30; / / Set the frame rate

[0084] webCamTexture.requestedWidth = 1280; / / Set the resolution width

[0085] webCamTexture.requestedHeight = 720; / / Set the resolution height

[0086] webCamTexture.Play();

[0087] 2). Video frame buffering mechanism:

[0088] The capture of video frames also uses a buffer mechanism to ensure smooth frame capture, especially when the device processing power is limited or the network bandwidth fluctuates. The buffer is used to store the captured video frames to ensure that the frame data will not be lost or delayed due to fluctuations in device load.

[0089] 3). Buffer size configuration:

[0090] The video frame buffer size can be dynamically adjusted according to the frame rate and resolution. For example, a larger buffer is suitable for video capture with higher resolution and frame rate, ensuring stable video data capture under high load conditions.

[0091] int videoBufferSize = 5; / / Video buffer size in frames

[0092] Color32[][]videoBuffer = new Color32[videoBufferSize][]; / / buffer storage

[0093] videoBuffer[0] = webCamTexture.GetPixels32(); / / Get a frame of video data

[0094] 4). Collection part of audio and video synchronization mechanism:

[0095] When capturing video, the system will also record the timestamp of each frame of video to ensure synchronization with the audio timestamp. The video timestamp is used for subsequent transmission and rendering processing to ensure that the audio and video can maintain time consistency during playback.

[0096] float videoTimestamp = Time.time; / / Get the video timestamp of the current frame

[0097] 3. Audio and video synchronization mechanism at the acquisition end:

[0098] At the acquisition end, it is very important to synchronize the timestamps of audio and video to ensure that they are consistent during the acquisition process. By recording the timestamps of audio and video frames at the same time, the system can play the audio and video data synchronously based on these timestamps during the subsequent transmission, decoding and rendering process.

[0099] Timestamp recording and calibration: Each time a frame of audio or video is captured, the current timestamp is recorded and the timestamp and the corresponding audio and video data are stored in the buffer. The system will calibrate the timestamps of audio and video as needed to ensure that the two can be seamlessly synchronized during transmission and rendering.

[0100] / / Collect the timestamps of audio and video frames at the same time

[0101] float currentTimestamp=Time.time;

[0102] audioTimestamp=currentTimestamp;

[0103] videoTimestamp=currentTimestamp;

[0104] Step 3, compressing the collected audio and video data;

[0105] The method for compressing the collected audio and video data is as follows:

[0106] Opus codec is used to compress audio data, VP8 or H.264 codec is used to compress video data, and compression parameters (bit rate, resolution, frame rate) are dynamically adjusted according to network conditions to ensure the best transmission quality. In the process of data transmission, SRTP protocol is used to encrypt audio and video data to ensure the security of audio and video during transmission.

[0107] In this embodiment, after the audio and video data collection is completed, the system will compress the audio and video data to reduce the data transmission bandwidth and delay. The core of the compression strategy is to select a suitable codec and configure corresponding parameters according to the actual scenario, such as bit rate, frame rate and resolution.

[0108] The working principle of step 3 is:

[0109] 1. Audio data compression:

[0110] Audio data is compressed using the Opus codec, a highly efficient codec designed for real-time communications that supports multiple modes from narrowband to full-band and can dynamically adapt to network conditions.

[0111] 1). Compression parameter configuration:

[0112] The Opus codec can automatically adjust audio compression parameters, such as bitrate, based on network bandwidth. Typically, voice calls can choose a lower bitrate (such as 16kbps), while high-definition audio can choose a higher bitrate (such as 64kbps or higher). Developers can manually configure or use the automatic adjustment mechanism.

[0113] int audioBitrate = 32000; / / Set the audio bit rate, unit: bps

[0114] byte[]compressedAudioData = OpusEncoder.Encode(audioData); / / compressed audio data

[0115] 2) Adaptability:

[0116] Opus has good adaptability and can dynamically adjust the audio compression ratio and bit rate according to the network quality to ensure clear voice quality when the network fluctuates. This feature is very suitable for network environments with unstable bandwidth or delay.

[0117] 2. Video data compression:

[0118] Video data is usually compressed using codecs such as VP8, VP9, ​​or H.264. VP8 and VP9 are the default video compression formats for WebRTC and support low-latency communications, while H.264 has wide hardware compatibility and high compression efficiency.

[0119] Compression format selection: Developers can choose the appropriate compression format based on the device and network conditions. VP8 is suitable for low-latency scenarios, while H.264 is suitable for scenarios with higher video quality requirements. VP9 can still provide relatively high video quality under low bandwidth.

[0120] string videoCodec = "VP8"; / / Select video compression format

[0121] byte[]compressedVideoData = VideoEncoder.Encode(videoFrameData); / / compress video data

[0122] Compression parameter configuration: The bit rate, frame rate and resolution of the video can be adjusted dynamically according to the scene. Low-bandwidth scenes can choose lower bit rates and frame rates (such as 15fps, 500kbps), while high-quality scenes can use higher bit rates and frame rates (such as 30fps, 1Mbps).

[0123] int videoBitrate = 1000000; / / Set the video bit rate, unit: bps

[0124] 3. Compression strategy combined with encoding buffer:

[0125] In order to ensure the real-time performance of the encoding process, the audio and video data will be buffered before entering the encoder to ensure that the quality will not be degraded due to data delay or loss during the encoding process. This buffering mechanism can prevent compression failures caused by data fluctuations or delays, while ensuring the synchronization of audio and video frame sequences.

[0126] The audio and video encoding buffer is used to store the audio and video frames to be encoded, ensuring that each frame can be encoded smoothly. The size of the audio and video encoding buffer can be dynamically adjusted according to the frame rate, sampling rate and resolution.

[0127] / / Encoding buffer example

[0128] int audioEncodeBufferSize=512;

[0129] int videoEncodeBufferSize=5;

[0130] byte[]audioEncodeBuffer=new byte[audioEncodeBufferSize];

[0131] Color32[][]videoEncodeBuffer=new Color32[videoEncodeBufferSize][];

[0132] Step 4: Transmit the compressed audio and video data between clients;

[0133] The specific method of transmitting the compressed and encrypted audio and video data between clients is as follows:

[0134] Through the P2P mechanism of WebRTC, compressed audio and video data are transmitted directly between clients, reducing server load and latency. When the P2P connection cannot be established, multiple TURN servers are used for relaying, and when a server fails or is overloaded, it automatically switches to a backup server to ensure transmission stability.

[0135] In this embodiment, in the point-to-point connection process of WebRTC, the configuration of the server is crucial, and is used to coordinate the connection of devices in different network environments to ensure the smooth transmission of audio and video data.

[0136] WebRTC uses a variety of server types to solve network penetration and connection problems, mainly including signaling servers, STUN servers, and TURN servers.

[0137] The working principle of step 4 is:

[0138] 1. Signaling Server:

[0139] The main function of the signaling server is to help clients exchange connection information, such as SDP (Session Description Protocol) and ICE (Interactive Connection Establishment) candidates, to help establish P2P connections. The signaling server does not transmit audio and video data, it is only used for initial connection negotiation. In this system, the signaling server is used to create a room mechanism, which is identified by a unique room ID, allowing multiple clients to join the same room for audio and video calls.

[0140] 1). Room Mechanism:

[0141] Each audio / video call or conference scenario is identified by a room, and clients can join the same room using the room ID. The signaling server is responsible for managing room creation, client joining and leaving, and the member list in the room. When two or more clients join the same room, the signaling server distributes their respective connection information (SDP and ICE candidates) to enable point-to-point connections between clients.

[0142] 2). Connection process:

[0143] The client sends a request to join the room to the signaling server.

[0144] The signaling server creates or joins a room and returns a list of room members.

[0145] The signaling server distributes SDP and ICE candidate information between clients to help establish P2P connections.

[0146] / / The client joins the room managed by the signaling server through the room ID

[0147] string roomID = "12345";

[0148] signalingServer.JoinRoom(roomID);

[0149] 2.STUN server (Session Traversal Utilities for NAT):

[0150] The STUN server is used to help the client obtain its IP address in the public network so that it can penetrate NAT (Network Address Translation). In this system, the configuration of the STUN server is very important, especially when the clients are in different network environments. The STUN server can resolve the public IP and port of each client, so that the two devices can establish a direct connection through the public network.

[0151] STUN server configuration: Developers can configure multiple STUN server addresses according to actual needs to increase the probability of successful connection, especially in complex network conditions. Multiple STUN servers can improve the stability of the connection and ensure P2P connection under different network conditions.

[0152]

[0153] 3.TURN server (Traversal Using Relays around NAT):

[0154] TURN servers are used to relay audio and video data when a point-to-point connection cannot be established. Especially when both parties are behind strict NAT or firewalls, TURN servers can act as intermediaries to ensure smooth data transmission. To ensure high availability and fault tolerance of the system, multiple TURN servers can be configured to prevent connection interruptions when a single TURN server goes down or is overloaded.

[0155] 1). High availability configuration of TURN server:

[0156] In enterprise-level or large-scale application scenarios, multiple TURN servers are usually configured to ensure system stability and high availability. These TURN servers can be distributed in different data centers or regions, and the client can automatically select the best TURN server for connection based on the network conditions. When a TURN server is unavailable due to load or network problems, the system can automatically switch to a backup TURN server to avoid connection failures caused by server failures, ensuring continuous transmission of audio and video and unaffected user experience.

[0157] 2). Load balancing of TURN servers:

[0158] In order to prevent a single TURN server from being overloaded, the system can configure a load balancing strategy to ensure that the resources of each TURN server can be reasonably utilized. When selecting a TURN server, the client can dynamically select the optimal server for relaying based on latency, bandwidth, or load conditions.

[0159]

[0160] 3) Room management and data transmission process:

[0161] Creating and joining a room: After the client is started, it first creates or joins a room through the signaling server, which manages the member information of each room. When multiple clients join the same room, the signaling server distributes connection information to each client to help establish a P2P connection.

[0162] NAT penetration and connection establishment: The client first tries to resolve its public IP address through the STUN server. If the client is behind NAT, the STUN server can help establish a direct connection between devices. If STUN penetration fails, the TURN server will act as a relay to help forward audio and video data between clients.

[0163] Transmission and encryption of audio and video data: After the connection is established, the client directly transmits audio and video data through P2P, and the data is encrypted through SRTP (Secure Real-time Transport Protocol) to ensure data security during transmission. If the P2P connection cannot be established, the audio and video data will be transmitted through the TURN server relay, and the TURN server also ensures encrypted data transmission.

[0164]

[0165] Step 5: The receiving end receives and decodes the data:

[0166] The specific steps of receiving and decoding data at the receiving end include:

[0167] (1) At the receiving end of audio and video communication, the client receives audio and video data transmitted by the remote user through the WebRTC protocol, and then the receiving end decodes the audio and video data through a decoder. The received data is first decoded into the audio and video encapsulation format (RTP) through the network layer, and then handed over to the corresponding codec (such as Opus, VP8, H.264) for decoding.

[0168] (2) A buffer mechanism is used to handle network jitter to ensure the continuity of audio and video data. A timestamp synchronization mechanism is used to keep audio and video data synchronized during playback to avoid the problem of audio and video being out of sync.

[0169] In this embodiment, at the receiving end of the audio and video communication, the client receives the audio and video data transmitted by the remote user through the WebRTC protocol. The received data is first decoded into the audio and video encapsulation format (RTP) by the network layer, and then handed over to the corresponding codec (such as Opus, VP8, H.264) for decoding.

[0170] Receiving audio and video data streams: After successfully establishing a P2P connection or relaying through a TURN server, the client begins to receive the audio and video data streams transmitted by the remote user.

[0171] Audio and video data are transmitted in the form of RTP (Real-time Transport Protocol) packets, which contain encoded frame information of audio and video.

[0172] The working principle of step 5 is:

[0173] 1. Audio reception and decoding:

[0174] After the audio data is decrypted by SRTP (Secure Real-time Transport Protocol), it enters the audio codec for processing. The Opus codec is responsible for decoding the received audio data and converting it back to PCM (Pulse Code Modulation) format for subsequent audio playback.

[0175] The decoded audio data can be stored in a buffer in the form of frames to ensure audio continuity and be able to handle delays or packet loss caused by network jitter.

[0176] / / Simulate audio data reception and decoding

[0177] byte[]receivedAudioData=ReceiveDataFromNetwork(); / / Received audio data

[0178] float[] decodedAudioData = OpusDecoder.Decode(receivedAudioData); / / decode to PCM data

[0179] 2. Video reception and decoding:

[0180] The video data is also decrypted by SRTP and decoded by video codecs such as VP8 or H.264. The decoded data is frame image data, usually presented in YUV or RGB format.

[0181] During the video decoding process, the timestamp information in the transmission is used to ensure the correct order and time synchronization of the video frames, thereby maintaining the synchronization of audio and video.

[0182] / / Simulate video data reception and decoding

[0183] byte[]receivedVideoData=ReceiveDataFromNetwork(); / / Received video data

[0184] Color32[]decodedVideoFrame=VideoDecoder.Decode(receivedVideoData); / / decode to image frame data

[0185] 3. Network jitter and delay processing:

[0186] In audio and video transmission, network fluctuations and delays may cause data loss or unstable transmission. To ensure the smoothness of audio and video calls, the system uses a jitter buffer mechanism on the receiving end to handle network jitter. By temporarily storing and reordering received data frames in the buffer, stable audio and video playback can be ensured when the network jitters.

[0187] Unity's WebRTC library is a plug-in officially developed and maintained by Unity, which is used to implement real-time audio and video communication and peer-to-peer data transmission in Unity projects. It is based on the open source WebRTC technology, supports P2P transmission of audio, video and data, and is widely used in video calls, online conferences, multiplayer games and other scenarios.

[0188] In WebRTC, Opus, VP8, and H.264 are the core audio and video codecs responsible for the compression and decompression of audio and video data to achieve low-latency, high-quality real-time communication. These codecs are crucial to WebRTC's audio and video processing:

[0189] Opus Codec: Opus is an efficient audio codec designed for real-time audio communication. It supports multiple modes from narrowband (such as voice) to full-band (such as music), and can automatically adjust the bit rate and sound quality according to network conditions. In WebRTC, Opus is widely used for voice and audio transmission, and has good anti-network jitter and bandwidth adaptability, ensuring clear audio quality when network conditions fluctuate.

[0190] VP8 codec: VP8 is the default video codec for WebRTC, designed for low-latency video communication and suitable for scenarios with poor network conditions or high real-time requirements. It can provide high video quality under limited bandwidth and supports dynamic adjustment of compression parameters to cope with network fluctuations.

[0191] H.264 codec: As a widely used video compression standard, H.264 is also important in WebRTC, especially for scenarios with high video quality requirements. H.264 has good hardware support and high compression efficiency, and can provide high-definition and stable video transmission experience under high bandwidth.

[0192] In WebRTC, audio and video data are compressed by these codecs, transmitted in smaller data volumes, and then decoded and restored at the receiving end, thereby effectively reducing network bandwidth usage and latency while maintaining high quality.

[0193] Step 6: The decoded audio data is rendered in real time and then output to the audio device and the display device respectively;

[0194] The specific method of performing real-time audio and video rendering on the decoded audio data is as follows:

[0195] In the Unity scene, play the decoded audio data through the AudioSource component and render the video frames to Texture2D or RenderTexture to achieve smooth display of real-time audio and video calls.

[0196] In this embodiment, after decoding the audio and video data, the next step is to render these data and output them to the audio device and the display device respectively, to ensure that the user can receive the audio and video content in real time.

[0197] The working principle of step 6 is:

[0198] 1. Audio rendering:

[0199] The decoded PCM audio data is transmitted to the `AudioSource` component in Unity for playback. The `AudioSource` component can process mono or stereo audio data. Developers can achieve a more immersive audio experience by setting different audio parameters (such as volume, 3D sound effects).

[0200] Audio playback and synchronization: To ensure the synchronization of audio and video, audio playback needs to be synchronized and controlled according to timestamps. The playback buffer of audio data must be consistent with the timestamp of the video frame to ensure that the two are played synchronously.

[0201] / / Audio playback and synchronization processing

[0202] AudioSource audioSource=gameObject.AddComponent <audiosource>();

[0203] audioSource.clip=AudioClip.Create("ReceivedAudio",audioData.Length,channels,sampleRate,false);

[0204] audioSource.SetData(decodedAudioData,0); / / Write the decoded PCM data into the audio source

[0205] audioSource.Play();

[0206] 2. Video rendering:

[0207] The frame image data after the video data is decoded is transmitted to the Unity material or UI component. The display is usually updated by assigning the decoded frame image data to the `RenderTexture` or `Texture2D` material.

[0208] 1). Video frame rendering:

[0209] Video frames are rendered in the order they are received and synchronized with the audio data. Developers can use the `Texture2D` class to render the decoded RGB image data as a texture onto a 3D object or UI plane to display real-time video.

[0210] / / Video frame rendering and synchronization processing

[0211] Texture2D videoTexture=new Texture2D(videoWidth, videoHeight, TextureFormat.RGBA32, false);

[0212] videoTexture.SetPixels32(decodedVideoFrame); / / Decoded frame data

[0213] Writing to Texture

[0214] videoTexture.Apply(); / / Apply texture update

[0215] / / Render to 3D object or UI interface

[0216] Renderer renderer=gameObject.GetComponent <renderer>();

[0217] renderer.material.mainTexture=videoTexture;

[0218] 2). Video synchronization processing:

[0219] To ensure synchronous playback of audio and video, the timestamps of video frame rendering and audio playback must be strictly aligned. If network latency causes video frames to arrive late, the system can discard the lagging frames to ensure synchronization.

[0220] A system for audio and video calls, comprising:

[0221] WebRTC library integration module, which is used to implement the WebRTC protocol in Unity and handle the real-time transmission, encoding, decoding and connection management of audio and video data;

[0222] Audio and video acquisition module, used to collect audio and video data;

[0223] Compression and encryption module, used to compress and encrypt the collected audio and video data;

[0224] The transmission module transmits the compressed audio and video data between clients;

[0225] Receiving module, the receiving end receives data and decodes it:

[0226] The rendering module performs real-time audio and video rendering on the decoded audio data and outputs them to the audio device and display device respectively.

[0227] The audio and video acquisition module is also used for:

[0228] Audio and video data are collected through microphones and camera devices, and the frame rate, resolution and sampling rate are dynamically adjusted. In the process of audio and video data collection, a buffer mechanism and timestamp synchronization strategy are used.

[0229] A storage medium stores a computer program, which implements an audio and video call method when executed by a processor.

[0230] The technical problems to be solved by the present invention are:

[0231] 1. Audio and video collection and compression issues: In audio and video communications, data collection often faces delays and poor synchronization issues. Especially in multi-person interactive scenarios, differences in device adaptation and platform differences exacerbate these problems, resulting in inconsistent quality of collected data and affecting audio and video transmission. At the same time, audio and video data needs to be compressed, but excessive compression will affect quality, and the lack of effective encryption protection can easily lead to data leakage.

[0232] Solution: This invention develops a unified audio and video acquisition module by utilizing the cross-platform characteristics of Unity to support multi-device adaptation. Through adaptive acquisition strategies, the acquisition parameters (such as resolution, frame rate, etc.) are dynamically adjusted to reduce delays, ensure the synchronization of audio and video, and ensure data consistency under multiple platforms. At the same time, efficient audio and video encoding and decoding algorithms (such as Opus, VP8) are used for compression, which can not only ensure the quality of audio and video, but also reduce bandwidth consumption. During the compression process, combined with the end-to-end encryption function of the WebRTC protocol, the security of data during transmission is ensured to prevent data tampering or theft.

[0233] 2. Data transmission issues: During audio and video data transmission, network fluctuations and bandwidth limitations often lead to transmission delays and instability, affecting call quality. At the same time, insufficient coordination between audio and video data transmission and graphics rendering systems, especially in multi-person interaction scenarios, leads to synchronization issues between audio and video streams and graphics rendering, affecting the overall user experience.

[0234] Solution: This invention integrates the WebRTC protocol and takes advantage of the Unity platform to optimize real-time data transmission. Adopt an adaptive bandwidth control mechanism to dynamically adjust the video quality when the network fluctuates, reduce latency, and improve the stability of data transmission. At the same time, optimize the collaborative work of audio and video data and graphics rendering to ensure that the audio and video streams are output synchronously with the rendering, and reduce the phenomenon of audio and video and graphics rendering being out of sync in multi-person interactive scenarios. Transmit audio and video streams in P2P mode to reduce dependence on the server and improve the transmission efficiency of the system.

[0235] 3. Decoding and rendering issues: When receiving and decoding audio and video data, high decoding delay or low decoding efficiency may cause audio and video asynchrony or freezes. At the same time, in the audio and video rendering process, especially in multi-person interaction scenarios, how to ensure low-latency rendering and synchronous playback of multiple audio and video streams is a technical challenge.

[0236] Solution: The present invention uses the built-in decoding library of WebRTC, directly uses Opus, VP8 or H.264 codecs to decode audio and video streams, and uses WebRTC's optimized decoding algorithm to reduce delays and improve decoding efficiency. Combined with Unity's hardware acceleration support, low latency and high performance are ensured during the decoding process to avoid freezes or asynchrony problems. In addition, by utilizing Unity's powerful graphics rendering capabilities, multi-threaded rendering and resource scheduling optimization are used to reduce rendering delays and ensure synchronous output of audio and video streams. In multi-person real-time interactive scenarios, the rendering pipeline technology is optimized to ensure smoothness during the rendering process and eliminate screen freezes.

[0237] The innovation of the present invention lies in:

[0238] 1. P2P transmission mechanism: Through WebRTC's point-to-point transmission, audio and video data are directly transmitted between clients, reducing latency and improving real-time performance. Through the P2P transmission mechanism, the latency of audio and video calls is effectively reduced, improving user experience.

[0239] 2. Multi-server redundant configuration: Supports multiple TURN server configurations to ensure high availability and automatically switch to backup servers when the server goes down or the load is too high. Use multi-server redundant configuration to ensure high system availability and is suitable for multi-user scenarios.

[0240] 3. Audio and video synchronization and buffering mechanism: During the audio and video acquisition, transmission and decoding process, a buffering mechanism and timestamp synchronization are used to ensure the stability and synchronization of audio and video.

[0241] 4. Encrypted transmission: Use the SRTP protocol to encrypt audio and video data to ensure data security during transmission. The SRTP protocol is used for encrypted transmission to ensure data security, which is suitable for application scenarios with high requirements for privacy protection.

[0242] 5. It has the ability to dynamically adapt to network bandwidth and can maintain high-quality audio and video communications under different network conditions.

[0243] The detection method used in the actual product of the present invention is:

[0244] 1. Audio and video acquisition and compression strategy:

[0245] Check whether Unity's Microphone and WebCamTexture components are used for audio and video capture, and check whether codecs such as Opus, VP8, and H.264 and mechanisms for dynamically adjusting compression parameters are applied in the product.

[0246] 2.P2P transmission and server redundancy:

[0247] Use network monitoring tools to verify whether audio and video data are transmitted in P2P mode and whether multiple TURN servers are configured for relay and automatic switching.

[0248] 3. Audio and video synchronization and buffering mechanism:

[0249] Check the product's audio and video playback module to confirm whether there is buffer and timestamp synchronization logic.

[0250] 4. Data encryption and transmission security:

[0251] Use a network analysis tool to check whether the SRTP protocol is used to encrypt audio and video data.

[0252] 5. Audio and video rendering in Unity:

[0253] Check your product project files to see if you use an AudioSource component for audio playback, and a Texture2D or RenderTexture for video rendering.

[0254] Application scenarios of the present invention:

[0255] The audio and video call method of the present invention provides a high-quality, low-latency, and secure solution for real-time interaction in scenarios such as virtual reality (VR) equipment, online education platforms, remote conference software, multiplayer online games, virtual exhibitions and online shopping platforms, and social interactive applications.

[0256] Scenario 1: Real-time teaching in VR virtual classroom

[0257] In the virtual classroom, teachers collect audio and video data through Unity's microphone and camera, and the system automatically adjusts the frame rate and resolution to ensure clarity and stability when the network fluctuates. With P2P transmission, the interaction between teachers and students does not need to be transferred to the server, reducing latency. The audio and video synchronization and buffering mechanism ensures that the voice and actions received by students are consistent, improving the classroom experience. At the same time, SRTP encryption ensures the security of classroom content.

[0258] Scenario 2: Product display in a virtual exhibition

[0259] Product managers can use the audio and video call method of the present invention to show the latest products to global customers in real time. P2P transmission and multi-server redundant configuration are used to ensure that customers have a smooth interactive experience in the virtual exhibition hall. The compression and encryption transmission mechanism ensures the security of data and avoids information leakage while ensuring the quality of audio and video.

[0260] Scenario 3: Real-time communication in multiplayer online games

[0261] In multiplayer online games, players communicate in real time through the invented audio and video call solution. The system dynamically adjusts the audio and video compression parameters according to the network status of each player to ensure smooth communication in a low-bandwidth environment. The P2P transmission mechanism reduces latency, and the audio and video synchronization strategy avoids the sound and picture being out of sync between players.

[0262] Scenario 4: Team collaboration in remote meetings

[0263] Enterprise teams use virtual conference rooms for remote collaboration, and employees use cameras to collect video and microphones to collect voice to achieve real-time communication. Multi-server configuration ensures the stability of communication, the P2P mechanism reduces latency, and the audio and video synchronization and buffering mechanism ensures synchronization of speech and action, improving meeting efficiency.

[0264] It should be emphasized that the embodiments of the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation modes. Any other implementation modes derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.< / renderer> < / audiosource>

Claims

1. A method for audio and video calls, characterized in that: The following steps are involved: Integrate the WebRTC library in the Unity engine; Collect audio and video data; Compressing the collected audio and video data; Transmit compressed audio and video data between clients; The receiving end receives the data and decodes it: The decoded audio data is rendered in real time and then output to the audio device and display device respectively.

2. The method for audio and video communication according to claim 1, characterized in that: The specific method of integrating the WebRTC library in the Unity engine is: Open Package Manager in the Unity project and choose to add the WebRTC package via Git URL; Enter the Git address of the WebRTC library and complete the installation; After installation, the WebRTC API is automatically loaded, allowing developers to call the WebRTC protocol in Unity for audio and video capture, compression, transmission, and rendering.

3. The method for audio and video communication according to claim 1, characterized in that: The specific method for collecting audio and video data is: Audio and video data are collected through microphones and camera devices, and the frame rate, resolution and sampling rate are dynamically adjusted. In the process of audio and video data collection, a buffer mechanism and timestamp synchronization strategy are used.

4. The method for audio and video communication according to claim 1, characterized in that: The method for compressing and encrypting the collected audio and video data is as follows: Opus codec is used to compress audio data, VP8 or H.264 codec is used to compress video data, and the bit rate, resolution, and frame rate in the compression parameters are dynamically adjusted according to network conditions to ensure the best transmission quality. During data transmission, the SRTP secure real-time transport protocol is used to encrypt audio and video data.

5. The method for audio and video communication according to claim 1, characterized in that: The specific method of transmitting the compressed and encrypted audio and video data between clients is as follows: Through WebRTC's P2P peer-to-peer mechanism, compressed audio and video data are transmitted directly between clients; and when the P2P connection cannot be established, multiple TURN servers are used for relaying, and automatically switching to backup servers when the server fails or is overloaded.

6. The method for audio and video calls according to claim 1, characterized in that: The specific steps of receiving and decoding data at the receiving end include: At the receiving end of audio and video communication, the client receives audio and video data transmitted by the remote user through the WebRTC protocol, and then the receiving end decodes the audio and video data through the decoder. The received data is first decoded into the audio and video encapsulation format RTP real-time transport protocol through the network layer, and then handed over to the corresponding codec for decoding; A buffering mechanism is used to handle network jitter, and a timestamp synchronization mechanism is used to keep audio and video data synchronized during playback.

7. The method for audio and video communication according to claim 1, characterized in that: The specific method of performing real-time audio and video rendering on the decoded audio data and outputting them to the audio device and the display device respectively is as follows: In the Unity scene, the decoded audio data is played and the video frames are rendered to achieve smooth display of real-time audio and video calls.

8. A system for audio and video calls, characterized in that: include: WebRTC library integration module, which is used to implement the WebRTC protocol in Unity and handle the transmission, encoding, decoding and connection of audio and video data; Audio and video acquisition module, used to collect audio and video data; Compression and encryption module, used to compress and encrypt the collected audio and video data; The transmission module transmits the compressed audio and video data between clients; Receiving module, the receiving end receives data and decodes it: The rendering module performs real-time audio and video rendering on the decoded audio data and outputs them to the audio device and display device respectively.

9. The audio and video call system according to claim 8, characterized in that: The audio and video acquisition module is also used for: Audio and video data are collected through microphones and camera devices, and the frame rate, resolution and sampling rate are dynamically adjusted. In the process of audio and video data collection, a buffer mechanism and timestamp synchronization strategy are used.

10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for audio and video calls as described in any one of claims 1 to 7 is implemented.

Images