Voice monitoring intercom system front-end interaction method compatible with H.265 and H.264

Abstract

The invention provides a voice monitoring intercom system front-end interaction method compatible with H.265 and H.264, which needs to build a three-layer system framework in advance, and comprises a core component layer, a functional module layer and an external dependency layer. The core component layer comprises a Player component and a Video component, and the functional module layer comprises a WebRTC connection management module, a media stream processing module, a decoding and playing module and a decoding and playing module; the external dependency layer comprises a WebRTC API (Application Program Interface) used for signaling and media stream processing, WebCoodecs used for hard decoding, a WebGL (Web Graphics Library) used for video rendering and an external decoder used for soft decoding; the method comprises the following steps: creating a player instance, and initializing parameters; establishing connection; after receiving the audio or video track, selecting a corresponding decoding mode according to the coding format; creating a video interface; and calling a story method to clean a media track, a decoder, a data channel and connection, and releasing all resources. According to the invention, the native two-way communication is realized, the stability and reliability of the system are improved, the support for the H.265 / H.264 double-coding format is realized, and the transmission efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of building intercom technology, and in particular to a front-end interaction method for a voice monitoring intercom system compatible with H.265 and H.264. Background Technology

[0002] With the rapid development of smart building and IoT technologies, building intercom systems, as a key component of modern security systems, have expanded their functions from simple voice communication to include video calls, remote unlocking, visitor identification, and more. This expansion of functionality places higher demands on the system's real-time audio and video transmission capabilities, network adaptability, and terminal compatibility.

[0003] Traditional building intercom systems typically use dedicated hardware and closed communication protocols for audio communication, such as proprietary audio and video transmission schemes based on RTSP streams, etc. Figure 1 As shown, the traditional private audio and video transmission scheme based on RTSP streams uses a server as a protocol relay for two-way intercom and monitoring architecture. This type of scheme has the following problems: 1. Traditional protocols are not suitable for two-way intercom: The RTSP protocol was originally designed for the control and transmission of unidirectional media streams, mainly for video playback and monitoring scenarios. It adopts a client-server model and lacks native support for two-way audio and video communication. To implement audio return functionality in building intercom systems, additional reverse stream configuration or complex architectural modifications are required, resulting in high implementation difficulty and poor compatibility. 2. High latency and poor interactivity: Due to the RTSP transmission mechanism's reliance on buffering and segmented fetching, its inherent latency is high, generally unsuitable for scenarios with high real-time requirements. Slow response during web front-end playback makes it difficult to meet the real-time voice interaction experience requirements of building intercom systems, severely impacting user experience. 3. Poor browser compatibility: Mainstream browsers (such as Chrome, Firefox, and Safari) do not directly parse RTSP streams, requiring plugins, transcoding services, or intermediate proxy servers for protocol conversion. This not only increases system complexity but also leads to compatibility issues across different browsers and platforms, impacting deployment and user experience.

[0004] Therefore, in order to meet the needs of modern building intercom systems, more advanced and compatible audio and video transmission technologies and protocols are required to improve the system's real-time performance, interactivity, and compatibility. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a front-end interaction method for a voice monitoring intercom system that is compatible with H.265 and H.264, so as to overcome the problems of complex structure, poor real-time performance, poor interactivity and compatibility of existing building intercom systems.

[0006] The problem addressed by this invention is as follows: a front-end interaction method for a voice monitoring and intercom system compatible with H.265 and H.264 requires the pre-construction of a three-layer system framework, comprising a core component layer, a functional module layer, and an external dependency layer. The core component layer includes a Player component and a Video component. The functional module layer includes a WebRTC connection management module, a media stream processing module, a decoding and playback module, and a decoding and playback module. The external dependency layer includes a WebRTC API for signaling and media stream processing, WebCoodecs for hardware decoding, WebGL for video rendering, and an external decoder for software decoding.

[0007] The WebRTC connection management module is responsible for signaling processing, SDP negotiation, and establishing a data channel, which is used to transmit H.265 (HEVC) data streams. The media stream processing module is responsible for receiving remote audio and video media streams and passing them to the Video component, as well as sending local audio streams to the remote component. The decoding and playback module is responsible for receiving the H.265 stream from the Player component and decoding it, and finally passing the decoded data to the Video component. The user interface module provides interface control functions.

[0008] The method includes the following steps:

[0009] Step S1: Create a player instance and initialize parameters;

[0010] Step S2: Trigger the onnegotiationneeded event, generate a local SDP and use the SDP exchange function to exchange SDP Offer / Answer signaling, set the remote description and create an ordered data channel, and automatically complete the signaling negotiation and media format confirmation;

[0011] Step S3: After receiving the audio or video track, select the corresponding decoding method according to the encoding format. If the media format is H.265, decode it and determine whether the current browser supports hardware decoding. If it supports it, use hardware decoding. If it does not support it, use software decoding. Otherwise, if the media format is H.264, the browser will directly perform H.264 decoding.

[0012] Step S4: Create a video interface. Use the video element to render H.264 (AVC) format data and WebGL to render video frames for H.265 format data.

[0013] Step S5: Call the destroy method to clean up the media tracks, decoders, data channels and connections, release all resources, and clean up video elements.

[0014] Furthermore, step S1 specifically includes:

[0015] Step S11: Create a player instance. Create a Player instance based on the necessary configuration parameters provided by the user to prepare for subsequent connection and playback. The necessary configuration parameters include container ID, SDP exchange function, event callback and video configuration. The SDP exchange function is responsible for signaling exchange and obtaining remote media description information.

[0016] Step S12: Parameter initialization, setting default values ​​for unspecified options;

[0017] Step S13: After parameter initialization, create an RTCPeerConnection instance. During the creation process, trigger the onBeforeConnect event callback to notify the application that the connection is about to be established. If the connection creation fails, catch the exception and notify the application through the onError callback function.

[0018] Step S14: If the audio function is enabled, the local microphone permission is obtained through the browser's getUserMedia interface and the audio stream is captured. The captured audio stream will be encapsulated as a MediaStreamTrack object and transmitted through the established RTCPeerConnection connection.

[0019] Furthermore, step S4 specifically involves: creating a video interface by creating video elements and binding media streams, initializing the audio and video control interface, triggering the onVideoLoaded event, rendering H.264 (AVC) format data using the video element, and rendering H.265 format data using WebGL to render video frames.

[0020] Furthermore, the cleanup of video elements in step S6 includes unloading event listeners, removing DOM elements, and releasing WebGL resources.

[0021] Furthermore, the construction process of the three-layer system framework also includes designing the data flow, which includes signaling flow, media flow, control flow, and degradation flow. The signaling flow is used to transmit SDP negotiation data and connection information to the Player component through the WebRTC connection management module. The media flow is used to transmit remote audio and video streams to the media processing module for decoding and then to the Video component. The control flow is used to transmit user interface control commands to the Video component to achieve interaction and response. The degradation flow is used to downgrade to software decoding when hardware decoding is not supported.

[0022] Furthermore, the Player component is responsible for the initialization and management of WebRTC connections, handling remote and local media streams, providing a unified interface and error handling for easy access by upper-layer modules, while the Video component is responsible for managing audio and video rendering and providing interface control.

[0023] The advantages of this invention are:

[0024] 1. The two-way intercom function architecture has been simplified, enabling native two-way communication without the need for additional reverse flow configuration or complex architecture modifications, thus improving system stability and reliability.

[0025] 2. It supports H.265 / H.264 dual encoding formats. Through the codec selection mechanism, it can automatically switch between hardware acceleration and software decoding schemes according to browser compatibility. At the same time, this design significantly reduces video stream transmission latency and greatly improves transmission efficiency compared with traditional HTTP streaming media protocols.

[0026] 3. Simplified cross-platform deployment: Based on Web standard technologies, it supports multiple terminal devices such as PCs, mobile devices, and tablets. One set of code can be adapted to multiple platforms, reducing development and maintenance costs. Attached Figure Description

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0028] Figure 1 This is a timing diagram for two-way intercom interaction in a traditional private audio and video transmission scheme based on RTSP streams.

[0029] Figure 2 This is an execution flowchart of a front-end interaction method for a voice monitoring and intercom system compatible with H.265 and H.264, according to the present invention.

[0030] Figure 3 This is a schematic diagram of the three-layer system framework in an embodiment of the present invention.

[0031] Figure 4 This is a timing diagram of communication between the web front-end and remote devices under the system framework of this invention.

[0032] Figure 5 This is a schematic diagram of the lifecycle of the Player component of this invention. Detailed Implementation

[0033] The overall technical solution in this application is as follows: A front-end solution for a voice monitoring and intercom system compatible with H.265 and H.264 is provided. The system consists of three layers: core components, functional modules, and external dependencies. The system architecture diagram is shown below. Figure 2As shown. The core components include Player and Video components. The functional module layer includes a WebRTC connection management module, a media stream processing module, a decoding and playback module, and a user interface module. External dependencies include the WebRTC API, WebCoodecs, WebGL, and external decoders. The data flow design includes signaling flow: SDP negotiation data and connection information are passed to the Player component through the WebRTC connection management module. Media stream: Remote audio and video streams arrive at the media processing module for decoding and are then passed to the Video component. Control flow: User interface control commands are passed to the Video component to achieve interaction and response. Degradation stream: When hardware decoding is not supported, it degrades to software decoding. Interaction between the web frontend and remote devices is achieved through initialization, connection establishment, media processing, and rendering interaction phases. A two-way intercom functional architecture is built to achieve native two-way communication without the need for additional reverse stream configuration or complex architecture modifications, improving system stability and reliability. It also supports H.265 / H.264 dual encoding formats, significantly reducing video stream transmission latency and improving transmission efficiency.

[0034] To make the present invention more apparent and understandable, a preferred embodiment is described in detail below with reference to the accompanying drawings.

[0035] like Figures 2 to 4 As shown, the present invention provides a front-end interaction method for a voice monitoring and intercom system compatible with H.265 and H.264. This method requires the pre-construction of a three-layer system framework, comprising a core component layer, a functional module layer, and an external dependency layer. The core component layer includes a Player component and a Video component. The functional module layer includes a WebRTC connection management module, a media stream processing module, a decoding and playback module, and a decoding and playback module. The external dependency layer includes a WebRTC API for signaling and media stream processing, WebCoodecs for hardware decoding, WebGL for video rendering, and an external decoder for software decoding.

[0036] The WebRTC connection management module is responsible for signaling processing, SDP negotiation, and establishing a data channel for transmitting H.265 (HEVC) data streams. The media stream processing module is responsible for receiving remote audio and video media streams and passing them to the Video component for playback, as well as sending local audio streams to the remote component. The decoding and playback module is responsible for receiving the H.265 stream from the Player component and decoding it. The decoded data is then passed to the Video component. When receiving an H.265 stream, the module first checks whether the browser supports hardware decoding; if not, it uses software decoding. The user interface module provides interface control functions, such as volume control, full-screen switching, pause playback, and screenshot functions.

[0037] The method includes the following steps:

[0038] Step S1, Initialization Phase: Create a player instance and initialize parameters;

[0039] Step S2, Connection Establishment Phase: Trigger the onnegotiationneeded event, generate local SDP and use SDP exchange functions to exchange SDP Offer / Answer signaling, set remote description and create an ordered data channel, and automatically complete signaling negotiation and media format (such as H.265) confirmation;

[0040] Step S3, Media Processing Stage: After receiving the audio or video track in the ontrack callback, update the MediaStream object and the mediaType state machine. Determine whether to decode based on the encoding format and select the corresponding decoding method. If the media format is H.265, decode it and determine whether the current browser supports hardware decoding. If it does, use hardware decoding; if it does not, use software decoding. Otherwise, if the media format is H.264, bind the mediastream and let the browser directly perform H.264 decoding.

[0041] Step S4, Rendering Interaction Stage: Create video interface, create video elements and bind media stream, initialize audio and video control interface, trigger onVideoLoaded event, render H.264 (AVC) format data using video element, and render video frames using WebGL for H.265 format data;

[0042] Step S5, Resource Release Phase: Call the destroy method to clean up media tracks, decoders, data channels and connections, release all resources, clean up video elements, including unloading event listeners, removing DOM elements, and releasing WebGL resources.

[0043] like Figure 4 As shown in the figure, this diagram mainly describes the interaction process between the device and the front-end (browser) in a specific embodiment, illustrating the various modules and their cooperation mechanisms from signaling exchange and media stream transmission to decoding and playback:

[0044] 1. Signaling Exchange (Offer / Answer)

[0045] The device sends an Offer message to the browser via a signaling protocol;

[0046] After receiving the offer, the browser calls WebRTC.createConnection() to create a PeerConnection object and returns it to the device.

[0047] The browser generates an Answer message and sends it back to the device to complete the signaling exchange.

[0048] Add audio and video tracks;

[0049] The device adds local audio and video tracks (such as media streams captured by the microphone and camera) to the PeerConnection;

[0050] The browser also adds the local audio and video tracks to its PeerConnection;

[0051] 2. Media streaming transmission

[0052] The device begins transmitting encoded media frames to the browser.

[0053] After receiving these encoded frames, the browser decodes them into video frames using WebCodecs.decode() or an external API.

[0054] 3. Decoding and playback

[0055] The decoded video frames are transmitted to <video>Rendering is performed within the element to enable playback of the media stream;

[0056] Control commands and UI updates;

[0057] Users can send control commands (such as pause playback) through the user interface module, and these commands will be passed to the decoding playback module;

[0058] The decoding and playback module adjusts the playback status according to the instructions and updates the user interface to reflect the current status (such as displaying "paused").

[0059] Preferably, step S1 is the initialization phase, specifically including:

[0060] Step S11: Create a player instance. Create a Player instance based on the necessary configuration parameters provided by the user to prepare for subsequent connection and playback. The necessary configuration parameters include container ID, SDP exchange function, event callback and video configuration, etc. The SDP exchange function is a key parameter, responsible for signaling exchange and obtaining remote media description information.

[0061] Step S12, parameter initialization: First, process the configuration parameters passed in by the user, setting default values ​​for unspecified options. For example, by default, audio is disabled and video control (play / pause, full screen) is enabled. These parameters will affect the player's behavior and interface display.

[0062] Step S13: After parameter initialization, create an RTCPeerConnection instance. During the creation process, trigger the onBeforeConnect event callback to notify the application that the connection is about to be established. If the connection creation fails, catch the exception and notify the application through the onError callback function.

[0063] Step S14: If the audio function is enabled, the local microphone permission is obtained through the browser's getUserMedia interface and the audio stream is captured. The captured audio stream will be encapsulated as a MediaStreamTrack object and transmitted through the established RTCPeerConnection connection.

[0064] Preferably, step S4 specifically involves: creating a video interface by creating video elements and binding media streams, initializing the audio and video control interface, triggering the onVideoLoaded event, rendering H.264 (AVC) format data using the video element, and rendering H.265 format data using WebGL to render video frames.

[0065] Preferably, the cleanup of video elements in step S6 includes unloading event listeners, removing DOM elements, and releasing WebGL resources.

[0066] Preferably, the construction process of the three-layer system framework also includes designing the data flow, which includes signaling flow, media flow, control flow, and degradation flow. The signaling flow is used to transmit SDP negotiation data and connection information to the Player component through the WebRTC connection management module. The media flow is used to transmit remote audio and video streams to the media processing module for decoding and then to the Video component. The control flow is used to transmit user interface control commands to the Video component to achieve interaction and response. The degradation flow is used to downgrade to software decoding when hardware decoding is not supported.

[0067] Preferably, the Player component is responsible for the initialization and management of the WebRTC connection, handles remote and local media streams, provides a unified interface and performs error handling, facilitating calls from upper-layer modules. The lifecycle of the Player component is as follows: Figure 5 As shown. The Video component is responsible for managing audio and video rendering and providing interface controls, such as pause, play, and full-screen switching.

[0068] In summary, since the RTSP protocol is primarily used for unidirectional video streaming, traditional building intercom systems typically require a "reverse stream" to achieve two-way communication. This means the remote device receives the audio stream from the front end, leading to high latency and complexity. It also necessitates plugins or specific platform support, making cross-platform deployment difficult. This application, however, achieves native two-way audio and video communication based on WebRTC technology. The system automatically negotiates a two-way transmission channel when establishing a connection, eliminating the need for additional reverse stream configuration or complex relay architectures as in traditional solutions. Using standard browser APIs (such as RTCPeerConnection and getUserMedia), local audio can be automatically captured and sent to the remote end, while simultaneously receiving and playing the other party's audio and video streams, simplifying the implementation process. Furthermore, it supports both H.265 and H.264 encoding formats. Through a codec selection mechanism, it automatically switches between hardware acceleration and software decoding schemes based on browser compatibility. This design significantly reduces video stream transmission latency, outperforming traditional HTTP streaming media protocols. Furthermore, this application is entirely based on browser standard technology, without relying on a specific operating system or installing dedicated client software. The system runs in a browser environment and can be naturally adapted to various terminal devices such as PCs, mobile devices, and tablets, achieving "develop once, run on multiple platforms".

[0069] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.< / video>

Claims

1. A front-end interaction method for a voice monitoring and intercom system compatible with H.265 and H.264, characterized in that: A three-tier system framework needs to be pre-built, comprising a core component layer, a functional module layer, and an external dependency layer. The core component layer includes a Player component and a Video component. The functional module layer includes a WebRTC connection management module, a media stream processing module, a decoding and playback module, and a decoding and playback module. The external dependency layer includes a WebRTC API for signaling and media stream processing, WebCoodecs for hardware decoding, WebGL for video rendering, and an external decoder for software decoding. The WebRTC connection management module is responsible for signaling processing, SDP negotiation, and establishing a data channel, which is used to transmit H.265 data streams. The media stream processing module is responsible for receiving remote audio and video media streams and passing them to the Video component, as well as sending local audio streams to the remote component. The decoding and playback module is responsible for receiving the H.265 stream from the Player component and decoding it, and finally passing the decoded data to the Video component. The user interface module provides interface control functions. The method includes the following steps: Step S1: Create a player instance and initialize parameters; Step S2: Trigger the onnegotiationneeded event, generate a local SDP and use the SDP switching function to exchange SDPoffer / Answer signaling, set the remote description and create an ordered data channel, and automatically complete the signaling negotiation and media format confirmation; Step S3: After receiving the audio or video track, select the corresponding decoding method according to the encoding format. If the media format is H.265, decode it and determine whether the current browser supports hardware decoding. If it supports it, use hardware decoding. If it does not support it, use software decoding. Otherwise, if the media format is H.264, the browser will directly perform H.264 decoding. Step S4: Create a video interface, using the video element to render H.264 format data and WebGL to render video frames for H.265 format data; Step S5: Call the destroy method to clean up the media tracks, decoders, data channels and connections, release all resources, and clean up video elements.

2. The front-end interaction method of a voice monitoring and intercom system compatible with H.265 and H.264 as described in claim 1, characterized in that: Step S1 specifically includes: Step S11: Create a player instance. Create a Player instance based on the necessary configuration parameters provided by the user to prepare for subsequent connection and playback. The necessary configuration parameters include container ID, SDP exchange function, event callback and video configuration. The SDP exchange function is responsible for signaling exchange and obtaining remote media description information. Step S12: Parameter initialization, setting default values ​​for unspecified options; Step S13: After parameter initialization, create an RTCPeerConnection instance. During the creation process, trigger the onBeforeConnect event callback to notify the application that the connection is about to be established. If the connection creation fails, catch the exception and notify the application through the onError callback function. Step S14: If the audio function is enabled, the local microphone permission is obtained through the browser's getUserMedia interface and the audio stream is captured. The captured audio stream will be encapsulated as a MediaStreamTrack object and transmitted through the established RTCPeerConnection connection.

3. The front-end interaction method of a voice monitoring and intercom system compatible with H.265 and H.264 as described in claim 1, characterized in that: Step S4 specifically involves: creating a video interface by creating video elements and binding media streams, initializing the audio and video control interface, triggering the onVideoLoaded event, rendering H.264 format data using the video element, and rendering H.265 format data using WebGL to render video frames.

4. The front-end interaction method of a voice monitoring and intercom system compatible with H.265 and H.264 as described in claim 1, characterized in that: The cleanup of video elements in step S6 includes unloading event listeners, removing DOM elements, and releasing WebGL resources.

5. The front-end interaction method of a voice monitoring and intercom system compatible with H.265 and H.264 as described in claim 1, characterized in that: The construction process of the three-layer system framework also includes designing the data flow, which includes signaling flow, media flow, control flow, and degradation flow. The signaling flow is used to transmit SDP negotiation data and connection information to the Player component through the WebRTC connection management module. The media flow is used to transmit remote audio and video streams to the media processing module for decoding and then to the Video component. The control flow is used to transmit user interface control commands to the Video component to achieve interaction and response. The degradation flow is used to downgrade to software decoding when hardware decoding is not supported.

6. The front-end interaction method of a voice monitoring and intercom system compatible with H.265 and H.264 as described in claim 1, characterized in that: The Player component is responsible for the initialization and management of WebRTC connections, handling remote and local media streams, providing a unified interface and error handling for easy access by upper-layer modules. The Video component is responsible for managing audio and video rendering and providing interface control.

Images