A method for realizing audio and video stream direct connection acquisition by a streaming media service integrator SDK

Abstract

The application discloses a method for realizing audio and video stream direct connection acquisition by streaming media service integrated vendor SDK, aiming at the defects of high resource consumption, large transmission delay and high service coupling degree of the existing three-level forwarding architecture, and reconstructing a flattened transmission architecture. The streaming media service integrates a private protocol SDK of the vendor, combines a data sharing center to complete device information query and direct connection permission verification, directly establishes communication with a front-end device, authenticates and issues a stream acquisition instruction, and directly pushes the audio and video original code stream to the streaming media service. After being unpacked, analyzed and converted by the SDK, the audio and video original code stream is distributed to a client. The application realizes complete decoupling of signaling and media stream, and the device access service only carries control type signaling service, thereby omitting the media data transfer forwarding link, effectively reducing server load and transmission delay, improving system stability, concurrent processing capacity and device access capacity, and being suitable for large-scale security monitoring platform deployment.

Description

Technical Field

[0001] This invention relates to the field of streaming media transmission technology, and more specifically to a method for a streaming media service integration vendor SDK to achieve direct connection and acquisition of audio and video streams. Background Technology

[0002] With the rapid development of industries such as the Internet of Things, security monitoring, online education, and interactive entertainment, the scale of streaming media service access is growing exponentially.

[0003] In application scenarios such as smart cities, large-scale park monitoring, or live streaming rooms with tens of thousands of users, video surveillance platforms typically need to connect to a massive number of front-end devices (such as IP cameras, encoders, NVRs, etc.) and provide real-time video preview services to command centers, mobile police terminals, or third-party application systems.

[0004] In the current security monitoring field, because front-end devices from different manufacturers often use proprietary protocols for communication, video surveillance platforms find it difficult to directly interconnect with all devices through standard protocols (such as GB / T28181). Therefore, mainstream solutions typically adopt a layered architecture of "device access service + streaming media service" to achieve stream aggregation and distribution.

[0005] In this architecture, the "Device Access Service" is responsible for maintaining long-term connections with front-end devices, signaling interaction, and private protocol parsing; the "Platform Streaming Media Service" is responsible for the aggregation, forwarding, and distribution of streams.

[0006] The typical preview streaming process is as follows: When a client initiates a video preview request, the request first reaches the platform's streaming media service; the platform's streaming media service sends a streaming instruction to the device access service; the device access service establishes a connection with the target front-end device through a private protocol and sends a playback instruction; the front-end device sends the encoded audio and video data to the device access service; the device access service receives the data packets, unpacks and repackages them, and then forwards them to the platform's streaming media service; the platform's streaming media service then distributes the streaming data to each requesting client for decoding and playback according to the client's needs.

[0007] While the aforementioned existing technical solutions have resolved the compatibility issues of connecting devices from multiple vendors, they have the following significant shortcomings in large-scale practical applications: Multi-level forwarding leads to huge consumption of server resources: audio and video data must flow through the device access service for relay, and each frame of data needs to be received-copied-sent in the server memory. This results in a large amount of CPU, memory and bandwidth resources being occupied, network card throughput increasing exponentially, the number of concurrent paths supported by a single server being limited, and high hardware deployment costs.

[0008] Redundant transmission links and high response latency: Before reaching the client, the data stream needs to undergo two network jumps and memory processing via the device access service and the platform streaming media service, increasing uncontrollable network queuing latency and internal server processing latency. Especially in high-concurrency or weak network environments, the cumulative latency of multi-level forwarding can seriously affect the timeliness of monitoring.

[0009] System scalability and stability are limited: The device access service simultaneously couples the functions of "signaling control" and "media forwarding." When the number of access devices surges, leading to heavy media forwarding pressure, it is difficult to independently scale up the media forwarding processing capacity. Furthermore, if a device access service node fails, all devices it manages will go offline, resulting in a high risk of a single point of failure in the system.

[0010] Limited device access capabilities: Since the device access service undertakes both signaling processing and media forwarding tasks, the device access capabilities of a single service are difficult to improve due to the differences in SDKs from different vendors and performance bottlenecks. This is especially true for multi-channel devices, which often cannot meet the needs of large-scale deployment.

[0011] In summary, the multi-level forwarding and functional coupling problems existing in the current streaming media transmission architecture have become key bottlenecks restricting the performance improvement, cost control and large-scale deployment of video surveillance platforms. There is an urgent need for a streaming media acquisition solution that can effectively eliminate intermediate forwarding links, reduce resource consumption and improve system scalability. Summary of the Invention

[0012] The purpose of this invention is to overcome the shortcomings of existing security video surveillance platform streaming media acquisition solutions, such as multi-level forwarding, high resource consumption, high transmission latency, coupled system architecture, and insufficient device access capabilities. It provides a method for direct audio and video stream acquisition using a streaming media service integration vendor SDK. By eliminating intermediate media stream forwarding links and decoupling signaling processing and media forwarding functions, it effectively reduces server CPU, memory, and bandwidth resource consumption, shortens transmission link latency to improve monitoring real-time performance, optimizes system architecture to enhance scalability and stability, and releases device access service resources to improve the overall device access capability of the platform, thereby meeting the deployment and low-latency usage requirements of large-scale security monitoring scenarios.

[0013] To achieve the above objectives, this invention provides a method for a streaming media service integration vendor SDK to achieve direct acquisition of audio and video streams, comprising: Receive a video preview request sent by a client for a target front-end device, wherein the video preview request carries the device identifier of the target front-end device; The data sharing center is queried based on the device identifier to determine whether the target front-end device meets the preset direct connection access conditions, and if it does, the network configuration information and access permission token of the target front-end device are obtained. The streaming media service calls the vendor's proprietary protocol SDK integrated within it and initiates a direct network connection with the target front-end device based on the network configuration information. After the connection is established, the streaming media service uses the vendor's private protocol SDK to complete the identity authentication with the target front-end device based on the access token, and sends a real-time stream request transmission command to the target front-end device; The receiving port of the streaming media service directly receives the raw audio and video data packets sent by the target front-end device in response to the transmission command, and performs unpacking and protocol conversion processing on the raw audio and video data packets within the streaming media service through the vendor's private protocol SDK, so as to push the processed audio and video stream to the corresponding client according to the session association relationship between the device identifier and the video preview request.

[0014] Furthermore, the integration of the vendor's proprietary protocol SDK includes: the vendor's proprietary protocol SDK being embedded within the process of the streaming media service as a dynamic link library or a static library, or the vendor's proprietary protocol SDK running in an independent proprietary protocol library proxy process, with the streaming media service using inter-process communication to schedule the proxy process to achieve direct streaming.

[0015] Furthermore, the streaming media service internally constructs a unified scheduling framework for vendor-specific protocol SDKs. This unified scheduling framework uses preset interface protocols to shield the underlying implementation differences of different vendor-specific protocol SDKs, enabling dynamic instantiation and resource reclamation of heterogeneous vendor-specific protocol SDKs.

[0016] Furthermore, the method also includes a signaling and media stream separation step, wherein the audio and video streams of the target front-end device are directly obtained by the streaming media service through the vendor's private protocol SDK, and the non-media stream service functions of the target front-end device are still implemented by an independent device access service through private protocol signaling interaction, and the non-media stream service functions include image capture, alarm and PTZ control.

[0017] Furthermore, before querying the data sharing center based on the device identifier, the method further includes: when the device access service starts, reading the preset front-end device configuration information and uploading it to the data sharing center, wherein the configuration information includes the device IP address, port number, username, password and device unique identifier; and when the streaming media service starts, loading the vendor's private protocol SDK and initializing the data interaction interface with the data sharing center.

[0018] Furthermore, the unpacking and protocol conversion processing of the original audio and video data packets includes: the vendor's proprietary protocol SDK performing sorting, verification, and demultiplexing on the received original media data packets to restore audio and video frame data conforming to the standard encoding format, which includes H.264 or H.265.

[0019] Furthermore, before initiating a direct network connection with the target front-end device, the following steps are also included: the streaming media service monitors its own CPU utilization, memory availability, and network card throughput in real time; only when the current resource load of the streaming media service is lower than a preset resource load threshold is the target front-end device determined to meet the direct connection streaming conditions; otherwise, the video preview request is forwarded to the device access service, which obtains the audio and video streams through a relay method.

[0020] Furthermore, the method also includes performing a dual-path adaptive switching step, that is, if a connection timeout, handshake failure or incompatibility of private protocol versions is detected during the process of the streaming media service attempting to establish a direct connection, the current request is automatically rolled back to the device access service, which establishes a connection with the front-end device according to the private protocol and receives the audio and video streams, and then forwards them to the streaming media service for distribution.

[0021] Furthermore, for the target front-end device having multiple sub-channels, the method further includes establishing a single physical long connection with the target front-end device through the vendor's proprietary protocol SDK, and opening multiple logical channels based on the physical long connection to extract streams in parallel.

[0022] Furthermore, when receiving raw audio and video data packets, the method also includes performing a zero-copy memory management step, that is, pre-allocating a continuously managed shared memory area for each instance of the vendor's proprietary protocol SDK, the vendor's proprietary protocol SDK writing the received raw audio and video data packets into the shared memory area, and the streaming media service reading data from the shared memory area through pointer swapping operations to achieve lock-free data synchronization between the protocol parsing logic and the stream distribution logic.

[0023] Furthermore, the method also includes: during transmission, the vendor's proprietary protocol SDK periodically sends heartbeat packets to the target front-end device to maintain session activity; in response to a stop request or timeout event, the streaming media service calls the vendor's proprietary protocol SDK to send a stop command to the target front-end device, disconnect the connection, and release the corresponding network port and memory resources.

[0024] Compared with the prior art, the present invention has the following significant advantages: This invention achieves direct acquisition of audio and video streams through the proprietary protocol SDK of streaming media service integration vendors. Compared with the existing multi-level forwarding architecture, it has the following substantial technical advantages: significantly reducing server resource consumption and hardware costs. This invention, by constructing a two-level direct connection architecture, eliminates the media stream relay link in device access services, effectively avoiding repeated reception of audio and video data across multiple services, redundant memory copying, and network card forwarding. Combined with a zero-copy memory management mechanism, lock-independent data synchronization is achieved through shared memory and pointer swapping, significantly reducing server CPU usage, memory overhead, and network card bandwidth load. Under the same hardware configuration, it can multiply the concurrent carrying capacity and reduce system hardware deployment costs. It also significantly reduces transmission latency and improves monitoring real-time performance. This invention significantly shortens the streaming media transmission link, reducing the accumulated latency caused by multi-level network jumps, intermediate service queuing, and multiple protocol conversions. By enabling direct access to streaming data, it accelerates the initial video loading speed, improves the smoothness of preview images, and accurately meets the stringent low-latency performance requirements of scenarios such as emergency command and real-time monitoring. It also achieves compatibility with multiple vendors' devices and unified scheduling of heterogeneous resources. By constructing a unified scheduling framework for vendor-specific protocol SDKs, this invention shields the differences in underlying interface implementations and calling mechanisms among different vendor SDKs, supporting dynamic loading, instantiation, and resource reclamation of SDK instances on heterogeneous devices. This mechanism enables large-scale integration without requiring customized modifications for a single vendor, significantly improving the system's versatility and compatibility. It also enhances the stability, availability, and self-healing capabilities of the system architecture. This invention employs a dual-path adaptive switching mechanism of direct connection streaming and relay streaming, combined with a resource load threshold judgment strategy. It can automatically and seamlessly fall back to the traditional mode when direct connection failure, resource overload, or protocol version incompatibility is detected. This dynamic defense strategy eliminates the risk of single point of failure, ensuring the continuity and high availability of the video preview service while pursuing ultimate performance. It optimizes data processing efficiency under large-scale concurrency: For multi-channel devices such as NVRs, this invention establishes a single physical long connection and a multi-logical channel parallel streaming mode through the vendor's proprietary protocol SDK, effectively solving the device overload problem caused by too many connections in the traditional architecture. Combined with a lock-free data synchronization mechanism, it further improves the streaming efficiency and data throughput stability of multi-channel devices in large-scale concurrency scenarios. It deeply decouples business functions, increasing the upper limit of platform access scale: This invention achieves complete decoupling of signaling services and media stream processing. Device access services are freed from heavy media forwarding tasks, allowing them to focus on non-media signaling services such as image capture, alarms, and PTZ control. This functional divide-and-conquer strategy reduces the workload and resource consumption of a single service, significantly increasing the maximum number of devices a single service can support, perfectly adapting to large-scale security monitoring deployments. It also improves resource lifecycle management and enhances system maintainability. Session stability is maintained through a heartbeat keep-alive mechanism integrated into the SDK, and connection closure and memory resource release are automatically triggered when a session stops, times out, or an exception occurs. This fine-grained session management mechanism avoids port deadlock and resource leakage problems during long-term operation, greatly simplifying system operation and maintenance and subsequent management. Attached Figure Description

[0025] Figure 1 This is a flowchart of the method for directly acquiring audio and video streams according to the present invention; Figure 2 This is a timing interaction diagram of the direct audio / video stream acquisition method of the present invention. Detailed Implementation

[0026] like Figure 1 As shown in the figure, this embodiment provides a method for a streaming media service integration vendor SDK to achieve direct connection and acquisition of audio and video streams, including: The service receives a video preview request from a client for a target front-end device, wherein the video preview request carries the device identifier of the target front-end device. Specifically, the streaming media service receives and registers the request through an internally maintained session manager, establishing a logical mapping relationship between the client request and the target device to facilitate accurate distribution later. Its beneficial effect is that it realizes standardized access and status traceability for streaming services.

[0027] The data sharing center is queried based on the device identifier to determine whether the target front-end device meets the preset direct connection streaming conditions, and if it does, the network configuration information and access permission token of the target front-end device are obtained. The query process involves comparing the manufacturer protocol version of the target device with the SDK compatibility list currently integrated with the streaming media service. Its beneficial effect is that it realizes dynamic decision-making of the streaming path, ensuring that the direct access mode is enabled when the environment supports it and the configuration is complete, thus guaranteeing the success rate of streaming from the source.

[0028] The streaming media service calls the vendor's proprietary protocol SDK integrated within it and initiates a direct network connection with the target front-end device based on the network configuration information. The direct network connection refers to the streaming media service process directly creating a socket and connecting to the designated port of the device, bypassing the traditional access relay link. Its beneficial effect is to shorten the network transmission path and eliminate the processing delay caused by intermediate node forwarding.

[0029] After the connection is established, the streaming media service completes the identity authentication with the target front-end device through the vendor's private protocol SDK based on the access token, and sends a real-time stream request transmission instruction to the target front-end device. The instruction includes playback control instructions that conform to the private protocol specification and specifies the streaming media service's own receiving address as the media stream push target. Its beneficial effect is that it establishes the streaming media service's direct control over the front-end device and simplifies the signaling interaction chain.

[0030] The receiving port of the streaming media service directly receives the raw audio and video data packets sent by the target front-end device in response to the transmission command. Within the streaming media service, the raw audio and video data packets are unpacked and converted using the vendor's proprietary protocol SDK. Based on the session association between the device identifier and the video preview request, the processed audio and video stream is pushed to the corresponding client. The conversion process includes converting the proprietary encapsulation format to a standard encapsulation format (such as FLV or HLS) that meets the client's playback requirements. Its advantage lies in avoiding cross-network / cross-memory copying of data between different service processes through "on-site processing," significantly reducing the server's CPU and bus bandwidth usage.

[0031] As one implementation method, the integration of the vendor's proprietary protocol SDK in this embodiment includes: the vendor's proprietary protocol SDK being embedded within the streaming media service process as a dynamic link library or a static library, or the vendor's proprietary protocol SDK running in an independent proprietary protocol library proxy process, with the streaming media service using inter-process communication to schedule the proxy process for direct streaming. When using an independent proxy process, independent runtime sandboxes can be allocated to SDKs from different vendors; its advantage lies in achieving fault isolation between the third-party library and the main service process, preventing the main streaming media service from crashing due to an anomaly in a single vendor's SDK, and improving system stability.

[0032] As one implementation method, the streaming media service in this embodiment internally constructs a unified scheduling framework for vendor-specific protocol SDKs. This unified scheduling framework masks the underlying implementation differences of different vendor-specific protocol SDKs through preset interface protocols, enabling dynamic instantiation and resource reclamation of heterogeneous vendor-specific protocol SDKs. The framework defines standard abstract interfaces for initialization, login, streaming, and stopping. Its advantage lies in improving system scalability, allowing for the addition of support for a particular vendor's device without modifying the main logic code; only the new SDK needs to be integrated according to the interface specifications.

[0033] As one implementation method, the method in this embodiment further includes a signaling and media stream separation step. The audio and video streams of the target front-end device are directly obtained by the streaming media service through the vendor's proprietary protocol SDK. The non-media stream service functions of the target front-end device are still implemented by an independent device access service through proprietary protocol signaling interaction. These non-media stream service functions include image capture, alarms, and PTZ control. This step decouples the "heavy media stream" from the "light control stream." Its beneficial effect is to reduce the computational and I / O pressure on the access service, allowing it to focus on high-frequency signaling processing and increasing the overall device access limit of the platform.

[0034] As one implementation method, this embodiment further includes the following steps before querying the data sharing center based on the device identifier: When the device access service starts, it reads preset front-end device configuration information and uploads it to the data sharing center. The configuration information includes the device IP address, port number, username, password, and unique device identifier. The streaming media service loads the vendor's proprietary protocol SDK upon startup and initializes the data interaction interface with the data sharing center. Through this automated configuration synchronization mechanism, the streaming media service can perceive changes in the front-end device in real time. Its advantages include reduced operation and maintenance costs and ensured strong consistency of configuration information in a large-scale deployment environment.

[0035] As one implementation method, the unpacking and protocol conversion processing of raw audio and video data packets in this embodiment includes: the vendor's proprietary protocol SDK performing sorting, verification, and demultiplexing on the received raw media data packets to restore audio and video frame data conforming to the standard encoding format. Specifically, after the streaming media service receives raw audio and video data packets (such as PS streams, ES streams, or proprietary encapsulation format streams) from the front-end device through the vendor's proprietary protocol SDK integrated internally, the SDK automatically performs data verification and reassembly logic; specifically, the vendor's proprietary protocol SDK sorts out-of-order data packets that may be generated due to network jitter by sequence number, and uses a checksum mechanism to verify the integrity and legality of the data packets to eliminate damaged data packets. Subsequently, the SDK performs a demultiplexing operation, extracting the original audio and video bitstream from the proprietary protocol encapsulation and re-encapsulating it into standard audio and video frame data according to the streaming media service's push requirements (for example, converting received proprietary format data into video frames conforming to H.264 or H.265 encoding standards, and audio frames conforming to AAC or G.711 standards). This allows the streaming media service to perform secondary encapsulation and distribution according to the client's playback protocol (such as RTMP, HLS, FLV, etc.). Its beneficial effect is that through refined data reconstruction and standard frame restoration, it ensures the continuity and image quality stability of the output video stream and improves the tolerance to complex network environments.

[0036] As one implementation method, this embodiment further includes the following steps before initiating a direct network connection with the target front-end device: the streaming media service monitors its own CPU utilization, memory availability, and network card throughput in real time; only when the current resource load of the streaming media service is lower than a preset resource load threshold is the target front-end device determined to meet the direct connection streaming conditions; otherwise, the video preview request is forwarded to the device access service, which obtains the audio and video streams through a relay method. Through this adaptive load balancing strategy, the system can dynamically fall back to cascading mode when the pressure is too high; its beneficial effect is to prevent service unavailability caused by overload of the streaming media service, ensuring a basic preview experience under extreme conditions.

[0037] As one implementation method, the method in this embodiment further includes a dual-path adaptive switching step. That is, during the process of the streaming media service attempting to establish a direct connection, if a connection timeout, handshake failure, or incompatibility of the private protocol version is detected, the current request is automatically rolled back to the device access service. The device access service then establishes a connection with the front-end device according to the private protocol, receives the audio and video streams, and forwards them to the streaming media service for distribution. This step provides a logical "safe exit"; its beneficial effect is improved business robustness, ensuring that even when SDK compatibility issues arise, users can still watch videos normally without being aware of the path switching.

[0038] As one implementation method, this embodiment targets a target front-end device with multiple sub-channels. The method further includes establishing a single physical long-lived connection with the target front-end device via the vendor's proprietary protocol SDK, and opening multiple logical channels based on the physical long-lived connection for parallel streaming. For multi-channel devices such as NVRs, multiplexing technology reduces socket consumption; its beneficial effect is a significant reduction in the connection load of the front-end device, avoiding slow device response or disconnections due to excessive connections.

[0039] As one implementation, this embodiment further includes a zero-copy memory management step when receiving raw audio and video data packets. This involves pre-allocating a contiguous shared memory area for each instance of the vendor's proprietary protocol SDK. The vendor's proprietary protocol SDK writes the received raw audio and video data packets into the shared memory area. The streaming media service reads data from the shared memory area through pointer swapping operations, achieving lock-free data synchronization between the protocol parsing logic and the stream distribution logic. Pointer swapping replaces traditional memory mirror copying; its advantages include eliminating memory bus contention under high concurrency, significantly improving the system's data throughput per second, and reducing the hardware cost of processing each stream.

[0040] As one implementation method, the method in this embodiment further includes: during transmission, the vendor's proprietary protocol SDK periodically sends heartbeat packets to the target front-end device to maintain session activity; in response to a stop request or timeout event, the streaming media service calls the vendor's proprietary protocol SDK to send a stop command to the target front-end device, disconnect, and release the corresponding network port and memory resources. This step includes comprehensive resource cleanup logic; its beneficial effect is to prevent device-side connection hangs or server-side memory leaks caused by abnormal disconnections, ensuring the long-term operational stability of the system.

[0041] like Figure 1 , Figure 2As shown, this embodiment provides a method for directly acquiring audio and video streams using a streaming media service integration vendor's SDK. Building upon the aforementioned basic implementation, this embodiment further discloses detailed implementation details based on a signaling and media stream decoupling architecture. This invention, by removing the relay from the traditional three-level cascading mode, breaks down the existing multi-level transmission links of front-end devices, device access services, platform streaming media services, and clients, constructing a two-level direct connection mode of front-end devices, platform streaming media services (integrating vendor's proprietary protocol SDK), and clients. This eliminates the resource redundancy, high transmission latency, and strong architectural coupling defects caused by multi-level forwarding from the underlying transmission path.

[0042] The core components involved in this embodiment have the following specific functional limitations: Front-end equipment includes various IP cameras, NVRs, or encoders that support vendor-specific protocols, used for on-site audio and video signal acquisition and encoding output. Utilizing the high compression ratio and custom transmission characteristics of proprietary protocols, the integrity and transmission quality of the original bitstream can be effectively guaranteed.

[0043] The platform streaming media service serves as the core execution entity of this invention, directly loading and deploying the vendor's proprietary protocol SDK module. Unlike traditional streaming media services that only perform the single function of stream forwarding, the streaming media service in this solution possesses comprehensive capabilities for both signaling interaction control and direct media data reception and processing. It can establish streaming sessions on demand based on business requests, enabling real-time streaming retrieval and immediate response. Through business function integration, the cross-process communication overhead between multiple services is significantly reduced.

[0044] The vendor-specific protocol SDK module fully encapsulates underlying capabilities such as device discovery, identity authentication and login, device permission verification, streaming media control command issuance, media data packet unpacking, and data reconstruction and parsing. It effectively shields the hardware and protocol differences between different brands and models of front-end devices, providing standardized and unified calling interfaces for upper-layer platform streaming media services, and reducing the cost of adapting and modifying devices from multiple vendors.

[0045] As the core hub for cross-service data collaboration, the data sharing center uniformly stores all front-end device configuration data uploaded periodically by the device access service. This data primarily includes key information such as device IP address, access port, login credentials, unique device identifier, protocol type, and direct connection permission configuration. It enables data interoperability and permission linkage between the signaling control layer and the media transport layer, providing a legitimate authorization basis for streaming media services to directly access devices.

[0046] The device access service completes the processing of non-media signaling services such as device registration and keep-alive, parameter configuration distribution, alarm event reporting, PTZ control, and image capture. It does not participate in the reception, forwarding, or copying of audio and video media data, thus achieving physical isolation and logical decoupling between signaling services and media streaming services.

[0047] The specific implementation process is as follows: Step S201: SDK Initialization and Configuration Distribution. During the overall system startup phase of the platform, the device access service prioritizes reading and organizing the configuration information of all front-end devices, and uploads standardized device data to the data sharing center for persistent storage. Simultaneously, after the platform's streaming media service starts, it automatically loads the vendor's proprietary protocol SDK dynamic library files, completing SDK runtime environment initialization, interface registration, and the establishment of the data sharing center interaction channel. This deployment method creates a globally unified view of device resources, providing stable and accurate data support for subsequent differentiated direct connection streaming scheduling.

[0048] Step S202: Receiving Streaming Requests and Determining Direct Connection. The streaming media module continuously listens for and receives real-time video preview streaming requests initiated by clients, parsing the unique identifier of the target device carried in the request message. Based on the parsed device identifier, it actively accesses the data sharing center to query and verify the target device's operating status, protocol compatibility, and direct connection permission configuration, accurately determining whether the current device is suitable for the direct streaming mode. Relying on a centralized data verification and policy determination mechanism, differentiated scheduling processing can be performed on heterogeneous devices, balancing the efficient streaming of new direct-connected devices with the compatibility and adaptation of older non-direct-connected devices, ensuring a smooth transition and operation of the entire network system.

[0049] Step S203: Establish direct connection session and obtain manufacturer information. After verifying that the target front-end device meets the conditions for direct connection streaming, the streaming media module retrieves key configurations such as the manufacturer type, communication parameters, and access tokens corresponding to the device from the data sharing center. It then calls the internally integrated standardized interface of the manufacturer's private protocol SDK to proactively initiate a network connection establishment request to the front-end device, completing the handshake and setup of the underlying communication link. A dynamic resource scheduling mode of establishing connections on demand is adopted, establishing device connections only when there is an actual preview service, avoiding the waste of system resources caused by idle persistent connections.

[0050] Step S204: Response Return and Streaming Trigger. After receiving the connection establishment request, the front-end device completes the protocol handshake and validity verification, and returns interactive response data upwards. After receiving the device's response, the streaming media module completes local session creation and context caching, and synchronously sends a request-response message back to the client. Upon receiving the response, the client formally initiates a continuous streaming command. Based on this, the streaming media service relies on the SDK to encapsulate encrypted authentication messages, and combines access tokens to complete two-way device authentication, building an end-to-end secure and reliable communication link to avoid the risks of unauthorized device access and unauthorized streaming.

[0051] Step S205: Instruction Issuance Based on Private Protocol. After successful identity authentication, the streaming media service issues dedicated playback control instructions to the front-end device via the vendor's private protocol SDK, in accordance with the device's private communication specifications. These instructions include control logic such as initiating real-time stream push, stream format restrictions, and transmission channel binding. By directly issuing device instructions through the streaming media service, the signaling relay and instruction pass-through processes of the device access service in traditional architectures are eliminated, effectively reducing the interaction latency caused by multi-level signaling forwarding.

[0052] Step S206: Direct transmission and encapsulation of audio and video streams. The front-end device parses and responds to the playback instructions issued by the streaming media service, continuously pushing the acquired and encoded raw audio and video data packets directly to the platform's dedicated media receiving port in a proprietary encapsulation format. The streaming media service, relying on its built-in SDK, completes the parsing and processing of the raw bitstream, sequentially performing operations such as data sorting, integrity verification, bitstream demultiplexing, and proprietary protocol unpacking, stripping away proprietary encapsulation fields to restore the raw audio and video frames in standard H.264 and H.265 universal encoding formats. The streaming media service, combined with the client's playback protocol requirements, performs secondary encapsulation and session binding on the standard audio and video frames, accurately pushing them to the corresponding requesting client for decoding and playback. Media data is not relayed through the device access service throughout the entire process, achieving single-level bandwidth utilization of the server's network card, significantly reducing server I / O load and bus pressure caused by multi-service data copying and forwarding.

[0053] This embodiment adopts an architecture design that completely separates signaling services from media stream processing, and deploys them separately according to different service load characteristics. High-bandwidth, high-IO-load audio and video media services are all directly acquired, processed, and distributed by the platform's streaming media service using a proprietary protocol SDK. Low-frequency, lightweight control services such as device capture, anomaly alarms, PTZ rotation, lens adjustment, and device inspection are still centrally processed within the device access service. This differentiated division of labor enables heavy-load media services to be directly connected and light-load control services to be centrally managed, significantly simplifying the business logic and operational load of the device access service, removing the limitation of media forwarding performance bottlenecks on the number of connected devices, and significantly increasing the scale of front-end devices that a single access service can support. This fully meets the usage needs of high-density device deployment scenarios such as ultra-large-scale campuses and comprehensive security systems.

[0054] Meanwhile, this solution features a comprehensive lifecycle session management mechanism. During audio and video transmission, the SDK periodically sends heartbeat keep-alive messages to the front-end device according to a preset cycle to maintain long-term connection stability and prevent silent disconnection. When scenarios such as client-initiated stoppage of preview, session timeout exceptions, or network link interruptions are detected, the streaming media service immediately calls the SDK to issue a stop streaming command, proactively disconnects the device connection, and simultaneously destroys the session context, releases occupied network ports, memory caches, and SDK instance resources. Through this proactive resource reclamation mechanism, resource residue and memory leaks caused by abnormal disconnections and unauthorized exits are effectively avoided, ensuring the platform's stable operation under long-term, high-concurrency conditions.

[0055] The solution provided in this embodiment completely eliminates the media relay link in the device access service, eliminates the problems of duplicate data copying and multi-node bandwidth consumption caused by multi-layer forwarding, significantly reduces the resource consumption of server CPU, memory and network card, and reduces the cost of project hardware expansion and maintenance.

[0056] The solution provided in this embodiment compresses the media transmission link layers, reduces intermediate service processing queuing and cross-node forwarding latency, effectively reduces the overall latency of video preview, shortens the first screen loading time, and is suitable for high-timeliness business scenarios such as emergency command and real-time analysis.

[0057] The solution architecture provided in this embodiment is clearly layered and decoupled. Media services and signaling services can be expanded and maintained independently, eliminating the risk of single-point failure propagation and improving the fault tolerance and horizontal scalability of the entire network system.

[0058] The solution provided in this embodiment focuses on signaling control, with lightweight business logic, significantly improved device access capacity, and an overall system architecture that is more adaptable to the iterative upgrade needs of large-scale, clustered security platforms.

[0059] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for directly acquiring audio and video streams using a streaming media service integration vendor's proprietary protocol SDK, characterized in that... include: Receive a video preview request sent by a client for a target front-end device, wherein the video preview request carries the device identifier of the target front-end device; The data sharing center is queried based on the device identifier to determine whether the target front-end device meets the preset direct connection access conditions, and if it does, the network configuration information and access permission token of the target front-end device are obtained. The streaming media service calls the vendor's proprietary protocol SDK integrated within it and initiates a direct network connection with the target front-end device based on the network configuration information. After the connection is established, the streaming media service uses the vendor's private protocol SDK to complete the identity authentication with the target front-end device based on the access token, and sends a real-time stream request transmission command to the target front-end device; The receiving port of the streaming media service directly receives the raw audio and video data packets sent by the target front-end device in response to the transmission command, and performs unpacking and protocol conversion processing on the raw audio and video data packets through the vendor's private protocol SDK within the streaming media service, so as to push the processed audio and video stream to the corresponding client according to the session association relationship between the device identifier and the video preview request; The method further includes a signaling and media stream separation step, wherein the audio and video streams of the target front-end device are directly obtained by the streaming media service through the vendor's private protocol SDK, and the non-media stream service functions of the target front-end device are still implemented by an independent device access service through private protocol signaling interaction. The non-media stream service functions include image capture, alarm and PTZ control. Before initiating a direct network connection with the target front-end device, the following steps are also included: the streaming media service monitors its own CPU utilization, memory availability, and network card throughput in real time; only when the current resource load of the streaming media service is lower than a preset resource load threshold is the target front-end device determined to meet the direct connection streaming conditions; otherwise, the video preview request is forwarded to the device access service, which obtains the audio and video streams through a relay method. The method further includes performing a dual-path adaptive switching step, that is, if a connection timeout, handshake failure or incompatibility of private protocol version is detected during the process of the streaming media service attempting to establish a direct connection, the current request is automatically rolled back to the device access service, which establishes a connection with the front-end device according to the private protocol and receives the audio and video streams, and then forwards them to the streaming media service for distribution.

2. The method according to claim 1, characterized in that, The integration of the vendor's private protocol SDK includes: the vendor's private protocol SDK being embedded within the process of the streaming media service as a dynamic link library or a static library, or the vendor's private protocol SDK running in an independent private protocol library proxy process, with the streaming media service using inter-process communication to schedule the proxy process to achieve direct streaming.

3. The method according to claim 2, characterized in that, The streaming media service has a unified scheduling framework built on vendor private protocol SDKs. This unified scheduling framework uses a preset interface protocol to shield the underlying implementation differences of different vendor private protocol SDKs, enabling dynamic instantiation and resource reclamation of heterogeneous vendor private protocol SDKs.

4. The method according to claim 1, characterized in that, Before querying the data sharing center based on the device identifier, the method further includes: when the device access service starts, reading the preset front-end device configuration information and uploading it to the data sharing center, wherein the configuration information includes the device IP address, port number, username, password and device unique identifier; when the streaming media service starts, loading the vendor's private protocol SDK and initializing the data interaction interface with the data sharing center.

5. The method according to claim 1, characterized in that, For the target front-end device having multiple sub-channels, the method further includes establishing a single physical long connection with the target front-end device through the vendor's proprietary protocol SDK, and opening multiple logical channels based on the physical long connection to extract streams in parallel.

6. The method according to claim 1, characterized in that, When receiving raw audio and video data packets, the method further includes performing a zero-copy memory management step, that is, pre-allocating a continuously managed shared memory area for each instance of the vendor's proprietary protocol SDK, the vendor's proprietary protocol SDK writing the received raw audio and video data packets into the shared memory area, and the streaming media service reading data from the shared memory area through pointer swapping operations to achieve lock-free data synchronization between the protocol parsing logic and the stream distribution logic.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: during transmission, the vendor's proprietary protocol SDK periodically sends heartbeat packets to the target front-end device to maintain session activity; in response to a stop request or timeout event, the streaming media service calls the vendor's proprietary protocol SDK to send a stop command to the target front-end device, disconnect the connection, and release the corresponding network port and memory resources.

Images