Server-side docking method and device for real-time audio and video communication system

Abstract

The present application discloses a method and device for docking a server of a real-time audio and video communication system, which can be applied to a real-time audio and video communication system agent, wherein the method comprises: in response to a push-pull stream interface call instruction sent by the server, sending a push-pull stream path establishment request to the real-time audio and video communication system; based on the push-pull stream path establishment request, establishing a push-pull stream path between the target media server in the real-time audio and video communication system; based on the push-pull stream path, performing push-pull stream data transmission between the server and the target media server. The present application can simplify the process of docking a push-pull stream module of the server with a push-pull stream of a real-time audio and video communication system, reduce docking and access costs, and reduce the difficulty of docking development.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a server-side docking method, device, computer equipment and storage medium for a real-time audio and video communication system. Background Art

[0002] In the Web Real-Time Communication (WebRTC) system, a common scenario is to pull streams from and push streams to the real-time audio and video system on the server side. In related technologies, streams are usually pushed and pulled through the standard WebRTC access process. However, when using standard WebRTC access, the ICE (Interactive Connectivity Establishment) connection establishment mechanism is cumbersome and complicated. For example, the communicating parties are required to collect Candidates (a data structure that describes the connection endpoints in WebRTC) and exchange Candidates under the coordination of the signaling service. In addition, the cost of standard WebRTC push-pull stream docking is high, and docking development is difficult. Summary of the invention

[0003] The present application provides a server-side docking method, device, computer equipment and storage medium for a real-time audio and video communication system, which can simplify the process of docking the server-side push-pull stream module with the push-pull stream of the real-time audio and video communication system, reduce the docking and access costs, and reduce the difficulty of docking development.

[0004] According to a first aspect of the present application, a server-side docking method of a real-time audio and video communication system is provided, which is applied to an agent of the real-time audio and video communication system, and the method includes:

[0005] In response to the push-pull stream interface call instruction sent by the server, sending a push-pull stream path establishment request to the real-time audio and video communication system;

[0006] Based on the push-pull stream path establishment request, establish a push-pull stream path with a target media server in the real-time audio and video communication system;

[0007] Based on the push-pull stream path, push-pull stream data transmission is performed between the service end and the target media server.

[0008] According to a second aspect of the present application, a server-side docking device of a real-time audio and video communication system is provided, which is applied to an agent of the real-time audio and video communication system, and the device includes:

[0009] A relay endpoint module, configured to send a push-pull stream path establishment request to the real-time audio and video communication system in response to a push-pull stream interface call instruction sent by the server;

[0010] The relay endpoint module is further used to establish a push-pull stream path with a target media server in the real-time audio and video communication system based on the push-pull stream path establishment request;

[0011] The data transmission module is used to perform push-pull stream data transmission between the server and the target media server based on the push-pull stream path.

[0012] According to a third aspect of the present application, a computer device is provided, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the server-side docking method of the real-time audio and video communication system described in the first aspect above.

[0016] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer instructions are used to enable the computer to execute the server-side docking method of the real-time audio and video communication system described in the first aspect.

[0017] According to a fifth aspect of the present application, a computer program product is provided, comprising a computer program, wherein the computer program implements the steps of the method described in the first aspect when executed by a processor.

[0018] According to the technical solution of the present application, the push-pull stream path is established and maintained by the real-time audio and video communication system agent, which simplifies the process of connecting the push-pull stream module of the server with the push-pull stream of the real-time audio and video communication system. In addition, in the present application, the server can complete the push-pull stream by calling the interface in the real-time audio and video communication system agent, shielding the complexity of the system behind it, greatly reducing the development cost of connecting and accessing the real-time audio and video communication system, reducing the difficulty of connection development, and reducing the connection and access cost.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0021] Figure 1It is a flow chart of a server-side docking method of a real-time audio and video communication system provided in an embodiment of the present application;

[0022] Figure 2 A flowchart of another server-side docking method of a real-time audio and video communication system provided in an embodiment of the present application;

[0023] Figure 3 A flowchart of a server-side docking method of another real-time audio and video communication system provided in an embodiment of the present application;

[0024] Figure 4 is an example diagram of an interaction of a docking system provided according to an embodiment of the present application;

[0025] Figure 5 It is a server-side docking device of a real-time audio and video communication system provided in an embodiment of the present application;

[0026] Figure 6 A schematic block diagram of an example computer device that can be used to implement embodiments of the present application is shown. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0028] Figure 1 This is a flow chart of a server-side docking method of a real-time audio and video communication system provided according to an embodiment of the present application. It should be noted that the server-side docking method of the real-time audio and video communication system of the embodiment of the present application can be applied to the real-time audio and video communication system agent. In other words, the server-side docking method of the real-time audio and video communication system of the embodiment of the present application is described from the real-time audio and video communication system agent side.

[0029] like Figure 1 As shown, the server-side docking method may include but is not limited to the following steps.

[0030] In step 101, in response to a push-pull stream interface call instruction sent by a server, a push-pull stream path establishment request is sent to a real-time audio and video communication system.

[0031] Optionally, the server can send a push-pull stream interface call instruction to the real-time audio and video communication system agent. When receiving the push-pull stream interface call instruction sent by the server, the real-time audio and video communication system agent can send a push-pull stream path establishment request to the real-time audio and video communication system.

[0032] For example, the server has a push-pull stream module, and the real-time audio and video communication system agent can provide a push-pull stream interface. The push-pull stream module in the server can start the push-pull stream process by calling the push-pull stream interface provided by the real-time audio and video communication system agent, and at this time, a push-pull stream path establishment request can be sent to the real-time audio and video communication system.

[0033] In step 102, based on the push-pull stream path establishment request, a push-pull stream path is established with a target media server in the real-time audio and video communication system.

[0034] Optionally, when sending a push-pull stream path establishment request to the real-time audio and video communication system, the real-time audio and video communication system agent can establish a push-pull stream path between the server and the real-time audio and video communication system based on the push-pull stream path establishment request.

[0035] In step 103, push-pull stream data transmission is performed between the service end and the target media server based on the push-pull stream path.

[0036] Optionally, based on the push-pull stream path, push-pull stream data transmission between the server and the target media server in the real-time audio and video communication system can be realized. For example, a push stream path is established between the server and the real-time audio and video communication system through the real-time audio and video communication system agent, and the real-time audio and video communication system agent can push the audio and video stream data in the server to the corresponding receiving streaming media server in the real-time audio and video communication system through the push stream path. For another example, a pull stream path is established between the server and the real-time audio and video communication system through the real-time audio and video communication system agent, and the real-time audio and video communication system agent can read the audio and video stream data to be pulled from the target media server in the real-time audio and video communication system through the pull stream path, and send the audio and video stream data to the server, thereby realizing the push-pull stream connection between the server and the real-time audio and video communication system.

[0037] According to the server-side docking method of the real-time audio and video communication system of the embodiment of the present application, the push-pull stream path can be established and maintained through the real-time audio and video communication system agent, simplifying the process of docking the server-side push-pull stream module with the real-time audio and video communication system push-pull stream. In addition, in the present application, the server can complete the push-pull stream by calling the interface in the real-time audio and video communication system agent, shielding the complexity of the system behind it, greatly reducing the development cost of docking and accessing the real-time audio and video communication system, reducing the difficulty of docking development, and reducing the cost of docking and access.

[0038] It should be noted that the push-pull stream process in the embodiment of the present application includes a push stream process and a pull stream process. Optionally, when the real-time audio and video communication system agent receives a push-pull stream interface call instruction sent by the server as a pull stream interface call instruction, it can be considered that the server wants to connect to the real-time audio and video communication system for pull stream. When the real-time audio and video communication system agent receives a push-pull stream interface call instruction sent by the server as a start push stream interface call instruction, it can be considered that the server wants to connect to the real-time audio and video communication system for push stream. The following will describe the process of the server connecting to the real-time audio and video communication system for pull stream and the process of push stream connection in conjunction with the accompanying drawings.

[0039] Figure 2 A flowchart of another method for connecting to a server of a real-time audio and video communication system provided in an embodiment of the present application. It should be noted that the method for connecting to a server in an embodiment of the present application can be described from the perspective of the agent side of the real-time audio and video communication system. Figure 2 As shown, the server-side docking method may include but is not limited to the following steps.

[0040] In step 201, in response to a stream pulling interface calling instruction sent by a server, a stream pulling path establishment request is sent to a real-time audio and video communication system.

[0041] Optionally, the server may send a stream pulling interface call instruction to the real-time audio and video communication system agent. Upon receiving the stream pulling interface call instruction sent by the server, the real-time audio and video communication system agent may send a stream pulling path establishment request to the real-time audio and video communication system.

[0042] For example, the server has a stream pulling module, and the real-time audio and video communication system agent can provide a stream pulling interface. The stream pulling module in the server can start stream pulling by calling the stream pulling interface provided by the real-time audio and video communication system agent, for example, calling the video stream pulling interface to start pulling the video stream, and calling the audio stream interface to start pulling the audio stream. At this time, the real-time audio and video communication system agent can call the path establishment interface of the relay endpoint RelayIO module of the local end, and start sending a stream pulling path establishment request to the real-time audio and video communication system.

[0043] In step 202, based on the pull stream path establishment request, the target stream path is obtained.

[0044] Optionally, the relay endpoint RelayIO module of the real-time audio and video communication system agent can obtain the target flow path from the scheduling module.

[0045] In step 203, a path establishment request is sent to a target media server corresponding to the target stream path, so as to establish a stream pulling path with the target media server.

[0046] Optionally, the relay endpoint RelayIO module of the real-time audio and video communication system agent can establish a path to the RelayIO module of the target media server according to the target stream path, and maintain the path status.

[0047] In step 204, based on the stream pulling path, the audio and video stream data to be pulled corresponding to the server is read from the target media server.

[0048] Optionally, when completing the creation of the above-mentioned streaming path, the target media server needs to create related objects, such as sendTransport (transport module), Consumer (consumer), etc., to form the audio and video stream data to be streamed, wherein the audio and video stream data can be SDP (Session Description Protocol) and RTP (Real-time Transport Protocol) data.

[0049] In this embodiment, the real-time audio and video communication system agent can read the audio and video stream data to be pulled from the target media server, such as the SDP and RTP data of the stream to be pulled.

[0050] In step 205, the audio and video stream data is parsed to obtain audio and video frame data, and the audio and video frame data is sent to the server.

[0051] Optionally, the audio and video stream data of the stream to be pulled can be parsed to obtain the audio and video frame data of the stream, and the audio and video frame data can be sent to the server so that the server can send the audio and video frame data to the corresponding client for audio and video playback.

[0052] In order to reduce the delay, optionally, in some embodiments of the present application, before parsing the audio and video stream data, the audio and video stream data may be subjected to network anti-jitter processing. For example, when the real-time audio and video communication system agent reads the audio and video stream data to be pulled from the target media server, the audio and video stream data may be subjected to network anti-jitter processing, and then the audio and video stream data subjected to network anti-jitter processing may be parsed to obtain the audio and video frame data of the stream, so as to send the audio and video frame data to the server.

[0053] To further avoid delays, optionally, in some embodiments of the present application, when building a streaming path, multiple streaming paths can be created between the streaming module in the server and the real-time audio and video communication system. The real-time audio and video communication system agent detects the delay on each streaming path and selects a streaming path with low latency for data transmission.

[0054] According to the server-side docking method of the real-time audio and video communication system of the embodiment of the present application, the streaming path can be established and maintained through the real-time audio and video communication system agent, which simplifies the process of docking the server-side streaming module with the real-time audio and video communication system. In addition, in the present application, the server can complete the streaming by calling the interface in the real-time audio and video communication system agent, shielding the complexity of the system behind it, greatly reducing the development cost of docking and accessing the real-time audio and video communication system, reducing the difficulty of docking development, and reducing the docking and access costs.

[0055] Figure 3 A flowchart of another method for connecting to a server of a real-time audio and video communication system provided in an embodiment of the present application. It should be noted that the method for connecting to a server in an embodiment of the present application can be described from the perspective of the agent side of the real-time audio and video communication system. Figure 3 As shown, the server-side docking method may include but is not limited to the following steps.

[0056] In step 301, in response to a start streaming interface call instruction sent by a server, the address information of a target streaming media receiving server is obtained.

[0057] Optionally, the server may send a start streaming interface call instruction to the real-time audio and video communication system agent. When the real-time audio and video communication system agent receives the start streaming interface call instruction sent by the server, it may obtain the address information of the target streaming media receiving server, which may include the IP (Internet Protocol Address) address and port number of the target streaming media receiving server.

[0058] For example, the server has a streaming module, and the real-time audio and video communication system agent can provide a start streaming interface. The streaming module in the server can start streaming by calling the start streaming interface provided by the real-time audio and video communication system agent, for example, calling the video stream start streaming interface to start pushing the video stream, and calling the audio stream start streaming interface to start pushing the audio stream. At this time, the real-time audio and video communication system agent can call the open interface getPublishWorker to obtain the address information of the target streaming media server. Among them, the target streaming media server can be understood as a media server in the real-time audio and video communication system that receives the audio and video streams pushed by the server.

[0059] In step 302, a streaming path establishment request is sent to the real-time audio and video communication system according to the address information.

[0060] Optionally, the real-time audio and video communication system agent can call the push stream path establishment interface of the relayIO module of the local end to send a push stream path establishment request to the real-time audio and video communication system to start establishing a push stream path to the real-time audio and video communication system.

[0061] In step 303, based on the streaming path establishment request, a path establishment data packet is sent to the target streaming media receiving server, so that the target streaming media receiving server performs the streaming receiving preparation work upon receiving the path establishment data packet.

[0062] Optionally, when the routing data packet arrives at the target streaming media receiving server in the real-time audio and video communication system, the target streaming media receiving server starts to create streaming receiving related objects and creates media layer streaming receiving related objects, so that the target streaming media receiving server can prepare for streaming receiving.

[0063] In step 304, a notification message sent by the target streaming media receiving server is received, where the notification message is used to notify the real-time audio and video communication system proxy end that the streaming path creation is completed.

[0064] Optionally, after the target streaming media receiving server completes the streaming preparation work, it notifies the real-time audio and video communication system proxy end that the streaming path creation is completed. When the real-time audio and video communication system proxy end receives the notification message sent by the target streaming media receiving server, it can confirm that the streaming path creation is completed.

[0065] In step 305, in response to the streaming interface calling instruction sent by the server, the audio and video stream data is pushed to the target streaming media receiving server based on the streaming path.

[0066] Optionally, the real-time audio and video communication system agent can call back the streaming module in the server to inform the streaming module that the streaming path is created. The server calls the streaming interface to actually push the audio and video data to the real-time audio and video communication system, such as calling the video stream streaming interface to send video data and calling the audio stream streaming interface to send audio data.

[0067] To further avoid delays, optionally, in some embodiments of the present application, when establishing a streaming path, multiple streaming paths can be created between the streaming module in the server and the real-time audio and video communication system. The real-time audio and video communication system agent detects the delay on each streaming path and selects a streaming path with low latency for data transmission.

[0068] According to the server-side docking method of the real-time audio and video communication system of the embodiment of the present application, the push stream path can be established and maintained through the real-time audio and video communication system agent, which simplifies the process of docking the server-side push stream module with the real-time audio and video communication system push stream. In addition, in the present application, the server can complete the push stream by calling the interface in the real-time audio and video communication system agent, shielding the complexity of the system behind it, greatly reducing the development cost of docking and accessing the real-time audio and video communication system, reducing the difficulty of docking development, and reducing the cost of docking and access.

[0069] In order to facilitate those skilled in the art to understand the present application more clearly, Figure 4 Describe in detail.

[0070] like Figure 4 As shown, the docking system can be used to achieve docking between the server and the real-time audio and video communication system. Among them, the docking system may include a server 410, a real-time audio and video communication system agent 420 and a real-time audio and video communication system 430. As an example, the real-time audio and video communication system agent can be integrated on the server, or the real-time audio and video communication system agent and the server are deployed separately, which is not specifically limited in this application.

[0071] Alternatively, if Figure 4 As shown, the server 410 has a push-pull stream module 411, which may include a push stream module and a pull stream module. The real-time audio and video communication system agent 420 may include an SDK (Software Development Kit, software development kit) module 421, a relay endpoint RelayIO module 422, and an anti-jitter module 423. The real-time audio and video communication system 430 includes multiple media servers, Figure 4 Taking a certain media server as an example, the media server may include a control module 431, a RelayIO module 432 and a signaling module 433. The real-time audio and video communication system 430 may include a scheduler 41 and an open interface 42. The real-time audio and video communication system agent 420 is a bridge for the push-pull stream module 411 and the real-time audio and video communication system 430 to connect. The function of the real-time audio and video communication system agent 420 is to simplify the push-pull stream process and reduce the push-pull stream access cost. Figure 4 The system architecture diagram describes the detailed process of pulling and pushing streams.

[0072] The flow pulling logic is as follows:

[0073] 1) The stream pulling module calls the stream pulling interface of the SDK module 421 to start stream pulling. For example, the audio stream pulling interface is called to start pulling the video stream, and the video stream pulling interface is called to start pulling the audio stream.

[0074] 2) The SDK module 421 calls the path building interface of the RelayIO module 422 on the local side to start building a streaming path to the real-time audio and video communication system 430 .

[0075] 3) The RelayIO module 422 of the SDK module 421 obtains the target flow path from the scheduler 41 .

[0076] 4) The RelayIO module 422 of the SDK module 421 builds a path to the RelayIO module 432 of the target media server according to the target stream path, and maintains the path status.

[0077] 5) The control module 431 in the target media server creates related objects (such as sendTransport, Consumer, etc.) and sends the SDP and RTP data of the streaming to the RelayIO module 432.

[0078] 6) The RelayIO module 422 of the SDK module 421 reads the SDP and RTP data of the stream from the peer RelayIO module 432 and outputs it to the SDK module 421 through a callback function.

[0079] 7) The SDK module 421 passes the SDP and RTP data of the stream to the anti-jitter module 423 for network anti-jitter processing.

[0080] 8) The SDK module 421 unpacks the data processed by the anti-jitter module 423 into Frame (frame data), and submits the data to the stream pulling module through a callback function.

[0081] The streaming logic is as follows:

[0082] 1) The streaming module calls the start streaming interface of the SDK module 421 to start streaming. For example, the video stream start streaming interface is called to start pushing the video stream, and the audio stream start streaming interface is called to start pushing the audio stream.

[0083] 2) The SDK module 421 calls the open interface 42 to obtain the address information of the target streaming media receiving server, such as the IP address and port number.

[0084] 3) The SDK module 421 calls the streaming path establishment interface of the local RelayIO module 422 to start establishing a streaming path to the real-time audio and video communication system 430 .

[0085] 4) The RelayIO module 422 of the SDK module 421 establishes a path to the RelayIO module 432 of the target receiving streaming media server and maintains the path status.

[0086] 5) When the routing packet reaches the target streaming media receiving server of the real-time audio and video communication system 430, the control module 431 in the target streaming media receiving server notifies the signaling module 433 to start creating streaming related objects.

[0087] 6) The signaling module 433 notifies the control module 431 to create objects related to media layer streaming.

[0088] 7) After the target streaming media receiving server has completed the streaming preparation work, it notifies the RelayIO module 422 of the SDK module 421 that the streaming path creation is completed.

[0089] 8) The SDK module 421 calls back the streaming module to complete the creation of the streaming path.

[0090] 9) The streaming module calls the streaming interface to actually push the audio and video data to the real-time audio and video communication system 430. For example, the video stream streaming interface is called to send video data, and the audio stream streaming interface is called to send audio data.

[0091] In an embodiment of the present application, when establishing a push-pull stream path, multiple paths are actually created between the push-pull stream module and the real-time audio and video communication system. The RelayIO module detects the delay on each path and selects the path with low delay for data transmission.

[0092] Figure 5 It is a server-side docking device of a real-time audio and video communication system provided in an embodiment of the present application. It should be noted that the server-side docking device of a real-time audio and video communication system can be applied to a real-time audio and video communication system agent. Figure 5 As shown, the server docking device 500 may include: a relay endpoint module 510 and a data transmission module 520 .

[0093] The relay endpoint module 510 is used to send a push-pull stream path establishment request to the real-time audio and video communication system in response to a push-pull stream interface call instruction sent by the server.

[0094] The relay endpoint module 510 is also used to establish a push-pull stream path with a target media server in the real-time audio and video communication system based on a push-pull stream path establishment request.

[0095] The data transmission module 520 is used to perform push-pull stream data transmission between the server and the target media server based on the push-pull stream path.

[0096] In some embodiments of the present application, the push-pull stream interface call instruction may be a pull stream interface call instruction. In the embodiment of the present application, the relay endpoint module 510 is specifically used to: respond to the pull stream interface call instruction sent by the server, and send a pull stream path establishment request to the real-time audio and video communication system.

[0097] In the embodiment of the present application, the relay endpoint module 510 is specifically used to: obtain the target stream path based on the pull stream path establishment request; send a path establishment request to the target media server corresponding to the target stream path to establish the pull stream path with the target media server.

[0098] In an embodiment of the present application, the relay endpoint module 510 is also used to read the audio and video stream data to be pulled corresponding to the server from the target media server based on the stream pulling path; the data transmission module 520 is specifically used to: parse and process the audio and video stream data to obtain audio and video frame data, and send the audio and video frame data to the server.

[0099] In one implementation, Figure 5Based on the above, the server-side docking device may further include an anti-jitter module, wherein the anti-jitter module is used to perform network anti-jitter processing on the audio and video stream data before the data transmission module parses and processes the audio and video stream data.

[0100] In some embodiments of the present application, the push-pull stream interface call instruction may be a start push stream interface call instruction; wherein, in an embodiment of the present application, the relay endpoint module 510 is specifically used to: respond to the start push stream interface call instruction sent by the server, obtain the address information of the target receiving streaming media server; and send a push stream path establishment request to the real-time audio and video communication system based on the address information.

[0101] In an embodiment of the present application, the relay endpoint module 510 is specifically used to: based on a streaming path establishment request, send a path establishment data packet to a target streaming media receiving server, so that the target streaming media receiving server performs streaming preparation work upon receiving the path establishment data packet; receive a notification message sent by the target streaming media receiving server, and the notification message is used to notify the real-time audio and video communication system agent end that the streaming path creation is complete.

[0102] In the embodiment of the present application, the data transmission module 520 is specifically used to: in response to the streaming interface call instruction sent by the server, push the audio and video stream data to the target streaming media server based on the streaming path.

[0103] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0104] According to the server-side docking device of the real-time audio and video communication system of the embodiment of the present application, the push-pull stream path is established and maintained by the real-time audio and video communication system agent, which simplifies the process of docking the server-side push-pull stream module with the real-time audio and video communication system push-pull stream. In addition, in the present application, the server can complete the push-pull stream by calling the interface in the real-time audio and video communication system agent, shielding the complexity of the system behind it, greatly reducing the development cost of docking and accessing the real-time audio and video communication system, reducing the difficulty of docking development, and reducing the cost of docking and access.

[0105] Based on the embodiments of the present application, the present application also provides a computer device, at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the server-side docking method of the real-time audio and video communication system described in any of the aforementioned embodiments.

[0106] Based on the embodiments of the present application, the present application also provides a computer-readable storage medium, wherein computer instructions are used to enable a computer to execute the server-side docking method of the real-time audio and video communication system described in any of the aforementioned embodiments provided according to the embodiments of the present application.

[0107] Figure 6 A schematic block diagram of an example computer device that can be used to implement an embodiment of the present application is shown. The computer device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0108] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0109] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0110] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the server docking method of the real-time audio and video communication system. For example, in some embodiments, the server docking method of the real-time audio and video communication system may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the server docking method of the real-time audio and video communication system described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured as a server-side docking method of a real-time audio and video communication system in any other appropriate manner (eg, by means of firmware).

[0111] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0113] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0115] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0116] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0118] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A server docking method for a real-time audio and video communication system, characterized in that: The method is applied to a real-time audio and video communication system agent end, and the real-time audio and video communication system agent end is used to provide a push-pull stream interface. The method includes: In response to the push-pull stream interface call instruction sent by the server, sending a push-pull stream path establishment request to the real-time audio and video communication system; Based on the push-pull stream path establishment request, establish a push-pull stream path with a target media server in the real-time audio and video communication system; Based on the push-pull stream path, performing push-pull stream data transmission between the server and the target media server; Wherein, based on the push-pull stream path, the push-pull stream data transmission between the server and the target media server includes: Based on the streaming path, read the audio and video stream data to be streamed corresponding to the server from the target media server; Performing network anti-jitter processing on the audio and video stream data; The audio and video stream data after network anti-jitter processing is parsed to obtain audio and video frame data, and the audio and video frame data is sent to the server.

2. The method according to claim 1, characterized in that The push-pull stream interface calling instruction is a pull stream interface calling instruction; the sending of a push-pull stream path establishment request to the real-time audio and video communication system in response to the push-pull stream interface calling instruction sent by the server includes: In response to the stream pulling interface calling instruction sent by the server, a stream pulling path establishment request is sent to the real-time audio and video communication system.

3. The method according to claim 2, characterized in that The step of establishing a push-pull stream path with a target media server in the real-time audio and video communication system based on the push-pull stream path establishment request includes: Based on the pull stream path establishment request, obtain a target stream path; A path establishment request is sent to a target media server corresponding to the target stream path, so as to establish a stream pulling path with the target media server.

4. The method according to claim 1, characterized in that The push-pull stream interface calling instruction is a start stream push interface calling instruction; The step of sending a push-pull stream path establishment request to the real-time audio and video communication system in response to the push-pull stream interface calling instruction sent by the server includes: In response to the start streaming interface call instruction sent by the server, the address information of the target streaming media receiving server is obtained; According to the address information, a streaming path establishment request is sent to the real-time audio and video communication system.

5. The method according to claim 4, characterized in that The step of establishing a push-pull stream path with a target media server in the real-time audio and video communication system based on the push-pull stream path establishment request includes: Based on the streaming path establishment request, a path establishment data packet is sent to the target streaming media receiving server, so that the target streaming media receiving server performs streaming receiving preparation work upon receiving the path establishment data packet; A notification message sent by the target streaming media receiving server is received, wherein the notification message is used to notify the real-time audio and video communication system agent that the streaming path is created.

6. The method according to claim 5, characterized in that The performing push-pull stream data transmission between the server and the target media server based on the push-pull stream path includes: In response to the streaming interface calling instruction sent by the server, the audio and video stream data is pushed to the target streaming media receiving server based on the streaming path.

7. A server-side docking device for a real-time audio and video communication system, characterized in that: The device is applied to a real-time audio and video communication system agent end, and the real-time audio and video communication system agent end is used to provide a push-pull stream interface, and the device includes: A relay endpoint module, configured to send a push-pull stream path establishment request to the real-time audio and video communication system in response to a push-pull stream interface call instruction sent by the server; The relay endpoint module is further used to establish a push-pull stream path with a target media server in the real-time audio and video communication system based on the push-pull stream path establishment request; A data transmission module, used for performing push-pull stream data transmission between the server and the target media server based on the push-pull stream path; Wherein, the data transmission module is specifically used for: Based on the streaming path, read the audio and video stream data to be streamed corresponding to the server from the target media server; Performing network anti-jitter processing on the audio and video stream data; The audio and video stream data after network anti-jitter processing is parsed to obtain audio and video frame data, and the audio and video frame data is sent to the server.

8. The device according to claim 7, characterized in that The push-pull stream interface calling instruction is a pull stream interface calling instruction; the relay endpoint module is specifically used for: In response to the stream pulling interface calling instruction sent by the server, a stream pulling path establishment request is sent to the real-time audio and video communication system.

9. The device according to claim 8, characterized in that The relay endpoint module is specifically used for: Based on the pull stream path establishment request, obtain a target stream path; A path establishment request is sent to a target media server corresponding to the target stream path, so as to establish a stream pulling path with the target media server.

10. The device according to claim 7, characterized in that The push-pull stream interface calling instruction is a start stream push interface calling instruction; the relay endpoint module is specifically used for: In response to the start streaming interface call instruction sent by the server, the address information of the target streaming media receiving server is obtained; According to the address information, a streaming path establishment request is sent to the real-time audio and video communication system.

11. The device according to claim 10, characterized in that The relay endpoint module is specifically used for: Based on the streaming path establishment request, a path establishment data packet is sent to the target streaming media receiving server, so that the target streaming media receiving server performs streaming receiving preparation work upon receiving the path establishment data packet; A notification message sent by the target streaming media receiving server is received, wherein the notification message is used to notify the real-time audio and video communication system agent that the streaming path is created.

12. The device according to claim 11, characterized in that The data transmission module is specifically used for: In response to the streaming interface calling instruction sent by the server, the audio and video stream data is pushed to the target streaming media receiving server based on the streaming path.

13. A computer device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the server-side docking method of the real-time audio and video communication system described in any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: The computer instructions are used to enable the computer to execute the server-side docking method of the real-time audio and video communication system according to any one of claims 1 to 6.

15. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images