Cross-platform audio and video multi-protocol conversion transmission method and system

Abstract

The invention discloses a cross-platform audio and video multi-protocol conversion transmission method and system, and relates to the technical field of audio and video transmission. The method comprises an audio and video processing module construction process, a signaling negotiation process, a media transmission process and a signaling negotiation exit process. The audio and video processing module comprises a signaling layer and a media layer; the signaling layer supports various protocols to access in the form of signaling plug-ins, receives and processes various signaling information, and then a signaling processing module schedules the processing of the signaling information to realize the signaling conversion among different protocols; the media layer supports various streaming media transmission protocols to access in the form of media stream plug-ins, receives and processes various audio and video data, and then forwards the audio and video data; the audio and video processing module can be deployed on a service layer and / or an access layer. By introducing the audio and video processing module, access and conversion of various audio and video protocols are supported, the platform opening capacity is improved, and meanwhile the audio and video processing capacity of an application is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of audio and video transmission, and in particular to a cross-platform audio and video multi-protocol conversion transmission method and system. Background Art

[0002] With the rapid development of IoT technology, smart devices are becoming more and more popular, and the demand for audio and video transmission between devices, apps, and web applications is becoming more and more urgent.

[0003] Most existing systems only support single-protocol access for audio and video transmission, without conversion between protocols. This can cause several problems for the system: (1) It is not conducive to device access, as third-party devices may use other protocols or legacy system devices may use private protocols due to resource constraints, which may result in inability to access the system. (2) Web applications cannot or find it difficult to implement point-to-point direct audio and video data transmission. For offline applications, once the server is offline or down, audio and video call functions cannot be performed. (3) Application scenarios are not rich enough. Single-protocol access application scenarios are relatively simple. For example, it is impossible or difficult to simultaneously meet the application requirements of multiplexed monitoring and group call intercom. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a cross-platform audio and video multi-protocol conversion and transmission method and system, which supports the access and conversion of various audio and video protocols by introducing an audio and video processing module, increases the openness of the platform, and enhances the ability of applications to process audio and video.

[0005] In a first aspect, the present invention provides a cross-platform audio and video multi-protocol conversion and transmission method, comprising:

[0006] The process of building the audio and video processing module: building the audio and video processing module, including the signaling layer and the media layer; the signaling layer supports various protocols to access in the form of signaling plug-ins, receives and processes various signaling information, and then the signaling processing module schedules the processing of signaling information to realize the conversion of signaling between different protocols; the media layer supports various streaming media transmission protocols to access in the form of media stream plug-ins, receives and processes various audio and video data, and then forwards them; the audio and video processing module can be deployed in the service layer and / or access layer;

[0007] Signaling negotiation process: When the signaling plug-in receives a signaling request, if it is an initiation command, the signaling processing module is requested to generate a SessionID to identify the entire communication process, and the initial signaling negotiation is completed based on the matching signaling plug-in; if it is not an initiation command, the corresponding information is recorded based on the SessionID identifier, and the signaling plug-in converts it into the corresponding protocol and sends it to the recipient;

[0008] Media transmission process: When the signaling plug-in receives the media stream information to be transmitted, the signaling processing module requests the media layer to create a media stream. The media layer creates the media stream through the corresponding media stream plug-in and updates the transmission relationship. The audio and video data is parsed based on the matching media stream plug-in and then re-packaged and sent to the media stream receiver;

[0009] Signaling exit negotiation process: When the signaling plug-in receives an exit request, the signaling processing module unbinds the transmission relationship between the corresponding media streams according to the SessionID and updates it. If a valid transmission graph can no longer be formed between the media streams, the entire communication is terminated.

[0010] Furthermore, the media layer of the audio and video processing module is transmitted after the audio and video data are transferred by the selective forwarding unit or the multi-point control unit re-encodes the mixed audio and video data.

[0011] Furthermore, the signaling negotiation process specifically includes the following steps:

[0012] Step A1, load the plug-in and initialize the configuration information;

[0013] Step A2: When the signaling plug-in receives a signaling request, if it is an initiation command, the signaling processing module is requested to generate a SessionID to identify the entire communication process, and then the sender information, receiver information and media stream information are parsed, and the SessionID is carried to the signaling processing module; if it is not an initiation command, the corresponding information is recorded based on the SessionID identifier, and the signaling plug-in converts it into the corresponding protocol and sends it to the receiver;

[0014] Step A3: The signaling processing module records the relevant information of the entire communication process based on the SessionID, finds the receiver and the matching signaling plug-in according to the SessionID, inputs the signaling command, sender information and media stream information, and converts the information into the corresponding protocol by the signaling plug-in and sends it to the receiver;

[0015] Step A4: After receiving the signaling, the receiving end replies with the corresponding signaling information and repeats steps A2 and A3 to complete the initial signaling negotiation.

[0016] Furthermore, the media transmission process specifically includes the following steps:

[0017] Step B1: When the signaling plug-in receives the media stream information to be transmitted, it carries the SessionID and the media stream information and transmits it to the signaling processing module. The signaling processing module integrates the transmission relationship between the media stream IDs according to the SessionID and requests the media layer to create a media stream.

[0018] Step B2: The media layer creates media stream information according to the media stream plug-in corresponding to the media stream information, which is used to receive and send media data, and updates the transmission relationship between the media stream IDs to the selective forwarding unit or the multipoint control unit based on the SessionID;

[0019] Step B3: After successfully creating the media stream, the signaling processing module returns a message to the corresponding signaling plug-in for signaling interaction;

[0020] Step B4: After receiving the media stream data, the media stream plug-in parses the original audio and video data packet and transmits it together with the media stream ID to the selective forwarding unit or the multipoint control unit. After determining the media stream receiver, the corresponding media stream plug-in is called to re-pack the original audio and video data packet and send it to the media stream receiver.

[0021] Furthermore, the signaling exit negotiation process specifically includes the following steps:

[0022] Step C1: When the signaling plug-in receives a logout request, it parses the sender information and carries the SessionID to the signaling processing module;

[0023] Step C2: The signaling processing module finds the communication-related information according to the SessionID, obtains the corresponding media stream ID, and unbinds the transmission relationship between the related media streams. If it is detected that a valid transmission graph can no longer be formed between the media streams, the entire communication is terminated and an exit signaling is sent to all communication objects. Otherwise, the transmission relationship between the media stream IDs of the media layer is updated.

[0024] In a second aspect, the present invention provides a cross-platform audio and video multi-protocol conversion and transmission system, comprising:

[0025] The audio and video processing module building module is used to build the audio and video processing module, including the signaling layer and the media layer; the signaling layer supports various protocols to access in the form of signaling plug-ins, receive and process various signaling information, and then the signaling processing module schedules the processing of signaling information to realize the conversion of signaling between different protocols; the media layer supports various streaming media transmission protocols to access in the form of media stream plug-ins, receive and process various audio and video data, and then forward them; the audio and video processing module can be deployed in the service layer and / or access layer;

[0026] The signaling negotiation module is used to request the signaling processing module to generate a SessionID to identify the entire communication process when the signaling plug-in receives a signaling request, if it is an initiation command, and complete the initial signaling negotiation based on the matching signaling plug-in; if it is not an initiation command, the corresponding information is recorded based on the SessionID identifier, and the signaling plug-in converts it into the corresponding protocol and sends it to the recipient;

[0027] The media transmission module is used for when the signaling plug-in receives the media stream information to be transmitted, the signaling processing module requests the media layer to create a media stream, the media layer creates the media stream through the corresponding media stream plug-in and updates the transmission relationship, parses the audio and video data based on the matching media stream plug-in, and then re-packages and sends it to the media stream receiver;

[0028] The signaling exit negotiation module is used for when the signaling plug-in receives an exit request, the signaling processing module unbinds the transmission relationship between the corresponding media streams according to the SessionID and updates it. If a valid transmission graph can no longer be formed between the media streams, the entire communication is terminated.

[0029] Furthermore, the media layer of the audio and video processing module also includes a selective forwarding unit or a multi-point control unit, the selective forwarding unit is used to forward the audio and video data, and the multi-point control unit is used to re-encode the mixed audio and video data before transmission.

[0030] Furthermore, the signaling negotiation module is specifically used to perform the following steps:

[0031] Step A1, load the plug-in and initialize the configuration information;

[0032] Step A2: When the signaling plug-in receives a signaling request, if it is an initiation command, the signaling processing module is requested to generate a SessionID to identify the entire communication process, and then the sender information, receiver information and media stream information are parsed, and the SessionID is carried to the signaling processing module; if it is not an initiation command, the corresponding information is recorded based on the SessionID identifier, and the signaling plug-in converts it into the corresponding protocol and sends it to the receiver;

[0033] Step A3: The signaling processing module records the relevant information of the entire communication process based on the SessionID, finds the receiver and the matching signaling plug-in according to the SessionID, inputs the signaling command, sender information and media stream information, and converts the information into the corresponding protocol by the signaling plug-in and sends it to the receiver;

[0034] Step A4: After receiving the signaling, the receiving end replies with the corresponding signaling information and repeats steps A2 and A3 to complete the initial signaling negotiation.

[0035] Furthermore, the media transmission module is specifically used to perform the following steps:

[0036] Step B1: When the signaling plug-in receives the media stream information to be transmitted, it carries the SessionID and the media stream information and transmits it to the signaling processing module. The signaling processing module integrates the transmission relationship between the media stream IDs according to the SessionID and requests the media layer to create a media stream.

[0037] Step B2: The media layer creates media stream information according to the media stream plug-in corresponding to the media stream information, which is used to receive and send media data, and updates the transmission relationship between the media stream IDs to the selective forwarding unit or the multipoint control unit based on the SessionID;

[0038] Step B3: After successfully creating the media stream, the signaling processing module returns a message to the corresponding signaling plug-in for signaling interaction;

[0039] Step B4: After receiving the media stream data, the media stream plug-in parses the original audio and video data packet and transmits it together with the media stream ID to the selective forwarding unit or the multipoint control unit. After determining the media stream receiver, the corresponding media stream plug-in is called to re-pack the original audio and video data packet and send it to the media stream receiver.

[0040] Furthermore, the signaling exit negotiation module is specifically used to perform the following steps:

[0041] Step C1: When the signaling plug-in receives a logout request, it parses the sender information and carries the SessionID to the signaling processing module;

[0042] Step C2: The signaling processing module finds the communication-related information according to the SessionID, obtains the corresponding media stream ID, and unbinds the transmission relationship between the related media streams. If it is detected that a valid transmission graph can no longer be formed between the media streams, the entire communication is terminated and an exit signaling is sent to all communication objects. Otherwise, the transmission relationship between the media stream IDs of the media layer is updated.

[0043] The technical solution provided in the embodiments of the present invention has at least the following technical effects:

[0044] By building audio and video processing modules, it supports the access and conversion of various audio and video protocols, and can support the access of different manufacturers, new and old equipment, etc., which increases the openness of the platform. At the same time, it supports deployment at the service layer and / or access layer. As long as one protocol is implemented, multiplexed monitoring and group calling functions between different devices can be realized, which greatly simplifies application development. It can also add functions such as direct point-to-point audio and video calls for web applications located in the access layer, enhancing the application's ability to process audio and video.

[0045] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0047] Figure 1 Schematic diagram of the framework of the audio and video processing module according to an embodiment of the present invention;

[0048] Figure 2 This is a system architecture diagram of an embodiment of the present invention;

[0049] Figure 3 is an overall flow chart of the method in Embodiment 1 of the present invention;

[0050] Figure 4 This is a flow chart of the signaling negotiation process in the first embodiment of the present invention;

[0051] Figure 5 This is a flow chart of a media transmission process in Embodiment 1 of the present invention;

[0052] Figure 6 This is a flowchart of the signaling exit negotiation process in the first embodiment of the present invention;

[0053] Figure 7 It is a schematic diagram of the structure of the system in the second embodiment of the present invention. DETAILED DESCRIPTION

[0054] The embodiment of the present invention provides a cross-platform audio and video multi-protocol conversion and transmission method and system, and supports the access and conversion of various audio and video protocols by introducing an audio and video processing module, thereby increasing the openness of the platform and enhancing the ability of applications to process audio and video.

[0055] The technical solution in the embodiment of the present invention has the following general idea:

[0056] A solution for audio and video processing module is proposed, such as Figure 1 As shown in the figure, the audio and video processing module consists of a signaling layer and a media layer. The signaling layer supports various protocols to access in the form of plug-ins, receive and process their respective signaling information, and the signaling processing module schedules the processing of signaling information to realize the conversion of signals between different protocols. The media layer supports various streaming media transmission protocols to access in the form of plug-ins, receive and process their respective audio and video data, and the selective forwarding unit (SFU) forwards the audio and video data or the multipoint control unit (MCU) re-encodes the mixed audio and video data to realize the transmission of data between different streaming media protocols. The system architecture diagram is shown in the figure. Figure 2 As shown in the figure, the service layer can horizontally expand and deploy audio and video processing modules according to business needs and performance. The access layer is composed of devices, web applications, and apps. Since the audio and video processing module has cross-platform characteristics, it can also be installed in the access layer application. This allows the web application to have the ability to make direct point-to-point calls. At the same time, when MCU processing is required, it can also be directly deployed to sink the calculation to the edge side, saving computing resources and simplifying access processing.

[0057] Embodiment 1

[0058] This embodiment provides a cross-platform audio and video multi-protocol conversion transmission method, such as Figure 3 As shown, including:

[0059] S1. Audio and video processing module construction process: Construct audio and video processing modules, including signaling layer and media layer; the signaling layer supports various protocols to be accessed in the form of signaling plug-ins (such as SIP plug-ins, RTSP plug-ins, WebRTC plug-ins, etc., and private protocol plug-ins can also be accessed), receive and process various signaling information, and then the signaling processing module schedules the processing of signaling information to realize the conversion of signaling between different protocols; the media layer supports various streaming media transmission protocols to be accessed in the form of media stream plug-ins (such as RTP / RTCP plug-ins, RTMP plug-ins, HLS plug-ins, SRT plug-ins, etc.), receive and process various audio and video data, and then forward them through selective forwarding units or multi-point control units; audio and video processing modules can be deployed at the service layer and / or access layer. The audio and video processing module can be deployed in the cloud or on the edge side, and can be deployed in a distributed manner to cope with the pressure of massive data processing.

[0060] S2. Signaling negotiation process: When the signaling plug-in receives a signaling request, if it is an initiation command, the signaling processing module is requested to generate a SessionID to identify the entire communication process, and complete the initial signaling negotiation based on the matching signaling plug-in; if it is not an initiation command, the corresponding information is recorded based on the SessionID identifier, and the signaling plug-in converts it into the corresponding protocol and sends it to the recipient.

[0061] S3, media transmission process: When the signaling plug-in receives the media stream information that needs to be transmitted, the signaling processing module requests the media layer to create a media stream. The media layer creates the media stream through the corresponding media stream plug-in and updates the transmission relationship (a SessionID identifier generally corresponds to an audio stream and a video stream). The audio and video data is parsed based on the matching media stream plug-in and then re-packaged and sent to the media stream receiver.

[0062] S4, signaling exit negotiation process: When the signaling plug-in receives the exit request, the signaling processing module unbinds the transmission relationship between the corresponding media streams according to the SessionID and updates it. If a valid transmission graph can no longer be formed between the media streams, the entire communication is terminated.

[0063] In a specific embodiment, the implementation process is as follows:

[0064] The first part is the signaling negotiation process, the steps are as follows:

[0065] A1. When the service starts, the plug-in information in the database is scanned, the plug-in library is loaded, and the plug-in Init function is called to pass in the initialization configuration information to complete the plug-in loading. The plug-ins include signaling plug-ins and media stream plug-ins.

[0066] A2. When the signaling plug-in receives a signaling request, if it is an initiation command, it will request the signaling processing module to generate a SessionID to identify the entire communication process, and then parse the sender information, receiver information and media stream information, and carry the SessionID to the signaling processing module; if it is not an initiation command, it will directly enter A3.

[0067] A3. The signaling processing module will record the relevant information of the entire communication process based on the SessionID, find the receiver and the matching signaling plug-in according to the SessionID, pass in the signaling command, sender information and media stream information, and convert it into the corresponding protocol by the signaling plug-in and send it to the receiver.

[0068] A4. After receiving the signaling, the receiver replies with the corresponding signaling information and repeats steps A2 and A3 to complete the initial signaling negotiation.

[0069] The above steps complete the initial signaling negotiation and match the audio and video transmission. The process is as follows: Figure 4 shown.

[0070] The second part is the media transmission process, the steps are as follows:

[0071] B1. When the signaling plug-in receives the media stream information and confirms that it needs to be transmitted, it will carry the SessionID and media stream information to the signaling processing module. The signaling processing module finds the communication-related information according to the SessionID, integrates the transmission relationship between the media stream IDs, and requests the media layer to create a media stream.

[0072] B2. The media layer finds the corresponding media stream plug-in based on the media stream information, creates media stream information for receiving and sending media data, and updates the transmission relationship between media stream IDs to the SFU / MCU module based on the SessionID.

[0073] B3. After the signaling processing module successfully creates the media stream, it records it and returns it to the corresponding signaling plug-in for signaling interaction.

[0074] B4. After receiving the audio and video data, the media stream plug-in parses the original audio and video data packets, determines the media stream recipient to be sent through the SFU / MCU module, finds the corresponding media stream plug-in, and the media stream plug-in encapsulates the original audio and video data packets according to the corresponding protocol and sends them to the media stream recipient.

[0075] Through the above steps, media transmission is completed to realize audio and video data communication. The process is as follows: Figure 5 shown.

[0076] The third part is the signaling exit negotiation phase, the steps are as follows:

[0077] C1. When the signaling plug-in receives a logout request, it parses the sender information and passes it to the signaling processing module with the SessionID.

[0078] C2. The signaling processing module finds the communication-related information according to the SessionID, obtains the corresponding media stream ID, and unbinds the transmission relationship between the related media streams. If it is detected that a valid transmission graph can no longer be formed between the media streams, the entire communication is terminated and an exit signaling is sent to all communication objects. Otherwise, the transmission relationship between the media stream IDs of the media layer is updated.

[0079] The signaling exit negotiation is completed through the above steps. The process is as follows: Figure 6 shown.

[0080] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0081] (1) Greatly increased the openness of the platform: Since it supports the access and conversion of various audio and video protocols, it can support the access of different manufacturers, new and old equipment, etc., increasing the openness of the platform.

[0082] (2) Enhanced point-to-point audio and video transmission capabilities: By installing an audio and video processing module, you can add functions such as direct point-to-point audio and video calls for web and other applications, thereby enhancing the application's ability to process audio and video.

[0083] (3) Richer application scenarios and wider adaptability: Through the audio and video processing module, as long as one protocol is implemented, multiplexing monitoring and group calling functions between different devices can be realized. For example, web applications can realize RTSP stream viewing, SIP intercom, and DVP private protocol monitoring and calling functions through WebRTC, which greatly simplifies application development.

[0084] (4) Stronger scalability: It can not only expand horizontally at the cloud service level to meet multi-protocol communication needs, but also sink to the edge side and interconnect with the cloud to meet the cloud audio and video needs of edge devices. It can also be deployed on the terminal side to enhance the open capabilities of the terminal side, reduce access costs, and sink computing resources to the edge side and terminal side.

[0085] Based on the same inventive concept, the present application also provides a system corresponding to the method in Example 1, see Example 2 for details.

[0086] Embodiment 2

[0087] In this embodiment, a cross-platform audio and video multi-protocol conversion transmission system is provided. Figure 7 As shown, including:

[0088] The audio and video processing module building module is used to build the audio and video processing module, including the signaling layer and the media layer; the signaling layer supports various protocols to access in the form of signaling plug-ins, receive and process various signaling information, and then the signaling processing module schedules the processing of signaling information to realize the conversion of signaling between different protocols; the media layer supports various streaming media transmission protocols to access in the form of media stream plug-ins, receive and process various audio and video data, and then forward them; the audio and video processing module can be deployed in the service layer and / or access layer;

[0089] The signaling negotiation module is used to request the signaling processing module to generate a SessionID to identify the entire communication process when the signaling plug-in receives a signaling request, if it is an initiation command, and complete the initial signaling negotiation based on the matching signaling plug-in; if it is not an initiation command, the corresponding information is recorded based on the SessionID identifier, and the signaling plug-in converts it into the corresponding protocol and sends it to the recipient;

[0090] The media transmission module is used for when the signaling plug-in receives the media stream information to be transmitted, the signaling processing module requests the media layer to create a media stream, the media layer creates the media stream through the corresponding media stream plug-in and updates the transmission relationship, parses the audio and video data based on the matching media stream plug-in, and then re-packages and sends it to the media stream receiver;

[0091] The signaling exit negotiation module is used for when the signaling plug-in receives an exit request, the signaling processing module unbinds the transmission relationship between the corresponding media streams according to the SessionID and updates it. If a valid transmission graph can no longer be formed between the media streams, the entire communication is terminated.

[0092] In this embodiment, the media layer of the audio and video processing module further includes a selective forwarding unit or a multipoint control unit, the selective forwarding unit is used to forward the audio and video data, and the multipoint control unit is used to re-encode the mixed audio and video data before transmission.

[0093] In this embodiment, the signaling negotiation module is specifically used to perform the following steps:

[0094] Step A1, load the plug-in and initialize the configuration information;

[0095] Step A2: When the signaling plug-in receives a signaling request, if it is an initiation command, the signaling processing module is requested to generate a SessionID to identify the entire communication process, and then the sender information, receiver information and media stream information are parsed, and the SessionID is carried to the signaling processing module; if it is not an initiation command, the corresponding information is recorded based on the SessionID identifier, and the signaling plug-in converts it into the corresponding protocol and sends it to the receiver;

[0096] Step A3: The signaling processing module records the relevant information of the entire communication process based on the SessionID, finds the receiver and the matching signaling plug-in according to the SessionID, inputs the signaling command, sender information and media stream information, and converts the information into the corresponding protocol by the signaling plug-in and sends it to the receiver;

[0097] Step A4: After receiving the signaling, the receiving end replies with the corresponding signaling information and repeats steps A2 and A3 to complete the initial signaling negotiation.

[0098] In this embodiment, the media transmission module is specifically used to perform the following steps:

[0099] Step B1: When the signaling plug-in receives the media stream information to be transmitted, it carries the SessionID and the media stream information and transmits it to the signaling processing module. The signaling processing module integrates the transmission relationship between the media stream IDs according to the SessionID and requests the media layer to create a media stream.

[0100] Step B2: The media layer creates media stream information according to the media stream plug-in corresponding to the media stream information, which is used to receive and send media data, and updates the transmission relationship between the media stream IDs to the selective forwarding unit or the multipoint control unit based on the SessionID;

[0101] Step B3: After successfully creating the media stream, the signaling processing module returns a message to the corresponding signaling plug-in for signaling interaction;

[0102] Step B4: After receiving the media stream data, the media stream plug-in parses the original audio and video data packet and transmits it together with the media stream ID to the selective forwarding unit or the multipoint control unit. After determining the media stream receiver, the corresponding media stream plug-in is called to re-pack the original audio and video data packet and send it to the media stream receiver.

[0103] In this embodiment, the signaling exit negotiation module is specifically used to perform the following steps:

[0104] Step C1: When the signaling plug-in receives a logout request, it parses the sender information and carries the SessionID to the signaling processing module;

[0105] Step C2: The signaling processing module finds the communication-related information according to the SessionID, obtains the corresponding media stream ID, and unbinds the transmission relationship between the related media streams. If it is detected that a valid transmission graph can no longer be formed between the media streams, the entire communication is terminated and an exit signaling is sent to all communication objects. Otherwise, the transmission relationship between the media stream IDs of the media layer is updated.

[0106] Since the system introduced in the second embodiment of the present invention is a system used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the system based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All systems used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0107] The present invention supports the access and conversion of various audio and video protocols by constructing an audio and video processing module, can support the access of different manufacturers, new and old equipment, etc., increases the openness of the platform, and supports deployment at the service layer and / or access layer. As long as one protocol is implemented, functions such as multiplexed monitoring and group calling between different devices can be realized, which greatly simplifies application development. Functions such as direct point-to-point audio and video calls of web applications located at the access layer can also be added, thereby enhancing the ability of applications to process audio and video.

[0108] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0112] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A cross-platform audio and video multi-protocol conversion and transmission method, characterized in that: include: The construction process of the audio and video processing module: The audio and video processing module is constructed, including the signaling layer and the media layer; the signaling layer supports various protocols to access in the form of signaling plug-ins, receive and process various signaling information, and then the signaling processing module schedules the processing of signaling information to realize the conversion of signaling between different protocols; the media layer supports various streaming media transmission protocols to access in the form of media stream plug-ins, receive and process various audio and video data, and then forward them; The audio and video processing module can be deployed at the service layer and / or access layer; Signaling negotiation process: When the signaling plug-in receives a signaling request, if it is an initiation command, it requests the signaling processing module to generate a SessionID to identify the entire communication process, and completes the initial signaling negotiation based on the matching signaling plug-in; If it is not an initiation command, the corresponding information is recorded based on the SessionID identifier, and the signaling plug-in converts it into the corresponding protocol and sends it to the recipient; Media transmission process: When the signaling plug-in receives the media stream information to be transmitted, the signaling processing module requests the media layer to create a media stream. The media layer creates the media stream through the corresponding media stream plug-in and updates the transmission relationship. The audio and video data is parsed based on the matching media stream plug-in and then re-packaged and sent to the media stream receiver; Signaling exit negotiation process: When the signaling plug-in receives an exit request, the signaling processing module unbinds the transmission relationship between the corresponding media streams according to the SessionID and updates it. If a valid transmission graph can no longer be formed between the media streams, the entire communication is terminated.

2. The method according to claim 1, characterized in that: The media layer of the audio and video processing module is transmitted after the audio and video data are transferred by the selective forwarding unit or the multi-point control unit re-encodes the mixed audio and video data.

3. The method according to claim 1, characterized in that: The signaling negotiation process specifically includes the following steps: Step A1, load the plug-in and initialize the configuration information; Step A2: When the signaling plug-in receives a signaling request, if it is an initiation command, the signaling processing module is requested to generate a SessionID to identify the entire communication process, and then the sender information, receiver information and media stream information are parsed, and the SessionID is carried into the signaling processing module; If it is not an initiation command, the corresponding information is recorded based on the SessionID identifier, and the signaling plug-in converts it into the corresponding protocol and sends it to the recipient; Step A3: The signaling processing module records the relevant information of the entire communication process based on the SessionID, finds the receiver and the matching signaling plug-in according to the SessionID, inputs the signaling command, sender information and media stream information, and converts the information into the corresponding protocol by the signaling plug-in and sends it to the receiver; Step A4: After receiving the signaling, the receiving end replies with the corresponding signaling information and repeats steps A2 and A3 to complete the initial signaling negotiation.

4. The method according to claim 1, characterized in that: The media transmission process specifically includes the following steps: Step B1: When the signaling plug-in receives the media stream information to be transmitted, it carries the SessionID and the media stream information and transmits it to the signaling processing module. The signaling processing module integrates the transmission relationship between the media stream IDs according to the SessionID and requests the media layer to create a media stream. Step B2: The media layer creates media stream information according to the media stream plug-in corresponding to the media stream information, which is used to receive and send media data, and updates the transmission relationship between the media stream IDs to the selective forwarding unit or the multipoint control unit based on the SessionID; Step B3: After successfully creating the media stream, the signaling processing module returns a message to the corresponding signaling plug-in for signaling interaction; Step B4: After receiving the media stream data, the media stream plug-in parses the original audio and video data packet and transmits it together with the media stream ID to the selective forwarding unit or the multipoint control unit. After determining the media stream receiver, the corresponding media stream plug-in is called to re-pack the original audio and video data packet and send it to the media stream receiver.

5. The method according to claim 1, characterized in that: The signaling exit negotiation process specifically includes the following steps: Step C1: When the signaling plug-in receives a logout request, it parses the sender information and carries the SessionID to the signaling processing module; Step C2: The signaling processing module finds the communication-related information according to the SessionID, obtains the corresponding media stream ID, and unbinds the transmission relationship between the related media streams; if it is detected that a valid transmission graph can no longer be formed between the media streams, the entire communication is terminated and an exit signaling is sent to all communication objects, otherwise the transmission relationship between the media stream IDs of the media layer is updated.

6. A cross-platform audio and video multi-protocol conversion and transmission system, characterized in that: include: The audio and video processing module building module is used to build the audio and video processing module, including the signaling layer and the media layer; The signaling layer supports various protocols to access in the form of signaling plug-ins, receive and process various signaling information, and then the signaling processing module schedules the processing of signaling information to realize the conversion of signaling between different protocols; the media layer supports various streaming media transmission protocols to access in the form of media stream plug-ins, receive and process various audio and video data, and then forward them; The audio and video processing module can be deployed at the service layer and / or access layer; The signaling negotiation module is used to request the signaling processing module to generate a SessionID to identify the entire communication process when the signaling plug-in receives a signaling request, if it is an initiation command, and complete the initial signaling negotiation based on the matching signaling plug-in; If it is not an initiation command, the corresponding information is recorded based on the SessionID identifier, and the signaling plug-in converts it into the corresponding protocol and sends it to the recipient; The media transmission module is used for when the signaling plug-in receives the media stream information to be transmitted, the signaling processing module requests the media layer to create a media stream, the media layer creates the media stream through the corresponding media stream plug-in and updates the transmission relationship, parses the audio and video data based on the matching media stream plug-in, and then re-packages and sends it to the media stream receiver; The signaling exit negotiation module is used for when the signaling plug-in receives an exit request, the signaling processing module unbinds the transmission relationship between the corresponding media streams according to the SessionID and updates it. If a valid transmission graph can no longer be formed between the media streams, the entire communication is terminated.

7. The system according to claim 6, characterized in that: The media layer of the audio and video processing module also includes a selective forwarding unit or a multi-point control unit. The selective forwarding unit is used to forward the audio and video data, and the multi-point control unit is used to re-encode the mixed audio and video data before transmission.

8. The system according to claim 6, characterized in that The signaling negotiation module is specifically used to perform the following steps: Step A1, load the plug-in and initialize the configuration information; Step A2: When the signaling plug-in receives a signaling request, if it is an initiation command, the signaling processing module is requested to generate a SessionID to identify the entire communication process, and then the sender information, receiver information and media stream information are parsed, and the SessionID is carried into the signaling processing module; If it is not an initiation command, the corresponding information is recorded based on the SessionID identifier, and the signaling plug-in converts it into the corresponding protocol and sends it to the recipient; Step A3: The signaling processing module records the relevant information of the entire communication process based on the SessionID, finds the receiver and the matching signaling plug-in according to the SessionID, inputs the signaling command, sender information and media stream information, and converts the information into the corresponding protocol by the signaling plug-in and sends it to the receiver; Step A4: After receiving the signaling, the receiving end replies with the corresponding signaling information and repeats steps A2 and A3 to complete the initial signaling negotiation.

9. The system according to claim 6, characterized in that The media transmission module is specifically used to perform the following steps: Step B1: When the signaling plug-in receives the media stream information to be transmitted, it carries the SessionID and the media stream information and transmits it to the signaling processing module. The signaling processing module integrates the transmission relationship between the media stream IDs according to the SessionID and requests the media layer to create a media stream. Step B2: The media layer creates media stream information according to the media stream plug-in corresponding to the media stream information, which is used to receive and send media data, and updates the transmission relationship between the media stream IDs to the selective forwarding unit or the multipoint control unit based on the SessionID; Step B3: After successfully creating the media stream, the signaling processing module returns a message to the corresponding signaling plug-in for signaling interaction; Step B4: After receiving the media stream data, the media stream plug-in parses the original audio and video data packet and transmits it together with the media stream ID to the selective forwarding unit or the multipoint control unit. After determining the media stream receiver, the corresponding media stream plug-in is called to re-pack the original audio and video data packet and send it to the media stream receiver.

10. The system according to claim 6, characterized in that The signaling exit negotiation module is specifically used to perform the following steps: Step C1: When the signaling plug-in receives a logout request, it parses the sender information and carries the SessionID to the signaling processing module; Step C2: The signaling processing module finds the communication-related information according to the SessionID, obtains the corresponding media stream ID, and unbinds the transmission relationship between the related media streams; if it is detected that a valid transmission graph can no longer be formed between the media streams, the entire communication is terminated and an exit signaling is sent to all communication objects, otherwise the transmission relationship between the media stream IDs of the media layer is updated.

Images