Method and device for processing streaming media data

Abstract

Provided are a method and device for processing streaming media data, which relate to the field of communication technology, and are used to solve the problem of excessive terminal load when multiple terminals communicate bidirectionally. The method for processing streaming media data includes: establishing a connection between a streaming media server and a first terminal and creating a room for real-time calls in the streaming media server; sending a room identifier of the room to the first terminal so that the first terminal joins the room; receiving a connection request from a second terminal, the connection request including the room identifier; establishing a connection between the streaming media server and the second terminal, and adding the second terminal to the room according to the room identifier; receiving first streaming media data sent by the first terminal, and sending the first streaming media data to the second terminal; receiving second streaming media data sent by the second terminal, and sending the second streaming media data to the first terminal.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a method and device for processing streaming media data. Background Art

[0002] With the development of network technology, live broadcasting via the Internet has become a popular communication method in recent years, for example, online lectures.

[0003] During the live broadcast, the host can send audio and video streams (i.e., audio and / or video streams) to the audience through a terminal (e.g., a sending terminal), and the audience can watch the audio and video streams sent by the host through other terminals (e.g., a receiving terminal).

[0004] However, in the above live broadcast mode, when multiple viewers watch the live broadcast through terminals, that is, when multiple terminals participate in the live broadcast, each terminal is required to send its own audio and video stream to other terminals while receiving the audio and video streams sent by other terminals. Figure 1 As shown, there are four terminals, denoted as A, B, C, and D. Any terminal A may need to directly communicate bidirectionally with the other three terminals. This will cause the terminal to be overloaded, making it impossible for the terminal to bear the load. The more terminals there are, the greater the load on the terminal will be. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for processing streaming media data, which are used to solve the problem of excessive terminal load when multiple terminals communicate bidirectionally.

[0006] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0007] In a first aspect, a method for processing streaming media data is provided, the method comprising: establishing a connection between a streaming media server and a first terminal and creating a room for real-time calls in the streaming media server; sending a room identifier of the room to the first terminal so that the first terminal joins the room; receiving a connection request from a second terminal, the connection request including the room identifier; establishing a connection between the streaming media server and the second terminal, and adding the second terminal to the room according to the room identifier; receiving first streaming media data sent by the first terminal, and sending the first streaming media data to the second terminal; receiving second streaming media data sent by the second terminal, and sending the second streaming media data to the first terminal.

[0008] In some embodiments, after receiving first streaming media data sent by a first terminal and before sending the first streaming media data to a second terminal, the processing method further includes: sending a first publish message to other terminals other than the first terminal that have joined the room, the first publish message being configured to indicate that the first terminal has published streaming media data; receiving a first subscription message sent by the second terminal, the first subscription message being configured to indicate that the second terminal has subscribed to the streaming media data of the first terminal; and / or, after receiving second streaming media data sent by the second terminal and before sending the second streaming media data to the first terminal, the processing method further includes: sending a second publish message to other terminals other than the second terminal that have joined the room, the second publish message being configured to indicate that the second terminal has published streaming media data; receiving a second subscription message sent by the first terminal, the second subscription message being configured to indicate that the first terminal has subscribed to the streaming media data of the second terminal.

[0009] In some embodiments, after the first terminal and the second terminal both join the room, the method for processing streaming media data also includes: creating a first PeerConnection object for the first terminal, and based on the first PeerConnection object, establishing a first P2P channel between the streaming media server and the first terminal; creating a second PeerConnection object for the second terminal, and based on the second PeerConnection object, establishing a second P2P channel between the streaming media server and the second terminal; receiving first streaming media data sent by the first terminal includes: receiving the first streaming media data sent by the first terminal through the first P2P channel; the method for processing streaming media data also includes: adding the received first streaming media data to the first PeerConnection object; copying the first streaming media data to the second PeerConnection object; sending the first streaming media data to the second terminal includes: sending the first streaming media data in the second PeerConnection object to the second terminal through the second P2P channel.

[0010] In some embodiments, after the first terminal and the second terminal both join the room, the method for processing streaming media data further includes: establishing a third P2P channel between the streaming media server and the first terminal based on the first PeerConnection object; establishing a fourth P2P channel between the streaming media server and the second terminal based on the second PeerConnection object; receiving second streaming media data sent by the second terminal includes: receiving the second streaming media data sent by the second terminal through the fourth P2P channel; the method for processing streaming media data further includes: adding the received second streaming media data to the second PeerConnection object; copying the second streaming media data to the first PeerConnection object; sending the second streaming media data to the first terminal includes: sending the second streaming media data in the first PeerConnection object to the first terminal through the third P2P channel.

[0011] In some embodiments, based on the first PeerConnection object, establishing a first P2P channel between the streaming media server and the first terminal includes: saving SDP information sent from the first terminal and creating response SDP information; saving network information sent from the first terminal; based on the second PeerConnection object, establishing a second P2P channel between the streaming media server and the second terminal includes: saving the SDP information sent by the first terminal to the second PeerConnection object; sending the SDP information sent by the first terminal to the second terminal; saving the network information sent from the second terminal.

[0012] In a second aspect, an embodiment of the present invention provides a method for processing streaming media data, which includes: establishing a connection between a first terminal and a streaming media server, and creating a sending PeerConnection object and a receiving PeerConnection object; based on the sending PeerConnection object, establishing a first P2P channel between the first terminal and the streaming media server; through the first P2P channel, the first terminal sends first streaming media data to the streaming media server; based on the receiving PeerConnection object, establishing a third P2P channel between the first terminal and the streaming media server; through the third P2P channel, the first terminal receives second streaming media data sent by the streaming media server.

[0013] In some embodiments, before the first terminal receives the second streaming media data sent by the streaming media server through the third P2P channel, the method for processing the streaming media data also includes: the first terminal receives a second publish message sent by the streaming media server, and the second publish message is configured to indicate that the second terminal has published the streaming media data; the first terminal sends a second subscription message to the streaming media server, and the second subscription message is configured to indicate that the first terminal subscribes to the streaming media data of the second terminal.

[0014] In some embodiments, based on sending a PeerConnection object, establishing a first P2P channel between the first terminal and the streaming media server includes: setting an event response operation for sending a PeerConnection object; obtaining streaming media data from the first terminal and transmitting the streaming media data to the sending PeerConnection object; creating SDP information for the streaming media data and sending the SDP information to the sending PeerConnection object; and sending the SDP information to the streaming media server.

[0015] In a third aspect, an embodiment of the present invention provides a streaming media data processing device, which includes a connection module, a creation module, a receiving module, a sending module and a joining module; the connection module is configured to establish a connection between a streaming media server and a first terminal, and is also configured to establish a connection between the streaming media server and a second terminal; the creation module is configured to create a room for real-time calls in the streaming media server; the sending module is configured to send a room identifier of the room to the first terminal so that the first terminal joins the room; the receiving module is configured to receive a connection request from the second terminal, and the connection request includes a room identifier; the receiving module is also configured to receive first streaming media data sent by the first terminal, and is also configured to receive second streaming media data sent by the second terminal; the joining module is configured to add the second terminal to the room according to the room identifier after the connection module establishes a connection between the streaming media server and the second terminal; the sending module is also configured to send the first streaming media data to the second terminal after the receiving module receives the first streaming media data, and is also configured to send the second streaming media data to the first terminal after the receiving module receives the second streaming media data.

[0016] In some embodiments, after receiving first streaming media data sent by a first terminal and before sending the first streaming media data to a second terminal, the sending module is further configured to send a first publish message to other terminals that have joined the room except the first terminal, and the first publish message is configured to indicate that the first terminal has published the streaming media data; the receiving module is further configured to receive a first subscription message sent by the second terminal, and the first subscription message is configured to indicate that the second terminal has subscribed to the streaming media data of the first terminal; and / or, after receiving second streaming media data sent by the second terminal and before sending the second streaming media data to the first terminal, the sending module is further configured to send a second publish message to other terminals that have joined the room except the second terminal, and the second publish message is configured to indicate that the second terminal has published the streaming media data; the receiving module is further configured to receive a second subscription message sent by the first terminal, and the second subscription message is configured to indicate that the first terminal has subscribed to the streaming media data of the second terminal.

[0017] In a fourth aspect, an embodiment of the present invention provides a device for processing streaming media data, and the processing device of the terminal includes a connection module, a creation module, a sending module and a receiving module; the connection module is configured to establish a connection between a first terminal and a streaming media server; the creation module is configured to create a sending PeerConnection object and a receiving PeerConnection object; the creation module is also configured to establish a first P2P channel between the first terminal and the streaming media server based on the sending PeerConnection object; the sending module is configured to send first streaming media data to the streaming media server through the first P2P channel established by the creation module; the creation module is also configured to establish a third P2P channel between the first terminal and the streaming media server based on the receiving PeerConnection object; the receiving module is configured to receive second streaming media data sent by the streaming media server through the third P2P channel established by the creation module.

[0018] In a fifth aspect, an embodiment of the present invention provides a streaming media server, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the streaming media server implements the steps of the method for processing streaming media data as described in the first and second aspects.

[0019] In a sixth aspect, an embodiment of the present invention provides a first terminal, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the first terminal implements the steps of the method for processing streaming media data as described in the first and second aspects.

[0020] In the seventh aspect, an embodiment of the present invention provides a non-volatile computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for processing streaming media data as described in the first aspect and the second aspect are implemented.

[0021] The method and device for processing streaming media data provided by the embodiment of the present invention can utilize the way of bidirectional transmission of streaming media data by the streaming media server, so that the streaming media server can receive streaming media data sent by one terminal (such as the first terminal) and forward the streaming media data to another terminal (such as the second terminal), and can also receive streaming media data sent by the second terminal and forward the streaming media data to the first terminal. Compared with the prior art, since the streaming media server is added as an intermediate communication node, each terminal only needs to communicate bidirectionally with the streaming media server, without directly communicating with other terminals; for example, if there are 3 terminals that need to communicate bidirectionally between two terminals, each terminal in the prior art needs to communicate directly with the other two terminals, while each terminal in the solution provided by the embodiment of the present invention can communicate directly with the streaming media server, so the load of the terminal can be reduced. When the number of terminals is more, the effect of reducing the load will be more prominent. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A multi-party communication structure diagram provided for related technologies;

[0024] Figure 2 A communication system provided for some embodiments of the present invention;

[0025] Figure 3 A schematic diagram of a method for processing streaming media data provided by some embodiments of the present invention;

[0026] Figure 4 A schematic diagram of another method for processing streaming media data provided by some embodiments of the present invention;

[0027] Figure 5 A system architecture diagram provided for some embodiments of the present invention;

[0028] Figure 6 A structural diagram of a streaming media server provided for some embodiments of the present invention;

[0029] Figure 7 A schematic diagram of the structure of a streaming media data processing device provided in some embodiments of the present invention;

[0030] Figure 8 A schematic structural diagram of another streaming media data processing device provided in some embodiments of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] Unless the context requires otherwise, throughout the specification and claims, the term "include" and its other forms, such as the third person singular form "include" and the present participle form "include", are to be interpreted as open, inclusive, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "example", "specific example" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner.

[0033] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0034] When describing some embodiments, the expressions "coupled" and "connected" and their derivatives may be used. For example, when describing some embodiments, the term "connected" may be used to indicate that two or more components are in direct physical or electrical contact with each other. For another example, when describing some embodiments, the term "coupled" may be used to indicate that two or more components are in direct physical or electrical contact. However, the term "coupled" or "communicative coupling" may also refer to two or more components that are not in direct contact with each other, but still cooperate or interact with each other. The embodiments disclosed herein are not necessarily limited to the contents of this document.

[0035] “At least one of A, B, and C” has the same meaning as “at least one of A, B, or C” and both include the following combinations of A, B, and C: A only, B only, C only, the combination of A and B, the combination of A and C, the combination of B and C, and the combination of A, B, and C.

[0036] “A and / or B” includes the following three combinations: A only, B only, and a combination of A and B.

[0037] "Plurality" means at least two.

[0038] The use of "adapted to" or "configured to" herein is meant to be open and inclusive language that does not exclude devices adapted or configured to perform additional tasks or steps.

[0039] Additionally, the use of “based on” is meant to be open and inclusive, as a process, step, calculation, or other action “based on” one or more stated conditions or values ​​may, in practice, be based on additional conditions or values ​​beyond those stated.

[0040] As used herein, "about" or "approximately" includes the stated value and the average value that is within an acceptable range of deviation from the particular value, where the acceptable range of deviation is determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0041] like Figure 2 As shown, an embodiment of the present invention provides a communication system 100. The communication system 100 includes multiple (at least two) terminals 10 and a streaming media server 20. The communication system 100 can be applied to application scenarios such as live broadcasting and video conferencing. In this embodiment, the terminal 10 can be a mobile terminal or a non-mobile terminal. Exemplarily, the mobile terminal can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle terminal, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc., and the non-mobile terminal can be a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present invention.

[0042] Those skilled in the art should understand that streaming media refers to a media format that transmits audio, video and multimedia files in a streaming manner over the network. Compared with the online playback format of downloading and watching, the characteristic of streaming media is that continuous audio and video information is compressed and put on the streaming media server, so that users can watch while downloading without having to wait for the entire file to be downloaded.

[0043] The streaming media server 20 is the basis of the streaming media application system, and its main function is to collect, cache, schedule, transmit and play streaming media content (streaming media data).

[0044] In the communication system 100, the multiple terminals that communicate with each other may be 2, 3, 5 or more than 5, etc. For the convenience of description, in the embodiment of the present invention, two of the multiple terminals 10 are referred to as the first terminal and the second terminal. For example, the first terminal may be any one of the multiple terminals, and the second terminal may be any one or more of the multiple terminals. For example, the communication system 100 includes three terminals 10, namely terminals 10_A, 10_B, and 10_C; wherein terminal 10_A may be the party that initiates the session, and terminals 10_B and 10_C join the session. At this time, the first terminal and the second terminal may be any two of terminals 10_A, 10_B, and 10_C. For example, the first terminal is terminal 10_A, and the second terminal is terminal 10_B or terminal 10_C.

[0045] like Figure 3 As shown, the embodiment of the present invention provides a method for processing streaming media data, which is applied to Figure 1 In the communication system shown, the method for processing streaming media data includes the following S101 to S116:

[0046] S101: Establish a connection between the streaming media server 20 and the first terminal, and create a room for real-time communication in the streaming media server 20.

[0047] For example, the terminal 10_A sends a WebSocket connection request, specifically, the browser included in the terminal 10_A sends a WebSocket connection request, the streaming server 20 receives the connection request and sends a response, and this process is usually called a "handshake" (i.e., a WebSocket connection is established). After the browser and the streaming server 20 complete the handshake action, it can be considered that a fast channel is formed between the browser and the streaming server 20, through which commands or data other than audio and video streams can be transmitted between the terminal 10_A and the streaming server 20. For example, the fast channel can be used to transmit Session Description Protocol (SDP) information. After the terminal 10_A and the streaming server 20 establish a connection, the streaming server 20 creates a room for real-time audio and video calls.

[0048] S102: The streaming media server 20 sends the room identifier of the room created in S101 to the first terminal, so that the first terminal joins the room.

[0049] Those skilled in the art should understand that a room is a virtual concept, and a room identifier may be a room ID (roomid). The streaming media server 20 generates a unique roomid when creating a room, and sends the roomid to the first terminal so that the first terminal joins the room. When the terminal 10 (for example, the terminal 10_A, the terminal 10_B or the terminal 10_C) receives the roomid of the room, it can be considered that the terminal 10 has joined the room, and only terminals in the same room can communicate with each other.

[0050] S103: The streaming media server 20 receives a connection request from the second terminal, wherein the connection request includes a room identifier.

[0051] S104: Establish a connection between the streaming media server 20 and the second terminal, and add the second terminal to the room according to the room identifier.

[0052] For example, the terminal 10_B sends a WebSocket connection request, the streaming server 20 receives the connection request to establish a connection, and according to the identifier of the room created by the terminal 10_A and the streaming server 20, adds the terminal 10_B to the room.

[0053] In this embodiment, after a terminal establishes a connection with the streaming server and creates a room for real-time communication, other terminals only need to join the room without creating a room. For example, the browser loaded in the terminal 10_B sends a WebSocket connection request to the streaming server 20, and the WebSocket connection request carries a room identifier for requesting to join the room. The streaming server 20 also responds, so that the two establish a WebSocket connection. The process of establishing a connection with the streaming server 20 and joining the room of other terminals is the same as the process performed by the terminal 10_B mentioned above, so it will not be repeated here.

[0054] It should be noted that the first terminal and the second terminal described in the embodiment of the present invention can be any two terminals added to the room. For example, the first terminal can be a terminal that applies to create a room, such as terminal 10_A; the second terminal can be the second or any subsequent terminal added to the room, such as terminal 10_B or terminal 10_C. For another example, the second terminal can be a terminal that applies to create a room, and the first terminal can be the second or any subsequent terminal added to the room. For another example, the first terminal and the second terminal can both refer to the second or any subsequent terminal added to the room, such as terminal 10_B and terminal 10_C, respectively. S105. After establishing a connection with the streaming media server 20, the first terminal creates a sending PeerConnection object and a receiving PeerConnection object.

[0055] S106: After the first terminal joins the room, the streaming media server 20 creates a first PeerConnection object for the first terminal.

[0056] S107 : A first P2P channel is established between the first terminal and the streaming media server 20 , for the first terminal to send streaming media data to the streaming media server 20 .

[0057] For example, the first terminal establishes a first P2P channel (P2P channel) between the first terminal and the streaming server 20 based on the sent PeerConnection object; correspondingly, the streaming server 20 establishes a first P2P channel between the streaming server 20 and the first terminal based on the first PeerConnection object. In other words, the first P2P channel is established between the sent PeerConnection object of the first terminal and the first PeerConnection object of the streaming server 20.

[0058] The sending PeerConnection object is a PeerConnection object used to send streaming media data to the streaming media server 20 , and the receiving PeerConnection object is a PeerConnection object used to receive streaming media data sent by the streaming media server 20 .

[0059] Those skilled in the art should understand that the step of creating an object (eg, sending a PeerConnection object) described in S105 and S106 above can refer to the step of creating a PeerConnection object (also referred to as a PeerConnection instance) in the WebRTC communication process, and therefore will not be repeated herein.

[0060] In the embodiment, the first terminal establishes two PeerConnection objects, which are used for sending and receiving streaming media data respectively. In this way, only data sending can be processed in one PeerConnection object, and only data receiving can be processed in another PeerConnection object, which improves the efficiency of sending and receiving data by the terminal.

[0061] S108: The first terminal sends first streaming media data to the streaming media server 20; correspondingly, the streaming media server 20 receives the first streaming media data sent by the first terminal.

[0062] For example, the first terminal sends the first streaming media data to the streaming media server 20 through the first P2P channel; correspondingly, the streaming media server 20 receives the first streaming media data sent by the first terminal through the first P2P channel.

[0063] In this embodiment, there is no restriction on the type of streaming media data. For example, the streaming media data may be audio, video, or both audio and video.

[0064] It should be noted that the communication system provided in the embodiment of the present invention includes multiple terminals, such as a first terminal and a second terminal. In order to distinguish the streaming media data sent by different terminals, "first", "second", etc. are added in front of them, that is, the "first streaming media data" above and the "second streaming media data" below are obtained. For example, the streaming media data sent by the first terminal is called "first streaming media data".

[0065] In some embodiments, Figure 4 As shown, the process of establishing the first P2P channel between the first terminal and the streaming media server 20 in the above S105 to S107 can be implemented through the following S201 to S214.

[0066] S201: The first terminal creates and sends a PeerConnection object.

[0067] S202: After creating a sending PeerConnection object, the first terminal sets an event response operation of the sending PeerConnection object in the sending PeerConnection object, that is, sets a PeerConnection.addEventListner(icecandidate, onicecandidate) event, so that in subsequent steps, when a network candidate becomes available, this function is executed.

[0068] Those skilled in the art should understand that addEventListner is a function for listening to events. In the PeerConnection.addEventListner(icecandidate, onicecandidate) event, PeerConnection.addEventListner can be a function included in the first terminal that sends the PeerConnection object to execute the listening event; icecandidate can be expressed as a parameter for determining whether a network candidate is available; onicecandidate can be expressed as a function, which is called when the network candidate is available.

[0069] S203: The first terminal obtains first streaming media data from the first terminal through the getUserMedia function. For example, a video stream collected by an image collection device (such as a camera) in the first terminal is obtained, and an audio stream collected by an audio collection device (such as a microphone) is obtained.

[0070] S204: The first terminal transmits the first streaming media data acquired in S203 to the sent PeerConnection object included in the first terminal 10 by calling the addStream function included in the sent PeerConnection object.

[0071] S205: The streaming media server 20 creates a first PeerConnection object for the first terminal.

[0072] S206: The first terminal creates SDP information of a first streaming media data by sending a createOffer function included in the PeerConnection object, and the SDP information stores relevant parameters of the first streaming media data sent by the first terminal 10. For example, when the first streaming media data is a video stream, the SDP information may include the resolution of the video stream, etc.

[0073] S207: The first terminal saves the SDP information generated in S206 into the sent peerConnection object of the first terminal by using the setLocalDescription function included in the sent PeerConnection object.

[0074] S208. The first terminal sends the SDP information saved in S207 from the first terminal to the streaming server 20 by sending the SendOfferSDP function included in the PeerConnection object. Specifically, the SDP information can be sent from the first terminal to the signaling service module included in the streaming server 20. The signaling service module includes the first PeerConnection object.

[0075] S209 . The streaming media server 20 (eg, the first PeerConnection object in the streaming media server 20 ) saves the SDP information sent in S208 through the setRemoteDescription function.

[0076] S210. The streaming media server 20 (eg, the first PeerConnection object in the streaming media server 20) creates a response SDP information through the CreateAnswer function.

[0077] S211. The streaming media server 20 (eg, the first PeerConnection object in the streaming media server 20) sends the answer SDP information created in S210 to the first terminal through the Relay Answer SDP function, which means telling the first terminal that the streaming media server has saved the SDP information of the first streaming media data published by the first terminal.

[0078] S212: The first terminal sends the Send Candidate function included in the PeerConnection object. When the network candidate becomes available, the event response operation of sending the PeerConnection object introduced in S202 is executed, and the network information of the first terminal is sent to the first PeerConnection object in the streaming server 20.

[0079] S213: The streaming media server 20 (eg, the first PeerConnection object in the streaming media server 20) saves the network information of the first terminal sent in S212 through the AddIceCandidate function.

[0080] S214, the streaming media server 20 (for example, the first PeerConnection object in the streaming media server 20) sends the saved network information of other terminals (except the first terminal) to the first terminal through the AddRemoteCandidate function. In this embodiment, after completing the above S201 to S214, it can be considered that a first P2P channel is established between the sending PeerConnection object in the first terminal and the first PeerConnection object in the streaming media server 20.

[0081] S215: After the first P2P channel is established, the first PeerConnection object included in the streaming media server 20 receives the first streaming media data sent by the sending PeerConnection object through the OnAddStream function.

[0082] S109 . After establishing a connection with the streaming media server 20 , the second terminal creates a sending PeerConnection object and a receiving PeerConnection object.

[0083] S110: After the second terminal joins the room, the streaming media server 20 creates a second PeerConnection object for the second terminal.

[0084] S111 . A second P2P channel is established between the second terminal and the streaming media server 20 , for the second terminal to receive streaming media data sent by the streaming media server 20 .

[0085] For example, the second terminal establishes a second P2P channel between the second terminal and the streaming server 20 based on the received PeerConnection object; correspondingly, the streaming server 20 establishes a second P2P channel between the streaming server 20 and the second terminal based on the second PeerConnection object. In other words, a second P2P channel is established between the received PeerConnection object of the second terminal and the second PeerConnection object of the streaming server 20.

[0086] S112: The streaming media server 20 sends the first streaming media data to the second terminal; correspondingly, the second terminal receives the first streaming media data sent by the streaming media server.

[0087] For example, the streaming media server sends the first streaming media data to the second terminal through the second P2P channel; correspondingly, the second terminal receives the first streaming media data sent by the streaming media server through the second P2P channel.

[0088] In some embodiments, Figure 4 As shown, the process of establishing the second P2P channel between the streaming media server and the second terminal in the above S109 to S111 can be implemented through the following S301 to S311.

[0089] S301: The second terminal creates and receives a PeerConnection object.

[0090] S302: After creating a receiving PeerConnection object, the second terminal sets an event response operation of the receiving PeerConnection object in the receiving PeerConnection object, that is, sets a PeerConnection.addEventListner(icecandidate, onicecandidate) event, so that in subsequent steps, when a network candidate becomes available, this function is executed.

[0091] S303 : Create a second PeerConnection object for the second terminal 30 in the streaming media server 20 .

[0092] S304: The streaming media server 20 saves the created SDP information of the first streaming media data into the second peerConnection object through the setLocalDescription function included in the second PeerConnection object.

[0093] S305. The streaming media server sends the SDP information stored in S304 to the second terminal through the SendOfferSDP function included in the second PeerConnection object.

[0094] S306: The second terminal saves the SDP information sent from S305 by receiving the setRemoteDescription function included in the PeerConnection object.

[0095] S307: The second terminal creates a response SDP information by receiving the CreateAnswer function included in the PeerConnection object.

[0096] S308. The second terminal sends the answer SDP information created in S307 to the streaming server 20 by receiving the Relay Answer SDP function included in the PeerConnection object, which means telling the streaming server 20 that the second terminal has saved the SDP information of the first streaming data published by the first terminal 10.

[0097] S309: The second terminal receives the Send Candidate function included in the PeerConnection object. When the network candidate becomes available, the event response operation of receiving the PeerConnection object introduced in S302 is executed, and the network information of the second terminal is sent to the second PeerConnection object in the streaming server 20.

[0098] S310. The streaming media server 20 saves the network information of the second terminal 30 sent in S309 through the AddICECandidate function included in the second PeerConnection object.

[0099] S311. The streaming media server sends the saved network information of other terminals (except the second terminal) to the second terminal through the AddRemoteCandidate function included in the second PeerConnection object.

[0100] In this embodiment, after the above S301 to S311 are completed, it can be considered that a second P2P channel is established between the receiving PeerConnection object in the second terminal and the second PeerConnection object in the streaming media server 20 .

[0101] S312: After the second P2P channel is established, the second terminal (eg, the receiving PeerConnection object in the second terminal) can receive the first streaming media data sent by the second PeerConnection object through the OnAddStream function.

[0102] In the embodiment of the present invention, the second terminal can not only receive the first streaming media data sent by the first terminal, but also send the second streaming media data to the first terminal through the streaming media server 20. Similarly, the first terminal can not only send the first streaming media data to the second terminal 30, but also receive the second streaming media data sent by the second terminal through the streaming media server. Figure 3 As shown, the method for processing streaming media data provided by the embodiment of the present invention may further include the following S113 to S116.

[0103] S113 : A third P2P channel is established between the first terminal and the streaming media server 20 , for the first terminal to receive streaming media data sent by the streaming media server 20 .

[0104] For example, the first terminal establishes a third P2P channel between the first terminal and the streaming server 20 based on the received PeerConnection object; correspondingly, the streaming server 20 establishes a third P2P channel between the streaming server and the first terminal based on the first PeerConnection object. In other words, the third P2P channel is established between the received PeerConnection object of the first terminal and the first PeerConnection object of the streaming server 20.

[0105] S114: The streaming media server 20 sends second streaming media data to the first terminal; correspondingly, the first terminal receives the second streaming media data sent by the streaming media server.

[0106] For example, the streaming media server sends the second streaming media data to the first terminal through the third P2P channel; correspondingly, the first terminal receives the second streaming media data sent by the streaming media server through the third P2P channel. It should be noted that the process of establishing the third P2P channel in S112 does not include S303 compared with the process of establishing the second P2P channel.

[0107] S115 . A fourth P2P channel is established between the second terminal and the streaming media server 20 , for the second terminal to send streaming media data to the streaming media server 20 .

[0108] For example, the second terminal establishes a fourth P2P channel (P2P channel) between the second terminal and the streaming server 20 based on the sent PeerConnection object; correspondingly, the streaming server 20 establishes a fourth P2P channel between the streaming server 20 and the second terminal based on the second PeerConnection object. In other words, the fourth P2P channel is established between the sent PeerConnection object of the second terminal and the second PeerConnection object of the streaming server 20.

[0109] S116: The second terminal sends the second streaming media data to the streaming media server 20; correspondingly, the streaming media server receives the second streaming media data sent by the second terminal. It should be noted that the process of establishing the fourth P2P channel in S115 does not include S205 compared with the process of establishing the first P2P channel.

[0110] In some embodiments, after the streaming media server receives the first streaming media data sent by the first terminal and before the streaming media server sends the first streaming media data to the second terminal, the method for processing the streaming media data further includes:

[0111] S401: The streaming media server sends a first publishing message to other terminals that have joined the room except the first terminal.

[0112] The first publishing message is configured to indicate that the first terminal has published the first streaming media data.

[0113] For example, after receiving the first streaming media data from the terminal 10_A, the streaming media server sends a first publishing message to the terminal 10_B and the terminal 10_C.

[0114] S402: The second terminal receives a first publishing message and sends a first subscription message to a streaming media server.

[0115] For example, after receiving the first publishing message, terminal 10_B sends a first subscription message to the streaming media server. After receiving the first publishing message, terminal 10_C does not respond (does not feed back any message) or sends a message to the streaming media server indicating that it does not subscribe to the first streaming media data.

[0116] S403: The streaming media server receives a first subscription message sent by the second terminal.

[0117] After receiving the first subscription message sent by the second terminal, the streaming media server sends the first streaming media data to the second terminal.

[0118] For example, the streaming media server sends the first streaming media data to the terminal 10_B, but does not send the first streaming media data to the terminal 10_C.

[0119] The first subscription message is configured to indicate that the second terminal subscribes to the first streaming media data published by the first terminal.

[0120] In some embodiments, a first terminal sends first streaming media data to a streaming media server through a first P2P channel. After the streaming media server receives the first streaming media data, the streaming media server sends a first publishing message to other terminals (e.g., a second terminal) other than the first terminal to inform other terminals (e.g., the second terminal) that the first terminal has published the first streaming media data. After the second terminal receives the first publishing message, if the second terminal needs to subscribe to the first streaming media data, the second terminal sends a first subscription message to the streaming media server. After receiving the first subscription message, the streaming media server forwards the first streaming media data to the second terminal through a second P2P channel.

[0121] In some embodiments, after the streaming media server receives the first streaming media data sent by the first terminal 10 through the first P2P channel, the streaming media data processing method provided by the embodiment of the present invention may further include the following S1051 to S1053.

[0122] S1051 . The streaming media server 20 adds the first streaming media data to the first PeerConnection object created in S205 .

[0123] Specifically, Figure 5 As shown, the streaming media server can add the first streaming media data to pubA in the first PeerConnection object.

[0124] S1052: The streaming media server copies the first streaming media data in the first PeerConnection object to the second PeerConnection object created in S303.

[0125] Specifically, Figure 5 As shown, the streaming media server can copy the first streaming media data in pubA to subBA in the second PeerConnection object.

[0126] S1053: The streaming media server sends the first streaming media data in the second PeerConnection object to the second terminal through the second P2P channel.

[0127] Specifically, Figure 5 As shown, the streaming media server can send the first streaming media data in the subBA to the sunBA in the receiving PeerConnection object included in the second terminal.

[0128] In some embodiments, after the streaming media server receives the second streaming media data sent by the second terminal and before the streaming media server sends the second streaming media data to the first terminal, the method for processing the streaming media data further includes:

[0129] S501. The streaming media server 20 sends a second publishing message to other terminals that have joined the room except the second terminal.

[0130] The second publishing message is configured to indicate that the second terminal has published the second streaming media data.

[0131] S502: The first terminal receives a second publish message and sends a second subscribe message to the streaming media server 20.

[0132] The second subscription message is configured to indicate that the first terminal subscribes to the streaming media data of the second terminal.

[0133] S503: The streaming media server 20 receives the second subscription message sent by the first terminal, and after receiving the second subscription message sent by the first terminal, the streaming media server sends second streaming media data to the first terminal.

[0134] In some embodiments, the second terminal sends the second streaming media data to the streaming media server 20 through the fourth P2P channel. After the streaming media server 20 receives the second streaming media data, the streaming media server 20 will send a second publishing message to other terminals (for example, the first terminal) except the second terminal to inform the other terminals that the second terminal has published the second streaming media data. After the first terminal receives the second publishing message, if the first terminal needs to subscribe to the second streaming media data, the first terminal will send a second subscription message to the streaming media server 20. After receiving the second subscription message, the streaming media server 20 will forward the second streaming media data to the first terminal through the third P2P channel.

[0135] In some embodiments, after the streaming media server 20 receives the second streaming media data sent by the second terminal through the fourth P2P, the streaming media data processing method provided by the embodiment of the present invention may further include the following S1081 to S1083 implementations.

[0136] S1081. The streaming media server adds the received second streaming media data to the second PeerConnection object created in S303.

[0137] Specifically, Figure 5 As shown, the streaming media server can add the second streaming media data to pubB in the second PeerConnection object.

[0138] S1082: The streaming media server 20 copies the second streaming media data in the second PeerConnection object to the first PeerConnection object created in S205.

[0139] Specifically, Figure 5 As shown, the streaming media server can copy the second streaming media data in pubB to subAB in the first PeerConnection object.

[0140] S1083: The streaming media server sends the second streaming media data in the first PeerConnection object to the first terminal through the third P2P channel.

[0141] Specifically, Figure 5 As shown, the streaming media server can send the second streaming media data in subAB to subAB in the receiving PeerConnection object included in the first terminal.

[0142] Among them, pubA represents the first streaming media data sent by the first terminal, pubB represents the second streaming media data sent by the second terminal, pubC represents the streaming media data sent by the third terminal, subAB represents the first terminal subscribing to the streaming media data sent by the second terminal, subAC represents the first terminal subscribing to the streaming media data sent by the third terminal, subBA represents the second terminal subscribing to the streaming media data sent by the first terminal, subBC represents the second terminal subscribing to the streaming media data sent by the third terminal, subCA represents the third terminal subscribing to the streaming media data sent by the first terminal, and subCB represents the third terminal subscribing to the streaming media data sent by the second terminal.

[0143] In this embodiment, after the first terminal sends the first streaming media data to the streaming media server and the second terminal sends the second streaming media data to the streaming media server, the streaming media server can realize the forwarding of the streaming media data, that is, the first streaming media data can be forwarded to the second PeerConnection object of the second terminal, and the second streaming media data can be forwarded to the first PeerConnection object of the first terminal, so that the two-way communication of the streaming media server can be utilized without directly communicating with other terminals. In this way, the load of the terminal can be reduced. When the number of terminals is large, the effect of reducing the load will be more prominent.

[0144] Correspondingly, an embodiment of the present invention further provides a streaming media server for implementing the streaming media data processing method described in the above embodiment. Figure 6 3 is a structural diagram of a streaming media server 300 provided in an embodiment of the present application. In some embodiments, Figure 6The streaming media server shown may include at least one processor 301 and a memory 302 .

[0145] The processor 301 includes one or more central processing units (CPUs), which are single-core CPUs or multi-core CPUs.

[0146] The memory 302 includes, but is not limited to, a random access memory (RAM), a read only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or an optical memory, etc. The memory 302 stores operating system codes.

[0147] For example, the processor 301 enables the streaming media server to implement the method for processing streaming media data in the above embodiment by reading the instructions stored in the memory 302, or the processor 301 enables the streaming media server to implement the method for processing streaming media data in the above embodiment through the instructions stored internally. In the case where the processor 301 implements the method in the above embodiment by reading the instructions stored in the memory 302, the memory 302 stores instructions for implementing the method for processing streaming media data provided in the embodiment of the present application.

[0148] In some embodiments, the streaming media server further includes: a receiver 303 and a transmitter 304. The processor 301 is configured to connect the streaming media server to the first terminal, create a room for real-time calls in the streaming media server, and connect the streaming media server to the second terminal, and add the second terminal to the room according to the room identifier; the transmitter 304 is configured to send the room identifier of the room to the first terminal so that the first terminal is added to the room; the receiver 303 is configured to receive a connection request from the second terminal, the connection request includes the room identifier, and is also configured to receive first streaming media data sent by the first terminal; it is also configured to receive second streaming media data sent by the second terminal; the transmitter 304 is also configured to send the first streaming media data to the second terminal, and is also configured to send the second streaming media data to the first terminal.

[0149] In some embodiments, after the streaming media server receives first streaming media data sent by the first terminal, before sending the first streaming media data to the second terminal, the transmitter 304 is also configured to send a first publish message to other terminals that have joined the room except the first terminal, and the first publish message is configured to indicate that the first terminal has published the streaming media data; the receiver 303 is also configured to receive a first subscription message sent by the second terminal, and the first subscription message is configured to indicate that the second terminal has subscribed to the streaming media data of the first terminal; and / or, after the streaming media server receives second streaming media data sent by the second terminal, before sending the second streaming media data to the first terminal, the transmitter 304 is also configured to send a second publish message to other terminals that have joined the room except the second terminal, and the second publish message is configured to indicate that the second terminal has published the streaming media data; the receiver 303 is also configured to receive a second subscription message sent by the first terminal, and the second subscription message is configured to indicate that the first terminal has subscribed to the streaming media data of the second terminal.

[0150] In some embodiments, after the first terminal and the second terminal are both added to the room, the processor 301 is further configured to create a first PeerConnection object for the first terminal, and based on the first PeerConnection object, establish a first P2P channel between the streaming media server and the first terminal; it is also configured to create a second PeerConnection object for the second terminal, and based on the second PeerConnection object, establish a second P2P channel between the streaming media server and the second terminal; the receiver 303 is specifically configured to receive the first streaming media data sent by the first terminal through the first P2P channel by the streaming media server; the processor 301 is further configured to add the received first streaming media data to the first PeerConnection object; and copy the first streaming media data to the second PeerConnection object; the transmitter 304 is specifically configured to send the first streaming media data in the second PeerConnection object to the second terminal through the second P2P channel by the streaming media server.

[0151] In some embodiments, after the first terminal and the second terminal are both added to the room, the processor 301 is further configured to establish a third P2P channel between the streaming media server and the first terminal based on the first PeerConnection object; and is further configured to establish a fourth P2P channel with the second terminal based on the second PeerConnection object; the receiver 303 is specifically configured to receive the second streaming media data sent by the second terminal through the fourth P2P channel; the processor 301 is further configured to add the received second streaming media data to the second PeerConnection object, and copy the second streaming media data to the first PeerConnection object; the transmitter 304 is specifically configured to send the second streaming media data in the first PeerConnection object to the first terminal through the third P2P channel.

[0152] In some embodiments, the processor 301 is specifically configured to save the SDP information sent from the first terminal and create response SDP information; save the network information sent from the first terminal; save the SDP information sent by the first terminal to the second PeerConnection object; send the SDP information sent by the first terminal to the second terminal; save the network information sent from the second terminal.

[0153] In some embodiments, the streaming media server includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the streaming media server implements each process of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0154] The embodiment of the present invention further provides a first terminal for implementing the method for processing streaming media data described in the above embodiment.

[0155] In some embodiments, the first terminal includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the first terminal implements each process of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0156] In some embodiments, the first terminal further includes a processor, a transmitter, and a receiver. The processor is configured to establish a connection between the first terminal and the streaming media server, and create a sending PeerConnection object and a receiving PeerConnection object; the processor is also configured to establish a first P2P channel between the first terminal and the streaming media server based on the sending PeerConnection object; the processor is also configured to establish a third P2P channel between the first terminal and the streaming media server based on the receiving PeerConnection object; the transmitter is configured to send first streaming media data to the streaming media server through the first P2P channel established by the processor; the receiver is configured to receive second streaming media data sent by the streaming media server through the third P2P channel established by the processor.

[0157] In some embodiments, before the first terminal receives the second streaming media data sent by the streaming media server through the third P2P channel, the receiver is also configured for the first terminal to receive a second publishing message sent by the streaming media server, and the second publishing message is configured to indicate that the second terminal has published the streaming media data; the transmitter is also configured for the first terminal to send a second subscription message to the streaming media server, and the second subscription message is configured to indicate that the first terminal subscribes to the streaming media data of the second terminal.

[0158] In some embodiments, the processor is specifically configured to set an event response operation for sending a PeerConnection object; create SDP information for streaming media data; the receiver is configured to obtain streaming media data from the first terminal; the transmitter is configured to transmit the streaming media data to the sending PeerConnection object; and is also configured to send the SDP information to the sending PeerConnection object; and is also configured to send the SDP information to the streaming media server.

[0159] like Figure 7As shown, an embodiment of the present invention provides a streaming media data processing device 500, including a connection module 501, a creation module 502, a receiving module 503, a sending module 504 and a joining module 505. The connection module 501 is configured to establish a connection between the streaming media server and the first terminal, and is also configured to establish a connection between the streaming media server and the second terminal; the creation module 502 is configured to create a room for real-time calls in the streaming media server; the sending module 504 is configured to send the room identifier of the room to the first terminal so that the first terminal joins the room; the receiving module 503 is configured to receive a connection request from the second terminal, the connection request includes the room identifier, and is also configured to receive first streaming media data sent by the first terminal, and is also configured to receive second streaming media data sent by the second terminal; the joining module 505 is configured to add the second terminal to the room according to the room identifier after the connection module 501 establishes a connection between the streaming media server and the second terminal; the sending module 504 is also configured to send the first streaming media data to the second terminal after the receiving module 503 receives the first streaming media data, and is also configured to send the second streaming media data to the first terminal after the receiving module 503 receives the second streaming media data.

[0160] In some embodiments, after receiving first streaming media data sent by a first terminal and before sending the first streaming media data to a second terminal, the sending module 504 is further configured to send a first publish message to other terminals that have joined the room except the first terminal, and the first publish message is configured to indicate that the first terminal has published the streaming media data; the receiving module 503 is further configured to receive a first subscription message sent by the second terminal, and the first subscription message is configured to indicate that the second terminal has subscribed to the streaming media data of the first terminal; and / or, after receiving second streaming media data sent by the second terminal and before sending the second streaming media data to the first terminal, the sending module 504 is further configured to send a second publish message to other terminals that have joined the room except the second terminal, and the second publish message is configured to indicate that the second terminal has published the streaming media data; the receiving module 503 is further configured to receive a second subscription message sent by the first terminal, and the second subscription message is configured to indicate that the first terminal has subscribed to the streaming media data of the second terminal.

[0161] In some embodiments, an embodiment of the present invention provides a streaming media data processing device 500 that also includes a copy module 506. After the first terminal and the second terminal both join the room, the creation module 502 is also configured to create a first PeerConnection object for the first terminal, and based on the first PeerConnection object, establish a first P2P channel between the streaming media server and the first terminal; the creation module 502 is also configured to create a second PeerConnection object for the second terminal, and based on the second PeerConnection object, establish a second P2P channel between the streaming media server and the second terminal; the receiving module 503 is specifically configured to receive the first streaming media data sent by the first terminal through the first P2P channel; the joining module 505 is also configured to add the first streaming media data received by the receiving module 503 to the first PeerConnection object; the copy module 506 is configured to copy the first streaming media data to the second PeerConnection object; the sending module 504 is specifically configured to send the first streaming media data in the second PeerConnection object to the second terminal through the second P2P channel.

[0162] In some embodiments, after the first terminal and the second terminal both join the room, the creation module 502 is further configured to establish a third P2P channel between the streaming media server and the first terminal based on the first PeerConnection object; and is further configured to establish a fourth P2P channel between the streaming media server and the second terminal based on the second PeerConnection object; the receiving module 503 is specifically configured to receive the second streaming media data sent by the second terminal through the fourth P2P channel; the joining module 505 is further configured to add the second streaming media data received by the receiving module 503 to the second PeerConnection object; the copying module 506 is further configured to copy the second streaming media data to the first PeerConnection object; and the sending module 504 is specifically configured to send the second streaming media data in the first PeerConnection object to the first terminal through the third P2P channel.

[0163] In some embodiments, an embodiment of the present invention provides a streaming media data processing device 500 that also includes a saving module, the saving module is configured to save the SDP information sent from the first terminal; the creation module 502 is also configured to create response SDP information; the saving module is also configured to save the network information sent from the first terminal; it is also configured to save the SDP information sent by the first terminal to the second PeerConnection object; the sending module 504 is configured to send the SDP information sent by the first terminal to the second terminal; the saving module is also configured to save the network information sent from the second terminal.

[0164] like Figure 8 As shown, an embodiment of the present invention provides a streaming media data processing device 600, including a connection module 601, a creation module 602, a sending module 603 and a receiving module 604. The connection module 601 is configured to establish a connection between a first terminal and a streaming media server; the creation module 602 is configured to create a sending PeerConnection object and a receiving PeerConnection object; the creation module 602 is also configured to establish a first P2P channel between the first terminal and the streaming media server based on the sending PeerConnection object; the sending module 603 is configured to allow the first terminal to send first streaming media data to the streaming media server through the first P2P channel established by the creation module 602; the creation module 602 is also configured to establish a third P2P channel between the first terminal and the streaming media server based on the receiving PeerConnection object; the receiving module 604 is configured to receive second streaming media data sent by the streaming media server through the third P2P channel established by the creation module 602.

[0165] In some embodiments, before the first terminal receives the second streaming media data sent by the streaming media server through the third P2P channel, the receiving module 604 is also configured for the first terminal to receive a second publishing message sent by the streaming media server, and the second publishing message is configured to indicate that the second terminal has published the streaming media data; the sending module 603 is also configured to send a second subscription message to the streaming media server, and the second subscription message is configured to indicate that the first terminal subscribes to the streaming media data of the second terminal.

[0166] In some embodiments, an embodiment of the present invention provides a streaming media data processing device 600 that also includes an acquisition module 605 and a setting module. The setting module is configured to set an event response operation for sending a PeerConnection object; the acquisition module 605 is configured to acquire streaming media data from a first terminal; the sending module 603 is configured to transmit the streaming media data to the sending PeerConnection object; the creation module 602 is configured to create SDP information for the streaming media data; the sending module 603 is also configured to send the SDP information to the sending PeerConnection object, and send the SDP information to the streaming media server.

[0167] Figure 7 and Figure 8 The device embodiments described are only schematic. For example, the division of the above modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. The functional modules in the various embodiments of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. Figure 7 and Figure 8 The above modules can be implemented in the form of hardware or software functional units. For example, when software is used, the connection module 501 and the creation module 502 can be implemented by Figure 6 At least one processor reads the program code stored in the memory and generates a software function module to implement it. Figure 7 and Figure 8 The above modules can also be implemented by different hardware in the device, for example, the connection module 501 is implemented by Figure 6 The creation module 502 is implemented by a part of the processing resources of at least one processor 301 (for example, one core or two cores of a multi-core processor). Figure 6 The remaining processing resources in at least one processor 301 (such as other cores in a multi-core processor) may be completed by using a field-programmable gate array (FPGA) or a programmable device such as a coprocessor. Obviously, the above functional modules can also be implemented in a combination of software and hardware. For example, the connection module 501 is implemented by a hardware programmable device, and the creation module 502 is a software functional module generated by the CPU after reading the program code stored in the memory.

[0168] Figure 7 and Figure 8For more details on how each module implements the above functions, please refer to the description in the previous method embodiments, which will not be repeated here.

[0169] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0170] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instruction is loaded and executed on a computer, the process or function in accordance with the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) mode or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a magnetic disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state drive (SSD)).

[0171] The embodiment of the present invention further provides a computer-readable storage medium, for example, a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned method embodiments is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing streaming media data, characterized in that: include: Establishing a connection between a streaming media server and a first terminal and creating a room for real-time communication in the streaming media server; sending a room identifier of the room to the first terminal so that the first terminal joins the room; receiving a connection request from a second terminal, wherein the connection request includes the room identifier; Establishing a connection between the streaming media server and the second terminal, and adding the second terminal to the room according to the room identifier; receiving first streaming media data sent by the first terminal, and sending the first streaming media data to the second terminal; receiving second streaming media data sent by the second terminal, and sending the second streaming media data to the first terminal; After receiving the first streaming media data sent by the first terminal and before sending the first streaming media data to the second terminal, the processing method further includes: Sending a first publishing message to other terminals except the first terminal that have joined the room, wherein the first publishing message is configured to indicate that the first terminal has published streaming media data; receiving a first subscription message sent by the second terminal, where the first subscription message is configured to indicate that the second terminal subscribes to streaming media data of the first terminal; and / or, After receiving the second streaming media data sent by the second terminal and before sending the second streaming media data to the first terminal, the processing method further includes: Sending a second publishing message to other terminals except the second terminal that have joined the room, wherein the second publishing message is configured to indicate that the second terminal has published streaming media data; receiving a second subscription message sent by the first terminal, where the second subscription message is configured to indicate that the first terminal subscribes to streaming media data of the second terminal; After both the first terminal and the second terminal join the room, the method for processing streaming media data further includes: Creating a first PeerConnection object for the first terminal, and establishing a first P2P channel between the streaming media server and the first terminal based on the first PeerConnection object; Creating a second PeerConnection object for the second terminal, and establishing a second P2P channel between the streaming media server and the second terminal based on the second PeerConnection object; The receiving the first streaming media data sent by the first terminal comprises: receiving the first streaming media data sent by the first terminal through the first P2P channel; The method for processing streaming media data also includes: Adding the received first streaming media data to the first PeerConnection object; Copying the first streaming media data to the second PeerConnection object; The sending the first streaming media data to the second terminal includes: sending the first streaming media data in the second PeerConnection object to the second terminal through the second P2P channel.

2. The method for processing streaming media data according to claim 1, characterized in that: After both the first terminal and the second terminal join the room, the method for processing streaming media data further includes: Based on the first PeerConnection object, establish a third P2P channel between the streaming media server and the first terminal; Based on the second PeerConnection object, establish a fourth P2P channel between the streaming media server and the second terminal; The receiving the second streaming media data sent by the second terminal comprises: receiving the second streaming media data sent by the second terminal through the fourth P2P channel; The method for processing streaming media data also includes: Adding the received second streaming media data to the second PeerConnection object; Copying the second streaming media data to the first PeerConnection object; The sending the second streaming media data to the first terminal includes: sending the second streaming media data in the first PeerConnection object to the first terminal through the third P2P channel.

3. The method for processing streaming media data according to claim 1, characterized in that: Establishing a first P2P channel between the streaming media server and the first terminal based on the first PeerConnection object includes: Saving the SDP information sent from the first terminal and creating a response SDP information; storing network information sent from the first terminal; Establishing a second P2P channel between the streaming media server and the second terminal based on the second PeerConnection object includes: Saving the SDP information sent by the first terminal to the second PeerConnection object; Sending the SDP information sent by the first terminal to the second terminal; The network information sent from the second terminal is saved.

4. A streaming media data processing device, characterized in that: It includes a connection module, a creation module, a receiving module, a copy module, a sending module and a joining module; The connection module is configured to establish a connection between the streaming media server and the first terminal, and is also configured to establish a connection between the streaming media server and the second terminal; The creation module is configured to create a room for real-time communication in the streaming media server; The sending module is configured to send the room identifier of the room to the first terminal so that the first terminal joins the room; The receiving module is configured to receive a connection request from a second terminal, the connection request including the room identifier, and is further configured to receive first streaming media data sent by the first terminal, and is further configured to receive second streaming media data sent by the second terminal; The joining module is configured to join the second terminal to the room according to the room identifier after the connection module establishes a connection between the streaming media server and the second terminal; The sending module is further configured to send the first streaming media data to the second terminal after the receiving module receives the first streaming media data, and is further configured to send the second streaming media data to the first terminal after the receiving module receives the second streaming media data; After receiving the first streaming media data sent by the first terminal and before sending the first streaming media data to the second terminal, the sending module is further configured to send a first publishing message to other terminals other than the first terminal that have joined the room, wherein the first publishing message is configured to indicate that the first terminal has published the streaming media data; The receiving module is further configured to receive a first subscription message sent by the second terminal, where the first subscription message is configured to indicate that the second terminal subscribes to the streaming media data of the first terminal; and / or, After receiving the second streaming media data sent by the second terminal and before sending the second streaming media data to the first terminal, the sending module is further configured to send a second publishing message to other terminals except the second terminal that have joined the room, wherein the second publishing message is configured to indicate that the second terminal has published the streaming media data; The receiving module is further configured to receive a second subscription message sent by the first terminal, where the second subscription message is configured to indicate that the first terminal subscribes to streaming media data of the second terminal; The creation module is further configured to create a first PeerConnection object for the first terminal after both the first terminal and the second terminal join the room, and establish a first P2P channel between the streaming media server and the first terminal based on the first PeerConnection object; The creation module is further configured to create a second PeerConnection object for the second terminal, and establish a second P2P channel between the streaming media server and the second terminal based on the second PeerConnection object; The receiving module is further configured to receive the first streaming media data sent by the first terminal, including: receiving the first streaming media data sent by the first terminal through the first P2P channel; The adding module is further configured to add the received first streaming media data to the first PeerConnection object; The copy module is configured to copy the first streaming media data to the second PeerConnection object; The sending module is specifically configured to send the first streaming media data in the second PeerConnection object to the second terminal through the second P2P channel.

5. A streaming media server, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the streaming media server implements the steps of the method for processing streaming media data as described in any one of claims 1 to 3.

6. A first terminal, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the first terminal implements the steps of the method for processing streaming media data as described in any one of claims 1 to 3.

7. A non-transitory computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for processing streaming media data as described in any one of claims 1 to 3 are implemented.

Images