A processing method and device for audio and video data transmission, an apparatus and a storage medium

Abstract

The present disclosure provides a kind of audio and video data transmission processing method, device, equipment and storage medium, the method comprises: first, transmission channel is established between first terminal and second terminal, then, first terminal receives the video picture data collected by second terminal through transmission channel, first terminal calibrates the video picture data synchronously collected with the audio data collected in advance, obtains audio and video data.Further, first terminal pushes audio and video data to streaming media server, wherein, streaming media server is used to push audio and video data to third terminal.Visibly, the present disclosure embodiment establishes transmission channel between first terminal and second terminal, guarantees the stability and reliability of audio and video data transmission between them, so that first terminal can stably and reliably push the audio and video data after time calibration to streaming media server, guarantee the experience of online teaching party.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of data processing, and particularly relates to a processing method and device for audio and video data transmission, an apparatus and a storage medium. BACKGROUND

[0002] At present, online network technology is increasingly applied in the field of education and teaching. Teachers can teach through intelligent terminal devices and the Internet, and students can watch online courses and audio and video to learn knowledge through intelligent terminal devices and the Internet.

[0003] Specifically, the mobile phone and the iPad establish a TCP (Transmission Control Protocol) connection. The mobile phone is used to collect teacher teaching video data, and then transmits the video data to the iPad through the TCP connection. The iPad is used to synchronously collect audio data, and then synchronously collects audio data. After the video data from the mobile phone and the locally synchronously collected audio data are time-synchronized and calibrated, the video data is pushed to a media server for realizing online teaching.

[0004] Since the mobile phone and the iPad of the teacher terminal transmit video data based on the TCP connection, when the network environment is not good, the network packet loss rate increases, which easily triggers the congestion control of the TCP, so as to cause the sending bandwidth to become small, affect the transmission of the video data, and further cause the video data received by the iPad to have problems such as picture delay and picture lag. Moreover, when the network environment recovers, since the congestion data is concentratedly received, the video picture may be accelerated to play, which affects the experience of both parties of online teaching.

[0005] Therefore, how to ensure the stable reliability of the audio and video data transmission between the two devices of the teacher terminal of online teaching and improve the experience of both parties of online teaching is a technical problem to be solved at present. SUMMARY

[0006] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a processing method for audio and video data transmission. The transmission channel is established between the first terminal and the second terminal, the stable reliability of the audio and video data transmission between the first terminal and the second terminal is ensured, so that the first terminal can stably and reliably push the time-synchronized and calibrated audio and video data to the streaming media server, and the experience of both parties of online teaching is ensured.

[0007] In a first aspect, the present disclosure provides a processing method for audio and video data transmission, comprising:

[0008] A transmission channel is established between the first terminal and the second terminal.

[0009] The first terminal receives the video picture data collected by the second terminal through the transmission channel;

[0010] The first terminal calibrates the video picture data with pre-collected audio data to obtain audio-video data; wherein the audio data and the video picture data are collected synchronously;

[0011] The first terminal pushes the audio-video data to a streaming media server; wherein the streaming media server is configured to push the audio-video data to a third terminal.

[0012] In an optional embodiment, the first terminal and the second terminal establish a transmission channel, comprising:

[0013] The first terminal and the second terminal establish an application layer connection;

[0014] The first terminal sends a request offer message to the second terminal through the application layer connection; wherein the offer message carries session description protocol (SDP) information of the first terminal;

[0015] The first terminal receives an answer message from the second terminal through the application layer connection; wherein the answer message carries SDP information of the second terminal;

[0016] The first terminal establishes a peer-to-peer (P2P) transmission channel with the second terminal based on the SDP information of the second terminal.

[0017] In an optional embodiment, the first terminal and the second terminal establish a transmission channel, comprising:

[0018] The first terminal and the second terminal establish an application layer connection;

[0019] The first terminal receives an offer message from the second terminal through the application layer connection; wherein the offer message carries session description protocol (SDP) information of the second terminal;

[0020] The first terminal collects its own SDP information and generates an answer message based on the SDP information;

[0021] The first terminal sends the answer message to the second terminal through the application layer connection; wherein the answer message is used to establish a P2P connection between the first terminal and the second terminal.

[0022] In an optional implementation, the first terminal and the second terminal establish an application layer connection, including:

[0023] The first terminal establishes a TCP connection with the second terminal based on a transmission control protocol (TCP).

[0024] In an optional implementation, the first terminal establishes a TCP connection with the second terminal based on a transmission control protocol (TCP), including:

[0025] The first terminal establishes a TCP connection with the second terminal based on a two-dimensional code of the first terminal or the second terminal.

[0026] In an optional implementation, the first terminal and the second terminal establish a transmission channel, including:

[0027] The first terminal and the second terminal establish a user datagram protocol (UDP) data transmission channel.

[0028] In an optional implementation, the first terminal starts timing when it detects that it starts to establish the application layer connection with the second terminal, and ends timing when it detects that it successfully establishes the P2P transmission channel with the second terminal, and obtains a timing result.

[0029] The first terminal determines the timing result as a connection time between the first terminal and the second terminal.

[0030] In a second aspect, the present disclosure also provides a processing device for audio and video data transmission, including:

[0031] A establishing module is configured to establish a transmission channel with a second terminal.

[0032] A receiving module is configured to receive video picture data collected by the second terminal through the transmission channel.

[0033] A calibration module is configured to calibrate the video picture data with pre-collected audio data to obtain audio and video data, wherein the audio data and the video picture data are collected synchronously.

[0034] A pushing module is configured to push the audio and video data to a streaming media server, wherein the streaming media server is configured to push the audio and video data to a third terminal.

[0035] In a third aspect, the present disclosure provides a computer readable storage medium, which stores instructions, when the instructions are run on a terminal device, the terminal device implements the method described above.

[0036] In a fourth aspect, the present disclosure provides a device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the computer program.

[0037] In a fifth aspect, the present disclosure provides a computer program product, comprising computer programs / instructions, wherein the computer programs / instructions implement the method described above when executed by a processor.

[0038] The technical solution provided by the embodiments of the present disclosure has the following advantages compared with the prior art:

[0039] The embodiments of the present disclosure provide a processing method for audio and video data transmission. First, a transmission channel is established between a first terminal and a second terminal. Then, the first terminal receives video picture data collected by the second terminal through the transmission channel. The first terminal calibrates the video picture data and pre-collected audio data to obtain audio and video data, wherein the audio data and the video picture data are collected synchronously. Further, the first terminal pushes the audio and video data to a streaming media server, wherein the streaming media server is configured to push the audio and video data to a third terminal. It can be seen that the processing method for audio and video data transmission provided by the embodiments of the present disclosure establishes a transmission channel between the first terminal and the second terminal to ensure the stability and reliability of the audio and video data transmission between the two terminals, so that the first terminal can stably and reliably push the audio and video data calibrated in time to the streaming media server, thereby ensuring the experience of both parties in online teaching. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure together with the specification.

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0042] Figure 1 A flowchart of a processing method for audio and video data transmission provided by the embodiments of the present disclosure;

[0043] Figure 2 A schematic diagram of terminal connection time consumption provided by the embodiments of the present disclosure;

[0044] Figure 3 A data interaction schematic diagram of a processing method for audio and video data transmission provided by the embodiments of the present disclosure;

[0045] Figure 4 A structural schematic diagram of a processing device for audio and video data transmission provided by an embodiment of the present disclosure is shown in FIG. 1.

[0046] Figure 5 A structural schematic diagram of a processing device for audio and video data transmission provided by an embodiment of the present disclosure is shown in FIG. 1. DETAILED DESCRIPTION

[0047] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0048] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present disclosure, but the present disclosure can also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some of the embodiments of the present disclosure, not all the embodiments.

[0049] At present, network online technology is more and more applied in the field of education and teaching. Teachers can teach through intelligent terminal equipment and the Internet, and students can watch online courses and audio and video to learn knowledge through intelligent terminal equipment and the Internet.

[0050] Among them, the way for teachers to teach through intelligent terminal equipment and the Internet is that teachers use mobile phones and iPads to teach. Specifically, the mobile phone and the iPad establish a TCP connection. The mobile phone is used to collect teacher teaching video data, and then the video data is transmitted to the iPad through the TCP connection. The iPad is used to synchronously collect audio data, and then the video data from the mobile phone and the locally synchronously collected audio data are time-synchronized and calibrated, and then pushed to a media server for realizing online teaching.

[0051] Since the mobile phone and the iPad of the teacher end transmit video data based on TCP connection, when the network environment is not good, the network packet loss rate increases, which can easily trigger the congestion control of TCP, so as to cause the sending bandwidth to become smaller, affect the transmission of video data, and further cause the video data received by the iPad end to have problems such as picture delay and picture lag. Moreover, when the network environment recovers, since the congestion data is concentratedly received, the video picture may be accelerated to play, which affects the experience of both parties of online teaching.

[0052] Based on the above application scenario, how to ensure the stable reliability of the audio and video data transmission between the two devices of the teacher end of online teaching and improve the experience of both parties of online teaching is a technical problem to be solved by the embodiments of the present disclosure.

[0053] To this end, the embodiment of the present disclosure provides a processing method for audio and video data transmission. First, a transmission channel is established between a first terminal and a second terminal. Then, the first terminal receives video picture data collected by the second terminal through the transmission channel. The first terminal calibrates the video picture data with pre-collected audio data to obtain audio and video data, wherein the audio data and the video picture data are collected synchronously. Further, the first terminal pushes the audio and video data to a streaming media server, wherein the streaming media server is configured to push the audio and video data to a third terminal. It can be seen that the processing method for audio and video data transmission provided by the embodiment of the present disclosure establishes a transmission channel between the first terminal and the second terminal to ensure the stability and reliability of the audio and video data transmission between the two terminals, so that the first terminal can stably and reliably push the audio and video data calibrated in time to the streaming media server to ensure the experience of both parties in online teaching.

[0054] Based on this, the embodiment of the present disclosure provides a processing method for audio and video data transmission, which is described with reference to Figure 1 The embodiment of the present disclosure provides a flow chart of a processing method for audio and video data transmission, which comprises the following steps:

[0055] S101: A transmission channel is established between a first terminal and a second terminal.

[0056] In the embodiment of the present disclosure, the first terminal and the second terminal are in the same local area network, and the first terminal and the second terminal can be intelligent terminals with a camera and a microphone, such as a smart phone, a tablet computer, a computer, etc. In an application scenario, for example, in an online teaching application scenario, the first terminal and the second terminal can be a teacher terminal, for example, the first terminal can be a tablet computer client (such as an iPad teaching terminal), and the second terminal can be a smart phone client (such as a mobile phone streaming terminal), etc. Wherein, the iPad teaching terminal and the mobile phone streaming terminal are in the same local area network.

[0057] In the embodiment of the present disclosure, a transmission channel is established between the first terminal and the second terminal, for example, a P2P (Peer to Peer, Peer-to-Peer) transmission channel can be established, wherein P2P is a distributed application architecture for distributing tasks and workloads between peers (Peers), which is a networking or network form formed by the peer-to-peer computing model in the application layer. The first terminal and the second terminal establish a peer-to-peer connection P2P transmission channel, and the first terminal and the second terminal in the P2P transmission channel are in a peer-to-peer position. The first terminal acts as a server to provide services for the second terminal, and also enjoys the services provided by the second terminal. Conversely, the second terminal acts as a server to provide services for the first terminal, and also enjoys the services provided by the first terminal.

[0058] In an alternative embodiment, the peer-to-peer connection P2P transmission channel between the first terminal and the second terminal can be established according to the following steps A1-A4.

[0059] Step A1: Establish an application layer connection between the first terminal and the second terminal.

[0060] In the embodiments of the present disclosure, the connection between the first terminal and the second terminal is established at the application layer, for example, a TCP (Transmission Control Protocol) connection can be established between the first terminal and the second terminal at the application layer, and TCP communication between the first terminal and the second terminal is realized based on the TCP protocol; and the like.

[0061] In an alternative embodiment, the first terminal establishes a TCP connection with the second terminal based on the transmission control protocol TCP.

[0062] In the embodiments of the present disclosure, the TCP connection is a secondary data transmission channel established at the application layer, and is used to transmit the respective SDP (Session Description Protocol) information of the first terminal and the second terminal, thereby providing services for subsequent establishment of a P2P connection.

[0063] In an alternative embodiment, the first terminal establishes a TCP connection with the second terminal based on a two-dimensional code of the first terminal.

[0064] In the embodiments of the present disclosure, the first terminal generates a two-dimensional code, and the second terminal can perform a scanning operation on the two-dimensional code, and establish a TCP connection between the first terminal and the second terminal based on the two-dimensional code generated by the first terminal.

[0065] In another alternative embodiment, the first terminal establishes a TCP connection with the second terminal based on a two-dimensional code of the second terminal.

[0066] In the embodiments of the present disclosure, the second terminal generates a two-dimensional code, and the first terminal can perform a scanning operation on the two-dimensional code, and establish a TCP connection between the first terminal and the second terminal based on the two-dimensional code generated by the second terminal.

[0067] For example, taking an online teaching application scenario as an example, the first terminal can be an iPad teaching terminal, and the second terminal can be a mobile phone streaming terminal. When the teacher terminal user clicks the "start" button, the iPad teaching terminal automatically starts the camera and microphone, and generates a two-dimensional code. The mobile phone streaming terminal can perform a scanning operation on the two-dimensional code, and establish a TCP connection between the iPad teaching terminal and the mobile phone streaming terminal based on the two-dimensional code.

[0068] In the embodiments of the present disclosure, after scanning the code between the first terminal and the second terminal, a WebRTC (Web Real-Time Communication) connection can be established, and the SDP information between the first terminal and the second terminal is exchanged by using a signaling server. The signaling server is used to help two terminals to establish a connection with as little privacy exposure as possible, thereby improving security. The signaling delivery mode can include establishing a socket connection.

[0069] Step A2: The first terminal sends a request offer message to the second terminal through an application layer connection.

[0070] The offer message carries the session description protocol (SDP) information of the first terminal.

[0071] In the embodiments of the present disclosure, the first terminal sends a request offer message to the second terminal through a socket connection, and waits for an answer message from the second terminal. The offer message includes the message type (indicated by type), the sender's username (indicated by name), the receiver's username (indicated by target), and the SDP information of the first terminal. For example, the offer message includes type: "video-offer", name: the first terminal, target: the second terminal, and the SDP information of the first terminal.

[0072] Step A3: The first terminal receives an answer message from the second terminal through an application layer connection.

[0073] The answer message carries the SDP information of the second terminal.

[0074] In the embodiments of the present disclosure, after the second terminal receives the offer message sent by the first terminal, the second terminal sends an answer message to the first terminal through a socket connection. The answer message includes the message type (indicated by type), the sender's username (indicated by name), the receiver's username (indicated by target), and the SDP information of the second terminal. For example, the answer message includes type: "video-answer", name: the second terminal, target: the first terminal, and the SDP information of the second terminal.

[0075] Step A4: The first terminal establishes a peer-to-peer (P2P) transmission channel with the second terminal based on the SDP information of the second terminal.

[0076] In the embodiment of the present disclosure, after the steps A2 and A3, the ICE (Interactive Connectivity Establishment) protocol can be applied to complete the establishment of the P2P transmission channel by exchanging the SDP information between the first terminal and the second terminal. If the P2P transmission channel is not successfully established, the step A4 is continued, and the ICE protocol is applied to establish the P2P transmission channel between the first terminal and the second terminal based on the SDP information. If the P2P transmission channel is successfully established, the audio and video data between the first terminal and the second terminal can be transmitted through the P2P transmission channel.

[0077] In an optional embodiment, the peer-to-peer connection P2P transmission channel between the first terminal and the second terminal can be established according to the following steps B1-B4.

[0078] Step B1: An application layer connection is established between the first terminal and the second terminal.

[0079] In the embodiment of the present disclosure, the specific process of the step B1 has been described in detail in the step A1 of the above embodiment, and will not be repeated here.

[0080] Step B2: The first terminal receives an offer message from the second terminal through the application layer connection.

[0081] The offer message carries the session description protocol (SDP) information of the second terminal.

[0082] In the embodiment of the present disclosure, the first terminal receives the offer message from the second terminal through the socket connection, wherein the offer message includes the message type (such as represented by type) of the offer message, the sender's username (such as represented by name), the receiver's username (such as represented by target), and the SDP information of the second terminal. For example, the offer message includes type: "video-offer", name: the second terminal, target: the first terminal, the SDP information of the second terminal, and the like.

[0083] Step B3: The first terminal collects the SDP information of itself, and generates an answer message based on the SDP information.

[0084] In the embodiment of the present disclosure, the first terminal collects its own SDP information, and generates an answer message based on the SDP information, wherein the answer message includes a message type (indicated by type) of the answer message, a sender username (indicated by name), an acceptor username (indicated by target), and the SDP information of the first terminal, and the like. For example, the answer message includes type: "video-answer", name: the first terminal, target: the second terminal, the SDP information of the first terminal, and the like.

[0085] Step B4: The first terminal sends the answer message to the second terminal through an application layer connection.

[0086] The answer message is used to establish a P2P connection between the first terminal and the second terminal.

[0087] In the embodiment of the present disclosure, the first terminal sends the answer message to the second terminal through a socket connection, and then, by applying the ICE protocol, the establishment of the P2P transmission channel is completed by exchanging the SDP information between the first terminal and the second terminal. If the P2P transmission channel is successfully established, the audio and video data between the first terminal and the second terminal can be transmitted through the P2P transmission channel.

[0088] In an optional implementation, a user datagram protocol (UDP) data transmission channel is established between the first terminal and the second terminal.

[0089] In the embodiment of the present disclosure, the UDP (User Datagram Protocol) is used to process data packets, and the UDP data transmission channel can be used to transmit the audio and video data between the first terminal and the second terminal. The implementation of establishing the P2P transmission channel between the first terminal and the second terminal includes establishing a UDP data transmission channel between the first terminal and the second terminal.

[0090] S102: The first terminal receives the video picture data collected by the second terminal through the transmission channel.

[0091] In the embodiment of the present disclosure, the second terminal sends the video picture data to the first terminal through the P2P transmission channel after collecting the video picture data. For example, taking an online teaching application scenario as an example, the first terminal can be an iPad teaching terminal, and the second terminal can be a mobile phone streaming terminal. When the teacher terminal user clicks the "start" button, the mobile phone streaming terminal automatically starts the camera and collects the video picture data.

[0092] S103: The first terminal calibrates the video picture data with the pre-collected audio data to obtain audio and video data.

[0093] The audio data is synchronously collected with the video picture data.

[0094] In the embodiment of the present disclosure, the first terminal synchronously collects the audio data while the second terminal collects the video picture data, and after receiving the video picture data transmitted by the second terminal through the P2P transmission channel, the video picture data and the audio data are calibrated, for example, the video picture data and the audio data can be time-aligned to obtain the audio-video data.

[0095] For example, taking the online teaching application scenario as an example, the first terminal can be an iPad teaching end, and the second terminal can be a mobile phone streaming end. When the teacher end user clicks the “start” button, the mobile phone streaming end automatically starts the camera and collects the video picture data. At the same time, the iPad teaching end automatically starts the camera and the microphone and collects the audio data. After receiving the video picture data transmitted by the mobile phone streaming end through the P2P transmission channel, the video picture data and the audio data are time-aligned to obtain the final audio-video data.

[0096] S104: The first terminal pushes the audio-video data to the streaming media server.

[0097] The streaming media server is configured to push the audio-video data to the third terminal.

[0098] In the embodiment of the present disclosure, the streaming media server includes an RTC (Real-Time Communications) server, and the first terminal, the second terminal and the third terminal can apply to join the same RTC room to the RTC server. The first terminal applies to join the RTC room to the RTC server, and after successfully joining the RTC room, can push the audio-video data to the RTC server. The third terminal applies to join the RTC room to the RTC server, and after successfully joining the RTC room, can pull the audio-video data of the RTC server.

[0099] In an optional implementation, the first terminal starts timing when detecting that the first terminal starts to establish an application layer connection with the second terminal, and ends timing when detecting that the P2P transmission channel between the first terminal and the second terminal is successfully established, and obtains the timing result. The first terminal determines the timing result as the connection time consumption between the first terminal and the second terminal.

[0100] As shown in FIG. 1, a schematic diagram of connection time consumption between terminals provided by an embodiment of the present disclosure is shown, wherein the connection time consumption between the first terminal and the second terminal is divided into two parts. The first part is the connection time consumption, which is the timing result of t1+t2+t3+t4+t5 in FIG. 1, wherein the connection time consumption is based on the angle statistics of the first terminal; and the second part is the transmission time consumption. Figure 2 Figure 2 ​​

[0101] In the embodiments of the present disclosure, the first terminal starts timing when it detects that it starts to establish a TCP connection with the second terminal, and ends timing when it detects that a P2P transmission channel is successfully established between the first terminal and the second terminal, and the corresponding timing result is the connection time consumption between the first terminal and the second terminal. As shown in FIG. 8, t1 is the time consumption for the first terminal to start creating an offer to successfully creating the offer, and the local SDP information of the first terminal is set; t2 is the time consumption for successfully creating the offer to the first terminal starting to send the offer to the second terminal; t3 is the time consumption for starting to send the offer to receiving the answer returned by the second terminal, and the remote SDP information of the second terminal is set; t4 is the time consumption for setting the answer of the second terminal to the local of the first terminal, and the ICE protocol connection is started; and t5 is the connection time consumption of the ICE protocol connection, and data transmission is started. Figure 2

[0102] In the embodiments of the present disclosure, during the implementation of t3+t4+t5, the second terminal synchronously performs connection, as shown in t6+t7 in FIG. 8, where t6 is the time consumption for the second terminal receiving the offer of the first terminal to creating its own answer, and the SDP information of the second terminal is set as the remote SDP information; and t7 is the time consumption for the second terminal sending the answer to the successful ICE connection. Figure 2

[0103] In the embodiments of the present disclosure, the transmission time consumption is the time required for a frame of video from being collected to being received by the opposite terminal, as shown in FIG. 8, t8 is the time consumption for the successful ICE connection to receiving the first frame of video. Figure 2

[0104] In the processing method for audio and video data transmission provided by the embodiments of the present disclosure, first, a transmission channel is established between the first terminal and the second terminal, then the first terminal receives the video picture data collected by the second terminal through the transmission channel, and the first terminal calibrates the video picture data with the pre-collected audio data to obtain audio and video data, where the audio data and the video picture data are collected synchronously. Further, the first terminal pushes the audio and video data to a streaming media server, where the streaming media server is configured to push the audio and video data to a third terminal. It can be seen that the processing method for audio and video data transmission provided by the embodiments of the present disclosure establishes a transmission channel between the first terminal and the second terminal to ensure the stability and reliability of the audio and video data transmission between the two terminals, so that the first terminal can stably and reliably push the audio and video data calibrated in time to the streaming media server, and the experience of both parties in online teaching is ensured.

[0105] ​​​Based on the above audio and video data transmission processing method, taking the online teaching application scenario as an example, the first terminal can be an iPad teaching end, the second terminal can be a mobile phone push stream end, and the third terminal can be a student end. The embodiment of the present disclosure also provides an audio and video data transmission processing method, which is referred to Figure 3 The data interaction diagram of the audio and video data transmission processing method provided by the embodiment of the present disclosure is shown in Figure 3 The method comprises the following steps:

[0106] S301: The iPad teaching end generates a two-dimensional code.

[0107] In the embodiment of the present disclosure, the iPad teaching end starts the camera and generates a two-dimensional code.

[0108] S302: The mobile phone push stream end scans the two-dimensional code.

[0109] In the embodiment of the present disclosure, the mobile phone push stream end starts the camera, and performs a scanning operation on the two-dimensional code generated by the iPad teaching end, so as to establish a TCP connection between the iPad teaching end and the mobile phone push stream end.

[0110] S303: The iPad teaching end sends a request offer message to the mobile phone push stream end through an application layer connection.

[0111] The offer message carries the session description protocol (SDP) information of the iPad teaching end.

[0112] In the embodiment of the present disclosure, the iPad teaching end sends a request offer message to the mobile phone push stream end through a socket connection, wherein the offer message includes the message type type: “video-offer”, the sender username name: the iPad teaching end, the receiver username target: the mobile phone push stream end, the SDP information of the iPad teaching end, and the like.

[0113] S304: The mobile phone push stream end receives the offer message sent by the iPad teaching end, and generates a response answer message.

[0114] The answer message carries the SDP information of the mobile phone push stream end.

[0115] In the embodiment of the present disclosure, the mobile phone push stream end receives the offer message sent by the iPad teaching end, and generates a response answer message, wherein the answer message includes the message type type: “video-answer”, the sender username name: the mobile phone push stream end, the receiver username target: the iPad teaching end, the SDP information of the mobile phone push stream end, and the like.

[0116] S305: The iPad teaching terminal receives the answer message from the mobile phone push terminal through the application layer connection.

[0117] In the embodiment of the present disclosure, the iPad teaching terminal receives the answer message from the mobile phone push terminal through the socket connection.

[0118] S306: The iPad teaching terminal establishes a P2P transmission channel with the mobile phone push terminal based on the SDP information of the mobile phone push terminal.

[0119] In the embodiment of the present disclosure, through the above S303-S305, the exchange of SDP information between the iPad teaching terminal and the mobile phone push terminal is realized, and the ICE protocol is applied to complete the establishment of the P2P transmission channel through the exchange of the SDP information between the iPad teaching terminal and the mobile phone push terminal, wherein the P2P transmission channel can be used to transmit the audio and video data between the iPad teaching terminal and the mobile phone push terminal.

[0120] S307: The mobile phone push terminal collects video picture data, and the iPad teaching terminal synchronously collects audio data.

[0121] In the embodiment of the present disclosure, the mobile phone push terminal collects video picture data, and can perform operations such as adding beauty, filters, virtual background effects, etc. on the video picture data, and at the same time, the iPad teaching terminal synchronously collects audio data.

[0122] S308: The mobile phone push terminal sends the video picture data to the iPad teaching terminal through the P2P transmission channel.

[0123] S309: The iPad teaching terminal performs time alignment on the video picture data and the pre-collected audio data to obtain audio and video data.

[0124] In the embodiment of the present disclosure, after the iPad teaching terminal receives the video picture data transmitted by the mobile phone push terminal through the P2P transmission channel, the iPad teaching terminal performs time alignment on the video picture data and the audio data to obtain the audio and video data.

[0125] S310: The iPad teaching terminal pushes the audio and video data to the RTC server.

[0126] In the embodiment of the present disclosure, the iPad teaching terminal can apply to join the RTC room to the RTC server, and after successfully joining the RTC room, push the audio and video data to the RTC server.

[0127] S311: The RTC server pushes the audio and video data to the student terminal.

[0128] In the embodiments of the present disclosure, the student end can apply to join an RTC room to an RTC server, and after successfully joining the RTC room, pull the audio and video data of the RTC server, so as to realize online teaching of the teacher to the student.

[0129] The processing method for audio and video data transmission provided by the embodiments of the present disclosure first establishes a peer-to-peer (P2P) transmission channel between the iPad teaching end and the mobile phone streaming end, then the iPad teaching end receives the video picture data collected by the mobile phone streaming end through the P2P transmission channel, and the iPad teaching end time-aligns the video picture data and the pre-collected audio data to obtain audio and video data, wherein the audio data and the video picture data are synchronously collected. Furthermore, the iPad teaching end pushes the audio and video data to an RTC server, wherein the RTC server is configured to push the audio and video data to a student. It can be seen that the processing method for audio and video data transmission provided by the embodiments of the present disclosure establishes a P2P transmission channel between the iPad teaching end and the mobile phone streaming end, thereby ensuring the stability and reliability of the audio and video data transmission between the two, so that the iPad teaching end can stably and reliably push the time-aligned audio and video data to the RTC server, thereby ensuring the experience of both parties of online teaching.

[0130] Based on the above method embodiments, the present disclosure further provides a processing device for audio and video data transmission, which refers to Figure 4 A structural schematic diagram of a processing device for audio and video data transmission provided by the embodiments of the present disclosure is shown in FIG. 4. The processing device 400 for audio and video data transmission comprises:

[0131] The establishing module 401 is configured to establish a transmission channel with a second terminal;

[0132] The receiving module 402 is configured to receive video picture data collected by the second terminal through the transmission channel;

[0133] The calibration module 403 is configured to calibrate the video picture data and pre-collected audio data to obtain audio and video data; wherein the audio data and the video picture data are synchronously collected;

[0134] The pushing module 404 is configured to push the audio and video data to a streaming media server; wherein the streaming media server is configured to push the audio and video data to a third terminal.

[0135] In an optional embodiment, the establishing module 401 comprises:

[0136] The first establishing submodule is configured to establish an application layer connection with the second terminal;

[0137] The first sending sub-module is configured to send a request offer message to the second terminal through the application layer connection; wherein the offer message carries session description protocol (SDP) information.

[0138] The first receiving sub-module is configured to receive an answer message from the second terminal through the application layer connection; wherein the answer message carries SDP information of the second terminal.

[0139] The second establishing sub-module is configured to establish a peer-to-peer (P2P) transmission channel between the second terminal based on the SDP information of the second terminal.

[0140] In an optional implementation, the establishing module 401 comprises:

[0141] The first establishing sub-module is configured to establish an application layer connection with the second terminal.

[0142] The second receiving sub-module is configured to receive an offer message from the second terminal through the application layer connection; wherein the offer message carries session description protocol (SDP) information of the second terminal.

[0143] The generating sub-module is configured to collect SDP information of itself, and generate an answer message based on the SDP information.

[0144] The second sending sub-module is configured to send the answer message to the second terminal through the application layer connection; wherein the answer message is used to establish a P2P connection with the second terminal.

[0145] In an optional implementation, the first establishing sub-module comprises:

[0146] The third establishing sub-module is configured to establish a transmission control protocol (TCP) connection with the second terminal based on TCP.

[0147] In an optional implementation, the third establishing sub-module is configured to:

[0148] establish a TCP connection with the second terminal based on a two-dimensional code.

[0149] In an optional implementation, the establishing module 401 comprises:

[0150] The first establishing sub-module is configured to establish a user datagram protocol (UDP) data transmission channel with the second terminal.

[0151] In an optional implementation, the apparatus further comprises:

[0152] a timing module configured to start timing when the first terminal detects that the first terminal starts to establish the application layer connection with the second terminal, and stop timing when the first terminal detects that the P2P transmission channel is successfully established between the first terminal and the second terminal, and obtain a timing result;

[0153] a determining module configured to determine, by the first terminal, the timing result as a connection time between the first terminal and the second terminal.

[0154] The processing apparatus for audio and video data transmission provided in the embodiments of the present disclosure first establishes a transmission channel between the first terminal and the second terminal, then the first terminal receives the video picture data collected by the second terminal through the transmission channel, and the first terminal calibrates the video picture data with the pre-collected audio data to obtain audio and video data, wherein the audio data and the video picture data are collected synchronously. Furthermore, the first terminal pushes the audio and video data to a streaming media server, wherein the streaming media server is configured to push the audio and video data to a third terminal. It can be seen that the processing apparatus for audio and video data transmission provided in the embodiments of the present disclosure establishes a transmission channel between the first terminal and the second terminal to ensure the stability and reliability of the audio and video data transmission between the two terminals, so that the first terminal can stably and reliably push the audio and video data calibrated in time to the streaming media server to ensure the experience of both parties in online teaching.

[0155] In addition to the above method and apparatus, the embodiments of the present disclosure further provide a computer-readable storage medium, which stores instructions, and when the instructions are run on a terminal device, the terminal device implements the processing method for audio and video data transmission provided in the embodiments of the present disclosure.

[0156] The embodiments of the present disclosure further provide a computer program product, which includes computer programs / instructions, and when the computer programs / instructions are executed by a processor, the processing method for audio and video data transmission provided in the embodiments of the present disclosure is implemented.

[0157] In addition, the embodiments of the present disclosure further provide a processing device 500 for audio and video data transmission, as shown in Figure 5 may include:

[0158] a processor 501, a memory 502, an input device 503 and an output device 504. The number of processors 501 in the processing device for audio and video data transmission can be one or more, Figure 5 In some embodiments of the present disclosure, the processor 501, the memory 502, the input device 503 and the output device 504 can be connected through a bus or other means, wherein, Figure 5 for example, connected through a bus.

[0159] The memory 502 can be used to store software programs and modules, and the processor 501 executes various function applications and data processing of the audio / video data transmission processing device by running the software programs and modules stored in the memory 502. The memory 502 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function, and the like. In addition, the memory 502 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. The input device 503 can be used to receive input digital or character information, and generate signal input related to user settings and function control of the audio / video data transmission processing device.

[0160] In particular in the present embodiment, the processor 501 loads the executable file corresponding to the process of one or more application programs into the memory 502 according to the following instructions, and runs the application programs stored in the memory 502 by the processor 501, so as to realize the various functions of the audio / video data transmission processing device described above.

[0161] It should be noted that in this paper, the relationship terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0162] The above is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications of these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing audio and video data transmission, characterized in that, The method includes: A transmission channel is established between the first terminal and the second terminal; the transmission channel is a peer-to-peer (P2P) transmission channel established based on the exchange of Session Description Protocol (SDP) information between the first terminal and the second terminal. The first terminal receives video image data collected by the second terminal through the transmission channel; The first terminal calibrates the video image data with the pre-collected audio data to obtain audio-video data; wherein the audio data and the video image data are collected synchronously. The first terminal pushes the audio and video data to the streaming media server; wherein, the streaming media server is used to push the audio and video data to the third terminal.

2. The method according to claim 1, characterized in that, Establishing a transmission channel between the first terminal and the second terminal includes: An application layer connection is established between the first terminal and the second terminal; The first terminal sends a request offer message to the second terminal through the application layer connection; wherein the offer message carries the Session Description Protocol (SDP) information of the first terminal; The first terminal receives an answer message from the second terminal through the application layer connection; wherein the answer message carries the SDP information of the second terminal; The first terminal establishes a peer-to-peer (P2P) transmission channel with the second terminal based on the SDP information of the second terminal.

3. The method according to claim 1, characterized in that, Establishing a transmission channel between the first terminal and the second terminal includes: An application layer connection is established between the first terminal and the second terminal; The first terminal receives an offer message from the second terminal through the application layer connection; wherein the offer message carries the Session Description Protocol (SDP) information of the second terminal; The first terminal collects its own SDP information and generates an answer message based on the SDP information; The first terminal sends the answer message to the second terminal through the application layer connection; wherein, the answer message is used to establish a P2P connection between the first terminal and the second terminal.

4. The method according to claim 2 or 3, characterized in that, Establishing an application layer connection between the first terminal and the second terminal includes: The first terminal establishes a TCP connection with the second terminal based on the Transmission Control Protocol (TCP).

5. The method according to claim 4, characterized in that, The first terminal establishes a TCP connection with the second terminal based on the Transmission Control Protocol (TCP), including: The first terminal establishes a TCP connection with the second terminal based on the QR code of the first terminal or the second terminal.

6. The method according to claim 1, characterized in that, Establishing a transmission channel between the first terminal and the second terminal includes: A User Datagram Protocol (UDP) data transmission channel is established between the first terminal and the second terminal.

7. The method according to claim 2 or 3, characterized in that, The method further includes: When the first terminal detects that it has started to establish the application layer connection with the second terminal, it starts timing and ends timing when it detects that it has successfully established the P2P transmission channel with the second terminal, and obtains the timing result. The first terminal determines the timing result as the connection time between itself and the second terminal.

8. A processing apparatus for audio and video data transmission, characterized in that, The device includes: The module is used to establish a transmission channel with the second terminal; the transmission channel is a peer-to-peer (P2P) transmission channel established based on the exchange of Session Description Protocol (SDP) information between the first terminal and the second terminal. The receiving module is used to receive video image data collected by the second terminal through the transmission channel; A calibration module is used to calibrate the video frame data with pre-acquired audio data to obtain audio-visual data; wherein the audio data and the video frame data are acquired synchronously. A push module is used to push the audio and video data to a streaming media server; wherein the streaming media server is used to push the audio and video data to a third terminal.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method as described in any one of claims 1-7.

10. A device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program / instruction that, when executed by a processor, implements the method as described in any one of claims 1-7.

Images