Method, device, electronic device and storage medium for transmitting audio and video data
Through end-to-end communication connections between terminals, audio and video data is directly transmitted, which solves the problems of reduced bandwidth pressure and service processing performance of streaming media servers, and improves the audio and video data transmission efficiency and processing capabilities of streaming media servers.
Patent Information
- Application Number
- CN202010501587.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-06-04
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-06-04
AI Technical Summary
In high concurrent audio and video communication scenarios, the streaming media server needs to allocate a large amount of bandwidth resources for audio and video data transmission, resulting in a degradation of its own service processing performance.
The communication data between terminals is obtained through the streaming media server, and the terminals are directly established to prevent audio and video data from being transferred through the streaming media server, and directly transmit end-to-end.
It improves the efficiency of audio and video data transmission, reduces the bandwidth pressure of the streaming media server, and enhances its service processing performance and processing capabilities for high concurrent audio and video communication.
Smart Images

Figure CN111787261B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for sending audio and video data. Background Art
[0002] With the rapid development of network technology, two-way communications such as video conferencing and video teaching have become widely popular in users' lives, work, and study. In video or audio communications in the visual network, a streaming media server usually establishes a connection between the two communicating parties and forwards the audio and video data between the two parties through the established connection. For example, terminal 1 needs to communicate with terminal 2 via video. After establishing a communication connection between terminal 1 and terminal 2, the streaming media server obtains the audio and video data generated by terminal 1 in real time and sends it to terminal 2. At the same time, it obtains the audio and video data generated by terminal 2 in real time and sends it to terminal 1.
[0003] Since video or audio communication is real-time communication, a large amount of audio and video data needs to be transmitted in a short period of time, and the transmission of audio and video data is usually completed by streaming media servers. Therefore, in high-concurrency video communication business scenarios, streaming media servers need to allocate a large amount of additional bandwidth resources to complete the transmission of audio and video data, facing greater bandwidth pressure, resulting in reduced business processing performance. Summary of the invention
[0004] In view of the above problems, embodiments of the present invention are proposed to provide an audio and video data sending method, device, electronic device and storage medium that overcome the above problems or at least partially solve the above problems.
[0005] A first aspect of an embodiment of the present application provides a method for sending audio and video data, which is applied to a first terminal. The method includes:
[0006] Sending a communication connection request to a streaming media server that is in communication connection with the first terminal, wherein the communication connection request is used to request to establish a communication connection with the second terminal;
[0007] receiving first communication data sent by the streaming media server, where the first communication data is communication data corresponding to the second terminal;
[0008] Establishing a first communication connection for end-to-end communication with the second terminal based on the first communication data;
[0009] Based on the first communication connection, audio and video data are sent to the second terminal.
[0010] Optionally, the method further comprises:
[0011] Upon receiving the second communication connection request sent by the second terminal, returning a response indicating approval of the second communication connection request to the second terminal, so as to establish a second communication connection for end-to-end communication with the second terminal;
[0012] Based on the second communication connection, receiving the audio and video data sent by the second terminal;
[0013] The second communication connection request is sent by the second terminal to the first terminal according to second communication data, and the second communication data is sent by the streaming media server to the second terminal when receiving the first communication connection request.
[0014] Optionally, sending audio and video data to the second terminal based on the first communication connection includes:
[0015] Obtaining encoding and decoding parameters of the second terminal from the first communication data;
[0016] Processing the collected audio and video data according to the encoding and decoding parameters;
[0017] Based on the first communication connection, the processed audio and video data is sent to the second terminal.
[0018] Optionally, establishing a first communication connection for end-to-end communication with the second terminal based on the first communication data includes:
[0019] Obtaining address information of the second terminal from the first communication data;
[0020] A communication connection request is sent to the second terminal according to the address information to establish a first communication connection for end-to-end communication with the second terminal.
[0021] Optionally, after establishing the first communication connection for end-to-end communication with the second terminal, the method further includes:
[0022] A message indicating that the first communication connection is successfully established is sent to the streaming media server, so that the streaming media server releases the communication connection with the first terminal.
[0023] A second aspect of an embodiment of the present application provides a method for sending audio and video data, which is applied to a streaming media server, and the method includes:
[0024] Receiving a communication connection request sent by a first terminal in communication connection with the streaming media server, the communication connection request being used to request establishment of a communication connection between the first terminal and a second terminal;
[0025] Send first communication data corresponding to the second terminal to the first terminal, so that the first terminal establishes a first communication connection for end-to-end communication with the second terminal according to the first communication data, and sends audio and video data to the second terminal through the first communication connection.
[0026] Optionally, the method further comprises:
[0027] In response to a communication connection request sent by the first terminal, second communication data corresponding to the first terminal is sent to the second terminal, so that the second terminal establishes a second communication connection with the first terminal for end-to-end communication based on the second communication data, and receives the audio and video data sent by the first terminal through the second communication connection.
[0028] Optionally, before receiving a communication connection request sent by a first terminal that is in communication connection with the streaming media server, the method further includes:
[0029] Obtaining login information of the first terminal and login information of the second terminal;
[0030] Based on the login information of the first terminal, second communication data that complies with the visual network audio and video instant communication protocol is obtained, and based on the login information of the second terminal, first communication data that complies with the visual network audio and video instant communication protocol is obtained.
[0031] Optionally, the method further comprises:
[0032] After receiving a message from the first terminal indicating that the first communication connection is successfully established, releasing the original communication connection with the first terminal;
[0033] After receiving a message from the second terminal indicating that the second communication connection is successfully established, the original communication connection with the second terminal is released.
[0034] A third aspect of an embodiment of the present application provides an audio and video data sending device, which is applied to a first terminal, and the device includes:
[0035] A first sending module, used for sending a communication connection request to a streaming media server connected to the first terminal in communication, wherein the communication connection request is used for requesting to establish a communication connection with the second terminal;
[0036] A first receiving module, configured to receive first communication data sent by the streaming media server, wherein the first communication data is communication data corresponding to the second terminal;
[0037] A first establishing module, used to establish a first communication connection for end-to-end communication with the second terminal based on the first communication data;
[0038] The second sending module is used to send audio and video data to the second terminal based on the first communication connection.
[0039] A fourth aspect of the present application provides an audio and video data sending device, which is applied to a streaming media server, and the device includes:
[0040] A third receiving module, used to receive a communication connection request sent by a first terminal that is in communication connection with the streaming media server, wherein the communication connection request is used to request to establish a communication connection between the first terminal and a second terminal;
[0041] The fourth sending module is used to send first communication data corresponding to the second terminal to the first terminal, so that the first terminal establishes a first communication connection for end-to-end communication with the second terminal according to the first communication data, and sends audio and video data to the second terminal through the first communication connection.
[0042] A fifth aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor is executed, the steps in the method for sending audio and video data described in the first aspect or the second aspect of the present application are implemented.
[0043] The sixth aspect of the embodiments of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the audio and video data sending method described in the first aspect or the second aspect of the present application are implemented.
[0044] Through the audio and video data sending method of the present application, when the first terminal requests to establish a communication connection with the second terminal, the communication data corresponding to the second terminal is first obtained through the streaming media server, and then according to the communication data corresponding to the second terminal, a request for establishing an end-to-end communication connection is directly initiated to the second terminal, and after the establishment is successful, the audio and video data is transmitted to the second terminal through the end-to-end communication connection. On the one hand, since the end-to-end communication connection is established between the terminals through the communication data obtained from the streaming media server, the end-to-end transmission of audio and video data can be directly performed between the terminals, that is, the first terminal directly sends the audio and video data to the second terminal, and avoids the need for the first terminal in the related art to transfer the audio and video data via the streaming media server, thereby improving the efficiency of audio and video data transmission. On the other hand, since the streaming media server can establish an end-to-end communication connection between the terminals by sending communication data to the terminals, so that the terminals can perform end-to-end audio and video transmission, and the streaming media server no longer transmits audio and video data, thereby significantly reducing the bandwidth pressure of the streaming media server, and enhancing the service processing performance of the streaming media server and the ability to cope with high-concurrency audio and video communication services. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application are briefly introduced below.
[0046] Figure 1 It is a schematic diagram of an implementation environment shown in an embodiment of the present application;
[0047] Figure 2 is a flow chart of a method for sending audio and video data shown in an embodiment of the present application;
[0048] Figure 3 This is a process diagram of a method for sending audio and video data shown in an embodiment of the present application;
[0049] Figure 4 is a flowchart of another method for sending audio and video data according to an embodiment of the present application;
[0050] Figure 5 is a flow chart of a method for sending audio and video data shown in an embodiment of the present application;
[0051] Figure 6 is a flowchart of another method for sending audio and video data according to an embodiment of the present application;
[0052] Figure 7 It is a complete process diagram of a method for sending audio and video data shown in an embodiment of the present application;
[0053] Figure 8is a structural block diagram of an audio and video data sending device shown in an embodiment of the present application;
[0054] Fig. 9 is a structural block diagram of another audio and video data sending device shown in an embodiment of the present application;
[0055] Fig.10 is a schematic diagram of an electronic device shown in an embodiment of the present application;
[0056] Fig.11 This is a schematic diagram of a visual network shown in an embodiment of the present application;
[0057] Fig.12 It is a schematic diagram of the hardware structure of a node server shown in an embodiment of the present application;
[0058] Fig.13 It is a schematic diagram of the hardware structure of an access switch shown in an embodiment of the present application;
[0059] Fig.14 It is a schematic diagram of the hardware structure of an Ethernet protocol conversion gateway shown in an embodiment of the present application. DETAILED DESCRIPTION
[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0061] In view of the problems in the related technology, the present application provides a method for sending audio and video data. In the audio and video communication scenario of the visual network, the streaming media server is only responsible for establishing an end-to-end communication connection between the communicating parties. After the end-to-end communication connection between the communicating parties is established, the audio and video data are directly transmitted between the communicating parties through the established channel without passing through the streaming media server, so as to reduce the bandwidth pressure of the streaming media server and enhance the business processing performance of the streaming media server.
[0062] Figure 1 This is a schematic diagram of an implementation environment shown in an embodiment of the present application. Figure 1 In the embodiment, the streaming media server is respectively connected to the first terminal and the second terminal for communication, and the streaming media server, the first terminal and the second terminal are all deployed in a visual networking environment, and communicate and interact with each other through a visual networking protocol.
[0063] Of course, the first terminal and / or the second terminal can be deployed in the visual network or in the Internet ( Figure 1Only the case where the first terminal and / or the second terminal are deployed in the visual network is shown). When the terminal (the first terminal and / or the second terminal) is deployed in the Internet, a protocol conversion server is required as an intermediate bridge to establish a communication connection with the streaming media server. For example, when the first terminal is deployed on the Internet, the first terminal needs to establish a communication connection with the streaming media server through the protocol conversion server. Since the protocol conversion server supports the Internet network card and the visual network network card, it can communicate and interact with the first terminal through the Internet protocol, and can also communicate and interact with the streaming media server through the visual network protocol.
[0064] Figure 2 This is a flow chart of a method for sending audio and video data according to an embodiment of the present application. Figure 2 The method for sending audio and video data provided by this application is described in detail. The method can be applied to Figure 1 In the first terminal, the following steps may be specifically included:
[0065] Step S11: sending a communication connection request to a streaming media server in communication connection with the first terminal, wherein the communication connection request is used to request to establish a communication connection with the second terminal.
[0066] In this embodiment, the first terminal can establish a video communication connection with the second terminal, and can also establish an audio communication connection with the second terminal. When the first terminal requests to establish a video communication connection with the second terminal, a request for establishing a video communication connection is sent to the streaming media server. When the first terminal requests to establish an audio communication connection with the second terminal, a request for establishing an audio communication connection is sent to the streaming media server. In order to facilitate the description of the audio and video data sending method of the present application, it is specifically stated here that the communication connection referred to in each subsequent embodiment refers to a video communication connection.
[0067] Step S12: receiving first communication data sent by the streaming media server, where the first communication data is communication data corresponding to the second terminal.
[0068] In this embodiment, the first communication data is used to establish an end-to-end communication connection with the second terminal.
[0069] Step S13: Based on the first communication data, a first communication connection is established with the second terminal for end-to-end communication.
[0070] In this embodiment, the first terminal can directly establish an end-to-end communication connection with the second terminal through the first communication data, that is, the first terminal can bypass the streaming media server and directly establish a communication connection with the second terminal.
[0071] In one embodiment, if the first terminal and the second terminal are both deployed in the visual network, the first terminal can directly establish an end-to-end communication connection with the second terminal; in another embodiment, if the first terminal is deployed in the visual network and the second terminal is deployed in the Internet, the first terminal can establish an end-to-end communication connection with the second terminal through a protocol conversion server. Similarly, if the second terminal is deployed in the visual network and the first terminal is deployed in the Internet, the second terminal can establish an end-to-end communication connection with the first terminal through a protocol conversion server.
[0072] Step S14: Based on the first communication connection, send audio and video data to the second terminal.
[0073] In this embodiment, since the communication connection between the first terminal and the second terminal is an end-to-end communication connection, the first terminal can directly send audio and video data to the second terminal through the end-to-end communication connection without passing through the streaming media server.
[0074] Figure 3 This is a schematic diagram of a method for sending audio and video data according to an embodiment of the present application. Figure 3 The above steps S11 to S14 are exemplarily described.
[0075] Terminal A is located in the Internet and is connected to the streaming media server through a protocol conversion server. Terminal B is located in the visual network and is directly connected to the streaming media server. When terminal A requests video communication with terminal B, a communication connection request is sent to the streaming media server through the protocol conversion server. After receiving the communication connection request, the streaming media server obtains the communication data corresponding to terminal B and sends the communication data corresponding to terminal B to terminal A. After terminal A receives the communication data corresponding to terminal B, it sends a request to establish an end-to-end communication connection with terminal B to the protocol conversion server through the communication data corresponding to terminal B, so that the protocol conversion server sends the request to terminal B. If terminal B returns a response to terminal A agreeing to establish the communication connection, the end-to-end communication connection between terminal A and terminal B is successfully established. Specifically, the communication connection is composed of terminal A, the protocol conversion server and terminal B. Figure 3 As shown in the bold part in . Afterwards, terminal A can send the collected audio and video streams to terminal B through the established end-to-end communication connection.
[0076] Through the audio and video data sending method of this embodiment, when the first terminal requests to establish a communication connection with the second terminal, the streaming media server first obtains the communication data corresponding to the second terminal, and then directly initiates a request to establish an end-to-end communication connection to the second terminal according to the communication data corresponding to the second terminal, and after the establishment is successful, the audio and video data is transmitted to the second terminal through the end-to-end communication connection. On the one hand, since the end-to-end communication connection is established between the terminals through the communication data obtained from the streaming media server, the end-to-end transmission of audio and video data can be directly performed between the terminals, that is, the first terminal directly sends the audio and video data to the second terminal, and avoids the need for the first terminal to transfer the audio and video data via the streaming media server in the related art, thereby improving the efficiency of audio and video data transmission. On the other hand, since the streaming media server can establish an end-to-end communication connection between the terminals by sending communication data to the terminals, so that the terminals can perform end-to-end audio and video transmission, and the streaming media server no longer transmits audio and video data, thereby significantly reducing the bandwidth pressure of the streaming media server, and enhancing the service processing performance of the streaming media server and the ability to cope with high-concurrency audio and video communication services.
[0077] In the present application, after the first terminal sends a communication connection request to the streaming media server, the streaming media server will send the communication data corresponding to the second terminal to the first terminal, so that the first terminal actively establishes an end-to-end communication connection with the second terminal; at the same time, the communication data corresponding to the first terminal will also be sent to the second terminal, so that the second terminal actively establishes an end-to-end communication connection with the first terminal, thereby realizing a two-way communication connection between the first terminal and the second terminal.
[0078] Figure 4 FIG. 1 is a flowchart of another method for sending audio and video data according to an embodiment of the present application. Figure 4 In addition to the above steps S11 to S14, the audio and video data sending method of the present application may also include the following steps:
[0079] Step S15: upon receiving the second communication connection request sent by the second terminal, returning a response indicating approval of the second communication connection request to the second terminal, so as to establish a second communication connection for end-to-end communication with the second terminal.
[0080] In this embodiment, the communication connection actively established between the first terminal and the second terminal is the first communication connection, and the function of the first communication connection is to transmit the audio and video data generated by the first terminal to the second terminal. The communication connection actively established between the second terminal and the first terminal is the second communication connection, and the function of the second communication connection is to transmit the audio and video data generated by the second terminal to the first terminal.
[0081] In this embodiment, after receiving the first communication connection request sent by the first terminal, if the second terminal returns a response to the first terminal indicating approval of the first communication connection request, then the first communication connection is successfully established. Similarly, after receiving the second communication connection request sent by the second terminal, if the first terminal returns a response to the second terminal indicating approval of the second communication connection request, then the second communication connection is successfully established.
[0082] Step S16: Based on the second communication connection, receiving the audio and video data sent by the second terminal.
[0083] The second communication connection request is sent by the second terminal to the first terminal according to second communication data, and the second communication data is sent by the streaming media server to the second terminal when receiving the first communication connection request.
[0084] In this embodiment, after the second communication connection is successfully established, the second terminal can send the collected audio and video streams to the first terminal through the second communication connection, that is, the first terminal can receive the audio and video data sent by the second terminal through the second communication connection.
[0085] Through the audio and video data sending method of this embodiment, the first terminal and the second terminal can actively request to establish an end-to-end communication connection with the other end, and then establish a two-way communication connection to achieve two-way transmission of audio and video, which can effectively meet the user's interactive needs in audio and video communication and enhance the user's usage experience.
[0086] In combination with the above embodiments, in one implementation, the present application embodiment further provides a method for sending audio and video data, such as Figure 5 shown. Figure 5 FIG. 1 is a flow chart of a method for sending audio and video data according to an embodiment of the present application. Figure 5 , the above step S14 may include:
[0087] Step S141: Obtaining encoding and decoding parameters of the second terminal from the first communication data.
[0088] Step S142: Processing the collected audio and video data according to the encoding and decoding parameters.
[0089] Step S143: Based on the first communication connection, the processed audio and video data is sent to the second terminal.
[0090] In this embodiment, the codec parameters include video coding parameters and audio coding parameters. The video coding parameters may include: video frame rate, video bit rate, video resolution, video sampling rate, video sampling accuracy, video quality, etc. The audio coding parameters may include: audio bit rate, audio sampling rate, audio sampling accuracy, audio quality, etc.
[0091] In this embodiment, the first terminal can obtain the encoding and decoding capability of the second terminal based on the first communication data. In order to ensure that the second terminal can successfully decode the audio and video data after receiving it sent by the first terminal, the first terminal will first encode the collected audio and video data according to the encoding parameters of the second terminal after collecting the audio and video data, and then send it to the second terminal.
[0092] In actual implementation, after obtaining the codec parameters of the second terminal, the first terminal can set its own codec device according to these codec parameters so that its own codec device has the same codec capability as the codec device of the second terminal, thereby ensuring smooth transmission of audio and video data.
[0093] In this embodiment, when the first terminal requests to establish a communication connection with the second terminal, it can also obtain the encoding and decoding capabilities of the second terminal based on the first communication data, and then configure its own encoding and decoding device to a device with the same encoding and decoding capabilities as the second terminal, so that the second terminal can successfully decode after receiving the audio and video data sent by the first terminal, thereby ensuring the smooth transmission of the audio and video data.
[0094] In combination with the above embodiments, in one implementation, if the second terminal also establishes a second communication connection with the first terminal, the second terminal can also obtain the codec parameters of the first terminal from the second communication data, process the collected audio and video data according to the codec parameters, and then send the processed audio and video data to the first terminal based on the second communication connection. In this way, after the first terminal and the second terminal establish an end-to-end communication connection, they can also encode and decode the audio and video data collected by themselves according to the codec parameters of the other terminal, thereby ensuring the smooth transmission of audio and video data.
[0095] In combination with the above embodiments, in one implementation, the present application further provides a method for establishing a first communication connection. Specifically, the above step S13 may include:
[0096] Obtaining address information of the second terminal from the first communication data;
[0097] A communication connection request is sent to the second terminal according to the address information to establish a first communication connection for end-to-end communication with the second terminal.
[0098] In this embodiment, after receiving the first communication data, the first terminal obtains the address information of the second terminal from the first communication data, such as MAC address, TCP port number, UDP port number, etc. Then, according to the address information, a communication connection request for establishing the first communication connection is sent to the second terminal.
[0099] Accordingly, after receiving the second communication data, the second terminal obtains the address information of the first terminal from the second communication data, such as MAC address, TCP port number, UDP port number, etc. Then, according to the address information, sends a communication connection request to establish a second communication connection to the first terminal.
[0100] In this embodiment, after receiving the communication connection request sent by the first terminal, the second terminal automatically returns a response to the first terminal in a default state, agreeing to establish the first communication connection, or it may return a response to the first terminal whether to establish the first communication connection after obtaining the operation of the user on the second terminal side. Similarly, after receiving the communication connection request sent by the second terminal, the first terminal automatically returns a response to the second terminal in a default state, agreeing to establish the second communication connection, or it may return a response to the second terminal whether to establish the first communication connection after obtaining the operation of the user on the first terminal side. This embodiment does not impose specific restrictions on this.
[0101] In this embodiment, the communication data (including the first communication data and the second communication data) carries the address information of the other end, so that the first terminal and / or the second terminal can successfully establish a communication connection with the other end according to the address information of the other end, thereby realizing end-to-end transmission of audio and video data.
[0102] In combination with the above embodiments, in one implementation manner, after step S13, the following steps may also be included:
[0103] A message indicating that the first communication connection is successfully established is sent to the streaming media server, so that the streaming media server releases the communication connection with the first terminal.
[0104] In this embodiment, since an end-to-end communication connection is established between the first terminal and the second terminal, the streaming media server does not need to participate in the subsequent transmission of audio and video data. Therefore, after the first terminal determines that the first communication connection is successfully established, it can send a message indicating that the first communication connection is successfully established to the streaming media server, so that the streaming media server releases the original communication connection between the first terminal, thereby reducing the bandwidth pressure of the streaming media server and using the bandwidth resources to establish communication connections between other terminals, thereby enhancing the business processing performance of the streaming media server.
[0105] Accordingly, if a two-way communication connection is established between the first terminal and the second terminal, after the second terminal determines that the second communication connection is successfully established, it can send a message indicating that the second communication connection is successfully established to the streaming media server, so that the streaming media server releases the original communication connection between the second terminal.
[0106] In this embodiment, after learning that the first communication connection is successfully established, the streaming media server releases the original communication connection with the first terminal, thereby reducing its own bandwidth pressure and enhancing its own service processing performance.
[0107] The present application also provides another method for sending audio and video data. Figure 6 shown. Figure 6 FIG. 1 is a flowchart of another method for sending audio and video data according to an embodiment of the present application. The method for sending audio and video data of the present application is applied to a streaming media server, comprising:
[0108] Step S21: receiving a communication connection request sent by a first terminal in communication connection with the streaming media server, wherein the communication connection request is used to request to establish a communication connection between the first terminal and a second terminal.
[0109] Step S22: Send first communication data corresponding to the second terminal to the first terminal, so that the first terminal establishes a first communication connection for end-to-end communication with the second terminal according to the first communication data, and sends audio and video data to the second terminal through the first communication connection.
[0110] For the description of step S21 - step S22 , reference may be made to the aforementioned description of step S11 - step S14 , and the embodiments of the present application will not be elaborated herein.
[0111] In combination with the above embodiments, in one implementation manner, the audio and video data sending method of the present application may further include:
[0112] In response to a communication connection request sent by the first terminal, second communication data corresponding to the first terminal is sent to the second terminal, so that the second terminal establishes a second communication connection with the first terminal for end-to-end communication based on the second communication data, and receives the audio and video data sent by the first terminal through the second communication connection.
[0113] For the description of this part, please refer to the aforementioned description of step S15-step S16, and the embodiment of the present application will not be described in detail here.
[0114] In combination with the above embodiment, in one implementation manner, before step S21, the audio and video data sending method of the present application may further include the following steps:
[0115] Obtaining login information of the first terminal and login information of the second terminal;
[0116] Based on the login information of the first terminal, second communication data that complies with the visual network audio and video instant communication protocol is obtained, and based on the login information of the second terminal, first communication data that complies with the visual network audio and video instant communication protocol is obtained.
[0117] In this embodiment, the first terminal and the second terminal support the video network audio and video instant communication protocol. The format of the video network audio and video instant communication protocol is shown in the following table:
[0118]
[0119] In the visual network audio and video instant communication protocol, the version number, CRC check, host name, source address (ie: source IP address), source MAC, target address (ie: target IP address), target MAC, source TCP port, target TCP port, source UDP port and target UDP port are the header information of the protocol; network type, network bandwidth, network adoption time, maximum network bandwidth and minimum network bandwidth are the network information of the two communicating terminals (source terminal and target terminal); CPU model, CPU size, CPU utilization, memory model, memory size, memory utilization, graphics card model, graphics card size and graphics card utilization are the codec capability information of the current host; video protocol, video frame rate, video bit rate, video resolution, video sampling rate, video sampling accuracy and video quality are the video codec parameter information of the current host; audio protocol, audio sampling rate, audio sampling accuracy and audio quality are the audio codec parameter information of the current host. In this embodiment, some devices need to use IP addressing (such as terminals deployed in the Internet), and some devices need to use MAC addressing (such as terminals deployed in the visual network), so IP address and MAC address need to be set in the protocol.
[0120] In this embodiment, each time the first terminal logs in to the streaming media server, it obtains its own network information, codec capability information, video codec parameter information, and audio codec parameter information, carries this part of data in the login request, and sends the login request to the streaming media server based on the visual network audio and video instant communication protocol. After receiving the login request, the streaming media server parses the login request, obtains the login information, and uses the network information of the first terminal, the codec capability information of the first terminal, the video codec parameter information of the first terminal, and the audio codec parameter information of the first terminal in the login information as the communication data corresponding to the first terminal, and stores it in the database. The format of the communication data is shown in the following table:
[0121]
[0122] Similarly, each time the second terminal logs in to the streaming media server, it obtains its own network information, codec capability information, video codec parameter information, and audio codec parameter information, carries this part of data in the login request, and sends the login request to the streaming media server based on the visual network audio and video instant communication protocol. After receiving the login request, the streaming media server parses the login request, obtains the login information, obtains the network information of the second terminal, the codec capability information of the second terminal, the video codec parameter information of the second terminal, and the audio codec parameter information of the second terminal from the login information, and uses this part of information as the communication data corresponding to the second terminal and stores it in the database. The format of the communication data is shown in the following table:
[0123]
[0124] When the streaming media server receives a communication connection request sent by the first terminal, it obtains the communication data corresponding to the second terminal, takes the first terminal as the target terminal, sets the target address, target MAC, target TCP port, and target UDP port in the header information of the communication data, and after the setting is successful, sends the communication data corresponding to the second terminal to the first terminal.
[0125] If a two-way communication connection needs to be established between the first terminal and the second terminal, the streaming media server obtains the communication data corresponding to the first terminal, takes the second terminal as the target terminal, and sets the target address, target MAC, target TCP port, and target UDP port in the header information in the communication data. After the setting is successful, the communication data corresponding to the first terminal is sent to the second terminal.
[0126] In this embodiment, the first terminal and the second terminal actively report their own communication data to the streaming media server as backup data when logging in, so that after receiving the request to establish an end-to-end communication connection between the first terminal and the second terminal, the streaming media server automatically obtains the communication data corresponding to the second terminal and sends it to the first terminal, and obtains the communication data corresponding to the first terminal and sends it to the second terminal, thereby enabling the first terminal and the second terminal to actively establish a communication connection with the other terminal based on the received communication data, and finally realize an end-to-end communication connection that is not dependent on the maintenance of the streaming media server, thereby speeding up the transmission efficiency of audio and video data and reducing the bandwidth pressure of the streaming media server.
[0127] In combination with the above embodiment, in one implementation manner, the audio and video data sending method of this embodiment may further include the following steps:
[0128] After receiving a message from the first terminal indicating that the first communication connection is successfully established, releasing the original communication connection with the first terminal;
[0129] After receiving a message from the second terminal indicating that the second communication connection is successfully established, the original communication connection with the second terminal is released.
[0130] Please refer to the previous text for the description of this part, and this embodiment will not be described in detail here.
[0131] Figure 7 This is a complete process diagram of a method for sending audio and video data according to an embodiment of the present application. Figure 7 , the audio and video sending method of the present application is described in detail with a specific embodiment.
[0132] exist Figure 7 In the example, client A is installed in the first terminal, client B is installed in the second terminal, user A is on the client A side, and user B is on the client B side. Client A is located in the Internet and communicates with the streaming media server through a monitoring access server (the monitoring access server is used to complete the protocol conversion between the visual network and the Internet, and it can also be a protocol conversion server); client B is located in the visual network and can communicate directly with the streaming media server. The streaming media server is provided with a client online list, which displays all currently successfully logged in clients. The online client can select a client from all other online clients and initiate audio and video communication.
[0133] When client A logs in to the streaming media server, the streaming media server records the communication data corresponding to client A in the login information of client A, and returns the current online client list to client A; similarly, when client B logs in to the streaming media server, the streaming media server records the communication data corresponding to client B in the login information of client B, and returns the current online client list to client B. At this time, client A can see that client B is online, and client B can see that client A is online, and both parties can initiate audio and video communication.
[0134] Taking client A initiating audio and video communication with client B as an example, user A selects client B in the client online list, client A generates a communication connection request according to the Internet protocol, and then sends the communication connection request to the monitoring access server, the monitoring access server encapsulates the communication connection request according to the visual network protocol, and then sends the communication connection request to the streaming media server, the streaming media server parses the communication connection request, obtains the communication data corresponding to client B, and sends the data to client A, and at the same time, obtains the communication data corresponding to client A, and sends the data to client B. Client A establishes a first communication connection with client B according to the communication data corresponding to client B, and client B establishes a second communication connection with client A according to the communication data corresponding to client A, and at this time, the two-way communication connection between client A and client B is successfully established.
[0135] Afterwards, client A and client B can directly transmit audio and video data in both directions through the established channel, wherein the monitoring access server communicates and interacts with client A through Internet commands, and communicates and interacts with client B through visual network commands, thereby realizing the transmission of audio and video data between the Internet and the visual network. In this process, audio and video data can be directly transmitted between client A and client B, which can significantly speed up the transmission of audio and video data compared to the original transmission method that requires a streaming media server as a transit, and at the same time, reduce the bandwidth pressure of the streaming media server.
[0136] Based on the same inventive concept, the present application also provides an audio and video data sending device 800, which is applied to a first terminal, such as Figure 8 shown. Figure 8 1 is a structural block diagram of an audio and video data sending device shown in an embodiment of the present application. Figure 8 , the audio and video data sending device 800 of the present application may include:
[0137] A first sending module 801 is used to send a communication connection request to a streaming media server that is in communication connection with the first terminal, wherein the communication connection request is used to request to establish a communication connection with the second terminal;
[0138] A first receiving module 802 is used to receive first communication data sent by the streaming media server, where the first communication data is communication data corresponding to the second terminal;
[0139] A first establishing module 803, configured to establish a first communication connection for end-to-end communication with the second terminal based on the first communication data;
[0140] The second sending module 804 is used to send audio and video data to the second terminal based on the first communication connection.
[0141] Optionally, the device 800 further includes:
[0142] a second establishing module, configured to, upon receiving a second communication connection request sent by the second terminal, return a response indicating approval of the second communication connection request to the second terminal, so as to establish a second communication connection for end-to-end communication with the second terminal;
[0143] A second receiving module, configured to receive the audio and video data sent by the second terminal based on the second communication connection;
[0144] The second communication connection request is sent by the second terminal to the first terminal according to second communication data, and the second communication data is sent by the streaming media server to the second terminal when receiving the first communication connection request.
[0145] Optionally, the second sending module 804 includes:
[0146] A first obtaining submodule, used to obtain the encoding and decoding parameters of the second terminal from the first communication data;
[0147] A processing module, used for processing the collected audio and video data according to the encoding and decoding parameters;
[0148] The first sending submodule is used to send the processed audio and video data to the second terminal based on the first communication connection.
[0149] Optionally, the first establishing module 803 includes:
[0150] A second obtaining submodule, used to obtain address information of the second terminal from the first communication data;
[0151] The second sending submodule is used to send a communication connection request to the second terminal according to the address information, so as to establish a first communication connection for end-to-end communication with the second terminal.
[0152] Optionally, the device 800 further includes:
[0153] The third sending module is used to send a message indicating that the first communication connection is successfully established to the streaming media server, so that the streaming media server releases the communication connection with the first terminal.
[0154] Based on the same inventive concept, the present application also provides another audio and video data sending device 900, which is applied to a streaming media server, such as Fig. 9 shown. Fig. 9 FIG. 1 is a structural block diagram of another audio and video data sending device according to an embodiment of the present application. Fig. 9, the audio and video data sending device 900 of the present application may include:
[0155] A third receiving module 901 is used to receive a communication connection request sent by a first terminal that is in communication connection with the streaming media server, wherein the communication connection request is used to request to establish a communication connection between the first terminal and a second terminal;
[0156] The fourth sending module 902 is used to send first communication data corresponding to the second terminal to the first terminal, so that the first terminal establishes a first communication connection for end-to-end communication with the second terminal according to the first communication data, and sends audio and video data to the second terminal through the first communication connection.
[0157] Optionally, the device 900 further includes:
[0158] The fifth sending module is used to send second communication data corresponding to the first terminal to the second terminal in response to the communication connection request sent by the first terminal, so that the second terminal establishes a second communication connection for end-to-end communication with the first terminal according to the second communication data, and receives the audio and video data sent by the first terminal through the second communication connection.
[0159] Optionally, the device 900 further includes:
[0160] A first obtaining module, used to obtain login information of the first terminal and login information of the second terminal;
[0161] The second acquisition module is used to obtain second communication data that complies with the visual network audio and video instant communication protocol based on the login information of the first terminal, and to obtain first communication data that complies with the visual network audio and video instant communication protocol based on the login information of the second terminal.
[0162] Optionally, the device 900 further includes:
[0163] A first releasing module, configured to release the original communication connection with the first terminal after receiving a message from the first terminal indicating that the first communication connection is successfully established;
[0164] The second releasing module is configured to release the original communication connection with the second terminal after receiving a message from the second terminal indicating that the second communication connection is successfully established.
[0165] Based on the same inventive concept, another embodiment of the present application provides an electronic device 1000, such as Fig.10 shown. Fig.101 is a schematic diagram of an electronic device shown in an embodiment of the present application. The electronic device includes a memory 1002, a processor 1001, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method described in any of the above embodiments of the present application when executing the computer program.
[0166] Based on the same inventive concept, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor is executed, the steps in the audio and video data sending method described in any of the above embodiments of the present application are implemented.
[0167] Based on the same inventive concept, the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the audio and video data sending method described in any of the above embodiments of the present application are implemented.
[0168] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0169] The visual Internet is an important milestone in the development of the Internet. It is a real-time network that can realize real-time transmission of high-definition video and push many Internet applications towards high-definition video and high-definition face-to-face communication.
[0170] The visual network adopts real-time high-definition video exchange technology. It can integrate the required services such as high-definition video conferencing, video monitoring, intelligent monitoring analysis, emergency command, digital radio and television, delayed TV, online teaching, live broadcast, VOD on demand, TV mail, personalized recording (PVR), intranet (self-operated) channels, intelligent video broadcast control, information release and dozens of other video, voice, picture, text, communication, data and other services on a single system platform, and realize high-definition quality video playback through TV or computer.
[0171] In order to enable those skilled in the art to better understand the embodiments of the present invention, the visual network is introduced as follows:
[0172] Some of the technologies used in the visual Internet of Things are as follows:
[0173] Network Technology
[0174] The network technology innovation of visual networking improves the traditional Ethernet to cope with the huge potential video traffic on the network. Different from simple network packet switching or network circuit switching, visual networking technology uses packet switching to meet streaming needs. Visual networking technology has the flexibility, simplicity and low price of packet switching, and the quality and security guarantee of circuit switching, realizing the seamless connection of the whole network switching virtual circuit and data format.
[0175] Switching Technology
[0176] Visual networking uses the two advantages of Ethernet: asynchronism and packet switching. It eliminates Ethernet defects under the premise of full compatibility, and has seamless end-to-end connection across the entire network, directly connects to user terminals, and directly carries IP data packets. User data does not require any format conversion within the entire network. Visual networking is a more advanced form of Ethernet and a real-time switching platform that can achieve large-scale real-time transmission of high-definition video across the entire network, which is currently unattainable on the Internet, and pushes many network video applications towards high definition and unification.
[0177] Server Technology
[0178] The server technology on the visual network and unified video platform is different from the traditional server. Its streaming media transmission is based on connection-oriented. Its data processing capability is independent of traffic and communication time. A single network layer can include signaling and data transmission. For voice and video services, the complexity of streaming media processing on the visual network and unified video platform is much simpler than data processing, and its efficiency is more than 100 times higher than that of traditional servers.
[0179] Storage Technology
[0180] The ultra-high-speed storage technology of the unified video platform adopts the most advanced real-time operating system to adapt to the media content with ultra-large capacity and ultra-large flow, mapping the program information in the server command to the specific hard disk space. The media content no longer passes through the server, but is directly delivered to the user terminal in an instant. The user generally waits for less than 0.2 seconds. The optimized sector distribution greatly reduces the mechanical movement of the hard disk head seeking, and the resource consumption accounts for only 20% of the same level of IP Internet, but it generates 3 times more concurrent traffic than the traditional hard disk array, and the overall efficiency is improved by more than 10 times.
[0181] Network Security Technology
[0182] The structural design of the visual network completely eradicates the network security issues that plague the Internet through separate licensing for each service and complete isolation of equipment and user data. Generally, anti-virus programs and firewalls are not required, and attacks by hackers and viruses are eliminated, providing users with a structurally worry-free and secure network.
[0183] Service Innovation Technology
[0184] The unified video platform integrates services and transmission. Whether it is a single user, a private network user or a network, it is just an automatic connection. User terminals, set-top boxes or PCs are directly connected to the unified video platform to obtain rich and colorful multimedia video services in various forms. The unified video platform uses a "recipe-style" table configuration mode to replace traditional complex application programming. Complex applications can be implemented with very little code, realizing "unlimited" new business innovation.
[0185] The networking of the visual network is as follows:
[0186] Visual networking is a centrally controlled network structure. The network can be a tree network, star network, ring network, etc., but on this basis, there needs to be a centralized control node in the network to control the entire network.
[0187] Fig.11 FIG. 1 is a schematic diagram of a visual network shown in an embodiment of the present application. Fig.11 As shown in the figure, the visual network is divided into two parts: access network and metropolitan area network.
[0188] The equipment in the access network can be divided into three categories: node server, access switch, and terminal (including various set-top boxes, encoding boards, storage, etc.). The node server is connected to the access switch, and the access switch can be connected to multiple terminals and can be connected to Ethernet.
[0189] The node server is a node in the access network that performs centralized control functions and can control access switches and terminals. The node server can be directly connected to the access switch or directly connected to the terminal.
[0190] Similarly, the devices in the metropolitan area network can also be divided into three categories: metropolitan area servers, node switches, and node servers. The metropolitan area servers are connected to the node switches, and the node switches can be connected to multiple node servers.
[0191] The node server is a node server of the access network part, that is, the node server belongs to both the access network part and the metropolitan area network part.
[0192] The metropolitan area server is a node with centralized control function in the metropolitan area network, which can control the node switch and node server. The metropolitan area server can be directly connected to the node switch or the node server.
[0193] It can be seen from this that the entire visual network is a hierarchical and centrally controlled network structure, and the network controlled by the node server and the metropolitan area server can be various structures such as tree, star, and ring.
[0194] Figuratively speaking, the access network part can form a unified video platform (the part in the dotted circle), and multiple unified video platforms can form a visual network; each unified video platform can be interconnected through the metropolitan area and wide area visual networks.
[0195] Visual networking equipment classification
[0196] 1.1 The devices in the visual network of the embodiment of the present invention can be mainly divided into three categories: servers, switches (including Ethernet gateways), terminals (including various set-top boxes, encoding boards, storage devices, etc.). The visual network as a whole can be divided into a metropolitan area network (or a national network, a global network, etc.) and an access network.
[0197] 1.2 The equipment in the access network can be mainly divided into three categories: node server, access switch (including Ethernet gateway), terminal (including various set-top boxes, encoding boards, storage, etc.).
[0198] The specific hardware structure of each access network device is as follows:
[0199] Node server:
[0200] Fig.12 FIG. 1 is a schematic diagram of the hardware structure of a node server shown in an embodiment of the present application. Fig.12 As shown, it mainly includes a network interface module 201, a switching engine module 202, a CPU module 203, and a disk array module 204;
[0201] Among them, the packets coming from the network interface module 201, the CPU module 203, and the disk array module 204 all enter the switching engine module 202; the switching engine module 202 performs an operation of checking the address table 205 on the incoming packets, thereby obtaining the packet's guidance information; and according to the packet's guidance information, the packet is stored in the queue of the corresponding packet buffer 206; if the queue of the packet buffer 206 is close to full, it is discarded; the switching engine module 202 polls all packet buffer queues, and forwards if the following conditions are met: 1) the port sending buffer is not full; 2) the queue packet counter is greater than zero. The disk array module 204 mainly implements the control of the hard disk, including the initialization, reading and writing of the hard disk; the CPU module 203 is mainly responsible for the protocol processing between the access switch and the terminal (not shown in the figure), the configuration of the address table 205 (including the downstream protocol packet address table, the upstream protocol packet address table, and the data packet address table), and the configuration of the disk array module 204.
[0202] Access switch:
[0203] Fig.13 FIG. 1 is a schematic diagram of the hardware structure of an access switch shown in an embodiment of the present application. Fig.13 As shown, it mainly includes a network interface module (a downlink network interface module 301, an uplink network interface module 302), a switching engine module 303 and a CPU module 304;
[0204] Among them, the packet (uplink data) coming from the downlink network interface module 301 enters the packet detection module 305; the packet detection module 305 detects whether the destination address (DA), source address (SA), data packet type and packet length of the packet meet the requirements. If they meet the requirements, the corresponding stream identifier (stream-id) is allocated and enters the switching engine module 303, otherwise it is discarded; the packet (downlink data) coming from the uplink network interface module 302 enters the switching engine module 303; the data packet coming from the CPU module 304 enters the switching engine module 303; the switching engine module 303 processes the incoming packet The address table 306 is looked up to obtain the packet's guidance information; if the packet entering the switching engine module 303 is from the downstream network interface to the upstream network interface, the packet is stored in the queue of the corresponding packet buffer 307 in combination with the stream identifier (stream-id); if the queue of the packet buffer 307 is nearly full, it is discarded; if the packet entering the switching engine module 303 is not from the downstream network interface to the upstream network interface, the data packet is stored in the queue of the corresponding packet buffer 307 according to the packet's guidance information; if the queue of the packet buffer 307 is nearly full, it is discarded.
[0205] The switching engine module 303 polls all packet buffer queues. In the embodiment of the present invention, there are two situations:
[0206] If the queue is from the downlink network interface to the uplink network interface, it is forwarded if the following conditions are met: 1) the port send buffer is not full; 2) the queue packet counter is greater than zero; 3) a token generated by the bit rate control module is obtained;
[0207] If the queue is not going from the downstream network interface to the upstream network interface, it is forwarded if the following conditions are met: 1) the port send buffer is not full; 2) the queue packet counter is greater than zero.
[0208] The code rate control module 308 is configured by the CPU module 304 to generate tokens for all packet buffer queues from the downstream network interface to the upstream network interface within a programmable interval to control the code rate of the upstream forwarding.
[0209] The CPU module 304 is mainly responsible for protocol processing with the node server, configuration of the address table 306 , and configuration of the bit rate control module 308 .
[0210] Ethernet protocol conversion gateway :
[0211] Fig.14 FIG. 1 is a schematic diagram of the hardware structure of an Ethernet protocol conversion gateway shown in an embodiment of the present application. Fig.14 As shown, it mainly includes a network interface module (a downlink network interface module 401, an uplink network interface module 402), a switching engine module 403, a CPU module 404, a packet detection module 405, a bit rate control module 408, an address table 406, a packet buffer 407 and a MAC adding module 409, and a MAC deleting module 410.
[0212] The data packet from the downlink network interface module 401 enters the packet detection module 405; the packet detection module 405 detects whether the Ethernet MAC DA, Ethernet MAC SA, Ethernet length or frame type, visual network destination address DA, visual network source address SA, visual network data packet type and packet length of the data packet meet the requirements, and if they meet the requirements, a corresponding stream identifier (stream-id) is assigned; then, the MAC deletion module 410 subtracts the MAC DA, MAC SA, length or frame type (2 bytes) and enters the corresponding receiving buffer, otherwise it is discarded;
[0213] The downlink network interface module 401 detects the sending buffer of the port. If there is a packet, it obtains the Ethernet MAC DA of the corresponding terminal based on the visual network destination address DA of the packet, adds the Ethernet MAC DA of the terminal, the MACSA of the Ethernet protocol conversion gateway, and the Ethernet length or frame type, and sends it.
[0214] The functions of other modules in the Ethernet protocol conversion gateway are similar to those of the access switch.
[0215] terminal:
[0216] It mainly includes network interface module, business processing module and CPU module; for example, the set-top box mainly includes network interface module, audio and video encoding and decoding engine module, CPU module; the encoding board mainly includes network interface module, audio and video encoding engine module, CPU module; the memory mainly includes network interface module, CPU module and disk array module.
[0217] 1.3 The equipment in the metropolitan area network can be divided into two categories: node server, node switch, and metropolitan area server. Among them, the node switch mainly includes network interface module, switching engine module and CPU module; the metropolitan area server mainly includes network interface module, switching engine module and CPU module.
[0218] 2. Definition of Visual Network Data Packet
[0219] 2.1 Access Network Data Packet Definition
[0220] The data packet of the access network mainly includes the following parts: destination address (DA), source address (SA), reserved bytes, payload (PDU), and CRC.
[0221] As shown in the following table, the data packet of the access network mainly includes the following parts:
[0222] DA S Reserved Payload CRC
[0223] in:
[0224] The destination address (DA) consists of 8 bytes. The first byte indicates the type of data packet (such as various protocol packets, multicast data packets, unicast data packets, etc.), with a maximum of 256 possibilities. The second to sixth bytes are the metropolitan area network address, and the seventh and eighth bytes are the access network address.
[0225] The source address (SA) is also composed of 8 bytes and has the same definition as the destination address (DA).
[0226] The reserved bytes consist of 2 bytes;
[0227] The payload part has different lengths depending on the type of datagram. If it is a packet of various protocols, it is 64 bytes. If it is a single multicast data packet, it is 32+1024=1056 bytes. Of course, it is not limited to the above two types.
[0228] CRC consists of 4 bytes and its calculation method follows the standard Ethernet CRC algorithm.
[0229] 2.2 Definition of Metropolitan Area Network Data Packet
[0230] The topology of the metropolitan area network is a graph. There may be two or more connections between two devices, that is, there may be more than two connections between a node switch and a node server, a node switch and a node switch, and a node switch and a node server. However, the metropolitan area network address of a metropolitan area network device is unique. In order to accurately describe the connection relationship between metropolitan area network devices, a parameter: a label is introduced in the embodiment of the present invention to uniquely describe a metropolitan area network device.
[0231] The definition of labels in this manual is similar to the definition of labels in MPLS (Multi-Protocol Label Switch). Assuming that there are two connections between device A and device B, there are two labels for the data packet from device A to device B, and there are two labels for the data packet from device B to device A. Labels are divided into input labels and output labels. Assuming that the label of the data packet entering device A (input label) is 0x0000, the label of the data packet when leaving device A (output label) may become 0x0001. The access process of the metropolitan area network is a centralized access process, which means that the address allocation and label allocation of the metropolitan area network are dominated by the metropolitan area server, and the node switch and node server are only passively executed. This is different from the label allocation of MPLS, which is the result of mutual negotiation between the switch and the server.
[0232] As shown in the following table, the data packets of the metropolitan area network mainly include the following parts:
[0233] DA S Reserved Label Payload CRC
[0234] That is, destination address (DA), source address (SA), reserved byte (Reserved), label, payload (PDU), CRC. The label format can refer to the following definition: the label is 32 bits, of which the upper 16 bits are reserved and only the lower 16 bits are used. It is located between the reserved bytes and payload of the data packet.
[0235] Based on the above-mentioned characteristics of the visual network, one of the core concepts of an embodiment of the present invention is proposed. In accordance with the protocol of the visual network, when a first terminal requests to establish a communication connection with a second terminal, the communication data corresponding to the second terminal is first obtained through the streaming media server, and then a request to establish an end-to-end communication connection is directly initiated to the second terminal based on the communication data corresponding to the second terminal. After the establishment is successful, the audio and video data are transmitted to the second terminal through the end-to-end communication connection without passing through the streaming media server.
[0236] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0237] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0238] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0239] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0240] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0241] The above is a detailed introduction to an audio and video data sending method, device, electronic device and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A method for sending audio and video data, It is characterized in that Applied to a first terminal, the method includes: Sending a communication connection request to a streaming media server that is in communication connection with the first terminal, wherein the communication connection request is used to request to establish a communication connection with the second terminal; receiving first communication data sent by the streaming media server, where the first communication data is communication data corresponding to the second terminal; Establishing a first communication connection for end-to-end communication with the second terminal based on the first communication data; Based on the first communication connection, sending audio and video data to the second terminal; After establishing a first communication connection for end-to-end communication with the second terminal, the method further includes: Sending a message that the first communication connection is successfully established to the streaming media server, so that the streaming media server releases the communication connection with the first terminal; Among them, at least one of the first terminal and the second terminal is deployed in the visual network; when one of the first terminal and the second terminal is deployed in the Internet, the first communication connection is established through a protocol conversion server.
2. The method according to claim 1, It is characterized in that The method further comprises: Upon receiving the second communication connection request sent by the second terminal, returning a response indicating approval of the second communication connection request to the second terminal, so as to establish a second communication connection for end-to-end communication with the second terminal; Based on the second communication connection, receiving the audio and video data sent by the second terminal; The second communication connection request is sent by the second terminal to the first terminal according to second communication data, and the second communication data is sent by the streaming media server to the second terminal when receiving the first communication connection request.
3. The method according to claim 1, It is characterized in that Sending audio and video data to the second terminal based on the first communication connection includes: Obtaining encoding and decoding parameters of the second terminal from the first communication data; Processing the collected audio and video data according to the encoding and decoding parameters; Based on the first communication connection, the processed audio and video data is sent to the second terminal.
4. The method according to claim 1, It is characterized in that Establishing a first communication connection for end-to-end communication with the second terminal based on the first communication data includes: Obtaining address information of the second terminal from the first communication data; A communication connection request is sent to the second terminal according to the address information to establish a first communication connection for end-to-end communication with the second terminal.
5. A method for transmitting audio and video data, It is characterized in that Applied to a streaming media server, the method comprises: Receiving a communication connection request sent by a first terminal in communication connection with the streaming media server, the communication connection request being used to request establishment of a communication connection between the first terminal and a second terminal; Sending first communication data corresponding to the second terminal to the first terminal, so that the first terminal establishes a first communication connection for end-to-end communication with the second terminal according to the first communication data, and sends audio and video data to the second terminal through the first communication connection; The method further comprises: After receiving a message from the first terminal indicating that the first communication connection is successfully established, releasing the original communication connection with the first terminal; Among them, at least one of the first terminal and the second terminal is deployed in the visual network; when one of the first terminal and the second terminal is deployed in the Internet, the first communication connection is established through a protocol conversion server.
6. The method according to claim 5, It is characterized in that The method further comprises: In response to a communication connection request sent by the first terminal, second communication data corresponding to the first terminal is sent to the second terminal, so that the second terminal establishes a second communication connection with the first terminal for end-to-end communication based on the second communication data, and receives the audio and video data sent by the first terminal through the second communication connection.
7. The method according to claim 6, It is characterized in that Before receiving a communication connection request sent by a first terminal that is in communication connection with the streaming media server, the method further includes: Obtaining login information of the first terminal and login information of the second terminal; Based on the login information of the first terminal, second communication data that complies with the visual network audio and video instant communication protocol is obtained, and based on the login information of the second terminal, first communication data that complies with the visual network audio and video instant communication protocol is obtained.
8. The method according to claim 6, It is characterized in that The method further comprises: After receiving a message from the second terminal indicating that the second communication connection is successfully established, the original communication connection with the second terminal is released.
9. An audio and video data transmitting device, It is characterized in that Applied to a first terminal, the device includes: A first sending module, used for sending a communication connection request to a streaming media server connected to the first terminal in communication, wherein the communication connection request is used for requesting to establish a communication connection with the second terminal; A first receiving module, configured to receive first communication data sent by the streaming media server, wherein the first communication data is communication data corresponding to the second terminal; A first establishing module, configured to establish a first communication connection for end-to-end communication with the second terminal based on the first communication data; A second sending module, configured to send audio and video data to the second terminal based on the first communication connection; The device also includes: A third sending module, configured to send a message that the first communication connection is successfully established to the streaming media server, so that the streaming media server releases the communication connection with the first terminal; Among them, at least one of the first terminal and the second terminal is deployed in the visual network; when one of the first terminal and the second terminal is deployed in the Internet, the first communication connection is established through a protocol conversion server.
10. An audio and video data transmitting device, It is characterized in that Applied to a streaming media server, the device comprises: A third receiving module, used to receive a communication connection request sent by a first terminal that is in communication connection with the streaming media server, wherein the communication connection request is used to request to establish a communication connection between the first terminal and a second terminal; a fourth sending module, configured to send first communication data corresponding to the second terminal to the first terminal, so that the first terminal establishes a first communication connection for end-to-end communication with the second terminal according to the first communication data, and sends audio and video data to the second terminal through the first communication connection; The device also includes: A first releasing module, configured to release the original communication connection with the first terminal after receiving a message from the first terminal indicating that the first communication connection is successfully established; Among them, at least one of the first terminal and the second terminal is deployed in the visual network; when one of the first terminal and the second terminal is deployed in the Internet, the first communication connection is established through a protocol conversion server.
11. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, it implements the steps in the audio and video data sending method as described in any one of claims 1 to 4, or implements the steps in any one of claims 5 to 8.
12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When executed, the processor implements the steps of the audio and video data sending method as described in any one of claims 1-4, or implements the steps of any one of claims 5-8.
Citation Information
Patent Citations
Method and apparatus for establishing communication channel
CN101364976A
A communication connection establishment method and system based on an articulated naturality web
CN109743526A
Terminal control method and device, electronic equipment and storage medium
CN110248134A
Monitoring video processing method and device and storage medium
CN110392233A
Communication method, device and system, terminal, equipment and medium
CN110691110A
