Live audio switching method and stream media server

[0021] In order to make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

[0022] The embodiments of the present invention provide a live audio switching method and a streaming media server, which can be applied to a web live broadcast scene. When people watch a live web program, especially a sports event, all viewers can only share the same host in front of the screen However, viewers of different genders, different ages, and different regions have different preferences. Different viewers have their own positions on the same live program. The viewers naturally like to hear the commentary of the anchor who is consistent with their position. If, according to the existing solution, you can only switch between different webcast providers, it will not only waste time and traffic, but also may miss exciting game moments during the switching buffer process. The embodiment of the present invention provides a live audio switching method and a streaming media server to solve the problem that users cannot independently select live audio in the same live program. The live video stream and multiple live audio streams are collected in a distributed manner through the network. Each host Corresponding to a live audio stream, the streaming media server receives the collected live video stream and live audio stream and mixes them to generate a live media stream, so that a live media stream contains live video and multiple anchors corresponding to the live video Live audio. The live media stream will be sent directly to the client, and the user can select a favorite anchor in the client according to his own preferences, and the client can extract the corresponding audio data of the anchor through the program map table and play it. In this way, the user can independently select live audio in the same live program, and can switch the live audio, greatly improving the user experience.

[0023] It should be noted that the live audio switching method and streaming media server provided in the embodiments of the present invention can also be applied to other scenarios, which are not limited here.

[0024] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] Such as figure 1 As shown, the embodiment of the present invention provides a live audio switching method, including:

[0026] S101. The live video stream and the live audio stream received in real time are respectively encoded and divided into TS slices, a header file is added to each TS slice, and the PID of the TS slice is configured in the header file, where corresponding to TS slices of the same live audio stream have the same PID, TS slices corresponding to the same live video stream have the same PID, and TS slices corresponding to different live audio streams or live video streams have different PIDs;

[0027] S102. Insert the TS slice into a TS data packet with the same PID as the TS slice, and mix the TS data packet to generate a TS media stream;

[0028] S103: Establish a program mapping table, and add the corresponding relationship between the description information encoding of the live audio stream and the PID of the TS slice corresponding to the live audio stream into the program mapping table;

[0029] S104: Send the program mapping table and the TS media stream to the client.

[0030] The process of processing the live audio stream and the live video stream received by the streaming media server is as described in step S101. The audio stream and video stream transmitted by the audio collection terminal and the video collection terminal are non-segmented continuous streams. In order to realize the mixing of multiple continuous streams, the live audio stream and the live video stream are sliced ​​in step S101 to Obtain multiple small TS slices, and in step S102, the multiple TS slices are packaged and combined to form a TS media stream. The generated TS media stream will be sent to the client through the CDN network and based on the HTTP Live Streaming (HLS) streaming media transmission protocol. In order to ensure that the client can extract the TS slices related to the single-channel live audio stream from the TS media stream for playback, step S101 also adds a header file for each TS slice. The header file contains the data type (audio, video or other ) And PID (the ID of the TS packet). The PID in the TS slice header file is consistent with the PID of the TS packet it will be inserted into; moreover, in order to distinguish TS slices related to each live audio stream or live video stream, In step S101, TS slices corresponding to the same live audio stream are assigned the same PID, TS slices corresponding to the same live video stream are assigned the same PID, and TS slices corresponding to different live audio streams or live video streams are assigned the same PID. Each is different. The TS media stream is composed of TS data packets. In step S102, the TS slices related to the live video stream and the live audio stream are inserted into the corresponding TS data packets according to the PID according to the rules to realize the mixing of multiple streams.

[0031] According to the processing of the live video stream and the live audio stream in the above steps S101 and S102, one TS media stream can be obtained. In order to ensure that the client can realize the free switching of the live audio during the process of playing the TS media stream, the establishment in step S103 A program mapping table, the program mapping table will be sent to the client together with the TS media stream in step S104, the client reads the program mapping table through parsing, and can be based on the switching operation performed by the user in the client , Find the PID of the corresponding TS slice from the program map table, and then directly extract the TS slice from the corresponding TS data packet to switch and play.

[0032] In this embodiment, the mixing of live video and multiple channels of live audio is completed before the streaming media server sends the live file to the client, and audio switching is performed for the client through processing such as the program map table and the PID of the TS slice The data basis and switching logic basis are prepared, and the free switching of multiple channels of live audio in the same live program can be realized by using the existing client.

[0033] reference figure 2 In the following, another embodiment is used to describe in detail the solution of receiving a live audio stream in real time.

[0034] Online live broadcasts such as sports events have relatively high requirements for the real-time performance of the program, but because each audio collection terminal is distributed in different locations, this may cause the network delays of each live audio stream to be transmitted to the streaming media server to be different. After the TS media streams obtained in steps S101-104 are sent to the client, the user may experience audio and video out of synchronization when switching live audio.

[0035] In order to achieve synchronous switching of live audio, in this embodiment, before step S101, the method further includes:

[0036] S1011, establish VPN, connect multiple audio collection terminals through VPN channel;

[0037] S1012: Acquire live audio streams collected by each audio collection terminal in real time according to RTP/RTCP.

[0038] VPN (Virtual Private Network, virtual private network), the function of a virtual private network is to establish a private network on a public network and perform encrypted communication. In step S1011, the audio collection terminal and the video collection terminal can be set up in different locations according to the needs of different anchors, and the collected live audio stream and live video stream will be sent to the streaming media server for processing. The audio collection terminal can be added to the VPN of the streaming media server through the client, etc. In step S1012, each audio collection terminal establishes the RTP/RTCP media transmission and control path with the streaming media server. There are many ways to establish a connection between the two. It can be through RTSP or VOIP. After the connection is established, the live audio stream can be transmitted to the streaming media server through RTP, and the RTP transmission process can be controlled through RTCP, such as flow control and congestion control. RTP and RTCP can be used together to effectively The feedback and minimal overhead optimize the transmission efficiency, so it is particularly suitable for transmitting real-time data on the network.

[0039] Among them, RTP (RealTime Transport Protocol) is the most effective way of real-time transmission. RTP uses UDP as the transmission channel, but there are two problems: a. Cannot traverse firewalls and NAT; b. RTP is an unreliable protocol, sometimes Packet loss. In the embodiment of the present invention, the problem that RTP cannot traverse the firewall is effectively solved through VPN; the transmission process of RTP is controlled through RTCP, which effectively solves the problem of RTP transmission packet loss.

[0040] In this embodiment, in order to ensure that the VPN of the streaming media server is connected to each audio collection terminal safely and quickly, preferably, step S1011 includes:

[0041] Before the live broadcast starts, actively add the audio collection terminal to the VPN according to the known network address of the audio collection terminal; and

[0042] During the live broadcast, the network connection request initiated by the audio collection terminal is monitored in real time, and when the network connection request of the audio collection terminal is monitored, the audio collection terminal is added to the VPN in real time after security verification.

[0043] For a fixed anchor, a fixed network address can be assigned to its audio collection terminal. In a predetermined time or scenario, the streaming media server can actively add the audio collection terminal to the VPN. For non-fixed anchors, this embodiment can also realize that the audio collection terminal dynamically joins the VPN in real time. According to a request independently initiated by the audio collection terminal, the streaming media server can perform security verification on it and allow it to join the VPN after confirming its identity. In addition, because the data volume of audio files is relatively small, the network requirements are not too high. It can be connected to the streaming media server through the exclusive wireless network of the stadium, so that the anchor can be in any corner of the stadium, such as the fan area of ​​the major camps. Collect the environmental sounds of the fans in this camp and cooperate with the commentary with standpoints to get the attention of different users.

[0044] Hereinafter, step S103 will be described in detail with an embodiment.

[0045] TS media stream is composed of TS data packets, each TS data packet is composed of Packet Header (packet header) and Packet Data (packet data), among which Packet Header indicates the attributes of the TS data packet, and gives the Packet Data The unique network identifier PID of the data. The program mapping table in step S103 points to the PID in the Packet Header, and the program mapping table as a kind of index file data can also be inserted into the TS data packet. In this embodiment, step S103 inserts the program map table into the dedicated TS packet, and configures the PID in the Packet Header of the dedicated TS packet to a fixed public value, for example, the PID of the dedicated TS packet It is configured as 0x0000. At the same time, in order to realize the specialization of the TS data packet, the blank bytes in the special TS data packet are filled by the adaptation field. After receiving the TS media stream, the client can find the TS data packet where the program map is located according to the fixed public PID and read the program map.

[0046] In this embodiment, the program map table is inserted into the dedicated TS data packet in the TS media stream, and there is no need to send it separately, which realizes the integration of data and avoids the time that the program map table is sent to the client later than the TS due to network delay. The media stream causes the client to wait, and can avoid the data loss of the program map.

[0047] The basis for switching live audio in the client is the description information of the live audio stream. The description information includes the host’s avatar, style introduction, position, popularity, audio track, etc. Preferably, the live broadcast can be received in real time In the case of audio streaming, the corresponding host is determined according to the network address and other information related to the live audio stream, and the description information file is generated and sent to the client for display. At the same time, a unique description information encoding is configured for each description information file , The encoding corresponds to the description information encoding of each live audio stream in the program map table. Of course, in the present invention, the description information and description information encoding of the live audio stream can also be directly obtained from each audio collection terminal, which is not limited in the present invention.

[0048] In order to make full use of the functions of the existing client and improve the convenience of user operations, in another embodiment of the present invention, in step S103, the corresponding relationship between the encoding of the description information of the live audio stream and the audio track of the client can be added to To the program map table.

[0049] When the client switches the audio track, it can find the corresponding description information encoding according to the corresponding relationship between the audio track in the program map table and the description information encoding of the live audio stream, and then encode and live broadcast according to the description information in the program map table The corresponding relationship of the PID of the TS slice corresponding to the audio stream is extracted, and the TS slice is extracted for switching playback.

[0050] Another embodiment of the present invention proposes a technical solution that can switch multiple live video streams while switching live audio. The details are as follows.

[0051] Users always want to see more details of the scenes they are interested in from the live broadcast or pay attention to their favorite character activities. For example, in a sports game, different users like different players. At the match point, users prefer the shooting perspective. Target your favorite players to observe their performance.

[0052] Based on steps S101 to 103, if the live video stream received in step S101 is multiple, then in step S103 the PID of the TS slice of the live video stream is bound with the PID of the TS slice of the live audio stream The play relationship is configured in the program mapping table.

[0053] Each anchor sends a live audio stream to the streaming server, and at the same time, can also send a live video stream to the streaming server. In this way, each anchor can shoot live video and explain the game process from the perspective of his own attention according to his own position. In this embodiment, the live video stream and the live audio stream provided by each anchor are bound, and the binding playback relationship is configured in the program map table. When the client reads the program map table, it can search At the same time as the live audio stream, the information of the live video stream that has a binding play relationship with the live audio stream is obtained.

[0054] In order to avoid synchronous switching to a live video with a binding playback relationship when the user switches audio in this embodiment, the user's autonomous expectations are ignored.

[0055] Preferably, in the program map table, the priority of the correspondence between the description information encoding of the live audio stream in the program map table and the PID of the TS slice corresponding to the live audio stream is set higher than the priority The priority of the binding play relationship between the PID of the TS slice of the live video stream and the PID of the TS slice of the live audio stream, and when the client switches to the live audio, whether to switch to the same The query information of the live video to which the live audio has a binding relationship.

[0056] In this way, according to the user's switching operation, only the live audio will be switched, and the user will be provided with the right to choose independently as to whether to switch the live video, and the user's experience will be fully considered.

[0057] The following takes the webcast including one live audio stream and two live audio streams as an example for detailed description of the embodiment.

[0058] Two audio collection terminals distributed in different locations can respectively transmit their live audio streams to the streaming media server via VPN and according to RTP/RTCP. Here, the live video stream is officially transmitted as an example. Of course, the live video stream can also be captured by video Terminal acquisition. The streaming media server slices 1 live video stream and 2 live audio streams separately to obtain TS slices, inserts the TS slices corresponding to the live video stream into the TS data packet with PID 10, and divides the 2 live audio streams corresponding to TS slices are inserted into TS data packets with PID 20 and 30 respectively, and the description information encoding of the live video stream is configured as 01, and the description information encoding of the 2 live audio streams is configured as 02 and 03 respectively. In the program map table, the description The correspondence between information code and PID 01-10, 02-20, 03-30 will be recorded. The program map table can contain the following information:

[0059] Stream_type 0x0f//where 0f represents audio

[0060] Elementary_PID 10//The audio data of this channel is stored in the TS packet with PID 10

[0061] Stream_type 0x0f

[0062] Elementary_PID 20//The audio data of this channel is stored in the TS packet with PID 20

[0063] Stream_type 0x1b//where 1b represents the video

[0064] Elementary_PID 30////The video data of this channel is stored in the TS packet with PID 30

[0065] When the user switches the anchor according to the description information displayed in the client, the client can find the PID of the corresponding TS slice from the program map according to the encoding of the selected description information, and then find the corresponding TS data packet in the TS media stream Extract the corresponding TS slice to switch and play.

[0066] reference image 3 , The embodiment of the present invention also provides a streaming media server, including:

[0067] The data acquisition module 11 is used to receive live video streams and multiple live audio streams in real time;

[0068] The TS slice processing module 12 is configured to separately encode and divide the live video stream and the live audio stream into TS slices, add a header file to each TS slice, and configure the PID of the TS slice in the header file , Where TS slices corresponding to the same live audio stream have the same PID, TS slices corresponding to the same live video stream have the same PID, and TS slices corresponding to different live audio streams or live video streams have the same PID. Not the same;

[0069] The data mixing module 13 is used to insert the TS slice into a TS data packet with the same PID, and mix the TS data packet to generate a TS media stream;

[0070] The index module 14 is configured to establish a program mapping table, and add the correspondence relationship between the description information encoding of the live audio stream and the PID of the TS slice corresponding to the live audio stream into the program mapping table;

[0071] The data sending module 15 is used to send the program mapping table and the TS media stream to the client.

[0072] Among them, in the data acquisition module 11, the live video stream and the live audio stream can be collected in a distributed manner on the network, and the audio collection terminal and the video collection terminal can be set up in different locations according to the needs of different anchors, and the collected live audio Streaming and live video streams will be sent to the streaming server for processing.

[0073] The processing of the live audio stream and the live video stream received by the streaming media server is completed by the TS slice processing module 12. The audio and video streams transmitted by the audio collection terminal and the video collection terminal are non-segmented continuous streams. In order to realize the mixing of multiple continuous streams, the TS slice processing module 12 slices the live audio stream and the live video stream. Processing to obtain multiple small TS slices, the data mixing module 13 packs and composites multiple TS slices to form a TS media stream, and the generated TS media stream is passed through the CDN network by the data sending module 15 and is based on HTTP Live Streaming (HLS) The streaming media transmission protocol is sent to the client. In order to ensure that the client can extract TS slices related to a single live audio stream from the TS media stream for playback, the TS slice processing module 12 also adds a header file for each TS slice, and the header file contains data types (audio, Video or other) and PID (the ID of the TS packet). The PID in the TS slice header file is consistent with the PID of the TS packet that will be inserted into it. In addition, in order to distinguish between the live audio streams or live video streams related TS slice, the TS slice processing module 12 also assigns the same PID to the TS slices corresponding to the same live audio stream, and the TS slices corresponding to the same live video stream to assign the same PID, corresponding to different live audio streams or live videos The PIDs of TS slices of the stream are different. The TS media stream is composed of TS data packets, and the TS slices related to the live video stream and the live audio stream in the data mixing module 13 are inserted into the corresponding TS data packets according to the rules according to the PID to realize the mixing of multiple streams.

[0074] According to the processing of the live video stream and the live audio stream by the TS slice processing module 12 and the data mixing module 13 above, one TS media stream can be obtained. In order to ensure that the client can realize the freedom of the live audio during the process of playing the TS media stream Switching, a program mapping table is established in the index module 14. The program mapping table will be sent to the client together with the TS media stream in the data sending module 15. The client reads the program mapping table through parsing, and can According to the switching operation performed by the user in the client, the PID of the corresponding TS slice is found from the program map table, and then the TS slice is directly extracted from the corresponding TS packet for switching and playing.

[0075] In this embodiment, the mixing of live video and multiple channels of live audio is completed before the streaming media server sends the live file to the client, and audio switching is performed for the client through processing such as the program map table and the PID of the TS slice The data basis and switching logic basis are prepared, and the free switching of multiple channels of live audio in the same live program can be realized by using the existing client.

[0076] The data acquisition module 11 will be described in detail in an embodiment below.

[0077] Online live broadcasts such as sports events have relatively high requirements for the real-time performance of the program, but because each audio collection terminal is located in different locations, this may cause the network delays of each live audio stream to be transmitted to the streaming media server to be different, so , After the TS media stream is sent to the client, the user may experience audio and video synchronization issues when switching live audio.

[0078] In order to achieve synchronous switching of live audio, in this embodiment, the data acquisition module 11 is further used for:

[0079] Establish VPN and connect multiple audio collection terminals through VPN channels;

[0080] Acquire live audio streams collected by various audio collection terminals in real time according to RTP/RTCP.

[0081] In the data acquisition module 11, audio collection terminals can be added to the VPN of the streaming media server through a client, etc., each audio collection terminal and the streaming media server establish RTP/RTCP media transmission and control channels, and there are many ways to establish a connection between the two There are many kinds of methods, which can be through RTSP or VOIP. After the connection is established, the live audio stream can be transmitted to the streaming media server through RTP, and the RTP transmission process can be controlled through RTCP, such as flow control and congestion control. RTP and RTCP are used together. It can optimize the transmission efficiency with effective feedback and minimal overhead, so it is particularly suitable for transmitting real-time data on the network.

[0082] Among them, RTP (RealTime Transport Protocol) is the most effective way of real-time transmission. RTP uses UDP as the transmission channel, but there are two problems: a. Cannot traverse firewalls and NAT; b. RTP is an unreliable protocol, sometimes Packet loss. In the embodiment of the present invention, the problem that RTP cannot traverse the firewall is effectively solved through VPN; the transmission process of RTP is controlled through RTCP, which effectively solves the problem of RTP transmission packet loss.

[0083] In this embodiment, in order to ensure that the VPN of the streaming media server is connected to each audio collection terminal safely and quickly, preferably, the data acquisition module 11 is also used for:

[0084] Before the live broadcast starts, actively add the audio collection terminal to the VPN according to the known network address of the audio collection terminal; and

[0085] During the live broadcast, the network connection request initiated by the audio collection terminal is monitored in real time, and when the network connection request of the audio collection terminal is monitored, the audio collection terminal is added to the VPN in real time after security verification.

[0086] For a fixed anchor, a fixed network address can be assigned to its audio collection terminal. In a predetermined time or scenario, the streaming media server can actively add the audio collection terminal to the VPN. For non-fixed anchors, this embodiment can also realize that the audio collection terminal dynamically joins the VPN in real time. According to a request independently initiated by the audio collection terminal, the streaming media server can perform security verification on it and allow it to join the VPN after confirming its identity.

[0087] The index module 14 will be described in detail in an embodiment below.

[0088] The program mapping table in the index module 14 points to the PID in the Packet Header of the TS data packet, and the program mapping table as a kind of index file data can also be inserted into the TS data packet. In this embodiment, the index module 14 inserts the program map table into a dedicated TS packet, and configures the PID in the Packet Header of the dedicated TS packet to a fixed public value, such as setting the value of the dedicated TS packet The PID is configured as 0x0000, and at the same time, in order to realize the specialization of the TS data packet, the blank bytes in the special TS data packet are filled with the adaptation field. After receiving the TS media stream, the client can find the TS data packet where the program map is located according to the fixed public PID and read the program map.

[0089] In this embodiment, the program map table is inserted into the dedicated TS data packet in the TS media stream, and there is no need to send it separately, which realizes the integration of data and avoids the time that the program map table is sent to the client later than the TS due to network delay. The media stream causes the client to wait, and can avoid the data loss of the program map.

[0090] The basis for switching live audio in the client is the description information of the live audio stream. The description information includes the host’s avatar, style introduction, position, popularity, audio track, etc., preferably in the data acquisition module 11 When acquiring the live audio stream, determine the corresponding host according to the network address and other information related to the live audio stream, and generate a description information file, and send it to the client for display. At the same time, the data acquisition module 11 also provides description information for each The file is configured with a unique description information encoding, which corresponds to the description information encoding of each live audio stream in the program map table. Of course, in the present invention, the description information and description information encoding of the live audio stream can also be directly obtained from each audio collection terminal, which is not limited in the present invention.

[0091] In order to make full use of the functions of the existing client and improve the convenience of user operations, in another embodiment of the present invention, the index module 14 may also encode the corresponding relationship between the description information of the live audio stream and the audio track of the client Add to the program map table.

[0092] When the client switches the audio track, it can find the corresponding description information encoding according to the corresponding relationship between the audio track in the program map table and the description information encoding of the live audio stream, and then encode and live broadcast according to the description information in the program map table The corresponding relationship of the PID of the TS slice corresponding to the audio stream is extracted, and the TS slice is extracted for switching playback.

[0093] In another embodiment of the present invention, a solution for switching multiple live video streams is proposed. The details are as follows.

[0094] Users always want to see more details of the scenes they are interested in from the live broadcast or pay attention to their favorite character activities. For example, in a sports game, different users like different players. At the match point, users prefer the shooting perspective. Target your favorite players to observe their performance.

[0095] If the live video stream received in the data acquisition module 11 is multi-channel, the index module 14 configures the binding play relationship between the PID of the TS slice of the live video stream and the PID of the TS slice of the live audio stream To the program map table.

[0096] Each anchor sends a live audio stream to the streaming server, and at the same time, can also send a live video stream to the streaming server. In this way, each anchor can shoot live video and explain the game process from the perspective of his own attention according to his own position. In this embodiment, the live video stream and the live audio stream provided by each anchor are bound, and the binding playback relationship is configured in the program map table. When the client reads the program map table, it can search At the same time as the live audio stream, the information of the live video stream that has a binding play relationship with the live audio stream is obtained.

[0097] In order to avoid synchronous switching to a live video with a binding playback relationship when the user switches audio in this embodiment, the user's autonomous expectations are ignored.

[0098] Preferably, in the program map table, the priority of the correspondence between the description information encoding of the live audio stream in the program map table and the PID of the TS slice corresponding to the live audio stream is set higher than the priority The priority of the binding play relationship between the PID of the TS slice of the live video stream and the PID of the TS slice of the live audio stream, and when the client switches to the live audio, whether to switch to the same The query information of the live video to which the live audio has a binding relationship.

[0099] In this way, according to the user's switching operation, only the live audio will be switched, and the user will be provided with the right to choose independently as to whether to switch the live video, and the user's experience will be fully considered.

[0100] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the objectives of the solutions of the embodiments. Those of ordinary skill in the art can understand and implement without creative work.

[0101] Through the description of the above implementation manners, those skilled in the art can clearly understand that each implementation manner can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic A disc, an optical disc, etc., include a number of instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiment.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions recorded in the foregoing embodiments are modified, or some of the technical features are equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.