Live relay method, live data receiving method, apparatus, device, and medium

Abstract

The application relates to a live relay method, a live data receiving method, a device, computer equipment and a storage medium. A live stream obtained from a live source is sent to a live client through a main path, live cache information is obtained according to image group information of live video frames cached from the live stream, frame indexes corresponding to the image group information are recorded, when a frame loss retransmission request of the live client is received through a backup path, corresponding image group information is obtained from the live cache information according to comparison between a frame index carried by the frame loss retransmission request and the recorded frame indexes, and the image group information is sent to the live client through the backup path. Compared with a traditional live relay method realized by switching a low code rate, the scheme simultaneously establishes the main path and the backup path, caches live video frames and frame indexes, and sends the cached image group information to the client through the backup path when the live broadcast is interrupted, so that the quality of a live picture during live relay is improved.

Description

Technical Field

[0001] This application relates to the field of live streaming technology, and in particular to a live streaming relay method, a live streaming data receiving method, an apparatus, a computer device, a storage medium, and a computer program product. Background Technology

[0002] In traditional internet video live streaming scenarios, the live stream is typically relayed through a cloud center. However, this increases latency, and when users experience packet loss, the media assets need to be re-acquired from the central server, further accelerating video quality degradation. To ensure the quality of the live stream for users, multi-path transmission carrying multi-bitrate data streams is usually used. When packet loss occurs on the user side, the system switches to a lower bitrate to ensure smooth live streaming. However, even using low bitrate switching for live stream relay still results in a decrease in video quality.

[0003] Therefore, current live streaming relay methods and live streaming data reception methods have the drawback of reducing image quality. Summary of the Invention

[0004] Therefore, it is necessary to provide a live streaming relay method, live streaming data receiving method, device, computer equipment, computer-readable storage medium, and computer program product that can improve the picture quality in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a live streaming relay method, the method comprising:

[0006] Establish connections between the main and backup paths with the live streaming client;

[0007] The system obtains the live stream from the live stream source, sends the live stream to the live stream client through the main path, obtains live stream cache information based on the image group information of the cached live video frames in the live stream, and records the frame index corresponding to the image group information.

[0008] When a frame loss retransmission request is received from the live streaming client through the backup path, the corresponding image group information is obtained from the live streaming cache information by comparing the frame index carried in the frame loss retransmission request with the recorded frame index.

[0009] The corresponding image group information is sent to the live streaming client via the backup route.

[0010] In one embodiment, obtaining the live stream from the live stream source includes:

[0011] Forward the live streaming request initiated by the live streaming client to the corresponding live streaming source;

[0012] Receive one live stream sent by the live stream source.

[0013] In one embodiment, the frame index corresponding to the recorded image group information includes:

[0014] Based on the frame packet header of the live stream on the main path, the timestamp and synchronization identifier of the live video frame are obtained;

[0015] The timestamps and synchronization identifiers of the live video frames are recorded as frame indices for the corresponding image group information.

[0016] In one embodiment, the step of obtaining the corresponding image group information from the live stream cache information by comparing the frame index carried in the frame loss retransmission request with the recorded frame index includes:

[0017] Based on the timestamp and synchronization identifier carried in the frame loss retransmission request, and by comparing them with the recorded timestamp and synchronization identifier, the image group information corresponding to the timestamp and synchronization identifier carried in the frame loss retransmission request is obtained from the live streaming cache information.

[0018] In one embodiment, establishing the connection between the primary and backup paths with the live streaming client includes:

[0019] Establish a connection with the live streaming client based on the Fast User Datagram Protocol (FAP) as the main path;

[0020] Establish a connection with the live streaming client based on the Hypertext Transfer Protocol as a backup path.

[0021] Secondly, this application provides a method for receiving live data, the method comprising:

[0022] Establish primary and backup connections with network devices;

[0023] The network device receives the live stream sent by the network device through the main path; the live stream is obtained by the network device from the live stream source.

[0024] When a frame drop is detected during live streaming, a frame drop retransmission request carrying the frame index is sent to the network device through the backup path;

[0025] The backup path is used to receive the image group information corresponding to the frame index sent by the network device;

[0026] The corresponding image group information is obtained by the network device from the live stream cache information by comparing the frame index carried in the frame loss retransmission request with the recorded frame index; the live stream cache information is obtained by the network device based on the image group information of the live stream cached live video frames obtained from the live stream source, and the recorded frame index is recorded by the network device when caching the image group information.

[0027] In one embodiment, when a live stream frame loss is detected, sending a frame loss retransmission request carrying a frame index to the network device through the backup path includes:

[0028] When the number of consecutive lost frames from the live stream from the main channel exceeds a threshold, a retransmission request carrying the timestamp and synchronization identifier is generated based on the timestamp and synchronization identifier corresponding to the lost live video frame.

[0029] The frame loss retransmission request is sent to the network device via the backup path.

[0030] Thirdly, this application provides a live broadcast relay device, the device comprising:

[0031] The first module is used to establish the connection between the main path and the backup path with the live streaming client;

[0032] The live streaming recording module is used to obtain the live stream from the live streaming source, send the live stream to the live streaming client through the main path, obtain live streaming cache information based on the image group information of the cached live video frames of the live stream, and record the frame index corresponding to the image group information.

[0033] The frame loss retransmission module is used to obtain the corresponding image group information from the live streaming cache information when it receives a frame loss retransmission request from the live streaming client through the backup path, based on the comparison between the frame index carried in the frame loss retransmission request and the recorded frame index.

[0034] The sending module is used to send the corresponding image group information to the live streaming client through the backup path.

[0035] Fourthly, this application provides a live data receiving device, the device comprising:

[0036] The second module is used to establish the connection between the primary and backup paths with network devices;

[0037] The live streaming module is used to receive the live stream sent by the network device through the main path; the live stream is obtained by the network device from the live streaming source.

[0038] The request module is used to send a retransmission request carrying a frame index to the network device through the backup path when a live frame loss is detected.

[0039] The receiving module is configured to receive image group information corresponding to the frame index sent by the network device via the backup path;

[0040] The corresponding image group information is obtained by the network device from the live stream cache information by comparing the frame index carried in the frame loss retransmission request with the recorded frame index; the live stream cache information is obtained by the network device based on the image group information of the live stream cached live video frames obtained from the live stream source, and the recorded frame index is recorded by the network device when caching the image group information.

[0041] Fifthly, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0042] Sixthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0043] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0044] The aforementioned live streaming relay method, live streaming data receiving method, apparatus, computer equipment, storage medium, and computer program product, through sending the live stream obtained from the live streaming source to the live streaming client via the main path, obtaining live streaming cache information based on the image group information of the cached live video frames, recording the frame index corresponding to the image group information, and when receiving a frame loss retransmission request from the live streaming client via the backup path, retrieving the corresponding image group information from the live streaming cache information by comparing the frame index carried in the frame loss retransmission request with the recorded frame index, and sending it to the live streaming client via the backup path. Compared to the traditional method of achieving live streaming relay by switching to a lower bitrate, this solution improves the quality of the live streaming image during live streaming relay by simultaneously establishing a main path and a backup path, caching live video frames and their frame indexes, and sending the cached image group information to the client via the backup path when the live stream is interrupted. Attached Figure Description

[0045] Figure 1 This is a diagram illustrating the application environment of a live relay method in one embodiment.

[0046] Figure 2 This is a flowchart illustrating a live relay method in one embodiment;

[0047] Figure 3 This is a schematic diagram of the structure of live streaming cache information in one embodiment;

[0048] Figure 4 This is a schematic diagram of a live data receiving method in one embodiment;

[0049] Figure 5This is a flowchart illustrating a live relay method in another embodiment;

[0050] Figure 6 This is a structural block diagram of a live broadcast relay device in one embodiment;

[0051] Figure 7 This is a structural block diagram of a live data receiving device in one embodiment;

[0052] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] The live streaming relay method and live streaming data receiving method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the live stream source connects to the network device via the main path, and the network device connects to the live stream client via the main path and backup path. The network device can include multiple modules. The live stream forwarding module sends the live stream to the live stream client via the main path. The network device buffers live video frames via the main path. When the live stream client experiences frame loss, it sends a frame loss retransmission request to the network device's stream scheduling module via the backup path. The network device then sends the buffered live video frames from the video frame buffering module to the live stream client.

[0055] In addition, in some embodiments, the aforementioned network devices can also be connected to other network devices. When network devices are connected in a multi-level cascade, each network device can obtain the live stream from its upstream network device or live stream source through the main path and forward it to the corresponding live stream client through the backup path. The live stream client can be, but is not limited to, various personal computers, laptops, smartphones, and tablets.

[0056] In one embodiment, such as Figure 2 As shown, a live streaming relay method and a live streaming data receiving method are provided, which are applied to... Figure 1 Taking network devices as an example, the explanation includes the following steps:

[0057] Step S202: Establish the connection between the main path and the backup path with the live streaming client.

[0058] The live streaming client can be the device terminal used by the user, who can trigger live streaming requests and watch live streams through the live streaming client. The live streaming client and network devices can establish a dual-path connection to achieve live stream forwarding and frame loss retransmission. This dual-path connection includes a primary path and a backup path; that is, the network device can establish a primary path connection and a backup path connection with the live streaming client.

[0059] The live streaming client can initiate a live streaming request to the network device, enabling the network device to establish a primary connection and a backup connection with the live streaming client upon receiving the request. These primary and backup connections can be established based on different protocols. For example, in one embodiment, the network device can establish a QUIC-UDP (Quick User Datagram Protocol Internet Connections) connection with the live streaming client as the primary connection. Alternatively, the network device can establish an HTTP (Hypertext Transfer Protocol) connection with the live streaming client as the backup connection. The live streaming request sent by the live streaming client can be a request for either QUIC-UDP or Hypertext Transfer Protocol.

[0060] Step S204: Obtain the live stream from the live stream source, send the live stream to the live stream client through the main path, obtain the live stream cache information based on the image group information of the cached live video frames, and record the frame index corresponding to the image group information.

[0061] In this process, a live streaming request initiated by a live streaming client can connect to the nearest network device. This network device can connect to a higher-level network device or directly to the live streaming source. The network device can then obtain the live stream from the higher-level network device or directly from the live streaming source. The live stream obtained by the network device from the live streaming source can be the live stream corresponding to the live streaming client's request. For example, if the live streaming request carries information about the corresponding live stream, the network device can obtain the corresponding live stream from the live streaming source based on that information.

[0062] After acquiring a live stream, the network device can send the live stream to the live streaming client via the main path while maintaining a connection with the backup path of the live streaming client. For example, the network device can quickly forward the live stream to the live streaming client via the main path, thereby improving the playback speed of the live stream on the live streaming client. Simultaneously, after sending the live stream to the live streaming client, the network device can also cache the live stream. The live streaming client can decode and play the live stream after receiving it. The live stream can include multiple live video frames. The network device can obtain live stream cache information based on the image group information of the cached live video frames, and the network device can also record the frame index corresponding to the image group information.

[0063] The aforementioned image group information may include multiple live video frames, which can be keyframes in the live stream. That is, the network device can cache multiple live video keyframes in the live stream to form image group information, thereby obtaining live stream cache information. The aforementioned frame index can be the index of the live video frame included in the image group information, such as the timestamp and synchronization identifier of the live video frame. Specifically, the aforementioned image group information can be in the form of GOP (Group of Pictures). That is, the network device can cache the live video frames in the aforementioned live stream in the form of Group of Pictures image group information. Therefore, the frame index recorded by the network device can be the frame index corresponding to each Group of Pictures image group information.

[0064] Step S206: When a frame loss retransmission request is received from the live streaming client through the backup path, the corresponding image group information is obtained from the live streaming cache information by comparing the frame index carried in the frame loss retransmission request with the recorded frame index.

[0065] The live stream from the live streaming client may be interrupted due to network issues. For example, if the main connection between the live streaming client and the network device fails, the live stream cannot be transmitted to the live streaming client, resulting in frame drops. When frame drops occur, the live streaming client can send a frame drop retransmission request to the network device via a backup path. The live streaming client can generate the retransmission request based on the frame index of the lost live video frame. That is, the retransmission request carries the frame index of the lost live video frame. The network device records the frame index of the corresponding image group information for the live stream. The network device can then retrieve the corresponding image group information from the live stream cache by comparing the frame index carried in the retransmission request with the recorded frame index. In other words, the corresponding image group information contains information about the lost live video frames from the live streaming client.

[0066] Step S208: Send the corresponding image group information to the live streaming client via the backup path.

[0067] The network device obtains the corresponding image group information by comparing frame indexes. The network device can then send this image group information to the live streaming client via a backup path. Upon receiving the image group information, the live streaming client can decode and play it. For example, the client can decode the live video frames within the image group information and then play the decoded live video frames, thus relaying the live stream after an interruption.

[0068] In the aforementioned live streaming relay method and live streaming data receiving method, the live stream obtained from the live streaming source is sent to the live streaming client via the main path. Live streaming cache information is obtained based on the image group information of the cached live video frames. The frame index corresponding to the image group information is recorded. When a frame loss retransmission request is received from the live streaming client via the backup path, the corresponding image group information is obtained from the live streaming cache information by comparing the frame index carried in the retransmission request with the recorded frame index, and then sent to the live streaming client via the backup path. Compared to the traditional method of relaying live streams by switching to a lower bitrate, this solution improves the quality of the live stream during relay by simultaneously establishing a main path and a backup path, caching live video frames and their frame indexes, and sending the cached image group information to the client via the backup path when the live stream is interrupted.

[0069] In one embodiment, obtaining a live stream from a live streaming source includes: forwarding a live streaming request initiated by a live streaming client to the corresponding live streaming source; and receiving a live stream sent by the live streaming source.

[0070] In this embodiment, the network device can obtain the corresponding live stream based on the live stream request from the live streaming client. Specifically, after receiving the live stream request from the live streaming client, the network device can determine the corresponding live stream source and forward the request to that source. In some embodiments, the network device can forward the request directly to the live stream source connected to it. In other embodiments, the network device can forward the request to the corresponding live stream source through a higher-level network device connected to it.

[0071] After receiving a live streaming request, the live streaming source can send one live stream to the aforementioned network device. For example, the live streaming source may send only one live stream via a Fast User Datagram Protocol (HTP) connection to the network device. Specifically, the live stream may include multiple live video frames based on the HTP protocol. The header of each live video frame may contain information such as a timestamp and a synchronization identifier. The timestamp may be in 32-bit format, specifically Unix time. The synchronization identifier may be a 32-bit integer incremented by a preset size by each HTP packet. For example, each HTP packet increments by 1. The synchronization identifier may be randomly generated by the live streaming source. Each HTP packet may contain a corresponding live video frame. For example, each synchronization identifier corresponds to one live video frame. The network device can receive one live stream sent by the live streaming source, and then forward the live stream to the live streaming client via a HTP connection.

[0072] In this embodiment, the network device can obtain a live stream from the live stream source based on the live stream request. Then, the network device can send the corresponding live stream to the live stream client through the connection of the main path and the backup path based on the live stream. When the live stream is interrupted, the live stream is sent to the live stream client through the backup path, which improves the quality of the picture during live stream relay.

[0073] In one embodiment, recording the frame index corresponding to the image group information includes: obtaining the timestamp and synchronization identifier of the live video frame based on the frame packet header of the live stream on the main path; and recording the timestamp and synchronization identifier of the live video frame as the frame index of the corresponding image group information.

[0074] In this embodiment, the live stream transmitted via the main channel may include multiple live video frames. Each live video frame may include a frame header containing a timestamp and a synchronization identifier. The network device can then obtain the timestamp and synchronization identifier of the live video frames based on the frame headers in the live stream on the main channel. When sending the live stream to the live client via the main channel, the network device can also record the timestamps and synchronization identifiers of the live video frames as corresponding image group information. The network device can record the timestamps and synchronization identifiers of multiple live video frames according to a preset selection rule, thereby obtaining the frame indexes corresponding to the image group information of these live video frames.

[0075] Specifically, the frame index recorded by the network device can be the frame header information of the User Datagram Protocol (UDP) corresponding to the Group of Pictures information, including timestamps and synchronization identifiers. For example... Figure 3 As shown, Figure 3This is a schematic diagram of the structure of live streaming cache information in one embodiment. The network device can cache multiple image group information, each image group information corresponding to a GOP ID. The GOP ID is a 32-bit integer, where the first 16 bits can be the identifier of the live stream from the aforementioned live streaming client, and the last 16 bits can be an auto-incrementing integer, for example, the network device increments from 0. Each image group information can correspond to a specific frame index, which includes the GOP ID corresponding to the frame index, multiple timestamps, and each timestamp can correspond to multiple synchronization identifiers. Each synchronization identifier can correspond to one live video frame. For example, one second can include multiple live video frames. The length of the timestamp can be set based on actual conditions. The synchronization identifier can indicate the position of each live video frame in the live stream.

[0076] Through this embodiment, the network device can cache the live video frames in the live stream when sending the live stream to the live client via the main path, and record the timestamps and synchronization identifiers of the live video frames in the live stream as frame indexes. In this way, when the live stream is interrupted, the image group information requested by the live client can be determined by the frame index, thereby improving the picture quality during live relay.

[0077] In one embodiment, the corresponding image group information is obtained from the live streaming cache information by comparing the frame index carried in the frame loss retransmission request with the recorded frame index, including: obtaining the image group information corresponding to the timestamp and synchronization identifier carried in the frame loss retransmission request from the live streaming cache information by comparing the timestamp and synchronization identifier carried in the frame loss retransmission request with the recorded timestamp and synchronization identifier.

[0078] In this embodiment, when a live streaming client experiences frame loss due to a main path interruption, for example, if the number of consecutively lost frames reaches a preset threshold, the live streaming client can send a frame loss retransmission request to the network device via a backup path. This retransmission request carries a timestamp and a synchronization identifier. The network device can compare the timestamp and synchronization identifier carried in the retransmission request with recorded timestamps and synchronization identifiers. Based on this comparison, the network device can retrieve the image group information corresponding to the timestamp and synchronization identifier carried in the retransmission request from the live streaming cache information. The network device can then send this image group information to the live streaming client via the backup path connection, allowing the live streaming client to decode and play the video based on the received image group information.

[0079] Specifically, the live streaming client can generate a frame loss retransmission request (X-Stream-restransmit) using a Hypertext Transfer Protocol (HTTP) GET request, with the timestamp and synchronization identifier of the retransmission request included in the HTTP HEAD field. Upon receiving the frame loss retransmission request, the network device can query the image group information table based on the timestamp and synchronization identifier in the extended fields carried by the HTTP HEAD field, for example, querying... Figure 3 The table shown is used to compare the corresponding timestamps and synchronization identifiers to obtain the corresponding image group information. This image group information is then sent to the live streaming client via a backup path, such as a TCP (Transmission Control Protocol) based Hypertext Transfer Protocol (HTTP) method. Additionally, in some embodiments, the network device can perform congestion control using the Transmission Control Protocol after sending the corresponding image group information to the live streaming client to prevent excessive data transmission to the client via the backup path from causing line congestion.

[0080] In this embodiment, when the main path is interrupted, the network device can query the corresponding cached image group information through a frame loss retransmission request and send it to the live streaming client through the backup path, thereby improving the picture quality during live streaming relay.

[0081] In one embodiment, such as Figure 4 As shown, a method for receiving live data is provided, which can be applied to... Figure 1 Taking a live streaming client as an example, the steps are as follows:

[0082] Step S302: Establish the connection between the primary and backup paths with the network device.

[0083] The live streaming client can be the device terminal used by the user, who can trigger live streaming requests and watch live streams through the live streaming client. The live streaming client and network devices can establish a dual-path connection to achieve live stream forwarding and frame loss retransmission. This dual-path connection includes a primary path and a backup path; that is, the network device can establish a primary path connection and a backup path connection with the live streaming client.

[0084] The live streaming client can initiate a live streaming request to the network device, enabling the network device to establish a primary connection and a backup connection with the live streaming client upon receiving the request. These primary and backup connections can be established based on different protocols. For example, in one embodiment, the network device can establish a QUIC-UDP (Quick User Datagram Protocol Internet Connections) connection with the live streaming client as the primary connection. Alternatively, the network device can establish an HTTP (Hypertext Transfer Protocol) connection with the live streaming client as the backup connection. The live streaming request sent by the live streaming client can be a request for either QUIC-UDP or Hypertext Transfer Protocol.

[0085] Step S304: Receive the live stream sent by the network device through the main path; the live stream is obtained by the network device from the live source.

[0086] The live streaming request initiated by the live streaming client can be connected to the nearest network device. The live stream obtained by the network device from the live streaming source can be the live stream corresponding to the live streaming request of the live streaming client. For example, if the live streaming request carries information about the corresponding live stream, the network device can obtain the corresponding live stream from the live streaming source based on that information.

[0087] Specifically, after receiving the live stream via a connection based on the Fast User Datagram Protocol (HUDP), the live streaming client can check the frame header information in the HUDP data packets of the received live stream to determine the timestamp and synchronization identifier corresponding to each live video frame, thereby enabling the live streaming client to decode and play the live stream.

[0088] Step S306: When a frame loss is detected during live streaming, a frame loss retransmission request carrying the frame index is sent to the network device through the backup path.

[0089] When the primary connection between the live streaming client and the network device fails, frame drops will occur in the live stream played by the client. The live streaming client and the network device are also connected via a backup path. When the live streaming client detects frame drops, it can generate a corresponding retransmission request based on the frame index of the lost video frame. The live streaming client can then send the retransmission request, carrying the frame index, to the network device via the backup path.

[0090] The live streaming client can determine when to send a frame loss retransmission request based on the number of lost frames. For example, in one embodiment, the live streaming client can detect whether the number of consecutive lost frames from the main stream exceeds a threshold. If the live streaming client detects that the number of consecutive lost frames from the main stream exceeds the threshold, the live streaming client can generate a frame loss retransmission request carrying the timestamp and synchronization identifier corresponding to the lost live video frames, and send the frame loss retransmission request to the network device through the backup path. Specifically, the live streaming client can send the aforementioned frame loss retransmission request carrying the frame index to the network device through a connection based on the Hypertext Transfer Protocol.

[0091] Step S308: Receive image group information corresponding to the frame index sent by the network device via the backup path; wherein, the corresponding image group information is obtained by the network device from the live stream cache information by comparing the frame index carried in the frame loss retransmission request with the recorded frame index; the live stream cache information is obtained by the network device based on the image group information of the live video frames cached from the live stream source, and the recorded frame index is recorded by the network device when caching the image group information.

[0092] When a live streaming client sends a frame retransmission request to a network device, the network device can record the frame index of the corresponding image group information for the live stream. The network device can then retrieve the corresponding image group information from the live stream cache by comparing the frame index carried in the retransmission request with the recorded frame index. In other words, the corresponding image group information contains information about the live video frames lost by the live streaming client.

[0093] Network devices can send the corresponding image group information to the live streaming client via a backup path. After receiving the image group information, the live streaming client can decode and play it. For example, the live streaming client can decode the live video frames in the image group information and then play the decoded live video frames, thus achieving relaying after a live stream interruption.

[0094] In the aforementioned live data reception method, when a live stream is received from the live source and a frame loss is detected, a frame loss retransmission request is sent to the network device. The network device then compares the frame index carried in the retransmission request with the recorded frame index to retrieve the corresponding image group information from the live stream cache information and sends it to the live client via a backup path. Compared to the traditional method of relaying live streams by switching to a lower bitrate, this solution improves the quality of the live stream during relay by simultaneously establishing a primary and backup path, caching live video frames and their frame indices, and sending the cached image group information to the client via the backup path when the live stream is interrupted.

[0095] In one embodiment, such as Figure 5 As shown, Figure 5This is a flowchart illustrating a live streaming relay method in another embodiment. In this embodiment, the network device can be a live streaming relay device, the primary connection can be a Fast User Datagram Protocol (HUDP) based connection, and the backup connection can be a Hypertext Transfer Protocol (HTP) based connection. When a live streaming client needs to play live video, it can send a Fast Live Request to the live streaming relay device. The live streaming relay device forwards the request to the live streaming source and obtains the live stream from the source through a HUDP connection. Each user datagram frame header in the live stream contains a timestamp and a synchronization identifier. The live streaming relay device establishes both a HUDP-based connection and a HTP-based connection with the live streaming client.

[0096] After receiving the live stream, the live relay device can transmit the live stream to the live client via the aforementioned connection based on the Fast User Datagram Protocol (HTP). Simultaneously, the live relay device can also establish a cache information table for image group information, recording the HTP header frame information corresponding to the current image group, including timestamps and synchronization identifiers.

[0097] When a live streaming client sends a lost frame, such as detecting packet loss in the frame header information of a user datagram, the client can send a retransmission request carrying the frame index of the lost live video frame to the live streaming relay device via a Hypertext Transfer Protocol (HTTP) GET request, extending the fields in the HTTP header, and using the HTTP connection. The live streaming relay device, based on the frame index in the extended fields of the header, queries its image group information table. By comparing the frame index carried in the retransmission request with the frame index recorded in the relay device, the relay device determines the image group information corresponding to the frame index in the retransmission request. The relay device then sends this corresponding image group information to the live streaming client via the HTTP connection, allowing the client to decode and play the image group information, thus restoring the live stream.

[0098] Through the above embodiments, the terminal establishes both a primary and a backup path, introducing a dual-path approach for QUIC live relay. This constructs a mechanism that meets live streaming quality requirements for fast QUIC-UDP transmission and HTTP packet loss retransmission. By caching live video frames and their frame indexes, and sending cached image group information to the client via the backup path when the live stream is interrupted, the quality of the live stream during relay is improved. Furthermore, in the dual-path live relay device, one stream uses fast transmission to maintain stable and continuous video streaming, while the other stream uses on-demand TCP transmission with guaranteed delivery, reducing the pressure on the live streaming service while ensuring video service quality.

[0099] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0100] Based on the same inventive concept, this application also provides a live relay device and a live data receiving device for implementing the live relay method and live data receiving method described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the live relay device and live data receiving device provided below can be found in the limitations of the live relay method and live data receiving method described above, and will not be repeated here.

[0101] In one embodiment, such as Figure 6 As shown, a live streaming relay device is provided, comprising: a first establishment module 500, a live streaming recording module 502, a frame loss retransmission module 504, and a sending module 506, wherein:

[0102] The first module 500 is used to establish the connection between the main path and the backup path with the live streaming client.

[0103] The live streaming recording module 502 is used to obtain the live stream from the live streaming source, send the live stream to the live streaming client through the main path, obtain the live streaming cache information based on the image group information of the cached live video frames, and record the frame index corresponding to the image group information.

[0104] The frame loss retransmission module 504 is used to obtain the corresponding image group information from the live streaming cache information when a frame loss retransmission request is received from the live streaming client through the backup path, based on the comparison between the frame index carried in the frame loss retransmission request and the recorded frame index.

[0105] The sending module 506 is used to send the corresponding image group information to the live streaming client via the backup path.

[0106] In one embodiment, the live streaming recording module 502 is used to forward the live streaming request initiated by the live streaming client to the corresponding live streaming source; and to receive a live streaming stream sent by the live streaming source.

[0107] In one embodiment, the live streaming recording module 502 is used to obtain the timestamp and synchronization identifier of the live video frame based on the frame packet header of the live stream on the main path; and to record the timestamp and synchronization identifier of the live video frame as the frame index of the corresponding image group information.

[0108] In one embodiment, the above-mentioned frame loss retransmission module 504 is used to obtain the image group information corresponding to the timestamp and synchronization identifier carried in the frame loss retransmission request from the live broadcast cache information by comparing the timestamp and synchronization identifier carried in the frame loss retransmission request with the recorded timestamp and synchronization identifier.

[0109] In one embodiment, the first establishment module 500 is used to establish a connection with the live streaming client based on the Fast User Datagram Protocol (FAP) as the primary path; and to establish a connection with the live streaming client based on the Hypertext Transfer Protocol (HTP) as the backup path.

[0110] In one embodiment, such as Figure 7 As shown, a live data receiving device is provided, including: a second establishment module 600, a live streaming module 602, a request module 604, and a receiving module 606, wherein:

[0111] The second module 600 is used to establish the connection between the primary and backup paths with the network devices.

[0112] The live streaming module 602 is used to receive live streams sent by network devices through the main path; the live streams are obtained by the network devices from the live streaming source.

[0113] The request module 604 is used to send a retransmission request for the lost frame, carrying the frame index, to the network device through the backup path when a frame loss is detected in the live broadcast.

[0114] The receiving module 606 is used to receive image group information corresponding to the frame index sent by the network device through a backup path; wherein, the corresponding image group information is obtained by the network device from the live streaming cache information by comparing the frame index carried in the frame loss retransmission request with the recorded frame index; the live streaming cache information is obtained by the network device based on the image group information of the live video frames cached from the live stream obtained from the live streaming source, and the recorded frame index is recorded by the network device when caching the image group information.

[0115] In one embodiment, the request module 604 is used to generate a frame loss retransmission request carrying the timestamp and synchronization identifier based on the timestamp and synchronization identifier corresponding to the lost live video frame when the number of consecutive frame loss from the live stream from the main path exceeds a threshold; and send the frame loss retransmission request to the network device through the backup path.

[0116] Each module in the aforementioned live broadcast relay device and live broadcast data receiving device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0117] In one embodiment, a computer device is provided, which may be a network device or a client, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a live broadcast relay method or a live data reception method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0118] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0119] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described live relay method and live data receiving method.

[0120] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described live broadcast relay method and live broadcast data receiving method.

[0121] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described live relay method and live data receiving method.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0123] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0124] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A live broadcast relay method, characterized in that, The method includes: Establishing a primary and backup connection with the live streaming client includes: establishing a connection with the live streaming client based on the Fast User Datagram Protocol as the primary connection, and establishing a connection with the live streaming client based on the Hypertext Transfer Protocol as the backup connection. The system obtains the live stream from the live stream source, sends the live stream to the live stream client through the main path, obtains live stream cache information based on the image group information of the cached live video frames in the live stream, and records the frame index corresponding to the image group information. When a frame loss retransmission request is received from the live streaming client through the backup path, the corresponding image group information is obtained from the live streaming cache information by comparing the frame index carried in the frame loss retransmission request with the recorded frame index. The corresponding image group information is sent to the live streaming client via the backup route.

2. The method according to claim 1, characterized in that, The process of obtaining the live stream from the live stream source includes: Forward the live streaming request initiated by the live streaming client to the corresponding live streaming source; Receive one live stream sent by the live stream source.

3. The method according to claim 1, characterized in that, The frame index corresponding to the recorded image group information includes: Based on the frame packet header of the live stream on the main path, the timestamp and synchronization identifier of the live video frame are obtained; The timestamps and synchronization identifiers of the live video frames are recorded as frame indices for the corresponding image group information.

4. The method according to claim 3, characterized in that, The step of comparing the frame index carried in the lost frame retransmission request with the recorded frame index to obtain the corresponding image group information from the live stream cache information includes: Based on the timestamp and synchronization identifier carried in the frame loss retransmission request, and by comparing them with the recorded timestamp and synchronization identifier, the image group information corresponding to the timestamp and synchronization identifier carried in the frame loss retransmission request is obtained from the live streaming cache information.

5. A method for receiving live streaming data, characterized in that, The method includes: Establishing a primary and backup connection with a network device includes: establishing a connection with the network device based on the Fast User Datagram Protocol as the primary connection, and establishing a connection with the network device based on the Hypertext Transfer Protocol as the backup connection. The network device receives the live stream sent by the network device through the main path; the live stream is obtained by the network device from the live stream source. When a frame drop is detected during live streaming, a frame drop retransmission request carrying the frame index is sent to the network device through the backup path; The backup path is used to receive the image group information corresponding to the frame index sent by the network device; The corresponding image group information is obtained by the network device from the live stream cache information by comparing the frame index carried in the frame loss retransmission request with the recorded frame index; the live stream cache information is obtained by the network device based on the image group information of the live stream cached live video frames obtained from the live stream source, and the recorded frame index is recorded by the network device when caching the image group information.

6. The method according to claim 5, characterized in that, When a live stream frame loss is detected, a retransmission request carrying the frame index is sent to the network device through the backup path, including: When the number of consecutive lost frames from the live stream from the main channel exceeds a threshold, a retransmission request carrying the timestamp and synchronization identifier is generated based on the timestamp and synchronization identifier corresponding to the lost live video frame. The frame loss retransmission request is sent to the network device via the backup path.

7. A live broadcast relay device, characterized in that, The device includes: The first establishment module is used to establish a connection between the main path and the backup path with the live streaming client. Specifically, it is used to establish a connection with the live streaming client based on the Fast User Datagram Protocol as the main path, and to establish a connection with the live streaming client based on the Hypertext Transfer Protocol as the backup path. The live streaming recording module is used to obtain the live stream from the live streaming source, send the live stream to the live streaming client through the main path, obtain live streaming cache information based on the image group information of the cached live video frames of the live stream, and record the frame index corresponding to the image group information. The frame loss retransmission module is used to obtain the corresponding image group information from the live streaming cache information when it receives a frame loss retransmission request from the live streaming client through the backup path, based on the comparison between the frame index carried in the frame loss retransmission request and the recorded frame index. The sending module is used to send the corresponding image group information to the live streaming client through the backup path.

8. A live data receiving device, characterized in that, The device includes: The second establishment module is used to establish a primary path and a backup path connection with the network device. Specifically, it is used to establish a connection with the network device based on the Fast User Datagram Protocol as the primary path and to establish a connection with the network device based on the Hypertext Transfer Protocol as the backup path. The live streaming module is used to receive the live stream sent by the network device through the main path; the live stream is obtained by the network device from the live streaming source. The request module is used to send a retransmission request carrying a frame index to the network device through the backup path when a live frame loss is detected. The receiving module is configured to receive image group information corresponding to the frame index sent by the network device through the backup path; The corresponding image group information is obtained by the network device from the live stream cache information by comparing the frame index carried in the frame loss retransmission request with the recorded frame index; the live stream cache information is obtained by the network device based on the image group information of the live stream cached live video frames obtained from the live stream source, and the recorded frame index is recorded by the network device when caching the image group information.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images