Live broadcast stream pulling method and device, equipment, system, storage medium and product

Abstract

The invention discloses a live broadcast stream pulling method and apparatus, a device, a system, a storage medium and a product. The method comprises the steps of obtaining first information of a live broadcast stream; if it is determined that the first information triggers a preset protocol switching condition, switching a stream pulling protocol of the live broadcast stream; wherein the protocol switching condition comprises a first protocol switching condition for switching a pull stream protocol based on the live broadcast keyword and / or a second protocol switching condition for switching the pull stream protocol based on the live broadcast configuration parameter, and the first information comprises the live broadcast keyword and / or the live broadcast configuration parameter. According to the method, the current pull flow protocol is switched after determining that the current pull flow protocol cannot meet the requirement of the live broadcast service based on the obtained live broadcast configuration parameters of the live broadcast stream, and / or the current pull flow protocol is switched after determining the current requirement of the live broadcast service based on the live broadcast keyword appearing in the live broadcast stream; therefore, the live broadcast quality of the live broadcast service is improved by fully utilizing the service advantages of the pull stream protocols based on the mode of dynamically switching the pull stream protocols of the live broadcast stream.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of live streaming, and particularly relates to a live streaming pulling method and device, equipment, system, storage medium and product. BACKGROUND

[0002] A live streaming system usually includes a live streaming client device, a live streaming background and a user terminal device. The live streaming client device initiates a live streaming service and pushes a live streaming flow to the live streaming background. The live streaming background receives the live streaming flow, transcodes the live streaming flow, and sends a live streaming pulling address to the user terminal device after receiving a live streaming pulling request of the user terminal device. The live streaming service process includes multiple steps such as audio and video collection, audio and video encoding, live streaming flow pushing, background transcoding, live streaming flow pulling, audio and video decoding, and audio and video playing.

[0003] In the related art, a live streaming system usually adopts an RTMP pushing and pulling flow architecture, that is, based on a RealTime Messaging Protocol (RTMP), to realize pushing and pulling of a live streaming flow. With the development of the live streaming service, the RTMP pushing and pulling flow architecture is difficult to meet the needs of the live streaming service with large concurrency and frequent live streaming interaction, which causes excessive resource consumption and uneven resource allocation. Moreover, the RTMP pushing and pulling flow architecture requires a user terminal device to use a specific multimedia program player to play a live streaming flow, and many commonly used browsers are not compatible with the multimedia program player. SUMMARY

[0004] Therefore, the embodiments of the present application provide a live streaming pulling method and device, equipment, system, storage medium and product, aiming to improve the live streaming quality based on a switching live streaming pulling protocol.

[0005] The technical scheme of the embodiments of the present application is as follows:

[0006] In a first aspect, the embodiments of the present application provide a live streaming pulling method applied to a user terminal device, and the method includes the following steps.

[0007] obtaining first information of a live streaming flow;

[0008] determining that the first information triggers a preset protocol switching condition, and then switching a pulling protocol of the live streaming flow;

[0009] The protocol switching condition includes a first protocol switching condition of switching a pulling protocol based on a live streaming keyword and / or a second protocol switching condition of switching a pulling protocol based on a live streaming configuration parameter. The first information includes a live streaming keyword and / or a live streaming configuration parameter.

[0010] In the above scheme, the method further includes the following steps.

[0011] The initial streaming protocol for the live stream is set to RMTP.

[0012] In the above scheme, obtaining the first information of the live stream includes:

[0013] Based on the video data of the live stream, obtain the live stream keywords that appear;

[0014] The step of determining that the first information triggers a preset protocol switching condition, and then switching the streaming protocol of the live stream, includes:

[0015] If the live stream keyword triggers the first protocol switching condition, then the pull protocol of the live stream is switched.

[0016] In the above scheme, the step of determining that the live stream keyword triggers the first protocol switching condition, and then switching the live stream pull protocol, includes:

[0017] Obtain the current first pull protocol of the live stream;

[0018] If it is determined that the live streaming keyword matches any keyword in the preset keyword list, then based on the first streaming protocol and the matched keyword, the corresponding second streaming protocol is determined from the first protocol switching conditions;

[0019] Switch the streaming protocol of the live stream to the second streaming protocol.

[0020] In the above scheme, the method further includes:

[0021] Based on the Deterministic Finite Automaton (DFA) algorithm, the keyword list is traversed to determine whether the live streaming keyword matches any keyword in the keyword list.

[0022] In the above scheme, determining that the first information triggers a preset protocol switching condition, and then switching the streaming protocol of the live stream, includes:

[0023] If the live streaming configuration parameters trigger the second protocol switching condition, then the streaming protocol of the live stream is switched.

[0024] In the above scheme, the live streaming configuration parameters include one or more of the following: resolution, bitrate, resolution, and live streaming duration. The step of determining that the live streaming configuration parameters trigger the second protocol switching condition, and then switching the live streaming protocol, includes:

[0025] If the resolution in the live streaming configuration parameters is determined to be greater than a set resolution threshold, then the live streaming protocol is switched to HTTP Live Streaming (HLS) protocol; and / or,

[0026] If it is determined that the bitrate in the live streaming configuration parameters is greater than a set bitrate threshold, and / or, if it is determined that the resolution in the live streaming configuration parameters is greater than a set resolution threshold, then the streaming protocol of the live stream will be switched to the Dynamic Adaptive Streaming over HTTP (DASH) protocol; and / or,

[0027] If the live streaming duration in the live streaming configuration parameters is determined to be greater than the set duration threshold, and if the current live streaming duration is determined to have reached the set duration threshold, then the streaming protocol of the live stream is switched to the Web Real Time Communication (WebRTC) protocol.

[0028] The second protocol switching condition is set to one or more of the following conditions: the set resolution threshold, the set bitrate threshold, the set resolution threshold, and the set duration threshold.

[0029] In the above scheme, the method further includes:

[0030] Obtain live streaming monitoring information;

[0031] Based on the live streaming monitoring information of the live stream, switch the streaming protocol of the live stream;

[0032] The live streaming monitoring information includes at least one of the following: the current latency of the live stream and the browser type used to play the live stream.

[0033] In the above scheme, the live streaming configuration information includes a latency threshold, and the live streaming monitoring information based on the live stream, and the conditions for switching the live stream pull protocol, include:

[0034] Based on the live streaming monitoring information, if it is determined that the current latency of the live stream is greater than the latency threshold, then the streaming protocol of the live stream is switched to HTTP streaming media format (HTTP-Flash Video, HTTP-FLV) protocol; and / or,

[0035] Based on the live streaming monitoring information, if it is determined that the browser type used by the user's device to play the live stream does not belong to the set browser set, then the streaming protocol condition for switching the live stream is changed to HTTP-FLV protocol.

[0036] Secondly, embodiments of this application provide a live streaming device and an application client device, the device comprising:

[0037] The acquisition module is used to acquire the initial information of the live stream;

[0038] The determining module is used to determine the preset protocol switching conditions triggered by the first information;

[0039] The switching module is used to trigger preset protocol switching conditions based on the first information to switch the streaming protocol of the live stream;

[0040] The protocol switching conditions include: a first protocol switching condition based on live streaming keywords and / or a second protocol switching condition based on live streaming configuration parameters, wherein the first information includes: live streaming keywords and / or live streaming configuration parameters.

[0041] Thirdly, embodiments of this application provide a live streaming system, which includes: a live streaming client device, a first live streaming backend, and a user terminal device, wherein...

[0042] The live streaming client device is used to initiate live streaming services and push the live stream to the first live streaming backend.

[0043] The first live streaming backend is used to transcode the live stream and send the pull stream address to the user terminal device;

[0044] The user terminal device is used to obtain the live stream based on the pull address; the user terminal device is also used to perform the steps of the method as described in the first aspect.

[0045] Fourthly, embodiments of this application provide a user terminal device, the user terminal being used for live streaming services, comprising: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, performs the steps of the method described in the first aspect.

[0046] Fifthly, embodiments of this application provide a storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0047] In a sixth aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0048] The live streaming pull method for a user terminal device provided in this application includes: acquiring first information of the live stream; determining that the first information triggers a preset protocol switching condition, and then switching the live stream pull protocol; wherein, the protocol switching condition includes: a first protocol switching condition based on live stream keywords and / or a second protocol switching condition based on live stream configuration parameters, and the first information includes: live stream keywords and / or live stream configuration parameters. This application embodiment, based on the acquired live stream configuration parameters, determines that the current pull protocol cannot meet the live stream service requirements, and switches the current pull protocol; and / or, based on the live stream keywords appearing in the live stream, determines the current requirements of the live stream service, and switches the current pull protocol. Thus, this application embodiment, based on dynamically switching the live stream pull protocol, fully utilizes the business advantages of each pull protocol, improving the live stream quality. Attached Figure Description

[0049] Figure 1 A schematic diagram of the structure of a live streaming system with an RTMP push-pull streaming architecture;

[0050] Figure 2 This is a flowchart illustrating the live streaming method according to an embodiment of this application;

[0051] Figure 3 This is a flowchart illustrating the first protocol switching condition in an embodiment of this application;

[0052] Figure 4 This is a flowchart illustrating the first protocol switching condition in an application example of this application;

[0053] Figure 5 This is a schematic diagram of the live streaming device according to an embodiment of this application;

[0054] Figure 6 This is a schematic diagram of the structure of a live streaming system according to an embodiment of this application;

[0055] Figure 7 This is a schematic diagram of the structure of a live streaming system according to another embodiment of this application;

[0056] Figure 8 This is a schematic diagram of the user terminal device according to an embodiment of this application. Detailed Implementation

[0057] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0059] Before describing the live streaming method of the embodiments of this application, the architecture of the live streaming system will be described first in order to better understand the technical solution of the embodiments of this application.

[0060] In related technologies, such as Figure 1 As shown, live streaming systems typically employ an RTMP push-pull streaming architecture, specifically including live streaming client devices, a live streaming backend, and user-end devices. The live streaming client device initiates the live streaming service, uses a streaming media recording program to merge multiple streams recorded by the recording device, and pushes the live stream to a push address specified by the live streaming backend based on the RTMP protocol. After receiving the live stream, the live streaming backend transcodes it and, upon receiving a pull request from the user-end device, sends a pull address to the user-end device. The user-end device then obtains the live stream based on the RTMP protocol, allowing users to watch the live stream. Specifically, the live streaming service process includes multiple steps such as audio and video acquisition, audio and video encoding, live stream push, backend transcoding, live stream pull, audio and video decoding, and audio and video playback.

[0061] Here, RTMP is a real-time transmission protocol that supports push and pull streaming. RTMP addresses begin with rtmp: / / , and the push and pull addresses are the same. RTMP is based on the Transmission Control Protocol (TCP) and forcibly segments the encapsulated audio and video data during live stream transmission to ensure real-time data transmission. Due to its low latency, most live streaming services choose to use an RTMP push-pull streaming architecture.

[0062] It should be noted that with the development of live streaming services, using a fixed RTMP protocol for streaming is difficult to meet the needs of live streaming services such as high concurrency and frequent live interactions. This will cause excessive resource consumption and uneven resource allocation. Furthermore, the RTMP push-pull streaming architecture requires user devices to use specific multimedia program players to play audio and video. Currently, many commonly used browsers are not compatible with these multimedia program players, making the RTMP push-pull streaming architecture unsuitable for some user devices.

[0063] Based on this, this application provides a live streaming pull method for user terminal devices, which aims to improve live streaming quality by switching the pull protocol.

[0064] This application provides a live streaming pull method, applied to user terminal devices, such as... Figure 2 As shown, the method includes:

[0065] Step 201: Obtain the first information of the live stream.

[0066] Step 202: If the first information triggers the preset protocol switching condition, then switch the live stream pull protocol.

[0067] The protocol switching conditions include: a first protocol switching condition based on live streaming keywords and / or a second protocol switching condition based on live streaming configuration parameters. The first information includes: live streaming keywords and / or live streaming configuration parameters.

[0068] For example, the method further includes setting the initial streaming protocol of the live stream to the RMTP protocol.

[0069] It is understood that when the user terminal device in this application embodiment pulls the stream, the default streaming protocol is the commonly used RMTP protocol, in order to ensure the stability of the live streaming service in a normal live streaming scenario.

[0070] It should be noted that some real-time transmission protocols have significant advantages over RTMP protocols in certain live streaming scenarios. Based on the first information obtained, this application embodiment determines the current requirements of the live streaming service and switches the streaming protocol to a streaming protocol that meets the requirements in order to improve the quality of the live streaming.

[0071] Here, the first piece of information in the live stream is information that reflects the current needs of the live streaming business, including: live keywords and / or live configuration parameters. Live keywords are those written and mentioned by the live streamer during the live stream. Based on the recognition of these keywords, the user-end device can determine that the live stream is about to play key content, and the live streaming business needs will change. For example, the live streaming business needs may change from a high-stability scenario to a high-concurrency scenario, making it difficult to determine if RTMP-based streaming is suitable for the current high-concurrency live streaming scenario. Live configuration parameters are the requirements for this live stream configured by the live streamer on the live streaming client device, including one or more of the following: resolution, bitrate, and live stream duration. The user-end device can obtain the live streaming configuration parameters configured by the live streamer to determine whether the current streaming protocol meets the current live streaming needs.

[0072] It is understood that the embodiments of this application pre-set protocol switching conditions. The protocol switching conditions are based on the current needs of the live stream and / or the current streaming protocol, indicating that the streaming protocol of the live stream should be switched to a suitable streaming protocol.

[0073] It is understood that, based on the live streaming configuration parameters obtained from the live streaming stream, this application embodiment determines that the current streaming protocol cannot meet the needs of the live streaming business, and switches the current streaming protocol; and / or, based on the live streaming keywords appearing in the live streaming stream, determines the current needs of the live streaming business, and switches the current streaming protocol. In this way, this application embodiment, based on the method of dynamically switching the streaming protocol of the live streaming stream, makes full use of the business advantages of each streaming protocol and improves the live streaming quality of the live streaming business.

[0074] For example, in a scenario where the first information of the acquired live stream is a live stream keyword, acquiring the first information of the live stream includes: acquiring the live stream keyword that appears based on the video data of the live stream.

[0075] Understandably, user devices capture images and text from live video content to obtain keywords that appear during the live stream in order to determine the importance of the current live content.

[0076] For example, determining that the first information triggers a preset protocol switching condition, and then switching the streaming protocol of the live stream, includes: determining that the live stream keyword triggers the first protocol switching condition, and then switching the streaming protocol of the live stream.

[0077] Specifically, if the live stream keyword triggers the first protocol switching condition, then the live stream pull protocol is switched, including: obtaining the current first pull protocol of the live stream; determining that the live stream keyword matches any keyword in the preset keyword list, then determining the corresponding second pull protocol from the first protocol switching condition based on the first pull protocol and the matched keyword; and switching the live stream pull protocol to the second pull protocol.

[0078] It should be noted that the keyword list can be pre-set based on the type of live streaming business, consisting of business type keywords and key guiding words.

[0079] In one example, if the live stream is for popularizing medical knowledge related to hypertension, the business type keywords can be set as commonly used terms related to hypertension, such as: blood pressure, blood pressure escalation, blood pressure control, and blood pressure risks. Key guiding words can be set as: "Here comes the important part," "Here comes the important content," and "Here comes the important knowledge," indicating the importance of the live stream content. When the user's device determines that the acquired live stream keywords match any keyword in the keyword list, it determines that the current live stream content is of high interest to the user. At this point, the live stream service may have requirements such as high concurrency and high definition, necessitating an adaptive switch to a streaming protocol that meets the scenario's requirements.

[0080] For example, the method further includes: based on the deterministic DFA algorithm, traversing the keyword list to determine whether the live streaming keyword matches any keyword in the keyword list.

[0081] It's important to note that the DFA algorithm is a string matching algorithm, commonly used for sensitive word filtering and string matching. The mechanism of the DFA algorithm involves constructing a finite-state automaton, which consists of a finite number of states and the transition relationships between them. During string matching, it starts from the beginning of the string and gradually transitions to the next state based on the character, until the entire automaton is matched or no further matching is possible. Crucially, in the automaton, each state transition is deterministic; that is, for the current state and the input character, there is only one unique next state.

[0082] It should be noted that although the DFA algorithm requires the pre-construction of the transformation relationship between each state and the keyword list before string matching, and the string matching rule setting is relatively complex, in the live streaming scenario, due to the relatively simple type and content of the live streaming business, the number of keywords in the keyword list constructed based on the live streaming business type is small, and the transformation relationship between the streaming protocols is limited. Therefore, the DFA algorithm is suitable for the method of switching the live streaming protocol based on live streaming keywords in the embodiments of this application.

[0083] It is understandable that, in the first protocol switching condition, based on the DFA algorithm, the pull streaming protocols supported by the embodiments of this application are used as a set of states, and the keyword list is used as the triggering event for the transition between states. For example... Figure 3 As shown, the next state of the streaming protocol 1 can be determined by matching the live streaming keyword 1 with the live streaming keyword 2, and the next state can be determined by matching the live streaming keyword 4 with the live streaming keyword 4. Therefore, in the first protocol switching condition, a unique second streaming protocol can be determined based on the current first streaming protocol and the matched keyword.

[0084] In the application example of a live broadcast for popularizing medical knowledge related to hypertension, the pre-set keyword list includes ["blood pressure", "blood pressure escalation", "blood pressure control", "blood pressure danger", "key point", "important"]. Among them, blood pressure, blood pressure escalation, and blood pressure control are business type keywords, while "key point" and "important" are key guiding words. Based on the above keyword list and the streaming protocol supported by this example, a finite automaton is constructed to obtain the first protocol switching condition, such as... Figure 4 As shown.

[0085] Here, during the live stream, the user's device acquires the live stream keywords and iterates through the keyword list based on the DFA algorithm. If the live stream keyword matches a keyword in the keyword list, the system iterates through a specific sequence according to the matching keyword. This example can pre-set specific sequences to reduce the number of iterations. For example, when it is determined that the live stream keyword contains "blood," the system will only iterate through the specific sequence "blood pressure" and not through sequences like "important" or "important."

[0086] Understandably, in this example, the initial streaming protocol for the live stream is RTMP. When the live stream keyword matches "blood pressure", the streaming protocol is switched to HTTP-FLV; when the live stream keyword matches "dangerous blood pressure", the streaming protocol is switched to WebRTC.

[0087] For example, in a scenario where the first information of the acquired live stream is the live stream configuration parameters, determining that the first information triggers a preset protocol switching condition, and then switching the live stream pull protocol, includes: determining that the live stream configuration parameters trigger a second protocol switching condition, and then switching the live stream pull protocol.

[0088] It is understood that the embodiments of this application combine the business advantages of each streaming protocol, set the second protocol switching conditions, and determine the live streaming scenario and requirements based on the live streaming configuration parameters configured by the live streamers, and switch the live streaming streaming protocol to a suitable streaming protocol.

[0089] Here, the second protocol switching condition is set to one or more of the following conditions: setting a resolution threshold, setting a bitrate threshold, setting a resolution threshold, and setting a duration threshold.

[0090] For example, if the live streaming configuration parameters trigger the second protocol switching condition, then the live streaming pull protocol is switched, including: if the resolution in the live streaming configuration parameters is greater than a set resolution threshold, then the live streaming pull protocol is switched to the HLS protocol.

[0091] It should be noted that the resolution in the live streaming configuration parameters can be configured by the live streamer through the live streaming client before the live stream starts, or the resolution can be switched during the live stream.

[0092] Here, the HLS protocol plays video in segments, using the TS video format. Live stream data transmitted via HLS consists of TS video segments and M3U8 files. The M3U8 file records the playlist of the TS video segments. When a user device pulls a stream using the HLS protocol, it first requests the M3U8 file of the live stream data via HTTP, and then obtains the pointers to the TS video segments from the M3U8 file. In a live streaming scenario, the user device continuously polls the M3U8 file to obtain the pointers to new TS video segments, thus enabling live playback.

[0093] Understandably, TS segment video files typically store only a few seconds of video data. Therefore, the HLS protocol is suitable for live streaming or video-on-demand streaming services that require high resolution. However, in live streaming scenarios, user devices need to continuously poll the M3U8 file, resulting in higher latency for live streams streamed using the HLS protocol. When the resolution of the live stream selected or switched to by the broadcaster exceeds a set resolution threshold, the current streaming protocol is adaptively switched to the HLS protocol.

[0094] For example, if the live streaming configuration parameters trigger the second protocol switching condition, then the live streaming pull protocol is switched, including: if the bitrate in the live streaming configuration parameters is greater than a set bitrate threshold, and / or if the resolution in the live streaming configuration parameters is greater than a set resolution threshold, then the live streaming pull protocol is switched to the DASH protocol.

[0095] Here, the DASH protocol, also known as the MPEG-DASH protocol, is similar to the HLS protocol in that it plays video by slicing video files. The DASH protocol supports multiple bitrates and resolutions for live streams, allowing broadcasters to adjust the quality based on their requirements.

[0096] Understandably, when the video stream bitrate selected by the live streamer is greater than the set bitrate threshold, and / or the video stream resolution is greater than the set resolution threshold, it is determined that the current streaming protocol of the live stream does not meet the bitrate and / or resolution requirements, and the current streaming protocol of the live stream is adaptively switched to the DASH protocol.

[0097] For example, if the live streaming configuration parameters trigger the second protocol switching condition, then the live streaming pull protocol is switched, including: determining that the live streaming duration in the live streaming configuration parameters is greater than a set duration threshold; if it is determined that the current live streaming duration has reached the set duration threshold, then the live streaming pull protocol is switched to the WebRTC protocol.

[0098] Here, WebRTC is a peer-to-peer video / voice call protocol that enables audio and video communication between browsers, making it suitable for small-scale, high-stability live streaming services.

[0099] Specifically, in the WebRTC protocol, the user device is first prompted to obtain the live stream based on the getUserMedia API interface, and the RTCPeerConnection interface is called to establish communication with the live streaming backend; a network streaming media SDP (Session Description Protocol) session is created and sent to the live streaming backend to notify the live streaming backend that the media stream type to be received by the user device is live; finally, the live stream sent by the live streaming backend is received based on the RTCPeerConnection interface.

[0100] Understandably, long-duration live streaming services typically have low concurrency and high stability requirements. When the live streaming duration configured by the broadcaster exceeds a set duration threshold, the current streaming protocol (e.g., RTMP) is used within the set duration threshold. When the current live streaming duration is detected to exceed the set duration threshold, the streaming protocol is adaptively switched to WebRTC.

[0101] In one application example, the duration threshold is set to 1 hour, the live streamer configures the live stream duration to be 1.5 hours, the initial streaming protocol for this live stream service is the RTMP protocol, and the initial value of the current live stream duration is 0. After the live stream starts, the user device pulls the stream based on the RTMP protocol, and updates the current live stream duration after each pull. If the current live stream duration is greater than 1 hour, the second protocol switching condition is triggered, and the streaming protocol is switched to the WebRTC protocol.

[0102] For example, the method further includes: obtaining live streaming monitoring information of the live stream; and switching the streaming protocol conditions of the live stream based on the live streaming monitoring information. The live streaming monitoring information includes at least one of the following: the current latency of the live stream and the browser type used to play the live stream.

[0103] It is understood that, in addition to switching the streaming protocol of the live stream based on the obtained live keywords and the live configuration parameters configured by the live streamers, the embodiments of this application also obtain live monitoring information such as the current latency and browser type during the live stream, and switch the streaming protocol based on the live monitoring information.

[0104] For example, the live streaming configuration information also includes a latency threshold, and conditions for switching the live streaming protocol based on the live streaming monitoring information, including: if the current latency of the live streaming is determined to be greater than the latency threshold based on the live streaming monitoring information, then the conditions for switching the live streaming protocol to HTTP-FLV protocol; and / or, if the browser type used by the user terminal device to play the live stream is determined not to belong to the set browser set based on the live streaming monitoring information, then the conditions for switching the live streaming protocol to HTTP-FLV protocol.

[0105] Here, the HTTP-FLV protocol is a video slice transmission protocol based on the HTTP protocol. It is usually only used as a streaming protocol. Compared with the RTMP protocol, the HTTP-FLV protocol is suitable for more playback scenarios and has the characteristics of low latency.

[0106] It is understandable that when streaming based on the RTMP protocol, the user device needs to use a specific multimedia program player to play audio and video. However, many commonly used browsers are not compatible with this multimedia program player. Therefore, this application embodiment pre-sets a set of browsers that support specific multimedia program players. If it is determined that the browser type currently used to play the live stream does not belong to the set of browsers, and it is identified that the user device does not currently support streaming based on the RTMP protocol, then the streaming protocol of the live stream is switched to the HTTP-FLV protocol, which is more compatible with playback scenarios.

[0107] Understandably, when pulling streams based on the RTMP protocol, the live streaming latency is high in high-concurrency live streaming scenarios. Therefore, based on the latency threshold configured by the live streamers, if the current latency is greater than the latency threshold, it is determined that the current live streaming quality is poor, and the live stream pulling protocol is switched to the HTTP-FLV protocol, which is suitable for high-concurrency scenarios.

[0108] It should be noted that the user terminal device used in this application to execute the aforementioned live streaming pull method can be a first user terminal device for playing live streams, or a second user terminal device for managing the live streaming status of the live streaming service. When the first user terminal device executes the aforementioned live streaming pull method, the first user terminal device switches the current pull protocol based on the acquired live stream when the protocol switching condition is triggered; when the second user terminal device executes the aforementioned live streaming pull method, the second user terminal device uniformly instructs the first user terminal device to switch the pull protocol based on the acquired live stream.

[0109] Here, the user terminal device can be a mobile phone, computer, television, or other terminal device.

[0110] It should be noted that determining the live streaming configuration parameters to trigger the second protocol switching condition can be a switch from the RTMP protocol to the streaming protocol indicated by the second protocol switching condition, or a switch from the current streaming protocol other than RTMP to the streaming protocol indicated by the second protocol switching condition. For example, when the user device determines that the performance of the server node used for live streaming services in the live streaming backend is reduced, the DASH protocol currently used for streaming will be switched to the WebRTC protocol.

[0111] In one application example, if the current live streaming scenario is a remote surgical demonstration, a medical conference seminar, or an academic exchange between large hospitals or medical alliances, such live streams usually have high latency requirements. The live streamer can set the latency threshold to a low latency. When the live stream starts, the user terminal device obtains the live stream based on the RTMP protocol. When it detects that the current latency is greater than the latency threshold, it dynamically switches to HTTP-FLV to obtain the live stream.

[0112] Meanwhile, the aforementioned live streaming scenarios, in addition to having high latency requirements, also feature low concurrency and long live streaming durations. When it is determined that the live streaming duration configured by the live streamer exceeds the set duration threshold, the current live streaming duration is timed after the live stream begins. When the current live streaming duration exceeds the set duration threshold, the system dynamically switches to the WebRTC protocol to obtain the live stream.

[0113] In order to implement the method of the embodiments of this application, the embodiments of this application also provide a live streaming pull device, which corresponds to the above-described live streaming pull method. The steps in the above-described live streaming pull method embodiments are also fully applicable to the embodiments of this device.

[0114] like Figure 5 As shown in the illustration, this application provides a live streaming pull device and an application client device. The device includes an acquisition module 501, a determination module 502, and a switching module 503. The acquisition module 501 acquires first information about the live stream; the determination module 502 determines that the first information triggers preset protocol switching conditions; the switching module 503 switches the live stream pull protocol based on the preset protocol switching conditions triggered by the first information. The protocol switching conditions include: a first protocol switching condition based on live stream keywords and / or a second protocol switching condition based on live stream configuration parameters. The first information includes: live stream keywords and / or live stream configuration parameters.

[0115] In some embodiments, the live streaming device further includes a streaming module 504, which is used to set the initial streaming protocol of the live stream to the Real-Time Message Transfer Protocol (RMTP).

[0116] In some embodiments, the acquisition module 501 is specifically used to: acquire live streaming keywords based on the video data of the live stream.

[0117] In some embodiments, the determining module 502 is specifically used to: determine the conditions for triggering a first protocol switching based on the live streaming keyword. The switching module 503 is specifically used to: switch the streaming protocol of the live stream based on the conditions for triggering a first protocol switching based on the live streaming keyword.

[0118] In some embodiments, the determining module 502 is specifically used to: obtain the current first streaming protocol of the live stream; determine that the live stream keyword matches any keyword in a preset keyword list, and then determine the corresponding second streaming protocol from the first protocol switching conditions based on the first streaming protocol and the matched keyword. The switching module 503 is specifically used to: switch the streaming protocol of the live stream to the second streaming protocol.

[0119] In some embodiments, the determining module 502 is further configured to: based on the DFA algorithm, traverse the keyword list and determine whether the live streaming keyword matches any keyword in the keyword list.

[0120] In some embodiments, the determining module 502 is specifically used to: determine the conditions for triggering the second protocol switching based on the live streaming keyword. The switching module 503 is specifically used to: switch the streaming protocol of the live stream based on the conditions for triggering the second protocol switching based on the live streaming keyword.

[0121] In some embodiments, the live streaming configuration parameters include one or more of the following: resolution, bitrate, resolution, and live streaming duration; the second protocol switching condition setting includes one or more of the following conditions: setting a resolution threshold, setting a bitrate threshold, setting a resolution threshold, and setting a duration threshold;

[0122] The determining module 502 is specifically used to: determine that the resolution in the live streaming configuration parameters is greater than a set resolution threshold. The switching module 503 is specifically used to: switch the live stream's streaming protocol to the HLS protocol based on the resolution in the live streaming configuration parameters being greater than the set resolution threshold. And / or,

[0123] The determining module 502 is specifically used to: determine that the bitrate in the live streaming configuration parameters is greater than a set bitrate threshold, and / or determine that the resolution in the live streaming configuration parameters is greater than a set resolution threshold; the switching module 503 is specifically used to: based on the fact that the bitrate in the live streaming configuration parameters is greater than the set bitrate threshold, and / or the resolution in the live streaming configuration parameters is greater than the set resolution threshold, switch the live stream's streaming protocol to the DASH protocol.

[0124] The determination module 502 is specifically used to: determine that the current live broadcast duration has reached the set duration threshold when the live broadcast duration in the live broadcast configuration parameters exceeds the set duration threshold; the switching module 503 is specifically used to: switch the live broadcast streaming protocol to the WebRTC protocol based on the current live broadcast duration reaching the set duration threshold.

[0125] In some embodiments, the acquisition module 501 is further configured to: acquire live streaming monitoring information of the live stream. The live streaming monitoring information includes at least one of the following: the current latency of the live stream and the type of browser used to play the live stream.

[0126] In some embodiments, the switching module 503 is further configured to: switch the streaming protocol of the live stream based on the live stream monitoring information of the live stream.

[0127] In some embodiments, the live streaming configuration information includes a latency threshold. The determining module 502 is further configured to: determine, based on live streaming monitoring information, that the current latency of the live stream is greater than the latency threshold; the switching module 503 is specifically configured to: switch the live stream's streaming protocol to HTTP-FLV if the current latency of the live stream is greater than the latency threshold. And / or,

[0128] The determining module 502 is also used to: determine, based on the live broadcast monitoring information, that the browser type used by the user terminal device to play the live broadcast does not belong to the set browser set; the switching module 503 is specifically used to: if the browser type used by the user terminal device to play the live broadcast does not belong to the set browser set, then switch the live broadcast streaming protocol to the HTTP-FLV protocol.

[0129] It should be noted that the live streaming device provided in the above embodiments is only illustrated by the division of the above program modules. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the live streaming device and the live streaming method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0130] This application also provides a live streaming system, such as... Figure 6 As shown, the live streaming system includes: a live streaming client device 601, a first live streaming backend 602, and a user terminal device 603. The live streaming client device 601 initiates the live streaming service and pushes the live stream to the first live streaming backend 602. The first live streaming backend 602 transcodes the live stream and sends a pull address to the user terminal device 603. The user terminal device 603 retrieves the live stream based on the pull address. The user terminal device 603 also executes the steps of the aforementioned live streaming pull method.

[0131] For example, such as Figure 7 As shown, the live streaming system also includes at least one second live streaming backend 701. The first live streaming backend 602 is also used to push the received live stream to at least one second live streaming backend 701. The second live streaming backend 701 is used to transcode the received live stream.

[0132] The first live streaming backend 602 includes a first node, and the second live streaming backend 602 includes a second node. The first node and the second node are Content Delivery Network (CDN) nodes used to store the transcoded live stream. Specifically, the first live streaming backend 602 is used to: use the address of the first node as the pull stream address and send the pull stream address to the user terminal device 603; if it is determined that the current live streaming quality of the live streaming service does not meet the set live streaming quality threshold, then update the pull stream address to the address of the second node of any second live streaming backend.

[0133] It should be noted that in a live streaming system with an RTMP push-pull streaming architecture, the live streaming client device and the user terminal device only connect to one live streaming backend. If the CDN node used by the live streaming backend to store the transferred live stream fails or its performance degrades, it will affect the live streaming quality and, in severe cases, affect the normal operation of the live streaming service.

[0134] It is understood that the live streaming system in this embodiment of the application is a cloud-backup architecture. In addition to the primary live streaming backend 602, which is mainly used for live streaming, the system also includes at least one backup secondary live streaming backend 701. After receiving the live stream pushed by the live streaming client device 601, the primary live streaming backend 602 converts the live stream to a push value of at least one secondary live streaming backend 701. The secondary live streaming backend 701 transcodes the received live stream and stores it on its own secondary node. When the live streaming service is running normally, the primary live streaming backend 602 acts as the pull source, and the address of the primary node of the primary live streaming backend 602 is the pull address. When the primary live streaming backend 602 malfunctions and affects the live streaming quality, the primary live streaming backend 602 updates the pull address sent to the user terminal device 603 to the address of the secondary node of any secondary live streaming backend 701, without affecting the normal operation of the live streaming service and without the user's awareness.

[0135] To implement the methods of the embodiments of this application, the embodiments of this application also provide a user terminal device. Figure 8 This is only an exemplary structure of the user terminal device, not the entire structure; implementation is possible as needed. Figure 8 The structure shown may be part or all of the structure.

[0136] like Figure 8 As shown, the user terminal device 800 provided in this embodiment includes at least one processor 801, a memory 802, a user interface 803, and at least one network interface 804. The various components in the user terminal device 800 are coupled together via a bus system 805. It can be understood that the bus system 805 is used to implement communication between these components. In addition to a data bus, the bus system 805 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 8 The general labeled all buses as Bus System 805.

[0137] The user interface 803 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0138] The memory 802 in this embodiment is used to store various types of data to support the operation of the user terminal device 800. Examples of such data include any computer program used to operate on the user terminal device 800.

[0139] The live streaming method for user terminal device 800 disclosed in this application can be applied to processor 801, or implemented by processor 801. Processor 801 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the live streaming method for user terminal device can be completed by integrated logic circuits in the hardware or by instructions in the software form of processor 801. The processor 801 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 801 can implement or execute the methods, steps, and logic block diagrams disclosed in this application for user terminal device. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in memory 802. The processor 801 reads the information in memory 802 and, in conjunction with its hardware, completes the steps of the live streaming method for user terminal devices provided in this application embodiment.

[0140] In an exemplary embodiment, the user terminal device 800 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the aforementioned live streaming method applied to the user terminal device.

[0141] It is understood that memory 802 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Sync Link Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0142] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 802 storing a computer program. This computer program can be executed by the processor 801 of the user terminal device 800 to complete the steps described in the live streaming method applied to the user terminal device 800 according to this application embodiment. The computer-readable storage medium can be a ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM, etc.

[0143] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by the processor 801 of the user terminal device 800 to complete the steps described in the live streaming method applied to the user terminal device 800 in this application embodiment.

[0144] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0145] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0146] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A live pull streaming method, characterized in that, The method is applied to a user terminal device, and comprises: obtaining first information of a live stream; determining that the first information triggers a preset protocol switching condition, and then switching a pulling protocol of the live stream; wherein the protocol switching condition comprises a first protocol switching condition of switching the pulling protocol based on a live keyword and / or a second protocol switching condition of switching the pulling protocol based on a live configuration parameter, and the first information comprises the live keyword and / or the live configuration parameter.

2. The method of claim 1, wherein, The method further comprises: setting an initial pulling protocol of the live stream as a real-time message transmission (RMTP) protocol.

3. The method of claim 1, wherein, The obtaining of the first information of the live stream comprises: obtaining the live keyword appearing based on video data of the live stream; The determining that the first information triggers the preset protocol switching condition, and then switching the pulling protocol of the live stream comprises: determining that the live keyword triggers the first protocol switching condition, and then switching the pulling protocol of the live stream.

4. The method of claim 3, wherein, The determining that the live keyword triggers the first protocol switching condition, and then switching the pulling protocol of the live stream comprises: obtaining a current first pulling protocol of the live stream; determining that the live keyword matches any keyword in a preset keyword list, and then determining a corresponding second pulling protocol from the first protocol switching condition based on the first pulling protocol and the matched keyword; switching the pulling protocol of the live stream to the second pulling protocol.

5. The method of claim 4, wherein, The method further comprises: based on a deterministic finite automaton (DFA) algorithm, traversing the keyword list to determine whether the live keyword matches any keyword in the keyword list.

6. The method of claim 1, wherein, The determining that the first information triggers the preset protocol switching condition, and then switching the pulling protocol of the live stream comprises: determining that the live configuration parameter triggers the second protocol switching condition, and then switching the pulling protocol of the live stream.

7. The method of claim 6, wherein, The live configuration parameter comprises one or more of the following: definition, code rate, resolution, and live duration, and the determining that the live configuration parameter triggers the second protocol switching condition, and then switching the pulling protocol of the live stream comprises: determining that the definition in the live configuration parameter is greater than a set definition threshold, and then switching the pulling protocol of the live stream to an HTTP live streaming (HLS) protocol; and / or, determining that the code rate in the live configuration parameter is greater than a set code rate threshold, and / or, determining that the resolution in the live configuration parameter is greater than a set resolution threshold, and then switching the pulling protocol of the live stream to a dynamic adaptive streaming over HTTP (DASH) protocol; and / or, determining that the live duration in the live configuration parameter is greater than a set duration threshold; if it is determined that the current live duration reaches the set duration threshold, then switching the pulling protocol of the live stream to a web real-time communication (WebRTC) protocol; wherein the second protocol switching condition sets one or more of the following: the set definition threshold, the set code rate threshold, the set resolution threshold, and the set duration threshold.

8. The method of claim 1, wherein, The method further comprises: obtaining live monitoring information of a live stream; switch a pull streaming protocol of the live streaming based on live monitoring information of the live streaming; wherein the live monitoring information comprises at least one of the following: a current latency of the live streaming and a browser type used for playing live streaming.

9. The method of claim 8, wherein, The live configuration information comprises a latency threshold, and the condition for switching the pull streaming protocol of the live streaming based on the live monitoring information of the live streaming comprises: determining, based on the live monitoring information, that the current latency of the live streaming is greater than the latency threshold, that the condition for switching the pull streaming protocol of the live streaming is HTTP streaming format HTTP-FLV protocol; and / or determining, based on the live monitoring information, that the browser type used for playing live streaming by the user terminal device does not belong to a set of specified browsers, that the condition for switching the pull streaming protocol of the live streaming is HTTP-FLV protocol.

10. A live pull stream device, comprising: An application client device, the apparatus comprising: an acquisition module configured to acquire first information of a live streaming; a determination module configured to determine that the first information triggers a preset protocol switching condition; a switching module configured to switch a pull streaming protocol of the live streaming based on the first information triggering the preset protocol switching condition; wherein the protocol switching condition comprises a first protocol switching condition for switching the pull streaming protocol based on a live keyword and / or a second protocol switching condition for switching the pull streaming protocol based on a live configuration parameter, and the first information comprises the live keyword and / or the live configuration parameter.

11. A live system, characterized by The live system comprises a live client device, a first live background, and a user terminal device, wherein the live client device is configured to initiate a live service and push a live streaming to the first live background; the first live background is configured to transcode the live streaming and send a pull streaming address to the user terminal device; the user terminal device is configured to acquire the live streaming based on the pull streaming address; and the user terminal device is further configured to perform the steps of the method according to any one of claims 1 to 9.

12. A user equipment, characterized by The user terminal device is configured to perform a live service, and comprises a processor and a memory for storing a computer program capable of running on the processor, wherein the processor is configured to execute the computer program to perform the steps of the method according to any one of claims 1 to 9.

13. A storage medium, characterized by The storage medium has a computer program stored thereon, and the computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 9.

14. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method according to any one of claims 1 to 9.

Images