Multimedia data stream processing method and apparatus, electronic device and storage medium

Abstract

Embodiments of the present disclosure provide a multimedia data stream processing method and apparatus, an electronic device and a storage medium. The method comprises: receiving stream pushing related data sent by a stream pushing node, wherein the stream pushing related data corresponds to a pushed multimedia data stream; determining a target processing policy on the basis of the stream pushing related data and reception related data of a received processed multimedia data stream forwarded by a trusted third party; and when the target processing policy is a stream repushing policy, creating a data transmission link corresponding to the trusted third party and a current receiver so as to push, on the basis of the created data transmission link, the multimedia data stream pushed by the stream pushing node to the trusted third party. The technical solution provided by the embodiments of the present disclosure can ensure that the trusted third party forwards the multimedia data stream to the current receiver, achieving the effect of improving the forwarding service quality and stability.

Description

Multimedia data stream processing methods, apparatuses, electronic devices and storage media

[0001] This application claims priority to Chinese Patent Application No. 202410487579.2, filed on April 22, 2024, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to a multimedia data stream processing method, apparatus, electronic device, and storage medium. Background Technology

[0003] In live streaming scenarios, after the broadcaster collects the multimedia data stream, it can push the stream to edge nodes (pushing nodes). Under certain business requirements, the pushing nodes may need to forward the multimedia data stream to other services so that these services can process it.

[0004] During the forwarding of multimedia data streams, at least one trusted third party will be involved to process the received multimedia data stream and push the processed multimedia data stream to other services.

[0005] The aforementioned trusted third parties are mostly not controlled by the edge servers corresponding to the edge nodes. Therefore, it is impossible to determine whether the trusted third parties push multimedia data streams to other services, resulting in abnormal multimedia data stream pushes. Consequently, viewers may experience black screens and stuttering when watching the multimedia data streams they pull, which means there are issues with the stability of the live streaming service. Summary of the Invention

[0006] This disclosure provides a multimedia data stream processing method, apparatus, electronic device, and storage medium to ensure the quality and stability of the forwarding service during the forwarding of multimedia data streams.

[0007] In a first aspect, embodiments of this disclosure provide a multimedia data stream processing method, the method comprising:

[0008] Receive streaming association data sent by the streaming node, wherein the streaming association data corresponds to the pushed multimedia data stream;

[0009] Based on the push-stream association data and the received association data of the processed multimedia data stream forwarded by the trusted third party, a target processing strategy is determined; wherein, the trusted third party is used to process the pushed multimedia data stream.

[0010] When the target processing strategy is a re-pushing strategy, a data transmission link is created between the trusted third party and the current receiver, so as to push the multimedia data stream pushed by the push node to the trusted third party according to the created data transmission link.

[0011] Secondly, embodiments of this disclosure also provide a multimedia data stream processing device configured in an editor, the device comprising:

[0012] The associated data receiving module is used to receive the associated data sent by the streaming node, wherein the associated data corresponds to the pushed multimedia data stream;

[0013] The strategy determination module is used to determine a target processing strategy based on the push-stream association data and the received association data of the processed multimedia data stream forwarded by the trusted third party; wherein the trusted third party is used to process the pushed multimedia data stream.

[0014] The link creation module is used to create a data transmission link corresponding to the trusted third party and the current receiver when the target processing strategy is a push stream strategy, so as to push the multimedia data stream pushed by the push stream node to the trusted third party according to the created data transmission link.

[0015] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0016] One or more processors;

[0017] Storage device for storing one or more programs.

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the multimedia data stream processing method as described in any embodiment of this disclosure.

[0019] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the multimedia data stream processing method as described in any of the embodiments of this disclosure. Attached Figure Description

[0020] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0021] Figure 1 is a schematic flowchart of a multimedia data stream processing method provided in an embodiment of this disclosure;

[0022] Figure 2 is a schematic flowchart of a multimedia data stream processing method provided in an embodiment of this disclosure;

[0023] Figure 3 is a schematic flowchart of a multimedia data stream processing method provided in an embodiment of this disclosure;

[0024] Figure 4 is a schematic diagram of a multimedia data stream processing device provided in an embodiment of this disclosure; and

[0025] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0026] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0027] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0028] The term "comprising" and its variations as used in this disclosure are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the description below.

[0029] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0030] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0031] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0032] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0033] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0034] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0035] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0036] It is understood that the data involved in this disclosed technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0037] Before introducing the technical solutions provided in the embodiments of this disclosure, the application scenario can be illustrated first. In the process of a streaming node pushing a multimedia data stream to the current recipient, the multimedia data stream needs to be sent to a trusted third party first. After processing the multimedia data stream, the trusted third party obtains a processed multimedia data stream. The trusted third party can then send the processed multimedia data stream to other services. The trusted third party can act as an intermediary. Since the intermediary is mostly not controlled by the edge nodes, when the stream is forwarded from the intermediary to the current recipient, there may be situations where the streaming node pushes the stream normally to the intermediary, but a problem in the intermediary causes it to fail to push the stream to the current recipient. In other words, when a problem occurs in the intermediary, a forwarding anomaly may occur. Problems in the intermediary can include delayed forwarding of data, the intermediary receiving data normally but not forwarding it, resulting in data loss (similar to a black hole in the intermediary), or the intermediary making forwarding errors due to configuration issues, causing the current recipient not to receive the multimedia data stream. In short, the introduction of an intermediary affects the stability of the forwarding link. Typically, because the intermediate link can receive multimedia data streams normally, it does not report problems to the push node. Therefore, the push node cannot detect the anomaly in the push forwarding and cannot take corresponding recovery measures, thus affecting the quality and stability of the push forwarding service. In this embodiment, the intermediate link can be a trusted third party such as a gateway, proxy, or load balancer.

[0038] Therefore, the stability of the push link can be ensured by using the technical solutions provided in the embodiments of this disclosure.

[0039] Figure 1 is a schematic flowchart of a multimedia data stream processing method provided in an embodiment of this disclosure. This embodiment can be applied to scenarios where a trusted third party processes the multimedia data stream pushed by the streaming node and then forwards it to other service nodes. This method can be implemented by a multimedia data stream processing device, which can be deployed in a computer device. Optionally, the computer device can be a PC or a mobile terminal, etc.

[0040] As shown in Figure 1, the method includes:

[0041] S110, Receive push-stream related data sent by the push-stream node.

[0042] During live streaming at the broadcaster's end, multimedia data streams corresponding to the live stream can be collected in real time and pushed to edge nodes, i.e., to the streaming nodes. These streaming nodes can then send the multimedia data streams to other service nodes, allowing them to further process the data. Optional processing may include transcoding, recording, etc. Other service nodes may include origin nodes. For some services, the multimedia data stream needs to be sent to an intermediary before being pushed to other service nodes. Optionally, this intermediary can be a trusted third party. This intermediary processes the multimedia data and then pushes it to other service nodes, which can then act as the current recipients.

[0043] To determine whether the intermediate link has pushed the processed multimedia data stream to the current receiver, the push node can periodically send push association data to the current receiver. This allows the current receiver to determine whether the trusted third party has a push problem based on the received push association data and the received association information of the processed multimedia data stream sent by the trusted third party.

[0044] In this embodiment, the streaming association data may include the live timestamp of the multimedia data stream that the streaming node has pushed to a trusted third party and / or the push timestamp of the multimedia data stream. That is, the streaming association data includes the live timestamp and / or push timestamp of the multimedia data stream. The live timestamp corresponds to the playback time of the pushed multimedia data stream, and the push timestamp is the push time of the pushed multimedia data stream. The playback time is relative to the entire live stream duration, while the push time is an absolute time.

[0045] Specifically, the streaming node can periodically send push-related data of the pushed multimedia data stream to the current receiver. Optionally, push-related data can be sent once every second, or push-related data can be sent to the current receiver once every time a pushed multimedia data stream is sent.

[0046] It should be noted that the transmission link corresponding to the push-related data sent by the push-stream node to the current receiver is different from the data transmission link of the multimedia data stream sent by the push-stream node to the trusted third party, and is also different from the data transmission link of the processed multimedia data stream sent by the trusted third party to the current receiver.

[0047] It should be noted that the streaming node cannot directly push the multimedia data stream to the current recipient because it needs to be processed again by a trusted third party. However, the streaming node and the current recipient can communicate directly to send streaming-related data.

[0048] S120. Determine the target processing strategy based on the push-stream associated data and the received associated data of the processed multimedia data stream forwarded by the trusted third party.

[0049] Typically, the streaming node pushes data to a trusted third party, which processes the data and then sends it back to the current recipient. The latency involved in this process is very short and can be ignored. Therefore, when the current recipient receives the streaming-related data, it can determine the received related data based on the processed multimedia data stream forwarded by the trusted third party. The received related data may include the live timestamp corresponding to the received multimedia data stream. The target processing strategy can be understood as the final strategy for processing the multimedia data stream.

[0050] Specifically, upon receiving push-related data, it is possible to determine whether there is a data forwarding failure between the trusted third party and the current receiver based on the received associated data of the processed multimedia data stream forwarded by the trusted third party, and thus determine the target processing strategy for the forwarding.

[0051] S130. When the target processing strategy is a push-stream strategy, a data transmission link is created between the trusted third party and the current receiver, so as to push the multimedia data stream pushed by the push-stream node to the trusted third party according to the created data transmission link.

[0052] The re-pushing strategy can be understood as the strategy for the streaming node to re-push the multimedia data stream. When the target processing strategy is the re-pushing strategy, it indicates a data transmission link failure between the trusted third party and the current receiver. In this case, the data transmission link between the trusted third party and the current receiver can be recreated.

[0053] Specifically, if the target processing strategy is a re-pushing strategy, it indicates a fault in the original data transmission link between the trusted third party and the current receiver. In this case, the data transmission link between the trusted third party and the current receiver can be recreated. The trusted third party can then use this newly created data transmission link to push the streaming node to the multimedia data stream processing stage before forwarding it to the current receiver.

[0054] Based on the above, after sending and receiving associated data at both ends of the push link (one end being the push node and the other end being the current receiver), real-time information on the push quality of the trusted third party can be determined (e.g., latency, whether the current receiver has received the data). Furthermore, based on this real-time information, a corresponding push strategy can be determined.

[0055] The technical solution of this disclosure, in the process of forwarding multimedia data streams based on a push node, can determine whether there is a push failure between the trusted third party and the current receiver based on the push-related data sent by the push node and the reception association information of the processed multimedia data stream pushed by the trusted third party. If so, the target processing strategy can be determined to be a re-pushing strategy. In the case of a re-pushing strategy, the transmission link between the trusted third party and the current receiver can be recreated, so that the trusted third party can continue to push the corresponding multimedia data stream based on the newly created data transmission link, thereby achieving smooth forwarding of multimedia data streams and improving the quality and stability of the forwarding service.

[0056] Figure 2 is a flowchart illustrating a multimedia data stream processing method provided in an embodiment of this disclosure. Based on the foregoing embodiments, the following can be further refined: "determining a target processing strategy based on push-stream associated data and received associated data of the processed multimedia data stream forwarded by a trusted third party". For specific implementation details, please refer to the detailed description of this embodiment. Technical features that are the same as or similar to those in the foregoing embodiments will not be repeated here.

[0057] As shown in Figure 2, the method of this embodiment may specifically include:

[0058] S210. Receive push-stream associated data sent by the push-stream node, wherein the push-stream associated data corresponds to the pushed multimedia data stream.

[0059] S220. Based on the first time information in the push-stream associated data and the second time information of the processed multimedia data stream in the received associated data, determine the push-stream delay duration corresponding to the current receiver.

[0060] The first time information includes the playback time corresponding to the playback duration of the sent multimedia data stream and / or the transmission time of sending the played multimedia data stream. The second time information includes the playback time and / or reception time of the processed multimedia data stream received by the current receiver. Based on the playback times in the first and second time information, a playback delay duration is determined. This playback delay duration can be used as the push streaming delay duration corresponding to the current receiver. The push streaming delay duration is used to determine whether the trusted third party has pushed the processed multimedia data stream to the current receiver. That is, the push streaming delay duration is used to assess whether there is a data transmission failure between the trusted third party and the current receiver.

[0061] Specifically, the delay between two playback times can be determined based on the playback time corresponding to the sent multimedia data stream in the push association data and the playback time of the processed multimedia data stream forwarded by the authorized third party in the received association data. This delay can be used as the push streaming delay.

[0062] S230. Determine the target processing strategy based on the streaming delay duration and the preset delay duration threshold.

[0063] The preset delay threshold is predetermined and is used to determine the duration of any link transmission failure between the trusted third party and the current receiver. The target processing strategy is ultimately determined and is used to determine how the streaming node processes the multimedia data stream.

[0064] Specifically, based on the relationship between a pre-set preset delay threshold and the push stream delay, it can be determined how the push stream node processes the multimedia data stream; and whether there is a data link transmission failure between the current receiver and the trusted third party. Based on this, the final target processing strategy can be determined.

[0065] Based on the above technical solution, the target processing strategy is determined based on the push stream delay duration and the preset delay duration threshold. This can be as follows: if the push stream delay duration is greater than or equal to the preset delay duration threshold, the target processing strategy is determined to be a re-push stream strategy; if the push stream delay duration is less than the preset delay duration threshold, the target processing strategy is determined to be a delay re-determination strategy. The delay re-determination strategy is used to determine whether to create a data transmission link.

[0066] This can be understood as follows: If the streaming delay is greater than or equal to a preset delay threshold, it indicates a data transmission link failure between the trusted third party and the current receiver. In this case, the target processing strategy can be determined as a re-pushing strategy. A re-pushing strategy can be understood as a strategy where the streaming node needs to re-push the multimedia data stream. If the streaming delay is less than the preset delay threshold, it can be preliminarily determined that there may be a data transmission link failure between the trusted third party and the current receiver. At this point, it can be further determined whether a data transmission link exists. That is, the target processing strategy is determined as a delay re-determination strategy. A delay re-determination strategy can be understood as a strategy to re-determine whether to create a data transmission link. The data transmission link at this time is the data transmission link between the trusted third party and the current receiver.

[0067] Based on the above technical solution, after determining the target processing strategy as a delay re-determination strategy, the method further includes: if the push frame rate in the push-related data and the receive frame rate in the receive-related data do not meet preset conditions, determining the delay re-determination strategy as a re-pushing strategy; if the push frame rate in the push-related data and the receive frame rate in the receive-related data meet preset conditions, determining the delay re-determination strategy as a continue-pushing strategy; wherein, the continue-pushing strategy is to continue pushing the processed multimedia data stream using the original data transmission link.

[0068] The preset condition is that the difference between the push frame rate and the received frame rate is greater than a preset frame rate threshold. The re-pull strategy is the strategy for when the push node needs to re-push the multimedia data stream. The continue-pull strategy can be understood as the push node can continue to push the multimedia data stream if there is no failure in the data transmission link.

[0069] Specifically, the frame rate difference can be determined based on the push frame rate in the push-related data and the receive frame rate in the receive-related data. If the frame rate difference does not meet the preset conditions, the delay re-determination strategy can be determined as a re-pushing strategy. If the frame rate difference meets the preset conditions, the delay re-determination strategy can be determined as a continue-pushing strategy, that is, if there are no gaps in the data transmission link, the pushing node and the trusted third party can continue to push multimedia data streams.

[0070] S240. When the target processing strategy is a push-stream strategy, a data transmission link is created between the trusted third party and the current receiver, so as to push the multimedia data stream pushed by the push-stream node to the trusted third party according to the created data transmission link.

[0071] In other words, when it is determined that there is a data link transmission failure between the trusted third party and the current recipient, a new data transmission link can be created or the original data transmission link can be restarted to continue transmitting multimedia data streams based on the newly created or restarted data transmission link.

[0072] The technical solution of this disclosure, in the process of forwarding multimedia data streams based on a push node, can determine whether there is a push failure between the trusted third party and the current receiver based on the push-related data sent by the push node and the reception association information of the processed multimedia data stream pushed by the trusted third party. If so, the target processing strategy can be determined to be a re-pushing strategy. In the case of a re-pushing strategy, the transmission link between the trusted third party and the current receiver can be recreated, so that the trusted third party can continue to push the corresponding multimedia data stream based on the newly created data transmission link, thereby achieving smooth forwarding of multimedia data streams and improving the quality and stability of the forwarding service.

[0073] Figure 3 is a flowchart illustrating a multimedia data stream processing method provided in an embodiment of this disclosure. Based on the foregoing embodiments, after determining that the target strategy is a re-pushing strategy, it is possible to determine how to process the multimedia data stream. For specific implementation methods, please refer to the detailed description of the embodiments of this disclosure. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated in this embodiment.

[0074] As shown in Figure 3, the method includes:

[0075] S310. Receive push-stream associated data sent by the push-stream node, wherein the push-stream associated data corresponds to the pushed multimedia data stream.

[0076] S320. Determine the target processing strategy based on the push-stream associated data and the received associated data of the processed multimedia data stream forwarded by the trusted third party.

[0077] The authorized third party is used to process the pushed multimedia data stream.

[0078] S330. When the target processing strategy is a re-pull streaming strategy, determine the re-pull streaming information.

[0079] The re-pushing stream information may include the time point at which the inference node re-pushes the multimedia data stream.

[0080] In this embodiment, determining the re-pushing information can be done by: determining the re-pushing information corresponding to the pushing node based on the pushing associated data and the receiving associated data; or by using the time point corresponding to determining the target strategy as the re-pushing strategy as the information in the re-pushing information.

[0081] This can be understood as follows: there are at least two ways to determine the re-streaming information. The first way is to determine the re-streaming information based on the push-streaming associated data and the receiving associated data. The second way is to use the playback time corresponding to the determination of the re-streaming strategy as the information in the re-streaming information.

[0082] In this embodiment of the disclosure, the re-pushing information corresponding to the pushing node is rated based on the pushing associated data and the receiving associated data, including: if the first time information in the pushing associated data is later than the second time information in the receiving associated data, the second time information is used as the information in the re-pushing information.

[0083] In other words, if the first time information is later than the second time information, it means that the streaming node has pushed more multimedia data streams than the authorized third party has pushed processed multimedia data streams to the current receiver. In this case, the second time information can be used as the information in the re-pushing stream information so that reasoning can be re-based on the second time information, thereby ensuring that no multimedia data streams are missed and improving the picture quality when the client watches the live stream.

[0084] S340 feeds back the re-pull information to the push node, so that the push node can determine whether to continue pushing the multimedia data stream to the trusted third party based on the re-pull information.

[0085] This can be understood as the current receiver being able to feed back the re-pull information to the push node, so that the push node can determine the multimedia data stream that the push node is pushing to the trusted third party based on the re-pull information.

[0086] After the streaming node pushes the corresponding multimedia data stream based on the re-pushing information, the receiver can re-verify the processed multimedia data stream sent by the trusted third party to ensure that no multimedia data stream has been missed.

[0087] Optionally, based on the data stream time information corresponding to the target multimedia data stream transmitted by the created data transmission link, and the second time information of the processed multimedia data stream received when the re-pushing stream strategy is determined, a target processing method for the target multimedia data stream is determined; and the target multimedia data stream is processed based on the target processing method.

[0088] The streaming node will use the multimedia data stream re-pushed based on the re-pushing information as the target multimedia data stream. Of course, after being pushed to the trusted third party, the target multimedia data stream can be considered as a previously pushed multimedia data stream. The target multimedia data stream corresponds to a specific timestamp, i.e., the playback timestamp. After the trusted third party processes the target multimedia data stream, it can transmit the processed target multimedia data stream to the current receiver via a newly created data transmission link. The current receiver can determine whether to process the target multimedia data stream based on its data stream time information and the second time information corresponding to the re-pushing strategy.

[0089] In this embodiment, based on the data stream time information corresponding to the target multimedia data stream transmitted by the created data transmission link, and the second time information of the processed multimedia data stream received when the re-pushing strategy is determined, the target processing method for the target multimedia data stream is determined, including: when the data stream time information is later than the second time information, the target processing method is determined to be to determine the re-pushing information based on the second time information, so that the pushing node re-pushes the target multimedia data stream based on the re-pushing information; when the data stream time information is earlier than the second time information, the target processing method is determined to perform deduplication processing on the target multimedia data stream based on the second time information.

[0090] This can be understood as follows: the target processing method is based on the data stream time information and second time information of the target multimedia data stream sent by the trusted third party to determine how to process the target multimedia data stream. The target processing method includes methods for redefining the target multimedia data stream and methods for deduplicating the target multimedia data stream.

[0091] S350. Create a data transmission link between the trusted third party and the current recipient, so that the trusted third party can push the processed multimedia data stream corresponding to the multimedia data stream based on the created data transmission link.

[0092] In this embodiment, after creating the data transmission link corresponding to the trusted third party and the current receiver, the method further includes: disconnecting the original transmission link between the trusted third party and the current receiver, so as to transmit the processed multimedia data stream based on the created data transmission link.

[0093] This can be understood as disconnecting the original transmission link between the trusted third party and the current recipient, and transmitting the processed multimedia data based on a newly created data transmission link.

[0094] The technical solution of this disclosure, in the process of forwarding multimedia data streams based on a push node, can determine whether there is a push failure between the trusted third party and the current receiver based on the push-related data sent by the push node and the reception association information of the processed multimedia data stream pushed by the trusted third party. If so, the target processing strategy can be determined to be a re-pushing strategy. In the case of a re-pushing strategy, the transmission link between the trusted third party and the current receiver can be recreated, so that the trusted third party can continue to push the corresponding multimedia data stream based on the newly created data transmission link, thereby achieving smooth forwarding of multimedia data streams and improving the quality and stability of the forwarding service.

[0095] Figure 4 is a schematic diagram of a multimedia data stream processing device provided in an embodiment of this disclosure. As shown in Figure 4, the device includes: an associated data receiving module 410, a strategy determination module 420, and a link creation module 430.

[0096] The associated data receiving module 410 is used to receive push-stream associated data sent by the push-stream node, wherein the push-stream associated data corresponds to the already pushed multimedia data stream; the strategy determination module 420 is used to determine the target processing strategy based on the push-stream associated data and the received associated data of the processed multimedia data stream forwarded by the trusted third party; wherein the trusted third party is used to process the already pushed multimedia data stream; the link creation module 430 is used to create a data transmission link corresponding to the trusted third party and the current receiver when the target processing strategy is a re-push-stream strategy, so as to push the multimedia data stream pushed by the push-stream node to the trusted third party according to the created data transmission link.

[0097] Based on the above technical solution, the push-stream related data includes first-time information corresponding to the pushed multimedia data stream, and the strategy determination module includes:

[0098] The delay duration determination unit is used to determine the current push delay duration corresponding to the receiver based on the first time information in the push-stream associated data and the second time information of the processed multimedia data stream in the received associated data; the strategy determination unit is used to determine the target processing strategy based on the push delay duration and the preset delay duration threshold.

[0099] Based on the above technical solutions, the strategy determination unit includes:

[0100] The re-pushing strategy determination sub-unit is used to determine the target processing strategy as the re-pushing strategy when the streaming delay duration is greater than or equal to a preset delay duration threshold.

[0101] The re-determination strategy determination subunit is used to determine the target processing strategy as the delay re-determination strategy when the push latency is less than a preset latency threshold; wherein, the delay re-determination strategy is used to determine whether to create a data transmission link.

[0102] Based on the above technical solution, the device also includes:

[0103] The re-pushing strategy determination module is used to determine the delay re-determination strategy as the re-pushing strategy when the push frame rate in the push-related data and the receive frame rate in the receive-related data do not meet the preset conditions.

[0104] The continue streaming strategy determination module is used to determine the delay re-determination strategy as the continue streaming strategy when the push frame rate in the push-related data and the receive frame rate in the receive-related data meet preset conditions; wherein, the continue streaming strategy is to continue pushing the processed multimedia data stream by using the original data transmission link.

[0105] Based on the above technical solutions, the device also includes:

[0106] The first information determination module is used to determine the re-pull stream information;

[0107] The push module is used to feed back the re-pull stream information to the push stream node, so that the push stream node can determine to continue pushing the multimedia data stream to the trusted third party based on the re-pull stream information, and enable the trusted third party to push the processed multimedia data stream corresponding to the multimedia data stream based on the created data transmission link.

[0108] Based on the above technical solutions, the first information determination module is further used to: determine the re-pushing information corresponding to the pushing node based on the pushing associated data and the receiving associated data; or, use the time point corresponding to the determination of the target strategy as the re-pushing strategy as the information in the re-pushing information.

[0109] Based on the above technical solutions, the first information determination module is further used to, when the first time information and / or the second time information in the push-stream associated data is later than the third time information and / or the fourth time information in the received associated data, use the time information that is furthest from the current time in the third time information and / or the fourth time information as the information in the re-pushing stream information.

[0110] Based on the above technical solution, the device also includes:

[0111] The processing method determination module is used to determine the target processing method for the target multimedia data stream based on the data stream time information corresponding to the target multimedia data stream transmitted by the created data transmission link, and the third time information of the processed multimedia data stream received when the re-pushing stream strategy is determined.

[0112] The first processing module is used to process the target multimedia data stream based on the target processing method.

[0113] Based on the above technical solutions, the processing method determination module is also used for:

[0114] When the data stream time information is later than the third time information, the target processing method is determined to be to determine the re-pull stream information based on the third time information;

[0115] If the data stream time information is earlier than the third time information, the target processing method is to deduplicatize the target multimedia data stream based on the third time information.

[0116] Based on the above technical solutions, the device also includes:

[0117] The link disconnect module is used to disconnect the existing transmission link between the trusted third party and the current receiver, so as to transmit multimedia data streams based on the created data transmission link.

[0118] The technical solution of this disclosure, in the process of forwarding multimedia data streams based on a push node, can determine whether there is a push failure between the trusted third party and the current receiver based on the push-related data sent by the push node and the reception association information of the processed multimedia data stream pushed by the trusted third party. If so, the target processing strategy can be determined to be a re-pushing strategy. In the case of a re-pushing strategy, the transmission link between the trusted third party and the current receiver can be recreated, so that the trusted third party can continue to push the corresponding multimedia data stream based on the newly created data transmission link, thereby achieving smooth forwarding of multimedia data streams and improving the quality and stability of the forwarding service.

[0119] The multimedia data stream processing apparatus provided in this disclosure can execute the multimedia data stream processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0120] It is worth noting that the various units and functional modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0121] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Referring to Figure 5 below, a schematic diagram of the structure of an electronic device (e.g., the terminal device or server in Figure 5) 500 suitable for implementing embodiments of this disclosure is shown. The terminal device in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in Figure 5 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this disclosure.

[0122] As shown in Figure 5, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An edit / output (I / O) interface 505 is also connected to the bus 504.

[0123] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 shows an electronic device 500 with various devices, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0124] According to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0125] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0126] The electronic device provided in this embodiment and the multimedia data stream processing method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0127] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the multimedia data stream processing method provided in the above embodiments.

[0128] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0129] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0130] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0131] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0132] Receive streaming association data sent by the streaming node, wherein the streaming association data corresponds to the pushed multimedia data stream;

[0133] Based on the push-stream association data and the received association data of the processed multimedia data stream forwarded by the trusted third party, a target processing strategy is determined; wherein, the trusted third party is used to process the pushed multimedia data stream.

[0134] When the target processing strategy is a re-pushing strategy, a data transmission link is created between the trusted third party and the current receiver, so as to push the multimedia data stream pushed by the push node to the trusted third party according to the created data transmission link.

[0135] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a functional module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0137] According to one or more embodiments of this disclosure, Example 1 provides a multimedia data stream processing method, the method comprising:

[0138] Receive streaming association data sent by the streaming node, wherein the streaming association data corresponds to the pushed multimedia data stream;

[0139] Based on the push-stream association data and the received association data of the processed multimedia data stream forwarded by the trusted third party, a target processing strategy is determined; wherein, the trusted third party is used to process the pushed multimedia data stream.

[0140] When the target processing strategy is a re-pushing strategy, a data transmission link is created between the trusted third party and the current receiver, so as to push the multimedia data stream pushed by the push node to the trusted third party according to the created data transmission link.

[0141] According to one or more embodiments of this disclosure, Example 2 provides a multimedia data stream processing method, the method comprising:

[0142] Optionally, the push-stream association data includes first-time information corresponding to the pushed multimedia data stream. The step of determining the target processing strategy based on the push-stream association data and the received association data of the processed multimedia data stream forwarded by the trusted third party includes:

[0143] Based on the first time information in the push-streaming associated data and the second time information of the processed multimedia data stream in the received associated data, the push-streaming delay duration corresponding to the current receiver is determined;

[0144] The target processing strategy is determined based on the streaming delay duration and the preset delay duration threshold.

[0145] According to one or more embodiments of this disclosure, Example 3 provides a multimedia data stream processing method, the method comprising:

[0146] Optionally, determining the target processing strategy based on the streaming delay duration and a preset delay duration threshold includes:

[0147] If the push delay duration is greater than or equal to the preset delay duration threshold, the target processing strategy is determined to be a re-pull strategy;

[0148] If the push latency is less than the preset latency threshold, the target processing strategy is determined to be a latency re-determination strategy; wherein, the latency re-determination strategy is used to determine whether to create a data transmission link.

[0149] According to one or more embodiments of this disclosure, Example 4 provides a multimedia data stream processing method, the method comprising:

[0150] Optionally, after determining that the target processing strategy is a delayed redetermination strategy, the method further includes:

[0151] If the push frame rate in the push data and the receive frame rate in the receive data do not meet the preset conditions, the delay re-determination strategy is determined to be a re-push strategy.

[0152] If the push frame rate in the push data and the receive frame rate in the receive data meet preset conditions, the delay re-determination strategy is determined to be the continue push strategy.

[0153] The continued streaming strategy involves using the existing data transmission link to continue pushing the processed multimedia data stream.

[0154] According to one or more embodiments of this disclosure, Example 5 provides a multimedia data stream processing method, the method comprising:

[0155] Optionally, after determining that the target strategy is a re-pushing strategy, the method further includes:

[0156] Determine the re-pull stream information;

[0157] The re-pushing information is fed back to the streaming node, so that the streaming node can determine to continue pushing the multimedia data stream to the trusted third party based on the re-pushing information, and the trusted third party can push the processed multimedia data stream corresponding to the multimedia data stream based on the created data transmission link.

[0158] According to one or more embodiments of this disclosure, Example Six provides a multimedia data stream processing method, the method comprising:

[0159] Optionally, determining the re-pull stream information includes:

[0160] If the first time information in the push-stream associated data is later than the second time information in the received associated data, the second time information shall be used as the information in the re-pull-stream information; or,

[0161] The time point at which the target strategy is determined to be a re-pull strategy is used as information in the re-pull information.

[0162] According to one or more embodiments of this disclosure, Example 7 provides a multimedia data stream processing method, the method comprising:

[0163] Optionally, after receiving the multimedia data stream pushed by the push node to the trusted third party according to the established data transmission link, the method further includes:

[0164] Based on the data stream time information corresponding to the target multimedia data stream transmitted by the created data transmission link, and the second time information of the processed multimedia data stream received when the re-pushing stream strategy is determined, the target processing method for the target multimedia data stream is determined.

[0165] The target multimedia data stream is processed based on the target processing method.

[0166] According to one or more embodiments of this disclosure, Example 8 provides a multimedia data stream processing method, the method comprising:

[0167] Optionally, the determination of the target processing method for the target multimedia data stream based on the data stream time information corresponding to the target multimedia data stream transmitted by the created data transmission link, and the second time information of the processed multimedia data stream received when the re-pushing stream strategy is determined, includes:

[0168] When the data stream time information is later than the second time information, the target processing method is determined to be to determine the re-pushing information based on the second time information, so that the streaming node re-pushes the target multimedia data stream based on the re-pushing information;

[0169] When the data stream time information is earlier than the second time information, the target processing method is determined to be deduplication of the target multimedia data stream based on the second time information.

[0170] According to one or more embodiments of this disclosure, Example Nine provides a multimedia data stream processing method, the method comprising:

[0171] Optionally, after creating the data transmission link corresponding to the trusted third party and the current receiver, the method further includes:

[0172] Disconnect the original transmission link between the trusted third party and the current recipient to transmit the processed multimedia data stream based on the created data transmission link.

[0173] According to one or more embodiments of this disclosure, Example 10 provides a multimedia data stream processing apparatus, the apparatus comprising:

[0174] The associated data receiving module is used to receive the associated data sent by the streaming node, wherein the associated data corresponds to the pushed multimedia data stream;

[0175] The strategy determination module is used to determine a target processing strategy based on the push-stream association data and the received association data of the processed multimedia data stream forwarded by the trusted third party; wherein the trusted third party is used to process the pushed multimedia data stream.

[0176] The link creation module is used to create a data transmission link corresponding to the trusted third party and the current receiver when the target processing strategy is a push stream strategy, so as to push the multimedia data stream pushed by the push stream node to the trusted third party according to the created data transmission link.

[0177] The above description is merely an embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0178] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0179] Although this disclosure has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A multimedia data stream processing method, comprising: Receive streaming association data sent by the streaming node, wherein the streaming association data corresponds to the pushed multimedia data stream; Based on the push-stream association data and the received association data of the processed multimedia data stream forwarded by the trusted third party, a target processing strategy is determined; wherein, the trusted third party is used to process the pushed multimedia data stream. When the target processing strategy is a re-pushing strategy, a data transmission link is created between the trusted third party and the current receiver, so as to push the multimedia data stream pushed by the push node to the trusted third party according to the created data transmission link.

2. The method according to claim 1, wherein, The push-stream association data includes first-time information corresponding to the pushed multimedia data stream. The determination of the target processing strategy based on the push-stream association data and the received association data of the processed multimedia data stream forwarded by the trusted third party includes: Based on the first time information and the second time information of the processed multimedia data stream in the received associated data, the push delay duration corresponding to the current receiver is determined; The target processing strategy is determined based on the streaming delay duration and the preset delay duration threshold.

3. The method according to claim 2, wherein, The step of determining the target processing strategy based on the streaming delay duration and a preset delay duration threshold includes: If the push delay duration is greater than or equal to the preset delay duration threshold, the target processing strategy is determined to be a re-pull strategy; If the push latency is less than the preset latency threshold, the target processing strategy is determined to be a latency re-determination strategy; wherein, the latency re-determination strategy is used to determine whether to create a data transmission link.

4. The method according to claim 3, wherein, After determining that the target processing strategy is a delayed re-determination strategy, the multimedia data stream processing method further includes: If the push frame rate in the push data and the receive frame rate in the receive data do not meet the preset conditions, the delay re-determination strategy is determined to be a re-push strategy. If the push frame rate in the push data and the receive frame rate in the receive data meet preset conditions, the delay re-determination strategy is determined to be the continue push strategy. The continued streaming strategy involves using the existing data transmission link to continue pushing the processed multimedia data stream.

5. The method according to any one of claims 1-4, wherein, After determining that the target strategy is a re-pushing strategy, the multimedia data stream processing method further includes: Determine the re-pull stream information; The re-pushing information is fed back to the streaming node, so that the streaming node can determine to continue pushing the multimedia data stream to the trusted third party based on the re-pushing information, and the trusted third party can push the processed multimedia data stream corresponding to the multimedia data stream based on the created data transmission link.

6. The method according to claim 5, wherein, The determination of the re-pull stream information includes: If the first time information in the push-stream associated data is later than the second time information in the received associated data, the second time information shall be used as the information in the re-pull-stream information; or, The time point at which the target strategy is determined to be a re-pull strategy is used as information in the re-pull information.

7. The method according to any one of claims 1-6, wherein, After receiving the multimedia data stream pushed by the push node to the trusted third party according to the established data transmission link, the multimedia data stream processing method further includes: Based on the data stream time information corresponding to the target multimedia data stream transmitted by the created data transmission link, and the second time information of the processed multimedia data stream received when the re-pushing stream strategy is determined, the target processing method for the target multimedia data stream is determined. The target multimedia data stream is processed based on the target processing method.

8. The method according to claim 7, wherein, The determination of the target processing method for the target multimedia data stream based on the data stream time information corresponding to the target multimedia data stream transmitted by the created data transmission link, and the second time information of the processed multimedia data stream received when the re-pushing strategy is determined, includes: When the data stream time information is later than the second time information, the target processing method is determined to be to determine the re-pushing information based on the second time information, so that the streaming node re-pushes the target multimedia data stream based on the re-pushing information; When the data stream time information is earlier than the second time information, the target processing method is determined to be deduplication of the target multimedia data stream based on the second time information.

9. The method according to any one of claims 1-8, wherein, After establishing the data transmission link between the trusted third party and the current receiver, the multimedia data stream processing method further includes: Disconnect the original transmission link between the trusted third party and the current recipient to transmit the processed multimedia data stream based on the created data transmission link.

10. A multimedia data stream processing apparatus, comprising: The associated data receiving module is configured to receive push-stream associated data sent by the push-stream node, wherein the push-stream associated data corresponds to the pushed multimedia data stream; The strategy determination module is configured to determine a target processing strategy based on the push-stream association data and the received association data of the processed multimedia data stream forwarded by the trusted third party; wherein the trusted third party is used to process the pushed multimedia data stream. The link creation module is configured to create a data transmission link corresponding to the trusted third party and the current receiver when the target processing strategy is a push stream strategy, so as to push the multimedia data stream pushed by the push stream node to the trusted third party according to the created data transmission link.

11. An electronic device, comprising: One or more processors; Storage device, configured to store one or more programs, Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the multimedia data stream processing method as described in any one of claims 1-9.

12. A storage medium containing computer-executable instructions, wherein, The computer-executable instructions, when executed by a computer processor, are used to perform the multimedia data stream processing method as described in any one of claims 1-9.

Images