Method for providing multimedia data, related apparatus and computer program product

Abstract

A method, related apparatus and computer program product for providing multimedia data, the application periodically requesting a data frame from an external data source, the data frame comprising at least one of an audio frame and an image frame; in response to the fact that the data frame can be obtained from the external data source, decoding the data frame to obtain multimedia data; based on a first timestamp standard used by an external data source and a locally used second timestamp standard, the first timestamp of the data frame is converted into a second timestamp, the first timestamp belongs to the first timestamp standard, and the second timestamp belongs to the second timestamp standard; and generating target multimedia data based on the multimedia data and the second timestamp, and presenting the target multimedia data. Therefore, not only can the occupation of the data frames on the cache be reduced, but also a large number of data frames provided by different external data sources can be received in parallel, and the acquisition and presentation efficiency of the multimedia data is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a method and apparatus for providing multimedia data, an electronic device, a computer readable medium and a computer program product. BACKGROUND

[0002] With the rapid development of cloud computing, virtualization technology and high-bandwidth networks, cloud-based broadcasting control platforms are gradually becoming a new type of business adopted by the multimedia content providing industry.

[0003] A cloud-based broadcasting control platform is a broadcasting control system based on cloud computing technology, mainly used for video program production, editing and real-time broadcasting. The cloud-based broadcasting control platform centralizes and virtualizes functions such as video signal acquisition, processing, editing and distribution through a cloud platform, and can provide more flexible and efficient services. Unlike traditional hardware director stations that require a large number of dedicated equipment, the cloud-based broadcasting control platform can realize remote and automatic operation and control through the Internet, thereby realizing more efficient program production and broadcasting processes.

[0004] In this context, how to further improve the performance and efficiency of the cloud-based broadcasting control platform to facilitate users to use the cloud-based broadcasting control platform more efficiently and effectively is a concern and urgent need. SUMMARY

[0005] The aspects of the present application provide a method and apparatus for providing multimedia data, an electronic device, a computer readable storage medium and a computer program product, which can not only reduce the cache occupancy of data frames in the cloud-based broadcasting control platform by actively requesting data frames from external data sources and decoding the data frames immediately after receiving them, but also enable the multimedia data after transcoding and decoding the data frames to be presented to the user more quickly, and can also enable the cloud-based broadcasting control platform to receive a large number of data frames provided by different external data sources in parallel by adjusting the time stamp, thereby improving the efficiency of acquiring and presenting multimedia data by the cloud-based broadcasting control platform.

[0006] In an aspect of the present application, a method for providing multimedia data is provided, comprising: periodically requesting data frames from an external data source, wherein the data frames comprise at least one of audio frames and image frames; decoding the data frames to obtain multimedia data in response to being able to obtain the data frames from the external data source; converting a first time stamp possessed by the data frames into a second time stamp based on a first time stamp standard used by the external data source and a second time stamp standard used locally, wherein the first time stamp belongs to the first time stamp standard, and the second time stamp belongs to the second time stamp standard; generating target multimedia data based on the multimedia data and the second time stamp, and presenting the target multimedia data.

[0007] Another aspect of this application provides an apparatus for providing multimedia data, comprising: a data frame request module configured to periodically request data frames from an external data source, wherein the data frames include at least one of audio frames and image frames; a data frame decoding module configured to decode the data frames to obtain multimedia data in response to the ability to obtain the data frames from the external data source; a timestamp conversion module configured to convert a first timestamp of the data frame into a second timestamp based on a first timestamp standard used by the external data source and a second timestamp standard used locally, wherein the first timestamp belongs to the first timestamp standard and the second timestamp belongs to the second timestamp standard; and a multimedia data presentation module configured to generate target multimedia data based on the multimedia data and the second timestamp, and to present the target multimedia data.

[0008] In another aspect of this application, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of providing multimedia data as provided above.

[0009] Another aspect of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the method of providing multimedia data as provided above.

[0010] Another aspect of this application is a computer program product that includes a computer program having computer program instructions stored thereon, which, when executed by a processor, can implement the method of providing multimedia data as provided above.

[0011] In the solution provided in this application embodiment, firstly, data frames are periodically requested from an external data source. These data frames include at least one of audio frames and image frames. Then, in response to the ability to obtain data frames from the external data source, the data frames are decoded to obtain multimedia data. Next, based on a first timestamp standard used by the external data source and a second timestamp standard used locally, the first timestamp of the data frame is converted to a second timestamp, where the first timestamp belongs to the first timestamp standard and the second timestamp belongs to the second timestamp standard. Finally, target multimedia data is generated based on the multimedia data and the second timestamp, and the target multimedia data is presented. Therefore, not only can the buffering of data frames in the cloud broadcasting station be reduced by actively requesting from external data sources and immediately decoding them upon receipt, allowing the multimedia data after transcoding and decoding to be presented to the user more quickly, but the timestamp adjustment also enables the cloud broadcasting station to receive a large number of data frames from different external data sources in parallel, improving the efficiency of multimedia data acquisition and presentation. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart illustrating a process for providing multimedia data, as provided in an embodiment of this application; Figure 2 A flowchart illustrating a process for determining supplementary data frames, provided as an embodiment of this application; Figure 3 A flowchart illustrating the process of providing multimedia data in a specific application scenario according to an embodiment of this application; Figure 4 A schematic diagram of a device for providing multimedia data is provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device suitable for implementing the solutions in the embodiments of this application.

[0014] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] In a typical configuration of this application, the terminal and the service network devices each include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0017] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0018] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer program instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0019] As discussed above, how to further improve the performance and efficiency of cloud broadcasting platforms, and make it easier for users to utilize them more effectively and efficiently, is a matter of concern and urgent need.

[0020] In some solutions, the cloud broadcast console can be configured to use a buffer to temporarily store data frames obtained from external data sources. When a certain number of data frames have accumulated, or when the cloud broadcast console wants to process these data frames, the cloud broadcast console can retrieve these data frames from the buffer.

[0021] However, this approach not only requires the use of the cloud broadcast console's storage to set up buffers, increasing the deployment cost and requirements of the cloud broadcast console, but also prevents the cloud broadcast console from "knowing" in a timely manner whether there are data frames that can be processed. This results in delays in the cloud broadcast console's processing, transcoding, and decoding of data frames, thus reducing the processing efficiency of the cloud broadcast console.

[0022] To address this situation, this application provides a technical solution. First, it periodically requests data frames from an external data source, where each data frame includes at least one of audio and image frames. Then, in response to obtaining the data frames from the external data source, it decodes the data frames to obtain multimedia data. Next, based on a first timestamp standard used by the external data source and a second timestamp standard used locally, it converts the first timestamp of the data frame into a second timestamp, where the first timestamp belongs to the first timestamp standard and the second timestamp belongs to the second timestamp standard. Finally, it generates and presents target multimedia data based on the multimedia data and the second timestamp. This not only reduces the buffer usage of data frames in the cloud broadcasting station by actively requesting from external data sources and immediately decoding them upon receipt, allowing the multimedia data after transcoding and decoding to be presented to the user more quickly, but also enables the cloud broadcasting station to receive a large number of data frames from different external data sources in parallel through timestamp adjustment, improving the efficiency of multimedia data acquisition and presentation.

[0023] In practical scenarios, the execution entity of this method can be a user device implemented as a cloud broadcasting station, or a device composed of user devices and network devices integrated through a network, or an application running on the aforementioned devices. User devices include, but are not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and wristbands. Network devices include, but are not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0024] When the executing entity is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software programs or software modules, or as a single software program or software module, without specific limitations.

[0025] Figure 1 This application illustrates a process 100 for providing multimedia data, which includes at least the following processing steps: (Step) S101, periodically request data frames from an external data source; In the embodiments of this application, for example, a "server" or "cloud server" can be used as a "cloud broadcasting station" to implement the process of providing multimedia data provided in the embodiments of this application. That is, such a "cloud broadcasting station" can be used as the execution subject to implement the process of providing multimedia data based on the embodiments of this application.

[0026] In this step, the executing entity can periodically request data frames from an external data source. For example, the executing entity can send requests to an external data source that can provide content to the executing entity at fixed intervals (i.e., at intervals of a time period or cycle) to detect and inquire whether this additional external data source exists and has produced data frames that can be used to provide multimedia data or multimedia content.

[0027] The external data source can be, for example, an external device or signal source that provides multimedia content such as images, audio, and video, independent of the executing entity. For instance, the external data source could be a provider or copyright holder of a live stream for a sporting event or gala, or a provider or copyright holder of video content such as films or short videos.

[0028] The data frame can be at least one of an audio frame and an image frame. For example, in the case where a radio station is the external data source, the data frame can be an audio frame. For a live stream provider, as mentioned above, the data frame can be both an audio frame and an image frame, thus combining them to deliver live content in "video" format. It should be understood that there can be multiple external data sources. For example, in a cloud broadcasting station, where it is desirable to simultaneously edit and combine content from multiple different formats and sources, the executing entity can obtain "edited" and "combined" materials by simultaneously acquiring these contents from multiple external data sources, and then combine and interweave them to provide to other viewers through a broadcasting process. For example, a user of the cloud broadcasting station (e.g., the content's "director") can switch between content provided by one external data source and content provided by another external data source (i.e., switching from one stream to another) to achieve a combination of the two streams of content.

[0029] In some embodiments, in this step, the executing entity may select a target period of less than the minimum duration of the data frame as the "fixed time" and periodically "interrogate" as discussed above.

[0030] Therefore, by using a target period that is shorter than the minimum duration of a data frame, the executing entity can acquire and query data frames more "greedily" and "frequently" to avoid missing data frames due to an excessively long period.

[0031] S102, in response to being able to obtain data frames from an external data source, decode the data frames to obtain multimedia data; In the embodiments of this application, based on the above S101, if the execution subject can obtain a data frame from an external data source, the execution subject can respond to this and, after obtaining the data frame, perform a decoding operation locally to decode the data frame into multimedia data (i.e., individual text, graphics, images, sounds, videos, etc., or information forms composed of them).

[0032] For example, if the data frame is in binary data form, the executing entity can perform a decompression operation based on the compression algorithm used by the external data source to compress the original multimedia data into the data frame in binary data form, so as to decode the data frame and restore it to (the original) multimedia data.

[0033] For example, for multimedia data in video format, an external data source can obtain data frames in binary form by lossily compressing the video's luminance, color signals, or audio sampling signals using a certain algorithm. Correspondingly, in such cases, the executing entity can use a corresponding decompression algorithm to restore the data frames to the original luminance, color, or audio sampling signals.

[0034] It should be understood that if one or more external data sources return multiple data frames consecutively, the executing entity can choose to decode these data frames continuously in a serial or parallel manner, based on the return time, the processing priority of the external data source, etc., to obtain the multimedia data corresponding to each data frame.

[0035] S103, based on the first timestamp standard used by the external data source and the second timestamp standard used locally, converts the first timestamp of the data frame into the second timestamp; In the embodiments of this application, after obtaining multimedia data in step S102, the executing entity can convert the first timestamp of the data frame into a second timestamp based on the first timestamp standard used by the external data source and the second timestamp standard used locally. This allows the data frame, or the multimedia data corresponding to the data frame, to be marked using timestamps that all belong to the second timestamp standard. This avoids audio-visual desynchronization caused by differences in the (first) timestamp standards used by (different) external data sources.

[0036] For example, for multimedia data in the form of video, if its images and audio come from different external data sources, and these two external data sources use different timestamp standards (e.g., one generates the timestamp based on real-world time, and the other generates it based on its own system time, such as the distance from the current time to the system startup time), simply referring to the numerical values ​​corresponding to the timestamps may lead to problems such as incompatibility or desynchronization when combining the images and audio provided by these two external data sources. Similarly, even if the data frames come from the same external data source, a similar problem will occur if the external data sources use different timestamp standards to generate (first) timestamps for different types of data frames. Therefore, in this application, this problem can be solved by unifying them to a second timestamp under a second timestamp used by the executing entity.

[0037] In some embodiments, during this step, the executing entity may use the first timestamp, which is present and used in the first data frame of the data stream provided by the external data source, to determine the difference between the first timestamp standard (used by the external data source) and the second timestamp standard (local to the executing entity).

[0038] Specifically, if the executing entity is able to obtain a data frame from an external data source, it can first detect whether the data frame is the first data frame in the data stream provided by the external data source this time (e.g., a data stream used to provide a complete video or live content).

[0039] Accordingly, if the data frame is the first data frame in the data stream provided by the external data source this time, the executing entity can respond to this by determining the time distance between the first timestamp standard used by the external data source and the second timestamp standard used locally based on the first timestamp.

[0040] Then, using this time distance, the first timestamp of the data frame is converted into a second timestamp.

[0041] For example, the executing entity can use the distance between the first timestamp and the starting time point, such as "0", under the second timestamp standard as the time distance, or in other words, the time offset value. Then, when other data frames in the data stream are received subsequently, the corresponding first timestamp is subtracted from the "time distance" to obtain the corresponding second timestamp, thereby completing the conversion between the first timestamp standard and the second timestamp standard.

[0042] In some embodiments, if the second timestamp standard is not based on "0", for example, it is based on a non-"0" value "X", the executing entity can associate the time of the first timestamp with the "X", and then similarly determine the time distance between the two (for example, subtract "X" from the first timestamp), and subsequently convert the first timestamp into the second timestamp by subtracting the time distance.

[0043] Therefore, this method enables the executing entity to adjust timestamps even without pre-maintaining or obtaining the timestamp standard used by the external data source, or obtaining the corresponding conversion relationship between the two. This allows the executing entity to more easily access new and different external data sources.

[0044] Correspondingly, this approach also allows the executing entity to establish communication connections with external data sources earlier without worrying about timestamp errors caused by establishing connections too early, and without waiting for the external data source to actually generate and send data frames before establishing a connection (i.e., the executing entity can establish communication with external data sources before some live content begins, without worrying about timestamp confusion or errors caused by changing the timestamp marking logic due to establishing connections too early).

[0045] S104, Generate target multimedia data based on multimedia data and second timestamp, and present the target multimedia data.

[0046] In the embodiments of this application, after decoding the multimedia data based on S102 and converting the second timestamp in S103, the executing entity can combine the two, that is, add a second timestamp to the multimedia data. Then, the multimedia data corresponding to each of the multiple data frames is presented in the order of the second timestamp, or in other words, multiple data frames are presented consecutively to achieve the purpose of "playing the data stream".

[0047] When the executing entity is represented as a "cloud broadcasting station", in the process of presenting target multimedia data, the executing entity can actually use the presentation interface to present the multimedia data provided by various external data sources in a separate manner, so that users, such as broadcasting directors, can understand the current playback status and content of each multimedia data through the presentation results, and combine and switch them accordingly to achieve broadcasting.

[0048] Therefore, in some embodiments, in order to enable the data streams corresponding to each external data source to be presented independently, the executing entity may also choose to use different channels to transmit, provide and present these multimedia contents in this step.

[0049] Accordingly, in this step, the executing entity can first generate target multimedia data based on the multimedia data and the second timestamp. Then, the executing entity allocates target channels to the external data source from the (pre-configured) channel set.

[0050] Typically, each channel in a channel set can be used by an external data source or by an independent data stream to achieve isolation between data sources. Accordingly, the executing entity can use these independent channels to provide the multimedia content corresponding to each data stream to the presentation unit, and use the presentation unit to present them independently and separately.

[0051] In some embodiments, the presentation unit may be a unit for generating a presentation interface for users, such as directors, to select a "data stream". For example, the presentation unit may be a data stream presentation and selection unit deployed in a cloud broadcast control console.

[0052] Accordingly, the presentation unit can pre-allocate multiple presentation sub-interfaces within the presentation interface. Then, after receiving multimedia data corresponding to the data stream through the (target) channel, it selects a presentation sub-interface to present the multimedia data received from that channel. Thus, through the presentation interface composed of multiple presentation sub-interfaces, the executing entity can provide multimedia data from multiple channels in parallel to the user acting as the director, allowing the director to view the current content of this multimedia data in real time and switch and combine it as needed. This approach enhances the executing entity's parallel capabilities with different external data sources while enabling the director to more accurately and effectively switch content and provide operations.

[0053] Subsequently, the method for providing multimedia data provided in this application periodically requests data frames from an external data source, wherein the data frames include at least one of audio frames and image frames; in response to being able to obtain data frames from the external data source, the data frames are decoded to obtain multimedia data; based on a first timestamp standard used by the external data source and a second timestamp standard used locally, the first timestamp of the data frame is converted into a second timestamp, wherein the first timestamp belongs to the first timestamp standard and the second timestamp belongs to the second timestamp standard; target multimedia data is generated based on the multimedia data and the second timestamp, and the target multimedia data is presented. Therefore, not only can the buffering of data frames in the cloud broadcasting station be reduced by actively requesting from external data sources and immediately decoding the received data frames, and the multimedia data after transcoding and decoding of the data frames be presented to the user more quickly, but also the timestamp adjustment method enables the cloud broadcasting station to receive a large number of data frames provided by different external data sources in parallel, improving the efficiency of the cloud broadcasting station in acquiring and presenting multimedia data.

[0054] In some embodiments, if the executing entity fails to request a data frame within one or more cycles (e.g., after the request is made, the external data source does not respond to the request or return a data frame within a preset time period), this may cause the continuous playback of the data stream or multimedia data to be interrupted (e.g., due to the lack of new data frames, resulting in missing playback content). In such cases, the executing entity can use supplementary data frames as temporary fillers to display the content even if no new data frames are obtained, in order to avoid interruption.

[0055] In some embodiments, if the failure to acquire a data frame occurs during the transmission of a data stream, that is, before the "current" cycle, the executing entity has acquired one or more historical data frames through other prior historical cycles, the executing entity may choose to use these historical data frames (e.g., the historical data frame whose acquisition time is closest to the current time) as supplementary data frames to supplement and avoid interruptions such as black screens.

[0056] In such a case, if a data frame cannot be obtained from an external data source, but there are historical data frames that have been obtained in the past, the executing entity can respond by determining a supplementary data frame based on the historical data frames.

[0057] Accordingly, after the supplementary data frame is determined, the executing entity can perform decoding and other operations on the supplementary data frame in a similar manner as discussed above, so as to convert it into the corresponding target multimedia data, so as to "supplement" the continuously playing multimedia data and avoid interruption.

[0058] In some embodiments, depending on the length of the period, such an interruption may actually be caused by a request period that is too short (e.g., a shorter period length may be used to avoid losing data frames). In such cases, it is actually expected and acceptable that no data frames are acquired for one or more periods. In such cases, to avoid wasting computing resources by repeatedly performing such supplementation and filling operations and causing the playback content composed of multiple multimedia data (i.e., the content corresponding to the data stream) to erroneously stutter, the executing entity can further determine whether such temporary filling is actually necessary based on the number of consecutive periods in which data frames have not been acquired from the external data source.

[0059] Accordingly, for a better discussion of this process, we can also refer to... Figure 2 . Figure 2This application illustrates a process 200 for determining supplementary data frames according to an embodiment of the present application. For example, this process 200 can be executed after S101 in the above-described process 100 when a data frame cannot be obtained from an external data source.

[0060] Process 200 includes at least the following processing steps: S201, in response to the inability to obtain a data frame from an external data source, detect the number of consecutive cycles in which a data frame could not be obtained from an external data source; Specifically, if a data frame cannot be obtained from an external data source, the executing entity can first detect the number of consecutive cycles in which a data frame cannot be obtained from an external data source.

[0061] For example, the executing entity can determine which period is the period in which a data frame could not be obtained from an external data source by continuously searching upwards to the historical period in which a data frame was obtained, or to the first period, and thus determine the number of consecutive periods.

[0062] For example, if the execution entity finds the historical period in which the data frame was acquired after searching upwards for 3 periods, the execution entity can determine that the number of consecutive periods is 4 (i.e., the 4 periods consisting of the previous 3 periods and the current period).

[0063] S202, in response to the number of consecutive cycles being greater than or equal to the number of cycles threshold, detect whether there are historically acquired data frames; Specifically, after determining the number of consecutive cycles, the executing entity can compare it with a cycle count threshold. Typically, this cycle count threshold is determined based on the length of time during which a fluctuation or interruption in the data stream could be reliably considered to have resulted in the failure to acquire data frames. For example, if the length of time is X and the length of the cycle is Y, then the cycle count threshold can be the result of dividing Y by X.

[0064] In some embodiments, if the result is not an integer, the executing entity can round it by removing the remainder and keeping only the integer part, in order to avoid the situation where the filling is not timely due to the relaxation of the quantity limit.

[0065] Accordingly, if the number of consecutive cycles is greater than or equal to the cycle number threshold, the executing entity can detect whether there are any historically acquired data frames. For example, if during the upward search, instead of traversing all cycles and assuming that all cycles are such "consecutive cycles" without acquired data frames, the determination of the number of consecutive cycles is completed when a historical cycle in which a data frame has been acquired is reached, then the executing entity can determine the (historical) data frame corresponding to the historical cycle closest to the current time as a supplementary data frame.

[0066] In some optional implementations of this embodiment, if the number of consecutive cycles in S202 is less than the cycle number threshold, the execution subject may choose to jump back to S201 to wait for the data frame acquisition status in the next cycle, instead of temporarily filling the gap directly in the case of a cycle in which no data frame is acquired, as discussed above.

[0067] Accordingly, if the executing entity determines in this step that there are historically acquired data frames, the executing entity may respond by choosing to continue executing S203.

[0068] S203, determine supplementary data frames based on historical data frames.

[0069] Specifically, as discussed above, if the executing entity determines that there are historical data frames that have been acquired in the past, the executing entity can determine supplementary data frames based on the historical data frames, as discussed above, which will not be repeated here.

[0070] It should be understood that, in some embodiments, the executing entity may also choose to first determine whether a historical period in which historical data frames have been acquired exists, and only after such a historical period and historical data frames actually exist will it detect the number of consecutive periods in which data frames could not be acquired from the external data source, in order to adapt to different scenarios. For example, the executing entity may first detect whether at least one (historical) data frame has actually been acquired, and only after at least one data frame has actually been acquired will it perform such a detection of the number of consecutive periods, in order to avoid wasting computing resources by attempting to fill in the gaps when there is a lack of materials that can be used for temporary filling.

[0071] In some embodiments, if the multimedia data is actually composed of multiple types of data frames (e.g., a video composed of both image and audio frames), when determining whether to temporarily fill in the gaps using supplementary data frames for each type of data frame, the situation of other related types of data frames can also be used as a reference to determine whether temporary data frames are needed for filling. For example, if the multimedia data is composed of image and audio frames, the executing entity can choose to fill in both simultaneously only if neither image nor audio frame is acquired at the same time, so that they can be "synchronized" even if temporarily filled. Alternatively, the executing entity can also choose to fill in the data frame of the type that lags more if it determines that there is a significant difference in time or quantity between the two types of data frames (e.g., the difference reaches a predetermined filling quantity threshold).

[0072] In some embodiments, if a data frame cannot be obtained from an external data source and there is no historically obtained data frame, the executing entity may respond by presenting temporary multimedia data instead of attempting to fill the gap with supplementary data frames.

[0073] This temporary multimedia data can be content maintained in a pre-defined database, used for temporary playback, such as transition videos, promotional videos, etc. Therefore, by utilizing this temporary multimedia data, it is possible to temporarily replace the content provided by the external data source after communication has been established between the executing entity and the external data source, but the content provided by the external data source is missing. This avoids prolonged black screen situations due to factors such as fluctuations or malfunctions that temporarily lack external data source content, thus improving the viewing experience for viewers watching the content provided by the backend director.

[0074] To deepen understanding, this application also incorporates a specific application scenario, through... Figure 3 A flowchart illustrating the process of providing multimedia data in a specific application scenario is provided. For further details, please refer to... Figure 3 The process shown is 300.

[0075] In process 300, server 310 can be executored as an "executor" for implementing a cloud broadcasting station and providing multimedia data. Devices 311, 312, and 313 can all be executored as "external data sources" that can provide data frames to server 310.

[0076] Accordingly, in process 300, server 310 can first request data frames from devices 311, 312 and 313 by (periodically) executing S301.

[0077] It should be understood that during the execution of S301 on devices 311, 312, and 313, server 310 may make requests at different “periods” for different devices, and this application is not intended to limit this. For example, server 310 may execute S301 on device 311 once every Z time interval to request a data frame, while simultaneously executing S301 sequentially on device 312 every K time interval (a value different from Z) to request a data frame.

[0078] For example, in process 300, device 311 provides data frame 321 to server 310 by executing S302 because it is able to provide data frame 321.

[0079] Accordingly, upon receiving data frame 321, server 310 can respond accordingly and continue executing S303 to decode data frame 321 and obtain multimedia data 322.

[0080] Then, server 310 can continue to execute S304 to convert the timestamp 331 of data frame 321 under the timestamp standard of device 311 to the timestamp 332 under the timestamp standard used locally by server 310.

[0081] Next, server 310 can continue to execute S305, generate multimedia data 340 based on multimedia data 322 and timestamp 332, and present multimedia data 340.

[0082] For example, server 310 can use a presentation interface 350 that can be directly viewed by users, such as directors, to present the multimedia data 340.

[0083] For example, presentation interface 350 may include presentation sub-interfaces 351 and 352. Server 310 may choose to use presentation sub-interface 351 to present the multimedia data 340 based on the channel allocated to device 311.

[0084] This application also provides an apparatus for providing multimedia data, the structure of which is as follows: Figure 4The apparatus 400 shown includes: a data frame request module 410 configured to periodically request data frames from an external data source, wherein the data frames include at least one of audio frames and image frames; a data frame decoding module 420 configured to decode data frames to obtain multimedia data in response to the ability to obtain data frames from the external data source; a timestamp conversion module 430 configured to convert a first timestamp of the data frame to a second timestamp based on a first timestamp standard used by the external data source and a second timestamp standard used locally, wherein the first timestamp belongs to the first timestamp standard and the second timestamp belongs to the second timestamp standard; and a multimedia data presentation module 440 configured to generate target multimedia data based on the multimedia data and the second timestamp, and to present the target multimedia data.

[0085] In some embodiments, the apparatus 400 further includes: a supplementary frame determination module, configured to determine a supplementary data frame based on a historical data frame in response to the inability to obtain a data frame from an external data source and the existence of historically obtained data frames; and a supplementary frame decoding module, configured to decode the supplementary data frame to obtain multimedia data.

[0086] In some embodiments, the supplementary frame determination module includes: a cycle number detection submodule, configured to detect the number of consecutive cycles in which a data frame cannot be obtained from an external data source in response to the inability to obtain a data frame from an external data source; and a supplementary frame determination submodule, configured to determine a supplementary data frame based on a historical data frame in response to the number of consecutive cycles being greater than or equal to a cycle number threshold and the existence of historically obtained data frames.

[0087] In some embodiments, the apparatus 400 further includes a temporary multimedia data presentation module, configured to present temporary multimedia data in response to the inability to obtain data frames from an external data source and the absence of historically obtained data frames.

[0088] In some embodiments, the timestamp conversion module 430 includes: a standard difference determination submodule, configured to determine the time distance between a first timestamp standard used by the external data source and a second timestamp standard used locally, based on a first timestamp, in response to the data frame being the first data frame in the data stream provided by the external data source; and a timestamp conversion submodule, configured to convert the first timestamp of the data frame into a second timestamp using the time distance.

[0089] In some embodiments, the multimedia data presentation module 440 includes: a target multimedia data generation submodule configured to generate target multimedia data based on multimedia data and a second timestamp; a channel allocation submodule configured to allocate a target channel from a channel set to an external data source; and a target multimedia data presentation submodule configured to send the target multimedia data to the corresponding presentation unit using the target channel, and to present the target multimedia data using the presentation sub-interface corresponding to the target channel in the presentation interface presented by the presentation unit.

[0090] In some embodiments, the data frame request module 410 is further configured to periodically request data frames from an external data source based on a target period, wherein the target period is less than the minimum duration of the data frame.

[0091] This device embodiment corresponds to the method embodiment shown in the figures above. This device can not only reduce the buffer occupation of data frames in the cloud broadcasting station by actively requesting external data sources and decoding them immediately after receiving them, and enable the multimedia data after transcoding and decoding the data frames to be presented to the user faster, but also enable the cloud broadcasting station to receive a large number of data frames provided by different external data sources in parallel by adjusting the timestamps, thereby improving the efficiency of the cloud broadcasting station in acquiring and presenting multimedia data.

[0092] Based on the same inventive concept, this application also provides an electronic device, a readable storage medium, and a computer program product. The method corresponding to the electronic device can be the method for providing multimedia data in the foregoing embodiments, and its problem-solving principle is similar to that method. The electronic device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the foregoing embodiments of this application.

[0093] Electronic devices can be user devices, or devices composed of user devices and network devices integrated through a network, or applications running on the aforementioned devices. User devices include, but are not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and wristbands. Network devices include, but are not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, and can be used to implement some processing functions when setting an alarm clock. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0094] Figure 5 The diagram illustrates the structure of an electronic device suitable for implementing the methods and / or technical solutions in the embodiments of this application. The electronic device 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes based on a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage portion 508 into a Random Access Memory (RAM) 503. The RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An Input / Output (I / O) interface 505 is also connected to the bus 504.

[0095] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, touchscreen, microphone, infrared sensor, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), LED display, OLED display, etc., and speakers, etc.; a storage section 508 including one or more computer-readable media such as hard disk, optical disk, magnetic disk, semiconductor memory, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet.

[0096] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 501, it performs the functions defined in the methods of this application.

[0097] Another embodiment of this application provides a computer-readable storage medium and a computer program product having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0098] Specifically, this embodiment may employ any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, a system, apparatus, or device that is, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0099] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0100] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0101] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0102] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0103] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules and units is only a logical functional division, and in actual implementation, there may be other division methods. Taking units as examples, multiple units or page components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0105] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] Furthermore, the functional modules and units in the various embodiments of this application can be integrated into one processing module or unit, or each module or unit can exist physically separately, or two or more units can be integrated into one module or unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules and units.

[0107] The integrated modules and units implemented as software functional modules and units described above can be stored in a computer-readable storage medium. These software functional modules and units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0109] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A method for providing multimedia data, characterized in that, include: Periodically request data frames from an external data source, wherein the data frames include at least one of audio frames and image frames; In response to the ability to obtain the data frame from the external data source, the data frame is decoded to obtain multimedia data; Based on the first timestamp standard used by the external data source and the second timestamp standard used locally, the first timestamp of the data frame is converted into the second timestamp, wherein the first timestamp belongs to the first timestamp standard and the second timestamp belongs to the second timestamp standard; Target multimedia data is generated based on the multimedia data and the second timestamp, and then presented.

2. The method according to claim 1, characterized in that, The method further includes: In response to the inability to obtain the data frame from the external data source, and the existence of historically obtained data frames, a supplementary data frame is determined based on the historical data frames; The multimedia data is obtained by decoding the supplementary data frame.

3. The method according to claim 2, characterized in that, In response to the inability to obtain the data frame from the external data source, and the existence of historically obtained data frames, determining supplementary data frames based on the historical data frames includes: In response to the inability to obtain the data frame from the external data source, the number of consecutive periods in which the data frame could not be obtained from the external data source is detected; In response to the fact that the number of consecutive cycles is greater than or equal to the number of cycles threshold, and there are historical data frames that have been acquired in the past, a supplementary data frame is determined based on the historical data frame.

4. The method according to claim 1, characterized in that, The method further includes: In response to the inability to obtain the data frame from the external data source and the absence of historically obtained data frames, temporary multimedia data is presented.

5. The method according to claim 1, characterized in that, The process of converting the first timestamp of the data frame into a second timestamp based on a first timestamp standard used by the external data source and a second timestamp standard used locally includes: In response to the fact that the data frame is the first data frame in the data stream provided by the external data source, the time distance between the first timestamp standard used by the external data source and the second timestamp standard used locally is determined based on the first timestamp. Using the time distance, the first timestamp of the data frame is converted into a second timestamp.

6. The method according to claim 1, characterized in that, The step of generating target multimedia data based on the multimedia data and the second timestamp, and presenting the target multimedia data, includes: Target multimedia data is generated based on the multimedia data and the second timestamp; Assign a target channel to the external data source from the channel set; The target multimedia data is sent to the corresponding presentation unit using the target channel, and the target multimedia data is presented in the presentation sub-interface corresponding to the target channel in the presentation interface presented by the presentation unit.

7. The method according to any one of claims 1-6, characterized in that, The periodic request for data frames from an external data source includes: Data frames are periodically requested from an external data source based on a target period, wherein the target period is less than the minimum duration of the data frame.

8. An apparatus for providing multimedia data, characterized in that, include: A data frame request module is configured to periodically request data frames from an external data source, wherein the data frames include at least one of audio frames and image frames; A data frame decoding module is configured to decode the data frame to obtain multimedia data in response to the ability to obtain the data frame from the external data source; The timestamp conversion module is configured to convert the first timestamp of the data frame into the second timestamp based on the first timestamp standard used by the external data source and the second timestamp standard used locally, wherein the first timestamp belongs to the first timestamp standard and the second timestamp belongs to the second timestamp standard; The multimedia data presentation module is configured to generate target multimedia data based on the multimedia data and the second timestamp, and to present the target multimedia data.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

10. A computer-readable medium, characterized in that, It stores computer program instructions that can be executed by a processor to implement the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images