Display device and media asset playing method
By detecting and adjusting the order of media data transmission and playing preset media data, the problems of low playback efficiency and black screen silence in the existing technology are solved, and more efficient media data playback and user experience are achieved.
Patent Information
- Application Number
- PCT/CN2023/140315
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-07-05
- Filing Date
- 2023-12-20
- Publication Date
- 2024-11-28
AI Technical Summary
When existing display devices play media data, the playback efficiency may decrease due to the inconsistent order of media data transmission, resulting in inconsistent screens, and when the video data decoding time is too long, the device will keep the black screen silent for a long time. state.
By detecting the transmission order of MPU metadata, fragment metadata and MFU data in the media data transmission stream, and re-encapsulate and inject the player when the target sequence does not meet the target sequence, ensuring the correct order of the data transmission stream; at the same time, in the target When the video data decoding time exceeds the threshold, preset media data is played to avoid the black screen silent state.
It improves the playback efficiency and continuity of media data, reduces the waiting time for the device to decode videos, avoids long-term black screen silent state, and improves user experience.
Smart Images

Figure CN2023140315_28112024_PF_FP_ABST
Abstract
Description
Display device and media playback method
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on May 19, 2023, with application number 202310575869.8, and filed with the China Patent Office on July 5, 2023, with application number 202310817525.3, the entire contents of which are incorporated by reference into this application. Technical Field
[0003] The present application relates to the technical field of display devices, and in particular to a display device and a media playback method. Background Art
[0004] Display devices refer to terminal devices capable of outputting specific display images. These devices can include smart TVs, communication terminals, smart advertising screens, projectors, and other terminal devices. Taking smart TVs as an example, they are based on Internet application technologies, feature open operating systems and chips, and possess an open application platform. They enable two-way human-computer interaction and integrate multiple functions, including audio, video, entertainment, and data, to meet the diverse and personalized needs of users.
[0005] Display devices can play different types of media data based on the protocol stack. For example, the media data of TV programs can be transmitted through the MMTP (MPEG Media Transport Protocol) and ROUTE (Real-time Object Delivery Protocol for Unidirectional Transport) defined in the Advanced Television Systems Committee (ATSC) 3.0 protocol stack. When playing media data through MMTP, the media data needs to be encapsulated into MMT data packets for decoding and playback by the display device.
[0006] Summary of the Invention
[0007] A display device provided by an embodiment of the present application includes: a display configured to display an image and / or a user interface; a user interface configured to receive instructions from a user; a communication device configured to communicate with an external device according to a predetermined protocol; a memory configured to store computer instructions and data associated with the display device; at least one processor connected to the display, user interface, communication device and memory, and configured to execute computer instructions so that the display device performs: in response to a playback instruction of media data, obtaining a data transmission stream of the media data; the data transmission stream including MPU metadata, fragment metadata and MFU data; detecting the transmission order of the MPU metadata, fragment metadata and MFU data in the data transmission stream; if the transmission order is the target order, injecting the data transmission stream into a player so that the data transmission stream is decoded and played in real time by the player; if the transmission order is not the target order, encapsulating the data transmission stream into a media transmission packet according to the target order, and injecting the media transmission packet into the player.
[0008] Another display device provided by an embodiment of the present application includes: a display configured to display images and / or a user interface; a user interface configured to receive instructions from a user; a communication device configured to communicate with an external device according to a predetermined protocol; a memory configured to store computer instructions and data associated with the display device; at least one processor connected to the display, user interface, communication device and memory, and configured to execute computer instructions so that the display device performs: in response to a playback instruction of media data, receiving a data transmission stream of the media data, the data transmission stream including MPU metadata, fragment metadata and MFU data; caching the data transmission stream to a player cache area; when MPU metadata and fragment metadata are cached in the player cache area, decapsulating the data in the player cache area, and injecting the data in the player cache area into the player.
[0009] An embodiment of the present application provides a media playback method, including: in response to a playback instruction of media data, obtaining a data transmission stream of the media data; the data transmission stream includes MPU metadata, fragment metadata and MFU data; detecting the transmission order of the MPU metadata, fragment metadata and MFU data in the data transmission stream; if the transmission order is a target order, injecting the data transmission stream into a player to perform decoding and playback on the data transmission stream in real time through the player; if the transmission order is not a target order, encapsulating the data transmission stream into a media transmission packet according to the target order, and injecting the media transmission packet into the player. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] FIG1 is a schematic diagram of an operation scenario between a display device and a control apparatus according to an embodiment of the present application;
[0011] FIG2 is a schematic diagram of a hardware configuration of a display device according to an embodiment of the present application;
[0012] FIG3 is a schematic diagram of the hardware configuration of a control device according to an embodiment of the present application;
[0013] FIG4 is a schematic diagram of software configuration of a display device according to an embodiment of the present application;
[0014] FIG5 is a schematic diagram of a connection relationship between a display device and a server according to an embodiment of the present application;
[0015] FIG6 is a schematic diagram of the architecture of the ATSC 3.0 system protocol stack according to an embodiment of the present application;
[0016] FIG7 is a schematic diagram of an architecture for transmitting a media transport packet through an MMT protocol session according to an embodiment of the present application;
[0017] FIG8 is a diagram illustrating a data sequence arrangement of MMT packets in a normal order according to an embodiment of the present application;
[0018] FIG9 is a diagram illustrating a data sequence arrangement of MMT packets in an abnormal order according to an embodiment of the present application;
[0019] FIG10 is a diagram illustrating a conventional process interaction between a protocol stack middleware and a player for transmitting MPU data according to an embodiment of the present application;
[0020] FIG11 is a schematic diagram of a flow chart of a method for transmitting a media transmission stream according to an embodiment of the present application;
[0021] FIG12 is a flow chart of a method for playing media resources according to an embodiment of the present application;
[0022] FIG13 is a schematic diagram of a playback architecture of a display device according to an embodiment of the present application;
[0023] FIG14 is an interaction diagram of an optimized process for transmitting MPU data between a protocol stack middleware and a player according to an embodiment of the present application;
[0024] FIG15 is a schematic diagram of a process for performing data standard setting on a data transmission stream according to an embodiment of the present application;
[0025] FIG16 is a schematic diagram of a process for decapsulating player buffer data according to an embodiment of the present application;
[0026] FIG17 is a schematic diagram of a flow chart of a data transmission stream caching method according to an embodiment of the present application;
[0027] FIG18 is a schematic diagram of a determination process for decapsulating player buffer data according to an embodiment of the present application;
[0028] FIG19 is a diagram showing an audio and video playback principle according to an embodiment of the present application;
[0029] FIG20 is a signaling interaction diagram of the audio and video playback process executed by the underlying framework according to an embodiment of the present application;
[0030] FIG21 is a hardware connection block diagram of some functional modules of a display device 200 according to an embodiment of the present application;
[0031] FIG22 is a flow chart of a method for playing media resources executed by a display device 200 according to an embodiment of the present application;
[0032] FIG23 is a signaling interaction diagram of another underlying framework executing an audio and video playback process according to an embodiment of the present application;
[0033] FIG24 is a signaling interaction diagram of another underlying framework executing an audio and video playback process according to an embodiment of the present application;
[0034] FIG25 is a schematic diagram of a user interface of a display device 200 according to an embodiment of the present application;
[0035] FIG26 is a schematic diagram of a user interface of another display device 200 according to an embodiment of the present application;
[0036] FIG27 is a schematic diagram of a user interface of another display device 200 according to an embodiment of the present application;
[0037] FIG28 is a schematic diagram of a user interface of another display device 200 according to an embodiment of the present application;
[0038] FIG29 is a schematic diagram of a user interface of another display device 200 according to an embodiment of the present application;
[0039] FIG30 is a schematic diagram of a user interface of another display device 200 according to an embodiment of the present application;
[0040] FIG31 is a schematic diagram of a user interface of another display device 200 according to an embodiment of the present application;
[0041] FIG32 is a schematic diagram showing a principle of decoding multiple target media asset data according to an embodiment of the present application;
[0042] FIG33 is a schematic diagram of a specific application flow of the media resource playback method according to an embodiment of the present application. DETAILED DESCRIPTION
[0043] In order to make the purpose, content and advantages of the embodiments of the present application clearer, the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, not all of the embodiments.
[0044] Based on the exemplary embodiments shown in this application, all other embodiments obtained by persons of ordinary skill in the art without inventive effort are within the scope of protection of this application. In addition, although the disclosure in this application is described based on one or more exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete embodiment on its own.
[0045] It should be understood that the terms "first," "second," "third," etc., in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, for example, enabling implementation in an order other than that shown or described in the embodiments of this application.
[0046] The display device provided in the embodiments of the present application can have various implementation forms, for example, it can be a television, a smart TV, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figures 1 and 2 illustrate a specific embodiment of the display device of the present application.
[0047] FIG1 is a schematic diagram of an operation scenario between a display device and a control device according to an embodiment. As shown in FIG1 , a user can operate a display device 200 through a smart device 300 or a control device 100 .
[0048] In some embodiments, the control device 100 may be a remote controller. Communication between the remote controller and the display device may include infrared protocol communication, Bluetooth protocol communication, or other short-range communication methods, and the display device 200 may be controlled wirelessly or wired. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, and the like.
[0049] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) can also be used to control the display device 200. For example, the display device 200 can be controlled using an application running on the smart device.
[0050] In some embodiments, the display device may not use the aforementioned smart device or control device to receive instructions, but may receive user control through touch or gestures.
[0051] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the display device 200 can directly receive the user's voice command control through a module for obtaining voice commands configured inside the display device 200, or it can receive the user's voice command control through a voice control device set outside the display device 200.
[0052] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be connected to a local area network (LAN), a wireless local area network (WLAN), or other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be a single cluster or multiple clusters, and may include one or more types of servers.
[0053] As shown in Figure 2, the display device 200 includes: a display 260 configured to display an image and / or a user interface; a user interface 290 configured to receive instructions from a user; a communication device 220 configured to communicate with an external device according to a predetermined protocol; a memory 280 configured to store computer instructions and data associated with the display device; at least one processor 250 connected to the display 260, the user interface 290, the communication device 220 and the memory 280, and configured to execute computer instructions to cause the display device 200 to perform the following: in response to a playback instruction of media asset data, obtaining a data transmission stream of the media asset data; the data transmission stream including MPU metadata, segment metadata and MFU data; detecting a transmission order of the MPU metadata, segment metadata and MFU data in the data transmission stream; if the transmission order is a target order, injecting the data transmission stream into a player so that the data transmission stream is decoded and played in real time by the player; if the transmission order is not a target order, encapsulating the data transmission stream into a media transmission packet according to the target order, and injecting the media transmission packet into the player.
[0054] In some embodiments, the processor 250 includes a video processor, an audio processor, a graphics processor, etc. The display device may further include RAM, ROM, and first to nth interfaces for input / output.
[0055] The display 260 includes a display screen component for presenting images, a driving component for driving image display, a component for receiving image signals output from a processor, and a component for displaying video content, image content, and a menu control interface and a user control UI interface.
[0056] The display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.
[0057] Communication device 220 is a component used to communicate with external devices or servers using various communication protocols. For example, the communication device may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chip or a near-field communication protocol chip, as well as an infrared receiver. Display device 200 can use communication device 220 to send and receive control signals and data signals with control device 100 or server 400.
[0058] The user interface 290 may be configured to receive a control signal from the control device 100 (eg, an infrared remote controller, etc.).
[0059] In some embodiments, the display device 200 may further include a detector for collecting signals from the external environment or external interactions. For example, the detector may include a light receiver, such as a sensor for collecting ambient light intensity; or an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or a sound collector, such as a microphone, for receiving external sounds.
[0060] In some embodiments, the display device 200 may further include an external device interface. The external device interface may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. A composite input / output interface formed by multiple of the above interfaces may also be provided.
[0061] In some embodiments, the display device 200 may further include a tuner / demodulator that receives broadcast television signals via wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals.
[0062] In some embodiments, the processor 250 and the tuner-demodulator may be located in different separate devices, that is, the tuner-demodulator may also be located in an external device of the main device where the processor 250 is located, such as an external set-top box.
[0063] Processor 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. Processor 250 controls the overall operation of display device 200. For example, in response to receiving a user command to select a UI object to be displayed on display 260, processor 250 may perform operations related to the object selected by the user command.
[0064] The user may input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific voice or gesture, and the user input interface may recognize the voice or gesture through a sensor to receive the user input command.
[0065] Figure 3 is a block diagram of a control device 100 according to an embodiment of the present application. As shown in Figure 3, the control device 100 includes a processor 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 receives user input commands and converts them into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200.
[0066] Figure 4 is a software configuration diagram of the display device 200 according to an embodiment of the present application. In some embodiments, as shown in Figure 4, the system of the display device may include a kernel, a command parser (shell), a file system, and an application. The kernel, shell, and file system together constitute the basic operating system structure, which allows users to manage files, run programs, and use the system. After power-on, the kernel starts, activates the kernel space, abstracts the hardware, initializes the hardware parameters, etc., runs and maintains virtual memory, scheduler, signals, and inter-process communication (IPC). After the kernel starts, the shell and user application are loaded. After startup, the application is compiled into machine code to form a process.
[0067] As shown in Figure 4, the display device system is divided into three layers, namely, the application layer, the middleware layer, and the hardware layer from top to bottom. In some embodiments, the display device system also includes a UI layer (not shown in the figure), which is located above the application layer and receives data transmission from the application layer to realize the display of the display 260.
[0068] The application layer mainly includes commonly used applications on the TV and the application framework. Common applications are mainly applications developed based on the browser, such as HTML5 apps; and native apps.
[0069] The Application Framework is a complete programming model that provides all the basic functions required by standard application software, such as file access, data exchange, etc., as well as the user interfaces for these functions (toolbars, status bars, menus, and dialog boxes).
[0070] Native apps can support online or offline, message push or local resource access.
[0071] The middleware layer includes various TV protocols, multimedia protocols, and system components. Middleware uses the basic services (functions) provided by system software to connect various parts of the application system or different applications on the network, enabling resource and function sharing.
[0072] The hardware layer mainly includes the HAL interface, hardware, and drivers. The HAL interface is a unified interface for all TV chips, and the specific logic is implemented by each chip. Drivers mainly include: audio driver, display driver, Bluetooth driver, camera driver, WiFi driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver.
[0073] In some embodiments, the application layer of the display device 200 includes at least one application, such as a live TV application icon control, a video on demand application icon control, a media center application icon control, an application center icon control, a game application icon control, etc.
[0074] In some embodiments, a live TV application can provide live TV and broadcast TV from different signal sources. For example, the live TV application can use an input from a cable TV, over-the-air broadcast, satellite service, or other type of live TV service to provide a TV signal. Furthermore, the live TV application can display media asset data of the live TV signal on the display device 200.
[0075] In some embodiments, a video-on-demand application can provide videos from various storage sources. Unlike live TV applications, video-on-demand provides media data from a variety of storage sources. For example, video-on-demand can be served from a cloud storage server or from a local hard drive containing stored video programs.
[0076] In some embodiments, a media center application may be an application that provides various multimedia content playback applications. For example, a media center may be a service that is different from live TV or video on demand, and a user may access various images or audio services through the media center application.
[0077] In some embodiments, the application center can provide storage for various applications. These applications can be games, apps, or other applications related to computer systems or other devices that can be run on a smart TV. The application center can obtain these applications from various sources, store them in local storage, and then run them on the display device 200.
[0078] It should be noted that the media asset data described in the embodiments of the present application includes audio data and video data, and can be one or a combination of the above two types of data.
[0079] Based on the above application, in order to play the corresponding media asset data on the display device 200, as shown in Figure 5, in some embodiments, the display device 200 can communicate with the server 400 during use to achieve data exchange. For example, the user can trigger the display device 200 to display a program list through interactive instructions. The program list may include the title, start time, detailed description, program rating and media asset items of programs on multiple channels. Each media asset item corresponds to a network address, which is used to download the corresponding media asset data. The display device 200 can respond to the interactive instructions input by the user to obtain a request for media asset data from the server 400. The user can select a media asset item in the program list and request the server 400 to download the corresponding media asset data for playback on the display device 200.
[0080] After receiving the acquisition request, the server 400 can extract the media asset items included in the corresponding channel from the storage module according to the acquisition request and feed the extracted media asset item information back to the display device 200. The display device 200 then generates a program list based on the media asset item information fed back by the server 400 and displays the program list on the display 260, providing a good program navigation mechanism for the display device 200.
[0081] After the user selects the corresponding media item in the program list for playback, the display device 200 can obtain media data from the server 400 in real time during the playback of the media item to form a media data stream, and continuously obtain media images through decoding, rendering and other processing.
[0082] In order to implement data interaction between the display device 200 and the server 400, the display device 200 needs to establish a communication connection with the server 400. For example, the display device 200 and the server 400 can establish a communication connection via a transmission network, and the interactive data is transmitted between the display device 200 and the server 400 via the transmission network.
[0083] In some embodiments, the display device 200 and the server 400 each need to be provided with a component for establishing a communication connection. Specifically, as shown in FIG5 , the display device 200 may be provided with a communication device 220, while the server 400 may be provided with a communication module. The communication device 220 and the communication module may simultaneously support at least one of the same communication methods to establish a communication connection. For example, the communication device 220 on the display device 200 may include a fiber optic interface, allowing the display device 200 to connect to the network via the fiber optic interface. Simultaneously, the communication module on the server 400 may also include a fiber optic interface, allowing the display device 200 to connect to the network via the fiber optic interface, thereby achieving a communication connection between the display device 200 and the server 400.
[0084] It should be noted that the display device 200 and the server 400 may also establish a communication connection relationship using other connection methods, such as wired broadband, wireless local area network, cellular network, Bluetooth, infrared, radio frequency communication, etc.
[0085] The connection between the display device 200 and the server 400 can be a "many-to-one" relationship, that is, multiple display devices 200 can establish communication connections with the same server 400, allowing the server 400 to provide services for multiple display devices 200. The connection between the display device 200 and the server 400 can also be a "many-to-many" relationship, that is, multiple display devices 200 can establish communication connections with multiple servers 400, allowing the multiple servers 400 to provide different services for the display devices 200. Obviously, in some application scenarios, the connection between the display device 200 and the server 400 can also be a "one-to-one" relationship, that is, one server 400 specifically provides services for one display device 200.
[0086] In order to provide services for the display device 200, the server 400 may also include a storage module that can store various resource data, information files, and control programs. In response to user interaction, the display device 200 can obtain different data from the storage module of the server 400. For example, when the display device 200 requests a certain media item on demand, it can send a request to the server 400 to obtain playback data. After receiving the request, the server 400 can extract the media data to be played from the storage module and transmit the media data to the display device 200 so that the display device 200 can decode and display the media data. The control program stored in the storage module can be run by the control module of the server 400 so that the control module can perform corresponding functions according to the control program.
[0087] Based on the above-mentioned display device 200, in order to play the media data of the accessed channel, in some embodiments, the display device 200 can transmit the media data corresponding to the channel accessed by the display device 200 based on the Advanced Television Systems Committee (ATSC) 3.0 protocol stack to form a data transmission stream for the media data. As shown in Figure 6, the protocol layer of the ATSC 3.0 protocol stack uses the full IP protocol. Due to the support of bidirectional channels, the ATSC 3.0 protocol stack can not only provide broadcast services for the display device 200, but also provide interactive services for the display device 200. Among them, the broadcast service is based on the UDP / IP protocol, and the interactive service is based on the TCP / IP protocol.
[0088] Therefore, the display device 200 can transmit media data through the broadcast service provided by the ATSC3.0 protocol stack. In some embodiments, the broadcast service provided by the ATSC3.0 protocol stack includes the MPEG Media Transport Protocol (MMTP) and the Real-time Object delivery over Unidirectional Transport (ROUTE). Among them, MMTP is used to transmit the Media Processing Unit (MPU) and MMT specific signaling. The MPU is a basic encapsulation unit based on the ISO Base Media File Format (ISO BMFF) in MPEG media transmission; the MMT specific signaling may include two types of signaling for consumption and presentation. ROUTE is used to transmit DASH segments, ROUTE specific signaling and non-sequential content. The DASH segment is an ISO basic media encapsulation format based on HTTP-based dynamic adaptive streaming. Non-sequential content may include non-sequential media content, EPG data, etc.
[0089] It should be noted that the MMTP protocol package includes the ALP / IP / UDP / MMTP protocols, which can implement the data required for the playback of specified MMTP programs. Among them, the ALP protocol refers to the link layer protocol in the ATSC3.0 protocol stack; the IP protocol refers to the protocol for interconnection between networks; and the UDP protocol refers to the User Datagram Protocol.
[0090] Furthermore, in some embodiments, non-sequential content can also be transmitted directly via UDP. ATSC 3.0 protocol stack signaling can be distributed via MMTP and / or ROUTE. Bootstrap signaling information can be provided in the form of a Service List Table (SLT).
[0091] As shown in Figure 6, in some embodiments, to enable heterogeneous services, one or more program elements in display device 200 are transmitted via a broadband path. On the broadband side of display device 200, the ATSC 3.0 protocol stack uses MPEG DASH over HTTP / TCP / IP at the protocol level, while using ISO BMFF-based MPU and DASH files as the formats for broadcast and broadband transmission, encapsulation, and synchronization.
[0092] In some embodiments, the display device 200 transmits MMT signaling information using a signaling information mode during the process of transmitting MMT data packets through an MMT protocol session. Each MMT protocol session needs to carry specific MMT signaling information and each component data transmitted. MMT signaling may include media presentation information (MPI) signaling, hypothetical receiver buffer model signaling, receiver buffer model removal signaling, and clock-related information (CRI) signaling. The media presentation information signaling includes all or part of the presentation information file.
[0093] As shown in Figure 7, in some embodiments, an MMT Asset corresponds to a content component and has a corresponding component ID, namely, packet-id. For example, 0x0001, 0x0002, and 0x0003 are shown in Figure 7. Each MMT Asset is a collection of one or more media processing units with the same Asset ID, and the media processing units do not overlap in presentation time; and an MMT package is a collection of one or more MMT Assets, such as Asset A, Asset B, and Asset C in Figure 7. As shown in Figure 7, the data transmission stream of media asset data is composed of one or more MMT packages (media transmission packets), and the MMT packages do not overlap in presentation time.
[0094] It should be noted that the mapping information between the MMT package and the MMT protocol session is transmitted to the receiving end of the display device 200 by the MMT signaling information.
[0095] In some embodiments, a complete MMT package is an MPU. The data in the MPU includes MPU metadata, movie fragment metadata, and MFU (Media Fragmentation Unit) data. Among them, an MFU is an I frame, and the I frame contains all the picture information, which may affect the playback picture presentation quality of the media data. When transmitting media data, the above data is divided into UDP packets and sent. Each MPU contains an MPU metadata, a fragment metadata, and a number of MFUs. The display device 200 can only play and display the received MFU data after receiving the MPU metadata and fragment metadata.
[0096] Therefore, in some embodiments, when transmitting an MMT package, the display device 200 also needs to detect the order of the data in the MMT package. Only when the order of the MMT package is MPU metadata, segment metadata, and MFU data, can the display device 200 play the media asset data normally.
[0097] For example, as shown in Figure 8, the MPU structure shown in Figure 8 is an MMT package in normal order. As can be seen from Figure 8, the order of the MMT package is MPU metadata, fragment metadata, and MFU data. Therefore, when transmitting the MMT package via the MMT protocol, there is no need to re-pack the MMT package; the complete MMT package can be directly transmitted to the player of the display device 200.
[0098] It should be noted that the "mdat" header data containing MFU data in the MPU structure shown in Figure 8 in the embodiment of the present application is also processed as fragment metadata.
[0099] Obviously, when the MPU structure is in an abnormal order, in order for the display device 200 to properly play the MMT packages of the media asset data, the display device 200 needs to repackage the MMT packages in the abnormal order to form MMT packages in the normal order. That is, in some embodiments, when the MPU structure is in an abnormal order, the display device 200 repackages the MMT packages in the normal order.
[0100] For example, as shown in FIG9 , the MPU structure shown in FIG9 is an MMT package arranged in a disordered order. The display device 200 needs to reassemble the MMT package in the order of MPU metadata-fragment metadata-MFU data (mpu metadata to fragment metadata to mdat) to form the MMT package in the normal order shown in FIG8 .
[0101] Therefore, when playing media data, the display device 200 needs to obtain media data composed of multiple MPU data to continuously form the playback screen of the media data on the display 260. In other words, in some embodiments, the display device 200 interacts with the player through the protocol stack to achieve media data interaction. When playing media data, the display device 200 activates the player through the protocol stack middleware, and the player requests MPU data from the protocol stack middleware to obtain the media data.
[0102] As shown in FIG10 , a conventional process for transmitting MPU data between a protocol stack middleware and a player provided in an embodiment of the present application includes:
[0103] S1001: The protocol stack sends a start player message to the player.
[0104] S1002: The player sends a message requesting MPU data to the protocol stack;
[0105] S1003, the protocol stack receives the complete MPU;
[0106] S1004: The protocol stack returns an MPU to the player.
[0107] S1005: The player requests the next MPU from the protocol stack.
[0108] S1006, the protocol stack receives the complete MPU;
[0109] S1007: The protocol stack returns an MPU to the player.
[0110] That is, in this embodiment, after receiving the request from the player, the protocol stack middleware of the display device 200 receives the MPU data corresponding to the media asset data sent by the server 400 in response to the player's request. After receiving the complete MPU data, the protocol stack middleware returns a complete set of MPU data to the player. After receiving the complete MPU data, the player can decode and play the MPU data and simultaneously request the protocol stack middleware for the next set of MPU data.
[0111] However, it takes a certain amount of time for the protocol stack of the display device 200 to obtain complete MPU data, which will slow down the display speed of the display device 200. In this way, when the user watches channel programs through the display device 200, a long waiting time will occur. For example, when the user continuously switches channels in the display device 200, the display device 200 needs to continuously obtain new signaling data and MPU data. Since the protocol stack of the display device 200 needs to wait for complete MPU data, the problem of discontinuous pictures will occur, resulting in a decrease in the playback efficiency of the media data. If the protocol stack of the display device 200 does not wait for the complete MPU data, and directly injects the received MPU data into the player, the display device 200 may not be able to play the media data normally due to the disorderly transmission of the MPU data.
[0112] Based on the above application scenarios, in order to improve the problem of reduced media data playback efficiency in the display device 200, some embodiments of the present application provide a display device 200, as shown in Figure 11, comprising a display 260 and a processor 250. The display 260 is configured to display a playback screen of the media data; as shown in Figure 12, the processor 250 is configured to execute computer instructions to cause the display device to perform the following processing:
[0113] S100: In response to a playback instruction of media asset data, a data transmission stream of the media asset data is obtained.
[0114] After receiving the playback instruction of the media data, the display device 200 obtains the MMT package of the media data, namely, MPU data, from the server 400 based on the protocol stack to form a data transmission stream of the media data, which includes MPU metadata, segment metadata, and MFU data.
[0115] Playback instructions for media asset data can be generated based on control events from upper-layer applications. Therefore, in some embodiments, the display device 200 monitors control events from upper-layer applications. Control events can include volume increase, brightness adjustment, channel switching, channel switching, and channel access. After detecting a channel switching or channel access control event, the display device 200 detects the target channel to which the display device 200 is connected in response to the channel switching or channel access control event.
[0116] For example, if the display device 200 is connected to channel DC a when it is turned on and broadcasting, the target channel is DC a. If the user uses the remote control device provided with the display device 200 to change the channel to DC b, the target channel is now DC b. In other words, the target channel is the channel currently connected to the display device 200 and is not fixed.
[0117] After detecting the target channel, the display device 200 queries the media data corresponding to the target channel and generates a play instruction according to the media data. After generating the play instruction of the media data, the display device 200 can respond to the play instruction and obtain the data transmission stream of the media data.
[0118] When querying the media data corresponding to the target channel, in order to facilitate filtering of the media data, in some embodiments, the display device 200 establishes a media transmission channel according to the media transmission protocol of the media data. For example, an MMT protocol session is created according to the MMT protocol. After the media transmission channel is created, the media presentation information signaling is obtained through the media transmission channel. Among them, the media presentation information signaling includes a presentation information table and a component description table. The above two tables are used for filtering media data, and the component description table (User Service Dsecription, USD) includes the component ID, namely packet_id. Therefore, querying the media data according to the presentation information table and the component description table is to query the MPU data corresponding to the media data.
[0119] In some embodiments, the display device 200 can obtain component data corresponding to the media asset data by matching the packet_id value of the MMT package. For example, the corresponding media asset data can be queried according to the packet_id recorded in the USD to form the MPU data of the media asset data.
[0120] Therefore, in some embodiments, when the display device 200 obtains the data transmission stream of the media asset data, it calls the channel interface of the protocol stack middleware and controls the target channel accessed by the display device 200 through the channel interface. Then, the media asset presentation information signaling and media asset data of the target channel are received.
[0121] For example, as shown in Figure 13, display device 200 transmits media data based on the ATSC 3.0 protocol stack, which includes two data transmission protocols: ROUTE and MMTP. When a user inputs a channel switching control event in the upper-layer application of display device 200, display device 200 calls the channel switching interface of the protocol stack middleware, initiates the channel switching process, and receives media presentation information signaling and media data from the target channel.
[0122] It can be understood that when switching the target channel, the display device 200 can not only receive media presentation signaling data, but also receive other signaling of more media data, such as CRI signaling, receiver buffer model removal signaling and assumed receiver buffer model signaling, so that the display device 200 can play the media data corresponding to the target channel.
[0123] S200: Detect the transmission order of MPU metadata, fragment metadata and MFU data in the data transmission stream.
[0124] After receiving the data transmission stream of the media asset data, the display device 200 also detects the transmission order of the data in the data transmission building. Since the display device 200 cannot play the media asset data normally if the MPU data is in a disordered state, the display device 200 needs to detect whether the transmission order of the MPU data is normal when receiving the data transmission stream. In other words, when the protocol stack middle layer of the display device 200 receives the data transmission stream, it detects whether the order of the data in the data transmission stream is MPU metadata, fragment metadata, and MFU data, that is, the order shown in Figure 8.
[0125] Therefore, the display device 200 can set the order shown in Figure 8 as the target order to measure whether the data transmission stream of the media asset data is out of order by the target order. That is, in some embodiments, the target order is MPU metadata, fragment metadata, and MFU data.
[0126] S300: If the transmission order is the target order, inject the data transmission stream into the player so that the player can decode and play the data transmission stream in real time.
[0127] Because the data transmission stream of media asset data consists of multiple MMT packages, that is, multiple MPU data, to ensure that the MPU data corresponding to the media asset data can be played normally, the display device 200 can detect the order of the data transmission stream and compare the detected transmission order with the target order. In other words, the display device 200 needs to detect the transmission order of each MPU data in the data transmission stream to determine whether the currently transmitted MPU data is in the target order.
[0128] Since the display device 200 can only display and play the received MFU data after receiving the MPU metadata and fragment metadata. Therefore, when the transmission order of the MPU data is the target order of MPU metadata, fragment metadata and MPU data, the MPU data can be played normally. At this time, in order to shorten the waiting time of the player, when the data transmission stream of the media data is in the target order, the display device 200 injects the current MPU data into the player, so that the player can perform decoding and playback on the data transmission stream in real time, thereby improving the player's picture output speed.
[0129] For example, the display device 200 detects that the transmission order of the current data transmission stream is MPU metadata, fragment metadata, and MPU data, indicating that the transmission order of the current data transmission stream is the target order.
[0130] As shown in FIG14 , an embodiment of the present application provides an optimized process for transmitting MPU data between a protocol stack middleware and a player, including:
[0131] S1401: The protocol stack sends a start player message to the player.
[0132] S1402: The player sends a message requesting MPU data to the protocol stack;
[0133] S1403: The protocol stack returns MPU metadata / fragment metadata / MFU data to the player.
[0134] S1404: The protocol stack returns MPU metadata / fragment metadata / MFU data to the player.
[0135] And so on;
[0136] S1405: The protocol stack sends a message to the player that the current MPU has completed sending.
[0137] S1406: The player requests the next MPU from the protocol stack.
[0138] That is to say, in this embodiment, the display device 200 sends the received data transmission stream to the player in real time, so that the player can decode and play the data transmission stream in real time. After the current MPU data is received, the player requests the next MPU data from the protocol stack middleware, and the display device 200 continues to detect the transmission order of the MPU to switch the transmission method of the MPU data according to the transmission order. Obviously, the interaction method shown in Figure 14 can shorten the waiting time of the player in the display device 200 and improve the picture output speed of the display device 200 compared to the interaction method shown in Figure 10.
[0139] S400: If the transmission order is not the target order, encapsulate the data transmission stream into a media transmission packet according to the target order, and inject the media transmission packet into the player.
[0140] Similarly, in order for the data transmission stream of the media data to be played normally, when the MPU data currently transmitted by the data transmission stream is not in the target order, it means that the MPU data currently transmitted is arranged in a disordered order. Since the player cannot play the MPU data arranged in a disordered order normally, the MPU data needs to be re-packaged. Therefore, when the data transmission stream of the media data is not in the target order, the display device 200 re-packages the data transmission stream into media transmission packets in the order shown in Figure 8 according to the target order. In this way, the order of the media transmission packets is MPU metadata, fragment metadata and MPU data, and the player of the display device can normally display the playback screen of the media transmission packet. After the packaging is completed, the display device 200 injects the packaged media transmission packet into the player of the display device 200, so that the display device 200 can decode and play the media transmission packet.
[0141] The process of display device 200 receiving and detecting the transmission sequence is implemented based on protocol stack middleware. In order to enable the protocol stack middleware of display device 200 to inject data into the player, the protocol stack middleware needs to establish a connection between the protocol stack middleware and the player before injecting data into the player.
[0142] Therefore, in some embodiments, the display device 200 starts the media server of the protocol stack middleware and establishes a connection between the protocol stack middleware and the player through the media server. After the connection between the protocol stack middleware and the player is established, the display device 200 can call the player based on the connection relationship and inject the data transmission stream or media transmission packet into the player, so that the player can decode and play the data transmission stream or media transmission packet.
[0143] For example, as shown in Figure 13, the protocol stack middleware of display device 200 includes an AV server module. Before transmitting the data transmission stream of media asset data, display device 200 activates the AV server module to establish a connection with the player. Once the connection is established, the protocol stack middleware can inject the media asset data transmission stream or media transmission packet into the player.
[0144] To facilitate detection of the transmission order of the transport data stream, in some embodiments, the FT field corresponding to the packet header of the fragment metadata is set to 0 or 1; the FT field corresponding to the packet header of the MFU data is set to 2. In this way, the FT field can clearly represent the boundary between the metadata or MFU data, and at the same time carry the minimum information used to restore the association between the MFU data and the metadata, such as the movie fragment sequence number and the sample sequence number.
[0145] Therefore, as shown in FIG15 , in some embodiments, the process of performing data standard setting on a data transmission stream includes:
[0146] S1501 : After receiving the data transmission stream of media asset data, the protocol stack middleware of the display device 200 further detects the data type of the data in the data transmission stream.
[0147] S1502. Determine whether the data type is MPU metadata;
[0148] S1503: If the data type is MPU metadata, the display device 200 sets the bit according to the MPU metadata standard;
[0149] S1504: If the data type is not MPU metadata, determine whether the data type is fragment metadata;
[0150] S1505: If the data type is segment metadata, the display device 200 sets the bit according to the segment metadata standard;
[0151] S1506: If the data type is not fragment metadata, determine whether the data type is MFU data;
[0152] S1507: If the data type is MFU data, the display device 200 sets the bit according to the MFU standard.
[0153] That is, the protocol stack middleware of the display device 200 sets the data transmission stream according to the corresponding data standard, so that the display device 200 can perform subsequent processing on the data transmission stream.
[0154] After the protocol stack middleware of display device 200 injects data into the player, the player can then decode and play the injected data. Therefore, in some embodiments, display device 200 initializes the player's decoder and uses it to decode the data transmission stream. After decoding the data transmission stream, display device 200 calls underlying resources to render the playback screen of the media asset data on display 260, forming a continuous playback screen.
[0155] Based on the above embodiments, some embodiments of the present application further provide a display device 200, comprising a display 260 and a processor 250. The display 260 is configured to display a playback screen of media asset data; as shown in FIG16 , the processor 250 is configured to execute computer instructions to cause the display device to perform the following processing:
[0156] S1: In response to a playback instruction of media asset data, a data transmission stream of the media asset data is received.
[0157] After receiving the playback instruction of the media data, the display device 200 obtains the MMT package of the media data, namely, MPU data, from the server 400 based on the protocol stack to form a data transmission stream of the media data, which includes MPU metadata, segment metadata, and MFU data.
[0158] S2: Cache the data transmission stream to the player buffer area.
[0159] To speed up player decoding, the display device 200 can create a player buffer on the player side to cache the data transmission stream of the media asset data in the player buffer. Therefore, by caching the data transmission stream on the player side, the transmission efficiency of the data transmission stream can be improved, and the response speed of the display device 200 can be accelerated.
[0160] In some embodiments, the player buffer includes a metadata buffer and an MFU data buffer, wherein the metadata buffer is used to cache MPU metadata and fragment metadata, and the MFU data buffer is used to cache MFU data.
[0161] Therefore, as shown in FIG17 , the process of the data transmission stream caching method provided in the embodiment of the present application includes:
[0162] S1701. When the display device 200 caches the data transmission stream into the player buffer area, it also detects the data type of the data in the data transmission stream.
[0163] S1702: Determine whether the data type is MPU metadata;
[0164] S1703. If the data type is MPU metadata, cache the metadata to the metadata cache area.
[0165] Among them, metadata includes MPU metadata and fragment metadata.
[0166] S1704: If the data type is not MPU metadata, determine whether the data type is fragment metadata.
[0167] S1705: If the data type is segment metadata, cache the metadata in a metadata cache area.
[0168] That is to say, when the data of the data transmission stream is MPU metadata or fragment metadata, the MPU metadata or fragment metadata is cached in the metadata cache area on the player side.
[0169] S1706: If the data type is not fragment metadata, determine whether the data type is MFU data;
[0170] S1707: Similarly, if the data type is MFU data, the MFU data is cached in the MFU data cache area. That is, when the data of the data transmission stream is MFU data, the MFU data is cached in the MFU cache area of the player end.
[0171] S1708. In addition, in some embodiments, when the display device 200 detects the data type of the data transmission stream, when the data type of the data transmission stream is neither metadata nor MFU data, the data is discarded to reduce the system resource consumption occupied by the player buffer area in the display device 200.
[0172] S3: When the MPU metadata and fragment metadata are cached in the player buffer, the data in the player buffer is decapsulated and the data in the player buffer is injected into the player.
[0173] After the display device 200 caches the data transmission stream of the media asset data in the buffer area of the player end, it can save the data in two types: metadata and MFU data. Since the display device 200 can only play and display the received MFU after receiving the MPU metadata and segment metadata, regardless of the order in which the data transmission stream is sent, the display device 200 only decapsulates the data in the player buffer area after receiving the MPU metadata and segment metadata in the player buffer area, and then injects the decapsulated data into the player, so that the player can decode and play the cached data in the player buffer area.
[0174] For example, as shown in FIG18 , a judgment process for decapsulating player buffer data provided in an embodiment of the present application includes:
[0175] S1801: The protocol stack of the display device 200 sends a data transmission stream of media asset data to the player, and the display device 200 detects the data type of the data transmission stream.
[0176] S1802: Determine whether the data type is MPU metadata. If yes, execute step S1803; otherwise, execute step S1804.
[0177] S1803: Cache the metadata into the metadata cache area;
[0178] S1804: Determine whether the data type is segment metadata; if so, execute step S1805; otherwise, execute step S1806;
[0179] S1805: Cache the metadata into the metadata cache area;
[0180] S1806. Determine whether the data type is MFU data; if so, execute step S1807; otherwise, execute step S1810;
[0181] S1807. Cache the MFU data into the MFU data cache area;
[0182] That is, in this embodiment, metadata and MFU data are cached in the corresponding player cache area;
[0183] S1808: Detect whether the player buffer area has received complete metadata. If so, execute step S1809; otherwise, execute step S1810;
[0184] S1809. When complete metadata is received in the player buffer, decapsulate the data in the player buffer so that the player can decode and play the data in the player buffer.
[0185] S1810. When the data type of the data transmission stream is neither metadata nor MFU data, and the metadata received in the player buffer is not complete, the data is discarded to reduce the system resource consumption occupied by the player buffer in the display device 200.
[0186] Based on the above-mentioned display device 200, some embodiments of the present application further provide a method for playing media resources, as shown in FIG12 . The method includes the following steps:
[0187] S100: In response to a playback instruction of media asset data, obtaining a data transmission stream of the media asset data; the data transmission stream includes MPU metadata, segment metadata, and MFU data;
[0188] S200: Detecting the transmission order of MPU metadata, fragment metadata, and MFU data in the data transmission stream;
[0189] S300: If the transmission order is the target order, inject the data transmission stream into the player so that the player can decode and play the data transmission stream in real time;
[0190] S400: If the transmission order is not the target order, encapsulate the data transmission stream into a media transmission packet according to the target order, and inject the media transmission packet into the player.
[0191] It can be seen from the above embodiments that the display device and media playback method provided in some embodiments of the present application can obtain the data transmission stream of the media data in response to the playback instruction of the media data. The data transmission stream includes MPU metadata, fragment metadata and MFU data. Then detect the transmission order of the MPU metadata, fragment metadata and MFU data in the data transmission stream. If the transmission order is the target order, the data transmission stream is injected into the player so that the data transmission stream is decoded and played in real time through the player. If the transmission order is not the target order, the data transmission stream is encapsulated into a media transmission packet according to the target order, and the media transmission packet is injected into the player. The method can directly inject the data transmission stream into the player when the transmission order of the data transmission stream is the target order, which can speed up the picture output speed of the display device and improve the playback efficiency of the media data.
[0192] In addition to the above embodiments, the present application also provides other embodiments of display devices and media resource playback methods, which prevent the display device from remaining in a black screen and silent state for a long time when playing a type of broadcast or television program whose video data takes too long to decode, as described below:
[0193] In some embodiments, the audio and video playback principle is shown in Figure 19.
[0194] Referring to FIG. 20 , the signaling interaction process of the underlying framework provided in the embodiment of the present application for executing the audio and video playback process includes:
[0195] S2001, the audio decoder completes decoding of the first frame of audio data;
[0196] S2002, the audio decoder sends an audio data presence message to the playback controller;
[0197] S2003, the video decoder completes decoding of the first frame of video data;
[0198] S2004, the video decoder sends a video data presence message to the playback controller;
[0199] S2005: The playback controller generates an audio and video synchronization message;
[0200] S2006: The playback controller injects video data and audio data into the peripheral device.
[0201] That is to say, in this embodiment, when the video decoder completes decoding of the video data (here, it refers to the completion of decoding of the first frame of video data), it also reports the decoding status information to the monitoring interface and throws a video data appearance message to the playback processor of the middleware layer. When the audio decoder completes decoding of the audio data (here, it refers to the completion of decoding of the first frame of audio data), it also reports the decoding status information to the monitoring interface and throws an audio data appearance message to the playback processor of the middleware layer. The playback processor determines whether to control the display and speaker of the peripheral device to turn on by monitoring the status information of the audio decoder and the video decoder. After receiving the audio and video synchronization message, the playback processor controls the display and speaker of the peripheral device to turn on (calling the audio and video data interface, injecting video data into the display, and injecting audio data into the power amplifier for playback), so that the display plays the picture and the speaker plays the sound, and then the display device can display the device picture and output sound, thereby realizing the playback of radio and television programs.
[0202] Although audio data can be decoded about 1 second after starting playback, the decoding time of video data is often longer than that of audio data. If the decoding time of video data of some radio and television programs is too long, it will often cause the playback software to wait for a long time for the audio and video synchronization status message, and the corresponding display device will remain in a black screen and silent state for a long time.
[0203] For example, in broadcast and television programs featuring still pictures, the decoding time for the first frame of video data is approximately 5 seconds. Therefore, the display device needs to remain black and silent for approximately 5 seconds while the audio is synchronized. The user then waits approximately 5 seconds after entering the play command before the display device can see the picture and hear the audio.
[0204] In view of the above issues, some embodiments of the present application provide a display device 200. To facilitate understanding of some embodiments of the present application, the following detailed description of each step is provided in conjunction with some specific embodiments and accompanying drawings. Figure 21 is a hardware connection block diagram of some functional modules of the display device 200 according to an embodiment of the present application. Figure 22 is a flow chart of a method for media playback executed by the display device 200 according to an embodiment of the present application.
[0205] As shown in Figure 21, the functional modules of the display device 200 involved in the embodiment of the present application mainly include: a processor 250, a power supply, a display 260, an audio system, a video decoder, an audio decoder, and a memory. The functional modules mentioned above are only for the purpose of illustrating the modules described in the solution, and are not intended to implement all the functional modules of the present application.
[0206] The processor 250 is the control and signal processing core of the entire display device 200. It is responsible for controlling the operation of the entire display device 200 system, including receiving external image signals, image signal decoding, image quality processing, and image signal output; audio signal input, audio signal processing, and outputting audio signals to the power amplifier device 500, controlling the operation of the backlight component, and ensuring the normal operation of peripheral devices or components such as Wi-Fi and Bluetooth.
[0207] The power supply is the power output module of the entire display device 200, providing power guarantee for all modules of the display device 200. The display 260 is used to display video images; the audio system is used to play audio data. The video decoder is used to decode the received video data and then transmit the decoded video data to the display 260 to display the video image. The audio decoder is used to decode the received audio data and then transmit the decoded audio data to the audio system, which plays the sound according to the decoded audio data. The memory can store video data and audio data that do not need to be decoded, or video data and audio data that have a short decoding time.
[0208] Based on the functional modules shown in FIG21 , as shown in FIG22 , the media resource playback method performed by the display device 200 provided in the embodiment of the present application includes the following steps:
[0209] Step S10: In response to a play instruction input by the user, target media data is loaded, wherein the target media data includes target audio data and target video data, and the decoding time required for the target audio data is shorter than the decoding time required for the target video data.
[0210] The present application can be applied to scenarios where the display device 200 plays broadcast television programs, scenarios where the display device 200 plays online television programs, and scenarios where the display device 200 uses an HDMI (High Definition Multimedia) interface to connect to an external media resource device to obtain media resources. For example, the HDMI interface of the display device 200 can be used to connect to an external set-top box, DVD player, computer, etc., and then media resources can be obtained from the external set-top box, DVD player, computer, etc., and then played on the display device 200.
[0211] In the scenario of playing a radio or television program, a play command may be input by selecting a radio or television program channel. After receiving the play command, the server corresponding to the radio or television program channel sends the corresponding media data to the display device 200. In the scenario of playing an online television program, a play command may be input by selecting a certain online media resource on the media resource platform. After receiving the play command, the server corresponding to the media platform sends the corresponding media data to the display device 200. In the scenario of obtaining media resources using an HDMI external media resource device, a play command may be input by selecting a certain media resource icon in an application supported by the external device. After receiving the play command, the external media resource device transmits the corresponding media data to the display device 200.
[0212] In the various scenarios described above, the media data acquired by the display device 200 is the target media data in the embodiments of the present application. In each of these scenarios, after the display device 200 acquires the target media data, it is necessary to decode the target media data. The target media data includes target audio data and target video data. Therefore, if the target media data needs to be decoded, both the target video data and the target audio data need to be decoded simultaneously.
[0213] Before decoding the target audio data and the target video data separately, the target audio data and the target video data in the target media asset data need to be disassembled, or the target video data needs to be processed to extract the target audio data from the target video data. For example, the target video data can be processed using FFMpeg (Fast Forward Mpeg, an open source computer program that records, converts digital audio and video, and can convert them into streams) to extract data in a 1600-sample, signed 16-bit little-endian pulse code modulation (PCM) format. In other words, the target video data and the target audio data in the target media asset data can be loaded separately, or the target audio data and the target video data can be loaded together.
[0214] The target audio data acquired by the display device 200 is composed of audio frames. An audio frame is a data block with a fixed or non-fixed length at one end generated after audio data of a certain length is compressed by a specific compression algorithm. Such a data block is called a frame. The audio data before compression processing can be PCM (Pulse Code Modulation) data. For an audio file, after being encoded and compressed, the audio file is composed of a large number of audio frames and frame headers. Among them, the frame header includes some descriptive information of the audio frame, such as the encoding type, the number of channels, etc.
[0215] Video is a continuous sequence of images, consisting of consecutive frames, with each frame representing an image. Due to the persistence of vision, when a frame sequence is played at a certain rate, the human eye perceives a video of continuous motion. Because consecutive frames are highly similar, encoding and compression are typically performed to remove spatial and temporal redundancy for easier storage and transmission. Video decoding essentially reverses the encoding process.
[0216] Based on the above description, it can be known that audio frame and video frame are the basic units when encoding and decoding in audio and video technology. Both audio decoding and video decoding consume time. When decoding, target audio data is decoded with audio frame as the basic unit, and frame by frame decoding is performed. When decoding, the same target video data is decoded with video frame as the basic unit, and frame by frame decoding is performed. As shown in the audio and video playback principle diagram in Figure 19, the playback controller receives the audio and video synchronization message (audio data and video data have both appeared) sent by the audio decoder and the video decoder, and then controls the display and the speaker to turn on. Therefore, the decoding duration required for the target audio data involved in the embodiment of the present application refers to the time consumed by the decoding of the first frame audio frame in the target audio data, and the decoding duration required for the target video data involved in the embodiment of the present application refers to the time consumed by the decoding of the first frame video frame in the target video data.
[0217] That is, in the embodiment of the present application, the audio decoder generates an audio data present message when it completes decoding the first audio frame, and the video decoder generates a video data present message when it completes decoding the first video frame. When both the first video frame and the first audio frame are decoded, an audio and video synchronized message is generated.
[0218] An audio frame typically contains only the data content of the current frame, so decoding of audio data is relatively quick. However, a video frame can be an I-frame (intra-coded frame), a P-frame (predictive coded frame), or a B-frame (bidirectionally predictive coded frame). An I-frame is an independent frame that carries all its own information and can be decoded independently without reference to other images. It can be simply thought of as a static image. A P-frame requires reference to the previous I-frame for encoding. It represents the difference between the current frame and the previous frame (which can be either an I-frame or a P-frame). During decoding, the difference defined by the current frame is superimposed with the previously cached image to generate the final image. Compared to I-frames, P-frames typically occupy fewer data bits, but P-frames have complex dependencies on the previous P and I reference frames. B-frames, also known as bidirectionally predictive coded frames, record the difference between the current frame and the frames before and after it. This means that decoding a B-frame requires not only the previously cached image but also the decoded image afterward. The final image is obtained by superimposing the previous and subsequent images with the current frame's data. B-frames offer high compression rates but also place high demands on decoding performance.
[0219] The first frame in a video sequence is always an I-frame. However, since it is a key frame and an independent frame with all its own information, the decoding time of the first video frame of the target video data is usually longer than the decoding time of the first audio frame of the target audio data. Some embodiments of this application do not consider the case where the decoding time required for the target audio data is longer than the decoding time required for the target video data.
[0220] Step S20: If the decoding time required for the target video data exceeds the decoding time threshold, at least the preset media data is played, and when the decoding of the target video data is completed, the playback of the preset media data is canceled, and the target video data and the target audio data are played, wherein the preset media data is media data that is pre-stored in the display device and does not need to be decoded.
[0221] Step S30: If the decoding time required for the target video data does not exceed the preset decoding threshold, the preset media asset data is not played, and when the decoding of the target video data is completed, the target video data and the target audio data are played.
[0222] After acquiring the target media data, the display device 200 can determine the required decoding duration for the target audio data and target video data included in the target media data based on the type of the target media data. For example, the target media data itself carries a media identifier, and a configuration file is stored in the memory of the display device 200. This configuration file includes the media identifier and the required decoding duration for the target audio data and target video data. Therefore, upon acquiring the target media data, the display device 200 can retrieve the required decoding duration for the target audio data and target video data from the configuration file based on the media identifier of the target media data.
[0223] For example, a configuration file includes three media asset identifiers, A, B, and C, and records the decoding durations required for the target audio data and target video data corresponding to media asset identifier A, the decoding durations required for the target audio data and target video data corresponding to media asset identifier B, and the decoding durations required for the target audio data and target video data corresponding to media asset identifier C. Media asset identifier A corresponds to one type of media asset data, media identifier B corresponds to one type of media asset data, and media identifier C corresponds to one type of media asset data.
[0224] If the target media asset data M1 loaded into the display device 200 carries the media asset identifier A, this indicates that the target media asset data M1 belongs to the category of media asset data with media asset identifier A. The corresponding decoding durations X1 and Y1 of the target video data and target audio data are then retrieved from the configuration file. If the target media asset data M2 loaded into the display device 200 carries the media asset identifier B, this indicates that the target media asset data M1 belongs to the category of media asset data with media asset identifier B. The corresponding decoding durations X2 and Y2 of the target video data and target audio data are then retrieved from the configuration file.
[0225] Since the decoding time required to locate the target video data is greater than the decoding time required for the target audio data in the embodiment of the present application, it is only necessary to compare the decoding time required for the target video data with the decoding time threshold. For example, for the target media data M1 carrying the media identifier A, the decoding time X1 required for the target video data is compared with the decoding time threshold. If the decoding time X1 required for the target video data exceeds the decoding time threshold, then if it is necessary to wait for the decoding of the target video data and the target audio data to be completed at the same time, the user will need to wait for a long time when the display device is in a black screen and silent state.
[0226] The specific implementation of the above process at the bottom layer is: the decoding time required by the audio decoder to decode the target audio data is Y1, for example, the decoding time required for the target audio data is 1 second, that is, the audio decoder takes 1 second to decode the first frame of audio of the target audio data, and then it can throw an audio data appearance message to the playback controller. The decoding time required by the video decoder to decode the target video data is X1, for example, the decoding time required for the target video data is 5 seconds, that is, the video decoder takes 5 seconds to decode the first frame of video of the target video data, and then it can throw a video data appearance message to the playback controller. In other words, the playback controller needs to wait for at least 5 seconds before it receives the audio and video synchronization message. If the playback controller controls the display and speakers to turn on after receiving the audio and video synchronization message, the user experience is that after entering the playback command, the display device will be in a black screen and silent state for 5 seconds before the picture and sound can be played.
[0227] Referring to FIG. 23 , an embodiment of the present application provides a signaling interaction process for executing an audio and video playback process in a bottom-layer framework, including:
[0228] S2301: The playback controller receives a message that the decoding duration exceeds a duration threshold;
[0229] S2302: The playback controller injects video data and audio data into the peripheral device;
[0230] S2303: The audio decoder completes decoding of the first frame of audio data.
[0231] S2304: The audio decoder sends an audio data presence message to the playback controller.
[0232] S2305: The video decoder completes decoding of the first frame of video data.
[0233] S2306: The video decoder sends a video data presence message to the playback controller.
[0234] S2307: The playback controller determines that the audio and video are synchronized.
[0235] S2308: The playback controller injects video data and audio data into the peripheral device.
[0236] As shown in the audio and video playback signaling diagram in Figure 23, when the decoding time required for the target video data received by the playback controller from the chip layer exceeds the decoding time threshold, the playback controller directly injects the preset media data into the peripheral device for playback. Then, when the audio decoder completes decoding the first frame of audio data, it sends an audio data arrival message to the playback controller. When the video decoder completes decoding the first frame of video data, it sends a video data arrival message to the playback controller. Based on the audio data arrival message and the video data arrival message, the playback controller determines that the audio and video are synchronized and then injects the target media data into the peripheral device.
[0237] Referring to FIG. 24 , an embodiment of the present application provides a signaling interaction process for executing an audio and video playback process in a bottom-layer framework, including:
[0238] S2401: The playback controller receives a message that the decoding duration exceeds a duration threshold;
[0239] S2402: The audio decoder completes decoding of the first frame of audio data.
[0240] S2403: The audio decoder sends an audio data presence message to the playback controller.
[0241] S2404: The playback controller injects preset media data and target audio data into the peripheral device;
[0242] S2405: The video decoder completes decoding of the first frame of video data.
[0243] S2406: The video decoder sends a video data presence message to the playback controller.
[0244] S2407: The playback controller stops injecting preset media data into the peripheral device.
[0245] S2408: The playback controller injects target video data and target audio data into the peripheral device.
[0246] As can be seen in the audio and video playback signaling diagram shown in Figure 24, the playback controller may receive the target video data sent by the chip layer and the required decoding time exceeds the decoding time threshold. Then the playback controller waits for the audio data appearance message sent by the audio decoder. The playback controller determines the presence of audio data and video data, and injects the target audio data and preset media data (including only video data) into the peripheral device. Then, when the video decoder completes decoding the first frame of video data, it sends a video data appearance message to the playback controller. Then, the playback controller stops injecting the preset media data into the peripheral device and injects the target video data and target audio data into the peripheral device.
[0247] The decoding time threshold may be pre-stored in the display device 200 or may be a parameter set by the user. As shown in the user interface of FIG25 , the user can set the decoding time threshold according to the actual usage scenario, and can also set different decoding time thresholds for different media asset types. For example, in the user interface shown in FIG25 , a decoding time threshold N1 is set for target media asset data of the type corresponding to media asset identifier A, a decoding time threshold N2 is set for target media asset data of the type corresponding to media asset identifier B, and a decoding time threshold N3 is set for target media asset data of the type corresponding to media asset identifier C. The user can set different decoding time thresholds for different media asset types in the user interface shown in FIG25 .
[0248] After loading the target media asset data, the display device 200 searches the configuration file for the required decoding time of the corresponding target video data based on the media asset identifier carried in the target media asset data. It then searches for the corresponding decoding time threshold based on the media asset identifier and compares the required decoding time of the target video data with the decoding time threshold.
[0249] In the above embodiment, if the decoding time threshold found is 2 seconds and the decoding time required for the target video data is 5 seconds, the decoding time required for the target video data exceeds the decoding time threshold, and the step of playing the preset media asset data can be performed. If the decoding time threshold found is 6 seconds and the decoding time required for the target video data is 5 seconds, the decoding time required for the target video data does not exceed the decoding time threshold, and the preset media asset data can be omitted, and the step of simultaneously playing the target video data and target audio data can be performed.
[0250] The specific implementation of the above process at the bottom layer is as follows: the audio decoder needs to decode the target audio data for a decoding time of Y1. For example, the decoding time required for the target audio data is 1 second, that is, the audio decoder takes 1 second to decode the first frame of the target audio data before sending an audio data appearance message to the playback controller. The video decoder needs to decode the target video data for a decoding time of 5 seconds, that is, the video decoder takes 5 seconds to decode the first frame of the target video data before sending a video data appearance message to the playback controller. In other words, the playback controller needs to wait at least 5 seconds to receive the audio and video synchronization message.
[0251] If the decoding time threshold is 2 seconds, it is determined that the decoding time of the target video data exceeds the decoding time threshold. The determination process may be performed by the underlying chip, which then sends the determination result to the playback controller. The playback controller may control the display and / or speaker to turn on and then play the preset media data. The preset media data includes at least one of video data, image data, and audio data.
[0252] If the preset media data only includes video data or image data, the playback controller can only control the display to turn on, that is, before playing the target video data and target audio data, only the preset video data or preset image data is played on the display. If the preset media data includes video data and audio data, or includes image data and audio data, the playback controller can control the display and the speaker to turn on, that is, before playing the target video data and target audio data, the preset video data or preset image data can be played on the display, and at the same time, the speaker plays the sound according to the preset audio data. If the preset media data only includes audio data, the playback controller can only control the speaker to turn on, that is, before playing the target video data and target audio data, the speaker plays the sound according to the preset audio data.
[0253] The above three situations can ensure that when the decoding time required for the target video data exceeds the decoding time threshold, the preset video data and the preset audio data are played, or the preset video data is played, or the preset audio data is played, thereby avoiding the display device from remaining in a black screen and silent state for a long time. It should be noted that this is an embodiment of the present application that avoids the display device from remaining in a black screen and silent state for a long time. (By playing preset video data and preset audio data at the same time, a long black screen and silent state is avoided; by playing only preset video data, a long black screen state is avoided but a long silent state does not need to be avoided; by playing only preset audio data, a long silent state is avoided but a long black screen state does not need to be avoided).
[0254] In some embodiments, after decoding the target audio data, but the decoding time required for the target video data exceeds a decoding time threshold, the target audio data and the preset media asset data can be played simultaneously. For example, the target media asset data is a video data type of the "still pictures" type. In this video data type, what is actually played is a group of pictures, that is, the target video data is a group of pictures, not dynamic pictures, and the target audio data is music-type audio data. The user is actually not very concerned about the video data being played, but is actually more concerned with the target audio data. Therefore, in this case, the target audio data can be played first (no need to synchronize the target audio data with the target video data), and the preset media asset data can be played. In the user interface shown in Figure 26, before loading the target video data, the target audio data is played and a static music cover image is displayed. Alternatively, in the user interface shown in Figure 27, before loading the target video data, the target audio data is played and a prompt message "Still pictures program, loading pictures..." is displayed. From this prompt message, the user can understand that the target media asset data being played is of the "still pictures" type.
[0255] The above process is specifically implemented at the bottom layer as follows: the audio decoder takes 1 second to decode the target audio data, meaning it takes 1 second to decode the first frame of the target audio data before sending an audio data appearance message to the playback controller. The video decoder takes 5 seconds to decode the target video data, meaning it takes 5 seconds to decode the first frame of the target video data before sending a video data appearance message to the playback controller. In other words, the playback controller needs to wait at least 5 seconds to receive the audio and video synchronization message. If the target media asset data type is determined to be still pictures, the playback controller can control the display and speakers to turn on simultaneously, with the display presenting the music cover image or prompt information, while the speakers play sound based on the target audio data.
[0256] It should be noted that in this case, after acquiring the target media data, the display device 200 needs to wait for the target audio data to be decoded before playing the sound. However, when presenting a music cover image or prompt information, it is possible to wait for the target audio data to be decoded, or it is possible to present the music cover image or prompt information as soon as the target media data is acquired, and then play the target audio data after the target audio data is decoded. If the preset media data includes preset audio data and preset video data, the preset audio data and preset video data can be played simultaneously when the target media data is acquired, without waiting for the target audio data to be decoded.
[0257] If the decoding time required for the target video data exceeds the decoding time threshold, after at least the preset media asset data has been played, if the target video data is decoded, in order to synchronize the audio data with the video data, the following playback situations exist:
[0258] If the target audio data is played while the preset media data is being played, the preset media data is canceled when the target video data is decoded. The target audio data is then played starting from the first audio frame of the completed target audio data, and the target video data is played starting from the first video frame. This allows the target audio data and the target video data to be played synchronously.
[0259] If the target audio data is not played while the preset media data is being played, the preset media data is canceled when the target video data is decoded. Then the target audio data is played starting from the first audio frame of the target audio data, and the target video data is played starting from the first video frame at the same time. This also achieves the effect of synchronous playback of the target audio data and the target video data.
[0260] In some embodiments, if the decoding time required for the target video data exceeds a decoding time threshold, but no preset media asset data is currently stored in the system, a prompt message may be displayed to the user, informing the user that the system does not currently store the preset media asset data. For example, a play command input by the user may be obtained, and the media asset identifier carried in the play command may be determined to be a type of media asset requiring playback of preset media asset data. Therefore, it is determined that the decoding time required for the target video data does not exceed the decoding time threshold. Furthermore, it is determined that no preset media asset data is currently stored in the system. In this case, a prompt message may be displayed in the user interface, as shown in FIG28 , stating that the system does not store the preset media asset data. Please confirm that you wish to play this media asset.
[0261] Below the prompt message, there are also "OK" and "Cancel" buttons. If the user selects the "OK" button, in response to the confirmation instruction input by the user by selecting the "OK" button, it is necessary to wait for the audio data and video data to be decoded, that is, it is necessary to show the user a black screen and silence for a long time before the display appears and the speaker plays sound. If the user selects the "Cancel" button, in response to the cancellation instruction input by the user by selecting the "Cancel" button, the user interface shown in Figure 28 can be jumped back to the user interface for selecting media resources, for example, jumping back to the main page of the media resource platform.
[0262] In some embodiments, if the decoding time required for the target video data exceeds a decoding time threshold, but preset media asset data is currently stored in the system, a prompt message may be displayed to the user to remind the user that the current system stores preset media asset data. If the current media asset is selected, the preset media asset data must be played before the selected media asset is played. For example, a play instruction input by the user is obtained, and it is determined that the play instruction carries a media asset identifier and is a type of media asset that requires playback of preset media asset data. Therefore, it is determined that the decoding time required for the target video data exceeds the decoding time threshold. At the same time, it is also determined that preset media asset data is currently stored in the system. In this case, as shown in FIG29 , a prompt message "The system stores preset media asset data. Are you sure you want to play this media asset" may be displayed in the user interface.
[0263] Below the prompt message, there are also "OK" and "Cancel" buttons. If the user selects the "OK" button, in response to the confirmation instruction input by the user by selecting the "OK" button, it is necessary to wait for the audio data and video data to be decoded, at which time the preset media asset data can be displayed to the user. If the user selects the "Cancel" button, in response to the cancellation instruction input by the user by selecting the "Cancel" button, the user interface shown in Figure 29 will jump back to the user interface for selecting media assets, for example, jumping back to the main page of the media asset platform.
[0264] In some embodiments, the setting function of the display device 200 can also be used to set whether to determine whether the decoding time of the target video data included in the media data exceeds the decoding time threshold when a certain media asset is clicked. The setting can be performed as a whole, uniformly for a certain type of media asset, or for a single media asset.
[0265] For example, the user interface shown in FIG30 includes an "Overall Settings" option, a "Media Asset Type" option, and a "Media Asset Name" option. Below each option are also provided an "OK" button and a "Cancel" button. If the user selects the "OK" button below the "Overall Settings" option, in response to the confirmation instruction input by the user, after the setting is successful, no matter which type of media asset the user chooses to play, it is necessary to first determine whether the decoding time required for the target video data exceeds the decoding time threshold. Then, if the decoding time required for the target video data exceeds the decoding time threshold, at least the preset media asset data will be played. If the decoding time required for the target video data does not exceed the decoding time threshold, the preset media asset data will not be played.
[0266] If the user enters media asset identifier A in the "Media Asset Type" input box and selects the "OK" button below the "Media Asset Type" option, in response to the user's confirmation instruction, after the setting is successfully completed, whenever the user selects to play media of the type corresponding to media asset identifier A, it is necessary to first determine whether the decoding time required for the target video data exceeds the decoding time threshold. If the decoding time required for the target video data exceeds the decoding time threshold, at least the preset media asset data will be played. If the decoding time required for the target video data does not exceed the decoding time threshold, the preset media asset data will not be played. If the user selects another type of media, it is not necessary to determine whether the decoding time required for the target video data exceeds the decoding time threshold. The process of synchronously playing the target video data and target audio data can be directly waited for the decoding of both the target video data and the target audio data to be completed. The type of media corresponding to media asset identifier A can be a type of media that the user knows from experience that the decoding time required for the target video data exceeds the decoding time threshold. The user interface shown in Figure 28 only shows one input box for entering a media type. In actual applications, multiple input boxes for entering media types can be provided. For example, media assets of the type corresponding to media asset identifier A and media assets of the type corresponding to media asset identifier B are selected at the same time. In this way, if the user chooses to play these types of media assets, he or she can first determine whether the decoding time required for the target video data and target audio data exceeds the decoding time threshold.
[0267] If the user enters the media asset name XXX in the input box of the "Media Asset Name" option and selects the "OK" button below the "Media Asset Name" option, in response to the confirmation instruction entered by the user, after the setting is successful, as long as the user chooses to play the media asset with the media asset name XXX, or the media asset with the media asset name containing XXX, it is necessary to first determine whether the decoding time required for the target video data exceeds the decoding time threshold. Then, if the decoding time required for the target video data exceeds the decoding time threshold, at least the preset media asset data will be played. If the decoding time required for the target video data does not exceed the decoding time threshold, the preset media asset data will not be played. If the user chooses to play media with other names, it is not necessary to determine whether the decoding time required for the target video data exceeds the decoding time threshold. You can directly wait for the decoding of the target video data and the target audio data to be completed, and then proceed with the process of synchronously playing the target video data and the target audio data.
[0268] In some embodiments, if the decoding time required for the target video data exceeds the decoding time threshold, at least the preset media data is played. However, if the preset media data has not been played when the target video data is decoded, the option of continuing to play the preset media data or playing the target media data can also be provided.
[0269] For example, if the target video data has been decoded and the preset media asset data has not yet been played, a prompt message will pop up in the user interface shown in FIG31: "The target media asset data has been loaded. Do you want to play it?" Below the prompt message, there are also "OK" and "Cancel" buttons. If the user selects the "OK" button, the playback of the preset media asset data is canceled in response to the user's input confirmation instruction, and the target media asset data is played. If the user selects the "Cancel" button, the playback of the preset media asset data is not canceled in response to the user's input confirmation instruction. Instead, the playback of the preset media asset data is waited for to complete, and then the target media data is played.
[0270] In some embodiments, if the display device 200 stores multiple preset media asset data, the preset media asset data can be categorized, and the categorization can also be performed based on the media asset type of the target media asset data. For example, the target media asset data may include the type corresponding to media asset identifier A, the type corresponding to media asset identifier B, the type corresponding to media asset identifier C, and so on. When categorizing the preset media asset data, the categorization can also be performed based on the type corresponding to media asset identifier A, the type corresponding to media identifier B, the type corresponding to media identifier C, and so on. In this way, if the decoding time required for the target video data exceeds a decoding time threshold, the system can search for preset media data of the same media type based on the media asset type of the target video data for playback.
[0271] In some embodiments, if the playback instruction input by the user carries the media identifiers of multiple target media data, that is, the user may choose to play multiple target media data, and the multiple target media data can be played in the order in which they are selected, or they can be played in the order in which they are loaded. In this way, when multiple target media data are played in sequence, the intervals of the video frames included in different target media data may be different. When one of the target media data is played, due to the different video frame intervals, the next target media data may not have been loaded yet, or there may be a black screen and silent state for a certain period of time. Therefore, when different target media data are played continuously, not only the first target media data needs to determine whether the decoding time required for the target video data exceeds the decoding time threshold, but other target media data also need to determine whether the decoding time required for the target video data exceeds the decoding time threshold.
[0272] For example, the play instruction carries a media asset identifier A and a media asset identifier B, where media asset identifier A corresponds to the first target media asset data and media asset identifier B corresponds to the second target media asset data, and the first target media asset data includes the first target video data and the first target audio data, and the second target media asset data includes the second target video data and the second target audio data. If the play order of the first target media asset data is before the play order of the second target media asset data, it is first determined whether the decoding time of the first target video data exceeds the decoding time threshold. If the decoding time of the first target video data exceeds the decoding time threshold, the preset media asset data is played, and when the decoding of the first target video data is completed, the first target video data and the first target audio data are played synchronously.
[0273] Then, when the playback of the first target media data is complete, a determination is made as to whether the difference between the playback duration of a single frame of the second target video data and the decoding duration of the first target video data exceeds a decoding duration threshold. As shown in the decoding principle diagram of FIG32 , this is because when the decoding of the last frame of the first target video data is complete, the video decoder begins decoding the first frame of the second target video data. In other words, while the last frame of the first target media data is being played, the first frame of the second target video data is being decoded.
[0274] The playback duration of a single video frame of the first target video data is t1. The decoding duration of a single video frame of the second target video data is t2. If t1 is greater than or equal to t2, it means that the last video frame of the first target video data has not been played yet, the first video frame of the second target video data has been decoded, or the first video frame of the second target video data has just been decoded when the last video frame of the first target video data has been played. At this time, regardless of whether the decoding duration of the second target video data exceeds the duration threshold, a black screen and silent state will not appear, and there is no need to determine whether the decoding duration of the second target video data exceeds the duration threshold.
[0275] If t1 is less than t2, this means that when the last frame of the first target video data is played, the first frame of the second target video data has not yet been decoded. In other words, the second target video data cannot be connected to the first target video data. In this case, it is necessary to determine whether the value of t2-t1 exceeds the decoding time threshold. If the value of t2-t1 exceeds the decoding time threshold, the preset media data needs to be played. If the value of t2-t1 does not exceed the decoding time threshold, the preset media data does not need to be played.
[0276] Based on the media resource playback method described in the above embodiment, as shown in the flowchart of FIG33 , the following is a specific application process of the media resource playback method provided in the embodiment of the present application, which specifically includes:
[0277] S3010: Start the display device 200 to prepare for playing media data;
[0278] S3011: After acquiring the target media asset data, start decoding the target video data and target audio data included in the target media asset data.
[0279] S3012: After the target audio data is decoded, the bottom chip sends an audio data presence message to the playback controller.
[0280] S3013: Determine whether the media asset is of the still pictures type. If not, execute step S3014; if so, execute step S3016.
[0281] S3014: Wait for the audio and video synchronization message.
[0282] S3015: The playback controller receives the audio and video synchronization message, opens the audio and video output, and plays the target audio data and the target video data.
[0283] S3016: Turn on audio output, play the target audio data, and notify the application to display a default image or prompt information.
[0284] S3017: Wait for a video data appearance message. If a video data appearance message is received, execute step S3018. If no video data appearance message is received, continue to execute step S3017.
[0285] S3018: Notify the application to stop displaying the default image or prompt information.
[0286] S3019: Open the video output and play the target video data.
[0287] The same and similar parts between the various embodiments of the present application can be referenced to each other and will not be described again here.
[0288] Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the parts of the embodiments of the present application that are essentially or contribute to the relevant technology can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.
[0289] For ease of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are intended to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.
Claims
1. A display device, comprising: a display configured to display an image and / or a user interface; A user interface configured to receive instructions from a user; a communication device configured to communicate with an external device according to a predetermined protocol; a memory configured to store computer instructions and data associated with a display device; At least one processor, connected to the display, user interface, communication device and memory, is configured to execute computer instructions to cause the display device to perform: In response to a playback instruction of the media asset data, a data transmission stream of the media asset data is acquired; the data transmission stream includes MPU metadata, segment metadata and MFU data; Detecting the transmission order of MPU metadata, fragment metadata and MFU data in the data transmission stream; If the transmission order is the target order, injecting the data transmission stream into the player so as to perform decoding and playback on the data transmission stream in real time through the player; If the transmission order is not the target order, encapsulating the data transmission stream into media transmission packets according to the target order, and injecting the media transmission packets into the player.
2. The display device according to claim 1, wherein the at least one processor is further configured to execute the computer instructions so that the display device performs: Monitor the manipulation events of upper-layer applications; In response to a manipulation event of switching channels or accessing channels, detecting a target channel accessed by the display device; Query the media resource data corresponding to the target channel; The play instruction is generated according to the media asset data.
3. The display device of claim 1 , wherein the at least one processor is further configured to execute the computer instructions so that the display device performs: Establish a media transmission channel according to the media transmission protocol of the media data; Acquire media presentation information signaling through the media transmission channel, wherein the media presentation information signaling includes a presentation information table and a component description table; The media asset data is queried according to the presentation information table and the component description table.
4. The display device according to claim 3, when acquiring the data transmission stream of the media asset data, the at least one processor is further configured to execute the computer instructions so that the display device executes: Call the channel interface of the protocol stack middleware; Controlling the target channel accessed by the display device through the channel interface; The media presentation information signaling and media data of the target channel are received.
5. The display device of claim 1 , wherein the at least one processor is further configured to execute the computer instructions so that the display device performs: Start the media server of the protocol stack middleware; Establishing a connection relationship between the protocol stack middleware and the player through the media server; The player is called based on the connection relationship, and the data transmission stream or the media transmission packet is injected into the player.
6. The display device according to claim 1, when decoding and playing the data transmission stream, the at least one processor is further configured to execute the computer instructions so that the display device executes: Initializing the decoder of the player; Decoding the data transmission stream by the decoder; The underlying resources are called to render a playback screen of the media asset data on the display.
7. The display device according to claim 1, wherein the target sequence is MPU metadata, fragment metadata, and MFU data.
8. A display device, comprising: a display configured to display an image and / or a user interface; A user interface configured to receive instructions from a user; a communication device configured to communicate with an external device according to a predetermined protocol; a memory configured to store computer instructions and data associated with a display device; At least one processor, connected to the display, user interface, communication device and memory, is configured to execute computer instructions to cause the display device to perform: In response to a playback instruction of the media asset data, a data transmission stream of the media asset data is received, wherein the data transmission stream includes an MPU element data, fragment metadata, and MFU data; Cache the data transmission stream into a player cache area; When MPU metadata and fragment metadata are cached in the player cache area, the data in the player cache area is decapsulated and the data in the player cache area is injected into the player.
9. The display device according to claim 8, wherein the player buffer area comprises a metadata buffer area and an MFU data buffer area, and when the data transmission stream is cached in the player buffer area, the at least one processor is further configured to execute the computer instructions so that the display device executes: Detecting the data type of the data in the data transmission stream; If the data type is metadata, cache the metadata in the metadata cache area, wherein the metadata includes the MPU metadata and the fragment metadata; If the data type is MFU data, the MFU data is cached in the MFU data cache area.
10. A media asset playing method, comprising: In response to a playback instruction of the media asset data, acquiring a data transmission stream of the media asset data; The data transmission stream includes MPU metadata, fragment metadata and MFU data; Detecting the transmission order of MPU metadata, fragment metadata and MFU data in the data transmission stream; If the transmission order is the target order, injecting the data transmission stream into the player so as to perform decoding and playback on the data transmission stream in real time through the player; If the transmission order is not the target order, encapsulating the data transmission stream into media transmission packets according to the target order, and injecting the media transmission packets into the player.
Citation Information
Patent Citations
Method for supporting media multicast, related device and system
CN104471895A
Multiserver-based stream media transmitting device and stream media transmitting method
CN104580067A
Transmission apparatus, transmission method, reproduction apparatus, reproduction method and reception apparatus
CN105308974A
Method and device for sending MMT packet and method for receiving MMT packet
CN105491394A
Reception device and reception method
CN112954435A
