Display device and audio and video synchronization method

Abstract

Embodiments of the invention provide a display device and an audio and video synchronization method. The method comprises the steps of receiving media asset data sent by a target application; setting the time of the video data of the first target frame number as a first reference clock, and performing first synchronization processing on the audio data and the video data according to the first reference clock; decoding the synchronously processed audio data and video data according to a second reference clock to obtain decoded audio data and video data; and setting the time corresponding to the decoded video data of the second target frame number as a third reference clock, performing second synchronization processing on the decoded audio data and video data according to the second reference clock and the third reference clock, and playing the audio data and video data after the second synchronization processing. According to the method, the audio data and the video data are synchronously processed twice, so that the synchronization precision can be improved, and the problem that the audio and the video are asynchronous is solved.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and an audio and video synchronization method. Background Art

[0002] Display devices are intelligent devices that can present user interfaces and support user interaction. Taking smart TVs as an example, smart TVs are based on Internet application technology, have open operating systems and chips, have open application platforms, can realize two-way human-computer interaction functions, and are TV products that integrate multiple functions such as audio and video, entertainment, and data to meet the diverse and personalized needs of users. Display devices can present different media recommendation interfaces, which can include multiple media options for users to select and play.

[0003] A variety of applications can be deployed in a display device, and the applications may contain media data such as audio data and video data. However, due to the different transmission speeds and data volumes of audio data and video data, there may be a problem of audio and video being out of sync. Summary of the invention

[0004] The present application provides a display device and an audio and video synchronization method to solve the problem of audio and video asynchrony.

[0005] In a first aspect, some embodiments of the present application provide a display device, including.

[0006] a display configured to display a user interface;

[0007] The controller is configured as:

[0008] Receiving media data sent by a target application; the media data includes audio data and video data;

[0009] Before decoding the audio data and the video data, setting the time corresponding to the video data of a first target frame number as a first reference clock, and performing a first synchronization process on the audio data and the video data according to the first reference clock;

[0010] Setting the system startup time as the second reference clock, and decoding the audio data and the video data after the first synchronization processing according to the second reference clock to obtain decoded audio data and video data; the system startup time at least includes the boot time or the networking time;

[0011] Setting the time corresponding to the decoded video data of the second target frame number as the third reference clock, and performing a second synchronization process on the decoded audio data and video data according to the second reference clock and the third reference clock;

[0012] Play the audio data and video data after the second synchronization process.

[0013] The above technical solution has the following advantages or beneficial effects: through the first synchronization processing and the second synchronization processing of audio data and video data, the synchronization accuracy of audio data and video data can be improved, the smoothness and continuity of the playback effect can be ensured, and the problem of audio and video asynchrony can be solved.

[0014] In some embodiments, the controller performs a first synchronization process on the audio data and the video data according to the first reference clock, and is specifically configured as follows:

[0015] Compare a first time difference between the video data and the audio data of the first target frame number; if the first time difference does not exceed a first threshold range, decode the video data and the audio data of the first target frame number; if the first time difference exceeds the first threshold range, adjust the audio data or the video data of the next frame of the first target frame number according to the first time difference.

[0016] The above technical solution has the following advantages or beneficial effects: when the first time difference exceeds the first threshold range, it indicates that there is a problem of audio and video asynchrony, and it is necessary to dynamically adjust the frame output time of the next frame of audio data or video data according to the time difference to correct the phenomenon of audio and video asynchrony, and ensure the coarse synchronization of audio data and video data before decoding through the first reference clock.

[0017] In some embodiments, the controller adjusts the audio data or the video data of the next frame of the first target frame number according to the first time difference, and is specifically configured as follows:

[0018] Determine the delay time according to the first time difference; obtain the actual frame sending time and the frame sending time threshold of the video data of the first target frame number; if the actual frame sending time is earlier than the frame sending time threshold, delay the fetching of the video data of the next frame of the first target frame number according to the delay time; if the actual frame sending time is later than the frame sending time threshold, delay the fetching of the audio data of the next frame of the first target frame number according to the delay time.

[0019] The above technical solution has the following advantages or beneficial effects: by dynamically adjusting the playback speed of audio data and video data, the problem of audio data and video data being out of sync is solved.

[0020] In some embodiments, before the controller performs a second synchronization process on the decoded audio data and video data according to the second reference clock and the third reference clock, the controller is further configured to:

[0021] Compare the frame delivery time corresponding to the video data of the second target frame number with the second time difference of the second reference clock; when the second time difference exceeds a second threshold range, discard the video data of the second target frame number.

[0022] The above technical solution has the following advantages or beneficial effects: when it is detected that the second time difference exceeds the second threshold range, the display device can choose to discard this part of the video data to ensure the synchronization and playback quality of subsequent video data.

[0023] In some embodiments, the controller performs a second synchronization process on the decoded audio data and video data according to the second reference clock and the third reference clock, and is specifically configured as follows:

[0024] When the second time difference does not exceed the second threshold range, render the decoded video data of the second target frame number; compare the decoded audio data with the third time difference of the third reference clock; if the third time difference does not exceed the third threshold range, render the audio data; if the third time difference exceeds the third threshold range, discard the audio data of the second target frame number.

[0025] The above technical solution has the following advantages or beneficial effects: by comparing the time difference between video data and audio data, if the time difference between the two is within the third threshold range, the audio data is rendered so that it is played synchronously with the video data, ensuring the synchronization of video data and audio data during playback.

[0026] In some embodiments, before receiving the media asset data sent by the target application, the controller is further configured to:

[0027] Obtain message types contained in the target application; the message types include control messages and data messages; the control message is used to characterize the playback status of the media data, and the data message includes at least the audio data and the video data; obtain the current playback status of the media data; adapt the control message according to the current playback status; when the control message is not compatible with the current playback status, discard the control message; when the control message is compatible with the current playback status, control the control message to be queued.

[0028] The above technical solution has the following advantages or beneficial effects: the display device can ensure that no erroneous operations are performed, thereby improving the stability and reliability of the system.

[0029] In some embodiments, the controller controls the control message to be enqueued, and is specifically configured to:

[0030] Obtain a configured control message queue; detect the adaptation status of the control message in the control message queue; if the adaptation status is that the control message is not connected to the control message queue, generate an add instruction; and, based on the add instruction, add the control message to the control message queue.

[0031] The above technical solution has the following advantages or beneficial effects: the control message queue can be used to store and manage all control messages to be processed, and the display device can ensure that no duplicate control messages are processed by checking whether the newly received control message already exists in the control message queue.

[0032] In some embodiments, after detecting the adaptation state of the control message in the control message queue, the controller is further configured to:

[0033] If the adaptation status is that the control message has been added to the message queue, the control message is adapted according to the current playing status; when the control message is not suitable for the current playing status, the control message is discarded.

[0034] The above technical solution has the following advantages or beneficial effects: after the control message has been added to the control message queue, the display device can determine whether the control message is applicable according to the current playback state. If it is found that a control message is not applicable to the current playback state, the display device can discard the control message to avoid performing invalid operations or generating erroneous results.

[0035] In some embodiments, after obtaining the message type included in the target application, the controller is further configured to:

[0036] Acquire the data format, data encoding and data length of the data message; verify the validity of the data message based on the data format, the data encoding and the data length; when the data message is valid, control the data message to be queued; when the data message is invalid, discard the data message.

[0037] The above technical solution has the following advantages or beneficial effects: by verifying the validity of the receipt message, the accuracy of data transmission and the stability of the system can be ensured. At the same time, by discarding invalid data messages, the system response speed can be improved.

[0038] In a second aspect, some embodiments of the present application provide a method for audio and video synchronization, including:

[0039] Receiving media data sent by a target application; the media data includes audio data and video data;

[0040] Before decoding the audio data and the video data, setting the time corresponding to the video data of a first target frame number as a first reference clock, and performing a first synchronization process on the audio data and the video data according to the first reference clock;

[0041] Setting the system startup time as the second reference clock, and decoding the audio data and the video data after the first synchronization processing according to the second reference clock to obtain decoded audio data and video data; the system startup time at least includes the boot time or the networking time;

[0042] Setting the time corresponding to the decoded video data of the second target frame number as the third reference clock, and performing a second synchronization process on the decoded audio data and video data according to the second reference clock and the third reference clock;

[0043] Play the audio data and video data after the second synchronization process.

[0044] The above technical solution has the following advantages or beneficial effects: through the first synchronization processing and the second synchronization processing of audio data and video data, the synchronization accuracy of audio data and video data can be improved, the smoothness and continuity of the playback effect can be ensured, and the problem of audio and video asynchrony can be solved.

[0045] It can be seen from the above technical solutions that some embodiments of the present application provide a display device and a method for audio and video synchronization, the method comprising: receiving media data sent by a target application; setting the time of the video data of the first target frame number as the first reference clock, and performing a first synchronization process on the audio data and the video data according to the first reference clock; decoding the synchronized audio data and the video data according to the second reference clock to obtain the decoded audio data and the video data; setting the time corresponding to the decoded video data of the second target frame number as the third reference clock, and performing a second synchronization process on the decoded audio data and the video data according to the second reference clock and the third reference clock. The method is based on the Driver Base architecture platform. When docking an application, it only needs to dock with the application docking module, which solves the problem of low application docking efficiency. By synchronizing the audio data and the video data twice, the synchronization accuracy can be improved and the problem of audio and video asynchrony can be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0048] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0049] Figure 3 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0050] Figure 4 A schematic diagram of a traditional media playback architecture provided in some embodiments of the present application;

[0051] Figure 5 A schematic diagram of an application docking adaptation scenario of a traditional media asset playback architecture provided in some embodiments of the present application;

[0052] Figure 6 A schematic diagram of a scenario after a traditional media asset playback architecture is adapted to an application in some embodiments of the present application;

[0053] Figure 7 A schematic flow chart of a method for performing audio and video synchronization on a display device provided in some embodiments of the present application;

[0054] Figure 8 A schematic diagram of a scenario of a Driver Base architecture platform provided in some embodiments of the present application;

[0055] Fig. 9 A schematic diagram of a scenario in which an application is connected to the Driver Base architecture platform provided in some embodiments of the present application;

[0056] Fig.10 A schematic diagram of the structure of an application docking module provided in some embodiments of the present application;

[0057] Fig.11 A schematic diagram of a process for a display device to perform queue management and control on application messages provided in some embodiments of the present application;

[0058] Fig.12 A schematic diagram of a flow chart of asynchronous processing of control messages and data messages by a display device provided in some embodiments of the present application;

[0059] Fig.13 A schematic diagram of the application playback module structure provided in some embodiments of the present application;

[0060] Fig.14 A timing diagram of audio and video synchronization performed by a display device provided in some embodiments of the present application;

[0061] Fig.15A schematic diagram of a flow chart of a display device performing pre-decoding synchronization on audio and video data provided by some embodiments of the present application;

[0062] Fig.16 A flowchart of a display device performing post-decoding synchronization on audio and video data provided in some embodiments of the present application. DETAILED DESCRIPTION

[0063] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0064] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and common meanings.

[0065] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0066] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0067] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0068] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0069] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1As shown in FIG. 1 , a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 is used to receive operation instructions input by the user and convert the operation instructions into control instructions that can be recognized and responded to by the display device 200. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0070] The mobile terminal 300 can be used as a control device for performing human-computer interaction between a user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0071] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application program for controlling the display device 200 .

[0072] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0073] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0074] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0075] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0076] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0077] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0078] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0079] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0080] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0081] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0082] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0083] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0084] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0085] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface, for example, the operating system can directly control the display device to provide a user interface, or can provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0086] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0087] The operating system can be divided into different modules or layers according to the functions implemented, for example Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications layer for short), the middleware layer, and the hardware layer.

[0088] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0089] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0090] like Figure 3 As shown, Figure 3 Some embodiments of the present application provide Figure 1 Schematic diagram of software configuration in the display device. In some embodiments, the system of the display device 200 can be divided into three layers, namely, application layer, middleware layer and hardware layer from top to bottom.

[0091] The application layer mainly includes commonly used applications on TV and application frameworks. Common applications are mainly applications developed based on browsers, such as HTML5 APPs and native APPs.

[0092] The Application Framework is a complete program model that has all the basic functions required by standard application software, such as file access, data exchange, etc., as well as the user interfaces of these functions (toolbars, status bars, menus, dialog boxes).

[0093] Native apps can support online or offline, message push or local resource access.

[0094] The middleware layer includes various TV protocols, multimedia protocols, system components and other middleware. The middleware can use the basic services (functions) provided by the system software to connect various parts of the application system or different applications on the network, and can achieve the purpose of resource sharing and function sharing.

[0095] The hardware layer mainly includes the hardware abstraction layer (HAL) interface, hardware and drivers. The HAL interface is a unified interface for all TV chips to connect, and the specific logic is implemented by each chip. Drivers mainly include: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver.

[0096] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0097] The display device 200 is an intelligent device that can present a user interface and support user interaction. Taking a smart TV as an example, a smart TV is based on Internet application technology, has an open operating system and chip, has an open application platform, can realize two-way human-computer interaction, and is a TV product that integrates multiple functions such as audio and video, entertainment, and data to meet the diverse and personalized needs of users. The display device can present different media recommendation interfaces, and the media recommendation interface can include multiple media options for users to select and play.

[0098] In some embodiments, multiple applications can be deployed in the display device, and the streaming media application layer can include multiple media data, such as application display pages (which can include various application apps), streaming media application pages such as Netflix and Amazon (which can display various media resources), mobile storage media application pages (which can include local multimedia storage), etc. The playback of media data relies on the media playback architecture.

[0099] Figure 4 A schematic diagram of a traditional media playback architecture provided in some embodiments of the present application, such as Figure 4 As shown, in some embodiments, the traditional media resource playback architecture can be divided into an application layer, an application management layer, an application playback layer, and an application display layer. The application layer refers to the aforementioned Netflix, Amazon, mobile storage media applications, etc., which are used to provide users with resources that can be played; the application management layer is used to manage different applications and provide the TV system with detailed playback information of each application; the application playback layer is used to parse specific media data, for example, the media data includes audio data, video data, and subtitle data, and the application playback layer is used to provide the hardware platform with renderable images and sounds; the application display layer is used to display the final audio and video images and sounds.

[0100] Figure 5 A schematic diagram of an application docking adaptation scenario of a traditional media asset playback architecture provided in some embodiments of the present application, such as Figure 5 As shown, in some embodiments, different applications usually have their own unique characteristics. In order to adapt different applications in the media asset playback platform, the traditional media asset playback architecture often needs to manage the newly connected applications separately according to the characteristics of the applications. If the hardware platform needs to expand new applications, it needs to be expanded according to the characteristics of the new applications.

[0101] Depend on Figure 5 It can be seen that every time a hardware platform expands an application, it needs to add the management module, playback module and display module of this application. In actual usage scenarios, each application is different and has its own characteristics. For the platform, the application that needs to be expanded is unpredictable, which brings high adaptation costs and low efficiency of application docking.

[0102] Figure 6 This is a schematic diagram of a scenario after the traditional media resource playback architecture is adapted to an application in some embodiments of the present application, such as Figure 6 As shown, after application n is adapted, the hardware platform includes application 1, application 2, ... application n. It is understandable that in actual use scenarios, in addition to the application needs to be updated, the platform will also be updated. When the platform has adapted all applications, if the platform needs to be upgraded, for example, the existing hardware platform is hardware platform n, and the platform to be upgraded is hardware platform n, then hardware platform n needs to adapt all applications of hardware platform 1 again. Moreover, if both the platform and the application need to be updated, the complexity of docking between the application and the platform may increase exponentially, and the docking efficiency is low.

[0103] In addition, the application may contain media data such as audio data and video data. Since the transmission speed and data volume of audio data and video data are different, there is a problem of audio and video asynchrony. For example, the data volume of audio data is small, the data volume of video data is large, and the video data may be more complex than the audio data. Therefore, the media playback platform may need more time to process the video data, resulting in the audio data being processed and played before the video data, resulting in audio and video asynchrony.

[0104] In summary, the traditional media playback architecture has problems such as low application docking efficiency and audio and video asynchrony.

[0105] Based on the above scenario, in order to improve the docking efficiency of applications in the media asset playback architecture and solve the problem of audio and video asynchronism, some embodiments of the present application provide a display device 200, wherein the display device 200 includes a display 260 and a controller 250. Among them, the display 260 is configured to display a user interface, and the controller 250 enables the display device 200 to execute an audio and video synchronization method by running a specific computer execution instruction. The media asset playback platform of the display device 200 is based on the DriverBase architecture. The Driver Base architecture is an audio and video decoding platform that is not related to the application. The data processing part of the display device 200 runs in the kernel space and is completely isolated from the user space. For the Driver Base architecture, the user's behavior is completely transparent. Based on the Driver Base architecture, an application docking module is added between the application and the application management module. Different applications are docked and adapted through the docking module, without the need to dock the application management module, the application playback module and the application display module, thereby improving the efficiency of application docking. At the same time, two-level synchronization is performed on the video data and audio data contained in the application to solve the problem of audio and video asynchronism.

[0106] In order to facilitate the understanding of the technical solutions in some embodiments of the present application, each step is described in detail below in conjunction with some specific embodiments and drawings. Figure 7 A flowchart of a method for performing audio and video synchronization on a display device provided in some embodiments of the present application is shown in FIG. Figure 7 As shown, in some embodiments, when the display device 200 performs the method for audio and video synchronization, the following steps may be included:

[0107] Step S1: the display device 200 receives media data sent by a target application; the media data includes audio data and video data.

[0108] In some embodiments, the display device 200 may receive media data sent by a target application, and the media data may include audio data and video data. The target application is deployed on a platform based on the Driver Base architecture, and the platform is provided with an application docking module, and the application docking module is used to dock multiple applications.

[0109] Figure 8 A schematic diagram of a scenario of a Driver Base architecture platform provided in some embodiments of the present application, such as Figure 8As shown, in some embodiments, considering that each application has its own uniqueness, but the overall playback has its commonality, an application docking module can be added between each application and the application management module based on the Driver Base architecture. The Driver Base architecture has the advantage of application versatility, does not rely on specific applications, and can seamlessly dock various applications. In this way, different applications can be docked through the application docking module, including but not limited to applications developed based on different programming languages, frameworks or protocols. In this way, the application docking module enables the platform to integrate multiple applications, reduce resource waste and repeated development costs caused by application incompatibility, and can not only solve the problem of low efficiency of platform docking applications, but also improve the compatibility and scalability of the platform.

[0110] Continue to combine Figure 8 The application docking module is used to dock each application upward, and adapts to the application management module, application playback module and application display module downward. When new applications such as application n need to be expanded, the traditional mode needs to be Figure 5 The application shown is connected to the application management module, application playback module and application display module respectively, while the platform based on the Driver Base architecture only needs to be connected to the application n in the application docking module. For example, the application docking module can be adapted in combination with the characteristics of application n. At the same time, due to the characteristics of the Driver Base architecture itself, it only needs to be adapted to the application display module, and no longer needs to be adapted to the application management module and the application playback module one by one. In this way, the application docking process will become a simplified process. Even if there are multiple applications to be docked, it is only necessary for multiple applications to be docked with the application docking module, thereby improving the efficiency of application docking.

[0111] Fig. 9 A schematic diagram of a scenario in which an application is connected to the Driver Base architecture platform provided in some embodiments of the present application, such as Fig. 9 As shown, application n has been connected to the application connection module. Since the application only needs to be connected to the application connection module when connecting to the Driver Base architecture platform, and the platform only needs to adapt the application display module, even if multiple applications need to be connected or the hardware platform needs to be upgraded, it is only necessary for multiple applications to be connected to the application connection module respectively, and only the application display module corresponding to the hardware platform needs to be upgraded. In this way, the application connection efficiency and the platform upgrade efficiency can be improved, the application connection and platform upgrade processes can be simplified, and the problem of low application connection efficiency can be solved.

[0112] Fig.10 A schematic diagram of the structure of the application docking module provided in some embodiments of the present application, such as Fig.10As shown, in some embodiments, the application docking module may include four parts: an application message receiving module, an application message classification module, an application message storage module, and an application message forwarding module. Among them, the application message receiving module is to receive messages from different applications, the application message classification module is to classify messages from different applications, the application message storage module is to store application messages, and the application message forwarding module is to forward application messages.

[0113] In some embodiments, before receiving the media data sent by the target application, the display device 200 can also obtain the message type contained in the target application, where the message type may include a control message and a data message, the control message is used to characterize the playback status of the media data, and the data message includes at least audio data and video data; thereafter, the current playback status of the media data can be obtained, and the control message can be adapted according to the current playback status; when the control message is not compatible with the current playback status, the display device 200 can discard the incompatible control message; when the control message is compatible with the current playback status, the display device 200 can control the control message to be queued.

[0114] That is to say, the control message is mainly used to characterize the playback status of the media data, such as operations such as play, pause, fast forward, etc. The data message is the actual transmitted media content. By obtaining the current playback status of the media data, the display device 200 can obtain the current playback progress and status in real time, providing a basis for the subsequent control message adaptation. When adapting the control message according to the current playback status, for example, if it is currently in a paused state, the adapted control message may include a "play" instruction. If it is currently in a playback state, the adapted control message may be a "pause" instruction, "fast forward" or "exit" instruction, etc. If the control message is not suitable for the current playback state, the display device 200 will discard these unsuitable control messages. If the control message is suitable for the current playback state, the display device 200 will queue the control message. In this way, when receiving the application message, the display device 200 can use the message queue to control the entry of different messages, and filter whether the control message is queued according to the current playback state, which can ensure that no wrong operation is performed, thereby improving the stability and reliability of the system.

[0115] In some embodiments, after obtaining the message type contained in the target application, the display device 200 can also obtain the data format, data encoding and data length of the data message, and verify the validity of the data message based on the data format, data encoding and data length; when the data message is valid, the display device 200 controls the data message to be queued; when the data message is invalid, the display device 200 discards the invalid data message.

[0116] For example, Fig.11A schematic diagram of a process for a display device to perform queue management on an application message provided in some embodiments of the present application, such as Fig.11 As shown, after receiving the message in the application, the application docking module can classify the message. The application message can be divided into two types of messages: control message and data message. The control message includes startup, start, audio track switching, video stream switching, pause, fast forward, fast rewind, exit, etc.; the data message can include audio data, video data and subtitle data. For the control message, the queue process is executed if it is adapted to the playback state, and the control message is directly discarded if it is not adapted. For the data message, its validity can be checked. For example, its data format, data encoding and data length are checked. If the data format, data encoding and data length meet the preset requirements, it is valid data, otherwise it is invalid data. If the data message is valid, the display device 200 can control these valid data messages to enter the queue, and if the data message is invalid, it is directly discarded. By verifying the validity of the receipt message, the accuracy of data transmission and the stability of the system can be ensured. At the same time, by discarding invalid data messages, the system response speed can be improved.

[0117] Fig.12 A flow chart of asynchronous processing of control messages and data messages by a display device provided in some embodiments of the present application. In some embodiments, the message asynchronous processing thread and the application message receiving module of the application docking module are started synchronously, and the message asynchronous processing thread can be used to judge and process messages in the message queue.

[0118] Exemplarily, for the processing flow of the control message asynchronous thread, after receiving the control message, the control message asynchronous thread can be started, and the control message queue can be checked to see if there is a control message. If there is a control message in the queue, the control message is controlled to be dequeued, and the control message is forwarded to execute subsequent steps. If there is no control message in the queue, the display device 200 can delay for a certain period of time and check again. For the processing flow of the data message asynchronous thread, after receiving the data message, the data message asynchronous thread can be started, and the data message queue can be checked to see if there is a data message. If there is no data message in the queue, the display device 200 can delay for a certain period of time and check again. If there is a data message in the queue, the data message can be controlled to be dequeued, and the data message can be forwarded.

[0119] In some embodiments, the display device 200 can control the enqueueing of control messages in the following manner. First, the display device 200 can obtain a configured control message queue, and then detect the adaptation status of the control message in the control message queue; if the adaptation status is that the control message is not connected to the control message queue, an add instruction is generated; and, based on the add instruction, the control message is added to the control message queue. In this way, the control message queue can be used to store and manage all pending control messages, and the display device 200 can ensure that no duplicate control messages are processed by checking whether the newly received control message already exists in the control message queue.

[0120] In some embodiments, after detecting the adaptation state of the control message in the control message queue, the display device 200 can adapt the control message according to the current playback state if the adaptation state is that the control message has been added to the message queue, and discard the control message when the control message is not suitable for the current playback state. In this way, after the control message has been added to the control message queue, the display device 200 can determine whether the control message is applicable according to the current playback state. If it is found that a control message is not applicable to the current playback state, the display device 200 can discard the control message to avoid performing invalid operations or generating erroneous results.

[0121] The following takes the platform expansion of an application based on the Driver Base architecture as an example to explain in detail the process of enqueuing and dequeuing control messages and data messages.

[0122] Exemplarily, the implementation steps of control message queueing are as follows: after the new application is docked, it starts to prepare to receive control messages and starts the control message receiving thread. When a new application has a control message arriving (control messages include but are not limited to startup, start broadcasting, audio track switching, video stream switching, pause, fast forward, fast rewind, exit instructions, etc.), the display device 200 can determine whether the current control message queue already contains this message. If not, add this new control message. If it is already included, it can be determined whether the control message is compatible with the current playback state. When judging whether the two are compatible, the current playback state can include but is not limited to not started, started, playing, paused, etc. If the control message is not compatible with the current playback state, the control message is discarded. If the control message is not compatible with the current playback state, the control message can be queued, and the queueing link is completed.

[0123] The implementation steps for controlling message dequeue are as follows: start the message asynchronous processing thread, and periodically query the control message queue to see if there is any message added. If no message is added, continue to periodically query the control message queue. If a message is added, the dequeue process can be executed for the control message.

[0124] The steps for implementing data message queueing are as follows: after the new application is docked, it starts to prepare to receive data messages and starts the data message receiving thread. When a data message arrives from the new application (the data message may include but is not limited to audio data, video data, subtitle data), the display device 200 may determine whether the current data message is valid, and the valid method includes but is not limited to data format, data encoding, data length, etc. If the current data message is invalid, the data message is discarded, and if the current data message is valid, the data message is queued.

[0125] The steps for dequeuing data messages are as follows: start the message asynchronous processing thread, and periodically query the data message queue to see if there are any data messages added. If no messages are added, continue to periodically query the data message queue. If there are any messages added, the dequeuing process can be executed for the data message.

[0126] Fig.13 A schematic diagram of the application playback module structure provided in some embodiments of the present application, such as Fig.13 As shown, in some embodiments, the application playback module based on the Driver Base architecture may include five parts: an audio and video data receiving module, an audio and video data decoding module, an audio and video data synchronization module, an audio and video data rendering module, and an audio and video data playback module. Among them, the audio and video data receiving module is used to receive audio and video data from the application, the audio and video data decoding module is used to decode the audio and video data, the audio and video data synchronization module is used to perform the audio and video data synchronization function, the audio and video data rendering module is used to render the audio and video data, and the audio and video data playback module can be used to play the synchronized audio data and video data.

[0127] In some embodiments, the decoding and rendering of audio and video data can be completed by a decoder and a driver, respectively, for example, an H264 decoder, an H265 decoder, a Dobly vision decoder, and a DoblyATMOS decoder, etc. In actual use scenarios, it can be selected according to actual needs, and this application does not make specific limitations on this.

[0128] Fig.14 A timing diagram of audio and video synchronization performed by a display device provided in some embodiments of the present application, such as Fig.14 As shown, in some embodiments, the audio and video data receiving module may receive the media data sent by the target application, wherein the media data includes audio data and video data. Afterwards, other modules may perform subsequent decoding, synchronization, and rendering processes. After step S1 is completed, the following step S2 may be performed.

[0129] Step S2: before decoding the audio data and the video data, the display device 200 sets the time corresponding to the video data of the first target frame number as the first reference clock, and performs a first synchronization process on the audio data and the video data according to the first reference clock.

[0130] In some embodiments, the audio data and video data have obtained the data reference timestamp before decoding. In order to reduce the pressure of audio and video synchronization after decoding, a coarse synchronization process, i.e., the first synchronization process, can be performed before decoding the audio data and video data. After decoding is completed, in order to further achieve the effect of audio and video synchronization, fine synchronization after decoding can be performed again. That is to say, in this application, a two-level synchronization method can be used for audio data and video data, i.e., coarse synchronization (the first synchronization before decoding) and fine synchronization (the second synchronization after decoding).

[0131] Therefore, after receiving the media data, the display device 200 can first set the video data of the first target frame number (such as the video data of the nth frame) as the first reference clock before entering the decoding stage. For example, the audio and video data receiving module can perform the first synchronization processing on the audio data and the video data according to the first reference clock. For example, the first reference clock is used to perform preliminary timestamp alignment on the audio data and the video data, that is, to adjust the playback rate or delay of the two so that they are consistent on the time axis. In this way, the initial synchronization of the audio data and the video data before decoding is ensured, the playback jamming or desynchronization caused by data asynchrony is reduced, and the data basis for subsequent decoding and playback is provided.

[0132] In some embodiments, when the display device 200 performs the first synchronization processing on the audio data and the video data according to the first reference clock, it can first compare the first time difference between the video data and the audio data of the first target frame number. If the first time difference does not exceed the first threshold range, the video data and the audio data of the first target frame number are decoded; if the first time difference exceeds the first threshold range, the audio data or video data of the next frame of the first target frame number is adjusted according to the first time difference.

[0133] Exemplarily, taking the time corresponding to the nth frame of video data as the first reference clock as an example, the time difference between the nth frame of video data and the nth frame of audio data can be compared. If the time difference is within the first threshold range, it means that the video data and the audio data are synchronized, and the audio data and the video data can be transmitted to the decoder for decoding. If the time difference is not within the first threshold range, it means that there is a problem of asynchrony between the video data and the audio data, and it is necessary to dynamically adjust the frame time of the next frame of audio data or video data according to the time difference to correct the phenomenon of audio and video asynchrony, and ensure that the rough synchronization of the audio data and the video data is completed before decoding through the first reference clock.

[0134] In some embodiments, when the display device 200 adjusts the audio data or video data of the next frame of the first target frame number according to the first time difference, it can first determine the delay time according to the first time difference, and obtain the actual frame sending time and the frame sending time threshold of the video data of the first target frame number; if the actual frame sending time is earlier than the frame sending time threshold, the video data of the next frame of the first target frame number is delayed according to the delay time; if the actual frame sending time is later than the frame sending time threshold, the audio data of the next frame of the first target frame number is delayed according to the delay time.

[0135] For example, Fig.15 A flow chart of a display device performing pre-decoding synchronization on audio and video data provided in some embodiments of the present application, such as Fig.15 As shown, after the data message is queued, it can enter the data buffer queue. The video buffer queue can be divided into a video data buffer queue and an audio data buffer queue. The nth frame of video data is taken out from the video data buffer queue, and the nth frame of audio data is taken out from the audio data buffer queue. After that, the time corresponding to the nth frame of video data can be set as the first reference clock, and the time difference between the nth frame of video data and the nth frame of audio data is compared to see whether it exceeds the first threshold range. If the first time difference exceeds the first threshold range, it is necessary to adjust the audio data or video data of the next frame of the first target frame number according to the first time difference.

[0136] When the first time difference exceeds the first threshold range, it is necessary to determine whether the n-th frame of video data is sent too quickly. In some embodiments, if the first time difference value is large, it means that the video data may be sent too quickly. To determine whether the frame is sent too quickly, a frame sending time threshold can be pre-set. If the actual frame sending time of the n-th frame of video data is earlier than the frame sending time threshold, it means that the video data playback speed is fast, that is, the n-th frame of video data is sent too quickly. At this time, combined with Fig.15 , the display device 200 can delay taking out the video data of the next frame, i.e., the n+1th frame, according to the delay time to slow down the video playback speed so that it keeps synchronization with the audio. On the contrary, if the actual frame sending time of the nth frame video data is later than the frame sending time threshold, it means that the video playback speed is slow. At this time, the display device 200 can delay taking out the audio data of the next frame according to the delay time so that it keeps synchronization with the video data. In this way, the display device 200 solves the problem of audio data and video data being out of sync by dynamically adjusting the playback speed of the audio data and the video data. And when the first time difference does not exceed the first threshold range, the nth frame video data and audio data can be input into the decoder to execute the decoding process. After step S2 is executed, the following step S3 can be executed.

[0137] Step S3: the display device 200 sets the system startup time as the second reference clock, and decodes the audio data and video data after the first synchronization processing according to the second reference clock to obtain decoded audio data and video data.

[0138] After completing the first synchronization of audio data and video data, the display device 200 can set the system startup time (such as the boot time or the networking time, etc.) as the second reference clock. Fig.14 The audio and video data decoding module can set a second reference clock, and then decode the audio data and video data after the first synchronization based on the clock to obtain decoded audio data and video data, and send the decoded audio data and video data to the audio and video data synchronization module through the audio and video data decoding module.

[0139] Exemplarily, the display device 200 can record and save the power-on time or networking time as the second reference clock, and use the second reference clock to decode the audio data and video data according to the preset decoding algorithm and parameters to obtain the decoded audio data and video data. In this way, by introducing the system startup time as the second reference clock, the time consistency of the decoding process can be ensured, which is convenient for subsequent time synchronization. After step S3 is executed, the following step S4 can be executed.

[0140] Step S4: the display device 200 sets the time corresponding to the decoded video data of the second target frame number as the third reference clock, and performs a second synchronization process on the decoded audio data and video data according to the second reference clock and the third reference clock.

[0141] In some embodiments, the display device 200 can select the time corresponding to the video data of the second target frame number in the decoded video data as the third reference clock. For example, the second synchronization process can be performed by the audio and video synchronization module. The audio and video synchronization module can select the time corresponding to the decoded video data of the nth frame (which can be the same as or different from the n of the first rough synchronization) as the third reference clock, and then combine the second reference clock and the third reference clock to make fine timestamp adjustments to the decoded audio data and video data to ensure that the final playback effect reaches the best synchronization state. In this way, through the second synchronization process, the synchronization accuracy of the audio data and the video data can be further improved, the smoothness and continuity of the playback effect can be ensured, and the problem of audio and video asynchronism can be solved. At the same time, since the application is deployed on a platform based on the Driver Base architecture, only the application docking module needs to be docked during the application docking process, thereby solving the problem of low application docking efficiency.

[0142] In some embodiments, before the display device 200 performs the second synchronization processing on the decoded audio data and video data according to the second reference clock and the third reference clock, it can compare the frame delivery time corresponding to the video data of the second target frame number with the second time difference of the second reference clock. When the second time difference exceeds the second threshold range, the video data of the second target frame number is discarded. Among them, the second threshold range can be used to determine whether the video data arrives within an acceptable time range. If the difference between the arrival time of the video data and the expected time exceeds this threshold range, it means that there is a large delay or advance in the video data, which may cause the video playback to be unsmooth or jamming. Therefore, when such a situation exceeding the threshold is detected, the display device 200 can choose to discard this part of the video data to ensure the synchronization and playback quality of subsequent video data.

[0143] In some embodiments, the display device 200 can perform a second synchronization processing on the decoded audio data and video data according to the following process: when the second time difference does not exceed the second threshold range, the decoded video data of the second target frame number is rendered; thereafter, the third time difference between the decoded audio data and the third reference clock can be compared; if the third time difference does not exceed the third threshold range, the audio data is rendered; if the third time difference exceeds the third threshold range, the audio data of the second target frame number is discarded.

[0144] Exemplary, the second synchronization can be the fine synchronization after decoding, and the next step after the audio data and the video data are decoded is the final rendering stage, and at this time, the display device 200 can perform fine synchronization processing on the audio data and the video data. For example, first, the display device 200 can set the system startup time (such as the power-on startup time or the networking time, etc.) as the reference clock, decode the video data and the audio data, and then compare the time difference between the decoded n-th frame video data and the system reference clock. If the above-mentioned time difference is within the second threshold range, the n-th frame video data can be rendered by the audio and video data rendering module. Afterwards, the video data after the n-th frame decoding can be set as the reference clock, and the time difference between the n-th frame video data and the n-th frame audio data is compared. If the time difference between the two is within the third threshold range, the n-th frame audio data is rendered, so that it is played synchronously with the video data, and the synchronization of the video data and the audio data when playing is ensured.

[0145] It should be noted that the specific values ​​of the reference clock, the first threshold range, the second threshold range, the third threshold range, etc. in the present application can be set according to actual needs. For example, the first threshold range, the second threshold range, and the third threshold range can be completely the same, or two of them can be the same or different. The present application does not limit the specific values. After step S4 is executed, the following step S5 can be executed.

[0146] Step S5: the display device 200 plays the audio data and video data after the second synchronization process.

[0147] After the audio data and the video data are processed synchronously twice, the display device 200 can play the audio data and the video data processed synchronously twice. For example, the audio data and the video data playing module can play the audio data and the video data processed synchronously twice.

[0148] The following takes platform upgrade as an example (such as changing the platform version) to further illustrate the secondary synchronization process of audio data and video data.

[0149] Combination Fig.15 During the first rough synchronization, the display device 200 can obtain the decoding capabilities supported by the new platform, which include but are not limited to audio decoding capabilities and video decoding capabilities, and receive data messages from the application from the data buffer queue. After that, the nth frame of video data can be taken out, and the time of the nth frame of video data can be set as the reference clock, and then the nth frame of audio and video can be taken out. After the audio data and video data are taken out, the display device can determine whether the time difference between the nth frame of video data and the nth frame of audio data exceeds the threshold. If the time difference does not exceed the threshold, it means that the nth frame of audio data meets the requirements of rough synchronization, and the nth frame of audio data and video data can be sent to the decoder for decoding. If the time difference exceeds the threshold, it can be determined whether the nth frame of video data is sent too fast. If the nth frame of video data is sent too fast, the n+1th frame of video data frame is delayed. If the nth frame of audio data is sent too fast, the n+1th frame of audio data frame is delayed. Each frame can be processed according to the above steps, and the rough synchronization is completed.

[0150] Fig.16 A flow chart of a display device performing post-decoding synchronization on audio and video data provided in some embodiments of the present application, such as Fig.16As shown, during the second fine synchronization, the display device 200 can set the system time as the reference clock, and decode the nth frame of video data and the nth frame of audio data. For example, the video decoder can decode the video data, and the audio decoder can decode the audio data. There is a corresponding decoding delay when decoding the video data and the audio data. After the decoding of the nth frame of video data and the nth frame of audio data is completed, it can be determined whether the time difference between the nth frame of video data and the system reference clock exceeds the threshold (such as the second threshold range). If the time difference between the nth frame of video data and the system reference clock exceeds the threshold, it means that the nth frame of video data is later than the current reference clock, and the current video data frame is discarded. If the time difference between the nth frame of video data and the system reference clock does not exceed the threshold, it means that the decoded video data is synchronized with the current reference clock, and the nth frame of video data is rendered. Furthermore, the time corresponding to the decoded video data of the nth frame can be set as a reference clock (such as a third reference clock), and it is determined whether the nth audio data frame and the synchronization reference clock (such as the third reference clock) exceed a threshold (such as a third threshold range). If the nth audio data frame and the reference clock exceed the threshold, it means that the nth frame of audio data is later than the current reference clock, and the current audio data frame is discarded. If the time difference between the nth audio data frame and the reference clock does not exceed the threshold, it means that the decoded audio data is synchronized with the current reference clock, and the nth audio data frame is rendered.

[0151] It can be seen from the above technical scheme that the above embodiment provides a display device 200, the display device 200 receives the media data sent by the target application; the media data includes audio data and video data; before decoding the audio data and video data, the time corresponding to the video data of the first target frame number is set as the first reference clock, and the audio data and video data are synchronized for the first time according to the first reference clock; the system startup time is set as the second reference clock, and the audio data and video data after the first synchronization processing are decoded according to the second reference clock to obtain the decoded audio data and video data; the system startup time at least includes the boot time or the networking time; the time corresponding to the decoded second target frame number of video data is set as the third reference clock, and the decoded audio data and video data are synchronized for the second time according to the second reference clock and the third reference clock. When the display device 200 is docked with the application based on the Driver Base architecture platform, it only needs to dock the application with the application docking module to solve the problem of low application docking efficiency. At the same time, through the first synchronization processing and the second synchronization processing of the audio data and the video data, the synchronization accuracy of the audio data and the video data can be improved, and the problem of audio and video asynchronism can be solved.

[0152] Based on the above display device 200, some embodiments of the present application further provide a method for audio and video synchronization, which can be applied to the display device 200 in the above embodiment. In some embodiments, the method may include the following content:

[0153] Receiving media data sent by a target application; the media data includes audio data and video data;

[0154] Before decoding the audio data and the video data, setting the time corresponding to the video data of a first target frame number as a first reference clock, and performing a first synchronization process on the audio data and the video data according to the first reference clock;

[0155] Setting the system startup time as the second reference clock, and decoding the audio data and the video data after the first synchronization processing according to the second reference clock to obtain decoded audio data and video data; the system startup time at least includes the boot time or the networking time;

[0156] The time corresponding to the decoded video data of the second target frame number is set as the third reference clock, and the decoded audio data and video data are subjected to a second synchronization process according to the second reference clock and the third reference clock.

[0157] From the above technical solutions, it can be seen that the above method only requires the application to dock with the application docking module when docking the application based on the Driver Base architecture platform, solving the problem of low application docking efficiency. At the same time, through the first synchronization processing and the second synchronization processing of audio data and video data, the synchronization accuracy of audio data and video data can be improved, solving the problem of audio and video asynchrony.

[0158] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0159] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0161] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that: include: a display configured to display a user interface; The controller is configured as: Receiving media data sent by a target application; the media data includes audio data and video data; Before decoding the audio data and the video data, setting the time corresponding to the video data of a first target frame number as a first reference clock, and performing a first synchronization process on the audio data and the video data according to the first reference clock; Setting the system startup time as the second reference clock, and decoding the audio data and the video data after the first synchronization processing according to the second reference clock to obtain decoded audio data and video data; The system startup time at least includes the boot time or the networking time; Setting the time corresponding to the decoded video data of the second target frame number as the third reference clock, and performing a second synchronization process on the decoded audio data and video data according to the second reference clock and the third reference clock; Play the audio data and video data after the second synchronization process.

2. The display device according to claim 1, characterized in that The controller performs a first synchronization process on the audio data and the video data according to the first reference clock, and is specifically configured as follows: Comparing a first time difference between the video data and the audio data of the first target frame number; If the first time difference does not exceed a first threshold range, decoding the video data and the audio data of the first target frame number; If the first time difference exceeds the first threshold range, the audio data or the video data of the next frame of the first target frame number is adjusted according to the first time difference.

3. The display device according to claim 2, characterized in that The controller adjusts the audio data or the video data of the next frame of the first target frame number according to the first time difference, and is specifically configured as follows: Determining a delay time according to the first time difference; Obtaining an actual frame sending time and a frame sending time threshold for the video data of the first target frame number; If the actual frame sending time is earlier than the frame sending time threshold, delaying the fetching of the video data of the next frame of the first target frame number according to the delay time; If the actual frame sending time is later than the frame sending time threshold, the audio data of the next frame of the first target frame number is retrieved with delay according to the delay time.

4. The display device according to claim 1, characterized in that Before the controller performs a second synchronization process on the decoded audio data and video data according to the second reference clock and the third reference clock, the controller is further configured to: Compare the frame delivery time corresponding to the video data of the second target frame number with a second time difference of the second reference clock; When the second time difference exceeds a second threshold range, the video data of the second target number of frames is discarded.

5. The display device according to claim 4, characterized in that The controller performs a second synchronization process on the decoded audio data and video data according to the second reference clock and the third reference clock, and is specifically configured as follows: When the second time difference does not exceed a second threshold range, rendering the decoded video data of the second target frame number; comparing a third time difference between the decoded audio data and the third reference clock; If the third time difference does not exceed a third threshold range, rendering the audio data; If the third time difference exceeds the third threshold range, the audio data of the second target number of frames is discarded.

6. The display device according to claim 1, characterized in that Before receiving the media asset data sent by the target application, the controller is further configured to: Acquire the message type included in the target application; the message type includes a control message and a data message; the control message is used to represent the playback status of the media asset data, and the data message includes at least the audio data and the video data; Obtain the current playback status of the media asset data; Adapting the control message according to the current playback state; When the control message is not suitable for the current playback state, discarding the control message; When the control message is adapted to the current playing state, the control message is controlled to enter a queue.

7. The display device according to claim 6, characterized in that The controller controls the control message to be enqueued, and is specifically configured as follows: Get the configured control message queue; Detecting an adaptation status of the control message in the control message queue; If the adaptation status is that the control message is not connected to the control message queue, generating an add instruction; and, based on the add instruction, adding the control message to the control message queue.

8. The display device according to claim 7, characterized in that After detecting the adaptation state of the control message in the control message queue, the controller is further configured to: If the adaptation state is that the control message has been added to the message queue, adapting the control message according to the current playing state; When the control message is not suitable for the current playing state, the control message is discarded.

9. The display device according to claim 6, characterized in that After obtaining the message type included in the target application, the controller is further configured to: Obtaining the data format, data encoding and data length of the data message; verifying the validity of the data message based on the data format, the data encoding and the data length; When the data message is valid, controlling the data message to enter a queue; When the data message is invalid, the data message is discarded.

10. A method for audio and video synchronization, applied to the display device according to any one of claims 1 to 9, characterized in that: The method comprises: Receiving media data sent by a target application; the media data includes audio data and video data; Before decoding the audio data and the video data, setting the time corresponding to the video data of a first target frame number as a first reference clock, and performing a first synchronization process on the audio data and the video data according to the first reference clock; Setting the system startup time as the second reference clock, and decoding the audio data and the video data after the first synchronization processing according to the second reference clock to obtain decoded audio data and video data; the system startup time at least includes the boot time or the networking time; Setting the time corresponding to the decoded video data of the second target frame number as the third reference clock, and performing a second synchronization process on the decoded audio data and video data according to the second reference clock and the third reference clock; Play the audio data and video data after the second synchronization process.

Images