Display device and audio-video synchronization method

Abstract

The application shows a display device and a sound-picture synchronization method. The display device comprises a display, an external device interface for connecting an external audio output device, a built-in audio output module, and a controller for obtaining sound-picture data of a media asset, the sound-picture data comprising image data and audio data, determining an output delay of the image data and an output delay of the audio data, the output delay of the image data being consistent with the output delay of the audio data, and generating a declaration containing the output delay of the image data and the output delay of the audio data to declare the output delay of the image data and the output delay of the audio data to the external audio output device or the built-in audio output module through the declaration. The application can complete the interaction between the display device and the external audio output device or the built-in audio output module of the display device through the declaration to realize the sound-picture synchronization between the display device and the external audio output device or the built-in audio output module of the display device.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device and an audio-visual synchronization method. Background Technology

[0002] The display device supports media asset playback. During playback, the device parses the audio and video data of the media asset into image and audio data. The image data is processed and displayed on the monitor, while the audio data is processed and played back through an audio output device. Furthermore, the display device supports media asset playback under various image settings, such as different image resolution settings and different image mode settings (e.g., standard mode, sports mode, game mode, etc.). The time required for the display device to parse the audio and video data varies depending on the image settings.

[0003] Because image data and audio data are processed using different data processing paths, and because the processing time for audio data is longer than that for image data, the display device is prone to audio-visual desynchronization when playing media, under the same or different image mode settings. Summary of the Invention

[0004] This application provides a display device and an audio-visual synchronization method, which can realize audio-visual synchronization between the display device and an external audio output device or the built-in audio output module of the display device by declaring the interaction between the display device and the external audio output device or the built-in audio output module of the display device.

[0005] In some embodiments of this application, a display device is provided, comprising: a display; an external device interface for connecting to an external audio output device; a built-in audio output module; and a controller configured to: acquire audio-visual data of media assets, the audio-visual data including image data and audio data; determine the output delay of the image data and the output delay of the audio data, wherein the output delay of the image data is consistent with the output delay of the audio data; and generate a declaration containing the output delay of the image data and the output delay of the audio data, so as to declare the output delay of the image data and the output delay of the audio data to the external audio output device or the built-in audio output module through the declaration. Using this implementation, the interaction between the display device and the external audio output device or the built-in audio output module of the display device can be completed through the declaration, thereby achieving audio-visual synchronization between the display device and the external audio output device or the built-in audio output module of the display device.

[0006] In some embodiments of this application, the controller, in determining the output delay of image data and audio data, is further configured to: acquire the current image setting parameters of the display device; and determine the output delay of image data and audio data based on the image setting parameters, with different output delays determined according to different image setting parameters. This implementation method enables the determination of the output delay of image data and audio data based on the current image setting parameters of the display device, thereby achieving audio-visual synchronization between the display device and an external audio output device or the display device's built-in audio output module.

[0007] In some embodiments of this application, the controller executes a declaration to generate output delays for image data and audio data, further configured to: generate a CEC message containing the output delays for image data and audio data; the controller, in executing the declaration to an external audio output device or a built-in audio output module, further configured to send a CEC message to the external audio output device. Using this implementation, the output delays for image data and audio data can be declared to the external audio output device in real time via CEC messages, enabling the external audio output device to obtain the output delays for image data and audio data in real time.

[0008] In some embodiments of this application, the controller is further configured to: when a change in the image setting parameters of the display device is detected, redetermine the output delay of the image data and the output delay of the audio data based on the new image setting parameters; generate a new CEC message based on the redetermined output delay of the image data and the output delay of the audio data; and send the new CEC message to the external audio output device. This implementation enables the audio and video data to declare the output delay of the image data and the output delay of the audio data to the external audio output device in real time under different image setting parameters, so that the external audio output device can obtain the output delay of the image data and the output delay of the audio data in real time.

[0009] In some embodiments of this application, it is declared as an extended display identification data declaration. Using this implementation, when the display device outputs audio data via a built-in audio output module or an external audio output device, audio-visual synchronization can be automatically achieved when image data and audio data are output at the display device.

[0010] In some embodiments of this application, the audio-visual data further includes the signal format of the image data, including P-mode and I-mode signal formats. The controller executes the determination of the output delay of the image data and the output delay of the audio data, and is further configured to: determine the output delay of the P-mode signal format image data and the output delay of the I-mode signal image data respectively; generate a declaration containing the output delay of the image data and the output delay of the audio data, including: generating a declaration containing the output delay of the P-mode signal format image data, the output delay of the I-mode signal format image data, and the output delay of the audio data. Using this implementation, the output delay of image data in different signal formats can be set separately, so that the display device supports audio-visual synchronization with an external audio output device or the built-in audio output module of the display device under multiple signal formats.

[0011] In some embodiments of this application, the controller, in determining the output delay of image data and the output delay of audio data, is further configured to: acquire the processing time of the audio data; subtract the processing time of the audio data from the output delay of the image data to obtain the delay compensation for the audio data; and determine the output delay of the audio data based on the processing time of the audio data and the delay compensation, so that the output delay of the image data is consistent with the output delay of the audio data. This implementation method enables the output delay of the audio data to be consistent with the output delay of the image data, thereby achieving audio-visual synchronization at the display device end.

[0012] In some embodiments of this application, the controller is further configured to send a declaration containing the output delay of image data and the output delay of audio data to the built-in audio output module, so that the built-in audio output module outputs audio data. Using this implementation, audio data can be output through the built-in audio output module, and audio-visual synchronization can be achieved on the display device.

[0013] In some embodiments of this application, the controller is further configured to: receive the external output delay of audio data sent by an external audio output device, wherein the external output delay is determined by the external audio output device based on the processing time of the audio data by the external audio output device; and determine the final output delay of the image data by summing the output delay of the image data and the external output delay of the audio data. Using this implementation, audio data can be output through an external audio output device, and audio-visual synchronization between the display device and the external audio output device can be achieved.

[0014] In some embodiments of this application, an audio-visual synchronization method is also provided. The method is applied to a display device, which includes an external device interface for connecting to an external audio output device and a built-in audio output module. The method includes: acquiring audio-visual data of media assets, the audio-visual data including image data and audio data; determining the output delay of the image data and the output delay of the audio data, wherein the output delay of the image data is consistent with the output delay of the audio data; generating a declaration containing the output delay of the image data and the output delay of the audio data, so as to declare the output delay of the image data and the output delay of the audio data to the external audio output device or the built-in audio output module through the declaration. Using this implementation, the interaction between the display device and the external audio output device or the built-in audio output module of the display device can be completed through the declaration, thereby achieving audio-visual synchronization between the display device and the external audio output device or the built-in audio output module of the display device.

[0015] Thus, in this application, the interaction between the display device and an external audio output device or the built-in audio output module of the display device can be completed by declaration, so as to achieve audio-visual synchronization between the display device and the external audio output device or the built-in audio output module of the display device. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram illustrating an operational scenario between a display device and a control device according to some embodiments is shown;

[0018] Figure 2 A configuration block diagram of a control device 100 according to some embodiments is shown;

[0019] Figure 3 A hardware configuration block diagram of a display device 200 according to some embodiments is shown;

[0020] Figure 4 A software configuration block diagram of a display device 200 according to some embodiments is shown;

[0021] Figure 5 A schematic diagram of an application icon control interface display of a display device 200 according to some embodiments is shown;

[0022] Figure 6 The diagram illustrates a media asset playback scenario according to some embodiments;

[0023] Figure 7A schematic diagram illustrating the media asset playback principle according to some embodiments is shown;

[0024] Figure 8 A schematic diagram of an image quality processing procedure according to some embodiments is shown;

[0025] Figure 9 A flowchart illustrating the configuration of a display device according to some embodiments is shown;

[0026] Figure 10 A flowchart illustrating the configuration of a display device according to some embodiments is shown;

[0027] Figure 11 A schematic diagram of an EDID declaration according to some embodiments is shown;

[0028] Figure 12 A flowchart illustrating the configuration of a display device according to some embodiments is shown;

[0029] Figure 13 A flowchart illustrating the configuration of a display device according to some embodiments is shown;

[0030] Figure 14 A schematic diagram illustrating image resolution setting parameters according to some embodiments is shown;

[0031] Figure 15 A schematic diagram illustrating image mode setting parameters according to some embodiments is shown. Detailed Implementation

[0032] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0033] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete implementation on its own. It should be noted that the brief descriptions of terminology in this application are merely for the convenience of understanding the embodiments described below, and are not intended to limit the implementation of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0034] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device according to one or more embodiments of this application, such as... Figure 1As shown, a user can operate the display device 200 via a mobile terminal 300 and a control device 100. The control device 100 can be a remote control, and communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, wireless or other wired methods to control the display device 200. The user can input user commands through buttons on the remote control, voice input, control panel input, etc., to control the display device 200. In some embodiments, a mobile terminal, tablet computer, computer, laptop computer, and other smart devices can also be used to control the display device 200.

[0035] In some embodiments, the mobile terminal 300 can install software applications with the display device 200 to achieve connection and communication via network communication protocols, enabling one-to-one control operations and data communication. Audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 for synchronous display. The display device 200 also communicates with the server 400 via various communication methods. The display device 200 can communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 can provide various content and interactive features to the display device 200. The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. In addition to providing broadcast television reception functions, the display device 200 can also be equipped with a smart network television function that provides computer support.

[0036] Figure 2 An exemplary block diagram of the configuration of the control device 100 according to an exemplary embodiment is shown. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input commands and convert them into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200. The communication interface 130 is used for external communication and includes at least one of a Wi-Fi chip, a Bluetooth module, NFC, or a replacement module. The user input / output interface 140 includes at least one of a microphone, a touchpad, a sensor, buttons, or a replacement module.

[0037] Figure 3 A hardware configuration block diagram of a display device 200 according to an exemplary embodiment is shown. For example... Figure 3The display device 200 shown includes at least one of the following: a tuner / demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to nth interface for input / output. The display 260 can be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and can also be a projection device and a projection screen. The tuner / demodulator 210 receives broadcast television signals via wired or wireless reception and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. The detector 230 is used to collect signals from the external environment or signals interacting with the external environment. The controller 250 and the tuner / demodulator 210 can be located in different separate devices; that is, the tuner / demodulator 210 can also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0038] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200. The user can input user commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, the user can input user commands by inputting specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0039] In some embodiments, a "user interface" is the medium through which an application or operating system interacts and exchanges information with a user, enabling the conversion between the internal form of information and a form acceptable to the user. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include at least one of the visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0040] Figure 4 This is a schematic diagram of the software configuration in a display device 200 according to one or more embodiments of this application, such as... Figure 4As shown, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the Android runtime and system library layer (referred to as the "System Runtime Layer"), and the kernel layer. The kernel layer contains at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, Wi-Fi driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0041] Figure 5 This is a schematic diagram showing the icon control interface of an application in a display device 200 according to one or more embodiments of this application, such as... Figure 5 As shown, the application layer contains at least one application whose corresponding icon control can be displayed on the screen, such as: live TV application icon control, video-on-demand application icon control, media center application icon control, application center icon control, game application icon control, etc. Live TV applications can provide live television from different signal sources. Video-on-demand applications can provide video from different storage sources. Unlike live TV applications, video-on-demand provides video display from certain storage sources. Media center applications can provide applications for playing various multimedia content. The application center can provide storage for various applications.

[0042] Figure 6 An exemplary illustration shows a media asset playback scenario according to an exemplary embodiment, such as... Figure 6 As shown, the media asset playback scenario may involve one or more devices, such as a control device 100, a display device 200, a smart device 300, a server 400, and an external audio output device 500. The display device 200 includes: a monitor 260; an external device interface 240 for connecting to the external audio output device 500; a built-in audio output module; and a controller. Users can input user commands through the control device 100 or the smart device 300 to control media asset playback. The audio-visual data of the media asset includes image data and audio data. The display device 200 plays image data through the monitor 260 and audio data through the built-in audio output module or the external audio output device 500.

[0043] Figure 7 An exemplary illustration shows a schematic diagram of the media asset playback principle according to an exemplary embodiment, such as... Figure 7As shown, when a display device plays media data, it parses the audio and video data of the media into image data and audio data, and processes the image data and audio data separately through different data processing paths. Specifically, the image data is output to the monitor after passing through the display device's image processing path, so that the monitor can play the image data. The audio data is output to the built-in audio output module or an external audio output device after passing through the display device's audio processing path, so that the built-in audio output module or external audio output device can play the audio data.

[0044] It should be noted that the image quality processing path is used to process image data. The processing steps of the image quality processing path may include: noise reduction, deinterlacing, scaling, and dynamic compensation. Noise reduction is used to reduce noise in the image data. Deinterlacing converts interlaced images into progressive scan images to eliminate or reduce the drawbacks of interlacing. Scaling makes the image fit the size of the display area, generating a thumbnail of the corresponding image. Dynamic compensation uses a dynamic imaging system, inserting a motion compensation frame between two traditional image frames, so that even fast-moving scenes can be clear and natural. In the embodiments shown in this application, the processing steps of the image quality processing path can be specifically implemented in the video processor of the display device.

[0045] It should be added that, in practical applications, the image quality processing path may include other processing processes in addition to the above-mentioned processes, such as brightness processing, contrast processing, chroma processing, hue processing, sharpness processing, dynamic contrast processing, gamma correction processing, color temperature processing, white balance processing, color correction processing, dynamic brightness processing, etc. This embodiment will not describe these in detail.

[0046] It should be noted that the audio processing path is used to process audio data, and it includes an audio processing module. The audio processing module can perform decoding, sound effects processing, and transmission processing. Sound effects processing mainly includes: Digital Theater System (DTS) sound effects processing, Dolby Atmos (ATMOS) sound effects processing, Graphic Equalizer (GEQ) processing, and Parametric Equalizer (PEQ) processing.

[0047] DTS and ATMOS audio processing both enhance the sound quality to improve playback. GEQ employs constant Q-factor technology, with a push-pull potentiometer at each frequency point. Regardless of boosting or attenuating a frequency, the filter's bandwidth remains constant. The distribution of push-pull buttons on the panel visually reflects the current audio data's equalization compensation and the boosting and attenuation of various frequencies. PEQ allows for fine-tuning of various equalization parameters and is often attached to a mixing console, but independent parametric equalizers are also available. Adjustable parameters include frequency bands, frequency points, gain, and Q-factor, enhancing and refining the sound to achieve a more distinct and artistic effect. In this embodiment, both the display device and the external audio output device include audio processors, and the specific processing steps of the audio processing pathway can be implemented within these audio processors.

[0048] It is worth noting that, compared to the audio processing of audio data, the image processing of image data takes longer. This means that the time required for image data to reach the display is longer than the time required for audio data to reach the built-in audio output module or external audio output device, which can easily lead to audio-visual desynchronization.

[0049] The following analysis explains the time-consuming nature of image quality processing. When processing a single frame of image data, the image quality processing path needs to read not only the current frame's image data but also the image data from several subsequent frames. Then, based on this combined data, the current frame is processed, and the processed data is written back to the current frame. Because processing the current frame requires reading image data from several subsequent frames, this process is time-consuming.

[0050] Figure 8 An exemplary illustration shows a schematic diagram of the image data quality processing procedure according to an exemplary embodiment, such as... Figure 8 As shown, in some embodiments, the threshold for the number of image data frames read is 4. This means that each time the display device reads a frame of image data, it needs to read the current frame and the next 3 frames. For example, if the current frame to be processed is frame n, then the image data of frames n, n+1, n+2, and n+3 need to be read. After processing these 4 frames, the processing result is written into frame n. The processing of frames n+1, n+2, and subsequent frames is similar and will not be elaborated here. Assuming a refresh rate of 60Hz, when the threshold for the number of frames read is 4, 4 frames of image data need to be read, which is equivalent to the image quality processing time for each frame being 1 / 60 * 4 = 66ms.

[0051] To address the audio-visual asynchrony issue caused by the time-consuming image data processing, the audio processing path includes an audio delay processing module. After the audio processing module processes the audio data, it undergoes further audio delay processing. This module buffers the audio data, caching it for a certain duration before outputting it to the built-in audio output module or external audio output device. This ensures that the audio data arrives at the built-in or external audio output device at the same time as the image data arrives at the display, thus eliminating audio-visual asynchrony.

[0052] For ease of description, in this application, the processing time of the image data by the image quality processing path is defined as the output delay of the image data. Figure 8 As described above, the output latency of image data is related to the threshold number of frames read by the video processor. Once the threshold number of frames read by the video processor is determined, the output latency of the image data corresponding to the video processor can be determined. The processing time of the audio data by the audio processing module in the audio processing path is defined as the audio data processing time, and the buffering time of the audio data by the audio delay processing module is defined as the audio data latency compensation. The sum of the audio data processing time and the audio data latency compensation is defined as the audio data output latency. It is worth noting that both the audio processing module and the audio delay processing module are internal modules of the display device, and the audio data latency compensation is the latency compensation performed by the audio processor of the display device.

[0053] It should be added that when the audio data of the display device is output from an external audio output device, the external audio output device cannot directly play the audio data of the display device. Instead, it needs to process the audio data before playback. The audio data is processed by the audio processor of the external audio output device. Specifically, the processing time of the audio data by the audio processing module in the audio processing path of the external audio output device is defined as the external processing time of the audio data, and the delay compensation of the audio data by the audio delay processing module in the audio processing path of the external audio output device is defined as the external delay compensation of the audio data. The sum of the external processing time and the external delay compensation is defined as the external output delay of the audio data. It is worth noting that in this embodiment, the audio delay processing module in the audio processing path of the external audio output device does not perform delay compensation for the audio data. Therefore, in this embodiment, the external output delay of the audio data is essentially the processing time of the audio data by the external audio output device. The sum of the output delay of the audio data and the external output delay is defined as the final output delay of the audio data.

[0054] The following describes the scenarios to which the embodiments of this application are applicable.

[0055] Scenario 1: The display device plays audio data through its built-in audio output module.

[0056] When the display device receives the audio and video data from the media assets, it parses the data into image data and audio data, and processes them separately through image processing and sound effect processing pathways. If the display device outputs through a built-in audio output module, the output delay of the audio data is determined based on the output delay of the image data to ensure consistency. The audio data is then output from the built-in audio output module for the duration of this output delay. The built-in audio output module may include speaker components. These components may include power amplifiers (AMPs) and speakers. Typically, speaker components can output at least two channels of sound; to achieve a panoramic surround sound effect, multiple speaker components are required to output multiple channels of sound, which will not be elaborated further here.

[0057] Scenario 2: The display device plays audio data through an external audio output device.

[0058] When a display device receives audio and video data from media resources, it parses this data into image data and audio data, processing them separately through image processing and audio processing pathways. If the display device outputs audio data via an external audio output device, the output delay of the audio data is determined based on the output delay of the image data, ensuring consistency between the image and audio data output delays. The audio data is then output from the external audio output device with the final output delay. The display device can connect to an external audio output device via an audio output interface, allowing the external audio output device to play audio data. External audio output devices can include Bluetooth speakers, power amplifiers, etc.

[0059] It is worth noting that since the display device and the external audio output device belong to different hardware control systems, they are unaware of each other's audio processing time. Therefore, the display device cannot keep the output duration of the image data of the display device consistent with the output duration of the audio data of the external audio output device, and thus cannot achieve real-time audio-visual synchronization between the display device and the external audio output device.

[0060] It's worth noting that when a display device plays audio via an external audio output device, the devices involved in the media asset playback scenario can be divided into sending devices and receiving devices based on the direction of the audio data output. The sending device, also known as the source, refers to the party that sends the audio data in the media asset playback scenario, such as... Figure 6The display device shown; the receiving device, also known as the sink, refers to the party that receives audio data in a media playback scenario, such as... Figure 6 The external audio output device is shown. Therefore, in a media asset playback scenario, the transmitting device can transmit the audio data it is playing to the receiving device via a wireless or wired connection, enabling the receiving device to play the received audio data in real time. In a media asset playback scenario, the transmitting and receiving devices are logically determined based on the audio output direction; therefore, depending on the audio output direction, the display device can function as either a transmitting or receiving device.

[0061] For example, regarding a display device and an external audio output device, when the display device outputs audio data to the external audio output device, the audio data is played through the external audio output device, with the display device being the transmitting device and the external audio output device being the receiving device. Conversely, when the external audio output device outputs audio data to the display device, the audio data is played through the display device's audio processor, with the external audio output device being the transmitting device and the display device being the receiving device.

[0062] It is important to note that, such as Figure 6 In the media playback scenario shown, a receiving device can only establish a connection with one transmitting device at a time. For example, when a display device establishes a connection with an external audio output device, and the external audio output device outputs the audio data of the media being played by the display device, the smart device cannot simultaneously establish a connection with the external audio output device, and the external audio output device cannot simultaneously output the audio data of the media being played by the smart device; conversely, when a smart device establishes a connection with an external audio output device, and the external audio output device outputs the audio data of the media being played by the smart device, the external audio output device cannot simultaneously output the audio data of the media being played by the display device.

[0063] Therefore, this application provides a display device that avoids the above-mentioned phenomenon of audio-visual asynchrony caused by the long processing time of image data, which makes the time required for image data to reach the display longer than the time required for audio data to reach the built-in audio output module or external audio output device, and the problem that the display device and external audio output device cannot achieve real-time audio-visual synchronization.

[0064] In its specific implementation, this application provides a display device to avoid the audio-visual desynchronization problem that may occur in scenario 1 above. The display device shown in this application includes a monitor; an external device interface for connecting an external audio output device; a built-in audio output module; and a controller for executing... Figure 9 The following steps S901-S904 are shown:

[0065] Step S901: Obtain the audio-visual data of the media assets, which includes image data and audio data;

[0066] In some embodiments, when a user inputs user commands to a display device via a control device or smart device, the display device can communicate with a server to obtain audio and video data of media assets. The display device can communicate with the server via various communication methods. In various embodiments of this application, the display device may be allowed to establish a wired or wireless communication connection with the server via a local area network, wireless local area network, or other networks. The server can provide the display device with various content and interactive features.

[0067] For example, the display device interacts by sending and receiving information, as well as with the Electronic Program Guide (EPG), receiving software updates, or accessing a remotely stored digital media library. Servers can be a group or multiple groups, and can be one or more types of servers. Servers can provide the display device with other network services such as video-on-demand and advertising services.

[0068] Display devices include, but are not limited to, acquiring audio and video data of media assets through data communication with a server. Display devices can also acquire audio and video data of media assets from smart devices through wired or wireless connections, or acquire audio and video data of media assets from local storage locations.

[0069] For example, smart devices can connect to display devices via screen mirroring, allowing them to project games (such as traditional arcade games) onto the display device to enhance the user's gaming experience.

[0070] For example, the display device may have local games stored on it, such as motion-sensing games (e.g., ball games, boxing games, running games, dancing games, etc.). The control device or smart device can open and run the local games from the local storage location by inputting user commands.

[0071] In some embodiments, the audio-visual data further includes the signal format of the image data, including P-mode and I-mode signal formats. Specifically, when the display device receives the audio-visual data of the media assets, the video processor is configured to automatically acquire the signal format of the image data.

[0072] It should be noted that both P-mode and I-mode signal formats are valid display formats for audio and video data. P-mode is a progressive scan signal format. Taking 1080p image data as an example, the effective display format is 1920×1080. 1080 indicates that there are 1080 scan lines in the vertical direction of the image data. Each horizontal scan line is simultaneously displayed on the screen, resulting in a smooth image display, suitable for playing motion graphics and movies. I-mode is an interlaced scan signal format. Taking 1080i image data as an example, the effective display format is 1920×1080. Each odd-numbered line of image data is displayed after each even-numbered line, resulting in a less smooth image display, suitable for playing documentaries and wildlife-themed media.

[0073] It should be added that P-type signal formats include, but are not limited to, 1080p, as well as other P-type signal formats such as 720p and 360p; I-type signal formats include, but are not limited to, 1080i, as well as other I-type signal formats such as 720i.

[0074] Step S902: Determine the output delay of image data and the output delay of audio data, and make the output delay of image data consistent with the output delay of audio data;

[0075] In some embodiments, the controller is used to perform Figure 10 The following steps S9021-S9023 are shown;

[0076] Step S9021: Obtain the processing time of the audio data.

[0077] The audio data processing time is the time taken by the audio processing module to process the audio data. The controller obtains the audio data processing time through the audio processor.

[0078] Step S9022: Subtract the processing time of the audio data from the output delay of the image data to obtain the delay compensation of the audio data.

[0079] The output latency of image data is the processing time of the image data in the image quality processing path. The controller obtains the processing time of the image data through the video processor. To ensure audio-visual synchronization, the output latency of image data must be consistent with the output latency of audio data. Therefore, the audio data latency compensation is obtained by subtracting the processing time of audio data from the output latency of image data.

[0080] Step S9023: Determine the output delay of the audio data based on the processing time of the audio data and the delay compensation of the audio data, so that the output delay of the image data is consistent with the output delay of the audio data.

[0081] In specific implementation, taking an image data output delay of 80ms and an audio data processing time of 30ms as an example, the controller obtains the audio data processing time of 30ms through the audio processor; by subtracting the audio data processing time of 30ms from the image data output delay of 80ms, the audio data delay compensation is obtained as 50ms; based on the audio data processing time of 30ms and the audio data delay compensation of 50ms, the output delay of the audio data is determined to be the sum of the audio data processing time of 30ms and the audio data delay compensation of 50ms, which is 80ms.

[0082] Step S903: Generate a declaration containing the output delay of image data and the output delay of audio data, so as to declare the output delay of image data and the output delay of audio data to the built-in audio output module through the declaration.

[0083] In some embodiments, the Extended Display Identification Data (EDID) is a standard for display identification data. The EDID is stored in the display's Direct Digital Control (DDC) memory. When a computer is connected to the display, the computer reads the stored EDID from the display's DDC memory via the DDC channel. The EDID includes: basic display parameters (such as the types of video display signals the display can receive, i.e., video signal formats); display size; and maximum horizontal and vertical image sizes. Because the EDID can identify the video signal format, delay settings within the EDID can separately determine the output delay of P-mode image data and I-mode image data.

[0084] Figure 11 An exemplary diagram illustrating the EDID declaration in an embodiment of this application is shown. Figure 11 Taking the delay settings shown as an example, in the P-mode signal format declaration 1101, the image output delay and audio output delay of the image data are both set to 80ms, and in the I-mode signal format declaration 1102, the image data output delay and audio data output delay are both set to 100ms, generating the following... Figure 11 The EDID declaration shown includes the output delay of image data in P-mode signal format, the output delay of image data in I-mode signal format, and the output delay of audio data.

[0085] Step S904: Send the declaration containing the output delay of image data and the output delay of audio data to the built-in audio output module so that the built-in audio output module outputs audio data.

[0086] Using this implementation method, the interaction between the display device and the built-in audio output module can be completed through EDID declaration. The EDID declaration ensures that the output delay of image data and audio data on the display device are consistent, so that the display device can achieve audio-visual synchronization when playing audio through the built-in audio output module.

[0087] In a specific implementation, this application also provides a display device to avoid the audio-visual desynchronization problem that may occur in scenario 2 above. The display device shown in this application includes a monitor; an external device interface for connecting an external audio output device; a built-in audio output module; and a controller for executing... Figure 12 The following steps S1201-S1204 are shown:

[0088] Step S1201: Obtain the audio-visual data of the media asset, which includes image data and audio data;

[0089] In practice, the specific method for obtaining the audio-visual data of media assets is the same as that in step S901 above, and will not be described in detail here.

[0090] Step S1202: Determine the output delay of image data and the output delay of audio data, and make the output delay of image data consistent with the output delay of audio data;

[0091] In the specific implementation, the method for determining the output delay of image data and the output delay of audio data is the same as in step S902 above, and will not be described in detail here.

[0092] Step S1203: Generate a declaration containing the output delay of image data and the output delay of audio data, so as to declare the output delay of image data and the output delay of audio data to an external audio output device through the declaration;

[0093] In some embodiments, the declaration is called an EDID declaration. The EDID declaration declares to the external audio output device that the image output delay and audio output delay of the display device are consistent, and that audio-visual synchronization has been achieved on the display device side.

[0094] Step S1204: Receive the external output delay of the audio data sent by the external audio output device. The external output delay is determined by the external audio output device based on the processing time of the audio data. The sum of the output delay of the image data and the external output delay of the audio data is determined as the final output delay of the image data.

[0095] In some embodiments, taking an image data output delay of 80ms and an audio data processing time of 30ms as an example, the controller obtains the audio data processing time of 30ms through the audio processor; by subtracting the audio data processing time of 30ms from the image data output delay of 80ms, the audio data delay compensation is obtained as 50ms; based on the audio data processing time of 30ms and the audio data delay compensation of 50ms, the audio data output delay is determined to be the sum of the audio data processing time of 30ms and the audio data delay compensation of 50ms, which is 80ms. At this time, the display device achieves audio-visual synchronization. Taking an external output delay of 40ms as an example, when the external audio output device reads the EDID declaration, the external audio output device sends audio data to the display device with an external output delay of 40ms. When the display device receives the audio data sent by the external audio output device with an external output delay of 40ms, the sum of the output delay of the image data (80ms) and the external output delay of the audio data (40ms) is determined as the final output delay of the image data (120ms). At the same time, the final output delay of the audio data is the sum of the output delay of the audio data (80ms) and the external output delay of 40ms (40ms), which is 120ms. Therefore, the final output delay of the image data is consistent with the final output delay of the audio data, realizing audio-visual synchronization between the display device and the external audio output device.

[0096] In a specific implementation, this application also provides another display device to avoid the audio-visual desynchronization problem that may occur in scenario 2 above. The display device shown in this application includes a monitor; an external device interface for connecting an external audio output device; a built-in audio output module; and a controller for executing... Figure 13 The following steps S1301-S1304 are shown:

[0097] Step S1301: Obtain the audio-visual data of the media asset, which includes image data and audio data;

[0098] In practice, the specific method for obtaining the audio-visual data of media assets is the same as that in step S901 above, and will not be described in detail here.

[0099] Step S1302: Determine the output delay of image data and the output delay of audio data, and make the output delay of image data consistent with the output delay of audio data;

[0100] In some embodiments, the current image setting parameters of the display device are obtained; the output delay of image data and the output delay of audio data are determined in combination with the image setting parameters, and the output delay varies depending on the different image setting parameters. The image setting parameters may include: image resolution setting parameters, image mode setting parameters, and image resource setting parameters.

[0101] Figure 14An exemplary illustration shows a schematic diagram of setting image resolution parameters according to an exemplary embodiment, such as... Figure 14 As shown, the user's overall settings interface 1400 includes: a boot scene control 1401, a screen adjustment control 1402, a resolution adjustment control 1403, a multi-screen interaction control 1404, and a device name control 1405. Clicking the corresponding control allows the user to enter its settings interface. For example, when the user clicks the resolution adjustment control 1403, the resolution of the image data can be set in the resolution adjustment interface. For instance, the resolution adjustment interface includes various resolutions such as 4K (2160×4096), 2K (1152×2048), 1080p (1080×1920), 720p (720×1280), and DV (480×720).

[0102] Figure 15 An exemplary illustration shows a schematic diagram of image mode setting parameters according to an exemplary embodiment, such as... Figure 15 As shown, the user's image mode settings interface 1500 includes: a standard mode control 1501, a sports mode control 1502, a game mode control 1503, a cinema mode control 1504, a concert mode control 1505, a studio mode control 1506, and a custom mode control 1507. Among these, clicking the game mode control 1503 sets the display device to game mode.

[0103] It should be noted that the output latency of image data varies under different image mode settings. Taking game mode as an example, when the display device is in game mode, the display device is allowed to optimize the screen response speed.

[0104] In practice, the image data latency is caused by the following: Image data output from the HDMI interface to the display device via the video cable has a latency of 5-10ms; image data is transmitted from the display device's HDMI interface to the controller, and the controller converts it into a data format that the video processor can process, which takes another 5-10ms; the video processor then processes the image data, taking 20-100ms. Therefore, the total latency from image data transmission to display on the monitor is 30-120ms, significantly impacting the user experience. Since the longest latency is generated by the video processor, the image quality processing path in the video processor includes various video processing effects to improve image quality. When the display device is in game mode, it disables these various video processing effects to improve image quality, allowing the image data to be displayed on the monitor more quickly.

[0105] It is worth noting that game mode does not completely eliminate the output latency of image data, but it will significantly reduce the latency by 2-3 times. For example, when the image mode setting is set to standard mode, the output latency of image data is 80ms, while when it is changed to game mode, the output latency of image data is 25ms.

[0106] Since game mode can reduce the output latency of image data, game mode is essentially a low-latency mode. Low-latency modes in display devices include, but are not limited to, game mode, as well as other low-latency modes such as automatic low-latency mode. The low-latency modes of display devices can be manually switched by user input operation commands. In automatic low-latency mode, the switching can also be automatic based on the media content played by the display device.

[0107] When a user connects to a PC device, the PC device sends the audio and video data of the media assets to the display device. When the audio and video data of the media assets is an image resource package, the image resource setting parameter is: image resource format (image / jpeg).

[0108] It should be noted that the image resource setting parameter is the content type of the media asset's audio-visual data. The image resource setting parameter is used to determine which content type to read the audio-visual data of the media asset. When the audio-visual data of the media asset is a game resource package, the image resource setting parameter is: game resource format (application / json).

[0109] In some embodiments, when a change in the image setting parameters of the display device is detected, the output delay of the image data and the output delay of the audio data are re-determined in conjunction with the new image setting parameters;

[0110] In practice, when any image setting parameter of the display device is detected to have changed, the output delay of the image data and the output delay of the audio data are re-determined based on the new image setting parameter.

[0111] Changes in the image settings parameters of the display device include the following scenarios:

[0112] Scenario 3: The image resolution setting of the display device is switched from 2k to 4k.

[0113] When the image resolution setting of the display device is switched from 2K to 4K, the processing time of the image data changes. Therefore, it is necessary to redetermine the output delay of the image data and the output delay of the audio data. In practice, the HDMI video monitor will continuously monitor the HDMI signal. When the image resolution setting is switched from 2K to 4K, the HDMI signal fluctuates. When the HDMI signal stabilizes, the output delay of the image data and the output delay of the audio data are redetermined.

[0114] Scenario 4: The display device's image mode settings are switched from standard mode to game mode.

[0115] When the display device switches its image mode settings from standard mode to game mode, the processing time of image data changes. Therefore, it is necessary to redetermine the output delay of image data and the output delay of audio data. In practice, the user enters a user command in the image mode settings interface through a control device or smart device to change the image mode settings parameters. At this time, since the image mode settings parameters have changed, it is necessary to redetermine the output delay of image data and the output delay of audio data.

[0116] Scenario 5: The image resource settings of the display device are switched from image resource format to game resource format.

[0117] When the display device is connected to the PC, the PC sends audio and video data of the media assets to the display device. When the audio and video data of the media assets is an image resource package, the image resource setting parameters are: image resource format; when the audio and video data of the media assets is a game resource package, the image resource setting parameters are: game resource format; when the audio and video data of the media assets is switched from an image resource package to a game resource package, the image resource setting parameters are switched from image resource format to game resource format. At this time, because the image resource setting parameters have changed, it is necessary to redetermine the output delay of the image data and the output delay of the audio data.

[0118] Therefore, in scenarios three through five above, taking a normal image data output latency of 80ms as an example, if a hard declaration (such as EDID declaration) is used, the image data output latency declared by the display device to the external audio output device will always be 80ms, regardless of whether the display device's image settings parameters change, thus only achieving automatic audio-visual synchronization. In reality, in low-latency mode, the image data output latency is less than 80ms. When any change in any image settings parameter of the display device is detected, the image data output latency will change, requiring a re-determination of the image data output latency. In this case, real-time determination of the image data output latency and declaration of this real-time determined image data output latency to the external audio output device enables real-time audio-visual synchronization between the display device and the external audio output device.

[0119] Step S1303: Generate a CEC message containing the output delay of image data and the output delay of audio data, so as to declare the output delay of image data and the output delay of audio data to an external audio output device through the CEC message;

[0120] The controller, in executing the declaration of the output delay of image data and the output delay of audio data to the external audio output device via CEC message, is further configured to send a CEC message to the external audio output device.

[0121] Consumer Electronics Control (CEC) is a complete single-wire protocol that allows display devices to control all HDMI-connected devices via CEC messages, enabling HDMI devices to command and control each other without user interference. HDMI includes the CEC bus and the Audio Return Channel (ARC). The CEC bus is a general-purpose control bus used for interconnecting HDMI devices. The ARC channel is used for digital audio output from the TV and can connect to external devices that also support ARC, transmitting the display device's audio to external audio output devices.

[0122] In some embodiments, when a change in the image setting parameters of the display device is detected, the output delay of the image data and the output delay of the audio data are re-determined in conjunction with the new image setting parameters;

[0123] A new CEC message is generated based on the redefined output delay of the image data and the output delay of the audio data;

[0124] Send the new CEC message to the external audio output device.

[0125] In scenarios three through five above, when any image setting parameter of the display device is detected to have changed, the display device sets a delay compensation scheme through a structure. This structure, contained within the CEC message, is used to declare the output delay of image data and the output delay of audio data to the external audio output device.

[0126] In some embodiments, taking the normal case where the output latency of image data and audio data are both declared as 80ms, if the current image mode setting parameter is low latency mode, when the image resolution setting parameter is switched from 2k to 4k, the output latency of image data and audio data is re-determined. Since the current image mode setting parameter is low latency mode, the actual output latency of image data is less than 80ms. Combining this with the current image setting parameter, the output latency of image data and audio data can be determined to be 60ms. Simultaneously, a new CEC message is sent to the external audio output device in real time, declaring to the external audio output device that the output latency of image data and audio data is 60ms.

[0127] In some embodiments, taking the normal case where the output latency of image data and audio data are both declared as 80ms, if the current image mode setting parameter is low latency mode, when the image mode setting parameter is switched from standard mode to game mode, the output latency of image data and audio data is re-determined. Since game mode is essentially low latency mode, the actual output latency of image data is less than 80ms. Combining this with the current image setting parameter, the output latency of image data and audio data can be determined to be 60ms. At the same time, a new CEC message is sent to the external audio output device in real time, declaring to the external audio output device that the output latency of image data and audio data is 60ms.

[0128] In some embodiments, taking the normal case where the output latency of image data and audio data are both declared as 80ms, if the current image mode setting parameter is low latency mode, when the image resource setting parameter is switched from image resource format to game resource format, the output latency of image data and audio data is re-determined. Since game mode is essentially low latency mode, the actual output latency of image data is less than 80ms. Combining this with the current image setting parameter, the output latency of image data and audio data can be determined to be 60ms. Simultaneously, a new CEC message is sent to the external audio output device in real time, declaring to the external audio output device that the output latency of image data and audio data is 60ms.

[0129] Therefore, taking the above embodiment as an example, when any image setting parameter of the display device is detected to change, the output delay of the image data changes. After the display device achieves real-time audio-visual synchronization, it declares the output delay of the image data and the output delay of the audio data to the external audio output device in real time.

[0130] Step S1304: Receive the external output delay of the audio data sent by the external audio output device. The external output delay is determined by the external audio output device based on the processing time of the audio data by the external audio output device.

[0131] The final output delay of the image data is determined by summing the output delay of the image data with the external output delay of the audio data.

[0132] In practice, step S1304 is implemented in the same way as step 804 above.

[0133] In a specific implementation, this application embodiment also illustrates an audio-visual synchronization method, including:

[0134] Acquire the audio-visual data of media assets, wherein the audio-visual data includes image data and audio data;

[0135] The output delay of the image data and the output delay of the audio data are determined, and the output delay of the image data is consistent with the output delay of the audio data;

[0136] Generate a declaration containing the output delay of the image data and the output delay of the audio data, so as to declare the output delay of the image data and the output delay of the audio data to the external audio output device or the built-in audio output module through the declaration.

[0137] It should be understood that the specific implementation methods of each step in the above-described audio-visual synchronization method can be found in the aforementioned display device embodiments, and will not be repeated here. As can be seen from the above embodiments, the display device and audio-visual synchronization method disclosed in this application can avoid the phenomenon of audio-visual asynchrony caused by the long processing time of image data, which makes the time required for image data to reach the display longer than the time required for audio data to reach the built-in audio output module or external audio output device, and the problem that the display device and external audio output device cannot achieve real-time audio-visual synchronization. This improves the user experience.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0139] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A display device, characterized by comprising: The display device comprises: a display; an external device interface for connecting an external audio output device; a built-in audio output module; a controller configured to: obtain audio-visual data of a media asset, the audio-visual data comprising image data and audio data; obtain current image setting parameters of the display device; determine an output delay of the image data and an output delay of the audio data in combination with the image setting parameters, the output delay of the image data being consistent with the output delay of the audio data, and the output delays determined in combination with different image setting parameters being different; when a change in the image setting parameters of the display device is monitored, re-determine the output delay of the image data and the output delay of the audio data in combination with new image setting parameters; generate a declaration containing the output delay of the image data and the output delay of the audio data, to declare the output delay of the image data and the output delay of the audio data to the external audio output device or the built-in audio output module through the declaration.

2. The display device of claim 1, wherein, The controller performing the generation of the declaration containing the output delay of the image data and the output delay of the audio data is further configured to: generate a CEC message containing the output delay of the image data and the output delay of the audio data; the controller performing the declaration of the output delay of the image data and the output delay of the audio data to the external audio output device through the declaration is further configured to send the CEC message to the external audio output device.

3. The display device of claim 2, wherein, The controller is further configured to: generate a new CEC message according to the re-determined output delay of the image data and the output delay of the audio data; send the new CEC message to the external audio output device.

4. The display device of claim 1, wherein, The declaration is an extended display identification data declaration.

5. The display device of claim 4, wherein, The audio-visual data further comprises a signal format of the image data, the signal format comprising a P-signal format and an I-signal format; The controller performing the determination of the output delay of the image data and the output delay of the audio data is further configured to: determine the output delay of the image data in the P-signal format and the output delay of the image data in the I-signal format respectively; The controller performing the generation of the declaration containing the output delay of the image data and the output delay of the audio data is further configured to: generate a declaration containing the output delay of the image data in the P-signal format, the output delay of the image data in the I-signal format and the output delay of the audio data.

6. The display device according to any one of claims 1 to 5, wherein The controller performing the determination of the output delay of the image data and the output delay of the audio data is further configured to: obtain a processing time consumption of the audio data; obtain a delay compensation of the audio data by subtracting the processing time consumption of the audio data from the output delay of the image data; determine the output delay of the audio data according to the processing time consumption of the audio data and the delay compensation of the audio data, so that the output delay of the image data is consistent with the output delay of the audio data.

7. The display device of claim 6, wherein, The controller is further configured to: sending a declaration containing the output delay of the image data and the output delay of the audio data to the built-in audio output module, so that the built-in audio output module outputs the audio data.

8. The display device of claim 6, wherein, The controller is further configured to: receive an external output delay of the audio data sent by the external audio output device, the external output delay being determined by the external audio output device according to a processing time of the external audio output device on the audio data; determine a sum of the output delay of the image data and the external output delay of the audio data as a final output delay of the image data.

9. A method for synchronizing audio and video, the method comprising: The method is applied to a display device, and the display device includes an external device interface for connecting an external audio output device and a built-in audio output module. The method includes: obtaining audio-visual data of a media asset, the audio-visual data including image data and audio data; obtaining current image setting parameters of the display device; determining an output delay of the image data and an output delay of the audio data in combination with the image setting parameters, the output delay of the image data being consistent with the output delay of the audio data, and the output delays determined according to different image setting parameters being different; when a change in the image setting parameters of the display device is monitored, redetermining the output delay of the image data and the output delay of the audio data in combination with new image setting parameters; generating a declaration containing the output delay of the image data and the output delay of the audio data, so as to declare the output delay of the image data and the output delay of the audio data to the external audio output device or the built-in audio output module through the declaration.

Images