Video and audio synchronization method, and device

Abstract

Disclosed are a video and audio synchronization method, and a device, which relate to the technical field of multimedia. A first device may rationally adjust delay compensation (a video delay value and an audio delay value) for a second device on the basis of the delay condition of video and audio playing by the second device, video and audio synchronization is implemented on a sending side (the first device) of a video and audio, and adjustment is not performed on a receiving side (the second device) of the video and audio, thereby facilitating an improvement in the synchronization effect of the video and audio in the second device, and improving the experience of a user watching the video and listening to the audio.

Description

Video and audio synchronization method and device Technical Field

[0001] The present application relates to the field of multimedia technology, and in particular to a method and device for synchronizing video and audio. Background Art

[0002] With the development of big data, artificial intelligence (AI), and cloud computing technology, various types of multimedia data are constantly emerging. Taking the wired transmission of multimedia data (such as audio or video) between different devices as an example, different devices can use the display port (DP) bus or the high definition multimedia interface (HDMI) bus to transmit multimedia data. In the DP protocol and the HDMI protocol, multimedia data is transmitted between the source device and the host device through the bus, and the source device sets a delay declaration between the audio and video to keep the audio and video synchronized, that is, the sound and the picture are synchronized with each other (lip synchronization). However, the value of the delay declaration is a preset fixed value, and the synchronization effect of the audio and video is poor.

[0003] Summary of the Invention

[0004] The present application provides a method and device for video and audio synchronization, which can set delay compensation information according to the delay information of a host device, thereby solving the problem of poor video and audio synchronization.

[0005] This application adopts the following technical solution.

[0006] In a first aspect, the present application provides a method for video and audio synchronization. This method is executed by a first device or a chip in the first device. Here, the method is described using the first device as an example. The method includes: the first device obtaining delay information of a second device, and setting a first video delay value and a first audio delay value of the second device based on the delay information. The delay information includes: video delay information, audio delay information, video delay compensation capability, and audio delay compensation capability.

[0007] In this way, the first device can reasonably adjust the delay compensation (video delay value and audio delay value) of the second device according to the delay of the video and audio playback of the second device, and realize video and audio synchronization on the sending side of the video and audio (first device), rather than adjusting on the receiving side of the video and audio (second device). This is conducive to improving the synchronization effect of the video and audio in the second device, thereby enhancing the user experience of watching videos and listening to audio.

[0008] In combination with the audio and video synchronization method provided in the first aspect, as an optional implementation method, after the first device sets the first video delay value and the first audio delay value of the second device according to the delay information, the audio and video synchronization method provided in this application also includes: the first device receives the updated delay information reported by the second device, and sets the second video delay value and the second audio delay value of the second device according to the updated delay information. In this way, during the playback of the audio and video, the first device can also adjust the video delay value and the audio delay value of the second device according to the current audio and video playback situation, that is, the delay value between the video and audio can be flexibly adjusted, avoiding the problem of audio and video asynchrony caused by a large delay value between the video and audio.

[0009] In conjunction with the video and audio synchronization method provided in the first aspect, as an optional implementation, a first device obtains latency information from a second device, including: the first device sending a device latency query request message to the second device; and the first device receiving a device latency query response message from the second device, the device latency query response message carrying the latency information. Exemplarily, the first device may proactively send a request to the second device to query the latency information of the second device, thereby setting a latency value for the second device to synchronize the video and audio on the second device, thereby improving the user's viewing and listening experience.

[0010] In conjunction with the audio and video synchronization method provided in the first aspect, as an optional implementation, the first device obtaining latency information from the second device includes: the first device receiving latency information reported by the second device. For example, the second device may periodically report local latency information, allowing the first device to set video and audio delay values ​​for the audio and video played on the second device, thereby improving audio and video synchronization on the second device.

[0011] In combination with the audio and video synchronization method provided in the first aspect, as an optional implementation method, the second device includes an audio playback device and a video playback device. The aforementioned first device sets the first video delay value and the first audio delay value of the second device according to the delay information, including: if the video delay information is greater than the audio delay information, the first device performs audio delay compensation on the audio playback device, and the compensation amount of the audio delay compensation is determined according to the difference between the video delay information and the audio delay information. In this way, the compensation amount of the audio delay compensation can be determined according to the delay information of the second device, so that the compensation amount of the audio delay compensation matches the delay between the video and audio played in the second device, thereby avoiding the problem of poor audio and video synchronization effect caused by the delay declaration value being a preset fixed value in the conventional technology, which is conducive to improving the user's experience of watching videos and listening to audio.

[0012] On the second aspect, the present application provides another method for video and audio synchronization. The video and audio synchronization method is executed by the second device, and the video and audio synchronization method includes: the second device sends video and audio delay information to the first device, and the delay information includes: video delay information, audio delay information, video delay compensation capability and audio delay compensation capability. In addition, the second device obtains the first video delay value and the first audio delay value set by the first device, and plays the video and audio according to the first video delay value and the first audio delay value. In the present application, the video delay value and the audio delay value of the second device are determined based on the delay information of the second device, which avoids the problem of poor video and audio synchronization caused by the fixed value of the delay statement in the conventional technology, which is conducive to improving the user experience of watching videos and listening to audio.

[0013] In combination with the audio and video synchronization method provided in the second aspect, as an optional implementation method, the second device sends audio and video delay information to the first device, including: the second device receives the query device delay request message of the first device, and sends a query device delay response message to the first device, and the query device delay response message carries delay information.

[0014] In combination with the audio and video synchronization method provided in the second aspect, as an optional implementation method, the second device sends delay information to the first device, including: in response to a change in the delay information of the second device, the second device sends updated delay information to the first device.

[0015] In conjunction with the video and audio synchronization methods provided in the first and second aspects, as an optional implementation, the difference between the video delay information and the audio delay information is a first value. The video delay value is determined based on the first value and the video delay compensation capability; and / or the audio delay value is determined based on the first value and the audio delay compensation capability.

[0016] In a third aspect, the present application provides a video and audio synchronization device, which includes a module for executing any optional implementation of the first aspect and the second aspect.

[0017] Exemplarily, the video and audio synchronization apparatus includes an acquisition module and a setting module. The acquisition module is configured to acquire delay information of the second device, the delay information including video delay information, audio delay information, video delay compensation capability, and audio delay compensation capability. The setting module is configured to set a first video delay value and a first audio delay value of the second device based on the delay information. The acquisition module and the setting module may be configured to collaboratively execute the method of any optional implementation of the first aspect.

[0018] In another exemplary embodiment, the video and audio synchronization apparatus includes a transceiver module and a playback module. The transceiver module is configured to transmit video and audio delay information to a first device, the delay information including video delay information, audio delay information, video delay compensation capability, and audio delay compensation capability; and obtain a first video delay value and a first audio delay value set by the first device. The playback module is configured to play the video and audio according to the first video delay value and the first audio delay value. The transceiver module and the playback module can be configured to collaboratively execute the method of any optional implementation of the second aspect.

[0019] In a fourth aspect, the present application provides a processing device. The processing device includes: a processor and a transceiver. The processor is configured to obtain delay information of a second device and, in collaboration with the transceiver, execute the method provided by any optional implementation of the first aspect.

[0020] In a fifth aspect, the present application provides an audio-visual playback device. The audio-visual playback device includes a transceiver and a playback unit. The transceiver is configured to transmit video delay information to a first device and collaborate with the playback unit to execute the method provided in any optional implementation of the second aspect.

[0021] In a sixth aspect, the present application provides an audio-visual playback system. The audio-visual playback system includes: the processing device provided in the fourth aspect, and one or more audio-visual playback devices provided in the fifth aspect, wherein the processing device and the audio-visual playback devices are connected via an audio-video interface network. Exemplarily, the processing device is configured to implement the functions of the first device in any optional implementation of the first or second aspect, and the audio-visual playback device is configured to implement the functions of the second device in any optional implementation of the first or second aspect.

[0022] In a seventh aspect, the present application provides a chip. The chip includes a control circuit and an interface circuit. The interface circuit is configured to receive or transmit delay information and collaborate with the control circuit to execute the method of any optional implementation of the first or second aspects.

[0023] In an eighth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium stores computer software instructions. When the computer software instructions are executed on a computer, the computer performs the method of any optional implementation of the first or second aspects. Exemplarily, the computer is the aforementioned first device, second device, processing device, or audio / video playback device.

[0024] In a ninth aspect, the present application provides a computer program product. When the computer program product is executed on a computer, the computer executes the method in any optional implementation of the first aspect or the second aspect. Exemplarily, the computer is the first device, the second device, the processing device, or the audio / video playback device described above.

[0025] In a tenth aspect, the present application provides another method for video and audio synchronization. The method includes: a first device obtaining delay information of a second device, the delay information including video delay information, audio delay information, video delay compensation capability, and audio delay compensation capability; the first device setting a first video delay value and a first audio delay value of the second device based on the delay information; and the second device playing the video and audio based on the first video delay value and the first audio delay value.

[0026] Regarding the beneficial effects of the third to tenth aspects, reference may be made to the description of any implementation in the first or second aspects, and no further details will be given here. Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] FIG1 is a schematic diagram of an audio and video transmission system provided by the present application;

[0028] FIG2 is a schematic diagram of an audio and video encoding and decoding system provided by the present application;

[0029] FIG3 is a flowchart of a method for synchronizing video and audio provided by the present application;

[0030] FIG4 is a second flow chart of a method for synchronizing video and audio provided by the present application;

[0031] FIG5 is a schematic diagram of a query device delay response message structure provided by the present application;

[0032] FIG6 is a third flow chart of a method for synchronizing video and audio provided by the present application;

[0033] FIG7 is a schematic diagram of a message structure for setting device delay request provided by the present application;

[0034] FIG8 is a schematic diagram of a message structure for setting device delay response provided by the present application;

[0035] FIG9 is a fourth flow chart of a method for synchronizing video and audio provided by the present application;

[0036] FIG10 is a flowchart diagram 5 of a video and audio synchronization method provided by the present application;

[0037] FIG11 is a schematic diagram of the structure of a device delay change notification message provided by the present application;

[0038] FIG12 is a schematic diagram of the structure of a device delay change notification response message provided by the present application;

[0039] FIG13A is a sixth flow chart of a video and audio synchronization method provided by the present application;

[0040] FIG13B is a flowchart diagram of a video and audio synchronization method provided by the present application;

[0041] FIG14 is a flowchart of an audio and video synchronization method provided by the present application;

[0042] FIG15 is a structural diagram of a video and audio synchronization device provided by the present application;

[0043] FIG16 is a second structural diagram of a video and audio synchronization device provided by the present application;

[0044] FIG17 is a schematic diagram of the structure of the processing equipment provided by this application;

[0045] FIG18 is a schematic structural diagram of the video and audio playback device provided in this application. DETAILED DESCRIPTION

[0046] The present application provides a method for synchronizing video and audio. After the first device queries the delay information of the second device, the first device sets the video delay value and audio delay value of the second device according to the delay information. The delay information includes: video delay information, audio delay information, video delay compensation capability, and audio delay compensation capability. In this way, the first device can reasonably adjust the delay compensation of the second device according to the delay of the second device playing the video and audio, and realize the synchronization of video and audio on the sending side (first device) of the video and audio, rather than adjusting it on the receiving side (second device) of the video and audio. This is conducive to improving the synchronization effect of the video and audio in the second device, so as to enhance the user's experience of watching videos and listening to audio.

[0047] The technical solutions involved in this application may be applicable not only to current audio and video transmission technologies or audio and video standards, but also to future audio and video transmission technologies or audio and video standards. The terms used in the implementation methods of this application are only used to explain the specific embodiments of this application and are not intended to limit this application. The following is a brief introduction to some concepts that may be involved in this application.

[0048] Descriptive information packet (DIP or DIP message): used to describe different data in a data packet.

[0049] Video streaming refers to the transmission of video data. For example, a video stream can be processed as a stable and continuous flow over a network. A video stream consists of multiple video frames, each corresponding to an image. Image parameters can include color and transparency. Color information describes the different colors used in the video image, such as red, green, or blue. Alpha refers to the degree of transparency of a transparent display, that is, the transparency of the display device background or the display screen background.

[0050] Audio stream: A data stream that supports streaming technology and may include one or more audio frames that carry audio data to be played.

[0051] Audio and video synchronization: The synchronization process between video and audio, also known as lip sync. For example, to ensure a good user experience, the audio delay should match the video delay as closely as possible. Typically, audio should not lag behind the video image by more than 45ms, nor should audio lead the video image by more than 15ms.

[0052] In this embodiment, video is a general term that includes a sequence of multiple consecutive frames, with one frame corresponding to one image. Video and audio are information application technology terms that refer to video, audio, or multimedia content including video and audio.

[0053] In order to make the description of the following embodiments clear and concise, an introduction to related technologies is first given.

[0054] Figure 1 is a schematic diagram of a video and audio transmission system provided by the present application. The video processing process may include but is not limited to: video and audio acquisition, video and audio encoding, video and audio transmission, video and audio decoding and playback.

[0055] The audio and video transmission system in Figure 1 includes a set-top box 110, a smart TV 120, multiple audio and video playback devices, and a server 130. Set-top box 110 accesses the operator's network via a network cable and receives audio and video streams from server 130. The network can implement audio and video transmission and may include one or more network devices, such as a router or switch. In some optional implementations, communication between set-top box 110 and server 130 may also be achieved via wireless communication, although this application is not limited thereto.

[0056] The set-top box 110 is a video and audio processing device that is used to implement functions such as receiving, processing, and pushing video streams or video and audio streams. In some possible scenarios, the set-top box 110 may also be called an Internet TV set-top box, a network high-definition player, or other similar devices. For example, the set-top box 110 may refer to a TV box provided by a network operator or a TV box purchased by a user. For the hardware implementation of the set-top box 110, reference may be made to the description of FIG. 17 below, which will not be described in detail here.

[0057] Smart TV 120 is a display device with audio and video processing capabilities that implements functions such as receiving, processing, pushing, and playing video or audio streams. In some possible scenarios, Smart TV 120 may refer to an audio and video device such as a conference tablet, smart TV, or projector, but this application is not limited to this. The hardware implementation of Smart TV 120 can be referred to in the description of Figures 17 or 18 below and will not be detailed here.

[0058] The multiple audio and video playback devices include audio and video playback devices 121 to 124. For example, these audio and video playback devices may include, but are not limited to, multimedia control platforms or other devices that support audio and video playback functions, such as virtual reality (VR) terminal devices or augmented reality (AR) terminal devices. The hardware implementation of the audio and video playback devices can be found in the description of FIG. 18 below and is not detailed here.

[0059] In this embodiment, the set-top box 110 and the smart TV 120 may be connected via a channel 125. The set-top box 110 and each audio and video playback device may also be connected via the channel 125. The smart TV 120 and each audio and video playback device may also be connected via the channel 125. Exemplarily, the channel 125 may be a data channel supporting video and audio transmission. For example, the channel 125 may be a unified multimedia interconnection channel, such as a UMI (unified multimedia interconnection, UMI) channel.

[0060] A unified multimedia interconnection channel (UMC) is a channel connected based on the unified multimedia interconnection interface provided by the source and sink devices. This UMC can be used to connect chargers to charge electronic devices (such as the aforementioned smart TVs, set-top boxes, or audio and video playback devices), transfer data between electronic devices and peripheral devices, and connect headphones to play audio through them. The unified multimedia interconnection interface can also be used to connect other electronic devices, such as augmented reality devices. When the UMC is used to implement data communication between devices, it can support both uncompressed and compressed video transmission, as well as various advanced features such as Quick Video Transport (QVT), Auto Low Latency Mode (ALLM), and Dynamic Frame Rate Refresh (DFR). Furthermore, the UMC supports LPCM audio and video formats defined by IEC60958, as well as various HDR protocols, such as those specified in T / UWA 005.1-2022, such as HDR Vivid. The UMC also supports encryption control and protection for audio and video data transmission. In some optional implementations, the channel 125 may also refer to other types of channels that can implement the functions supported by the above-mentioned unified multimedia interconnection channel.

[0061] In this embodiment, the unified multimedia interconnection interface is suitable for transmitting high-speed signals for audio and video devices, third-party protocol devices (such as USB / PCIe / Ethernet), etc. A device that uses the unified multimedia interconnection interface is called a unified multimedia interconnection device, and a system composed of multiple unified multimedia interconnection devices connected together is called a unified multimedia interconnection system. The capabilities provided by the unified multimedia interconnection interface include: ①, bidirectional transmission of audio and video, meeting the audio and video transmission requirements of devices such as TVs, PCs, and mobile phones, and supporting audio and video content transmission protection, visual lossless compression, etc. ②, third-party protocol data transmission, realizing the interaction between the unified multimedia interconnection device and the third-party protocol device, such as the supported third-party protocols include USB3, and Ethernet, PCIe, etc. will be supported in the future. ③, bidirectional power supply, meeting the power supply requirements of electronic devices whose power does not exceed a certain power value, such as the certain power value is 480W, 500W or other values.

[0062] In some examples, the communication network formed by the channels used to implement audio and video transmission between devices is also referred to as an audio and video interface network or an audio and video interface network. It is worth noting that in this embodiment, the term "channel" is a general term referring to a communication path or a medium for signal transmission. Channels may also be referred to as buses, data lines, communication channels, communication connections, links, or pathways, and this application does not limit these terms. In some optional implementations, the channel may be implemented using wired or wireless connection technologies.

[0063] The server 130 may be an application server or an authentication and authorization server. The server 130 may provide video services, game services, message services, music services, authentication and authorization services, and the like. In one example, the functions of multiple services may be integrated on the server 130. For example, a game service and a music service may be deployed on the server 130. In another example, the functions of some services may be integrated on the server 130. For example, some services of the game service and some services of the video service may be deployed on the server 130. The server 130 may also utilize virtualization technology to provide multiple virtual machines, and the virtual machines provide various services. The embodiments of the present application do not limit the deployment form of the server. The network device 131 is connected to the server 130 via a wireless or wired mode. FIG1 is merely a schematic diagram. Other devices may also be included in the network, which are not shown in FIG1.

[0064] It is understood that the above-mentioned audio and video is a general term. Audio and video include multiple video frames, each of which corresponds to a group of packets carrying the audio and video data to be parsed and played. Figure 1 is merely a schematic diagram. The audio and video transmission system may also include other devices, which are not shown in Figure 1. The embodiments of this application do not limit the number and type of each device included in this system.

[0065] Based on the audio and video transmission system shown in Figure 1, Figure 2 is a schematic diagram of an audio and video encoding and decoding system provided by this application, which includes a source device 210 and a host device 220. The source device 210 establishes a communication connection with the host device 220 through a unified multimedia interconnection channel.

[0066] The above-mentioned source device 210 can realize the function of audio and video encoding. As shown in Figure 1, the source device 210 can be a set-top box 110 or a smart TV 120. The source device 210 can also be an audio and video control center with audio and video encoding capabilities. For example, the audio and video control center includes one or more servers.

[0067] The source device 210 may include a data source 211 , a pre-processing module 212 , an audio and video transmission adapter 213 , and a communication interface 214 .

[0068] The data source 211 may include or may be any type of electronic device for capturing audio and video, and / or any type of source audio and video generating device, such as a computer graphics processor for generating computer animation scenes or any type of device for acquiring and / or providing source audio and video, or computer-generated source audio and video. The data source 211 may be any type of memory or storage for storing the above-mentioned source audio and video. The above-mentioned source audio and video may include multiple audio and video streams or images captured by multiple audio and video capturing devices (such as cameras), such as ultra-high definition (UHD) video, high definition (HD) video, 4K video, etc.

[0069] The pre-processing module 212 is configured to receive source audio and video and pre-process the source audio and video to obtain audio and video or multiple frames of images. For example, the pre-processing performed by the pre-processing module 212 may include color format conversion (e.g., from RGB to YCbCr), octree structuring, audio and video splicing, audio track merging and deletion, or channel number adjustment.

[0070] The video and audio transmission adapter 213 is used to receive video and audio or images and encode the video and audio, images, or images to obtain coded data. In some optional situations, the coded stream (coded data) obtained by encoding can also be called a bit stream. If the coded data is obtained by encoding video and audio data, the bit stream refers to the video and audio stream.

[0071] The communication interface 214 in the source device 210 can be used to receive encoded data (such as a video stream or an audio / video stream) and send the encoded data (or a version of the encoded data after any other processing) to another device such as the host device 220 or any other device through a unified multimedia interconnection channel for storage, display, playback, or image reconstruction.

[0072] In the example of FIG. 2 , the video and audio transmission adapter 213 and the communication interface 214 in the source device 210 may be referred to as the controller of the source device 210 , and the video and audio reception adapter 223 and the communication interface 223 in the sink device 220 may be referred to as the controller of the sink device 220 .

[0073] Optionally, the source device 210 includes a bitstream buffer, which is used to store bitstreams corresponding to one or more coding units.

[0074] The sink device 220 can implement the function of video and audio decoding. As shown in FIG1 , the sink device 220 can be any one of the smart TV 120 or the video and audio playback device shown in FIG1 .

[0075] The sink device 220 may include a video and audio playback unit 221 , a post-processing module 222 , a video and audio receiving adapter 223 , and a communication interface 224 .

[0076] The communication interface 224 in the sink device 220 is configured to receive the encoded data (or a version of the encoded data after any other processing has been performed on the encoded data) from the source device 210 or any other source device such as a storage device.

[0077] Communication interface 214 and communication interface 224 can be used to connect the source device 210 and the sink device 220 through a direct communication link, such as a direct wired connection, as shown in the unified multimedia interconnection channel in Figure 2. For details about the unified multimedia interconnection channel, please refer to the description of Figure 1 and will not be repeated here.

[0078] The communication interface 224 corresponds to the communication interface 214, for example, and can be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded data (such as video stream or audio and video stream).

[0079] Both the communication interface 224 and the communication interface 214 can be configured as a unidirectional communication interface or a bidirectional communication interface as indicated by the arrow pointing from the source device 210 to the corresponding unified multimedia interconnection channel of the sink device 220 in Figure 2, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link, or data transmission such as encoded compressed data transmission, etc.

[0080] The video and audio receiving adapter 223 is used to receive encoded data and decode the encoded data to obtain decoded data (video or video and audio, etc.).

[0081] The post-processing module 222 is used to post-process the decoded data to obtain post-processed data (such as an image to be displayed or audio and video to be played). The post-processing performed by the post-processing module 222 may include, for example, color format conversion (such as from YCbCr to RGB), octree reconstruction, audio and video splitting and fusion, or any other processing for generating data for output by the audio and video playback unit 221.

[0082] The audio and video playback unit 221 is used to receive post-processed data for display or playback to a user or viewer. The audio and video playback unit 221 can be or include any type of display for representing the reconstructed image, such as an integrated or external display screen or display. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen. The audio and video playback unit 221 can also include one or more audio and video playback modules, which can refer to a speaker, a smart speaker, or an amplifier (amplifier), etc.

[0083] As an optional implementation, the source device 210 and the sink device 220 can transmit encoded data through a data forwarding device. For example, the data forwarding device can be a router or a switch. It is worth noting that the data forwarding device needs to support a unified multimedia interconnection interface. It is understandable that the interface connection relationship between the modules illustrated in the embodiments of the present application is only a schematic illustration and does not constitute a structural limitation on the source device and the sink device. In other embodiments of the present application, the structural limitation of the source device and the sink device can also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0084] The implementation of the video transmission method provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0085] Here, the video transmission method according to an embodiment of the present application is applied to the source device 210 and the sink device 220 shown in FIG2 as an example. FIG3 is a flowchart of a video and audio synchronization method provided by the present application. In particular, the first device 31 is used to implement the functions of the source device 210, and the second device 32 is used to implement the functions of the sink device 220.

[0086] In this embodiment, the first device 31 and the second device 32 are connected via a channel. This channel may be a unified multimedia interconnection channel. The specific implementation of the unified multimedia interconnection channel can be found in the description of FIG2 , and is not described here in detail. In some examples, this channel may also be referred to as a communication channel, link, path, or other name.

[0087] In this embodiment, the first device 31 may also be referred to as an audio and video sending device, an audio and video sending terminal, or a video sending device, and the second device 32 may also be referred to as an audio and video receiving device, an audio and video playback device, or a video receiving device. It is worth noting that the second device may include one or more audio and video playback devices.

[0088] The following is an illustrative description of applicable scenarios of this embodiment with reference to FIG1 .

[0089] In a first possible application scenario, the first device 31 is the set-top box 110 in Figure 1, and the second device 32 includes the smart TV 120 or any audio and video playback device in Figure 1. For example, the set-top box pushes audio and video data to the smart TV and the audio and video playback device.

[0090] In a second possible application scenario, the first device 31 is the smart TV 120 in Figure 1, and the second device 32 includes any one or more video and audio playback devices in Figure 1. For example, the smart TV pushes video and audio data to the video and audio playback device.

[0091] The above two possible application scenarios are merely examples provided in this embodiment and should not be understood as limitations on this application. In some optional implementations, the second device 32 may include a larger number of video playback devices (such as televisions) and audio playback devices (such as speakers or amplifiers, etc.), which is not limited in this application.

[0092] The video and audio synchronization method provided by this embodiment is described in detail below with reference to FIG3 .

[0093] S310 : The first device 31 obtains delay information of the second device 32 .

[0094] The delay information includes one or a combination of the following: video delay information, audio delay information, video delay compensation capability, and audio delay compensation capability. It should be understood that in some optional situations, the delay information is also called delay information.

[0095] Video latency refers to the delay between the time the second device 32 receives the video and the time the video is played.

[0096] For example, if the second device 32 is a television, and the video is played on the television, the video delay information is the delay of the television in processing the video.

[0097] As another example, the video delay information is the sum of the video delay of the downstream device and the delay of the TV processing the video. The downstream device may include but is not limited to: a routing device or other type of transmission device between the first device 31 and the second device 32, etc.

[0098] For example, the video delay information can also be called the current video delay value. The delay unit of the video delay information is 2ms. That is, when the delay value is 50, the actual delay is 100ms. The value range is [0, 250], that is, 0 to 500ms, and 251 to 255 are reserved.

[0099] Audio latency refers to the delay between the time when the second device 32 receives the audio and the time when the audio is played.

[0100] For example, if the second device 32 is a power amplifier, and the audio is played by the power amplifier, the audio delay information is the delay of the power amplifier in processing the audio.

[0101] In another exemplary embodiment, the audio delay information is the sum of the audio delay of the downstream device and the delay of the power amplifier in processing the audio.

[0102] For example, the audio delay information can also be called the current audio delay value. The delay unit of the audio delay information is 2ms. That is, when the delay value is 50, the actual delay is 100ms. The value range is [0, 250], that is, 0 to 500ms, and 251 to 255 are reserved.

[0103] The video delay compensation capability refers to the maximum video delay, or video max latency, or maximum video latency, that the second device 32 can support. If the difference between the video delay information and the audio delay information is within the compensation range of the video delay compensation capability, the second device 32 can compensate for the video playback within the difference, synchronizing the compensated video and audio.

[0104] For example, the maximum video delay value has a delay unit of 2ms. That is, when the delay value is 50, the actual delay is 100ms. The value range is [0, 250], that is, 0 to 500ms, and 251 to 255 are reserved.

[0105] The audio delay compensation capability refers to the maximum audio delay, or audio max latency, or maximum audio latency, supported by the second device 32. If the difference between the video delay information and the audio delay information is within the compensation range of the audio delay compensation capability, the second device 32 can compensate for the audio playback within the difference range to synchronize the compensated audio and video.

[0106] For example, the maximum audio delay value has a delay unit of 2ms. That is, when the delay value is 50, the actual delay is 100ms. The value range is [0, 250], that is, 0 to 500ms, and 251 to 255 are reserved.

[0107] For example, the format of the delay information is: {video delay information, audio delay information, video delay compensation capability, audio delay compensation capability}. Taking the delay information in milliseconds (ms) as an example, if the delay information is {0, 100, 0, 200}, it indicates that the video delay information of the second device 32 is 0, the audio delay information is 100ms, video delay compensation is not supported, and the maximum value of audio delay compensation supported is 200ms.

[0108] Regarding the process of the first device 31 acquiring the delay information of the second device 32 , this embodiment provides two optional implementation methods.

[0109] In a first optional implementation manner, the first device 31 receives the delay information reported by the second device 32 .

[0110] In a second optional implementation, as shown in FIG4 , which is a second flow chart of a method for video and audio synchronization provided by the present application, an intermediate device may be included between the first device 31 and the second device 32 , and the intermediate device may be configured to forward communication information between the first device 31 and the second device 32 , such as the following query device delay request message and query device delay response message.

[0111] Referring to FIG. 4 , the above-mentioned S310 may include the following S311 and S312 .

[0112] S311 . The first device 31 sends a device latency query request message to the second device 32 .

[0113] Illustratively, Table 1 below provides a possible implementation of the query device delay request message structure.

[0114] Table 1 Message structure for querying device latency request

[0115] The general fields include the error code (Error Code), the sequence number of the current management data message (Sequence No. (SeqNo)), whether it is the last message (Last), the reserved field (Rsvd), the shuttle ID (Channel Shuttle ID) used by the service channel processed by the device delay request message at the current device output (Channel Shuttle ID), the service channel identifier (Channel), and the message identifier (Tag). The device address field is used to identify the audio and video playback device on the receiving side.

[0116] Regarding the description of the fields of the query device delay request message, the following Table 2 provides a possible example.

[0117] Table 2 Description of fields in the device latency query request message

[0118] The source adapter ID (SRC_AdapterID) may refer to an identification (ID) of the video and audio transmitting adapter in the first device 31 .

[0119] S312: The second device 32 sends a device latency query response message to the first device 31.

[0120] Corresponding to the process of S312 , the first device 31 receives the query device delay response message from the second device 32 .

[0121] The above-mentioned device delay query response message carries the delay information in S310.

[0122] The query device delay response message structure is exemplarily described below in conjunction with FIG5 and Table 3. FIG5 is a schematic diagram of a query device delay response message structure provided by the present application.

[0123] Table 3 Query device delay response message structure

[0124] The contents repeated in Table 3 and Table 1 and Table 2 are not repeated here. For the current video delay value, the current audio delay value, the maximum video delay value and the maximum audio delay value, please refer to the description of S310 above and will not be repeated here.

[0125] In some optional situations, the above S311 and S312 may also be referred to as the first device 31 using the “query device delay” command to obtain the delay information of the second device.

[0126] In this embodiment, the first device can actively send a request to the second device to query the delay information of the second device; the second device can also actively report the delay information to the first device, so that the first device can set the delay value of the second device, so that the video and audio in the second device are synchronized, thereby improving the user's experience of watching videos and listening to audio.

[0127] S320: The first device 31 sets delay compensation information of the second device 32 according to the delay information.

[0128] The delay compensation information includes one or both of a first video delay value and a first audio delay value.

[0129] The first video delay value refers to the time for the second device 32 to delay the video. In some optional examples, the video delay value may also be referred to as video delay compensation, video delay compensation information, video delay compensation value, video compensation value, or video compensation information.

[0130] The first audio delay value refers to the time for the second device 32 to delay the audio. In some optional examples, the audio delay value may also be referred to as audio delay compensation, audio delay compensation information, audio delay compensation value, audio compensation value, or audio compensation information.

[0131] Optionally, if the video delay information and the audio delay information are equal, the first device and the second device do not need to perform video and audio synchronization; if the video delay information and the audio delay information are not equal, the second device can perform audio delay compensation to reduce video and audio delay.

[0132] If the video delay information is greater than the audio delay information, the audio will play before the video. In this case, the device with audio and video processing capabilities should perform audio delay compensation. Exemplarily, the amount of audio delay compensation is determined based on the difference between the video delay information and the audio delay information. For example, the compensation amount Δt = Video_Latency - Audio_Latency, and should support audio delay compensation of no less than 200ms.

[0133] If the video delay is less than the audio delay, the video will play before the audio. If the video leads the audio by more than 80ms, this is noticeable to the user and should be avoided (e.g., by disabling certain audio processing modules or audio playback devices to reduce latency).

[0134] An audio and video playback device with audio and video processing capabilities should compensate for the delay of downstream devices (such as the aforementioned intermediate device) within its own compensation capabilities. For any delay exceeding its own compensation capabilities, it should request delay compensation from upstream devices (such as the first device). The compensation capabilities mentioned here include the aforementioned video delay compensation capabilities and audio delay compensation capabilities. The specific implementation of these capabilities can be found in S310 and will not be further elaborated here.

[0135] Optionally, with respect to the process of setting the device delay in S320, FIG6 below provides a feasible specific example, which is a flowchart diagram of a method for synchronizing video and audio provided in this application. An intermediate device may also be included between the first device 31 and the second device 32. The intermediate device may be configured to forward communication information between the first device 31 and the second device 32, such as the following set device delay request message and set device delay response message.

[0136] Referring to FIG. 6 , the above-mentioned S320 may include the following S321 and S322 .

[0137] S321 . The first device 31 sends a device delay setting request message to the second device 32 .

[0138] Regarding the message structure of the device delay setting request, an exemplary description is given below in conjunction with FIG7 and Table 4. FIG7 is a schematic diagram of a message structure of the device delay setting request provided in this application.

[0139] Table 4 Description of variables in the Set Device Delay Request message

[0140] The video delay value has a unit of 2ms. That is, when the delay value is 50, the actual delay is 100ms. The value range is [0, 250], from 0 to 500ms, with 251 to 255 reserved. The audio delay value has a unit of 2ms. That is, when the delay value is 50, the actual delay is 100ms. The value range is [0, 250], from 0 to 500ms, with 251 to 255 reserved. For descriptions of other fields in Table 4, refer to the previous table and are not repeated here.

[0141] S322 : The second device 32 sends a set device delay response message to the first device 31 .

[0142] Corresponding to the process of S322 , the first device 31 receives the device delay setting response message from the second device 32 .

[0143] [Corrected 14.12.2023 according to Rule 91] Regarding the structure of the device delay response message, an exemplary explanation is given below in combination with Figure 8 and Table 5. Figure 8 is a schematic diagram of a device delay response message structure provided by this application.

[0144] Table 5 Description of variables in the Set Device Delay Response message

[0145] For the description of each field in Table 5, please refer to the description of the previous table and will not be repeated here.

[0146] Exemplarily, when the second device receives the request message for setting device delay, it should process the message according to steps 1 to 5 shown in the following rules:

[0147] Step 1: The second device parses the set video delay (value) and audio delay (value).

[0148] Step 2. If the set delay parameters [video delay (value) and audio delay (value)] exceed the processing capability of this device, the maximum processing delay of this device (i.e., the above-mentioned video delay compensation capability and audio delay compensation capability) is filled in the response message (set device delay response message).

[0149] Step 3: The second device generates a device delay setting response message according to the content of step 2 above.

[0150] Step 4: The second device sends a set device delay response message to the remote initiating device (the first device).

[0151] Step 5: The second device reports the delay parameters obtained in steps 1 and 2, and the device completes the delay adjustment.

[0152] Note: If the adapter number queried by the second device is inconsistent with the expected number of the adapter of this device (for example, the type is inconsistent), the error code in the response message (set device delay response message) should be modified to the corresponding error type.

[0153] In some optional situations, the above S321 and S322 may also be referred to as the first device 31 using the “set device delay” command to set the delay value (video delay value and audio delay value) of the second device 32 .

[0154] In this way, the first device can reasonably adjust the delay compensation (video delay value and audio delay value) of the second device according to the delay of the video and audio playback of the second device, and realize video and audio synchronization on the sending side of the video and audio (first device), rather than adjusting on the receiving side of the video and audio (second device). This is conducive to improving the synchronization effect of the video and audio in the second device, thereby enhancing the user experience of watching videos and listening to audio.

[0155] 3 , the video and audio synchronization method provided in this embodiment further includes the following S330 .

[0156] S330. The second device 32 plays the video and audio according to the first video delay value and the first audio delay value.

[0157] In this application, the video delay value and audio delay value of the second device are determined based on the delay information of the second device, avoiding the problem of poor video and audio synchronization caused by the fixed value of the delay declaration in conventional technology, and flexibly adjusting the video and audio delay values, which is conducive to improving the user experience of watching videos and listening to audio.

[0158] In an optional implementation, if the delay state of the second device itself changes, in order to synchronize the delays of the video and audio, the embodiments of the present application further provide a possible example, as shown in Figure 9, which is a fourth flow chart of a video and audio synchronization method provided by the present application. The hardware implementation of Figure 9 can be referred to the description of the aforementioned embodiments and will not be repeated here.

[0159] Referring to FIG. 9 , after the above-mentioned step S330 , the video and audio synchronization method provided in this embodiment further includes the following steps S340 and S350 .

[0160] S340 . In response to a change in the delay information of the second device 32 , the second device 32 sends the updated delay information to the first device 31 .

[0161] Corresponding to the process of S340 , the first device 31 receives the updated delay information reported by the second device 32 .

[0162] The specific implementation of the delay information can refer to the description of S310 above and will not be repeated here.

[0163] S350. The second device 32 sets a second video delay value and a second audio delay value of the second device 32 according to the updated delay information.

[0164] The above delay information change can also be specifically implemented using the method shown in Figure 10 below, which is a flowchart diagram of a video and audio synchronization method provided by this application. An intermediate device may also be included between the initiating device (second device 32) and the target device (first device 31). The intermediate device can be used to forward communication information between the initiating device and the target device, such as the device delay change notification message and device delay change notification response message below.

[0165] Referring to FIG. 10 , the aforementioned S340 may include the following S1010 and S1020 .

[0166] S1010: The initiating device sends a device delay change notification message to the target device.

[0167] Exemplarily, the initiating device generates a device delay setting request message according to a unicast message structure and sends the message to the remote target device.

[0168] Corresponding to the process of S1010 , the target device receives the device delay change notification message from the initiating device.

[0169] The structure of the device delay change notification message is exemplified below in conjunction with FIG11 and Table 6. FIG11 is a schematic diagram of the structure of a device delay change notification message provided by this application. The difference between FIG11 and FIG8 is that the value of the device control command number in FIG11 is 13.

[0170] Table 6 Device delay change notification message structure

[0171] For the description of each field in Table 6, please refer to Table 3 and will not be repeated here.

[0172] S1020: The target device sends a device delay change notification response message to the initiating device.

[0173] Corresponding to the process of S1020, the initiating device receives the device delay change notification response message from the target device.

[0174] The structure of the device delay change notification message is illustrated below in conjunction with Figure 12 and Table 7. Figure 12 is a schematic diagram of the structure of a device delay change notification response message provided by this application. The difference between Figure 12 and Figure 8 is that the value of the device control command number in Figure 12 is 13.

[0175] Table 7 Device delay change notification response message variable description

[0176] For the description of each field in Table 7, please refer to Table 5 and will not be repeated here.

[0177] In some optional situations, the above S1010 and S1020 may also be referred to as the initiating device using the "change device delay" command to change the delay information (video delay information and audio delay information) stored in the target device.

[0178] In this embodiment, if the delay information of the audio and video playback device changes, the playback device (second device or host device) needs to actively report the updated delay information to the source device (first device or processing device), and the source device (first device or processing device) will re-match the audio and video synchronization based on the latest delay information.

[0179] A possible specific example is provided for two different scenarios: setting delay information and changing delay information.

[0180] Taking the second device 32 including a TV and a power amplifier as an example, the first device 31 transmits video information to the TV and transmits audio information to the power amplifier.

[0181] In the delay information setting scenario, the first device 31 is the initiator of the video and audio synchronization command, and the video and audio synchronization process is shown in Figure 13A, which is a flowchart of a video and audio synchronization method provided by this application. The video and audio synchronization method includes the following ① to ③.

[0182] ① The first device uses the "Request Current Latency" command to query the TV and amplifier for latency information. By obtaining the latency information, the first device can detect that the TV has a video latency of 100ms, while the amplifier has no audio latency.

[0183] That is, the TV has a video delay of 100ms and an audio delay of 0s, does not support video delay compensation, but supports 100ms audio delay compensation, then the TV's delay capability (delay information) can be marked as {100, 0, 0, 100}.

[0184] Also, the amplifier does not support video playback, has no video delay and audio delay, does not support video delay compensation, but supports 200ms audio delay compensation, so the delay capability (delay information) of the amplifier can be marked as {0, 0, 0, 200}.

[0185] ②. The first device sets a 100ms audio delay to the amplifier through "Set Current Latency".

[0186] ③. After receiving the command to set the device delay, the power amplifier sets the audio delay value and then reports the current delay status {0, 100, 0, 200}.

[0187] Finally, the first device confirms that video and audio synchronization has been achieved through the combined message, and the current channel delay status is: video delay 100ms, audio delay 100ms.

[0188] It is worth noting that if the audio compensation delay of the amplifier cannot make up for the video delay of the TV, the first device itself needs to make up for the audio and video delay difference.

[0189] In the following example, second device 32 includes a television, and first device 31 transmits video and audio information to the television. In the delay information setting scenario, first device 31 is the initiator of the video and audio synchronization command. The video and audio synchronization process is shown in Figure 13B, which is a flowchart diagram of a video and audio synchronization method provided by this application. This video and audio synchronization method includes the following steps ① to ③.

[0190] ① The first device queries the TV for latency status (latency information) using the "Request Current Latency" command. By obtaining the latency status (latency information), the first device discovers that the TV currently has a video delay of 100ms, but its audio delay compensation capability is only 50ms.

[0191] That is, the TV has a video delay of 100ms and an audio delay of 0s, does not support video delay compensation, and supports 500ms audio delay compensation. Then the TV's delay capability (delay information) can be marked as {100, 0, 0, 100}.

[0192] ②. The first device sets the maximum supported audio delay of 50ms to the TV through "Set Current Latency".

[0193] ③. After receiving the command, the TV sets the audio delay and then reports the current delay status (delay information) as {100, 50, 0, 50}.

[0194] At this time, there is still a 50ms difference between the video delay and the audio delay on the TV playback side, and the source tester (the first device 31) needs to make up for the audio and video delay difference.

[0195] In the scenario where the delay information changes, if the TV's own delay state (delay information) changes, such as switching from 100ms to 50ms (low latency mode), the audio and video synchronization process is shown in Figure 14, which is a flowchart of an audio and video synchronization method provided by this application. The audio and video synchronization method includes the following (1) to (2).

[0196] (1) After the TV delay status changes, the audio and video status (updated delay information) should be updated to the first device.

[0197] (2) The first device re-sets a new audio delay compensation for the power amplifier based on the current video delay.

[0198] At this point, the channel has achieved video and audio synchronization, and reports the current channel delay status as: video delay is 50ms, audio delay is 50ms.

[0199] It is understandable that in order to implement the functions in the above embodiments, the first device and the second device include hardware structures and / or software modules corresponding to the execution of each function. It should be readily apparent to those skilled in the art that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in hardware or in a computer software-driven hardware manner depends on the specific application scenario and design constraints of the technical solution.

[0200] The above describes in detail the video and audio synchronization method provided according to the present embodiment in conjunction with FIG. 1 to FIG. 14 . The following describes the video and audio synchronization device provided according to the present embodiment in conjunction with FIG. 15 and FIG. 16 .

[0201] FIG15 is a structural schematic diagram of a video and audio synchronization device provided by the present application. The video and audio synchronization device 1500 can be used to implement the functions of the first device in the above-mentioned method embodiment, and thus can also achieve the beneficial effects possessed by the above-mentioned method embodiment. In this embodiment, the video and audio synchronization device 1500 can be the set-top box 110, smart TV 120, or any video and audio playback device as shown in FIG1 , or the source device 210 as shown in FIG2 , or the first device provided in subsequent embodiments. It should be understood that the video and audio synchronization device 1500 can also be a module (such as a chip) applied to any of the aforementioned devices.

[0202] As shown in Figure 15 , the video and audio synchronization device 1500 includes an acquisition module 1510 and a settings module 1520. Acquisition module 1510 and settings module 1520 can collaboratively implement the various steps in the aforementioned method embodiment. A more detailed description of acquisition module 1510 and settings module 1520 can be directly obtained by referring to the description of the first device in the method embodiment shown in the aforementioned figures, and is not further elaborated here.

[0203] FIG16 is a second structural diagram of a video and audio synchronization device provided by the present application. The video and audio synchronization device 1600 can be used to implement the functions of the second device in the above-mentioned method embodiment, thereby also achieving the beneficial effects of the above-mentioned method embodiment. In this embodiment, the video and audio synchronization device 1600 can be the smart TV 120 or any video and audio playback device as shown in FIG1 , or the host device 220 as shown in FIG2 , or the second device provided in subsequent embodiments. It should be understood that the video and audio synchronization device 1600 can also be a module (such as a chip) applied to any of the aforementioned devices.

[0204] As shown in Figure 16 , the video and audio synchronization device 1600 includes a transceiver module 1610 and a playback module 1620. Transceiver module 1610 and playback module 1620 can collaboratively implement the various steps of the aforementioned method embodiment. A more detailed description of transceiver module 1610 and playback module 1620 can be directly obtained by referring to the description of the second device in the method embodiment shown in the aforementioned figures, and is not further elaborated here.

[0205] When the audio / video synchronization device implements any of the audio / video synchronization methods shown in the aforementioned figures through software, the audio / video synchronization device and its various units may also be software modules. The audio / video synchronization method described above is implemented by calling the software module via a processor. The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0206] It can be understood that the audio and video synchronization devices shown in Figures 15 and 16 are only examples provided in this embodiment. Depending on the different audio and video transmission processes, the audio and video synchronization devices may include more or fewer units, and this application does not limit this.

[0207] When the audio / video synchronization device is implemented via hardware, the hardware may be implemented via a processor, a chip, or a chip system. The chip system includes one or more chips, each of which includes an interface circuit and a control circuit. The interface circuit is used to receive data from devices outside the chip and transmit it to the control circuit, or to send data from the control circuit to devices outside the chip. The control circuit and the interface circuit implement the method of any possible implementation method in the above embodiments through logic circuits or executing code instructions. The beneficial effects can be found in the description of any aspect of the above embodiments and will not be repeated here.

[0208] It is understood that the processor in the embodiments of the present application may be a CPU, or other general-purpose processor, digital signal processor (DSP), ASIC, FPGA or other programmable logic device, transistor logic device, hardware component or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0209] In addition, the audio and video synchronization device 1500 shown in Figure 15 can also be implemented by a processing device, as shown in Figure 17. Figure 17 is a structural diagram of the processing device provided in this application. The processing device 700 includes: a memory 710 and at least one processor 720. The processor 720 can implement the audio and video synchronization method provided in the above embodiment. The memory 710 is used to store software instructions corresponding to the above audio and video synchronization method.

[0210] As an optional implementation, in hardware implementation, the processing device 700 may refer to a chip or chip system encapsulating one or more processors 720. For example, when the processing device 700 is used to implement the method steps in the above embodiment, the processor 720 included in the processing device 700 executes the steps of the first device and possible sub-steps in the above method. In an optional scenario, the processing device 700 may also include a communication interface 730, which can be used to transmit and receive data. For example, the communication interface 730 is used to receive audio data or transmit audio streams; the communication interface 730 may be implemented by the interface circuit included in the processing device 700. Therefore, in some examples, the communication interface 730 may also be referred to as the transceiver of the processing device. In this embodiment, the communication interface 730 supports wired connection using a unified multimedia interconnect interface (such as a UMI interface).

[0211] In an embodiment of the present application, the communication interface 730, the processor 720, and the memory 710 may be connected via a bus 740, which may be divided into an address bus, a data bus, a control bus, etc. The bus 740 may be a Peripheral Component Interconnect Express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), or other types of buses.

[0212] It is worth noting that the processing device 700 can also perform the functions of the video and audio synchronization device 1500 shown in Figure 15, which will not be described here in detail.

[0213] The processing device 700 provided in this embodiment can be the set-top box 110, smart TV 120, source device 210, or first device, or other devices with video processing capabilities, and this application is not limited thereto. For example, when the aforementioned audio and video playback device also has video processing capabilities, the processing device 700 can be any of the aforementioned display devices.

[0214] In addition, the video and audio synchronization device 1600 shown in FIG16 can also be implemented by a video and audio playback device. When the video and audio synchronization device 1600 is implemented by a video and audio playback device, this embodiment provides a possible example, as shown in FIG18 , which is a schematic structural diagram of the video and audio playback device provided by this application. The video and audio playback device includes: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a unified multimedia interconnection interface 231, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio unit 270, a speaker 270A, a receiver 270B, a microphone 270C, a sensor module 280, a button 260, an indicator 262, a camera 263, and a display screen 264.

[0215] Among them, the above-mentioned sensor module 280 may include sensors such as pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor and bone conduction sensor.

[0216] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the audio and video playback device. In other embodiments, the audio and video playback device may include more or fewer components than shown, or may combine or separate certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0217] The processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0218] The controller can be the nerve center and command center of the audio and video playback device. The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0219] Processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 210 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 210. If processor 210 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 210 latency, and thus improves system efficiency.

[0220] In some embodiments, the processor 210 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a USB interface, a unified multimedia interconnect interface, etc.

[0221] It is understood that the interface connection relationship between the modules illustrated in this embodiment is merely illustrative and does not constitute a structural limitation on the audio and video playback device. In other embodiments, the audio and video playback device may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0222] The wireless communication function of the video and audio playback device can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, a modem processor, and a baseband processor. In some embodiments, antenna 1 of the video and audio playback device is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, so that the video and audio playback device can communicate with the network and other devices through wireless communication technology.

[0223] The wired communication function of the video and audio playback device can be implemented through the USB interface 230 or the unified multimedia interconnection interface 231. For example, the video and audio playback device receives or sends video streams, audio streams and video metadata messages via the bus connected to the unified multimedia interconnection interface 231.

[0224] The video and audio playback device implements its display functionality through a GPU, display screen 264, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 264 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or modify display information.

[0225] The display screen 264 is used to display images, videos, etc. The display screen 264 includes a display panel.

[0226] The video and audio playback device can implement a capture function using an ISP, camera 263, a video codec, a GPU, a display 264, and an application processor. The ISP processes data fed back by camera 263. Camera 263 is used to capture still images or video. In some embodiments, the video and audio playback device may include one or N cameras 263, where N is a positive integer greater than 1.

[0227] In this embodiment, the above display screen 264, video codec, GPU, display screen 264 and application processor can also be collectively referred to as the display unit of the audio and video playback device 32, which is used to process and display the received video stream.

[0228] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the audio and video playback device. The external memory card communicates with the processor 210 via the external memory interface 220 to implement data storage. For example, files such as music and videos can be stored on the external memory card.

[0229] The internal memory 221 can be used to store computer-executable program code, which includes instructions. The processor 210 executes the instructions stored in the internal memory 221 to perform various functional applications and data processing of the audio and video playback device. For example, in an embodiment of the present application, the processor 210 can execute instructions stored in the internal memory 221, which can include a program storage area and a data storage area.

[0230] The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the audio and video playback device (such as audio data, a phone book, etc.). In addition, the internal memory 221 can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, a flash memory device, or a universal flash storage (UFS).

[0231] The audio and video playback device can implement audio functions such as music playback and recording through the audio unit 270, the speaker 270A, the receiver 270B, the microphone 270C, and the application processor.

[0232] Buttons 260 include a power button, a volume button, and the like. Buttons 260 may be mechanical buttons or touch buttons. Indicator 262 may be an indicator light that can be used to indicate charging status, battery level changes, messages, missed calls, notifications, and the like.

[0233] The method steps in the embodiments of the present application can also be implemented by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device or a terminal device. Of course, the processor and storage medium can also exist as discrete components in a processing device and an audio and video playback device.

[0234] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the process or function described in the embodiments of the present application is performed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as a solid state drive (SSD).

[0235] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A video-audio synchronization method, characterized in that, The method includes: Obtaining the latency information of a second device; the latency information includes: video latency information, audio latency information, video latency compensation capability, and audio latency compensation capability; Setting a first video latency value and a first audio latency value of the second device according to the latency information.

2. The method according to claim 1, wherein After setting the first video latency value and the first audio latency value of the second device according to the latency information, the method further includes: Receiving updated latency information reported by the second device; Setting a second video latency value and a second audio latency value of the second device according to the updated latency information.

3. The method according to claim 1 or 2, characterized in that The obtaining the latency information of the second device includes: Sending a query device latency request message to the second device; Receiving a query device latency response message of the second device, where the query device latency response message carries the latency information.

4. The method according to claim 1 or 2, characterized in that The obtaining the latency information of the second device includes: Receiving the latency information reported by the second device.

5. The method according to any one of claims 1-4, characterized in that, The second device includes an audio playback device and a video playback device. The setting the first video latency value and the first audio latency value of the second device according to the latency information includes: If the video latency information is greater than the audio latency information, performing audio latency compensation on the audio playback device, and the compensation amount of the audio latency compensation is determined according to the difference between the video latency information and the audio latency information.

6. The method according to any one of claims 1-5, characterized in that, The difference between the video latency information and the audio latency information is a first value; The first video latency value is determined according to the first value and the video latency compensation capability; And / or, the audio latency compensation is determined according to the first value and the audio latency compensation capability.

7. An audio-visual synchronization method, characterized in that, The method includes: Sending the latency information of video and audio to a first device; the latency information includes: video latency information, audio latency information, video latency compensation capability, and audio latency compensation capability; Obtaining the first video latency value and the first audio latency value set by the first device; Playing the video and audio according to the first video latency value and the first audio latency value.

8. The method according to claim 7, wherein The sending the latency information of video and audio to the first device includes: Receiving a query device latency request message of the first device; Sending a query device latency response message to the first device, where the query device latency response message carries the latency information.

9. The method according to claim 7 or 8, characterized in that, The sending the latency information to the first device includes: In response to a change in the latency information of the second device, sending the updated latency information to the first device.

10. A processing device, characterized in that, Includes: A processor and a transceiver; The processor is configured to obtain the latency information of the second device and cooperate with the transceiver to execute the method according to any one of claims from 1 to 6.

11. An audio-visual playback device, characterized in that, Includes: A transceiver and a player; The transceiver is configured to send the latency information of video and audio to the first device and cooperate with the player to execute the method according to any one of claims from 7 to 9.

12. An audio-visual playing system, characterized in that, Includes: The processing device according to claim 10, and one or more video and audio playback devices according to claim 11, where the processing device and the video and audio playback devices are connected through a unified multimedia interconnection channel.

13. A chip, characterized in that, Includes: A control circuit and an interface circuit; the interface circuit is configured to receive or transmit delay information, and cooperate with the control circuit to execute the method according to any one of claims 1-9.

14. A computer-readable storage medium, characterized in that, Computer software instructions are stored in the computer-readable storage medium, and when the computer software instructions run on a computer, the computer executes the method according to any one of claims 1-9.

15. A computer program product, characterized in that, When the computer program product runs on a computer, the computer executes the method according to any one of claims 1-9.

16. An audio-visual synchronization device, characterized in that, Comprising: An acquisition module, configured to acquire the delay information of a second device; The delay information includes: video delay information, audio delay information, video delay compensation capability, and audio delay compensation capability; A setting module, configured to set a first video delay value and a first audio delay value of the second device according to the delay information.

17. An audio-video synchronization device, characterized in that, Comprising: A transceiver module, configured to send the delay information of video and audio to a first device; The delay information includes: video delay information, audio delay information, video delay compensation capability, and audio delay compensation capability; acquire the first video delay value and the first audio delay value set by the first device; A playback module, configured to play the video and audio according to the first video delay value and the first audio delay value.

18. A video-audio synchronization method, characterized in that, The method includes: A first device acquires the delay information of a second device; the delay information includes: video delay information, audio delay information, video delay compensation capability, and audio delay compensation capability; The first device sets a first video delay value and a first audio delay value of the second device according to the delay information; The second device plays video and audio according to the first video delay value and the first audio delay value.

Images