Video transmission method, and device

Abstract

Provided are a video transmission method and a device, which relate to the technical field of media, and can dynamically start a dynamic frame rate refreshing mode for video data, thereby reducing display anomalies of video images. The method is applied to a source device or a chip in a source device. The method comprises: querying the capability of a sink device; and when the sink device has the dynamic frame rate refreshing capability, transmitting a first vertical blanking packet (VBP) and first video data to the sink device, wherein a dynamic frame rate refreshing field included in the first VBP is a first value, and the first value is used for instructing to start a dynamic frame rate refreshing mode for the first video data.

Description

Video transmission method and device Technical Field

[0001] The present application relates to the field of media technology, and in particular to a video transmission method and device. Background Art

[0002] With the development of big data, artificial intelligence (AI), and cloud computing technologies, various types of multimedia data are constantly emerging, and more requirements are being placed on the transmission performance of multimedia data between different devices. Taking video data as an example, the complexity of video images in some scenes varies, and therefore the rendering time of video images also varies. To ensure stable transmission and normal display of video data, the frame rate should be adjusted as the video image changes. How to achieve frame rate adjustment presents many challenges.

[0003] Summary of the Invention

[0004] The present application provides a video transmission method and device, which can dynamically enable a dynamic frame rate refresh mode for video data to reduce display anomalies of the video screen.

[0005] This application adopts the following technical solutions:

[0006] In a first aspect, the present application provides a video transmission method, which is applied to a source device or a chip in a source device, the method comprising: querying the capabilities of a host device; when the host device has a dynamic frame rate refresh capability, transmitting a first vertical blanking message VBP and first video data to the host device; wherein the dynamic frame rate refresh field included in the first VBP is a first value, and the first value is used to indicate that a dynamic frame rate refresh mode is enabled for the first video data.

[0007] In the present application, when the source device queries that the host device has the dynamic frame rate refresh capability, the source device can carry a field in the VBP that instructs the host device to turn on the dynamic frame rate refresh mode, and send the VBP and video data to the host device, so that the host device turns on the dynamic frame rate refresh mode according to the instruction of the dynamic frame rate refresh field. This method can dynamically turn on the dynamic frame rate refresh mode for the video data and reduce the display abnormalities of the video screen.

[0008] Furthermore, when the host device supports dynamic frame rate refresh capability and needs to enable the dynamic frame rate refresh mode, this method can more conveniently and dynamically control the activation of the dynamic frame rate refresh mode by carrying a field indicating activation of the dynamic frame rate refresh mode in the VBP, without the need for manual activation.

[0009] In one possible implementation, the frame rate of the first video data is a first frame rate, which is the frame rate after the source device adjusts the current frame rate. Optionally, the first frame rate may be greater than the current frame rate, or may be less than the current frame rate. The direction of frame rate adjustment (i.e., increase or decrease) is related to the content of the first video data. For example, if the first video frame (including the first video data) has more complex content than the previous frame, the source device needs to reduce the video frame rate.

[0010] In one possible implementation, before transmitting the first vertical blanking message VBP and the first video data to the host device, the video transmission method provided in the present application also includes: determining the maximum frame rate and the minimum frame rate supported by the host device; adjusting the video transmission bandwidth according to the minimum value of the maximum frame rate supported by the host device, the maximum frame rate supported by the source device, and the maximum frame rate allowed by the current link bandwidth; wherein the first frame rate is greater than or equal to the minimum frame rate supported by the host device, and less than or equal to the minimum value of the maximum frame rate supported by the host device, the maximum frame rate supported by the source device, and the maximum frame rate allowed by the current link bandwidth.

[0011] In one possible implementation, the video transmission method provided in the present application also includes: transmitting a second vertical blanking message VBP and second video data to the host device; wherein the dynamic frame rate refresh field included in the second VBP is a second value, and the second value is used to indicate that the dynamic frame rate refresh mode is turned off for the second video data.

[0012] In one possible implementation, the frame rate of the second video data is a second frame rate, which is greater than or equal to the minimum video transmission frame rate supported by the host device, and less than or equal to the minimum of the maximum video transmission frame rate supported by the host device, the maximum video transmission frame rate supported by the source device, and the maximum video transmission frame rate allowed by the current link bandwidth.

[0013] In one possible implementation, the above-mentioned transmission of the first VBP and the first video data to the host device specifically includes: transmitting a first video frame to the host device, the first video frame including at least one first VBP and multiple first valid video packets AVP, and the multiple first AVPs are used to carry the first video data.

[0014] In one possible implementation, when the first frame rate is lower than the current frame rate, the duration of the front shoulder of the vertical blanking area of ​​the first video frame is increased. The source device inserts an additional video line in the front shoulder area of ​​the vertical blanking area of ​​a video frame to extend the front shoulder of the vertical blanking area (i.e., increase the duration of the front shoulder of the vertical blanking area) to adjust the time interval between two adjacent video frames. It can be seen that the transmission time of a video frame is increased and the video frame rate is reduced.

[0015] In one possible implementation, when the first frame rate is greater than the frame rate of the video data currently transmitted by the source device, the duration of the front shoulder of the vertical blanking area of ​​the first video frame is reduced. The source device discards some video lines in the front shoulder area of ​​the vertical blanking area of ​​a video frame and reduces the front shoulder of the vertical blanking area (i.e., reduces the duration of the front shoulder of the vertical blanking area) to adjust the time interval between two adjacent video frames. It can be seen that the transmission duration of a video frame is reduced and the video frame rate is increased.

[0016] In one possible implementation, the above-mentioned transmission of the second VBP and the second video data to the host device specifically includes: transmitting a second video frame to the host device, the second video frame including at least one second VBP and multiple second valid video packets AVP, and the multiple second AVPs are used to carry the second video data.

[0017] In this application, VBP is sent in each video frame, and VBP includes a field for dynamic frame rate refresh mode. In this way, frame-level control of video data can be achieved to enable dynamic frame rate refresh mode for a certain video frame, which is highly flexible.

[0018] In the second aspect, the present application provides a video transmission method, which is applied to a host device or a chip in the host device. The video transmission method provided by the present application includes: receiving a first vertical blanking message VBP and first video data sent by a source device; when the dynamic frame rate refresh field included in the first VBP is a first value, starting a dynamic frame rate refresh mode for the first video data.

[0019] In the present application, after the host device receives the first VBP and the first video data sent by the source device, the source device can carry a field in the VBP to instruct the host device to turn on the dynamic frame rate refresh mode, and turn on the dynamic frame rate refresh mode according to the instruction of the dynamic frame rate refresh field. This method can dynamically turn on the dynamic frame rate refresh mode for the video data and reduce the display abnormalities of the video screen.

[0020] Furthermore, when the host device supports dynamic frame rate refresh capability and needs to enable the dynamic frame rate refresh mode, this method can more conveniently and dynamically control the activation of the dynamic frame rate refresh mode by carrying a field indicating activation of the dynamic frame rate refresh mode in the VBP, without the need for manual activation.

[0021] In a possible implementation, the frame rate of the first video data is a first frame rate, and the first frame rate is a frame rate obtained by adjusting a current frame rate by the source device.

[0022] In one possible implementation, the video transmission method provided herein further includes: processing the first video data at a first frame rate by the sink device. When the source device adjusts the video frame rate to the first frame rate, the sink device correspondingly adjusts the video frame rate to ensure frame rate synchronization between the sink device and the source device, thereby reducing display anomalies such as video tearing.

[0023] In one possible implementation, the first frame rate is greater than or equal to the minimum frame rate supported by the sink device, and less than or equal to the minimum of the maximum frame rate supported by the sink device, the maximum frame rate supported by the source device, and the maximum frame rate allowed by the current link bandwidth.

[0024] In one possible implementation, the video transmission method provided in the present application also includes: receiving a second vertical blanking message VBP and second video data sent by a source device; when the dynamic frame rate refresh field included in the second VBP is a second value, turning off the dynamic frame rate refresh mode for the second video data.

[0025] In one possible implementation, the frame rate of the second video data is a second frame rate, which is greater than or equal to the minimum video transmission frame rate supported by the host device, and less than or equal to the minimum of the maximum video transmission frame rate supported by the host device, the maximum video transmission frame rate supported by the source device, and the maximum video transmission frame rate allowed by the current link bandwidth.

[0026] In one possible implementation, the above-mentioned receiving the first vertical blanking message VBP and the first video data sent by the source device specifically includes: receiving the first video frame sent by the source device, the first video frame includes at least one first VBP and multiple first valid video messages AVP, and the multiple first AVPs are used to carry the first video data.

[0027] In one possible implementation, when the first frame rate is lower than the current frame rate, the duration of the front porch of the vertical blanking area of ​​the first video frame is increased. When the first frame rate is higher than the frame rate of the video data currently transmitted by the source device, the duration of the front porch of the vertical blanking area of ​​the first video frame is decreased.

[0028] In one possible implementation, the second vertical blanking message VBP and the second video data sent by the receiving source device specifically include: receiving the second video frame sent by the source device, the second video frame includes at least one second VBP and multiple second valid video messages AVP, and the multiple second AVPs are used to carry the second video data.

[0029] In a third aspect, the present application provides a video processing device, which includes a module for executing the method of any one of the implementations of the first aspect or the second aspect.

[0030] In a fourth aspect, the present application provides a source device comprising: a memory, a transceiver, and a processor; the memory, the transceiver, and the processor are configured to collaboratively execute the method described in the first aspect and any one of its possible implementations.

[0031] In a fifth aspect, the present application provides a host device, comprising: a transceiver and a processor; the transceiver and the processor are configured to collaboratively execute the method described in the second aspect and any one of its possible implementations.

[0032] In a sixth aspect, the present application provides a video transmission system. The video transmission system includes the source device provided in the fourth aspect and the sink device provided in the fifth aspect. The source device can be used to implement the functions of the source device in the first or second aspect, and the sink device can be used to implement the functions of the sink device in the first or second aspect. Therefore, the video transmission system can also achieve the beneficial effects of the methods described in the first and second aspects, which will not be elaborated here.

[0033] In a seventh aspect, the present application provides a computer-readable storage medium storing computer instructions, which, when executed on a computing device, execute the method of any one of the first aspect and its possible implementations, or the second aspect and its possible implementations. For example, the computing device is the aforementioned source device or sink device.

[0034] In an eighth aspect, the present application provides a computer program product comprising computer instructions that, when executed on a computing device, execute the method of the first aspect and any one of its possible implementations. For example, the computing device is the aforementioned source device or sink device.

[0035] In a ninth aspect, the present application provides a chip system, comprising: a processor configured to call and execute a computer program from a memory, so that a computing device equipped with the chip system executes the method described in any one of the first aspect and its possible implementations, or the second aspect and its possible implementations. The computing device may be the aforementioned source device or sink device.

[0036] It should be understood that the beneficial effects achieved by the technical solutions of the second to ninth aspects of this application and the corresponding possible implementation methods can be referred to the technical effects of the first aspect and its corresponding possible implementation methods mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG1 is a schematic diagram of a video transmission system provided in an embodiment of the present application;

[0038] FIG2 is a schematic diagram of an audio and video encoding and decoding system provided in an embodiment of the present application;

[0039] FIG3 is a block diagram of an audio and video transmission adapter provided in an embodiment of the present application;

[0040] FIG4 is a block diagram of an audio / video receiving adapter provided in an embodiment of the present application;

[0041] FIG5 is a flow chart of a video transmission method according to an embodiment of the present application;

[0042] FIG6 is a schematic diagram of the structure of a vertical blanking message VBP provided in an embodiment of the present application;

[0043] FIG7 is a schematic diagram of the structure of a video frame provided in an embodiment of the present application;

[0044] FIG8 is a schematic diagram of the structure of an effective video packet AVP provided in an embodiment of the present application;

[0045] FIG9 is a schematic diagram of a timing sequence of a video frame according to an embodiment of the present application;

[0046] FIG10 is a second timing diagram of a video frame provided in an embodiment of the present application;

[0047] FIG11 is a second flow chart of a video transmission method provided in an embodiment of the present application;

[0048] FIG12 is a third flow chart of a video transmission method provided in an embodiment of the present application;

[0049] FIG13 is a schematic structural diagram of a video processing device provided by the present application;

[0050] FIG14 is a schematic diagram of the structure of a video processing device provided by this application;

[0051] FIG15 is a schematic structural diagram of the display device provided in this application. DETAILED DESCRIPTION

[0052] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0053] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects rather than to describe a specific order of the objects. For example, "first video data" and "second video data" are used to distinguish different video data rather than to describe a specific order of the video data.

[0054] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0055] In the description of the embodiments of the present application, unless otherwise specified, “plurality” means two or more.

[0056] Understandably, in some scenarios, the complexity of video images varies, and therefore the rendering time of video images also varies. For example, in a game scenario, some video images have relatively simple content, and the source device's graphics processing unit (GPU) takes a shorter time to render the video images; some video images have relatively complex content, and the source device's GPU takes a longer time to render the video images. Therefore, the frame rate of the video output by the source device's GPU may be unstable. If the source device sends the video at a fixed frame rate and the sink device displays the video at a fixed frame rate (also called a refresh rate on the source device side), this may cause display anomalies such as screen tearing when the sink device displays the video.

[0057] To meet the needs of different scenarios, an embodiment of the present application provides a video transmission method for transmitting video between a source device and a sink device. The method supports a dynamic frame rate refresh (DFR) mode (or variable refresh rate mode). When the source device finds that the sink device has dynamic frame rate refresh capability, the source device can control the sink device to turn on or off the dynamic frame rate refresh mode. The source device can carry a field in the vertical blanking message indicating that the sink device has turned on the dynamic frame rate refresh mode, and send the vertical blanking packet (VBP) and video data to the sink device. The sink device turns on the dynamic frame rate refresh mode. In this way, when the sink device displays the video data, display anomalies such as video tearing can be reduced, thereby improving the viewing experience.

[0058] The technical solutions involved in the embodiments of this application can be applied not only to current audio and video transmission technologies or audio and video standards, but may also be applied to future audio and video transmission technologies or audio and video standards. The terms used in the implementation methods of this application are only used to explain the specific embodiments of this application and are not intended to limit this application. The following is a brief introduction to some concepts that may be involved in this application.

[0059] Video streaming refers to the transmission of video data. For example, a video stream can be processed as a stable and continuous flow across a network. Video (or video data) captured by a video capture device consists of multiple frames. This video data is packaged and encapsulated to create a video stream, which consists of multiple frames, each corresponding to a single image.

[0060] In this embodiment, video is a general term that includes a sequence of multiple consecutive frames, with one frame corresponding to one image. Video and audio are information application technology terms that refer to video, audio, or multimedia content including video and audio.

[0061] Dynamic frame rate refresh: refers to transmitting video at a variable frame rate and displaying video at a variable frame rate (or refresh rate). The essence of dynamic frame rate refresh is to dynamically adjust the video frame rate so that the time to transmit a video frame changes with the change of video content. When the video content is complex, the video frame rate is reduced (that is, the sending time of a video frame is extended). When the video content is simple, the video frame rate is increased (that is, the sending time of a video frame is shortened). In this way, the source device transmits video at a dynamic frame rate, and the sink device displays video at a dynamic frame rate, which can reduce display anomalies such as screen tearing.

[0062] The video formats supported by the video transmission method provided in the embodiments of the present application include but are not limited to RGB, YCbCr4:4:4, YCbCr4:2:2, YCbCr4:2:0, ARGB, Y-only, RAW, etc.

[0063] The video transmission method provided in the embodiments of the present application supports video data with component bit widths (or color depths) including, but not limited to, one or more of the following: 8-bit, 10-bit, 12-bit, or 16-bit. For example, for video data in RGB format, if the component bit width is 8-bit, the R component of a pixel occupies 8 bits, the G component occupies 8 bits, and the B component occupies 8 bits. This can also be described as supporting video data with 8 bits per component (bpc), 10 bits per component, 12 bits per component, or 16 bits per component.

[0064] In order to make the description of the following embodiments clear and concise, an introduction to related technologies is first given.

[0065] Figure 1 is a schematic diagram of a video transmission system provided by an embodiment of the present application. The video processing process may include but is not limited to: video acquisition, video encoding, video transmission, video decoding and playback processes.

[0066] The audio and video transmission system in Figure 1 includes a set-top box 110, a smart TV 120, multiple audio and video playback devices, and a server 130. The set-top box 110 is connected to the operator's network via a network cable and can receive audio and video streams from the server 130. The network can implement audio and video transmission and can include one or more network devices, such as a router or switch. In some optional implementations, the set-top box 110 and the server 130 can also communicate via wireless communication, which is not limited in this embodiment of the present application.

[0067] The set-top box 110 is a video and audio processing device that is used to implement functions such as receiving, processing, and pushing video streams or video and audio streams. In some possible scenarios, the set-top box 110 may also be called an Internet TV set-top box, a network high-definition player, or other similar devices. For example, the set-top box 110 may refer to a TV box provided by a network operator or a TV box purchased by a user. For the hardware implementation of the set-top box 110, reference may be made to the description of FIG. 11 below, which will not be described in detail here.

[0068] Smart TV 120 is a display device with audio and video processing capabilities that implements functions such as receiving, processing, pushing, and playing video or audio streams. In some possible scenarios, Smart TV 120 may refer to an audio and video device such as a conference tablet, smart TV, or projector, but this embodiment of the application is not limited to this. The hardware implementation of Smart TV 120 can be referred to the description of Figures 11 or 12 below and will not be repeated here.

[0069] The multiple audio and video playback devices include audio and video playback devices 121 to 124. For example, these audio and video playback devices may include, but are not limited to, multimedia control platforms or other devices that support audio and video playback functions, such as virtual reality (VR) terminal devices or augmented reality (AR) terminal devices. The hardware implementation of the audio and video playback devices can be found in the description of FIG. 12 below and is not detailed here.

[0070] In this embodiment, the set-top box 110 and the smart TV 120 are connected via a network, for example, via an audio and video interface network. The set-top box 110 and each audio and video playback device may also be connected via an audio and video interface network. The smart TV 120 and each audio and video playback device may also be connected via an audio and video interface network. Exemplarily, the audio and video interface network may be a wired network that supports video and audio and video transmission. Optionally, the audio and video interface network may be referred to as a unified multimedia internet network. In some implementations, the unified multimedia internet network may also be referred to by other names.

[0071] The unified multimedia internet network supports both uncompressed and compressed video transmission, as well as advanced features such as quick video transport (QVT), auto low latency mode (ALLM), and dynamic frame rate refresh (DFR). It also supports LPCM audio and video formats defined in IEC 60958, as well as various HDR protocols, such as HDR Vivid, as specified in T / UWA 005.1-2022. The unified multimedia internet network also supports encryption control and protection for audio and video data transmission.

[0072] The server 130 may be an application server or an authentication and authorization server. The server 130 may provide video services, game services, message services, music services, authentication and authorization services, and the like. In one example, the functions of multiple services may be integrated on the server 130. For example, a game service and a music service may be deployed on the server 130. In another example, the functions of some services may be integrated on the server 130. For example, some services of the game service and some services of the video service may be deployed on the server 130. The server 130 may also utilize virtualization technology to provide multiple virtual machines, which provide various services. The embodiments of the present application do not limit the deployment form of the server. The network device 131 is connected to the server 130 via a wireless or wired mode. The schematic diagram of FIG1 is merely an example. Other devices may also be included in the network, which are not shown in FIG1.

[0073] Figure 1 is only a schematic diagram, and the video and audio transmission system may also include other devices, which are not shown in Figure 1. The embodiment of the present application does not limit the number and type of each device included in the system.

[0074] Based on the audio and video transmission system shown in Figure 1, Figure 2 is a schematic diagram of an audio and video encoding and decoding system provided in an embodiment of the present application. The audio and video encoding and decoding system includes a source device 210 and a host device 220. The source device 210 establishes a communication connection with the host device 220 through a unified multimedia Internet network.

[0075] The above-mentioned source device 210 can realize the function of audio and video encoding. As shown in Figure 1, the source device 210 can be the set-top box 110 or the smart TV 120 shown in Figure 1. The source device 210 can also be an audio and video control center with audio and video encoding capabilities. For example, the audio and video control center includes one or more servers.

[0076] The source device 210 may include a data source 211 , a pre-processing module 212 , an audio and video transmission adapter 213 , and a communication interface 214 .

[0077] The data source 211 may include or may be any type of electronic device for capturing audio and video, and / or any type of source audio and video generating device, such as a computer graphics processor for generating computer animation scenes or any type of device for acquiring and / or providing source audio and video, or computer-generated source audio and video. The data source 211 may be any type of memory or storage for storing the above-mentioned source audio and video. The above-mentioned source audio and video may include multiple audio and video streams or images captured by multiple audio and video capturing devices (such as cameras), such as ultra high definition (UHD) video, high definition (HD) video, 4K video, 8K video, etc.

[0078] The pre-processing module 212 is configured to receive source audio and video and pre-process the source audio and video to obtain audio and video or multiple frames of images. For example, the pre-processing performed by the pre-processing module 212 may include color format conversion (e.g., from RGB to YCbCr), octree structuring, audio and video splicing, audio track merging and deletion, or channel number adjustment.

[0079] The audio / video transmission adapter 213 is used to receive audio / video or images and encode the audio / video or images to obtain encoded data. In some optional situations, the encoded code stream (encoded data) may also be referred to as a bit stream. If the encoded data is obtained by encoding audio / video data, the bit stream refers to the audio / video stream; if the encoded data is obtained by encoding video data, the bit stream refers to the video stream.

[0080] The communication interface 214 in the source device 210 can be used to receive encoded data (such as a video stream or an audio / video stream) and send the encoded data (or a version of the encoded data after any other processing) to another device such as the host device 220 or any other device via the unified multimedia internet network for storage, display, playback, or image reconstruction.

[0081] Optionally, the source device 210 includes a bitstream buffer, which is used to store bitstreams corresponding to one or more coding units.

[0082] The above-mentioned sink device 220 can implement the function of audio and video decoding. As shown in Figure 1, when the source device 210 is a set-top box 110, the sink device 220 can be any of the smart TVs 120 or audio and video playback devices shown in Figure 1. When the source device 210 is a smart TV 120, the sink device 220 can be any of the audio and video playback devices.

[0083] The sink device 220 may include a video and audio playback unit 221 , a post-processing module 222 , a video and audio receiving adapter 223 , and a communication interface 224 .

[0084] The communication interface 224 in the sink device 220 is configured to receive the encoded data (or a version of the encoded data after any other processing has been performed on the encoded data) from the source device 210 or any other source device such as a storage device.

[0085] Communication interface 214 and communication interface 224 can be used to connect source device 210 and sink device 220 through a direct communication link, such as a direct wired connection, as shown in the unified multimedia internet network in FIG2 . For details about the unified multimedia internet network, please refer to the description of FIG1 and will not be repeated here.

[0086] The communication interface 224 corresponds to the communication interface 214 and can be used, for example, to transmit data and process the data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain audio and video data.

[0087] Both the communication interface 224 and the communication interface 214 can be configured as a unidirectional communication interface or a bidirectional communication interface as indicated by the arrow pointing from the source device 210 to the corresponding unified multimedia internet network of the sink device 220 in FIG2 , and can be used to send and receive messages, etc. to establish a connection, and transmit other information, etc., such as transmitting information related to the communication link, or, for example, transmitting information related to audio and video data (such as descriptive information DIP of audio and video).

[0088] The video and audio receiving adapter 223 is used to receive encoded data and decode the encoded data to obtain decoded data (video or video and audio, etc.).

[0089] The post-processing module 222 is used to post-process the decoded data to obtain post-processed data (such as an image to be displayed or audio and video to be played). The post-processing performed by the post-processing module 222 may include, for example, color format conversion (such as from YCbCr to RGB), octree reconstruction, audio and video splitting and fusion, or any other processing for generating data for output by the audio and video playback unit 221.

[0090] The audio and video playback unit 221 is used to receive post-processed data for display or playback to a user or viewer, etc. The audio and video playback unit 221 can be or include any type of display for representing the reconstructed image, such as an integrated or external display screen or display. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen. The audio and video playback unit 221 can also include one or more audio and video playback modules, each of which can refer to a speaker, a smart speaker, an amplifier, etc.

[0091] As an optional implementation, the source device 210 and the sink device 220 can transmit the encoded data through a data forwarding device. For example, the data forwarding device can be a router or a switch. It is worth noting that the data forwarding device needs to support the unified multimedia internet network.

[0092] Figure 3 shows a logical block diagram of an audio / video transmitter adapter. The adapter receives video and audio signals, including vertical synchronization (Vsync), horizontal synchronization (Hsync), pixel clock, video data, and display enable (DE), as well as descriptive information. These signals are then encapsulated into corresponding packets, such as the vertical blanking packet (VBP), horizontal blanking packet (HBP), active video packet (AVP), audio sample packet (ASP), and descriptive information packet (DIP). When content protection is enabled, the AVP and ASP are sent to the content protection encryption module for encryption, generating an encryption description packet (EDP) and a key distribution packet (KDP). The encrypted AVP is AVP', and the encrypted ASP is ASP'. These packets are combined into a single audio / video stream by the audio / video stream multiplexer and then sent to the audio / video receiver adapter of the sink device via the transport and electrical layers of the unified multimedia internetwork.

[0093] Figure 4 is a logic block diagram of an audio and video receiving adapter. The audio and video receiving adapter is responsible for receiving the audio and video stream, parsing it into various messages (VBP, HBP, DIP, AVP (AVP' in the case of encryption), ASP (ASP' in the case of encryption)), and restoring vertical synchronization (Vsync) / horizontal synchronization (Hsync) / pixel clock / pixel data / display enable (DE) / audio clock / audio data based on the messages.

[0094] The implementation of the video transmission method provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0095] Here, the video transmission method of an embodiment of the present application is described as being executed by the source device 210 or a chip in the source device 210 and the sink device 220 or a chip in the sink device 220 shown in FIG2 . FIG5 is a flow chart of a video transmission method provided by the present application. In this embodiment, the source device may also be referred to as an audio and video transmitting device, an audio and video transmitting terminal, etc., and the sink device may also be referred to as an audio and video receiving device, an audio and video playing device, etc. In this embodiment, the source device and the sink device are connected via a unified multimedia internet network.

[0096] In a first possible application scenario, the source device may be the set-top box 110 in Figure 1, and the sink device may be the smart TV 120 in Figure 1. For example, the set-top box pushes video and audio data to the smart TV.

[0097] In a second possible application scenario, the source device may be the set-top box 110 in FIG1 , and the sink device may be any video and audio playback device in FIG1 , such as any one of the video and audio playback devices 121 to 124. For example, the set-top box pushes video and audio data to the video and audio playback device.

[0098] In a third possible application scenario, the source device may be the smart TV 120 in FIG1 , and the sink device may be any of the video and audio playback devices in FIG1 , such as any of the video and audio playback devices 121 to 124. For example, the smart TV pushes video and audio data to the video and audio playback device.

[0099] The above three possible application scenarios are merely examples provided in this embodiment and should not be construed as limiting the present application. In other possible examples, the source device may be any of the video and audio playback devices in FIG1 (e.g., video and audio playback device 121), and the sink device may be another video and audio playback device different from the aforementioned video and audio playback device (e.g., video and audio playback device 122).

[0100] 5 , the video transmission method provided in the embodiment of the present application includes S501 - S503 .

[0101] S501: The source device queries the capability of the sink device.

[0102] In an embodiment of the present application, the sink device provides information indicating the capabilities of the sink device. The source device queries the capabilities of the sink device, including the source device obtaining capability description information of the sink device. The process of the source device obtaining the capability information of the sink device includes: the source device sends a query message to the sink device, where the query message is used to query the capabilities of the sink device; and then, the sink device sends (feeds back) the capability description information of the sink device to the source device, and the source device receives the capability description information sent by the sink device, where the capability description information may indicate the capabilities of the sink device.

[0103] The capability description information of the sink device may include information indicating the dynamic frame rate refresh capability of the sink device, that is, including information indicating whether the sink device has the dynamic frame rate refresh capability. In some possible situations, the capability description information may also be called a capability descriptor, which may include one or more bits of information.

[0104] In some cases, the sink device's capability description information may also be referred to as a device comprehensive capability description (DCCD). The area in the sink device that stores this DCCD information is referred to as a DCCD area or simply DCCD, though this application does not limit this. In the DCCD, the sink device declares whether it supports dynamic frame rate refresh capability, such as support for dynamic frame rate refresh capability. The source device queries the sink device's capabilities through the DCCD, and the sink device then provides feedback on its capabilities to the source device.

[0105] Optionally, the DCCD may also store other content describing the audio and video processing capabilities of the host device, such as audio processing capabilities (whether the host device supports processing pure audio data packets), HDR display capabilities (whether the host device supports displaying specific types of HDR videos, such as Colorful HDR), or others.

[0106] In an optional example, the sink device includes a memory storing the capability description information. The memory may include, but is not limited to, random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable hard disk, a CD-ROM, or any other form of storage medium known in the art.

[0107] In another optional example, the sink device includes a dedicated register whose state is used to indicate the capability description information. If the value indicated by the register is 1, the capability description information indicates that the sink device has dynamic frame rate refresh capability; if the value indicated by the register is 0, the capability description information indicates that the sink device does not have dynamic frame rate refresh capability.

[0108] S502: If the sink device has a dynamic frame rate refresh capability, the source device transmits a first VBP and first video data to the sink device, where the first VBP includes a dynamic frame rate refresh (DFR) field with a first value, indicating that a dynamic frame rate refresh mode is enabled for the first video data. Accordingly, the sink device receives the first VBP and first video data sent by the source device.

[0109] The first video data is a frame of data in the video data to be sent by the source device, that is, the first video data corresponds to valid data (or valid pixels, or valid pixel data) of a frame image in the video.

[0110] It is understood that when the source device determines that the sink device has dynamic frame rate refresh capability, the source device can choose whether to instruct the sink device to enable dynamic frame rate refresh mode based on actual needs. For example, the source device can choose whether to enable dynamic frame rate refresh mode based on the type of video content. Optionally, different types of video content may include, but are not limited to, graphics, images, videos, or games. The source device can also choose whether to enable dynamic frame rate refresh mode for all types of video content.

[0111] In the embodiment of the present application, the vertical blanking message is used to transmit the VBS (Vblank Start) signal, i.e., the vertical blanking start signal. The vertical blanking message VBP includes a 4-byte message header (also called a tunnel message header) and a 28-byte message payload (also called a payload, load), totaling 32 bytes.

[0112] For example, referring to the structural diagram of VBP shown in Figure 6, the tunnel message header occupies 4 bytes, the message payload occupies 28 bytes, and a single VBP occupies a total of 32 bytes. The following is an exemplary description of the tunnel message header and message payload of VBP in combination with Table 1 and Table 2.

[0113] Table 1: VBP tunnel header

[0114] The message payload of the VBP includes frame-level control information (VFC), which is used to describe information related to the video of this frame.

[0115] Table 2: VBP message payload

[0116] In conjunction with Table 2, DFR in Table 2 stands for the dynamic frame rate refresh field. The DFR field occupies 1 bit. A value of 1 in the DFR field indicates that the dynamic frame rate refresh mode is enabled, and a value of 0 in the DFR field indicates that the dynamic frame rate refresh mode is not enabled. For example, if the source device determines that the dynamic frame rate refresh mode needs to be enabled, the source device sets the 1st bit of the DFR field in the VBP to 1 to instruct the sink device to enable DFR mode.

[0117] In one possible implementation, the process of the source device transmitting the first VBP and the first video data to the host device specifically includes: the source device transmits a first video frame to the host device, the first video frame includes at least one first VBP and multiple first valid video packets AVP, and the multiple first AVPs are used to carry the first video data.

[0118] In the embodiment of the present application, after the source device (the video and audio transmission adapter in the source device) obtains the video data, it encapsulates the video data into an AVP message. It will be appreciated that for a frame of video data, the video data is encapsulated into multiple AVP messages. Optionally, the source device can send three VBPs to the sink device to ensure the success rate of VBP transmission.

[0119] As can be understood, referring to the block diagram of the audio and video transmission adapter shown in Figure 3 above, after the audio and video transmission adapter in the source device encapsulates the video data into AVP packets, the AVP packets in the source device are combined with other packets through the audio and video stream multiplexer to form a video stream. Other packets include vertical blanking packets (VBP), horizontal blanking packets (HBP), active video packets (AVP), audio sample packets (ASP), and descriptive information packets (DIP). The video stream includes multiple video frames, each of which corresponds to an image of the video.

[0120] Based on the above, it can be seen that in the embodiment of the present application, a video frame includes an AVP that carries video data and also includes a VBP that carries frame-level control information.

[0121] In an embodiment of the present application, VBP is sent in each video frame, and VBP includes a field for the dynamic frame rate refresh mode. In this way, frame-level control of video data can be achieved to enable the dynamic frame rate refresh mode for a certain video frame, which is highly flexible.

[0122] The format of a video frame in a video stream is described below in conjunction with FIG7 , which is a schematic diagram of the structure of a video frame provided by the present application. In FIG7 , a video frame includes: a multimedia message and a timing signal.

[0123] Multimedia packets include one or both of Active Video Packets (AVPs) and Audio Sample Packets (ASPs). AVPs are used to transmit video data, while ASPs are used to transmit audio data. As shown in Figure 7, multiple AVPs are arranged in rows within a video frame. Multiple AVPs belonging to the same row constitute a video row.

[0124] It is worth noting that the structure of the video frame in Figure 7 is only an example provided by this application and should not be understood as a limitation of this application. In some possible implementations, a single video frame may also include only one line of AVP message, which is not limited by this application.

[0125] Optionally, when the link bandwidth is greater than the video bandwidth, an ASP can be inserted between two AVPs, but the ASP has a lower priority than the AVP.

[0126] Timing signals include the horizontal synchronization signal (Hsync), the vertical synchronization signal (Vsync), and the display enable (DE) signal. These signals describe the video timing, that is, when which messages or signals should be transmitted. Hsync indicates the start of scanning a row of pixels (e.g., when the row register in the sink device receives a signal of 1), Vsync indicates the start of scanning a video frame, and display enable (DE) indicates whether valid data can be received (e.g., DE = 1 indicates that valid data can be received, and DE = 0 indicates that valid data cannot be received).

[0127] In conjunction with Figure 7, the video timing of the video frame is briefly explained: a video frame includes a vertical blanking area, a horizontal blanking area, and an active video area. In the blanking area, the video frame cannot transmit AVP messages, but can transmit other messages, such as ASP, DIP messages, etc.; in the video active area, the video frame can transmit AVP messages carrying audio and video data. Among them, the starting point of Hactive is marked by the horizontal blanking packet (HBP), which indicates the end of the line blanking area. Exemplarily, the audio and video sending adapter in the source device sends HBP for each video line and sends it immediately at the end of Hblank.

[0128] In the video frame shown in FIG7 , the line blanking area of ​​the video frame includes a VBP, which is used to indicate the video line where the rising edge of Vsync (in terms of positive polarity) of a video frame is located. Exemplarily, the audio and video transmission adapter in the source device sends one or more VBPs in each frame (such as three consecutive VBPs in FIG7 ), and ASP and DIP cannot be inserted between VBPs. For example, if Vsync is positive polarity, the audio and video transmission adapter replaces HBP with VBP on the video line where the rising edge of Vsync is located, and sends two more VBPs immediately thereafter; if Vsync is negative polarity, the audio and video transmission adapter replaces HBP with VBP on the video line where the falling edge of Vsync is located, and sends two more VBPs immediately thereafter.

[0129] Referring to the structural diagram of the valid video packet AVP shown in Figure 8, the AVP consists of two parts: a header and a payload (a total of 512 bytes). The length of the header part is 32 bits, and the payload part is used to transmit valid video pixel data (or called video data, or valid data of video, or valid pixels, or valid pixel data, or pixel data of valid video), and its length does not exceed 508 bytes.

[0130] The structure of the header of the valid video packet AVP is shown in Table 3 below.

[0131] Table 3: AVP header

[0132] As shown in Figure 8, each AVP is encapsulated as a 32-bit line, that is, in the payload of the AVP message, each line includes 32 bits of data. If 32 bits (i.e., 4 bytes) is defined as a symbol, then the payload of an AVP includes multiple symbols.

[0133] In an embodiment of the present application, when the dynamic frame rate refresh mode (i.e., DFR mode) is enabled, the source device adjusts the current frame rate. After the frame rate adjustment, the frame rate of the first video data is the first frame rate, which is the frame rate after the source device adjusts the current frame rate. Optionally, the first frame rate may be greater than the current frame rate, or the first frame rate may be less than the current frame rate. The direction of the frame rate adjustment (i.e., increase or decrease) is related to the content of the first video data. For example, if the first video frame (including the first video data) has more complex content than the previous frame, the source device needs to reduce the video frame rate.

[0134] In the embodiments of the present application, the purpose of dynamically refreshing the frame rate, i.e., dynamically adjusting the video frame rate, is achieved by adjusting the time interval between adjacent video frames. The longer the time interval between adjacent video frames, the lower the video frame rate; conversely, the higher the frame rate. It is understood that the interval between adjacent video frames refers to the time interval between the transmission times of two video frames, which is the transmission duration of a video frame. The longer the transmission duration of a video frame, the higher the frame rate, and the shorter the transmission duration, the lower the frame rate.

[0135] In one implementation, when the dynamic frame rate refresh mode is turned on, the frame rate is adjusted by adjusting the length of the front porch of the vertical blanking area, and other video timings remain unchanged.

[0136] When the first frame rate is less than the current frame rate, the duration of the vertical blanking area front porch (VFrontPorch) of the first video frame is increased. It can be understood that the vertical blanking area includes the vertical blanking area front porch, vertical synchronization (Vsync) and the vertical blanking area back porch (VBackPorch). Exemplarily, referring to Figure 9, the source device inserts additional video lines in the vertical blanking area of ​​a video frame to extend the vertical blanking area front porch (i.e., increase the duration of the vertical blanking area front porch) to adjust the time interval between two adjacent video frames. It can be seen that the transmission duration of a video frame is increased and the video frame rate is reduced.

[0137] Optionally, the total duration of the video lines inserted in the front porch of the vertical blanking area is N times the duration of one video line (N is a positive integer). As shown in FIG9 , the back porch of the vertical blanking area and vertical synchronization (Vsync) of video frame 2 remain unchanged, while the front porch of the vertical blanking area is extended, and the transmission of video frame 2 is delayed, thereby achieving the purpose of reducing the frame rate.

[0138] When the first frame rate is greater than the frame rate of the video data currently being transmitted by the source device, the duration of the vertical blanking area front porch of the first video frame is reduced. For example, referring to FIG. 10 , the source device reduces the vertical blanking area front porch (i.e., reduces the duration of the vertical blanking area front porch) by discarding some video lines in the vertical blanking area front porch of a video frame to adjust the time interval between two adjacent video frames. As can be seen, the transmission duration of a video frame is reduced and the video frame rate is increased.

[0139] Optionally, the total time of the video lines discarded in the front porch of the vertical blanking area is 1 / N times the duration of a video line (N is a positive integer). As shown in Figure 10, the back porch of the vertical blanking area and vertical synchronization (Vsync) of video frame 2 remain unchanged, the front porch of the vertical blanking area is shortened, and video frame 2 is sent in advance to achieve the purpose of increasing the frame rate.

[0140] It should be noted that after the dynamic frame rate refresh mode is enabled, although the duration of the vertical blanking area of ​​the video frame has changed, the timing parameters (such as the VFrontPorch parameter) in the frame-level control information of the vertical blanking message VBP (i.e., the message payload of the VBP) should remain unchanged, that is, the host device should ignore the VFrontPorch parameter.

[0141] 5 , as shown in FIG11 , when the source device determines that the sink device has a dynamic frame rate refresh capability, the source device further performs the following S504 - S505 before sending the first VBP and the first video data to the sink device.

[0142] S504: The source device determines the maximum frame rate and the minimum frame rate supported by the sink device.

[0143] Optionally, the source device may also obtain the maximum frame rate and minimum frame rate supported by the sink device during the process of querying the capabilities of the sink device (S501 above), for example, by determining the maximum frame rate and minimum frame rate supported by the sink device from the DCCD fed back by the source device from the sink device.

[0144] It should be understood that the maximum frame rate supported by the sink device can also be referred to as the maximum sending frame rate supported by the sink device, or the maximum frame rate of the video that can be sent by the sink device. The maximum frame rates supported by other devices can also be named in a similar manner to the maximum frame rate supported by the sink device.

[0145] S505: The source device adjusts the video transmission bandwidth according to the minimum value of the maximum frame rate supported by the sink device, the maximum frame rate supported by the source device, and the maximum frame rate allowed by the current link bandwidth.

[0146] The maximum frame rate allowed by the current link bandwidth is the maximum frame rate allowed by the available bandwidth for the current video transmission. The maximum frame rate allowed by the current link bandwidth may also be referred to as the maximum video frame rate allowed by the current link bandwidth (link bandwidth status). It should be understood that there is a conversion relationship between bandwidth and frame rate. If the bandwidth is known, the frame rate allowed by the bandwidth can be calculated. Similarly, if the frame rate is known, the corresponding bandwidth can also be calculated.

[0147] It can be understood that after the source device adjusts the video transmission bandwidth, correspondingly, in the above S502 , the source device sends the first video data using the adjusted video transmission bandwidth.

[0148] In S505, after determining the current link bandwidth, the maximum frame rate allowed by the current link bandwidth can be calculated based on the current link bandwidth. After determining the minimum value of the maximum frame rate supported by the host device, the maximum frame rate supported by the source device, and the maximum frame rate allowed by the current link bandwidth, the adjusted bandwidth (for example, the first bandwidth) can be calculated based on the frame rate of the minimum value, and then the video transmission bandwidth is adjusted to the first bandwidth, which is greater than the current video transmission bandwidth.

[0149] In an embodiment of the present application, if the source device is directly connected to the host device (i.e., there is no intermediate device between the source device and the host device), the above-mentioned link bandwidth is the current available bandwidth of the source device. If one or more intermediate devices (such as switches) are also included between the source device and the host device, the source device needs to interact with the intermediate devices to obtain the available bandwidth of all intermediate devices, and determine the minimum value (i.e., the minimum available bandwidth) of the available bandwidth of the source device itself and the available bandwidth of each intermediate device. When determining the current link bandwidth, the minimum value of the available bandwidth of the source device and all intermediate devices is taken into account, which can ensure that all devices transmitting video data support adjusting the bandwidth to the first bandwidth, thereby ensuring smooth transmission of video data.

[0150] It should be noted that when the bandwidth applied for at the beginning of the service is exactly the bandwidth corresponding to the above minimum value (that is, the minimum of the maximum frame rate supported by the host device, the maximum frame rate supported by the source device, and the maximum frame rate allowed by the current link bandwidth), there is no need to adjust the video transmission bandwidth. That is, when the current link has no bandwidth margin (that is, no excess bandwidth), there is no need to adjust the video transmission bandwidth.

[0151] It should be noted that when dynamic frame rate refresh mode is enabled, the above adjustment of the video frame rate also needs to consider the frame rate adjustment range, that is, the upper and lower frame rate limits. The upper frame rate limit is the minimum of the maximum frame rate supported by the sink device, the maximum frame rate supported by the source device, and the maximum frame rate allowed by the current link bandwidth, and the lower frame rate limit is the minimum frame rate supported by the sink device. In other words, the first frame rate is greater than or equal to the minimum frame rate supported by the sink device, and less than or equal to the minimum of the maximum frame rate supported by the sink device, the maximum frame rate supported by the source device, and the maximum frame rate allowed by the current link bandwidth.

[0152] It is understood that after the audio / video receiving adapter of the sink device receives the video stream, the sink device parses the first video frame (including the first VBP and the first video data) in the video stream to obtain the value of the DFR field in the VBP and determines whether to enable the dynamic frame rate refresh mode. Specifically, if the dynamic frame rate refresh field included in the first VBP has a first value (i.e., the value of the DFR field is 1), the sink device executes S503 below.

[0153] S503: The sink device enables a dynamic frame rate refresh mode for the first video data.

[0154] Corresponding to the VBP sent by the video sending adapter in the source device, the video receiving adapter in the sink device first performs a CRC check after receiving the VBP. If the check fails, the next VBP is received until the correct VBP is received. That is, the receiving end takes the first accurate VBP received as the basis.

[0155] In an embodiment of the present application, after the sink device receives the first video data and turns on the dynamic frame rate refresh mode, the sink device processes the first video data and displays the video picture corresponding to the first video data.

[0156] The above-mentioned source device adjusts the video frame rate to the first frame rate, and accordingly, the sink device also adjusts the video frame rate to ensure that the sink device and the source device have frame rate synchronization. Specifically, the sink device processes the first video data at the first frame rate, for example, the sink device displays the video image corresponding to the first video data at the first frame rate (for the sink device, the first frame rate can also be called the first refresh rate), or, when the frame rate is increased or decreased, the sink device performs some other processing on the first video data according to actual needs, which is not limited in the embodiments of the present application.

[0157] In summary, in the video transmission method provided in the embodiment of the present application, when the source device queries that the host device has the dynamic frame rate refresh capability, the source device can carry a field instructing the host device to turn on the dynamic frame rate refresh mode in the vertical blanking message, and send the vertical blanking packet (vertical blanking packet, VBP) and video data to the host device, so that the host device turns on the dynamic frame rate refresh mode according to the instruction of the dynamic frame rate refresh field. This method can dynamically turn on the dynamic frame rate refresh mode for the video data, thereby reducing display abnormalities of the video screen.

[0158] Furthermore, when the host device supports dynamic frame rate refresh capability and needs to enable the dynamic frame rate refresh mode, this method can more conveniently and dynamically control the activation of the dynamic frame rate refresh mode by carrying a field indicating activation of the dynamic frame rate refresh mode in the VBP, without the need for manual activation.

[0159] In one possible implementation, in the video transmission method provided in the embodiment of the present application, the dynamic frame rate refresh mode can also be turned off during the video data transmission process. In conjunction with Figure 11, as shown in Figure 12, after the above S503, the video transmission method provided in the embodiment of the present application further includes S506-S507.

[0160] S506: The source device transmits a second vertical blanking packet (VBP) and the second video data to the sink device. The dynamic frame rate refresh field included in the second VBP is a second value, which indicates that the dynamic frame rate refresh mode is disabled for the second video data. Accordingly, the sink device receives the second VBP and the second video data sent by the source device.

[0161] In one possible implementation, the process of the source device transmitting the second VBP and the second video data to the host device specifically includes: the source device transmits a second video frame to the host device, the second video frame includes at least one second VBP and multiple second valid video packets AVP, and the multiple second AVPs are used to carry the second video data.

[0162] Optionally, when the source device determines that the dynamic frame rate refresh mode needs to be turned off, the source device may adjust the current first frame rate (i.e., the frame rate adjusted when the dynamic frame rate refresh mode is turned on) to adjust the video frame rate to a fixed frame rate. That is, after the dynamic frame rate refresh mode is turned off, the video frame rate will remain unchanged. For example, if the source device adjusts the video frame rate from the first frame rate to the second frame rate, the frame rate of the second video data is the second frame rate.

[0163] It is understandable that when the source device adjusts the first frame rate to the second frame rate, it also needs to consider the frame rate adjustment range, that is, the upper and lower frame rate limits. When dynamic frame rate mode is disabled, the lower frame rate limit is still the minimum frame rate supported by the sink device, and the upper frame rate limit is the minimum of the maximum frame rate supported by the sink device, the maximum frame rate supported by the source device, and the maximum frame rate allowed by the current link bandwidth.

[0164] Based on the above description, the second frame rate is greater than or equal to the minimum video transmission frame rate supported by the sink device, and is less than or equal to the minimum of the maximum video transmission frame rate supported by the sink device, the maximum video transmission frame rate supported by the source device, and the maximum video transmission frame rate allowed by the current link bandwidth.

[0165] In some examples, the second frame rate can be set to the frame rate before the dynamic frame rate refresh mode is turned on, or the second frame rate can be set to other values, which is determined according to actual needs and is not limited in the embodiments of the present application.

[0166] After the audio and video receiving adapter of the host device receives the video stream, for the second video frame in the video stream (including the second VBP and the second video data), when the dynamic frame rate refresh field included in the second VBP is the second value (that is, the value of the DFR field is 0), the host device executes S507.

[0167] S507: The sink device disables the dynamic frame rate refresh mode for the second video data.

[0168] Optionally, in combination with the description of enabling the dynamic frame rate refresh mode on the sink device in the above embodiment, when the sink device disables the dynamic frame rate refresh mode for the second video data, the sink device displays the video at a fixed frame rate (second frame rate).

[0169] Based on the above content, it can be seen that the source device sends VBP in each video frame, and the VBP includes a field for the dynamic frame rate refresh mode. In this way, frame-level control of the video data can be achieved to dynamically turn on or off the dynamic frame rate refresh mode for the video frame.

[0170] It is understood that in order to implement the functions in the above embodiments, the source device and the sink device include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily appreciate that, in combination with the units and method steps of the various examples described in the embodiments disclosed in this application, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in hardware or in a manner driven by computer software depends on the specific application scenario and design constraints of the technical solution.

[0171] The video transmission method provided according to this embodiment is described in detail above with reference to FIG. 1 to FIG. 12 . The video processing device provided according to this embodiment will be described below with reference to FIG. 13 .

[0172] Figure 13 is a structural diagram of a video processing device provided in an embodiment of the present application. The video device can be used to implement the functions of any one of the devices in the above method embodiments, and therefore can also achieve the beneficial effects possessed by the above method embodiments. In this embodiment, the video processing device can be a set-top box 110, a smart TV 120, or any display device as shown in Figure 1, or a source device 210 or a sink device 220 as shown in Figure 2, or a source device or sink device provided in subsequent embodiments. It should be understood that the video processing device can also be a module (such as a chip) applied to any of the aforementioned devices.

[0173] As shown in Figure 13, the video processing device includes a transceiver module 1301 and a processing module 1302. Transceiver module 1301 and processing module 1302 can collaboratively implement the various steps in the aforementioned method embodiment. A more detailed description of transceiver module 1301 and processing module 1302 can be directly obtained by referring to the relevant description of the device in the method embodiment shown in the aforementioned figures, and is not repeated here.

[0174] When a video processing device implements any of the video transmission methods shown in the aforementioned figures through software, the video processing device and its various units may also be software modules. The aforementioned video transmission method is implemented by a processor calling the software module. The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0175] It can be understood that the video processing device shown in FIG13 is only an example provided in this embodiment. The video processing device may include more or fewer units according to different video transmission processes, and this application does not limit this.

[0176] When the video processing device is implemented via hardware, the hardware may be implemented via a processor or a chip system. The chip system includes one or more chips, each of which includes an interface circuit and a control circuit. The interface circuit is used to receive data from other devices outside the chip and transmit it to the control circuit, or to send data from the control circuit to other devices outside the chip. The control circuit and the interface circuit are used to implement any possible implementation method of the above embodiments through logic circuits or execution code instructions. The beneficial effects can be found in the description of any aspect of the above embodiments and will not be repeated here.

[0177] It is understood that the processor in the embodiments of the present application may be a CPU, or other general-purpose processor, digital signal processor (DSP), ASIC, FPGA or other programmable logic device, transistor logic device, hardware component or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0178] The video processing apparatus shown in FIG13 may also be implemented by a video processing device. FIG14 is a schematic structural diagram of the video processing device provided in this application. The video processing device includes: a memory 1401 and at least one processor 1402. The processor 1402 may implement the video transmission method provided in the above embodiment. The memory 1401 is used to store software instructions corresponding to the above video transmission method.

[0179] As an optional implementation, in hardware implementation, the video processing device may refer to a chip or chip system encapsulated with one or more processors 1402. For example, when the video processing device is used to implement the method steps in the above embodiment, the processor 1402 included in the video processing device executes the steps of the source device in the above method and its possible sub-steps. In an optional scenario, the video processing device may also include a communication interface 1403, which can be used to send and receive data. For example, the communication interface 1403 is used to receive audio and video data, or to send audio and video streams, etc.; the communication interface 1403 can be implemented by the interface circuit included in the video processing device. Therefore, in some examples, the communication interface 1403 can also be called a transceiver of the video processing device. In this embodiment, the communication interface 1403 supports the use of a unified multimedia internet network.

[0180] In an embodiment of the present application, the communication interface 1403, the processor 1402, and the memory 1401 may be connected via a bus 1404, which may be divided into an address bus, a data bus, a control bus, etc. The bus 1404 may be a Peripheral Component Interconnect Express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), or other types of buses.

[0181] It is worth noting that the video processing device can also perform the functions of the video processing apparatus shown in FIG13 , which will not be described in detail here.

[0182] The video processing device provided in this embodiment may be the set-top box 110, smart TV 120, source device 210, or other devices with video processing capabilities, and this application is not limited thereto. For example, when the aforementioned display device also has a video processing function, the video processing device may refer to any of the aforementioned display devices.

[0183] In addition, the video processing device shown in FIG13 can also be implemented by a display device. When the video processing device is implemented by a display device, this embodiment provides a possible example, as shown in FIG15 , which is a schematic structural diagram of the display device provided by the present application. The display device includes: a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a unified multimedia interconnection interface 331, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, a sensor module 380, a button 390, an indicator 392, a camera 393, a display screen 394, a subscriber identification module (SIM) card interface 1-N 395, etc.

[0184] Among them, the above-mentioned sensor module 380 may include sensors such as pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor and bone conduction sensor.

[0185] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the display device. In other embodiments, the display device may include more or fewer components than shown, or may combine or separate certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0186] The processor 310 may include one or more processing units. For example, the processor 310 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0187] The controller can be the nerve center and command center of the display device. The controller can generate operation control signals based on instruction opcodes and timing signals to complete the control of instruction fetching and execution.

[0188] Processor 310 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 310 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 310. If processor 310 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 310 latency, and thus improves system efficiency.

[0189] In some embodiments, the processor 310 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a USB interface, a unified multimedia interconnect interface, etc.

[0190] It is understood that the interface connection relationship between the modules illustrated in this embodiment is only for illustrative purposes and does not constitute a structural limitation on the display device. In other embodiments, the display device may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0191] The wireless communication function of the display device can be implemented through antenna 1, antenna 2, mobile communication module 350, wireless communication module 360, a modem processor, and a baseband processor. In some embodiments, antenna 1 of the display device is coupled to mobile communication module 350, and antenna 2 is coupled to wireless communication module 360, so that the display device can communicate with the network and other devices through wireless communication technology.

[0192] The wired communication function of the display device can be implemented through the USB interface 330 or the unified multimedia interconnection interface 331. For example, the display device receives or sends video streams and AVP messages through the bus connected to the unified multimedia interconnection interface 331.

[0193] The display device implements display functions through a GPU, display screen 394, and an application processor. A GPU is a microprocessor for image processing that connects display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 310 may include one or more GPUs that execute program instructions to generate or modify display information.

[0194] The display screen 394 is used to display images, videos, etc. The display screen 394 includes a display panel.

[0195] The display device can implement a camera function through an ISP, a camera 393, a video codec, a GPU, a display 394, and an application processor. The ISP processes data fed back by the camera 393. The camera 393 is used to capture still images or videos. In some embodiments, the display device may include one or N cameras 393, where N is a positive integer greater than 1.

[0196] In this embodiment, the above display screen 394, video codec, GPU, display screen 394 and application processor can also be collectively referred to as a display unit of the host device, which is used to process and display the received video stream.

[0197] External memory interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the display device. The external memory card communicates with processor 310 via external memory interface 320 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0198] The internal memory 321 can be used to store computer executable program code, which includes instructions. The processor 310 executes various functional applications and data processing of the display device by running the instructions stored in the internal memory 321. For example, in an embodiment of the present application, the processor 310 can execute instructions stored in the internal memory 321, and the internal memory 321 can include a program storage area and a data storage area.

[0199] The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the display device (such as audio and video data, a phone book, etc.). In addition, the internal memory 321 can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, a flash memory device, or a universal flash storage (UFS).

[0200] The display device can implement audio functions such as music playback and recording through the audio module 370, the speaker 370A, the receiver 370B, the microphone 370C, and the application processor.

[0201] Buttons 390 include a power button, a volume button, and the like. Buttons 390 may be mechanical buttons or touch buttons. Indicator 392 may be an indicator light that can be used to indicate charging status, battery level changes, messages, missed calls, notifications, and the like.

[0202] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using a software program, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions in accordance with the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a magnetic disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state drive (SSD)).

[0203] Through the description of the above embodiments, those skilled in the art will clearly understand that for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0204] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0205] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0206] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0207] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk.

[0208] The above is only a specific embodiment of the present application, but the scope of protection of this application is not limited to this. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A video transmission method, characterized in that, applied to a source device or a chip in the source device, the method includes: querying the capabilities of a destination device; when the destination device has the ability to refresh the dynamic frame rate, transmitting a first vertical blanking packet VBP and first video data to the destination device; wherein, the dynamic frame rate refresh field included in the first VBP is a first value, and the first value is used to indicate to enable the dynamic frame rate refresh mode for the first video data.

2. The method according to claim 1, characterized in that, the frame rate of the first video data is a first frame rate, and the first frame rate is the frame rate after the source device adjusts the current frame rate.

3. The method according to claim 1 or 2, characterized in that, before transmitting the first vertical blanking packet VBP and the first video data to the destination device, the method further includes: determining the maximum frame rate and the minimum frame rate supported by the destination device; adjusting the video transmission bandwidth according to the minimum value among the maximum frame rate supported by the destination device, the maximum frame rate supported by the source device, and the maximum frame rate allowed by the current link bandwidth; wherein, the first frame rate is greater than or equal to the minimum frame rate supported by the destination device, and less than or equal to the minimum value among the maximum frame rate supported by the destination device, the maximum frame rate supported by the source device, and the maximum frame rate allowed by the current link bandwidth.

4. The method according to any one of claims 1 to 3, characterized in that, the method further includes: transmitting a second vertical blanking packet VBP and second video data to the destination device; wherein, the dynamic frame rate refresh field included in the second VBP is a second value, and the second value is used to indicate to disable the dynamic frame rate refresh mode for the second video data.

5. The method according to any one of claims 1 to 4, characterized in that, transmitting the first VBP and the first video data to the destination device includes: transmitting a first video frame to the destination device, the first video frame includes at least one of the first VBP and a plurality of first valid video packets AVP, and the plurality of first AVP are used to carry the first video data.

6. The method according to claim 5, characterized in that, when the first frame rate is less than the current frame rate, the duration of the front shoulder of the vertical blanking area of the first video frame increases.

7. The method according to any one of claim 4, characterized in that, transmitting the second VBP and the second video data to the destination device includes: transmitting a second video frame to the destination device, the second video frame includes at least one of the second VBP and a plurality of second valid video packets AVP, and the plurality of second AVP are used to carry the second video data.

8. A video transmission method, characterized in that, applied to a destination device or a chip in the destination device, the method includes: receiving a first vertical blanking packet VBP and first video data sent by a source device; when the dynamic frame rate refresh field included in the first VBP is a first value, enabling the dynamic frame rate refresh mode for the first video data.

9. The method according to claim 8, characterized in that, The frame rate of the first video data is the first frame rate, and the first frame rate is the frame rate after the source device adjusts the current frame rate.

10. The method according to claim 8 or 9, wherein, the method further includes: the sink device processes the first video data at the first frame rate.

11. The method according to claim 9 or 10, wherein, the first frame rate is greater than or equal to the minimum frame rate supported by the sink device and less than or equal to the minimum value among the maximum frame rate supported by the sink device, the maximum frame rate supported by the source device, and the maximum frame rate allowed by the current link bandwidth.

12. The method according to any one of claims 8 to 11, wherein, the method further includes: receiving a second vertical blanking packet VBP and second video data sent by the source device; when the dynamic frame rate refresh field included in the second VBP is a second value, turning off the dynamic frame rate refresh mode for the second video data.

13. The method according to any one of claims 8 to 12, wherein, receiving a first vertical blanking packet VBP and first video data sent by the source device includes: receiving a first video frame sent by the source device, the first video frame including at least one of the first VBPs and a plurality of first valid video packets AVP, and the plurality of first AVPs being used to carry the first video data.

14. The method according to claim 13, wherein, when the first frame rate is less than the current frame rate, the duration of the front shoulder of the vertical blanking area of the first video frame increases.

15. The method according to claim 12, wherein, receiving a second vertical blanking packet VBP and second video data sent by the source device includes: receiving a second video frame sent by the source device, the second video frame including at least one of the second VBPs and a plurality of second valid video packets AVP, and the plurality of second AVPs being used to carry the second video data.

16. A source device, wherein, it includes: a memory, a transceiver, and a processor; the memory, the transceiver, and the processor are used to cooperate to execute the method according to any one of claims 1 to 7.

17. A sink device, wherein, it includes: a transceiver and a processor; the transceiver and the processor are used to cooperate to execute the method according to any one of claims 8 to 15.

18. A computer-readable storage medium, wherein, it stores computer instructions, and when the computer instructions run on a computing device, they execute the method described in any one of claims 1 to 7 or any one of claims 8 to 15.

Images