Video transmission method and device

Abstract

Provided are a video transmission method and device, relating to the technical field of media. According to the method, the consumed bandwidth during video data transmission can be reduced, so that the bandwidth utilization is improved. The method is applicable to a source device or a chip in a source device, and the method comprises: encapsulating video data into at least one active video packet (AVP), pixels being continuously arranged one after another in each row of the video data, and each AVP being encapsulated according to 32 bits in a row; and transmitting the at least one AVP to a sink device.

Description

Video transmission method and device Technical Field

[0001] The present application relates to the field of media technology, and in particular to a video transmission method and device. Background Art

[0002] With the development of big data, artificial intelligence (AI), and cloud computing, various types of multimedia data are constantly emerging, and how to transmit multimedia data between different devices has become increasingly important. Taking video data as an example, existing video data transmission technologies stipulate that the video transmitter transmits specific pixels on a specified transmission channel.

[0003] In existing video transmission methods, some transmission channels require a large amount of padding (such as filling with the value 0) due to insufficient pixels, which wastes a lot of bandwidth and leads to low bandwidth utilization.

[0004] Summary of the Invention

[0005] The present application provides a video transmission method and device, which can save bandwidth during the process of transmitting video streams between a source device and a sink device, thereby improving bandwidth utilization.

[0006] This application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a video transmission method, which is applied to a source device or a chip in a source device, the method comprising: encapsulating video data into at least one valid video packet AVP; wherein pixels within each row of the video data are arranged continuously, and each AVP is encapsulated as 32 bits per row; and transmitting at least one AVP to a host device.

[0008] In the present application, the source device can encapsulate the video data into an AVP message and transmit it to the host device. Since the pixels in each row of the video data are arranged continuously without gaps, there is no need to fill data between the pixels in the row when the video data is encapsulated. Compared with the method that requires a large amount of filling data, the present application can reduce the amount of invalid data filling, thereby saving the bandwidth consumed during the video data transmission process, thereby improving bandwidth utilization.

[0009] In a possible implementation, if the video data at the end of a line of video data cannot fill up 32 bits, data 0 is padded after the video data until 32 bits are filled up.

[0010] In the present application, pixels are arranged continuously within each row of video data, and data padding is performed only when the video data at the end of the row is less than 32 bits. The amount of padding is less than 32 bits, and the amount of padding is small. Compared with the case where more data "0" is filled, the encapsulation method of the video data in the present application can save bandwidth, thereby improving bandwidth utilization.

[0011] In one possible implementation, the video transmission method provided in an embodiment of the present application further includes: sending a vertical blanking message (VBP) to a sink device, wherein the value of a field for the number of horizontal valid pixels included in the VBP is a first value, and the first value is used to indicate the number of pixels in each row of the video data. In this way, the sink device can obtain the number of valid pixels in the video data, and thus also the number of padded data (i.e., the number of invalid data).

[0012] In a second aspect, the present application provides a video transmission method, which is applied to a host device or a chip in the host device, the method comprising: receiving at least one valid video message AVP sent by a source device; wherein, at least one AVP is a message obtained by encapsulating video data, the pixels in each row of the video data are arranged continuously, and each AVP is encapsulated according to 32 bits per row; and parsing at least one AVP to obtain video data.

[0013] In the present application, the pixels in each row of the video data in the AVP received by the host device are arranged continuously without gaps, so no data is filled between the pixels in the row. Compared with the method that requires a large amount of filling data, the present application can reduce the amount of invalid data filling, thereby saving the bandwidth consumed during the video data transmission process, thereby improving bandwidth utilization.

[0014] In a possible implementation, within a line of video data, data 0 at the end of the line is padding data, and the amount of data 0 is less than 32 bits.

[0015] In one possible implementation, the video transmission method provided by an embodiment of the present application also includes: receiving a vertical blanking message VBP sent by a source device, wherein the value of the field of the number of horizontal valid pixels included in the VBP is a first value, and the first value is used to indicate the number of pixels in each row of the video data.

[0016] Parsing at least one AVP to obtain video data specifically includes discarding padding data in at least one AVP based on the value of the horizontal active pixel count field in the VBP to obtain the video data. The sink device identifies the number of active pixels in a row of the video frame based on HActivePixels, retains the active pixels, discards the padding data at the end of the row, and recovers the video data based on the active pixels.

[0017] In combination with the first aspect or the second aspect, in one possible implementation method, the format of the video data is RGB format, and in each row of the video data, the arrangement order of the pixel components of each pixel is R component, G component, and B component, that is, the R component is arranged first, then the G component, and finally the B component.

[0018] In combination with the first aspect or the second aspect, in one possible implementation method, the format of the video data is YCbCr4:4:4 format, and in each row of the video data, the arrangement order of the pixel components of each pixel is Cr component, Y component, and Cb component, that is, the Cr component is arranged first, then the Y component, and finally the Cb component.

[0019] In combination with the first aspect or the second aspect, in a possible implementation method, the format of the video data is YCbCr4:2:2 format, and in each row of the video data, the arrangement order of the pixel components of every two pixels is Y component, Cb component, Y component, and Cr component, that is, the Y component is arranged first, and then Cb or Cr is arranged, such as a Y component is arranged first, then a Cb component, then a Y component, and finally a Cr component.

[0020] In combination with the first aspect or the second aspect, in a possible implementation method, the format of the video data is YCbCr4:2:0 format, and in the even rows of the video data, the arrangement order of the pixel components of each pixel is Y component, Y component, Cb component; in the odd rows of the video data, the arrangement order of the pixel components of each pixel is Y component, Y component, Cr component, that is, the two Y components are arranged first, and then the Cb or Cr components are arranged according to the odd and even rows, that is, the Cb component is arranged in the even rows, and the Cr component is arranged in the odd rows.

[0021] In combination with the first aspect or the second aspect, in one possible implementation method, the format of the video data is ARGB format, and in each row of the video data, the arrangement order of the pixel components of each pixel is R component, G component, B component, and A component, that is, the R component is arranged first, then the G component, the B component, and finally the A component.

[0022] In combination with the first aspect or the second aspect, in a possible implementation manner, the component bit width of the video data is any one of the following: 8 bits, 10 bits, 12 bits, or 16 bits.

[0023] In a third aspect, the present application provides a video processing device, which includes a module for executing the method of any one of the implementations of the first aspect or the second aspect.

[0024] In a fourth aspect, the present application provides a source device comprising: a memory, a transceiver, and a processor; the processor is used to encapsulate video data into at least one valid video packet AVP; the memory is used to cache the at least one AVP; the transceiver is used to transmit at least one AVP; the memory, the transceiver, and the processor are used to collaboratively execute the method described in the first aspect and any one of its possible implementations.

[0025] In a fifth aspect, the present application provides a host device comprising: a transceiver and a processor; the transceiver is used to receive at least one valid video packet AVP; the processor is used to parse at least one AVP; the transceiver and the processor are used to collaboratively execute the method described in the second aspect and any one of its possible implementation methods.

[0026] In a sixth aspect, the present application provides a video transmission system. The video transmission system includes the source device provided in the fourth aspect and the sink device provided in the fifth aspect. The source device can be used to implement the functions of the source device in the first or second aspect, and the sink device can be used to implement the functions of the sink device in the first or second aspect. Therefore, the video transmission system can also achieve the beneficial effects of the methods described in the first and second aspects, which will not be elaborated here.

[0027] In a seventh aspect, the present application provides a computer-readable storage medium storing computer instructions, which, when executed on a computing device, execute the method of any one of the first aspect and its possible implementations, or the second aspect and its possible implementations. For example, the computing device is the aforementioned source device or sink device.

[0028] In an eighth aspect, the present application provides a computer-readable storage medium, which stores the valid video packet AVP described in the first aspect and its possible implementation, or the second aspect and its possible implementation.

[0029] In a ninth aspect, the present application provides a computer program product comprising computer instructions that, when executed on a computing device, execute the method of the first aspect and any one of its possible implementations. For example, the computing device is the aforementioned source device or sink device.

[0030] In a tenth aspect, the present application provides a chip system, comprising: a processor configured to call and execute a computer program from a memory, so that a computing device equipped with the chip system executes the method described in any one of the first aspect and its possible implementations, or the second aspect and its possible implementations. The computing device may be the aforementioned source device or sink device.

[0031] It should be understood that the beneficial effects achieved by the technical solutions of the second to tenth aspects of this application and the corresponding possible implementation methods can be referred to the technical effects of the first aspect and its corresponding possible implementation methods mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG1 is a schematic diagram of a video transmission system provided in an embodiment of the present application;

[0033] FIG2 is a schematic diagram of an audio and video encoding and decoding system provided in an embodiment of the present application;

[0034] FIG3 is a block diagram of an audio and video transmission adapter provided in an embodiment of the present application;

[0035] FIG4 is a block diagram of an audio / video receiving adapter provided in an embodiment of the present application;

[0036] FIG5 is a flow chart of a video transmission method provided in an embodiment of the present application;

[0037] FIG6 is a schematic diagram of the structure of a video frame provided in an embodiment of the present application;

[0038] FIG7 is a schematic diagram of pixel data arrangement in RGB format provided by an embodiment of the present application;

[0039] FIG8 is a schematic diagram of pixel data arrangement in a YCbCr 4:4:4 format provided in an embodiment of the present application;

[0040] FIG9 is a schematic diagram of pixel data arrangement in a YCbCr 4:2:2 format provided in an embodiment of the present application;

[0041] FIG10 is a schematic diagram of pixel data arrangement in a YCbCr 4:2:0 format provided in an embodiment of the present application;

[0042] FIG11 is a schematic diagram of pixel data arrangement in an ARGB format provided by an embodiment of the present application;

[0043] FIG12 is a schematic diagram of the structure of an effective video packet AVP provided in an embodiment of the present application;

[0044] FIG13 is a schematic diagram of an arrangement of RGB format video data in an AVP according to an embodiment of the present application;

[0045] FIG14 is a schematic diagram showing an arrangement of video data in a YCbCr 4:4:4 format in an AVP according to an embodiment of the present application;

[0046] FIG15 is a schematic diagram showing an arrangement of video data in a YCbCr 4:2:2 format in an AVP according to an embodiment of the present application;

[0047] FIG16 is a schematic diagram showing one arrangement of video data in a YCbCr 4:2:0 format in an AVP according to an embodiment of the present application;

[0048] FIG17 is a second schematic diagram of an arrangement of video data in a YCbCr 4:2:0 format in an AVP according to an embodiment of the present application;

[0049] FIG18 is a schematic diagram of an arrangement of video data in ARGB format in an AVP according to an embodiment of the present application;

[0050] FIG19 is a schematic diagram showing an arrangement of video data in a Y-only format in an AVP according to an embodiment of the present application;

[0051] FIG20 is a schematic diagram of an arrangement of RAW format video data in an AVP provided in an embodiment of the present application;

[0052] FIG21 is a schematic structural diagram of a video processing device provided by the present application;

[0053] FIG22 is a schematic diagram of the structure of a video processing device provided by this application;

[0054] FIG23 is a schematic structural diagram of the display device provided in this application. DETAILED DESCRIPTION

[0055] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0056] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0057] In the description of the embodiments of the present application, unless otherwise specified, “plurality” means two or more.

[0058] Embodiments of the present application relate to a video transmission method for transmitting video between a source device and a sink device. The source device can encapsulate video data into an AVP message and transmit it to the sink device. The sink device then receives the AVP message, parses it, and obtains the video data for subsequent display. This video transmission method can save bandwidth consumed during video data transmission, thereby improving bandwidth utilization.

[0059] The technical solutions involved in the embodiments of this application may be applicable not only to current audio and video transmission technologies or audio and video standards, but also to future audio and video transmission technologies or audio and video standards. The terms used in the implementation methods of this application are only used to explain the specific embodiments of this application and are not intended to limit this application. The following is a brief introduction to some concepts that may be involved in this application.

[0060] 1. Video streaming

[0061] Video streaming refers to the transmission of video data. For example, a video stream can be processed as a stable and continuous flow across a network. Video (or video data) captured by a video capture device consists of multiple frames. This video data is packaged and encapsulated to create a video stream, which consists of multiple frames, each corresponding to a single image.

[0062] In this embodiment, video is a general term that includes a sequence of multiple consecutive frames, with one frame corresponding to one image. Video and audio are information application technology terms that refer to video, audio, or multimedia content including video and audio.

[0063] 2. Video Format

[0064] The video formats supported by the video transmission method provided in the embodiments of the present application include but are not limited to RGB, YCbCr4:4:4, YCbCr4:2:2, YCbCr4:2:0, ARGB, Y-only, RAW, etc.

[0065] Video data in RGB format includes three pixel components: red (Red, R) component, green (Green, G) component, and blue (Blue, B) component.

[0066] The video in YCbCr format includes three pixel components: Y component, Cb component, and Cr component. The Y component represents the brightness component, the Cb component represents the blue chrominance component, and the Cr component represents the red chrominance component.

[0067] The YCbCr format includes various sampling formats, such as the aforementioned YCbCr4:4:4, YCbCr4:2:2, and YCbCr4:2:0. YCbCr4:4:4 is a non-downsampled format, meaning the Y, Cb, and Cr components are sampled at the same rate, and each pixel contains complete information for all three components. YCbCr4:2:2 and YCbCr4:2:0 are downsampled formats. The YCbCr4:2:2 format means the Cb and Cr components are sampled at half the rate of the Y component, with four Y components, two Cb components, and two Cr components sampled for every four pixels. The YCbCr4:2:0 format means that for each scan line, only one chroma component is stored at a 2:1 sampling rate, and adjacent scan lines store different chroma components. That is to say, if in the first row of pixels, each pixel samples the Y component, every two pixels sample 1 Cb component, and no Cr component is sampled, then in the next row of pixels, each pixel samples the Y component, every two pixels sample 1 Cr component, and no Cb component is sampled.

[0068] The ARGB format includes four pixel components, namely red (Red, R) component, green (Green, G) component, blue (Blue, B) component and transparency (alpha, A) component.

[0069] The Y-only format contains only the Y component. The RAW format is an unprocessed and uncompressed format.

[0070] In order to make the description of the following embodiments clear and concise, an introduction to related technologies is first given.

[0071] Figure 1 is a schematic diagram of a video transmission system provided by an embodiment of the present application. The video processing process may include but is not limited to: video acquisition, video encoding, video transmission, video decoding and playback processes.

[0072] The audio and video transmission system in Figure 1 includes a set-top box 110, a smart TV 120, multiple audio and video playback devices, and a server 130. The set-top box 110 is connected to the operator's network via a network cable and can receive audio and video streams from the server 130. The network can implement audio and video transmission and can include one or more network devices, such as a router or switch. In some optional implementations, the set-top box 110 and the server 130 can also communicate via wireless communication, which is not limited in this embodiment of the present application.

[0073] The set-top box 110 is a video and audio processing device that is used to implement functions such as receiving, processing, and pushing video streams or video and audio streams. In some possible scenarios, the set-top box 110 may also be called an Internet TV set-top box, a network high-definition player, or other similar devices. For example, the set-top box 110 may refer to a TV box provided by a network operator or a TV box purchased by a user. For the hardware implementation of the set-top box 110, reference may be made to the description of FIG. 22 below, which will not be described in detail here.

[0074] Smart TV 120 is a display device with audio and video processing capabilities that implements functions such as receiving, processing, pushing, and playing video or audio streams. In some possible scenarios, Smart TV 120 may refer to an audio and video device such as a conference tablet, smart TV, or projector, but this embodiment of the application is not limited to this. The hardware implementation of Smart TV 120 can be referred to the description of Figures 22 or 23 below and will not be repeated here.

[0075] The multiple audio and video playback devices include audio and video playback devices 121 to 124. For example, these audio and video playback devices may include, but are not limited to, multimedia control platforms or other devices that support audio and video playback functions, such as virtual reality (VR) terminal devices or augmented reality (AR) terminal devices. The hardware implementation of the audio and video playback devices can be found in the description of FIG. 23 below and is not detailed here.

[0076] In this embodiment, the set-top box 110 and the smart TV 120 are connected via a network, for example, via an audio and video interface network. The set-top box 110 and each audio and video playback device may also be connected via an audio and video interface network. The smart TV 120 and each audio and video playback device may also be connected via an audio and video interface network. Exemplarily, the audio and video interface network may be a wired network that supports video and audio and video transmission. Optionally, the audio and video interface network may be referred to as a unified multimedia internet network. In some implementations, the unified multimedia internet network may also be referred to by other names.

[0077] The unified multimedia internet network supports LPCM audio and video formats defined in IEC 60958, as well as various HDR protocols, such as HDR Vivid, as specified in T / UWA 005.1-2022. The unified multimedia internet network also supports encryption control and protection for audio and video data transmission.

[0078] The server 130 may be an application server or an authentication and authorization server. The server 130 may provide video services, game services, message services, music services, authentication and authorization services, etc. In one example, the functions of multiple services may be integrated on the server 130. For example, a game service and a music service may be deployed on the server 130. In another example, the functions of some services may be integrated on the server 130. For example, some services of the game service and some services of the video service may be deployed on the server 130. The server 130 may also utilize virtualization technology to provide multiple virtual machines, which provide various services. The embodiments of the present application do not limit the deployment form of the server. The network device 131 is connected to the server 130 by wireless or wired means. The schematic diagram of FIG1 is merely an example. Other devices may also be included in the network, which are not shown in FIG1.

[0079] Figure 1 is only a schematic diagram, and the video and audio transmission system may also include other devices, which are not shown in Figure 1. The embodiment of the present application does not limit the number and type of each device included in the system.

[0080] Based on the audio and video transmission system shown in Figure 1, Figure 2 is a schematic diagram of an audio and video encoding and decoding system provided in an embodiment of the present application. The audio and video encoding and decoding system includes a source device 210 and a host device 220. The source device 210 establishes a communication connection with the host device 220 through a unified multimedia Internet network.

[0081] The above-mentioned source device 210 can realize the function of audio and video encoding. As shown in Figure 1, the source device 210 can be the set-top box 110 or the smart TV 120 shown in Figure 1. The source device 210 can also be an audio and video control center with audio and video encoding capabilities. For example, the audio and video control center includes one or more servers.

[0082] The source device 210 may include a data source 211 , a pre-processing module 212 , an audio and video transmission adapter 213 , and a communication interface 214 .

[0083] The data source 211 may include or may be any type of electronic device for capturing audio and video, and / or any type of source audio and video generating device, such as a computer graphics processor for generating computer animation scenes or any type of device for acquiring and / or providing source audio and video, or computer-generated source audio and video. The data source 211 may be any type of memory or storage for storing the above-mentioned source audio and video. The above-mentioned source audio and video may include multiple audio and video streams or images captured by multiple audio and video capturing devices (such as cameras), such as ultra high definition (UHD) video, high definition (HD) video, 4K video, 8K video, etc.

[0084] The pre-processing module 212 is configured to receive source audio and video and pre-process the source audio and video to obtain audio and video or multiple frames of images. For example, the pre-processing performed by the pre-processing module 212 may include color format conversion (e.g., from RGB to YCbCr), octree structuring, audio and video splicing, audio track merging and deletion, or channel number adjustment.

[0085] The audio / video transmission adapter 213 is used to receive audio / video or images and encode the audio / video, images, or images to obtain encoded data. In some optional situations, the encoded code stream (encoded data) may also be referred to as a bit stream. If the encoded data is obtained by encoding audio / video data, the bit stream refers to an audio / video stream; if the encoded data is obtained by encoding video data, the bit stream refers to a video stream.

[0086] The communication interface 214 in the source device 210 can be used to receive encoded data (such as a video stream or an audio / video stream) and send the encoded data (or a version of the encoded data after any other processing) to another device such as the host device 220 or any other device via the unified multimedia internet network for storage, display, playback, or image reconstruction.

[0087] Optionally, the source device 210 includes a bitstream buffer, which is used to store bitstreams corresponding to one or more coding units.

[0088] The above-mentioned sink device 220 can implement the function of audio and video decoding. As shown in Figure 1, when the source device 210 is a set-top box 110, the sink device 220 can be any of the smart TVs 120 or audio and video playback devices shown in Figure 1. When the source device 210 is a smart TV 120, the sink device 220 can be any of the audio and video playback devices.

[0089] The sink device 220 may include a video and audio playback unit 221 , a post-processing module 222 , a video and audio receiving adapter 223 , and a communication interface 224 .

[0090] The communication interface 224 in the sink device 220 is configured to receive the encoded data (or a version of the encoded data after any other processing has been performed on the encoded data) from the source device 210 or any other source device such as a storage device.

[0091] Communication interface 214 and communication interface 224 can be used to connect source device 210 and sink device 220 through a direct communication link, such as a direct wired connection, as shown in the unified multimedia internet network in FIG2 . For details about the unified multimedia internet network, please refer to the description of FIG1 and will not be repeated here.

[0092] The communication interface 224 corresponds to the communication interface 214 and can be used, for example, to transmit data and process the data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain audio and video data.

[0093] Both the communication interface 224 and the communication interface 214 can be configured as a unidirectional communication interface, as indicated by the arrow pointing from the source device 210 to the corresponding unified multimedia internet network of the sink device 220 in FIG2 , or a bidirectional communication interface, and can be used to send and receive messages, etc., to establish a connection, confirm and exchange any other information related to the communication link, or data transmission, such as encoded data transmission, etc.

[0094] The video and audio receiving adapter 223 is used to receive encoded data and decode the encoded data to obtain decoded data (video or video and audio, etc.).

[0095] The post-processing module 222 is used to post-process the decoded data to obtain post-processed data (such as an image to be displayed or audio and video to be played). The post-processing performed by the post-processing module 222 may include, for example, color format conversion (such as from YCbCr to RGB), octree reconstruction, audio and video splitting and fusion, or any other processing for generating data for output by the audio and video playback unit 221.

[0096] The audio and video playback unit 221 is used to receive post-processed data for display or playback to a user or viewer, etc. The audio and video playback unit 221 can be or include any type of display for representing the reconstructed image, such as an integrated or external display screen or display. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen. The audio and video playback unit 221 can also include one or more audio and video playback modules, each of which can refer to a speaker, a smart speaker, an amplifier, etc.

[0097] As an optional implementation, the source device 210 and the sink device 220 can transmit the encoded data through a data forwarding device. For example, the data forwarding device can be a router or a switch. It is worth noting that the data forwarding device needs to support the unified multimedia internet network.

[0098] Figure 3 shows a logical block diagram of an audio / video transmitter adapter. The adapter receives video and audio signals, including vertical synchronization (Vsync), horizontal synchronization (Hsync), pixel clock, video data, and display enable (DE), as well as descriptive information. These signals are then encapsulated into corresponding packets, such as the vertical blanking packet (VBP), horizontal blanking packet (HBP), active video packet (AVP), audio sample packet (ASP), and descriptive information packet (DIP). When content protection is enabled, the AVP and ASP are sent to the ADCP encryption module for encryption, generating an encryption description packet (EDP) and a key distribution packet (KDP). The encrypted AVP is AVP', and the encrypted ASP is ASP'. These packets are combined into a single audio / video stream by the audio / video stream multiplexer and then sent to the audio / video receiver adapter of the destination device via the transport and physical layers of the unified multimedia internetwork.

[0099] Figure 4 is a logical block diagram of an audio and video receiving adapter. The audio and video receiving adapter is responsible for receiving audio and video streams, parsing them into various messages (VBP, HBP, DIP, AVP, ASP), and restoring video signals and audio signals such as vertical synchronization (Vsync) / horizontal synchronization (Hsync) / pixel clock / pixel data / display enable (DE), as well as descriptive information based on the messages.

[0100] The implementation of the video transmission method provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0101] Here, the video transmission method of an embodiment of the present application is described as being executed by the source device 210 or a chip in the source device 210 and the sink device 220 or a chip in the sink device 220 shown in FIG2 . FIG3 is a flow chart of a video transmission method provided by the present application. In this embodiment, the source device may also be referred to as an audio and video transmitting device, an audio and video transmitting terminal, etc., and the sink device 32 may also be referred to as an audio and video receiving device, an audio and video playing device, etc. In this embodiment, the source device and the sink device are connected via a unified multimedia internet network.

[0102] In a first possible application scenario, the source device may be the set-top box 110 in Figure 1, and the display device 32 may be the smart TV 120 in Figure 1. For example, the set-top box pushes video and audio data to the smart TV.

[0103] In a second possible application scenario, the source device may be the set-top box 110 in FIG1 , and the display device 32 may be any video and audio playback device in FIG1 , such as any one of the video and audio playback devices 121 to 124. For example, the set-top box pushes video and audio data to the video and audio playback device.

[0104] In a third possible application scenario, the source device may be the smart TV 120 in FIG1 , and the display device 32 may be any video and audio playback device in FIG1 , such as any one of the video and audio playback devices 121 to 124. For example, the smart TV pushes video and audio data to the video and audio playback device.

[0105] The above three possible application scenarios are merely examples provided in this embodiment and should not be construed as limiting the present application. In other possible examples, the source device may be any of the video and audio playback devices in FIG1 (e.g., video and audio playback device 121), and the display device 32 may be another video and audio playback device different from the aforementioned video and audio playback device (e.g., video and audio playback device 122).

[0106] 5 , the video transmission method provided in the embodiment of the present application includes S501 - S503 .

[0107] S501: A source device encapsulates video data into at least one active video packet (AVP).

[0108] In the embodiment of the present application, after receiving the video, the above-mentioned source device (the audio and video transmission adapter in the source device) performs pixel arrangement on the original video data (or called the valid data of the video, or valid pixels, or valid pixel data, or pixel data of the valid video) in the video to obtain the pixel-arranged video data, and then encapsulates the pixel-arranged video data (for ease of understanding, the video data in the following embodiments refers to the pixel-arranged video data) and encapsulates the video data into AVP messages. It can be understood that for one frame of video data, the video data is encapsulated into multiple AVP messages.

[0109] As you can understand, after the video data is encapsulated into AVP packets, the AVP packets are combined with other packets to form a video stream. These other packets include vertical blanking packets (VBP), horizontal blanking packets (HBP), active video packets (AVP), audio sample packets (ASP), and descriptive information packets (DIP). A video stream includes multiple video frames, each of which corresponds to an image in the video.

[0110] The format of a video frame in a video stream is described below in conjunction with FIG6 , which is a schematic diagram of the structure of a video frame provided by the present application. In FIG6 , a video frame includes: a multimedia message and a timing signal.

[0111] Multimedia packets include one or both of Active Video Packets (AVPs) and Audio Sample Packets (ASPs). AVPs are used to transmit video data, while ASPs are used to transmit audio data. As shown in Figure 6, multiple AVPs are arranged in rows within a video frame. Multiple AVPs belonging to the same row constitute a video row.

[0112] It is worth noting that the structure of the video frame in Figure 6 is only an example provided by this application and should not be understood as a limitation of this application. In some possible implementations, a single video frame may also include only one line of AVP message, which is not limited by this application.

[0113] Optionally, when the link bandwidth is greater than the video bandwidth, an ASP can be inserted between two AVPs, but the ASP has a lower priority than the AVP.

[0114] Timing signals include a horizontal synchronization signal (Hsync), a vertical synchronization signal (Vsync), and a display enable (DE) signal. These signals describe the video timing, specifically when and what messages or signals should be transmitted. Hsync indicates the start of scanning a row of pixels (e.g., when the row register in display device 32 receives a signal of 1), Vsync indicates the start of scanning a video frame, and display enable (DE) indicates whether valid data can be received (e.g., DE = 1 indicates valid data can be received, and DE = 0 indicates invalid data cannot be received).

[0115] In conjunction with Figure 6, the video timing of a video frame is briefly explained: During the line blanking area, a video frame cannot transmit AVP messages, but can transmit other messages, such as ASP and DIP messages. During the line active area, a video frame can transmit AVP messages. The starting point of Hactive is marked by the horizontal blanking packet (HBP), which indicates the end of the line blanking area. For example, the audio and video transmission adapter in the source device sends the HBP for each video line and immediately sends it at the end of Hblank.

[0116] In the video frame shown in FIG6 , the line blanking area of ​​the video frame includes a VBP message, which is used to indicate the video line where the rising edge of Vsync (in terms of positive polarity) of a video frame is located. Exemplarily, the audio and video transmission adapter in the source device sends one or more VBPs in each frame (such as three consecutive VBPs in FIG6 ), and ASP and DIP cannot be inserted between VBPs. For example, if Vsync is positive polarity, the audio and video transmission adapter replaces HBP with VBP on the video line where the rising edge of Vsync is located, and sends two more VBPs immediately thereafter; if Vsync is negative polarity, the audio and video transmission adapter replaces HBP with VBP on the video line where the falling edge of Vsync is located, and sends two more VBPs immediately thereafter.

[0117] In an embodiment of the present application, the source device performs pixel arrangement on the valid pixel data in the received video to obtain video data. For consecutive video pixels, the first pixel data should be arranged first, and then the second pixel data should be arranged in the order received. The pixels in each row of the video data are arranged continuously, that is, except at the end of the row, the pixels in a row should be arranged without gaps, and there should be no other invalid data between the pixels in the row, such as padded data 0.

[0118] Optionally, the video transmission method provided in the embodiments of the present application supports video data and RAW data in various color spaces and sampling formats, such as RGB, YCbCr 4:4:4, YCbCr 4:2:2, YCbCr 4:2:0, Y-only, and ARGB. The following describes the arrangement of video data (i.e., pixel data) in several formats.

[0119] 1. Arrangement of pixel data in RGB format

[0120] When the video data is in RGB format, the pixel components of each pixel in each row of the video data are arranged in the order of R component, G component, and B component. In other words, for RGB format video, the audio and video transmission adapter arranges the R component first, then the G component, and finally the B component.

[0121] Exemplarily, referring to the schematic diagram of pixel data arrangement in RGB format shown in FIG7 , when DE is 1, the source device obtains pixel data at the rising edge of the pixel clock (PixelClk) and arranges the pixel data in a certain order. FIG7 illustrates the arrangement of a row of pixel data in the effective video area (i.e., a row of pixel data in an image of the video). In one frame of the original video (without format conversion or downsampling), a row includes n pixels, and the pixel points are numbered from 0 to n-1. Each pixel point includes 3 pixel components, namely R component, G component and B component. For example, as shown in FIG7 , the three pixel components of pixel point 0 are R0, G0, and B0. The arrangement order of the three pixel components of pixel point 0 is R0 first, then G0, and finally B0. The arrangement order of a row of pixels is R0, G0, B0, R1, G1, B1, R2, G2, B2, R3, G3, B3, ..., R n-2 , G n-2 、B n-2 、R n-1 , G n-1 、B n-1 , it can be seen that the pixels in a row are arranged continuously.

[0122] Optionally, for video data in RGB format, on the basis of satisfying the continuous arrangement between pixels in a row, the R component, G component and B component are also arranged in an order different from that shown in Figure 7 above. For example, the R component may be arranged first, then the B component, and finally the G component; or the B component may be arranged first, then the R component, and finally the G component; or the B component may be arranged first, then the G component, and finally the R component; or the G component may be arranged first, then the R component, and finally the B component; or the G component may be arranged first, then the R component, and finally the B component; or the G component may be arranged first, then the B component, and finally the R component. The arrangement method of each component in the RGB format is selected according to actual needs, and the embodiments of the present application are not limited thereto.

[0123] 2. Arrangement of pixel data in YCbCr4:4:4 format

[0124] When the video data format is YCbCr 4:4:4, the pixel components of each pixel in each row of the video data are arranged in the order of Cr component, Y component, and Cb component. In other words, for YCbCr 4:4:4 format video, the video and audio transmission adapter arranges the Cr component first, then the Y component, and finally the Cb component.

[0125] For example, referring to the pixel data arrangement diagram of the YCbCr4:4:4 format shown in FIG8 , when DE is 1, the source device obtains pixel data at the rising edge of the pixel clock (PixelClk) and arranges the pixel data in a certain order. Similarly, FIG8 illustrates the arrangement of a row of pixel data in the effective video area (i.e., a row of pixel data in an image of the video). In one frame of the original video (without format conversion or downsampling), a row includes n pixels, and the pixel points are numbered from 0 to n-1. Each pixel point includes three pixel components, namely, a Y component, a Cb component, and a Cr component. For example, as shown in FIG8 , the three pixel components of pixel point 0 are Y0, Cb0, and Cr0. The arrangement order of the three pixel components of pixel point 0 is Cr0 first, then Y0, and finally Cb0. The arrangement order of a row of pixels is Cr0, Y0, Cb0, Cr1, Y1, Cb1, Cr2, Y2, Cb2, Cr3, Y3, Cb3, ..., Cr n-2 、Y n-2 , Cb n-2 Cr n-1 、Y n-1 , Cb n-1 , it can be seen that the pixels in a row are arranged continuously.

[0126] Optionally, for video data in YCbCr 4:4:4 format, on the basis of satisfying the continuous arrangement between pixels in a row, the Y component, Cb component and Cr component are also arranged in an order different from that shown in Figure 8 above. For example, the Cr component may be arranged first, then the Cb component, and finally the Y component; or the Y component may be arranged first, then the Cb component, and finally the Cr component; or the Y component may be arranged first, then the Cr component, and finally the Cb component; or the Cb component may be arranged first, then the Y component, and finally the Cr component; or the Cb component may be arranged first, then the Y component, and finally the Cr component; or the Cb component may be arranged first, then the Y component, and finally the Cr component; or the Cb component may be arranged first, then the Cr component, and finally the Y component. The arrangement method of each component in the YCbCr 4:4:4 format is selected according to actual needs, and the embodiments of the present application are not limited thereto.

[0127] 3. Arrangement of pixel data in YCbCr4:2:2 format

[0128] When the video data format is YCbCr4:2:2, the pixel components of every two pixels in each row of the video data are arranged in the order of Y component, Cb component, Y component, and Cr component. In other words, for YCbCr4:2:2 format video, the audio and video transmission adapter prioritizes the Y component, then arranges Cb or Cr, for example, it prioritizes a Y component, then a Cb component, then a Y component, and finally a Cr component.

[0129] For example, referring to the schematic diagram of pixel data arrangement in the YCbCr 4:2:2 format shown in FIG9 , when DE is 1, the source device acquires pixel data on the rising edge of the pixel clock (PixelClk) and arranges the pixel data in a certain order. Similarly, FIG9 illustrates the arrangement of a row of pixel data in the active video area (i.e., a row of pixel data in an image of the video). In one frame of the original video (without format conversion or downsampling), a row includes n pixels, the pixels are numbered from 0 to n-1, and each pixel includes three pixel components, namely, a Y component, a Cb component, and a Cr component.

[0130] It should be noted that for video data in the YCbCr4:2:2 format, a row of pixel data in the effective video area is pixel data after downsampling. The pixel data shown in Figure 9 is pixel data after downsampling. A column of pixel data in the downsampling pixel data can be defined as a pixel point (also called a new pixel point). For example, the components Y0 and Cb0 of pixel point 0 in Figure 9 are a new pixel point, and the component Y1 of pixel point 1 and the component Cr0 of pixel point 0 are a new pixel point. The arrangement order of the pixel components of these two new pixel points is to arrange Y0 first, then Cb0, then Y1, and finally Cr0. The arrangement order of a row of pixels is Y0, Cb0, Y1, Cr0, Y2, Cb2, Y2, Cr2, ..., Y n-2 , Cb n-2 、Y n-1 Cr n-2 , it can be seen that the pixels in a row are arranged continuously.

[0131] In some embodiments, in each row of video data in YCbCr 4:2:2 format, the arrangement order of pixel components of every two pixels may also be Y component, Cr component, Y component, and Cb component, such as arranging a Y component first, then a Cr component, then a Y component, and finally a Cb component.

[0132] Optionally, on the basis of satisfying the continuous arrangement between pixels in a row, the arrangement order of pixel components in the video data in the YCbCr4:2:2 format may also be other orders, which is not limited in the embodiment of the present application.

[0133] 4. Arrangement of pixel data in YCbCr4:2:0 format

[0134] When the video data format is YCbCr4:2:0, the pixel components of each pixel in the even-numbered rows are arranged in the order of Y component, Y component, Cb component; and the pixel components of each pixel in the odd-numbered rows are arranged in the order of Y component, Y component, Cr component. In other words, for YCbCr420 format video, the audio and video transmission adapter prioritizes the two Y components and then arranges the Cb or Cr components based on the odd or even rows, i.e., the Cb component is arranged on even rows and the Cr component is arranged on odd rows.

[0135] For example, referring to the schematic diagram of pixel data arrangement in the YCbCr 4:2:0 format shown in FIG10 , when DE is 1, the source device acquires pixel data on the rising edge of the pixel clock (PixelClk) and arranges the pixel data in a certain order. Similarly, FIG10 illustrates the arrangement of a row of pixel data in the active video area (i.e., a row of pixel data in an image of the video), where, in a frame of the original video (without format conversion or downsampling), a row includes n pixels, the pixels are numbered from 0 to n-1, and each pixel includes three pixel components, namely, a Y component, a Cb component, and a Cr component.

[0136] It should be noted that for video data in the YCbCr4:2:0 format, a row of pixel data in the effective video area is pixel data after downsampling. In the YCbCr4:2:0 format video, the pixel data is arranged in such a way that odd rows arrange the Y component and Cb component, and even rows arrange the Y component and Cr component.

[0137] (a) in FIG10 shows the arrangement of pixel data in even rows, wherein a column of pixel data can be defined as a pixel point (also referred to as a new pixel point). For example, in (a) in FIG10 , the component Y1 of pixel point 1 in an even row, the component Y0 of pixel point 0, and the component Cb0 of pixel point 0 are a new pixel point, and the arrangement order of the pixel components of the new pixel point is to arrange Y1 first, then Y0, and finally Cb0; for another example, the component Y3 of pixel point 3 in an even row, the component Y2 of pixel point 2, and the component Cb2 of pixel point 2 are a new pixel point, and the arrangement order of the pixel components of the new pixel point is to arrange Y3 first, then Y2, and finally Cb2. For even rows, the arrangement order of a row of pixel points is Y1, Y0, Cb0, Y3, Y2, Cb2, Y5, Y4, Cb4, Y7, Y6, Cb6, ..., Y n-3 、Y n-4 , Cb n-4 、Y n-1 、Y n-2 , Cb n-2 , it can be seen that the pixels in a row are arranged continuously.

[0138] (b) in FIG10 shows the arrangement of pixel data in odd rows, wherein a column of pixel data can be defined as a pixel point (also referred to as a new pixel point). For example, in (b) in FIG10 , the component Y1 of pixel point 1 in the odd row, the component Y0 of pixel point 0, and the component Cr0 of pixel point 0 are a new pixel point, and the arrangement order of the pixel components of the new pixel point is to arrange Y1 first, then Y0, and finally Cr0; for another example, the component Y3 of pixel point 3 in the odd row, the component Y2 of pixel point 2, and the component Cr2 of pixel point 2 are a new pixel point, and the arrangement order of the pixel components of the new pixel point is to arrange Y3 first, then Y2, and finally Cr2. For odd rows, the arrangement order of a row of pixel points is Y1, Y0, Cr0, Y3, Y2, Cr2, Y5, Y4, Cr4, Y7, Y6, Cr6, ..., Y n-3 、Y n-4 Cr n-4 、Y n-1 、Y n-2 Cr n-2 , it can be seen that the pixels in a row are arranged continuously.

[0139] Optionally, on the basis of satisfying the continuous arrangement between pixels in a row, the arrangement order of pixel components in the video data in the YCbCr4:2:0 format may also be other orders, which is not limited in the embodiment of the present application.

[0140] 5. Arrangement of pixel data in ARGB format

[0141] When the video data is in ARGB format, the pixel components of each pixel in each row of the video data are arranged in the order of R component, G component, B component, and A component. In other words, for ARGB format video, the video and audio transmission adapter arranges the R component first, then the G component, then the B component, and finally the A component.

[0142] For example, referring to the pixel data arrangement diagram of the ARGB format shown in FIG11, when DE is 1, the source device obtains pixel data at the rising edge of the pixel clock (PixelClk) and arranges the pixel data in a certain order. FIG11 illustrates the arrangement of a row of pixel data in the effective video area (i.e., a row of pixel data in an image of the video). In one frame of the original video (without format conversion or downsampling), a row includes n pixels, and the pixel points are numbered from 0 to n-1. Each pixel point includes four pixel components, namely A component, R component, G component, and B component. For example, as shown in FIG11, the four pixel components of pixel point 0 are A0, R0, G0, and B0. The arrangement order of the four pixel components of pixel point 0 is R0 first, G0 second, B0 third, and A0 last. The arrangement order of a row of pixels is R0, G0, A0, B0, R1, G1, B1, A1, R2, G2, B2, A2, R3, G3, B3, A3..., R n-2 , G n-2 、B n-2 、A n-2 、R n-1 , G n-1 、B n-1 、A n-1 , it can be seen that the pixels in a row are arranged continuously.

[0143] Optionally, for video data in ARGB format, on the basis of satisfying the continuous arrangement between pixels in a row, the A component, R component, G component and B component are also arranged in an order different from that shown in Figure 11 above. For example, the R component can be arranged first, then the B component, then the A component, and finally the G component; or the four pixel components are arranged in other ways after arrangement and combination, which are not listed one by one here. The arrangement method of each component in the ARGB format is selected according to actual needs, and the embodiments of the present application are not limited to this.

[0144] In the above, the video data obtained after the pixel data is arranged is encapsulated into multiple AVPs, that is, AVP is used to transmit valid video pixel data, wherein each AVP is encapsulated according to 32 bits per line, that is, each line of the AVP message contains 32 bits.

[0145] Referring to the structural diagram of the valid video packet AVP shown in Figure 12, the AVP consists of two parts: a header and a payload (a total of 512 bytes). The length of the header part is 32 bits, and the payload part is used to transmit valid video pixel data, and its length does not exceed 508 bytes.

[0146] The structure of the header of the valid video packet AVP is shown in Table 1 below (Description of the structure of the valid video packet header).

[0147] Table 1

[0148] As shown in Figure 12, each AVP is encapsulated as a 32-bit line, that is, in the payload of the AVP message, each line includes 32 bits of data. If 32 bits (i.e., 4 bytes) is defined as a symbol, then the payload of an AVP includes multiple symbols.

[0149] The video transmission method provided in the embodiments of the present application supports video data with component bit widths (or color depths) of any one or more of the following: 8 bits, 10 bits, 12 bits, or 16 bits. For example, for video data in RGB format, if the component bit width is 8 bits, the R component of a pixel occupies 8 bits, the G component occupies 8 bits, and the B component occupies 8 bits. This can also be described as supporting video data with 8 bits per component (bpc), 10 bits, 12 bits, or 16 bits.

[0150] The following describes how the video data is arranged in the AVP during the process of encapsulating video data of different component bit widths in different video formats into the AVP.

[0151] Figure 13 illustrates the arrangement of RGB format video data in an AVP. Figure 13 (a) illustrates the arrangement of RGB format video data with an 8-bit component width in an AVP, Figure 13 (b) illustrates the arrangement of RGB format video data with a 10-bit component width in an AVP, Figure 13 (c) illustrates the arrangement of RGB format video data with a 12-bit component width in an AVP, and Figure 13 (d) illustrates the arrangement of RGB format video data with a 16-bit component width in an AVP.

[0152] Figure 14 illustrates the arrangement of video data in the YCbCr4:4:4 format in an AVP. Figure 14 (a) illustrates the arrangement of video data in the YCbCr4:4:4 format with an 8-bit component width in an AVP, Figure 14 (b) illustrates the arrangement of video data in the YCbCr4:4:4 format with a 10-bit component width in an AVP, Figure 14 (c) illustrates the arrangement of video data in the YCbCr4:4:4 format with a 12-bit component width in an AVP, and Figure 14 (d) illustrates the arrangement of video data in the YCbCr4:4:4 format with a 16-bit component width in an AVP.

[0153] Figure 15 illustrates the arrangement of video data in the YCbCr4:2:2 format in an AVP. Figure 15 (a) illustrates the arrangement of video data in the YCbCr4:2:2 format with an 8-bit component width in an AVP, Figure 15 (b) illustrates the arrangement of video data in the YCbCr4:2:2 format with a 10-bit component width in an AVP, Figure 15 (c) illustrates the arrangement of video data in the YCbCr4:2:2 format with a 12-bit component width in an AVP, and Figure 15 (d) illustrates the arrangement of video data in the YCbCr4:2:2 format with a 16-bit component width in an AVP.

[0154] Figures 16 and 17 illustrate the arrangement of video data in the YCbCr4:2:0 format in an AVP. Figure 16 (a) illustrates the arrangement of video data in the YCbCr4:2:0 format with an 8-bit component width in an AVP, Figure 16 (b) illustrates the arrangement of video data in the YCbCr4:2:0 format with a 10-bit component width in an AVP, Figure 17 (c) illustrates the arrangement of video data in the YCbCr4:2:0 format with a 12-bit component width in an AVP, and Figure 17 (d) illustrates the arrangement of video data in the YCbCr4:2:0 format with a 16-bit component width in an AVP.

[0155] Figure 18 illustrates the arrangement of ARGB format video data in an AVP. Figure 18 (a) illustrates the arrangement of ARGB format video data with a component bit width of 8 bits in an AVP, Figure 18 (b) illustrates the arrangement of ARGB format video data with a component bit width of 10 bits in an AVP, Figure 18 (c) illustrates the arrangement of ARGB format video data with a component bit width of 12 bits in an AVP, and Figure 18 (d) illustrates the arrangement of ARGB format video data with a component bit width of 16 bits in an AVP.

[0156] Figure 19 illustrates the arrangement of video data in the Y-only format in an AVP. Figure 19 (a) illustrates the arrangement of video data in the Y-only format with an 8-bit component width in an AVP, Figure 19 (b) illustrates the arrangement of video data in the Y-only format with a 10-bit component width in an AVP, Figure 19 (c) illustrates the arrangement of video data in the Y-only format with a 12-bit component width in an AVP, and Figure 19 (d) illustrates the arrangement of video data in the Y-only format with a 16-bit component width in an AVP.

[0157] Figure 20 illustrates the arrangement of RAW format video data in an AVP. Figure 20 (a) illustrates the arrangement of RAW format video data with an 8-bit bit width in an AVP, Figure 20 (b) illustrates the arrangement of RAW format video data with a 10-bit bit width in an AVP, Figure 20 (c) illustrates the arrangement of RAW format video data with a 12-bit bit width in an AVP, and Figure 20 (d) illustrates the arrangement of RAW format video data with a 16-bit bit width in an AVP.

[0158] It should be noted that, based on the arrangement of the video data described in the above embodiment, if the video data at the end of a row of the above video data cannot fill 32 bits, then data 0 is filled after the video data until 32 bits are filled. It can also be described as: when the valid video data at the end of the row cannot fill a symbol (1 symbol = 4 bytes, i.e. 32 bits), the audio and video sending adapter of the source device should fill the data "0" for symbol structure integer, that is, fill the data "0" after the valid video data until the last symbol containing valid video data is filled. The filling rules are as follows:

[0159] (1) Filling can only be done at the end of the row;

[0160] (2) Fill with data "0";

[0161] (3) The length of the padded data should be less than the length of one symbol (i.e. 32 bits), and there should not be a situation where the last symbol contains only padded data, i.e. the maximum padded data is 31 bits.

[0162] In an embodiment of the present application, pixels are arranged continuously within each row of video data, and data padding is performed only when the video data at the end of the row is less than 32 bits. The amount of padding is less than 32 bits, and the amount of padding is small. Compared with the case where more data "0" is filled, the encapsulation method of the video data in the embodiment of the present application can save bandwidth, thereby improving bandwidth utilization.

[0163] The following is an example of data padding calculation. Assuming the video data is in RGB format, the component bit width is 10 bits, and the resolution is 1366×768, the total amount of valid video data in one line in a video frame corresponding to the video data is: 1366×30=40980 bits

[0164] The above definition of a symbol size is 32 bits. After the data is arranged, the total number of symbols corresponding to the video data is: 40980 bits ÷ 32 bits = 1280.625

[0165] Round up the above result, so there are 1281 symbols in total. Then the data padding of the last symbol in this row is: 1281×32bit-40980bit=12bit

[0166] In this example, 12 bits of data 0 need to be filled.

[0167] According to the description of the above embodiment, the source device assembles VPB, HBP, AVP and other messages into a video stream and sends the video stream to the host device. Optionally, in the embodiment of the present application, the source device may carry an identifier indicating the video data format in the field of the VBP message, such as the PixelFormat field in Table 3 below; or it may carry an identifier of the number of valid pixels in a line of video data in the field of the VBP message to indicate the valid pixels in a line of video data to the host device. For example, the HActivePixels field in Table 3 below; the field of the VBP message also includes an identifier indicating the bit width of the pixel component, such as the BitsPerComp field in Table 3 below.

[0168] In the embodiment of the present application, the vertical blanking message VBP includes a 4-byte tunnel message header and a 28-byte message payload, totaling 32 bytes. The message payload of the VBP includes video frame control information (VFC) of a video frame.

[0169] For example, the tunnel header occupies 4 bytes, the message payload occupies 28 bytes, and a single VBP message occupies a total of 32 bytes. The tunnel header and message payload of a VBP message are illustrated below in conjunction with Tables 2 and 3. Table 3 illustrates some fields of the VBP message payload.

[0170] Table 2: Tunnel header of VBP message

[0171] Among them, ShuttleID is used to identify a data transmission channel on the link (Link), and the value ranges from 0 to 127. For audio and video adapters (audio and video transmitting adapters or audio and video receiving adapters), this field is the adapter ID set at the factory.

[0172] Table 3: Message payload of VBP message

[0173] Optionally, in an embodiment of the present application, the source device may further send a vertical blanking message VBP to the host device, wherein the value of the field for the number of horizontal valid pixels included in the VBP is a first value, and the first value is used to indicate the number of pixels in each row of the video data, that is, the number of valid pixels. In this way, the host device can obtain the number of valid pixels in the video data, and thus can also obtain the number of filled data (that is, the number of invalid data).

[0174] S502: The source device transmits at least one AVP to the sink device. Correspondingly, the sink device receives the at least one AVP sent by the source device.

[0175] After the source device arranges the pixel data in sequence, it generates at least one Active Video Message (AVP), and then sends the generated at least one Active Video Message to the sink device in sequence. The sink device receives the at least one AVP from the source device via the unified multimedia internet network.

[0176] S503: The sink device parses at least one AVP to obtain video data.

[0177] In an embodiment of the present application, the sink device also receives the VBP sent by the source device, and then the sink device can parse the AVP message through the relevant information in the video frame control information VFC in the VBP (i.e., the message payload of the VBP), such as PixelFormat (data arrangement format), BitsPerComp (component bit width), HActivePixels (horizontal number of effective pixels), and other fields.

[0178] The sink device discards the data padded in at least one AVP based on the value of the horizontal active pixel count field included in the VBP to obtain video data. Specifically, the sink device identifies the number of active pixels in a row of the video frame based on HActivePixels, retains the active pixels, discards the data padded at the end of the row, recovers the video data based on the active pixels, and restores the video. Optionally, after performing relevant post-processing on the video, the sink device displays the video.

[0179] Exemplarily, if the number of horizontal active pixels indicated by HActivePixels is M, the first M pixels in a row are active pixels, and the remaining data at the end of the row are padding data (ie, invalid data).

[0180] To sum up, in the video transmission method provided in the embodiment of the present application, the source device can encapsulate the video data into an AVP message and transmit it to the host device. Since the pixels in each row of the video data are arranged continuously without gaps, when the video data is encapsulated, there is no need to fill data between the pixels in the row. Compared with the method that requires a large amount of filling data, the present application can reduce the amount of invalid data filling, and therefore can save the bandwidth consumed during the video data transmission process, thereby improving bandwidth utilization.

[0181] It is understood that in order to implement the functions in the above embodiments, the source device and the sink device include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily appreciate that, in combination with the units and method steps of the various examples described in the embodiments disclosed in this application, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in hardware or in a manner driven by computer software depends on the specific application scenario and design constraints of the technical solution.

[0182] The video transmission method provided according to this embodiment is described in detail above with reference to FIG. 1 to FIG. 20 . The video processing device provided according to this embodiment will be described below with reference to FIG. 21 .

[0183] Figure 21 is a structural diagram of a video processing device provided in an embodiment of the present application. The video device can be used to implement the functions of any one of the devices in the above method embodiments, and therefore can also achieve the beneficial effects possessed by the above method embodiments. In this embodiment, the video processing device can be a set-top box 110, a smart TV 120, or any display device as shown in Figure 1, or a source device 210 or a sink device 220 as shown in Figure 2, or a source device or sink device provided in subsequent embodiments. It should be understood that the video processing device can also be a module (such as a chip) applied to any of the aforementioned devices.

[0184] As shown in Figure 21, the video processing device includes a transceiver module 2101 and a processing module 2102. Transceiver module 2101 and processing module 2102 can collaboratively implement the various steps in the aforementioned method embodiment. A more detailed description of transceiver module 2101 and processing module 2102 can be directly obtained by referring to the relevant description of the device in the method embodiment shown in the aforementioned figures, and is not repeated here.

[0185] When a video processing device implements any of the video transmission methods shown in the aforementioned figures through software, the video processing device and its various units may also be software modules. The aforementioned video transmission method is implemented by a processor calling the software module. The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0186] It can be understood that the video processing device shown in Figure 21 is only an example provided in this embodiment. Depending on the different video transmission processes, the video processing device may include more or fewer units, and this application does not limit this.

[0187] When the video processing device is implemented via hardware, the hardware may be implemented via a processor or a chip system. The chip system includes one or more chips, each of which includes an interface circuit and a control circuit. The interface circuit is used to receive data from other devices outside the chip and transmit it to the control circuit, or to send data from the control circuit to other devices outside the chip. The control circuit and the interface circuit are used to implement any possible implementation method of the above embodiments through logic circuits or execution code instructions. The beneficial effects can be found in the description of any aspect of the above embodiments and will not be repeated here.

[0188] It is understood that the processor in the embodiments of the present application may be a CPU, or other general-purpose processor, digital signal processor (DSP), ASIC, FPGA or other programmable logic device, transistor logic device, hardware component or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0189] The video processing apparatus shown in FIG21 can also be implemented by a video processing device. FIG22 is a schematic structural diagram of the video processing device provided in this application. The video processing device includes: a memory 2201 and at least one processor 2202. The processor 2202 can implement the video transmission method provided in the above embodiment. The memory 2201 is used to store software instructions corresponding to the above video transmission method.

[0190] As an optional implementation, in hardware implementation, the video processing device may refer to a chip or chip system encapsulated with one or more processors 2202. For example, when the video processing device is used to implement the method steps in the above embodiment, the processor 2202 included in the video processing device executes the steps of the source device in the above method and its possible sub-steps. In an optional scenario, the video processing device may also include a communication interface 2203, which can be used to send and receive data. For example, the communication interface 2203 is used to receive audio and video data, or to send audio and video streams, etc.; the communication interface 2203 can be implemented by the interface circuit included in the video processing device. Therefore, in some examples, the communication interface 2203 can also be called a transceiver of the video processing device. In this embodiment, the communication interface 2203 supports the use of a unified multimedia internet network.

[0191] In an embodiment of the present application, the communication interface 2203, the processor 2202, and the memory 2201 may be connected via a bus 2204, which may be divided into an address bus, a data bus, a control bus, etc. The bus 2204 may be a Peripheral Component Interconnect Express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), or other types of buses.

[0192] It is worth noting that the video processing device can also perform the functions of the video processing apparatus shown in FIG21 , which will not be described in detail here.

[0193] The video processing device provided in this embodiment may be the set-top box 110, smart TV 120, source device 210, or other devices with video processing capabilities, and this application is not limited thereto. For example, when the aforementioned display device also has a video processing function, the video processing device may refer to any of the aforementioned display devices.

[0194] In addition, the video processing device shown in FIG21 can also be implemented by a display device. When the video processing device is implemented by a display device, this embodiment provides a possible example, as shown in FIG23, which is a schematic diagram of the structure of the display device provided by the present application. The display device includes: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a unified multimedia interconnection interface 231, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a sensor module 280, a button 290, an indicator 292, a camera 293, a display screen 294, a subscriber identification module (SIM) card interface 1-N 295, etc.

[0195] Among them, the above-mentioned sensor module 280 may include sensors such as pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor and bone conduction sensor.

[0196] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the display device. In other embodiments, the display device may include more or fewer components than shown, or may combine or separate certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0197] The processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0198] The controller can be the nerve center and command center of the display device. The controller can generate operation control signals based on instruction opcodes and timing signals to complete the control of instruction fetching and execution.

[0199] Processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 210 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 210. If processor 210 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 210 latency, and thus improves system efficiency.

[0200] In some embodiments, the processor 210 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a USB interface, a unified multimedia interconnect interface, etc.

[0201] It is understood that the interface connection relationship between the modules illustrated in this embodiment is only for illustrative purposes and does not constitute a structural limitation on the display device. In other embodiments, the display device may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0202] The wireless communication function of the display device can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, a modem processor, and a baseband processor. In some embodiments, antenna 1 of the display device is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, so that the display device can communicate with the network and other devices through wireless communication technology.

[0203] The wired communication function of the display device can be implemented through the USB interface 230 or the unified multimedia interconnection interface 231. For example, the display device receives or sends video streams and AVP messages through the bus connected to the unified multimedia interconnection interface 231.

[0204] The display device implements display functions through a GPU, display screen 294, and an application processor. The GPU is a microprocessor for image processing that connects display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 210 may include one or more GPUs that execute program instructions to generate or modify display information.

[0205] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel.

[0206] The display device can implement a camera function through an ISP, a camera 293, a video codec, a GPU, a display screen 294, and an application processor. The ISP is responsible for processing data fed back by the camera 293. The camera 293 is responsible for capturing still images or videos. In some embodiments, the display device may include one or N cameras 293, where N is a positive integer greater than one.

[0207] In this embodiment, the above display screen 294, video codec, GPU, display screen 294 and application processor can also be collectively referred to as a display unit of the display device 32, which is used to process and display the received video stream.

[0208] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the display device. The external memory card communicates with the processor 210 via the external memory interface 220 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0209] The internal memory 221 can be used to store computer executable program code, which includes instructions. The processor 210 executes various functional applications and data processing of the display device by running the instructions stored in the internal memory 221. For example, in an embodiment of the present application, the processor 210 can execute instructions stored in the internal memory 221, and the internal memory 221 can include a program storage area and a data storage area.

[0210] The program storage area may store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the display device (such as audio and video data, a phone book, etc.). In addition, the internal memory 221 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, a flash memory device, or a universal flash storage (UFS).

[0211] The display device can implement audio functions such as music playback and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, and the application processor.

[0212] Buttons 290 include a power button, a volume button, and the like. Buttons 290 may be mechanical buttons or touch buttons. Indicator 292 may be an indicator light that can indicate charging status, battery level changes, messages, missed calls, notifications, and the like.

[0213] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using a software program, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions in accordance with the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a magnetic disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state drive (SSD)).

[0214] Through the description of the above embodiments, those skilled in the art will clearly understand that for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0215] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0216] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0217] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0218] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk.

[0219] The above is only a specific embodiment of the present application, but the scope of protection of this application is not limited to this. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A video transmission method, characterized in that: Applied to a source device or a chip in the source device, the method includes: Encapsulate the video data into at least one valid video packet AVP; wherein pixels in each row of the video data are arranged continuously, and each AVP is encapsulated as 32 bits per row; The at least one AVP is transmitted to a sink device.

2. The method according to claim 1, characterized in that If the video data at the end of a line of the video data cannot fill up 32 bits, data 0 is padded after the video data until 32 bits are filled up.

3. The method according to claim 1 or 2, characterized in that The format of the video data is RGB format. In each row of the video data, the arrangement order of pixel components of each pixel is R component, G component, and B component.

4. The method according to claim 1 or 2, characterized in that The format of the video data is YCbCr 4:4:4 format. In each row of the video data, the arrangement order of pixel components of each pixel is Cr component, Y component, and Cb component.

5. The method according to claim 1 or 2, characterized in that The format of the video data is YCbCr4:2:2 format. In each row of the video data, the arrangement order of pixel components of every two pixels is Y component, Cb component, Y component, Cr component.

6. The method according to claim 1 or 2, characterized in that The format of the video data is YCbCr4:2:0 format. In the even rows of the video data, the arrangement order of the pixel components of each pixel is Y component, Y component, Cb component; in the odd rows of the video data, the arrangement order of the pixel components of each pixel is Y component, Y component, Cr component.

7. The method according to claim 1 or 2, characterized in that The format of the video data is ARGB format. In each row of the video data, the arrangement order of pixel components of each pixel is R component, G component, B component, and A component.

8. The method according to any one of claims 1 to 7, characterized in that The component bit width of the video data is any one of the following: 8 bits, 10 bits, 12 bits or 16 bits.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: A vertical blanking message VBP is sent to the sink device, where the value of a field for the number of horizontal valid pixels included in the VBP is a first value, where the first value is used to indicate the number of pixels in each row of the video data.

10. A video transmission method, characterized in that: The method applied to a sink device or a chip in the sink device includes: Receive at least one valid video packet AVP sent by a source device; wherein the at least one AVP is a packet obtained by encapsulating video data; pixels in each row of the video data are arranged continuously, and each AVP is encapsulated according to 32 bits per row; Parse the at least one AVP to obtain the video data.

11. The method according to claim 10, characterized in that In a line of the video data, data 0 at the end of the line is padding data, and the amount of data 0 is less than 32 bits.

12. The method according to claim 10 or 11, characterized in that The method further comprises: receiving a vertical blanking message VBP sent by the source device, wherein the value of a field for the number of horizontal valid pixels included in the VBP is a first value, and the first value is used to indicate the number of pixels in each row of the video data; The parsing the at least one AVP to obtain the video data includes: According to the value of the field of the number of horizontal valid pixels included in the VBP, the data filled in the at least one AVP is discarded to obtain the video data.

13. The method according to any one of claims 10 to 12, characterized in that The format of the video data is RGB format. In each row of the video data, the arrangement order of pixel components of each pixel is R component, G component, and B component.

14. The method according to any one of claims 10 to 12, characterized in that The format of the video data is YCbCr 4:4:4 format. In each row of the video data, the arrangement order of pixel components of each pixel is Cr component, Y component, and Cb component.

15. The method according to any one of claims 10 to 12, characterized in that The format of the video data is YCbCr4:2:2 format. In each row of the video data, the arrangement order of pixel components of every two pixels is Y component, Cb component, Y component, Cr component.

16. The method according to any one of claims 10 to 12, characterized in that The format of the video data is YCbCr4:2:0 format. In the even rows of the video data, the arrangement order of the pixel components of each pixel is Y component, Y component, Cb component; in the odd rows of the video data, the arrangement order of the pixel components of each pixel is Y component, Y component, Cr component.

17. The method according to any one of claims 10 to 12, characterized in that The format of the video data is ARGB format. In each row of the video data, the arrangement order of pixel components of each pixel is R component, G component, B component, and A component.

18. The method according to any one of claims 10 to 17, characterized in that The color depth of the video data is any one of the following values: 8, 10, 12 or 16.

19. A video processing device, characterized in that: include: memories, transceivers, and processors; The processor is used to encapsulate the video data into at least one valid video packet AVP; The memory is configured to cache the at least one AVP; The transceiver is configured to transmit the at least one AVP; The memory, the transceiver, and the processor are configured to collaboratively execute the method according to any one of claims 1 to 9.

20. A sink device, characterized in that: include: transceivers and processors; The transceiver is used to receive at least one valid video packet AVP; The processor is configured to parse the at least one AVP; The transceiver and the processor are configured to collaboratively execute the method according to any one of claims 10 to 16.

21. A computer-readable storage medium, characterized in that The computer-readable storage medium stores the valid video packet AVP obtained by the method described in any one of claims 1 to 9 or any one of claims 10 to 18.

Images