Video metadata transmission method and device

Abstract

Disclosed are a video metadata transmission method and a device, relating to the technical field of multimedia. A video metadata packet comprises an organization to which a standard used by the packet belongs and types of metadata under the standard, so that a source device can encapsulate different types of metadata by using an encapsulating method, and the standard used by the video metadata packet and the types of metadata are marked in the video metadata packet, preventing the source device from consuming more computing power to realize encapsulation of the video metadata packet, thus facilitating improvement of the efficiency of transmission, among different devices, of a video carrying the video metadata packet.

Description

Video metadata data transmission method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 17, 2023, with application number 202311348266.0 and application name “A method and device for transmitting video metadata data”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of multimedia technology, and in particular to a method and device for transmitting video metadata. Background Art

[0003] With the development of big data, artificial intelligence (AI), and cloud computing technologies, various types of multimedia data are constantly emerging. For example, the wired transmission of video data between different devices can utilize the display port (DP) bus to transmit video data. In the DP protocol, when the display device playing the video supports high-dynamic range (HDR) display, the source device also sends video metadata to the display device to optimize the video display quality. However, video metadata includes both dynamic and static metadata. During the transmission of video metadata between the source and display devices, the source device encapsulates different types of video metadata into metadata packets in different formats and then sends these encapsulated metadata packets to the display device. The display device then selects different parsing methods based on the metadata packet format to optimize the video being played on the display device. Because the source device must select different encapsulation methods for different types of video metadata, this consumes computing power within the source device, potentially impacting the source device's ability to receive or push video streams, and reducing video metadata transmission efficiency.

[0004] Summary of the Invention

[0005] The present application provides a method and device for transmitting video metadata, which solves the problem that the transmission process of the video stream between the source device and the sink device (such as a display device) is affected and the transmission efficiency of the video metadata is reduced.

[0006] This application adopts the following technical solution.

[0007] In a first aspect, the present application provides a method for transmitting video metadata. The method includes encapsulating one or more video metadata messages for a video and sending the one or more video metadata messages. The video metadata message includes the organization to which the standard used by the video metadata message belongs, and the metadata type under the standard.

[0008] The video metadata message generated by this application includes the organization to which the standard adopted by the video metadata message belongs and the metadata type under the standard, so that the source device can use an encapsulation method to encapsulate different types of metadata, and the video metadata message is marked with the standard and metadata type adopted by the video metadata message, thereby avoiding the source device from consuming more computing power to realize the encapsulation of the video metadata message, which is conducive to improving the transmission efficiency of the video metadata message between different devices. Furthermore, the host device (such as a display device) can parse the video metadata message based on the aforementioned standard and metadata type, so as to process and display the video in the video stream according to the video metadata carried by the video metadata message, so as to improve the display effect of the video in the display device and improve the user's video viewing experience.

[0009] In combination with the video metadata data transmission method provided in the first aspect, in an optional implementation method, taking the video metadata data transmission method being executed by a source device as an example, the source device is connected to a display device (host device) via a first bus, and the video metadata data transmission method includes: the source device reads the capability description information of the display device via the first bus, and the capability description information is used to indicate the HDR display capability of the display device. If the capability description information indicates that the display device supports displaying Jingcai HDR video, the source device generates a video metadata message for the video, and the video metadata message includes: the organization to which the standard adopted by the video metadata message belongs, and the metadata type under the standard. And, the source device sends a video stream carrying the video metadata message to the display device via the first bus.

[0010] In the present application, the source device can directly read the capability description information of the display device through the bus, thereby knowing the HDR display capability of the display device, and only when the display device supports displaying the Jingcai HDR video, the source device sends the video stream to the display device. In this way, the invalid communication process between devices caused by the display device not supporting the processing of Jingcai HDR video and the problem of bus bandwidth resources between devices being occupied are avoided. In addition, compared with the conventional technology that requires different encapsulation methods to encapsulate metadata packets, the video metadata message in the video stream generated by the present application includes the organization to which the standard adopted by the video metadata message belongs and the metadata type under the standard, so that the source device can use one encapsulation method to encapsulate different types of metadata, and the video metadata message is marked with the standard and metadata type adopted by the video metadata message, thereby avoiding the source device from consuming more computing power to realize the encapsulation of the video metadata message, which is conducive to improving the transmission efficiency of the video carrying the video metadata message between different devices.

[0011] In combination with the video metadata transmission method provided in the first aspect, in an optional implementation method, the video metadata transmission method provided in the present application also includes: the source device determines the HDR working mode of the host device (display device) according to the capability description information. The HDR working mode is a first mode or a second mode, and the first mode is characterized by: the image rendering parameters of the video are provided by the host device (display device), also known as the monitor adaptation mode; the second mode is characterized by: the image rendering parameters of the video are provided by the source device, also known as the receiving end adaptation mode. In the present application, when the host device (display device) works in different HDR working modes, different devices are used to provide image rendering parameters, that is, the source device can determine the generation end of the image rendering parameters according to the HDR working mode of the display device, so as to reasonably allocate the processing resources of the source device and the display device, which is conducive to reducing the rendering delay during video transmission and video display, and improving the image rendering efficiency of the video, improving the display effect of the video in the display device, and improving the user's experience of watching the video.

[0012] In conjunction with the video metadata transmission method provided in the first aspect, in an optional implementation, if the HDR operating mode of the sink device is the monitor adaptation mode, the video metadata message includes: first video metadata and a metadata type whose value is a first value, the first value indicating that the first video metadata is static metadata, and the first video metadata is used to indicate: the association between the electrical parameters provided by the sink device for the video and the display brightness of the video. Exemplarily, if the HDR operating mode of the sink device (display device) is the first mode (monitor adaptation mode), the source device generates a video metadata message for the video, including: the source device generates the first video metadata of the video, and determines that the value of the metadata type is the first value; the first value indicates that the first video metadata is static metadata, and the first video metadata is used to indicate: the association between the electrical parameters provided by the sink device (display device) for the video and the display brightness of the video. And, the source device generates the video metadata message based on the first value and the first video metadata. In some examples, the association between the electrical parameters provided by the sink device (display device) for the video and the display brightness of the video can also be referred to as: the image transfer characteristic (transfer character) of the video.

[0013] In combination with the video metadata transmission method provided in the first aspect, in an optional implementation, if the HDR operating mode of the host device is the receiving end adaptation mode, the video metadata message includes: second video metadata and a metadata type with a second value as the value, the second value indicates that the second video metadata is dynamic metadata, and the second video metadata is used to indicate: the host device displays multiple sets of image rendering parameters for multiple frames of images in the video, and one frame of image corresponds to one set of image rendering parameters. Exemplarily, if the HDR operating mode of the host device (display device) is the second mode (receiving end adaptation mode), the source device generates a video metadata message for the video, including: the source device generates the second video metadata for the video, and determines that the value of the metadata type is the second value; the second value indicates that the second video metadata is dynamic metadata, and the second video metadata is used to indicate: the host device (display device) displays multiple sets of image rendering parameters for multiple frames of images in the video, and one frame of image corresponds to one set of image rendering parameters. And, the source device generates a video metadata message based on the second value and the second video metadata.

[0014] Exemplarily, the first value is 0x0 and the second value is 0x1.

[0015] In the above implementation, the source device determines the metadata type of the video metadata according to the HDR working mode of the display device, so that the video metadata message of the metadata type can adapt to the HDR working mode of the display device, such as static metadata adapting to the first mode (monitor adaptation mode) and dynamic metadata adapting to the second mode (receiving end adaptation mode). This avoids the problem that the type of video metadata is incompatible with the HDR working mode of the display device, resulting in the display device being unable to use the video metadata message to improve the display quality of the video, which is conducive to improving the display effect of the video on the display device and enhancing the user's viewing experience.

[0016] In a second aspect, the present application provides another method for transmitting video metadata. The method includes receiving one or more video metadata messages from a video, and processing and displaying the video based on the one or more video metadata messages. The video metadata message includes the organization to which the standard used by the video metadata message belongs, and the metadata type under the standard.

[0017] The sink device (display device) can parse the video metadata message based on the standard and metadata type adopted by the video metadata message, so as to process and display the video in the video stream according to the video metadata carried by the video metadata message, thereby improving the display effect of the video in the sink device and enhancing the user's video viewing experience.

[0018] For example, the video metadata data transmission method is described using a display device as an example. The display device is connected to a source device via a first bus. The display device provides capability description information indicating the display device's HDR display capabilities. The video metadata data transmission method includes: the display device receiving a video stream sent by the source device via the first bus. The video stream carries a video metadata message. The video metadata message includes the organization to which the standard used by the video metadata message belongs and the metadata type under the standard. Furthermore, the display device processes and displays the video stream based on the video metadata message carried by the video stream.

[0019] In the present application, the display device is provided with: capability description information indicating the HDR display capability of the display device. The source device can directly read the capability description information of the display device through the bus, thereby knowing the HDR display capability of the display device, and only when the display device supports displaying the Jingcai HDR video, the source device sends the video stream to the display device. In this way, the invalid communication process between devices caused by the display device not supporting the processing of Jingcai HDR video and the problem of bus bandwidth resources between devices being occupied are avoided. In the process of the display device parsing the video metadata message, since the video metadata message in the video stream of the present application includes the organization to which the standard adopted by the video metadata message belongs and the metadata type under the standard, the display device can adopt a parsing method to parse different types of metadata, and the video metadata message is marked with the standard and metadata type adopted by the video metadata message, thereby avoiding the display device from consuming more computing power to implement the parsing of the video metadata message.

[0020] In conjunction with the video metadata data transmission method provided in the second aspect, in an optional implementation, the video metadata is used to indicate the relationship between the electrical parameters provided by the display device for the video and the display brightness of the video. The display device processes and displays the video stream based on the video metadata message carried by the video stream, including: the display device generating image rendering parameters for the video based on the video metadata and device information of the display device; and the display device processing the video stream based on the image rendering parameters to obtain a rendered video, and displaying the rendered video.

[0021] In combination with the video metadata transmission method provided in the first aspect and the second aspect, in an optional implementation, the video metadata message includes a message header, and the message header carries the organization and metadata type.

[0022] In conjunction with the video metadata transmission methods provided in the first and second aspects, in one optional implementation, the video metadata message includes a data portion, which is used to carry the video metadata. The data portion has a size of N bytes, where N is any value between 1 and 32. In some possible examples, this data portion is also referred to as the payload of the video metadata message.

[0023] In conjunction with the video metadata transmission methods provided in the first and second aspects, in an optional implementation, the message start flag (S flag) of the first video metadata message among the multiple video metadata messages is 1, and the message end flag (E flag) is 0. Furthermore, the message start flag of the last video metadata message among the multiple video metadata messages is 0, and the message end flag is 1.

[0024] Exemplarily, a video stream carries multiple video metadata messages. The first video metadata message in the multiple video metadata messages carries first S / E standard information, and the first S / E standard information is: S=1, E=0. The last video metadata message in the multiple video metadata messages carries second S / E standard information, and the second S / E standard information is: S=0, E=1. In the present application, different S / E standard information is used to mark the start position (S) and end position (E) in multiple video metadata messages, thereby avoiding the situation where the display device cannot recognize the start and end of the video metadata message, resulting in errors in the image rendering of the video. Subsequently, the display device can effectively identify the video metadata message in the video stream, so as to perform image rendering on the video according to the video metadata carried by the video metadata message, thereby improving the display effect of the video in the display device and the user's experience of watching the colorful HDR video.

[0025] Optionally, the message start flag of the middle video metadata message among the multiple video metadata messages is 0, and the message end flag is 0.

[0026] Exemplarily, the above-mentioned multiple video metadata messages also include: an intermediate video metadata message located between the first video metadata message and the last video metadata message, and the intermediate video metadata message carries third S / E standard information, and the third S / E standard information is: S=0, E=0.

[0027] In conjunction with the video metadata transmission method provided in the first and second aspects, in an optional implementation, each of the plurality of video metadata messages carries an index. The index is used to indicate the position or order of each video metadata message in the plurality of video metadata messages.

[0028] In combination with the video metadata transmission method provided in the first aspect and the second aspect, in an optional implementation, the first bus provided in this application is a unified multimedia interconnect bus.

[0029] In combination with the video metadata transmission method provided in the first aspect and the second aspect, in an optional implementation, the above-mentioned video metadata message includes: a cyclic redundancy check (CRC) set at the end, which is used to verify the accuracy of the message header and data part in the video metadata message.

[0030] In a third aspect, the present application provides a video device, which includes a module for executing the method of any one of the implementations of the first aspect or the second aspect.

[0031] Exemplarily, the video device is a video metadata data transmission device, comprising modules for implementing the method of any one of the implementation methods of the first aspect, such as an encapsulation module and a transceiver module. The encapsulation module is configured to encapsulate one or more video metadata messages for a video, wherein the video metadata message includes: the organization to which the standard used by the video metadata message belongs, and the metadata type under the standard. The transceiver module is configured to send the one or more video metadata messages.

[0032] In another exemplary embodiment, the video device is a video metadata data transmission device, comprising modules for implementing the method of any implementation method of the second aspect, such as a transceiver module and a display module. The transceiver module is configured to receive one or more video metadata messages for a video, wherein the video metadata message includes: the organization to which the standard used by the video metadata message belongs, and the metadata type under the standard. The display module is configured to process and display the video according to the one or more video metadata messages.

[0033] In a fourth aspect, the present application provides a video processing device. The video processing device includes a memory, a transceiver, and a processor. The processor is configured to generate a video stream to be transmitted, the memory is configured to buffer the video stream, and the transceiver is configured to transmit the video stream. The memory, the transceiver, and the processor are configured to collaboratively perform the method of any implementation of the first aspect.

[0034] In a fifth aspect, the present application provides a display device comprising: a transceiver and a display unit. The transceiver is configured to receive a video stream carrying a video metadata message, wherein the video metadata message includes the organization to which the standard adopted by the video metadata message belongs and the metadata type under the standard. The display unit is configured to process and display the video stream based on the video metadata message carried by the video stream. The transceiver and the display unit are configured to collaboratively perform the method of any implementation method of the second aspect.

[0035] In a sixth aspect, the present application provides a video transmission system. This audio transmission system includes the source device provided in the fourth aspect and the display device provided in the fifth aspect. The source device and the display device are connected via a bus. The source device can be used to implement the functions of the source device in the first or second aspect, and the display device can be used to implement the functions of the display device in the first or second aspect. Therefore, this video transmission system can also achieve the beneficial effects of the video metadata data transmission methods described in the first and second aspects, which will not be elaborated here.

[0036] In a seventh aspect, the present application provides a computer-readable storage medium, which includes computer software instructions.

[0037] When the computer software instructions are executed in a computing device, the computing device is caused to perform the operating steps of the method described in the first aspect or any possible implementation of the first aspect, as well as the operating steps of the method described in the second aspect or any possible implementation of the second aspect. For example, the computing device is the aforementioned source device or display device.

[0038] In an eighth aspect, the present application provides a computer program product. When the computer program product is executed on a computer, the computer program product causes the computing device to perform the steps of the method described in the first aspect or any possible implementation of the first aspect, as well as the steps of the method described in the second aspect or any possible implementation of the second aspect. For example, the computer is the aforementioned source device or display device.

[0039] In a ninth aspect, the present application provides a video playback method. The method includes: a source device encapsulating one or more video metadata messages of a video, where the video metadata message includes: the organization to which the standard adopted by the video metadata message belongs, and the metadata type under the standard; the source device sending the one or more video metadata messages to a sink device; and the sink device processing and displaying the video based on the one or more video metadata messages.

[0040] Regarding the beneficial effects of the third to ninth aspects, reference may be made to the description of any implementation in the first or second aspects, and no further details will be given here. Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] FIG1 is a schematic diagram of a video transmission system provided by the present application;

[0042] FIG2 is a schematic diagram of an audio and video encoding and decoding system provided by the present application;

[0043] FIG3 is a flowchart of a method for transmitting video metadata provided by the present application;

[0044] FIG4 is a schematic diagram of the structure of a video metadata message provided by this application;

[0045] FIG5 is a second flow chart of a method for transmitting video metadata provided by the present application;

[0046] FIG6 is a third flow chart of a method for transmitting video metadata provided by the present application;

[0047] FIG7 is a structural diagram of a video metadata message provided by the present application;

[0048] FIG8 is a second structural diagram of a video metadata message provided by this application;

[0049] FIG9 is a third structural diagram of a video metadata message provided by this application;

[0050] FIG10 is a schematic structural diagram of a video device provided by the present application;

[0051] FIG11 is a schematic diagram of the structure of a video processing device provided by this application;

[0052] FIG12 is a schematic structural diagram of the display device provided in this application. DETAILED DESCRIPTION

[0053] This application provides a method for transmitting video metadata. The source device can directly read the capability description information of the display device through the bus, thereby knowing the HDR display capability of the display device. The source device will send the video stream to the display device only when the display device supports displaying the colorful HDR video. In this way, the problem of invalid communication between devices caused by the display device not supporting the processing of colorful HDR video and the occupation of bus bandwidth resources between devices is avoided.

[0054] Specifically, compared to conventional technologies that require different encapsulation methods to encapsulate metadata packets, the video metadata message in the video stream generated by this application includes the organization to which the standard adopted by the video metadata message belongs and the metadata type under the standard, so that the source device can use one encapsulation method to encapsulate different types of metadata, and the video metadata message is marked with the standard and metadata type adopted by the video metadata message, thereby avoiding the source device from consuming more computing power to realize the encapsulation of the video metadata message, which is conducive to improving the transmission efficiency of the video carrying the video metadata message between different devices. Furthermore, the display device can parse the video metadata message based on the aforementioned standard and metadata type, so as to process and display the video in the video stream according to the video metadata carried by the video metadata message, so as to improve the display effect of the video in the display device and enhance the user's viewing experience.

[0055] The technical solutions involved in this application may be applicable not only to current audio and video transmission technologies or audio and video standards, but also to future audio and video transmission technologies or audio and video standards. The terms used in the implementation methods of this application are only used to explain the specific embodiments of this application and are not intended to limit this application. The following is a brief introduction to some concepts that may be involved in this application.

[0056] Video streaming refers to the transmission of video data, i.e., a video stream can be processed as a stable and continuous flow over a network. A video stream consists of multiple video frames, each of which corresponds to an image.

[0057] Video metadata: Data describing video frames. For example, this metadata contains the image rendering parameters for each frame in the video stream. When a display device parses and plays the video frames based on this metadata, the video can be displayed better on the display device.

[0058] Descriptive information packet (DIP or DIP message): used to describe different data in a data packet.

[0059] In this embodiment, video is a general term that includes a sequence of multiple consecutive frames, with one frame corresponding to one image. Video and audio are information application technology terms that refer to video, audio, or multimedia content that includes both video and audio.

[0060] In order to make the description of the following embodiments clear and concise, an introduction to related technologies is first given.

[0061] Figure 1 is a schematic diagram of a video transmission system provided by the present application. The video processing process may include but is not limited to: video acquisition, video encoding, video transmission, video decoding and playback.

[0062] The audio and video transmission system in Figure 1 includes a set-top box 110, a smart TV 120, multiple audio and video playback devices, and a server 130. Set-top box 110 accesses the operator's network via a network cable and receives audio and video streams from server 130. The network can implement audio and video transmission and may include one or more network devices, such as a router or switch. In some optional implementations, communication between set-top box 110 and server 130 may also be achieved via wireless communication, although this application is not limited thereto.

[0063] The set-top box 110 is a video and audio processing device that is used to implement functions such as receiving, processing, and pushing video streams or video and audio streams. In some possible scenarios, the set-top box 110 may also be called an Internet TV set-top box, a network high-definition player, or other similar devices. For example, the set-top box 110 may refer to a TV box provided by a network operator or a TV box purchased by a user. For the hardware implementation of the set-top box 110, reference may be made to the description of FIG. 11 below, which will not be described in detail here.

[0064] Smart TV 120 is a display device with audio and video processing capabilities that implements functions such as receiving, processing, pushing, and playing video or audio streams. In some possible scenarios, Smart TV 120 may refer to an audio and video device such as a conference tablet, smart TV, or projector, but this application is not limited to this. The hardware implementation of Smart TV 120 can be referred to in the description of Figures 11 or 12 below and will not be detailed here.

[0065] The multiple audio and video playback devices include audio and video playback devices 121 to 124. For example, these audio and video playback devices may include, but are not limited to, multimedia control platforms or other devices that support audio and video playback functions, such as virtual reality (VR) terminal devices or augmented reality (AR) terminal devices. The hardware implementation of the audio and video playback devices can be found in the description of FIG. 12 below and is not detailed here.

[0066] In this embodiment, the set-top box 110 and the smart TV 120 may be connected via a bus 125. The set-top box 110 and each audio and video playback device may also be connected via the bus 125. The smart TV 120 and each audio and video playback device may also be connected via the bus 125. Exemplarily, the bus 125 may be a data bus that supports video and audio transmission. For example, the bus 125 may be a unified multimedia interconnection bus, such as a UMI (unified multimedia interconnection, UMI) bus.

[0067] A unified multimedia interconnect bus refers to a bus connected based on the unified multimedia interconnect interface provided by the source device and the sink device. This unified multimedia interconnect bus can be used to connect a charger to charge electronic devices (such as the smart TV, set-top box, or audio and video playback device mentioned above), can also be used to transmit data between electronic devices and peripheral devices, and can also be used to connect headphones to play audio through the headphones. The above-mentioned unified multimedia interconnect interface can also be used to connect other electronic devices, such as AR devices. When the unified multimedia interconnect bus is used to implement data communication functions between devices, the unified multimedia interconnect bus can support uncompressed and compressed video transmission, as well as various advanced features such as Quick Video Transport (QVT), Auto Low Latency Mode (ALLM), and Dynamic Frame Rate Refresh (DFR). In addition, the unified multimedia interconnect bus can also support LPCM format audio and video defined by IEC60958 and various HDR protocols, such as the HDR protocol specified in T / UWA 005.1-2022, such as HDR Vivid. The unified multimedia interconnect bus also supports encryption control protection for audio and video data transmission. In some optional implementations, the bus 125 may also refer to other types of buses that can implement the functions supported by the above-mentioned unified multimedia interconnect bus.

[0068] In this embodiment, the unified multimedia interconnection interface is suitable for transmitting high-speed signals for audio and video devices, third-party protocol devices (such as USB / PCIe / Ethernet), etc. A device that uses the unified multimedia interconnection interface is called a unified multimedia interconnection device, and a system composed of multiple unified multimedia interconnection devices connected together is called a unified multimedia interconnection system. The capabilities provided by the unified multimedia interconnection interface include: ①, bidirectional transmission of audio and video, meeting the audio and video transmission requirements of devices such as TVs, PCs, and mobile phones, and supporting audio and video content transmission protection, visual lossless compression, etc. ②, third-party protocol data transmission, realizing the interaction between the unified multimedia interconnection device and the third-party protocol device, such as the supported third-party protocols include USB3, and Ethernet, PCIe, etc. will be supported in the future. ③, bidirectional power supply, meeting the power supply requirements of electronic devices whose power does not exceed a certain power value, such as the certain power value is 480W, 500W or other values.

[0069] The server 130 may be an application server or an authentication and authorization server. The server 130 may provide video services, game services, message services, music services, authentication and authorization services, and the like. In one example, the functions of multiple services may be integrated on the server 130. For example, a game service and a music service may be deployed on the server 130. In another example, the functions of some services may be integrated on the server 130. For example, some services of the game service and some services of the video service may be deployed on the server 130. The server 130 may also utilize virtualization technology to provide multiple virtual machines, which provide various services. The embodiments of the present application do not limit the deployment form of the server. The network device 131 is connected to the server 130 via a wireless or wired mode. FIG1 is merely a schematic diagram. Other devices may also be included in the network, which are not shown in FIG1.

[0070] It is understood that the above-mentioned audio and video is a general term. Audio and video include multiple video frames, each of which corresponds to a group of packets carrying the audio and video data to be parsed and played. Figure 1 is merely a schematic diagram. The audio and video transmission system may also include other devices, which are not shown in Figure 1. The embodiments of this application do not limit the number and type of each device included in this system.

[0071] Based on the audio and video transmission system shown in Figure 1, Figure 2 is a schematic diagram of an audio and video codec system provided by this application. The audio and video codec system includes a source device 210 and a sink device 220. The source device 210 establishes a communication connection with the sink device 220 via a UMI bus.

[0072] The above-mentioned source device 210 can realize the function of audio and video encoding. As shown in Figure 1, the source device 210 can be a set-top box 110 or a smart TV 120. The source device 210 can also be an audio and video control center with audio and video encoding capabilities. For example, the audio and video control center includes one or more servers.

[0073] The source device 210 may include a data source 211 , a pre-processing module 212 , an audio and video transmission adapter 213 , and a communication interface 214 .

[0074] The data source 211 may include or may be any type of electronic device for capturing audio and video, and / or any type of source audio and video generating device, such as a computer graphics processor for generating computer animation scenes or any type of device for acquiring and / or providing source audio and video, or computer-generated source audio and video. The data source 211 may be any type of memory or storage for storing the above-mentioned source audio and video. The above-mentioned source audio and video may include multiple audio and video streams or images captured by multiple audio and video capturing devices (such as cameras), such as ultra-high definition (UHD) video, high definition (HD) video, 4K video, etc.

[0075] The pre-processing module 212 is configured to receive source audio and video and pre-process the source audio and video to obtain audio and video or multiple frames of images. For example, the pre-processing performed by the pre-processing module 212 may include color format conversion (e.g., from RGB to YCbCr), octree structuring, audio and video splicing, audio track merging and deletion, or channel number adjustment.

[0076] The video and audio transmission adapter 213 is used to receive video and audio or images and encode the video and audio, images, or images to obtain coded data. In some optional situations, the coded stream (coded data) obtained by encoding can also be called a bit stream. If the coded data is obtained by encoding video and audio data, the bit stream refers to the video and audio stream.

[0077] The communication interface 214 in the source device 210 may be configured to receive encoded data (e.g., a video stream or an audio / video stream) and send the encoded data (or a version of the encoded data after any other processing) to another device such as the sink device 220 or any other device via a UMI bus for storage, display, playback, or image reconstruction.

[0078] Optionally, the source device 210 includes a bitstream buffer, which is used to store bitstreams corresponding to one or more coding units.

[0079] The sink device 220 can implement the function of video and audio decoding. As shown in FIG1 , the sink device 220 can be any one of the smart TV 120 or the video and audio playback device shown in FIG1 .

[0080] The sink device 220 may include a video and audio playback unit 221 , a post-processing module 222 , a video and audio receiving adapter 223 , and a communication interface 224 .

[0081] The communication interface 224 in the sink device 220 is configured to receive the encoded data (or a version of the encoded data after any other processing has been performed on the encoded data) from the source device 210 or any other source device such as a storage device.

[0082] Communication interface 214 and communication interface 224 may be used to communicate via a direct communication link between source device 210 and sink device 220, such as a direct wired connection, as shown in the UMI bus in FIG2 . For details about the UMI bus, please refer to the description of FIG1 , and will not be repeated here.

[0083] The communication interface 224 corresponds to the communication interface 214, for example, and can be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded data (such as video stream or audio and video stream).

[0084] Both the communication interface 224 and the communication interface 214 may be configured as a unidirectional communication interface, as indicated by the arrow pointing from the source device 210 to the corresponding UMI bus of the sink device 220 in FIG2 , or a bidirectional communication interface, and may be used to send and receive messages, etc., to establish a connection, confirm and exchange any other information related to the communication link or data transmission, such as encoded compressed data transmission, etc.

[0085] The video and audio receiving adapter 223 is used to receive encoded data and decode the encoded data to obtain decoded data (video or video and audio, etc.).

[0086] The post-processing module 222 is used to post-process the decoded data to obtain post-processed data (such as an image to be displayed or audio and video to be played). The post-processing performed by the post-processing module 222 may include, for example, color format conversion (such as from YCbCr to RGB), octree reconstruction, audio and video splitting and fusion, or any other processing for generating data for output by the audio and video playback unit 221.

[0087] The audio and video playback unit 221 is used to receive post-processed data for display or playback to a user or viewer, etc. The audio and video playback unit 221 can be or include any type of display for representing the reconstructed image, such as an integrated or external display screen or display. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen. The audio and video playback unit 221 can also include one or more audio and video playback modules, each of which can refer to a speaker, a smart speaker, an amplifier, etc.

[0088] As an optional implementation, the source device 210 and the sink device 220 may transmit the encoded data via a data forwarding device. For example, the data forwarding device may be a router or a switch. It is worth noting that the data forwarding device must support the UMI interface.

[0089] The following describes in detail the implementation of the video metadata transmission method provided by the embodiment of the present application with reference to the accompanying drawings.

[0090] Here, the video metadata data transmission method of an embodiment of the present application is performed by the source device 210 and the sink device 220 shown in Figure 2 as an example for explanation. Figure 3 is a flow chart of a video metadata data transmission method provided by the present application. Among them, the source device 31 is used to implement the functions of the source device 210, and the display device 32 is used to implement the functions of the sink device 220. In this embodiment, the source device 31 can also be referred to as an audio and video sending device, an audio and video sending end, or an audio and video sending device, and the display device 32 can also be referred to as an audio and video receiving device, an audio and video playback device, or an audio and video receiving device. In this embodiment, the source device 31 and the display device 32 are connected via a bus 33, which can refer to a unified multimedia interconnect bus (such as a UMI bus). In some examples, the bus 33 can also be referred to as a connection bus, a first bus, or other names.

[0091] In a first possible application scenario, the source device 31 may be the set-top box 110 in Figure 1, and the display device 32 may be the smart TV 120 in Figure 1. For example, the set-top box pushes video and audio data to the smart TV.

[0092] In a second possible application scenario, the source device 31 may be the set-top box 110 in FIG1 , and the display device 32 may be any video and audio playback device in FIG1 , such as any one of the video and audio playback devices 121 to 124. For example, the set-top box pushes video and audio data to the video and audio playback device.

[0093] In a third possible application scenario, the source device 31 may be the smart TV 120 in FIG1 , and the display device 32 may be any video and audio playback device in FIG1 , such as any one of the video and audio playback devices 121 to 124. For example, the smart TV pushes video and audio data to the video and audio playback device.

[0094] The above three possible application scenarios are merely examples provided in this embodiment and should not be construed as limiting the present application. In other possible examples, the source device 31 may be any of the video and audio playback devices in FIG1 (e.g., the video and audio playback device 121 ), and the display device 32 may be another video and audio playback device different from the aforementioned video and audio playback device (e.g., the video and audio playback device 122 ).

[0095] Referring to FIG. 3 , the video metadata data transmission method provided in this embodiment includes the following steps S310 to S340 .

[0096] S310 , the source device 31 reads capability description information of the display device 32 through the bus 33 .

[0097] The capability description information is used to indicate the wide dynamic range (HDR) display capability of the display device 32. In some possible scenarios, the capability description information may also be referred to as a capability descriptor, which may include one or more bits of information.

[0098] In an optional example, the display device 32 includes a memory in which the capability description information is stored. The memory may include, but is not limited to, random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable hard disk, a CD-ROM, or any other form of storage medium known in the art.

[0099] In another optional example, the display device 32 includes a dedicated register whose state is used to indicate the capability description information. If the value indicated by the register is 1, the capability description information indicates that the display device 32 has the ability to display the colorful HDR video; if the value indicated by the register is 0, the capability description information indicates that the display device 32 does not have the ability to display the colorful HDR video.

[0100] It is worth noting that since the capability description information indicates the HDR display capability of the device, in some cases, the capability description information is also called device comprehensive capability description information, and the area in the display device that stores the DCCD information is called the DCCD area or simply DCCD, which is not limited in this application.

[0101] If the capability description information indicates that the display device 32 supports displaying colorful HDR video, the source device 31 executes S320 ; if the capability description information indicates that the display device 32 does not support displaying colorful HDR video, the source device 31 sends a regular video stream to the display device 32 .

[0102] S320: The source device 31 generates a video metadata message for the video.

[0103] The video metadata message includes: the organization to which the standard adopted by the video metadata message belongs, and the metadata type under the aforementioned standard.

[0104] The organization to which the standard adopted by the video metadata message belongs may include, but is not limited to, the following organizations: the UHD World Association (UWA), S Organization, C Organization, V Organization or other organizations.

[0105] The metadata type can be static metadata or dynamic metadata. Static metadata uses the same value at all times when the video data is used. For example, the same video metadata is used throughout a video to control the color and detail of each frame, and the video metadata does not change between frames. Dynamic metadata, on the other hand, changes as needed (possibly on a frame-by-frame basis), for example, allowing specific video metadata to be specified for each frame or scene switch. The following embodiments provide exemplary descriptions of video metadata of different metadata types, which will not be elaborated here.

[0106] The structure of the video metadata message of this embodiment is described below in conjunction with Figure 4, which is a schematic diagram of the structure of a video metadata message provided by this application. In Figure 4, the video metadata message includes a tunnel header, a message header, and a data portion.

[0107] As shown in Figure 4, the tunnel message header includes a reserved field (RSVD, 1 byte), an adapter ID (Shuttle ID, 7 bytes), an E flag (1 byte), an S flag (1 byte), an R flag (1 byte), a CP flag (1 byte), a type (type, 4 bytes), a length (length, 9 bytes), a T flag (1 byte), and an error correction code (ECC), where the ECC occupies 6 bytes.

[0108] The message header is also called the header byte (HB) portion of the video metadata message, and the HB portion includes one or more header bytes, such as HBx. The message header carries the aforementioned organization and metadata type.

[0109] The following is an exemplary description of the structure of the message header of the video metadata message in conjunction with Table 1.

[0110] Table 1

[0111] HB1 includes an index field. This index indicates the position or order of the current video metadata message within multiple generated video metadata messages. For example, the receiving end (e.g., display device 32) can use the value of this index to determine whether any video metadata messages were lost during transmission. The length field in HB3 records the valid data length of the data portion. For example, when the E flag is 1, the value of the length field in HB3 records the number of valid bytes (valid data length) carried in the video metadata message.

[0112] Continuing with Figure 4, the data portion is also referred to as the data byte (DB) portion of the video metadata message. The DB portion includes one or more data bytes, such as DBx. The data portion carries video metadata and is therefore sometimes referred to as the payload, payload area, or payload region of the video metadata message.

[0113] Exemplarily, the size of the data portion is N bytes, where N is any value between 1 and 32, such as 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, or 32 bytes. As shown in FIG4 , the data portion includes 32 bytes, such as DB0 to DB31.

[0114] In one possible scenario, if the length of the video metadata is less than or equal to 32 bytes, then one video metadata message can carry it. In this case, both the S and E flags in the message header of the video metadata message must be set to 1. At the same time, the HB3 byte of the message header stores the actual effective metadata length.

[0115] In another possible scenario, if the video metadata is longer than 32 bytes, multiple video metadata packets are encapsulated. Each video metadata packet transmits 32 bytes of video metadata (where the HB3 byte is 32). If the last of these multiple video metadata packets contains less than 32 bytes of video metadata, the remaining bytes are padded with zeros. At the same time, the HB3 byte of the packet header stores the actual effective metadata length.

[0116] It's worth noting that when the video metadata is longer than 32 bytes, it's encapsulated using multiple video metadata packets. The index (INDEX) in the packet header can be used to distinguish these video metadata packets. To further identify the first, middle, and last video metadata packets, the S / E standard information can be used. Middle video metadata packets refer to those between the first and last video metadata packets.

[0117] Exemplarily, the first video metadata message among the multiple video metadata messages carries first S / E standard information, and the first S / E standard information is: S=1, E=0.

[0118] As another example, the last video metadata message among the multiple video metadata messages carries the second S / E standard information, and the second S / E standard information is: S=0, E=1.

[0119] Also illustratively, the middle video metadata message among the multiple video metadata messages carries third S / E standard information, and the third S / E standard information is: S=0, E=0.

[0120] In this embodiment, different S / E standard information is used to mark the start and end positions of multiple video metadata messages, avoiding the situation where the display device cannot identify the start and end of the video metadata message, resulting in video image rendering errors. Subsequently, the display device can effectively identify the video metadata message in the video stream and render the video image according to the video metadata carried by the video metadata message, thereby improving the video display quality on the display device and the user's experience of watching brilliant HDR video.

[0121] In an optional implementation, the video metadata message includes: a cyclic redundancy check code (CRC) set at the end, and the CRC is used to check the accuracy of the message header and data part in the video metadata message.

[0122] Combining the CRC, FIG. 4 and the contents of Table 1, Table 2 below shows a possible complete structure of a video metadata message.

[0123] Table 2

[0124] The content repeated in Table 2 and Table 1 is not repeated here. For the DB and CRC parts, refer to the description of the above embodiment. Using CRC to check the header bytes (HB or message header) and data bytes (DB or payload) is beneficial to improving the robustness of the video metadata message.

[0125] Continuing to refer to FIG. 3 , the video metadata data transmission method provided in this embodiment further includes the following steps S330 and S340 .

[0126] S330 : The source device 31 sends a video stream carrying a video metadata message to the display device 32 via the bus 33 .

[0127] Corresponding to the process of S330 , the display device 32 receives the video stream sent by the source device 31 through the bus 33 .

[0128] The video stream also includes video data, which includes multiple frames of image data or compressed or encoded image data. In this embodiment, the video data is carried in an Active Video Packet (AVP or AVP message) in the video stream. The AVP message is used to transmit effective video pixel data (the pixel data actually to be displayed by the display device).

[0129] In this way, the source device can directly read the capability description information of the display device through the bus, thereby knowing the HDR display capability of the display device, and only sends the video stream to the display device when the display device supports displaying Colorful HDR video. In this way, the invalid communication process between devices caused by the display device not supporting the processing of Colorful HDR video and the problem of bus bandwidth resources between devices being occupied are avoided.

[0130] Compared with the conventional technology that requires different encapsulation methods to encapsulate metadata data packets, the video metadata message in the video stream generated by this application includes the organization to which the standard adopted by the video metadata message belongs and the metadata type under the standard, so that the source device can use one encapsulation method to encapsulate different types of metadata, that is: a video metadata message of one format only needs to set up a specific field to mark whether it is dynamic metadata or static metadata, without the need to use two different packages to carry it. Thus, the source device avoids consuming more computing power to realize the encapsulation of the video metadata message, which is conducive to improving the transmission efficiency of the video carrying the video metadata message between different devices.

[0131] S340: The display device 32 processes and displays the video stream according to the video metadata message carried by the video stream.

[0132] Exemplarily, the display device can parse the video metadata message based on the aforementioned standards and metadata types, so as to process and display the video in the video stream according to the video metadata carried by the video metadata message, so as to improve the display effect of the video in the display device and enhance the user's experience of watching the video.

[0133] Regarding the process of transmitting the video metadata message provided in S320 above, a possible example is provided below, as shown in FIG5 , which is a second flow diagram of a method for transmitting video metadata provided by this application. In conjunction with the embodiment shown in FIG3 above, after S310 , the method for transmitting video metadata provided in this embodiment further includes the following step S311 .

[0134] S311. The source device 31 determines the HDR operating mode of the display device 32 according to the capability description information.

[0135] The HDR working mode is the first mode or the second mode.

[0136] The first mode characterizes that the image rendering parameters of the video are provided by the display device 32, also known as the monitor adaptation mode. The second mode characterizes that the image rendering parameters of the video are provided by the source device 31, also known as the receiver adaptation mode.

[0137] In an optional implementation, if the HDR operating mode of the display device 32 is the first mode, the above S320 includes the following S321 and S322.

[0138] S321: The source device 31 generates first video metadata of the video, and determines that the value of the metadata type is a first value.

[0139] The first value indicates that the first video metadata is static metadata, such as the first value is 0x0

[0140] The first video metadata is used to indicate the relationship between the electrical parameters provided by the display device 32 for the video and the display brightness of the video. The electrical parameters may refer to voltage or current. For example, the first video metadata is also called the image transfer characteristics of the video.

[0141] For example, in the data portion of the video metadata message, there are two flags (flag bits or identifiers): the monitor adaptation mode support flag (monitor mode enable) and the image transmission characteristic flag. The monitor adaptation mode support flag is a 1-bit unsigned integer, which is fixed to 1, indicating that the interconnection adaptation processing method of the HDR Vivid signal at this time is the monitor adaptation mode. The image transmission characteristic is a 1-bit unsigned integer that represents the image transmission characteristic. It is fixed to 0, indicating that in the monitor adaptation mode, the image transmission characteristic is the ST2084EOTF curve.

[0142] S322: The source device 31 generates a video metadata message according to the first value and the first video metadata.

[0143] Corresponding to the video metadata message generated in S322, in the video metadata data transmission method provided by this embodiment, S340 includes the following steps S341 to S343.

[0144] S341 : The display device 32 generates image rendering parameters for the video according to the video metadata and the device information of the display device 32 .

[0145] The device information of the display device 32 may include, but is not limited to: the display panel type, screen size, color gamut standard, color gamut value, screen resolution, screen ratio, brightness, speaker power, number of speakers, refresh rate, backlight mode and number of backlight partitions, etc. of the display device 32.

[0146] The display panel type refers to LCD, LED, OLED or other types of display screens.

[0147] Screen size refers to the size of the display device 32, such as 85 inches. The color gamut standard refers to the standard used by the display device 32 for color rendering, such as BT.709. The color gamut value indicates the display device 32's ability to display pixels of different colors. Screen resolution refers to the number of pixels in both the vertical and horizontal directions, measured in pixels (px). For screens of the same size, lower screen resolutions display fewer pixels and larger individual pixels. Higher screen resolutions display more pixels and smaller individual pixels. Screen ratio refers to the aspect ratio of the display device 32. The unit of brightness for a display is nits; higher nits values ​​indicate higher brightness. The refresh rate refers to the number of times the electron beam repeatedly scans the image on the screen, i.e., the number of images switched per unit time, measured in Hertz (Hz). The backlight type is either edge-lit or direct-lit. Backlight partitioning divides the backlight source into multiple zones, each of which can be independently controlled. The number of backlight partitions refers to the number of zones into which the backlight source is divided.

[0148] Image rendering parameters may include, but are not limited to: output image size, brightness, contrast, saturation, rendering order of image blocks, image noise removal information, image filters, and the like.

[0149] S342: The display device 32 processes the video stream according to the image rendering parameters to obtain a rendered video.

[0150] Exemplarily, the display device 32 processes the video data in the video stream according to the image rendering parameters, such as image cropping, image scaling, noise removal, image optimization, etc., to obtain a rendered video.

[0151] S343: The display device 32 displays the rendered video.

[0152] In this way, the display device generates image rendering parameters according to the device information of the display device and the received video metadata, thereby avoiding the problem of image rendering parameters being lost during transmission, which is conducive to improving the accuracy of video display by the display device.

[0153] In another optional implementation, if the HDR operating mode of the display device 32 is the second mode, the above S320 includes the following S323 and S324.

[0154] S323: The source device 31 generates second video metadata for the video, and determines that the value of the metadata type is a second value.

[0155] The second value indicates that the second video metadata is dynamic metadata, and is used to indicate that the display device 32 displays multiple sets of image rendering parameters for multiple frames of images in the video, with one frame of image corresponding to one set of image rendering parameters.

[0156] S324: The source device 31 generates a video metadata message according to the second value and the second video metadata.

[0157] Corresponding to the video metadata message generated by S324, in the video metadata data transmission method provided by this embodiment, S340 includes: the display device 32 processes the video stream according to the image rendering parameters in the video metadata message to obtain a rendered video, and displays the rendered video.

[0158] In this way, the source device determines the metadata type of the video metadata according to the HDR working mode of the display device, so that the video metadata message of this metadata type can adapt to the HDR working mode of the display device, such as static metadata adapting to the first mode and dynamic metadata adapting to the second mode. This avoids the problem that the type of video metadata is incompatible with the HDR working mode of the display device, resulting in the display device being unable to use the video metadata message to improve the display quality of the video. This is beneficial to improving the display effect of the video on the display device and enhancing the user's viewing experience.

[0159] Regarding the above video metadata transmission process, this embodiment also provides an optional implementation method, as shown in Figure 6, which is a flowchart diagram of a video metadata data transmission method provided by this application. The specific implementation of the source device, sink device, and bus can be referred to the description of the previous embodiment and will not be repeated here.

[0160] Referring to FIG. 6 , the video metadata transmission method provided in this embodiment includes the following steps S610 to S630 .

[0161] S610: The source device encapsulates one or more video metadata messages of the video.

[0162] The video metadata message includes: the organization to which the standard adopted by the video metadata message belongs, and the metadata type under the standard.

[0163] For more details about S610 and the video metadata message, please refer to the description of S320 above, which will not be repeated here.

[0164] S620: The source device sends one or more video metadata messages to the sink device.

[0165] S630: The sink device processes and displays the video according to the one or more video metadata messages.

[0166] For more details about S630, please refer to the description of the aforementioned embodiment and will not be elaborated here.

[0167] In this way, the source device can use a single encapsulation method to encapsulate different types of metadata, and the video metadata message can be marked with the standard and metadata type used by the video metadata message. This avoids the source device from consuming a lot of computing power to encapsulate the video metadata message, which is beneficial for improving the transmission efficiency of the video metadata message between different devices. Furthermore, the sink device (such as a display device) can parse the video metadata message based on the aforementioned standard and metadata type, so as to process and display the video in the video stream according to the video metadata carried by the video metadata message, thereby improving the display effect of the video on the display device and enhancing the user's viewing experience.

[0168] Regarding the specific format of the video metadata, in combination with the examples of FIG. 4 to FIG. 6 , three possible specific formats of the video metadata message are provided below.

[0169] A first possible video metadata message is shown in Figure 7, which is a structural diagram of a video metadata message provided by this application. For the tunnel message header of Figure 7, reference can be made to the description of the above embodiment, which will not be repeated here.

[0170] The HB portion includes: 0x04, RSVD, INDEX, 0x1, 0x0 and 0x05 in sequence.

[0171] The DB section includes: system version code (version_code), monitor mode enable flag (monitor mode enable), image transfer characteristics (transfer_character), 0, 0, multiple 0x00s, and CRC 32. The contents of the DB are the above static metadata.

[0172] The system version code is a 4-bit unsigned integer that represents the system version number. For example, the current version is assigned a value of 1.

[0173] The image transfer characteristic is a 1-bit unsigned integer that indicates the image transfer characteristic. For example, if the transfer character is 0, it indicates the transfer characteristic is the ST2084EOTF curve. If the transfer character is 1, it indicates the transfer characteristic is the Hybrid Log-Gamma (HLG)OETF curve.

[0174] A second possible video metadata message is shown in Figure 8, which is a second structural diagram of a video metadata message provided by this application. For the tunnel message header in Figure 8, reference may be made to the description of the aforementioned embodiment, which will not be repeated here.

[0175] The HB part includes: 0x04, RSVD, INDEX, 0x1, 0x1 and 0x20 in sequence.

[0176] A third possible video metadata message is shown in Figure 9, which is a structural diagram of a video metadata message provided by this application. For the tunnel message header in Figure 9, please refer to the description of the previous embodiment and will not be repeated here.

[0177] The HB part includes: 0x04, RSVD, INDEX, 0x1, 0x0 and 0x18 in sequence.

[0178] The following describes the contents of the DB section in Figures 8 and 9: The image fixed curve interval parameter (graphic source display value pq) is an 8-bit unsigned integer representing the PQ domain nonlinearly normalized display luminance code value corresponding to the graphics transmitted by the source device (playback device) to the display device. The color gamut and transfer characteristics of the graphics content are consistent with those of video, and are determined by the image transfer characteristics. For example, a value of 0 for graphic_source_display_value_pq indicates no graphics are being transmitted.

[0179] The maximum display mastering luminance is a 16-bit unsigned integer, obtained by combining the LSB and MSB of the maximum display mastering luminance. 0x0001 represents 1 nit, and 0xFFFF represents 65535 nits. The remaining parameters in Figures 7 through 9 refer to the dynamic metadata syntax in UWA005-2021 and are not detailed here.

[0180] It is understood that in order to implement the functions in the above embodiments, the source device and the display device include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily appreciate that, in conjunction with the various exemplary units and method steps described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in hardware or in a manner driven by computer software depends on the specific application scenario and design constraints of the technical solution.

[0181] The video metadata data transmission method provided by this embodiment is described in detail above with reference to FIG. 1 to FIG. 5 . The video device provided by this embodiment will be described below with reference to FIG. 10 .

[0182] Figure 10 is a schematic structural diagram of a video device provided by the present application. The video device 600 can be used to implement the functions of any one of the devices in the above method embodiments, and thus can also achieve the beneficial effects possessed by the above method embodiments. In this embodiment, the video device 600 can be a set-top box 110, a smart TV 120, or any display device as shown in Figure 1, or a source device 210 or a sink device 220 as shown in Figure 2, or a source device or display device provided in subsequent embodiments. It should be understood that the video device 600 can also be a module (such as a chip) applied to any of the aforementioned devices.

[0183] As shown in Figure 10, the video device 600 includes a transceiver module 610 and a processing module 620. The transceiver module 610 and the processing module 620 can collaborate to implement the various steps in the aforementioned method embodiment. A more detailed description of the transceiver module 610 and the processing module 620 can be directly obtained by referring to the relevant description of the device in the method embodiment shown in the aforementioned figures, and is not repeated here.

[0184] When a video device implements the video metadata data transmission method shown in any of the aforementioned figures through software, the video device and its various units may also be software modules. The aforementioned video metadata data transmission method is implemented by a processor calling the software module. The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0185] It can be understood that the video device shown in FIG10 is only an example provided in this embodiment. The video device may include more or fewer units according to different audio transmission processes, and this application does not limit this.

[0186] When the video device is implemented via hardware, the hardware may be implemented via a processor or a chip system. The chip system includes one or more chips, each of which includes an interface circuit and a control circuit. The interface circuit is used to receive data from other devices outside the chip and transmit it to the control circuit, or to send data from the control circuit to other devices outside the chip. The control circuit and the interface circuit implement the method of any possible implementation method in the above embodiments through logic circuits or executing code instructions. The beneficial effects can be found in the description of any aspect of the above embodiments and will not be repeated here.

[0187] It is understood that the processor in the embodiments of the present application may be a CPU, or other general-purpose processor, digital signal processor (DSP), ASIC, FPGA or other programmable logic device, transistor logic device, hardware component or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0188] In addition, the video device 600 shown in Figure 10 can also be processed by a video processing device, as shown in Figure 11. Figure 11 is a structural diagram of the video processing device provided in this application. The video processing device 700 includes: a memory 710 and at least one processor 720. The processor 720 can implement the video metadata data transmission method provided in the above embodiment. The memory 710 is used to store software instructions corresponding to the above video metadata data transmission method.

[0189] As an optional implementation, in hardware implementation, the video processing device 700 may refer to a chip or chip system encapsulating one or more processors 720. For example, when the video processing device 700 is used to implement the method steps in the above embodiment, the processor 720 included in the video processing device 700 executes the steps of the source device 31 in the above method and its possible sub-steps. In an optional scenario, the video processing device 700 may also include a communication interface 730, which can be used to send and receive data. For example, the communication interface 730 is used to receive audio data or send audio streams; the communication interface 730 can be implemented by the interface circuit included in the video processing device 700. Therefore, in some examples, the communication interface 730 can also be referred to as the transceiver of the video processing device. In this embodiment, the communication interface 730 supports wired connection using a unified multimedia interconnect interface (such as a UMI interface).

[0190] In an embodiment of the present application, the communication interface 730, the processor 720, and the memory 710 may be connected via a bus 740, which may be divided into an address bus, a data bus, a control bus, etc. The bus 740 may be a Peripheral Component Interconnect Express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), or other types of buses.

[0191] It is worth noting that the video processing device 700 can also perform the functions of the video apparatus 600 shown in FIG. 10 , which will not be described in detail here.

[0192] The video processing device 700 provided in this embodiment can be the above-mentioned set-top box 110, smart TV 120, source device 210 or source device 31, etc., or other devices with video processing functions, and this application is not limited to this. For example, when the aforementioned display device also has a video processing function, the video processing device 600 can refer to any of the aforementioned display devices.

[0193] In addition, the video device 600 shown in FIG10 can also be implemented by a display device. When the video device 600 is implemented by a display device, this embodiment provides a possible example, as shown in FIG12, which is a schematic structural diagram of the display device provided by the present application. The display device includes: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a UMI interface 231, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio unit 270, a speaker 270A, a receiver 270B, a microphone 270C, a sensor module 280, a button 260, an indicator 262, a camera 263, and a display screen 264.

[0194] Among them, the above-mentioned sensor module 280 may include sensors such as pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor and bone conduction sensor.

[0195] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the display device. In other embodiments, the display device may include more or fewer components than shown, or may combine or separate certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0196] The processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0197] The controller can be the nerve center and command center of the display device. The controller can generate operation control signals based on instruction opcodes and timing signals to complete the control of instruction fetching and execution.

[0198] Processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 210 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 210. If processor 210 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 210 latency, and thus improves system efficiency.

[0199] In some embodiments, the processor 210 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a USB interface, a UMI interface, etc.

[0200] It is understood that the interface connection relationship between the modules illustrated in this embodiment is only for illustrative purposes and does not constitute a structural limitation on the display device. In other embodiments, the display device may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0201] The wireless communication function of the display device can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, a modem processor, and a baseband processor. In some embodiments, antenna 1 of the display device is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, so that the display device can communicate with the network and other devices through wireless communication technology.

[0202] The wired communication function of the display device can be implemented through the USB interface 230 or the UMI interface 231. For example, the display device receives or sends video streams and video metadata messages via a bus connected to the UMI interface 231.

[0203] The display device implements its display functions through a GPU, display screen 264, and an application processor. A GPU is a microprocessor for image processing that connects display screen 264 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 210 may include one or more GPUs that execute program instructions to generate or modify display information.

[0204] The display screen 264 is used to display images, videos, etc. The display screen 264 includes a display panel.

[0205] The display device can implement a camera function through an ISP, camera 263, video codec, GPU, display 264, and application processor. The ISP is responsible for processing data fed back by camera 263. Camera 263 is responsible for capturing still images or video. In some embodiments, the display device may include one or N cameras 263, where N is a positive integer greater than 1.

[0206] In this embodiment, the above display screen 264, video codec, GPU, display screen 264 and application processor can also be collectively referred to as a display unit of the display device 32, which is used to process and display the received video stream.

[0207] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the display device. The external memory card communicates with the processor 210 via the external memory interface 220 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0208] The internal memory 221 can be used to store computer executable program code, which includes instructions. The processor 210 executes various functional applications and data processing of the display device by running the instructions stored in the internal memory 221. For example, in an embodiment of the present application, the processor 210 can execute instructions stored in the internal memory 221, and the internal memory 221 can include a program storage area and a data storage area.

[0209] The program storage area may store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the display device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0210] The display device can implement audio functions such as music playback and recording through the audio unit 270, the speaker 270A, the receiver 270B, the microphone 270C, and the application processor.

[0211] Buttons 260 include a power button, a volume button, and the like. Buttons 260 may be mechanical buttons or touch buttons. Indicator 262 may be an indicator light that can be used to indicate charging status, battery level changes, messages, missed calls, notifications, and the like.

[0212] The method steps in the embodiments of the present application can also be implemented by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device or a terminal device. Of course, the processor and storage medium can also exist as discrete components in a video processing device and a display device.

[0213] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the process or function described in the embodiments of the present application is performed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as a solid state drive (SSD).

[0214] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A video metadata transmission method, characterized in that: The method comprises: One or more video metadata messages encapsulating a video, wherein the video metadata message includes: an organization to which a standard adopted by the video metadata message belongs, and a metadata type under the standard; The one or more video metadata messages are sent.

2. The method according to claim 1, characterized in that The video metadata message includes a message header, and the message header carries the organization and the metadata type.

3. The method according to claim 1 or 2, characterized in that: The video metadata message includes a payload, and the payload is used to carry the video metadata. The size of the payload is N bytes, where N is any value between 1 and 32.

4. The method according to any one of claims 1 to 3, characterized in that The message start flag of the first video metadata message among the multiple video metadata messages is 1, and the message end flag is 0; The message start flag of the last video metadata message among the multiple video metadata messages is 0, and the message end flag is 1.

5. The method according to claim 4, characterized in that The message start flag of the middle video metadata message of the multiple video metadata messages is 0, and the message end flag is 0.

6. The method according to any one of claims 1 to 5, characterized in that The video metadata message carries an index.

7. The method according to any one of claims 1 to 6, characterized in that Before encapsulating the one or more video metadata messages of the video, the method further includes: Get the HDR capability information of the sink device. The HDR working mode of the host device is determined according to the Jingcai HDR capability information; the HDR working mode is a monitor adaptation mode or a receiving end adaptation mode, the monitor adaptation mode represents: the image rendering parameters of the video are provided by the host device, and the receiving end adaptation mode represents: the image rendering parameters of the video are provided by the source device.

8. The method according to claim 7, characterized in that If the HDR working mode of the sink device is the monitor adaptation mode, the video metadata message includes: first video metadata and a metadata type whose value is a first value, wherein the first value indicates that the first video metadata is static metadata, and the first video metadata is used to indicate: an association relationship between an electrical parameter provided by the sink device for the video and a display brightness of the video.

9. The method according to claim 7 or 8, characterized in that: If the HDR working mode of the sink device is the receiving end adaptation mode, the video metadata message includes: second video metadata and a metadata type whose value is a second value, where the second value indicates that the second video metadata is dynamic metadata, and the second video metadata is used to indicate: the sink device displays multiple groups of image rendering parameters for multiple frames of images in the video, and one frame of image corresponds to one group of image rendering parameters.

10. The method according to any one of claims 7 to 9, characterized in that: The source device and the sink device are connected via a unified multimedia interconnect bus.

11. A video metadata transmission method, characterized in that: The method comprises Receive one or more video metadata messages of a video, wherein the video metadata messages include: an organization to which a standard adopted by the video metadata messages belongs, and a metadata type under the standard; The video is processed and displayed according to the one or more video metadata messages.

12. The method according to claim 11, characterized in that The video metadata message includes a message header, and the message header carries the organization and the metadata type.

13. The method according to claim 11 or 12, characterized in that: The video metadata message includes a payload, and the payload is used to carry the video metadata. The size of the payload is N bytes, where N is any value between 1 and 32.

14. The method according to any one of claims 11 to 13, characterized in that The message start flag of the first video metadata message among the multiple video metadata messages is 1, and the message end flag is 0; The message start flag of the last video metadata message among the multiple video metadata messages is the second value, and the message end flag is 1.

15. The method according to claim 14, characterized in that The message start flag of the middle video metadata message of the multiple video metadata messages is 0, and the message end flag is 0.

16. The method according to any one of claims 11 to 15, characterized in that The video metadata message carries an index.

17. The method according to any one of claims 11 to 16, characterized in that The video metadata is used to indicate: an association relationship between an electrical parameter provided by a sink device for the video and a display brightness of the video; The processing and displaying of the video according to the one or more video metadata messages includes: generating an image rendering parameter of the video according to the one or more video metadata messages and the device information of the sink device; Processing the video according to the image rendering parameters to obtain a rendered video; The rendered video is displayed.

18. The method according to any one of claims 11 to 16, characterized in that: The video metadata is used to indicate that: the sink device displays multiple groups of image rendering parameters for multiple frames of images in the video, and one frame of image corresponds to one group of image rendering parameters.

19. The method according to claim 17 or 18, characterized in that The sink device is connected to the source device via a unified multimedia interconnect bus.

20. A video processing device, characterized in that: include: memories, transceivers, and processors; The processor is used to generate a video metadata message; The memory is used to cache the video metadata message; The transceiver is used to send the video metadata message; The memory, the transceiver and the processor are configured to collaboratively execute the method of any one of claims 1-10.

21. A display device, characterized in that: include: transceiver and display unit; The transceiver is used to receive a video stream carrying a video metadata message, wherein the video metadata message includes an organization to which a standard adopted by the video metadata message belongs, and a metadata type under the standard; The display unit is used to process and display the video stream according to the video metadata message carried by the video stream; The transceiver and the display unit are used to collaboratively execute the method of any one of claims 11-19.

22. A video playback system, characterized in that: include: The video processing device of claim 20, and the display device of claim 21, wherein the video processing device and the display device are connected via a unified multimedia interconnect bus.

23. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer software instructions. When the computer software instructions are executed in a computer, the computer executes the method according to any one of claims 1 to 19.

24. A computer program product, characterized in that When the computer program product is run on a computer, the computer executes the method of any one of claims 1-19.

25. A video metadata transmission device, characterized in that: include: An encapsulation module, used to encapsulate one or more video metadata messages of a video, wherein the video metadata message includes: an organization to which a standard adopted by the video metadata message belongs, and a metadata type under the standard; The transceiver module is used to send the one or more video metadata messages.

26. A video metadata transmission device, characterized in that: include: A transceiver module, configured to receive one or more video metadata messages of a video, wherein the video metadata message includes: an organization to which a standard adopted by the video metadata message belongs, and a metadata type under the standard; The display module is used to process and display the video according to the one or more video metadata messages.

27. A video playback method, characterized in that: include: The source device encapsulates one or more video metadata messages of the video, wherein the video metadata message includes: an organization to which a standard adopted by the video metadata message belongs, and a metadata type under the standard; The source device sends the one or more video metadata messages to the sink device; The sink device processes and displays the video according to the one or more video metadata messages.

Images