Video processing method, device, system and client

Abstract

The present disclosure relates to a video processing method, device, system and client, and to the field of computer technology. The method of the present disclosure includes: for each frame image of a video with a transparent channel, separating the image data into first color mode data and transparent channel data, wherein the transparent channel data represents the transparency of the pixel; converting the first color mode data into second color mode data, wherein the second color mode belongs to a color mode that can be processed by the video encoder; mixing and arranging the second color mode data and the transparent channel data in a preset order to form the data of the extended image corresponding to the image; encoding the data of the extended image corresponding to each frame image of the video, generating a video code stream and sending it to the client, so that the client can restore each frame image according to the encoded data of the extended image corresponding to each frame image and then play it.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video processing method, device, system and client. Background Art

[0002] Video streams with transparent channels can be widely used in video conferencing, live broadcasting and other scenarios, and bring richer imagination space for the content presentation of these scenarios. Using this kind of video stream can greatly improve the flexibility of front-end display, bring more vivid visual effects to the video stream, realize the arbitrary replacement of background images, the layered display of special effects and video streams, the superposition of multiple video streams, etc., and the fusion after superposition is very natural without any sense of disobedience, making it easy to achieve lively and cool effects in scene display.

[0003] As known to the inventors, when processing a video stream with a transparent channel, the original video data and the transparent channel data are encoded separately, and the encoding can be performed using the VP8 or VP9 encoding standard, and then placed in a fixed format container for transmission (such as MKV or WebM). Summary of the invention

[0004] The inventors have discovered that the existing processing and transmission methods for video streams with transparent channels, the requirements for fixed format containers and the requirements for encoding methods, all greatly limit the usage scenarios of video streams with transparent channels.

[0005] A technical problem to be solved by the present disclosure is: how to improve the applicability and application flexibility of video streams with transparent channels.

[0006] According to some embodiments of the present disclosure, a video processing method is provided, comprising: for each frame image of a video with a transparent channel, separating the image data into first color mode data and transparent channel data, wherein the transparent channel data represents the transparency of the pixel; converting the first color mode data into second color mode data, wherein the second color mode belongs to a color mode that can be processed by a video encoder; mixing and arranging the second color mode data and the transparent channel data in a preset order to form data of an extended image corresponding to the image; encoding the data of the extended image corresponding to each frame image of the video, generating a video code stream and sending it to a client, so that the client can restore each frame image according to the encoded data of the extended image corresponding to each frame image and then play it.

[0007] In some embodiments, the second color mode data includes first channel data, second channel data and third channel data; mixing and arranging the second color mode data and the transparent channel data in a preset order includes: mixing and arranging the second color mode data and the transparent channel data in the order of first channel data, transparent channel data, second channel data, third channel data, third channel data and second channel data.

[0008] In some embodiments, the method further includes: storing the data of the extended image corresponding to each frame image of the video into a memory; recording the second color mode, and the width, height, starting address of the first channel data in the memory, and starting address of the first second channel data of the extended image corresponding to each frame image as storage information of the video; wherein, encoding the data of the extended image corresponding to each frame image of the video includes: reading the data of the extended image corresponding to each frame image of the video from the memory according to the storage information of the video for encoding.

[0009] In some embodiments, reading data of extended images corresponding to each frame of a video from a memory according to storage information of the video for encoding includes: for each frame of the image, determining a first length value and a second length value according to a second color mode and the width and height of the extended image corresponding to the image, wherein the first length value is the total length of the first channel data of the corresponding extended image as a whole of the first channel data and the transparent channel data in the image, and the second length value is the total length of the second channel data, the third channel data, the third channel data and the second channel data of the image as a whole as the second channel data and the third channel data of the corresponding extended image; reading the first channel data, the second channel data and the third channel data of the extended image corresponding to the image from the memory respectively according to the starting address of the first channel data in the memory, the starting address of the first second channel data, the first length value and the second length value, and dividing the extended image corresponding to the image into multiple macroblocks; encoding the multiple macroblocks of the extended image corresponding to the image according to a preset encoding method.

[0010] In some embodiments, the first color mode data includes red, green and blue (RGB) data, the transparent channel data includes alpha channel data, the second color mode data includes YUV data of a preset sampling method, and mixing and arranging the second color mode data and the transparent channel data in the order of first channel data, transparent channel data, second channel data, third channel data, third channel data and second channel data includes: mixing and arranging the YUV data of a preset sampling method and the Alpha channel data in the order of Y channel data, Alpha channel data, U channel data, V channel data, V channel data and U channel data.

[0011] In some embodiments, converting the first color mode data into the second color mode data includes: converting the first color mode data into the second color mode data according to a first conversion matrix.

[0012] According to some other embodiments of the present disclosure, a video processing method is provided, including: receiving a code stream of a video, wherein the code stream includes encoded data of an extended image corresponding to each frame image of the video; decoding the code stream of the video to obtain data of the extended image corresponding to each frame image after decoding, wherein the data of the extended image is data after second color mode data and transparent channel data of the image are mixed and arranged in a preset order; for each frame image, separating the data of the extended image corresponding to the decoded image into second color mode data and transparent channel data; for each frame image, converting the second color mode data into first color mode data, wherein the first color mode data is the original color mode data of the image; for each frame image, fusing the first color mode data and the transparent channel data to obtain data of an image with a transparent channel; and using the data of the image with a transparent channel of each frame of the video for rendering and display.

[0013] In some embodiments, the second color mode data includes first channel data, second channel data and third channel data, and the data of the extended image is the second color mode data and the transparent channel data mixed and arranged in the order of first channel data, transparent channel data, second channel data, third channel data, third channel data and second channel data.

[0014] In some embodiments, the method further includes: obtaining the second color mode and the width and height of the extended image corresponding to each frame of image; wherein, for each frame of image, separating the data of the extended image corresponding to the decoded image into the second color mode data and the transparent channel data includes: for each frame of image, according to the second color mode and the width and height of the extended image corresponding to each frame of image, determining the length of the first channel data, the second channel data length and the third channel data length of the image, wherein the second color mode data includes the first channel data, the second channel data and the third channel data; dividing the data of the extended image corresponding to the decoded image into two parts of data from the middle on average; according to the length of the first channel data, the second channel data length and the third channel data length, separating the first channel data, the second channel data and the third channel data from the first part of data; according to the length of the first channel data, the second channel data length and the third channel data length, separating the transparent channel data, the third channel data and the second channel data from the second part of data, and discarding the separated third channel data and second channel data.

[0015] In some embodiments, the first color mode data includes red, green, and blue (RGB) data, the transparent channel data includes alpha channel data, the second color mode data includes YUV data of a preset sampling method, and the data of the extended image is the YUV data of a preset sampling method and the Alpha channel data mixed and arranged in the order of Y channel data, Alpha channel data, U channel data, V channel data, V channel data, and U channel data.

[0016] In some embodiments, converting the second color mode data into the first color mode data includes: converting the second color mode data into the first color mode data according to a second conversion matrix.

[0017] According to some other embodiments of the present disclosure, a video processing device is provided, including: a separation module, for separating the data of each frame image of a video with a transparent channel into first color mode data and transparent channel data, wherein the transparent channel data represents the transparency of the pixel; a conversion module, for converting the first color mode data into second color mode data, wherein the second color mode belongs to a color mode that can be processed by a video encoder; an arrangement module, for mixing and arranging the second color mode data and the transparent channel data in a preset order to form data of an extended image corresponding to the image; a sending module, for encoding the data of the extended image corresponding to each frame image of the video, generating a video code stream and sending it to a client, so that the client can restore each frame image according to the encoded data of the extended image corresponding to each frame image and then play it.

[0018] According to some further embodiments of the present disclosure, a client is provided, comprising: a receiving module for receiving a code stream of a video, wherein the code stream includes encoded data of an extended image corresponding to each frame image of the video; a decoding module for decoding the code stream of the video to obtain data of an extended image corresponding to each frame image after decoding, wherein the data of the extended image is data obtained by mixing and arranging the second color mode data and transparent channel data of the image in a preset order; a separation module for separating, for each frame image, the data of the extended image corresponding to the decoded image into second color mode data and transparent channel data; a conversion module for converting, for each frame image, the second color mode data into first color mode data, wherein the first color mode data is the original color mode data of the image; a fusion module for fusion, for each frame image, of the first color mode data and the transparent channel data to obtain data of an image with a transparent channel; and a display module for rendering and displaying using the data of the image with a transparent channel of each frame of the video.

[0019] According to some further embodiments of the present disclosure, a video processing device is provided, including: a processor; and a memory coupled to the processor, for storing instructions, wherein when the instructions are executed by the processor, the processor executes the video processing method as described in any of the aforementioned embodiments.

[0020] According to some further embodiments of the present disclosure, a client is provided, comprising: a processor; and a memory coupled to the processor, for storing instructions, which, when executed by the processor, causes the processor to execute a video processing method as described in any of the aforementioned embodiments.

[0021] According to some further embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the steps of any of the aforementioned method embodiments.

[0022] According to some further embodiments of the present disclosure, a video processing system is provided, including: the video processing device of any of the aforementioned embodiments or the client of any of the aforementioned embodiments.

[0023] In the present disclosure, for each frame of a video with a transparent channel, the image data is separated into first color mode data and transparent channel data, the first color mode data is converted into second color mode data, and then the second color mode data and the transparent channel data are mixed and arranged in a preset order to form the data of an extended image corresponding to the image. The data of the extended image corresponding to each frame of the video is encoded to generate a video code stream and send it to the client. In the present disclosure, the second color mode data and the transparent channel data are mixed and arranged in a preset order to generate an extended image. The image data and the transparent channel data that originally need to be encoded and transmitted separately can be synthesized into the data of the extended image. In this way, the extended image can be directly encoded using a universal encoding method, which is not limited to the VP8 or VP9 encoding standards, and is adapted to all video encoding formats. The generated video code stream no longer relies on fixed format video container transmission, which improves the applicability and application flexibility of the video stream with a transparent channel.

[0024] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 A schematic flowchart of a video processing method according to some embodiments of the present disclosure is shown.

[0027] Figure 2A A schematic diagram showing the arrangement of image data according to some embodiments of the present disclosure.

[0028] Figure 2B A schematic diagram showing the arrangement of data of an extended image according to some embodiments of the present disclosure.

[0029] Figure 2C A schematic diagram showing the display effect of an extended image according to some embodiments of the present disclosure.

[0030] Figure 3 A schematic flowchart of a video processing method according to some other embodiments of the present disclosure is shown.

[0031] Figure 4A A schematic diagram showing a video processing method according to some embodiments of the present disclosure.

[0032] Figure 4B Schematic diagrams showing video processing methods according to other embodiments of the present disclosure.

[0033] Figure 5 A schematic diagram showing the structure of a video processing device according to some embodiments of the present disclosure.

[0034] Figure 6 A schematic diagram showing the structure of a client according to some embodiments of the present disclosure.

[0035] Figure 7 A schematic structural diagram of a video processing device according to some other embodiments of the present disclosure is shown.

[0036] Figure 8 A schematic diagram showing the structure of a video processing device according to some other embodiments of the present disclosure.

[0037] Fig. 9 A schematic diagram showing the structure of a client according to some embodiments of the present disclosure.

[0038] Fig.10 A schematic diagram showing the structure of a client according to other embodiments of the present disclosure.

[0039] Fig.11 A schematic diagram showing the structure of a video processing system according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0041] The present disclosure provides a video processing method. Figures 1 to 4B Give a description.

[0042] Figure 1 Flowcharts of some embodiments of the video processing method disclosed herein. Figure 1 As shown, the method of this embodiment includes: steps S102 to S108.

[0043] In step S102, for each frame of the video with a transparent channel, the image data is separated into first color mode data and transparent channel data.

[0044] In some embodiments, each frame image of a video with a transparent channel is represented by RGBA (Red Green Blue Alpha) data, the first color mode data includes RGB data, the transparent channel data includes Alpha channel data, and the transparent channel data represents the transparency of the pixel.

[0045] In step S104, the first color mode data is converted into second color mode data.

[0046] The second color mode belongs to a color mode that can be processed by the video encoder, and the second color mode may belong to a color mode that can be encoded using a general encoding standard, for example, various mainstream encoding standards such as H264, HEVC (H265), VP8, VP9, ​​SVAC, etc. In some embodiments, the second color mode is a YUV mode, and further, the second color mode is, for example, YUV4:2:1, YUV4:2:2, YUV4:4:4, etc., and may be a YUV mode with different sampling methods, not limited to the examples given.

[0047] In some embodiments, the first color mode data is converted to the second color mode data according to a first conversion matrix.

[0048] For example, RGB data is converted to YUV data according to the following formula:

[0049] Y=0.299R+0.587G+0.114B (1)

[0050] U=-0.147R-0.289G+0.436B (2)

[0051] V=0.615R-0.515G-0.100B (3)

[0052] In formulas (1)-(3), Y, U, and V respectively represent the Y channel data, U channel data, and V channel data in the YUV data of the preset sampling method, and R, G, and B respectively represent the R channel data, G channel data, and B channel data in the RGB data. After converting the RGB data into YUV data according to formulas (1)-(3), sampling can be performed according to the preset sampling method to obtain a YUV mode with the same sampling method.

[0053] In step S106, the second color mode data and the transparent channel data are mixed and arranged in a preset order to form data of an extended image corresponding to the image.

[0054] In some embodiments, the second color mode data includes first channel data, second channel data and third channel data; the second color mode data and transparent channel data are mixed and arranged in the order of first channel data, transparent channel data, second channel data, third channel data, third channel data and second channel data.

[0055] Taking the case where the first color mode data includes RGB data, the transparent channel data includes Alpha channel data, and the second color mode data includes YUV data of a preset sampling method, the YUV data of the preset sampling method and the Alpha channel data are mixed and arranged in the order of Y channel data, Alpha channel data, U channel data, V channel data, V channel data, and U channel data. In this way, for the encoder, the transparent channel data can be regarded as the first channel data, and combined with the added second channel data and third channel data for encoding, and the encoding and transmission can be performed normally without affecting the original image data.

[0056] Taking the 6×4 format image data as an example, the original YUV420 (YUV4:2:0) data without alpha channel is arranged as follows Figure 2A As shown, the Y channel data, U channel data, and V channel data are stored in the memory in sequence. Every four Y channel data correspond to a set of U channel data and V channel data, for example, Y1, Y2, Y7, and Y8 correspond to U1 and V1.

[0057] After the data is mixed and arranged in the order of Y channel data, Alpha channel data, U channel data, V channel data, V channel data, and U channel data, the image data increases, which is equivalent to expanding to a 6×8 extended image, that is, obtaining the data of the extended image. Figure 2BAs shown, the Y channel data, Alpha channel data, U channel data, V channel data, V channel data, and U channel data are stored in the memory in sequence. The data is mixed in the above manner to adapt to the encoding method. When encoding, the encoder will use the Alpha channel data as the Y channel data and encode it together with the newly added V channel data and U channel data, that is, the 6×8 extended image is encoded as a whole. This mixed arrangement can avoid the problem of color deviation in the restored video image.

[0058] like Figure 2C As shown, if the extended image is directly encoded and transmitted, and the client directly decodes and displays it, the displayed extended image is divided into two parts, the upper part is the original image (the image should be in color, but not shown), and the lower part is the Alpha channel data that is recognized as Y channel data and combined with the newly added V channel data and U channel data to form an image.

[0059] In step S108, the data of the extended image corresponding to each frame image of the video is encoded to generate a video code stream and send it to the client.

[0060] In some embodiments, the data of the extended image corresponding to each frame image of the video is stored in the memory; the second color mode, and the width, height, starting address of the first channel data in the memory, and starting address of the first second channel data of the extended image corresponding to each frame image are recorded as the storage information of the video; and the data of the extended image corresponding to each frame image of the video is read from the memory for encoding according to the storage information of the video.

[0061] In some embodiments, for each frame of image, a first length value and a second length value are determined according to a second color mode and a width and a height of an extended image corresponding to the image, and according to a starting address of the first channel data in the memory, a starting address of the first second channel data, the first length value and the second length value, the first channel data, the second channel data and the third channel data of the extended image corresponding to the image are respectively read from the memory, and the extended image corresponding to the image is divided into multiple macroblocks; the multiple macroblocks of the extended image corresponding to the image are encoded according to a preset encoding method.

[0062] The first length value is the total length of the first channel data and the transparent channel data in the image as a whole as the first channel data of the corresponding extended image, and the second length value is the total length of the second channel data, the third channel data, the third channel data and the second channel data of the image as a whole as the second channel data and the third channel data of the corresponding extended image. For example, according to YUV420, the width of the extended image is 6 and the height is 8, so it can be determined that the first length value is 48 and the second length value is 24. Further, according to the starting address of the Y channel data in the memory and the starting address of the first U channel data, the Y channel data, the U channel data and the V channel data of the extended image are read from the memory respectively, and divided into multiple macroblocks for encoding respectively. The method of dividing macroblocks can adopt the existing technology, which will not be repeated here.

[0063] The above method is equivalent to directly encoding the extended image to generate a video code stream, which can be transmitted to the client using a general streaming media transmission protocol, such as RTSP (Real Time Streaming Protocol), RTMP (Real Time Messaging Protocol), HTTP-FLV (HyperText Transfer Protocol-Flash Video), WS-FLV, etc.

[0064] In the above embodiment, for each frame of the video with a transparent channel, the image data is separated into first color mode data and transparent channel data, the first color mode data is converted into second color mode data, and then the second color mode data and the transparent channel data are mixed and arranged in a preset order to form the data of the extended image corresponding to the image. The data of the extended image corresponding to each frame of the video is encoded to generate a video code stream and send it to the client. In the above embodiment, the second color mode data and the transparent channel data are mixed and arranged in a preset order to generate an extended image. The method can synthesize the image data and the transparent channel data that originally need to be encoded and transmitted separately into the data of the extended image. In this way, the extended image can be directly encoded using a universal encoding method, which is not limited to the VP8 or VP9 encoding standards, and is adapted to all video encoding formats. The generated video code stream no longer relies on fixed format video container transmission, which improves the applicability and application flexibility of the video stream with a transparent channel.

[0065] Combine the following Figure 3 Describes how the client processes the video stream after receiving it.

[0066] Figure 3 Flowcharts of some embodiments of the video processing method disclosed herein. Figure 3 As shown, the method of this embodiment includes: steps S302 to S312.

[0067] In step S302, a video code stream is received.

[0068] The code stream includes the encoded data of the extended image corresponding to each frame image of the video.

[0069] In step S304, the code stream of the video is decoded to obtain data of an extended image corresponding to each frame of the decoded image.

[0070] The data of the extended image is the data after the second color mode data and the transparent channel data of the image are mixed and arranged in a preset order. In some embodiments, the second color mode data includes first channel data, second channel data and third channel data, and the data of the extended image is the data after the second color mode data and the transparent channel data are mixed and arranged in the order of first channel data, transparent channel data, second channel data, third channel data, third channel data and second channel data.

[0071] For example, the transparent channel data includes Alpha channel data, the second color mode data includes YUV data of a preset sampling method, and the data of the extended image is YUV data of a preset sampling method and Alpha channel data mixed and arranged in the order of Y channel data, Alpha channel data, U channel data, V channel data, V channel data, and U channel data.

[0072] The decoding may be performed using a method corresponding to the encoding method, which will not be described in detail here.

[0073] In step S306, for each frame of image, the data of the extended image corresponding to the decoded image is separated into the second color mode data and the transparent channel data.

[0074] In some embodiments, the second color mode data includes first channel data, second channel data and third channel data; the second color mode and the width and height of the extended image corresponding to each frame of the image are obtained; for each frame of the image, the length of the first channel data, the second channel data length and the third channel data length of the image are determined according to the second color mode and the width and height of the extended image corresponding to each frame of the image; the data of the extended image corresponding to the decoded image is evenly divided into two parts of data from the middle; according to the length of the first channel data, the second channel data length and the third channel data length, the first channel data, the second channel data and the third channel data are separated from the former part of the data; according to the length of the first channel data, the second channel data length and the third channel data length, the transparent channel data, the third channel data and the second channel data are separated from the latter part of the data, and the separated third channel data and second channel data are discarded.

[0075] The client can obtain the second color mode and the width and height of the extended image corresponding to each frame of the image from the sending end. The data of the extended image corresponding to the decoded image is data divided into multiple macroblocks, each macroblock in the first part of the data includes the first channel data, the second channel data and the third channel data, and each macroblock in the second part of the data includes the transparent channel data, the third channel data and the second channel data. Therefore, the above method can be used to separate the second color mode data from the transparent channel data.

[0076] In step S308, for each frame of image, the second color mode data is converted into the first color mode data.

[0077] The first color mode data is original color mode data of the image, such as RGB data. In some embodiments, the second color mode data is converted into the first color mode data according to a second conversion matrix.

[0078] In some embodiments, the YUV data of the preset sampling method is converted into RGB data according to the following formula:

[0079] R=Y+1.14V (4)

[0080] G=Y-0.39U-0.58V (5)

[0081] B=Y+2.03U (6)

[0082] In formulas (4)-(6), Y, U, and V respectively represent the Y channel data, U channel data, and V channel data in the YUV data of the preset sampling method, and R, G, and B respectively represent the R channel data, G channel data, and B channel data in the RGB data.

[0083] In step S310, for each frame of image, the first color mode data and the transparent channel data are merged to obtain an image with a transparent channel.

[0084] For example, merging RGB data with alpha channel data results in RGBA data of an image with a transparent channel.

[0085] In step S312, the data of the image with the transparent channel of each frame of the video is used for rendering and display.

[0086] The method of the above embodiment can be executed by the client, and the data received by the client can be decoded using the general standard corresponding to the encoding end. The video with a transparent channel can be restored according to the decoded data to achieve the effect of a transparent picture, thereby improving the applicability and application flexibility of the video stream with a transparent channel.

[0087] The following example takes the first color mode data as RGB data, the second color mode data as YUV4:2:0, and the transparent channel data as Alpha channel data as an example. Figure 4A and 4B Some application examples of the present disclosure are described.

[0088] like Figure 4A As shown, the video processing device can perform the following steps:

[0089] 1) For each frame of the video, the RGBA data corresponding to the image is extracted and separated into RGB data and Alpha channel data;

[0090] 2) Convert the separated RGB data into YUV4:2:0 (YUV420) data that can be video-encoded;

[0091] 3) Considering the influence of the encoder dividing macroblocks on the encoded data, the YUV420 data and the Alpha channel data are mixed and arranged in the order of Y channel data, Alpha channel data, U channel data, V channel data, V channel data, and U channel data to obtain the data of the extended image;

[0092] This arrangement can avoid the problem of color deviation in the restored video image.

[0093] 4) storing the data of the extended image in a memory;

[0094] The extended image data is stored in memory in the format of first storing all Y components, then storing U components, and finally storing V components.

[0095] 5) The video encoder in the video processing device reads the data of the extended image and encodes it using a video general encoding algorithm;

[0096] When encoding, the video encoder first divides the data of the extended image into macroblocks (for example, macroblocks are 4x4 pixels, 8x8 pixels, or 16x16 pixels) before encoding. In order to ensure that adjacent UV values ​​are fused and encoded after macroblock division, and to ensure that the UV data is not distorted after decoding, the most efficient and concise method is to arrange the UV data of the UV and Alpha channels as UV (Y channel) + VU (Alpha channel). For example, H264, HEVC, VP8, etc. are used for encoding.

[0097] 6) Use a common streaming media transmission protocol to transmit to the client.

[0098] For example, RTSP / RTMP / HTTP-FLV / WS-FLV, etc.

[0099] like Figure 4B As shown, the client can perform the following steps:

[0100] 1) The received video data is decoded into YUV420 and Alpha channel mixed data using the corresponding decoding format, that is, the data of the extended image;

[0101] 2) Separate the data of the extended image into YUV420 data and Alpha channel data;

[0102] 3) Convert YUV420 data to RGB channel data;

[0103] 4) Use RGB data and Alpha channel data to merge into RGBA four-channel data for rendering and display, to achieve the effect of transparent picture, which can be merged and superimposed with any background.

[0104] In the above embodiment, the original video data and the corresponding Alpha channel data are arranged in a special arrangement order, and then restored during decoding. By using the method of the above embodiment, the video can achieve the effect of not improving the bit rate, and can not only send the YUV420 data to the video encoding library for encoding and transmission, and then use the corresponding video decoding library for decoding without changing the process of sending the YUV420 data to the video encoding library for encoding and transmission, but also merge the encoded Alpha channel data and the decoded RGB rendering data into RGBA four-channel data for fusion display in other scenes. The scheme of the above embodiment realizes the mixing of Alpha channel data and YUV data before real-time video encoding, and then encodes and transmits them using any video encoding technology. In this way, only the video data is changed, and all video encoding formats are adapted.

[0105] The disclosed solution is fully compatible with the traditional video encoding and decoding transmission process. After special data arrangement and processing, the effect of audio and video code stream encoding and transmission with a transparent channel can be achieved. The disclosed solution proposes a transparent channel video data transmission solution that performs video data mixing before data compression and then performs video encoding and transmission, and no longer relies on fixed format video container transmission; the disclosed solution is not limited to VP8 or VP9 coding standards, but can be suitable for various mainstream coding standards such as H264, HEVC (H265), VP8, VP9, ​​SVAC, etc., which improves the universality of transparent channel video streams.

[0106] The present disclosure also provides a video processing device, Figure 5 Give a description.

[0107] Figure 5 FIG. 1 is a structural diagram of some embodiments of the video processing device disclosed in the present invention. Figure 5 As shown, the device 50 of this embodiment includes: a separation module 510 , a conversion module 520 , an arrangement module 530 , and a sending module 540 .

[0108] The separation module 510 is used to separate the image data into first color mode data and transparent channel data for each frame image of the video with a transparent channel, wherein the transparent channel data represents the transparency of the pixel.

[0109] The conversion module 520 is used to convert the first color mode data into the second color mode data, wherein the second color mode belongs to the color mode that can be processed by the video encoder.

[0110] In some embodiments, the conversion module 520 is used to convert the first color mode data into the second color mode data according to the first conversion matrix.

[0111] The arrangement module 530 is used to mix and arrange the second color mode data and the transparent channel data in a preset order to form data of an extended image corresponding to the image.

[0112] In some embodiments, the second color mode data includes first channel data, second channel data and third channel data; the arrangement module 530 is used to mix and arrange the second color mode data and the transparent channel data in the order of first channel data, transparent channel data, second channel data, third channel data, third channel data and second channel data.

[0113] In some embodiments, the first color mode data includes red, green, and blue (RGB) data, the transparent channel data includes alpha channel data, and the second color mode data includes YUV data of a preset sampling method. The arrangement module 530 is used to mix and arrange the YUV data of a preset sampling method and the Alpha channel data in the order of Y channel data, Alpha channel data, U channel data, V channel data, V channel data, and U channel data.

[0114] The sending module 540 is used to encode the data of the extended image corresponding to each frame image of the video, generate a video code stream and send it to the client, so that the client can restore each frame image according to the encoded data of the extended image corresponding to each frame image and play it.

[0115] In some embodiments, the sending module 540 is also used to store the data of the extended image corresponding to each frame image of the video into the memory; record the second color mode, and the width, height, starting address of the first channel data in the memory, and starting address of the first second channel data of the extended image corresponding to each frame image as the storage information of the video; read the data of the extended image corresponding to each frame image of the video from the memory according to the storage information of the video for encoding.

[0116] In some embodiments, the sending module 540 is used to determine a first length value and a second length value for each frame of image according to the second color mode and the width and height of the extended image corresponding to the image, wherein the first length value is the total length of the first channel data of the corresponding extended image as a whole of the first channel data and the transparent channel data in the image, and the second length value is the total length of the second channel data, the third channel data, the third channel data and the second channel data of the image as a whole as the second channel data and the third channel data of the corresponding extended image; according to the starting address of the first channel data in the memory, the starting address of the first second channel data, the first length value and the second length value, the first channel data, the second channel data and the third channel data of the extended image corresponding to the image are read from the memory respectively, and the extended image corresponding to the image is divided into multiple macroblocks; the multiple macroblocks of the extended image corresponding to the image are encoded according to a preset encoding method.

[0117] The sending module 540 may include a video encoder (which may be independently provided or integrated in the video processing device) for reading data of an extended image corresponding to each frame image of the video from the memory according to the storage information of the video for encoding.

[0118] The present disclosure also provides a client, Figure 6 Give a description.

[0119] Figure 6 FIG. 1 is a structural diagram of some embodiments of the client of the present disclosure. Figure 6 As shown, the client 60 of this embodiment includes: a receiving module 610 , a decoding module 620 , a separation module 630 , a conversion module 640 , a fusion module 650 , and a display module 660 .

[0120] The receiving module 610 is used to receive a code stream of a video, wherein the code stream includes the encoded data of the extended image corresponding to each frame image of the video.

[0121] The decoding module 620 is used to decode the code stream of the video to obtain the data of the extended image corresponding to each frame of the decoded image, wherein the data of the extended image is the data obtained by mixing and arranging the second color mode data and the transparent channel data of the image in a preset order.

[0122] In some embodiments, the second color mode data includes first channel data, second channel data and third channel data, and the data of the extended image is the second color mode data and the transparent channel data mixed and arranged in the order of first channel data, transparent channel data, second channel data, third channel data, third channel data and second channel data.

[0123] In some embodiments, the first color mode data includes red, green, and blue (RGB) data, the transparent channel data includes alpha channel data, the second color mode data includes YUV data of a preset sampling method, and the data of the extended image is the YUV data of a preset sampling method and the Alpha channel data mixed and arranged in the order of Y channel data, Alpha channel data, U channel data, V channel data, V channel data, and U channel data.

[0124] The separation module 630 is used to separate the data of the extended image corresponding to the decoded image into the second color mode data and the transparent channel data for each frame of the image.

[0125] In some embodiments, the separation module 630 is also used to obtain the second color mode and the width and height of the extended image corresponding to each frame of the image; for each frame of the image, according to the second color mode and the width and height of the extended image corresponding to each frame of the image, determine the length of the first channel data, the second channel data length and the third channel data length of the image, wherein the second color mode data includes the first channel data, the second channel data and the third channel data; divide the data of the extended image corresponding to the decoded image into two parts of data from the middle on average; according to the length of the first channel data, the second channel data length and the third channel data length, separate the first channel data, the second channel data and the third channel data from the former part of the data; according to the length of the first channel data, the second channel data length and the third channel data length, separate the transparent channel data, the third channel data and the second channel data from the latter part of the data, and discard the separated third channel data and second channel data.

[0126] The conversion module 640 is used to convert the second color mode data into the first color mode data for each frame of the image, wherein the first color mode data is the original color mode data of the image.

[0127] In some embodiments, the conversion module 640 is used to convert the second color mode data into the first color mode data according to the second conversion matrix.

[0128] The fusion module 650 is used to fuse the first color mode data and the transparent channel data for each frame of image to obtain data of an image with a transparent channel.

[0129] The display module 660 is used to render and display the data of the image with a transparent channel in each frame of the video.

[0130] The video processing device in the embodiment of the present disclosure can be implemented by various computing devices or computer systems. Figure 7 as well as Figure 7 Give a description.

[0131] Figure 7 FIG. 1 is a structural diagram of some embodiments of the video processing device disclosed in the present invention. Figure 7 As shown, the device 70 of this embodiment includes: a memory 710 and a processor 720 coupled to the memory 710 , and the processor 720 is configured to execute the video processing method in any of some embodiments of the present disclosure based on instructions stored in the memory 710 .

[0132] The memory 710 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0133] Figure 8 FIG. 1 is a structural diagram of some other embodiments of the video processing device disclosed in the present invention. Figure 8 As shown, the device 80 of this embodiment includes: a memory 810 and a processor 820, which are similar to the memory 710 and the processor 720 respectively. It can also include an input and output interface 830, a network interface 840, a storage interface 850, etc. These interfaces 830, 840, 850 and the memory 810 and the processor 820 can be connected, for example, through a bus 860. Among them, the input and output interface 830 provides a connection interface for input and output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 840 provides a connection interface for various networked devices, for example, it can be connected to a database server or a cloud storage server. The storage interface 850 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0134] The client in the embodiments of the present disclosure can be implemented by various computing devices or computer systems. Fig. 9 as well as Fig. 9 Give a description.

[0135] Fig. 9 FIG. 1 is a structural diagram of some embodiments of the client of the present disclosure. Fig. 9 As shown, the client 90 of this embodiment includes: a memory 910 and a processor 920 coupled to the memory 910 , and the processor 920 is configured to execute the video processing method in any of some embodiments of the present disclosure based on the instructions stored in the memory 910 .

[0136] The memory 910 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0137] Fig.10 FIG. 2 is a structural diagram of some other embodiments of the client disclosed in the present invention. Fig.10As shown, the client 100 of this embodiment includes: a memory 1010 and a processor 1020, which are similar to the memory 910 and the processor 920 respectively. It can also include an input and output interface 1030, a network interface 1040, a storage interface 1050, etc. These interfaces 1030, 1040, 1050 and the memory 1010 and the processor 1020 can be connected, for example, through a bus 1060. Among them, the input and output interface 1030 provides a connection interface for input and output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 1040 provides a connection interface for various networked devices, for example, it can be connected to a database server or a cloud storage server. The storage interface 1050 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0138] The present disclosure also provides a video processing system. Fig.11 Give a description.

[0139] Fig.11 FIG. 1 is a structural diagram of some embodiments of the video processing system disclosed in the present invention. Fig.10 As shown, the video processing system 11 of this embodiment includes: the video processing device 50 / 70 / 80 of any of the aforementioned embodiments and the client 60 / 90 / 100 of any of the aforementioned embodiments.

[0140] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0144] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A video processing method, comprising: For each frame image of the video with a transparent channel, separating the data of the image into first color mode data and transparent channel data, wherein the transparent channel data represents the transparency of the pixel; Converting the first color mode data into second color mode data, wherein the second color mode belongs to a color mode that can be processed by a video encoder; The second color mode data and the transparent channel data are mixed and arranged in a preset order to form data of an extended image corresponding to the image, including: the second color mode data and the transparent channel data are mixed and arranged in the order of first channel data, transparent channel data, second channel data, third channel data, third channel data, and second channel data, wherein the second color mode data includes first channel data, second channel data, and third channel data; The data of the extended image corresponding to each frame image of the video is encoded to generate a code stream of the video and send it to the client, so that the client can restore each frame image according to the encoded data of the extended image corresponding to each frame image and then play it.

2. The video processing method according to claim 1, further comprising: Storing the extended image data corresponding to each frame image of the video into a memory; Recording the second color mode, as well as the width and height of the extended image corresponding to each frame of the image, the starting address of the first channel data in the memory, and the starting address of the first second channel data as storage information of the video; The step of encoding the data of the extended image corresponding to each frame image of the video includes: The data of the extended image corresponding to each frame image of the video is read from the memory according to the storage information of the video for encoding.

3. The video processing method according to claim 2, wherein: The step of reading data of the extended image corresponding to each frame image of the video from the memory according to the storage information of the video for encoding comprises: For each frame of image, a first length value and a second length value are determined according to the second color mode and the width and height of the extended image corresponding to the image, wherein the first length value is the total length of the first channel data of the corresponding extended image taken as a whole by the first channel data and the transparent channel data in the image, and the second length value is the total length of the second channel data and the third channel data of the corresponding extended image taken as a whole by the second channel data, the third channel data, the third channel data and the second channel data of the image; According to the starting address of the first channel data in the memory, the starting address of the first second channel data, the first length value and the second length value, the first channel data, the second channel data and the third channel data of the extended image corresponding to the image are read from the memory respectively, and the extended image corresponding to the image is divided into a plurality of macroblocks; The multiple macroblocks of the extended image corresponding to the image are encoded according to a preset encoding method.

4. The video processing method according to claim 1, wherein: The first color mode data includes red, green and blue (RGB) data, the transparent channel data includes alpha channel data, the second color mode data includes YUV data in a preset sampling mode, and the mixing and arranging the second color mode data and the transparent channel data in the order of first channel data, transparent channel data, second channel data, third channel data, third channel data and second channel data includes: The YUV data of the preset sampling method and the Alpha channel data are mixed and arranged in the order of Y channel data, Alpha channel data, U channel data, V channel data, V channel data, and U channel data.

5. The video processing method according to claim 1, wherein: The converting the first color mode data into the second color mode data comprises: The first color mode data is converted into second color mode data according to a first conversion matrix.

6. A video processing method, comprising: Receive a code stream of a video, wherein the code stream includes encoded data of an extended image corresponding to each frame image of the video; Decoding the code stream of the video to obtain data of an extended image corresponding to each frame of the decoded image; For each frame of image, separating the data of the extended image corresponding to the decoded image into second color mode data and transparent channel data, wherein the second color mode data includes first channel data, second channel data and third channel data, and the data of the extended image is the data obtained by mixing and arranging the second color mode data and the transparent channel data in the order of first channel data, transparent channel data, second channel data, third channel data, third channel data and second channel data; For each frame of image, convert the second color mode data into first color mode data, wherein the first color mode data is the original color mode data of the image; For each frame of image, the first color mode data and the transparent channel data are merged to obtain data of an image with a transparent channel; The data of the image with the transparent channel in each frame of the video is used for rendering and display.

7. The video processing method according to claim 6, further comprising: Obtain the second color mode and the width and height of the extended image corresponding to each frame of the image; The step of separating the data of the extended image corresponding to the decoded image into the second color mode data and the transparent channel data for each frame of the image includes: For each frame of image, according to the second color mode and the width and height of the extended image corresponding to each frame of image, determine the length of the first channel data, the second channel data and the third channel data of the image, wherein the second color mode data includes the first channel data, the second channel data and the third channel data; Dividing the data of the extended image corresponding to the decoded image equally from the middle into two parts of data; Separating the first channel data, the second channel data and the third channel data from the first portion of data according to the length of the first channel data, the length of the second channel data and the length of the third channel data; According to the length of the first channel data, the length of the second channel data and the length of the third channel data, the transparent channel data, the third channel data and the second channel data are separated from the latter part of the data, and the separated third channel data and the second channel data are discarded.

8. The video processing method according to claim 6, wherein: The first color mode data includes red, green and blue (RGB) data, the transparent channel data includes alpha channel data, the second color mode data includes YUV data of a preset sampling method, and the data of the extended image is the YUV data of the preset sampling method and the Alpha channel data mixed and arranged in the order of Y channel data, Alpha channel data, U channel data, V channel data, V channel data and U channel data.

9. The video processing method according to claim 6, wherein: The converting the second color mode data into the first color mode data comprises: The second color mode data is converted into first color mode data according to a second conversion matrix.

10. A video processing device, comprising: A separation module, for separating the data of each frame of the video with a transparent channel into first color mode data and transparent channel data, wherein the transparent channel data represents the transparency of the pixel; A conversion module, used for converting the first color mode data into second color mode data, wherein the second color mode belongs to a color mode that can be processed by a video encoder; an arrangement module, used for mixing and arranging the second color mode data and the transparent channel data in a preset order to form data of an extended image corresponding to the image, comprising: mixing and arranging the second color mode data and the transparent channel data in the order of first channel data, transparent channel data, second channel data, third channel data, third channel data, and second channel data, wherein the second color mode data includes first channel data, second channel data, and third channel data; The sending module is used to encode the data of the extended image corresponding to each frame image of the video, generate the code stream of the video and send it to the client, so that the client can restore each frame image according to the encoded data of the extended image corresponding to each frame image and then play it.

11. A client, comprising: A receiving module, configured to receive a code stream of a video, wherein the code stream includes encoded data of an extended image corresponding to each frame image of the video; A decoding module, used to decode the code stream of the video to obtain data of an extended image corresponding to each frame of the decoded image; a separation module, for separating, for each frame of image, data of an extended image corresponding to the decoded image into second color mode data and transparent channel data, wherein the second color mode data includes first channel data, second channel data and third channel data, and the data of the extended image is data obtained by mixing and arranging the second color mode data and the transparent channel data in the order of first channel data, transparent channel data, second channel data, third channel data, third channel data and second channel data; A conversion module, configured to convert the second color mode data into first color mode data for each frame of image, wherein the first color mode data is original color mode data of the image; A fusion module, used for fusing the first color mode data and the transparent channel data for each frame of image to obtain data of an image with a transparent channel; The display module is used for rendering and displaying the data of the image with the transparent channel in each frame of the video.

12. A video processing device, comprising: processor; as well as A memory coupled to the processor is used to store instructions, and when the instructions are executed by the processor, the processor executes the video processing method according to any one of claims 1 to 5.

13. A client, comprising: processor; as well as A memory coupled to the processor is used to store instructions, and when the instructions are executed by the processor, the processor executes the video processing method as described in any one of claims 6-9.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

15. A video processing system, comprising: The video processing device according to claim 10 or 12 or the client according to claim 11 or 13.

Images