BOX structure video file and material fusion rendering method and device

Abstract

The invention discloses a BOX structure video file and material fusion rendering method and device. The method comprises the following steps: reading information combination of data blocks in a BOX structure video file to obtain predefined data block information; according to predefined data block information, constructing a to-be-fused texture containing an alpha channel for the to-be-fused material; for each video frame in the BOX structure video file, sequentially executing the following operations: converting the video frame into an initial texture containing an alpha channel, determining a to-be-displayed area of the initial texture according to video frame information contained in predefined data block information, and updating a color channel and the alpha channel of the to-be-displayed area according to each channel of the initial texture; and determining a fusion area of the to-be-fused texture in the to-be-displayed area according to the initial texture, updating an alpha channel of the to-be-fused texture according to each channel of the initial texture, and fusing the updated to-be-fused texture with the fusion area to obtain the to-be-displayed area which is fused and rendered with the to-be-fused material and displayed on the screen.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method and device for rendering a BOX structured video file and material fusion. Background Art

[0002] BOX structured video files, such as MP4 files, are a widely used multimedia container format that can be played on various devices and platforms, such as Windows, macOS, iOS, Android, smart TVs, browsers, etc.

[0003] However, when displaying special effects animations on BOX structured video files, there are the following disadvantages:

[0004] 1. Alpha channel is not supported. Alpha channel refers to the transparency and translucency of the image, which makes it impossible to achieve transparent and translucent effects.

[0005] 2. It only supports the special effects that come with the video, and does not support the additional information, Alpha channel, etc. contained in the integrated display material information. It also needs to be designed and developed separately for each material, which has poor flexibility and high cost. Summary of the invention

[0006] In view of the above problems, embodiments of the present application are proposed to provide a method and device for fusion rendering of BOX structured video files and materials that overcome the above problems or at least partially solve the above problems.

[0007] According to a first aspect of an embodiment of the present application, a method for fusion rendering of a BOX structured video file and a material is provided, which includes:

[0008] Read the information of each data block in the BOX structure video file and combine them to obtain the predefined data block information;

[0009] According to the material information contained in the predefined data block information, a texture to be fused including an alpha channel is constructed for the material to be fused;

[0010] For each video frame in the BOX structure video file, perform the following operations in sequence:

[0011] Convert the video frame into an initial texture including an alpha channel, determine the area to be displayed on the screen in the initial texture according to the video frame information included in the predefined data block information, and update the color channel and alpha channel of the area to be displayed on the screen according to each channel of the initial texture;

[0012] Determine the fusion area of the texture to be fused in the area to be displayed on the screen according to the initial texture, update the alpha channel of the texture to be fused according to each channel of the initial texture, and fuse the updated texture to be fused with the fusion area to obtain the area to be displayed on the screen after being fused and rendered with the material to be fused, and display it on the screen; the fusion area is located within the area to be displayed on the screen.

[0013] Optionally, read the BOX-structured video file, and the obtained predefined data block information further includes:

[0014] Read each data block in the BOX-structured video file, obtain the information of the predefined type in each data block, and splice them to obtain the predefined data block information; the predefined data block information is used to fuse each video frame in the BOX-structured video file with the material to be fused; the predefined data block information includes material information, video frame information, and / or rendering information.

[0015] Optionally, according to the material information included in the predefined data block information, constructing a texture to be fused including an alpha channel for the material to be fused further includes:

[0016] Construct a corresponding texture for the material to be fused according to the size of the material information in the predefined data block information, and store the identifier of the material to be fused and the corresponding texture in a key-value pair manner;

[0017] Obtain the material to be fused, and convert the material to be fused into a bitmap to be fused including an alpha channel; the alpha channel of the bitmap to be fused is set to an initial preset value;

[0018] Determine the corresponding texture according to the identifier of the material to be fused, and update the bitmap to be fused into the texture to obtain a texture to be fused including an alpha channel; the alpha channel of the texture to be fused is the initial preset value.

[0019] Optionally, converting the video frame into an initial texture including an alpha channel further includes:

[0020] Read the BOX-structured video file, sequentially obtain each video frame, and decode the video frame into image data in a preset format; the image data does not include an alpha channel;

[0021] Convert the image data into an initial texture including an alpha channel; the alpha channel of the initial texture is the initial preset value.

[0022] Optionally, according to the video frame information included in the predefined data block information, determine the area to be displayed on the screen in the initial texture, and update the color channel and the alpha channel of the area to be displayed on the screen according to each channel of the initial texture further includes:

[0023] According to the video frame information included in the predefined data block information, determine the first area and the second area of the initial texture, where the first area is used as the area to be displayed on the screen;

[0024] Set the color channels of the area to be displayed on the screen according to the color channels of the first area; the color channels include RGB channels.

[0025] Update and render the alpha channel of the area to be displayed on the screen according to the preset color channels of the second area; the preset color channels include the R channel.

[0026] Optionally, before determining the fusion area of the texture to be fused according to the initial texture, the method further includes:

[0027] Judge whether a video frame is included according to the rendering information contained in the predefined data block information;

[0028] If so, obtain the identifier of the material to be fused of the video frame, determine the corresponding texture to be fused, and determine the fusion area of the texture to be fused according to the initial texture;

[0029] If not, display the area to be displayed on the screen, obtain the next video frame for judgment until all video frames are judged.

[0030] Optionally, determining the fusion area of the texture to be fused in the area to be displayed on the screen according to the initial texture, and updating the alpha channel of the texture to be fused according to each channel of the initial texture further includes:

[0031] Calculate and determine the fusion area of the texture to be fused in the area to be displayed on the screen according to the third area of the initial texture and the area to be displayed on the screen;

[0032] Determine the fourth area of the texture to be fused in the initial texture according to the rendering information contained in the predefined data block information, scale the size of the fourth area to be the same as that of the texture to be fused, and then update and render the alpha channel of the texture to be fused according to the preset color channels of the fourth area.

[0033] Optionally, fusing the updated texture to be fused with the fusion area to obtain the area to be displayed on the screen after fusing and rendering with the material to be fused, and further including for displaying on the screen:

[0034] Scale the size of the updated texture to be fused to be the same as that of the fusion area, and then fuse it with the fusion area; wherein, each channel obtained after fusion is calculated according to each channel of the fusion area and each channel of the texture to be fused.

[0035] According to the second aspect of the embodiments of the present application, a BOX structure video file and material fusion rendering device is provided, which includes:

[0036] A data block module, adapted to read the information of each data block in the BOX structure video file and combine it to obtain predefined data block information;

[0037] The material module is adapted to construct a to-be-fused texture including an alpha channel for the to-be-fused material according to the material information included in the predefined data block information.

[0038] The loop module is adapted to sequentially execute the video frame processing module and the fusion module for each video frame in the BOX structure video file.

[0039] The video frame processing module is adapted to convert a video frame into an initial texture including an alpha channel, determine the to-be-displayed area in the initial texture according to the video frame information included in the predefined data block information, and update the color channel and the alpha channel of the to-be-displayed area according to each channel of the initial texture.

[0040] The fusion module is adapted to determine the fusion area of the to-be-fused texture in the to-be-displayed area according to the initial texture, update the alpha channel of the to-be-fused texture according to each channel of the initial texture, and fuse the updated to-be-fused texture with the fusion area to obtain the to-be-displayed area after being fused and rendered with the to-be-fused material for on-screen display; the fusion area is located within the to-be-displayed area.

[0041] According to the third aspect of the embodiments of the present application, a computing device is provided, including: a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface complete communication with each other through the communication bus;

[0042] The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the above BOX structure video file and material fusion rendering method.

[0043] According to the fourth aspect of the embodiments of the present application, a computer storage medium is provided, and at least one executable instruction is stored in the storage medium, and the executable instruction causes the processor to perform operations corresponding to the above BOX structure video file and material fusion rendering method.

[0044] According to the fifth aspect of the embodiments of the present application, a computer program product is provided, including at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the above BOX structure video file and material fusion rendering method.

[0045] The method and device for fusing and rendering a BOX-structured video file and materials provided by this application convert each video frame in the BOX-structured video file into an initial texture containing an alpha channel, construct the material to be fused into a texture to be fused containing an alpha channel, and update the alpha channels of the area to be displayed on the screen and the texture to be fused based on each channel of the initial texture, thereby enabling transparent and semi-transparent effects. Moreover, for each material to be fused, it can be converted into a texture to be fused and then fused with the area to be displayed on the screen, which can be applied to each material to be fused without the need for separate design for each material to be fused, making it more flexible and convenient.

[0046] The above description is only an overview of the technical solution of this application. In order to be able to understand the technical means of this application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of this application more obvious and understandable, the following specifically gives the specific implementation manners of this application. Description of the Drawings

[0047] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of this application. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0048] Figure 1 Shows a flowchart of a method for fusing and rendering a BOX-structured video file and materials according to an embodiment of this application;

[0049] Figure 2 Shows a flowchart of a method for fusing and rendering a BOX-structured video file and materials according to another embodiment of this application;

[0050] Figure 3a Shows a schematic diagram of the setting of the area to be displayed on the screen in the initial texture;

[0051] Figure 3b Shows a schematic diagram of the alpha channel area of the texture to be fused;

[0052] Figure 3c Shows a schematic diagram of the fusion area of the texture to be fused in the area to be displayed on the screen;

[0053] Figure 4 Shows a schematic diagram of the structure of a device for fusing and rendering a BOX-structured video file and materials according to an embodiment of this application;

[0054] Figure 5 Shows a schematic diagram of the structure of a computing device according to an embodiment of this application. Detailed Embodiments

[0055] Exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present application can be more thoroughly understood and the scope of the present application can be fully conveyed to those skilled in the art.

[0056] First, the noun terms related to one or more embodiments of the present application are explained.

[0057] JSON: JavaScript Object Notation, a notation for JavaScript objects, designed based on a subset of ECMAScript, an open standard file format and data exchange format, which is easy to read and write, and is also easy for machines to parse and generate.

[0058] BOX: The concept of BOX originated from atoms in QuickTime. A video file consists of individual Boxes, that is, it is a data block in a video file, composed of a Header and Data. A BOX can not only store audio-visual stream data but also store custom data.

[0059] YUV: A color encoding system, commonly used in the fields of video compression, image processing, and video transmission. By separating image information into luminance (Luma) and chrominance (Chroma) components, it can represent colors more efficiently and at the same time match the characteristics of the human visual system.

[0060] DirectX: A set of multimedia application programming interfaces (APIs) developed by Microsoft Corporation, mainly used for graphics rendering and multimedia processing under the Windows platform.

[0061] Texture: In computer graphics, a texture is an image or data used to represent the surface details of an object.

[0062] RGBA: A color space representing Red, Green, Blue, and Alpha.

[0063] Figure 1 The flowchart of the method for fusing and rendering a BOX-structured video file and materials according to an embodiment of the present application is shown, as Figure 1 shown, the method includes the following steps:

[0064] Step S101, read the information of each data block in the BOX-structured video file and combine it to obtain predefined data block information.

[0065] The BOX-structured video file includes multiple BOXes. Each BOX can be regarded as a data block of the video file, such as data recording each video frame, etc. Custom data can also be stored in the BOX. In this embodiment, predefined data required for fusion rendering can be stored in the BOX, such as setting video frames to be fused with the material, fusion rendering positions, etc.

[0066] By reading the information of each data block in the BOX-structured video file, the predefined data recorded in the BOX can be obtained, and the predefined data block information can be obtained by combining them. The predefined data block information can adopt formats such as JSON. For example, each predefined data is recorded using a JSON string. The predefined data block information can be used to fuse each video frame in the BOX-structured video file with the material to be fused. Based on the predefined data block information, it is convenient to process the video frames and the material to be fused subsequently, and fuse the processed video frames and the material to be fused for rendering to obtain the finally displayed video.

[0067] Step S102: According to the material information included in the predefined data block information, construct a texture to be fused with an alpha channel for the material to be fused.

[0068] The material can include various types, such as text, rich text, etc., like TXT text, or can be various pictures, such as compressed pictures, etc.

[0069] The material information included in the predefined data block information records the identifier of the material to be fused, type (such as picture, text, etc.), material loading method (such as local acquisition, network loading, etc.), material fusion content (such as fusion to replace the avatar, name, etc.), material size (such as height, width, etc.), adaptation method (such as scaling size, displaying in the original size, etc.), and can also include, for example, when the material is text, setting the text color, style, etc.

[0070] For the material to be fused, when it is text or a picture, to ensure that a transparent or semi-transparent effect can be set through the alpha channel when fusing with the video frame subsequently, the material to be fused can be first converted into a texture to be fused with an alpha channel. For example, the texture to be fused can be generated based on the material size in the material information, etc., to generate a texture of the corresponding size for the material to be fused in advance, convert the material to be fused into a picture with an alpha channel, and then update the picture with the alpha channel to the texture to obtain the texture to be fused with an alpha channel.

[0071] Step S103: For each video frame in the BOX-structured video file, convert the video frame into an initial texture with an alpha channel, determine the area to be displayed on the screen in the initial texture according to the video frame information included in the predefined data block information, and update the color channel and alpha channel of the area to be displayed on the screen according to each channel of the initial texture.

[0072] For a BOX-structured video file, an environment related to its decoder and renderer can be created. For example, a rendering environment can be built based on DirectX11 to parse the BOX-structured video file and obtain each video frame in the BOX-structured video file. Since the BOX-structured video file does not contain an alpha channel, the obtained video frames need to be converted into data containing an alpha channel first. For example, when parsing the BOX-structured video file, the obtained video frames are YUV420 data, and then the YUV420 data of the video frames are rendered into an initial texture containing an alpha channel through a rendering pipeline. The initial texture can adopt a format such as RGBA32, where A is the alpha channel. At this time, the alpha channel can be set to an initial preset value, such as 0, and corresponding values can be updated for the alpha channel subsequently to set transparent and semi-transparent special effects.

[0073] The initial texture can be recorded as YUV_TEX, and its size is YUV_SIZE, which contains the content to be displayed on the screen and the corresponding positions of the video frames to be fused with the material to be fused. The content to be displayed on the screen contains RGB color channels, and the data of the corresponding alpha channel is also included in the initial texture. For example, a certain area in the initial texture can be used as an alpha area to obtain the data of the corresponding alpha channel, etc. Specifically, the predefined data block information includes video frame information, and the video frame information includes information such as the version information of the BOX-structured video file, the number of frames, FPS (Frames Per Second), the size of the texture to be displayed on the screen (such as width, height, etc.), the size of the source video stream (such as width, height, etc.), and the relevant areas to be displayed on the screen (such as the RGB channel area, the alpha channel area, etc.). According to the video frame information included in the predefined data block information, the area to be displayed on the screen in the initial texture, such as the relevant areas to be displayed on the screen, can be determined. The area to be displayed on the screen is a part of the initial texture, and its RGBA channels can update the color channels and the alpha channel of the area to be displayed on the screen according to the channels of the initial texture. The area to be displayed on the screen can be the RGB channel area. The color channels of the area to be displayed on the screen are directly assigned according to the RGB-related data of the color channels in the RGB channel area of the initial texture, and the alpha channel of the area to be displayed on the screen is assigned according to the channel-related data of the alpha channel area in the initial texture, such as the R channel data, etc., so as to complete the rendering of the area to be displayed on the screen in the video frame. Since the alpha channel of the area to be displayed on the screen is updated, the area to be displayed on the screen can achieve effects such as transparency or semi-transparency.

[0074] Step S104: Determine the fusion area of the texture to be fused in the area to be displayed on the screen according to the initial texture, update the alpha channel of the texture to be fused according to the channels of the initial texture, and fuse the updated texture to be fused with the fusion area to obtain the area to be displayed on the screen after being fused and rendered with the material to be fused, and display it on the screen.

[0075] After rendering the alpha channel for the area to be displayed on the screen in the video frame, the texture to be fused in the video frame is determined according to the predefined data block information. One or more textures to be fused can be fused in a video frame, and each texture to be fused corresponds to its own fusion area, and the fusion area is located within the area to be displayed on the screen.

[0076] When determining the fusion area of the texture to be fused in the area to be displayed on the screen according to the initial texture, for example, in the RGB channel area of the area to be displayed on the screen, a partial area is taken as the fusion area, and the size of the fusion area can be calculated according to the size of the RGB channel area and the size of the texture to be fused. The RGB color channels of the texture to be fused itself are determined when the material to be fused is converted, and the alpha channel of the texture to be fused can be determined according to the channels of a partial area in the initial texture to ensure coordination and more natural fit when fused with the video frame. Specifically, the alpha channel of the texture to be fused can be updated based on the data of the channels of the initial texture. For example, a certain area A of the initial texture can be set in the predefined data block information, and according to the color channels of area A, such as the data of the R channel, the alpha channel of the texture to be fused is assigned a value to obtain the updated texture to be fused with RGBA channels. Then, the texture to be fused is fused with the fusion area. When fusing, if the size of the fusion area is inconsistent with the size of the texture to be fused, the size of the texture to be fused is scaled proportionally accordingly. After the size of the fusion area is consistent with the size of the texture to be fused, the fusion area and the texture to be fused are fused and rendered, so that the information such as text and pictures contained in the texture to be fused can be presented in the fusion area, thereby obtaining the area to be displayed on the screen after being fused and rendered with the material to be fused. The area to be displayed on the screen contains both the content of the video frame and the content of the material to be fused, and has an alpha channel, realizing a transparent or semi-transparent special effect, and can be displayed on the screen to present the fused and rendered effect.

[0077] Step S103 and step S104 can be executed cyclically until all video frames in the BOX structure video file are executed.

[0078] According to the BOX structure video file and material fusion rendering method provided by the present application, each video frame in the BOX structure video file is converted into an initial texture containing an alpha channel, the material to be fused is constructed into a texture to be fused containing an alpha channel, and the alpha channels of the area to be displayed on the screen and the texture to be fused are updated based on the channels of the initial texture, so that a transparent or semi-transparent effect can be realized. And for each material to be fused, it can be converted into a texture to be fused and then fused with the area to be displayed on the screen, which can be applied to each material to be fused, without the need to design separately for each material to be fused, which is more flexible and convenient.

[0079] Figure 2 Shows a flowchart of the BOX structure video file and material fusion rendering method according to an embodiment of the present application, asFigure 2 As shown in the figure, the method includes the following steps:

[0080] Step S201: Read each data block in the BOX-structured video file, obtain the information of the predefined type in each data block, and splice them to obtain the predefined data block information.

[0081] In the BOX-structured video file, information for material fusion can be preset in advance. For example, in the video frames recorded in each data block, the fusion area for material fusion, alpha channel settings, etc. can be set. In the BOX-structured video file, materials can be fused in each video frame or in some video frames. Specifically, corresponding information can be stored in each data block using the predefined type.

[0082] Before fusing the BOX-structured video file with the material, read each data block in the BOX-structured video file, obtain the information of the predefined type from each data block, and splice the information of the predefined type in each data block to obtain the predefined data block information. The predefined data block information is used to fuse each video frame in the BOX-structured video file with the material to be fused. Among them, the predefined data block information includes, for example, material information, video frame information, rendering information, etc. The material information records the relevant information of the material to be fused, such as the identifier of the material to be fused, type (such as picture, text, etc.), material loading method (such as local acquisition, network loading, etc.), material fusion content (such as fusing and replacing the avatar, name, etc.), material size (such as height, width, etc.), adaptation method (such as scaling size, displaying the original size, etc.), and can also include, for example, when the material is text, setting the text color, style, etc. The video frame information records the information of each video frame in the BOX-structured video file, such as the version information of the BOX-structured video file, the number of frames, FPS (Frames Per Second), the size of the on-screen texture (such as width, height, etc.), the size of the source video stream (such as width, height, etc.), the on-screen relevant area (such as RGB channel area, alpha channel area, etc.). Here, since each video frame in the BOX-structured video file does not contain an alpha channel and the transparency processing of the BOX-structured video file itself cannot be realized, in this embodiment, an alpha channel area is also set for each video frame itself to set an alpha channel for the video frame and achieve the transparent or semi-transparent effect of the video frame. The rendering information records the relevant information for the fusion rendering of the video frame and the material to be fused, such as which video frames are fused with the material to be fused, the fusion area, etc. The above is for illustrative purposes, and specific settings are made according to the implementation situation and are not limited here.

[0083] Step S202: Construct a texture to be fused with an alpha channel for the material to be fused according to the material information included in the predefined data block information.

[0084] The materials to be fused can include various types of materials such as text and pictures. To better fuse with the video frames in the BOX-structured video file and achieve a more suitable on-screen display effect, the materials to be fused can be constructed into a texture to be fused containing an alpha channel, which is convenient for subsequent fusion with the video frame data containing the alpha channel.

[0085] The texture to be fused contains the materials to be fused, which can be obtained through local acquisition or network acquisition, etc., for different types of materials to be fused. Convert the materials to be fused into a bitmap to be fused containing an alpha channel, such as an RGBA format bitmap. Among them, the alpha channel of the bitmap to be fused is set to an initial preset value, such as 0, and can be updated according to the texture in the video frame later. Construct a corresponding texture for the materials to be fused according to the size of the material information in the predefined data block information. When there are multiple materials to be fused, construct corresponding textures for each material to be fused respectively, and store the identifier of the material to be fused and the corresponding texture in a key-value pair manner, such as in the <key, tex> way, where key is the identifier of the material to be fused and tex is the corresponding texture. Construct corresponding textures for multiple materials to be fused at one time, which is convenient for subsequent fusion with the video frame. The corresponding texture can be directly used for fusion without having to process each material to be fused separately, saving processing time. After obtaining the texture, determine the corresponding texture according to the identifier of the material to be fused, and update the bitmap to be fused into the texture to obtain a texture to be fused containing an alpha channel. At this time, the alpha channel of the texture to be fused is the initial preset value. What is stored in the key-value pair is the updated texture to be fused.

[0086] Step S203, read the BOX-structured video file, sequentially obtain each video frame, decode the video frame into image data in a preset format, and convert the image data into an initial texture containing an alpha channel.

[0087] For the BOX-structured video file, sequentially read each video frame, decode the video frame to obtain image data in a preset format, such as YUV420 data, and the image data does not contain an alpha channel. According to the image data, convert it into an initial texture containing an alpha channel. For example, the initial texture is in the RGBA format, where the RGB color channels can be converted according to the YUV420 data. For example, R = Y + 1.13983 * (V - 128), G = Y - 0.39465 * (U - 128) - 0.58060 * (V - 128), B = Y + 2.03211 * (U - 128), etc. The above is for illustrative purposes, and specific conversion can be carried out according to the actual situation and is not limited here. The RGB colors in the initial texture can be correspondingly converted according to the image data of the video frame. The A channel, that is, the alpha channel, is set to an initial preset value, such as 0.

[0088] Step S204: Determine the area to be displayed on the screen in the initial texture according to the video frame information included in the predefined data block information, and update the color channels and alpha channels of the area to be displayed on the screen according to the channels of the initial texture.

[0089] After obtaining the initial texture, the initial texture contains RGBA channels, but the alpha channel is the initial preset value at this time. To achieve the transparent special effect on the screen, the alpha channel needs to be updated.

[0090] Specifically, according to the video frame information included in the predefined data block information, first determine the area to be displayed on the screen in the initial texture for each video frame, that is, the area to be displayed on the screen, and update the alpha channel for the area to be displayed on the screen. For example Figure 3a As shown in the initial texture, according to the video frame information included in the predefined data block information, determine the first area and the second area of the initial texture. The upper left red frame is the first area, and the lower left red frame is the second area. The first area is used as the area to be displayed on the screen. According to the color channels of the first area, the color channels of the area to be displayed on the screen can be directly set. The color channels include RGB channels. According to the preset color channels of the second area, such as the R channel, the alpha channel for rendering the area to be displayed on the screen can be updated.

[0091] When setting the channels of the area to be displayed on the screen, the following code can be used to process each pixel point one by one:

[0092] Texture2D tex_res_rgba; / / Initial texture

[0093] float4 output; / / output is the value of each pixel point in the area to be displayed on the screen

[0094] output.rgb = tex_res_rgba.Sample(rgb_pos).rgb; / / Take the rgb of the first area as the rgb of the area to be displayed on the screen

[0095] output.a = tex_res_rgba.Sample(aplah_pos).r; / / Take the r of the second area as the alpha of the area to be displayed on the screen

[0096] The above is for illustrative purposes. It is specifically set according to the actual situation and is not limited here.

[0097] Further, since the first region is the region to be displayed on the screen, its RGB channels can be directly assigned to the region to be displayed on the screen. However, there may be a problem that the size of the second region is inconsistent with the size of the region to be displayed on the screen. To ensure accuracy, before assignment, the size of the second region needs to be enlarged proportionally to make it the same as the size of the region to be displayed on the screen, and then sampling is performed. The data of the R channel of each pixel is assigned to the corresponding alpha channel of the region to be displayed on the screen. The settings of the first region and the second region can be set in the video frame information in advance, which is not limited here.

[0098] Through the above processing, the alpha channel of the region to be displayed on the screen in the video frame can be updated to achieve the transparent special effect processing of the region to be displayed on the screen.

[0099] Step S205, determine whether a video frame is included according to the rendering information contained in the predefined data block information.

[0100] After processing the region to be displayed on the screen of the video frame, it can be further determined whether the rendering information contained in the predefined data block information includes a video frame. If it does, it means that this video frame still needs to be fused and rendered with the material to be fused. If not, there is no need to continue processing, and the obtained region to be displayed on the screen can be directly displayed on the screen.

[0101] The rendering information records the frame number of the video frame, the corresponding identifier of the material to be fused, the fusion order, the rendering position, etc. By comparing the frame number of the video frame with the frame number recorded in the rendering information, it can be determined whether the rendering information includes a video frame. If so, execute step S206. If not, the region to be displayed on the screen is displayed on the screen, and after the display, step S203 (not shown in Figure 2 ) can be continued to obtain the next video frame for alpha channel update, and then it is determined whether to fuse and render with the material to be fused until all video frames are judged.

[0102] Step S206, obtain the identifier of the material to be fused of the video frame, determine the corresponding texture to be fused, determine the fusion region of the texture to be fused according to the initial texture, and update the alpha channel of the texture to be fused according to each channel of the initial texture.

[0103] When the video frame needs to be fused with the material to be fused, based on the frame number of the video frame, the identifier of the material to be fused is determined. According to the identifier of the material to be fused, the corresponding texture to be fused can be directly determined from the key-value pair.

[0104] The third region of the initial texture can be determined according to the rendering position and the like of the rendering information. Here, the third region is the position in the initial texture, and the fusion region of the texture to be fused in the region to be displayed on the screen also needs to be calculated and determined based on the region to be displayed on the screen. As Figure 3b shown, the 1 region in the upper left red frame is the third region in the initial texture.Figure 3c The 3 area in the red box in the middle is the fusion area in the area to be displayed on the screen. For example, area 1 is ATTACH_RGB_RECT, the area to be displayed on the screen is AIM_RECT, and the fusion area can be calculated by, for example, ATTACH_RGB_RECT - AIM_RECT.xy. Figure 3b The 2 area in the lower right red box in the middle is used to update the alpha channel of the texture to be fused. For example, determine the fourth area of the texture to be fused in the initial texture according to the rendering information included in the predefined data block information, that is Figure 3b For area 2 in the middle, considering that the size of the fourth area is inconsistent with the size of the texture to be fused, the size of the fourth area can be scaled to be the same as the texture to be fused, and then according to the preset color channels of the fourth area, such as the R channel, update the alpha channel of the texture to be fused. At this time, the texture to be fused has four complete RGBA channels.

[0105] Step S207, after scaling the size of the updated texture to be fused to be the same as the fusion area, fuse it with the fusion area to obtain the area to be displayed on the screen after fusing and rendering with the material to be fused, and display it on the screen.

[0106] After obtaining the texture to be fused with four complete RGBA channels, the updated texture to be fused can be fused and rendered into the area to be displayed on the screen for display on the screen.

[0107] During the fusion rendering, it is necessary to first scale the size of the texture to be fused to be the same as the fusion area and then fuse it with the fusion area. During the fusion, each channel is calculated according to the channels of the fusion area and the channels of the texture to be fused. For example, the fusion color = (the picture color of the fusion area in the area to be displayed on the screen) + (the texture color to be fused - the picture color of the fusion area in the area to be displayed on the screen) * the transparency of the texture to be fused. Obtain the fused area to be displayed on the screen, which contains both the video frame source picture information and the picture information of the material to be fused, and supports transparent special effects, and can better display the fused effect when displayed on the screen.

[0108] The fusion rendering of the BOX structure video file and the material can be applied to various video display scenarios, such as live broadcast, on-demand, etc. The information uploaded by the user can be used as the material to be fused with the BOX structure video file. While displaying the live broadcast and on-demand, user interaction, etc. can be realized, which is not limited here. The BOX structure video file can be, for example, an MP4 file, etc., which is not limited here.

[0109] According to the BOX structure video file and material fusion rendering method provided by the present application, the information of each data block in the BOX structure video file is read and combined to obtain predefined data block information. According to the predefined data block information, the video frame can be converted into an initial texture including an alpha channel, and the alpha channel of the area to be displayed on the screen is updated to achieve the transparent special effect when the video frame is displayed on the screen. According to the predefined data block information, a to-be-fused texture including an alpha channel is constructed for the to-be-fused material, and the alpha channel of the to-be-fused texture is updated by using the channels of the area of the initial texture to make it more compatible with the video frame picture. For each video frame, if there is no need to fuse with the to-be-fused material, the area to be displayed on the screen with the updated alpha channel can be directly displayed on the screen. If it is necessary to fuse with the to-be-fused material, the fusion area is determined, and the to-be-fused material is fused into the fusion area of the area to be displayed on the screen to achieve the fusion rendering with the material and include the transparent special effect.

[0110] Figure 4 FIG. shows a schematic structural diagram of a BOX structure video file and material fusion rendering device provided by an embodiment of the present application. As Figure 4 shown, the device includes:

[0111] A data block module 410, adapted to read the information of each data block in the BOX structure video file and combine it to obtain predefined data block information;

[0112] A material module 420, adapted to construct a to-be-fused texture including an alpha channel for the to-be-fused material according to the material information included in the predefined data block information;

[0113] A loop module 430, adapted to sequentially execute a video frame processing module 440 and a fusion module 450 for each video frame in the BOX structure video file;

[0114] A video frame processing module 440, adapted to convert the video frame into an initial texture including an alpha channel, determine the area to be displayed on the screen in the initial texture according to the video frame information included in the predefined data block information, and update the color channel and the alpha channel of the area to be displayed on the screen according to each channel of the initial texture;

[0115] A fusion module 450, adapted to determine the fusion area of the to-be-fused texture in the area to be displayed on the screen according to the initial texture, update the alpha channel of the to-be-fused texture according to each channel of the initial texture, and fuse the updated to-be-fused texture with the fusion area to obtain the area to be displayed on the screen after being fused and rendered with the to-be-fused material for display on the screen; the fusion area is located within the area to be displayed on the screen.

[0116] Optionally, the data block module 410 is further adapted to:

[0117] Read each data block in the BOX-structured video file, obtain the information of predefined types in each data block, and splice to obtain the predefined data block information; the predefined data block information is used to fuse each video frame in the BOX-structured video file with the material to be fused; the predefined data block information includes material information, video frame information, and / or rendering information.

[0118] Optionally, the material module 420 is further adapted to:

[0119] Construct a corresponding texture for the material to be fused according to the size of the material information in the predefined data block information, and store the identifier of the material to be fused and the corresponding texture in the form of key-value pairs;

[0120] Obtain the material to be fused, and convert the material to be fused into a bitmap to be fused including an alpha channel; the alpha channel of the bitmap to be fused is set to an initial preset value;

[0121] Determine the corresponding texture according to the identifier of the material to be fused, and update the bitmap to be fused into the texture to obtain a texture to be fused including an alpha channel; the alpha channel of the texture to be fused is the initial preset value.

[0122] Optionally, the video frame processing module 440 is further adapted to:

[0123] Read the BOX-structured video file, sequentially obtain each video frame, and decode the video frame into image data in a preset format; the image data does not include an alpha channel;

[0124] Convert the image data into an initial texture including an alpha channel; the alpha channel of the initial texture is the initial preset value.

[0125] Optionally, the video frame processing module 440 is further adapted to:

[0126] Determine the first region and the second region of the initial texture according to the video frame information included in the predefined data block information, where the first region is used as the region to be displayed on the screen;

[0127] Set the color channels of the region to be displayed on the screen according to the color channels of the first region; the color channels include RGB channels;

[0128] Update the alpha channel of the region to be displayed on the screen for rendering according to the preset color channels of the second region; the preset color channels include the R channel.

[0129] Optionally, the apparatus further includes: a determination module 460, configured to determine whether a video frame is included according to the rendering information included in the predefined data block information; if so, obtain an identifier of the material to be fused for the video frame, determine a corresponding texture to be fused, and determine a fusion area of the texture to be fused according to the initial texture; if not, perform on-screen display on the area to be displayed on the screen, obtain the next video frame for determination until all video frames are determined.

[0130] Optionally, the fusion module 450 is further configured to:

[0131] Calculate and determine a fusion area of the texture to be fused in the area to be displayed on the screen according to the third area of the initial texture and the area to be displayed on the screen;

[0132] Determine a fourth area of the texture to be fused in the initial texture according to the rendering information included in the predefined data block information, scale the size of the fourth area to be the same as that of the texture to be fused, and update the alpha channel of the texture to be fused for rendering according to the preset color channels of the fourth area.

[0133] Optionally, the fusion module 450 is further configured to:

[0134] After scaling the size of the updated texture to be fused to be the same as that of the fusion area, fuse it with the fusion area; wherein, each channel obtained after fusion is calculated according to each channel of the fusion area and each channel of the texture to be fused.

[0135] The descriptions of the above modules refer to the corresponding descriptions in the method embodiments and will not be elaborated here.

[0136] According to the BOX structure video file and material fusion rendering apparatus provided by the present application, each video frame in the BOX structure video file is converted into an initial texture including an alpha channel, the material to be fused is constructed into a texture to be fused including an alpha channel, and the alpha channels of the area to be displayed on the screen and the texture to be fused are updated based on each channel of the initial texture, so that transparent and semi-transparent effects can be achieved. Moreover, for each material to be fused, it can be converted into a texture to be fused and then fused with the area to be displayed on the screen, which can be applicable to each material to be fused, without the need to design separately for each material to be fused, and is more flexible and convenient.

[0137] The present application further provides a non-volatile computer storage medium, and the computer storage medium stores at least one executable instruction, and the executable instruction can execute the operations corresponding to the BOX structure video file and material fusion rendering method in any of the above method embodiments.

[0138] The present application further provides a computer program product, and the computer program product includes at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to execute the operations corresponding to the BOX structure video file and material fusion rendering method in any of the above method embodiments.

[0139] Figure 5 The structural schematic diagram of a computing device according to an embodiment of the present application is shown. The specific embodiments of the present application do not limit the specific implementation of the computing device.

[0140] As Figure 5 shown, the computing device may include: a processor 502, a communications interface 504, a memory 506, and a communication bus 508.

[0141] Wherein:

[0142] The processor 502, the communications interface 504, and the memory 506 communicate with each other through the communication bus 508.

[0143] The communications interface 504 is used to communicate with network elements of other devices such as clients or other servers.

[0144] The processor 502 is used to execute the program 510, and specifically can execute the relevant steps in the embodiment of the method for fusing and rendering the BOX structure video file and materials.

[0145] Specifically, the program 510 may include program code, and the program code includes computer operation instructions.

[0146] The processor 502 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the present application. One or more processors included in the computing device may be of the same type of processor, such as one or more CPUs; or may be of different types of processors, such as one or more CPUs and one or more ASICs.

[0147] The memory 506 is used to store the program 510. The memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

[0148] The program 510 can specifically be used to cause the processor 502 to execute the BOX structure video file and material fusion rendering method in any of the above method embodiments. For the specific implementation of each step in the program 510, reference can be made to the corresponding descriptions in the corresponding steps and units in the above BOX structure video file and material fusion rendering embodiments, which will not be elaborated here. Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the foregoing method embodiments, which will not be repeated here.

[0149] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings provided herein. Based on the above description, the structure required to construct such a system is obvious. In addition, the present application is not directed to any specific programming language. It should be understood that the content of the present application described herein can be implemented using various programming languages, and the description of a specific language above is for the purpose of disclosing the preferred embodiments of the present application.

[0150] In the specification provided herein, a large number of specific details are set forth. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and technologies have not been shown in detail so as not to obscure the understanding of this specification.

[0151] Similarly, it should be understood that, for the purpose of streamlining the present application and assisting in understanding one or more of the various inventive aspects, in the foregoing description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together in a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting the intention that the claimed subject matter of the present application requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, the inventive aspects lie in less than all the features of the single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate embodiment of the present application.

[0152] Those skilled in the art can understand that the modules in the devices in the embodiments can be adaptively changed and set in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition, they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination can be adopted to combine all the features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all the processes or units of any method or device so disclosed. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) can be replaced by an alternative feature that provides the same, equivalent, or similar purpose.

[0153] In addition, those skilled in the art can understand that although some of the embodiments herein include certain features included in other embodiments rather than other features, the combination of the features of different embodiments means that it is within the scope of this application and forms different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

[0154] Each component embodiment of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components according to the present application. The present application can also be implemented as a device or apparatus program (for example, a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0155] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In the unit claims listing several devices, several of these devices may be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not denote any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for fusion rendering of a BOX structure video file and material, comprising: Read the information of each data block in the BOX structure video file, and combine them to obtain predefined data block information; Constructing a texture to be fused including an alpha channel for the material to be fused according to the material information included in the predefined data block information; For each video frame in the BOX structure video file, perform the following operations in sequence: Convert the video frame into an initial texture including an alpha channel, determine a to-be-screened area in the initial texture according to the video frame information included in the predefined data block information, and update a color channel and an alpha channel of the to-be-screened area according to each channel of the initial texture; Determine a fusion region of the texture to be fused in the area to be displayed on the screen according to the initial texture, update the alpha channel of the texture to be fused according to each channel of the initial texture, fuse the updated texture to be fused with the fusion region, obtain the area to be displayed on the screen after fusion and rendering with the material to be fused, and display it on the screen; The fusion area is located in the area to be placed on the screen.

2. The method according to claim 1, wherein: The step of reading the BOX structure video file and obtaining the predefined data block information further includes: Read each data block in the BOX structure video file, obtain information of a predefined type in each data block, and splice to obtain predefined data block information; the predefined data block information is used to fuse each video frame in the BOX structure video file with the material to be fused; the predefined data block information includes material information, video frame information and / or rendering information.

3. The method according to claim 1 or 2, wherein: The step of constructing a texture to be fused including an alpha channel for the material to be fused according to the material information included in the predefined data block information further includes: Constructing a corresponding texture for the material to be fused according to the size of the material information in the predefined data block information, and storing the identifier of the material to be fused and the corresponding texture in a key-value pair manner; Acquire the material to be fused, and convert the material to be fused into a bitmap to be fused containing an alpha channel; the alpha channel of the bitmap to be fused is set to an initial preset value; The corresponding texture is determined according to the identifier of the material to be fused, and the bitmap to be fused is updated into the texture to obtain the texture to be fused including an alpha channel; the alpha channel of the texture to be fused is an initial preset value.

4. The method according to any one of claims 1 to 3, wherein: The converting the video frame into an initial texture including an alpha channel further comprises: Read the BOX structure video file, obtain each video frame in sequence, and decode the video frame into image data in a preset format; the image data does not include an alpha channel; The image data is converted into an initial texture including an alpha channel; the alpha channel of the initial texture is an initial preset value.

5. The method according to any one of claims 1 to 4, wherein: The step of determining the area to be put on screen in the initial texture according to the video frame information included in the predefined data block information, and updating the color channel and the alpha channel of the area to be put on screen according to each channel of the initial texture further comprises: Determine a first area and a second area of ​​the initial texture according to the video frame information included in the predefined data block information, wherein the first area is used as the area to be displayed on the screen; According to the color channel of the first area, setting the color channel of the area to be displayed on the screen; the color channel includes RGB channels; The alpha channel of the area to be displayed on the screen is updated and rendered according to a preset color channel of the second area; the preset color channel includes an R channel.

6. The method according to any one of claims 1 to 5, wherein: Before determining the fusion area of ​​the texture to be fused according to the initial texture, the method further includes: Determining whether the video frame is included according to the rendering information included in the predefined data block information; If yes, obtain the identifier of the to-be-fused material of the video frame, determine the corresponding to-be-fused texture, and determine the fusion area of ​​the to-be-fused texture according to the initial texture; If not, the area to be put on the screen is put on the screen and the next video frame is obtained for judgment until all video frames are judged.

7. The method according to any one of claims 1 to 6, wherein: The step of determining the fusion area of ​​the texture to be fused in the area to be put on the screen according to the initial texture, and updating the alpha channel of the texture to be fused according to each channel of the initial texture further comprises: According to the third area of ​​the initial texture and the area to be put on the screen, a fusion area of ​​the texture to be fused in the area to be put on the screen is calculated and determined; A fourth region of the texture to be fused in the initial texture is determined according to the rendering information included in the predefined data block information, a size of the fourth region is scaled to be consistent with the texture to be fused, and an alpha channel of the texture to be fused is updated and rendered according to a preset color channel of the fourth region.

8. The method according to any one of claims 1 to 7, wherein: The step of fusing the updated texture to be fused with the fusion area to obtain the area to be displayed on the screen after fusion and rendering with the material to be fused, and displaying on the screen further comprises: After scaling the size of the updated texture to be fused to be consistent with the fusion area, the texture is fused with the fusion area; wherein each channel obtained after the fusion is calculated based on each channel of the fusion area and each channel of the texture to be fused.

9. A BOX structure video file and material fusion rendering device, comprising: A data block module, adapted to read information of each data block in the BOX structure video file and combine them to obtain predefined data block information; A material module, adapted to construct a texture to be fused including an alpha channel for the material to be fused according to the material information included in the predefined data block information; A loop module, adapted to sequentially execute a video frame processing module and a fusion module for each video frame in the BOX structure video file; a video frame processing module, adapted to convert the video frame into an initial texture including an alpha channel, determine a to-be-screened area in the initial texture according to the video frame information included in the predefined data block information, and update a color channel and an alpha channel of the to-be-screened area according to each channel of the initial texture; a fusion module, adapted to determine a fusion region of the texture to be fused in the area to be displayed on the screen according to the initial texture, and update the alpha channel of the texture to be fused according to each channel of the initial texture, and fuse the updated texture to be fused with the fusion region to obtain the area to be displayed on the screen after fusion and rendering with the material to be fused, and display it on the screen; The fusion area is located in the area to be placed on the screen.

10. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the BOX structure video file and material fusion rendering method according to any one of claims 1 to 8.

11. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and wherein the executable instruction enables a processor to execute operations corresponding to the BOX structure video file and material fusion rendering method according to any one of claims 1 to 8.

12. A computer program product, comprising at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the BOX structure video file and material fusion rendering method according to any one of claims 1 to 8.

Images