Video rendering method and device, storage medium, and electronic device

Abstract

The present disclosure belongs to the field of video processing technology, and relates to a video rendering method and device, a storage medium, and an electronic device. The method includes: obtaining a first texture of a first video, and creating a shared environment for the first video, and creating a shared object for the first video, wherein the shared environment is a rendering environment on which the indirectly shared first texture depends, and the shared object is an object required to expose the first texture to the outside for indirect sharing; based on the shared environment, drawing the first texture in the shared object; calling the first texture to the second video corresponding to the first video through the shared object, so as to perform mixed rendering of the first video and the second video to obtain a mixed video. The present disclosure provides a safe and efficient way to share video textures, which avoids the situation where direct sharing causes the texture to be modified, prevents the mixed video from generating a black screen or flickering in advance, ensures the normal display of the color saturation and clarity of the original video, and optimizes the visual effect of the mixed video.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a video rendering method and device, a computer-readable storage medium, and an electronic device. Background Art

[0002] With the continuous development of computer technology, MP4 (Moving Picture Experts Group 4) video rendering technology with transparent channels has become relatively mature. For example, the open source project AlphaPlayer (a video animation special effects SDK (Software Development Kit) used by the live broadcast center) can support the rendering of MP4 videos with transparent channel information, or VAP (Video Animation Player, an implementation solution for playing cool animations) supports MP4 videos combined with image rendering solutions, which can be used to play special effects animation implementation solutions. Compared with dynamic image solutions such as Webp (picture format) and Apng (Animated Portable Network Graphics, bitmap animation format), this method has the advantage of high compression rate due to smaller materials, and faster decoding is achieved through hardware decoding.

[0003] AlphaPlayer only supports playing videos with transparency, so it only plays and renders MP4 videos. VAP supports playing and rendering transparent videos, and also adds the ability to render images and text. However, neither solution supports mixed video and video rendering.

[0004] In view of this, there is an urgent need in this field to develop a new video rendering method and device.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0006] The purpose of the present disclosure is to provide a video rendering method, a video rendering device, a computer-readable storage medium and an electronic device, thereby overcoming, at least to a certain extent, the technical problem of being unable to realize video texture sharing and two-video mixed rendering due to the limitations of related technologies.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0008] According to a first aspect of an embodiment of the present invention, a video rendering method is provided, the method comprising:

[0009] Obtaining a first texture of a first video, creating a shared environment for the first video, and creating a shared object for the first video, wherein the shared environment is a rendering environment on which the indirectly shared first texture depends, and the shared object is an object required to expose the first texture to the outside for indirect sharing;

[0010] Drawing the first texture in the shared object based on the shared environment;

[0011] The first texture is called to a second video corresponding to the first video through the shared object, so as to perform mixed rendering on the first video and the second video to obtain a mixed video.

[0012] In an exemplary embodiment of the present invention, creating a sharing environment for the first video includes:

[0013] Determine that the context object of the first video is a shared context object, and create a rendering thread;

[0014] Based on the rendering thread, a shared environment is created using the shared context object.

[0015] In an exemplary embodiment of the present invention, drawing the first texture in the shared object based on the shared environment includes:

[0016] Creating a temporary frame buffer, and binding the first texture and the temporary frame buffer to obtain a first binding relationship;

[0017] Based on the shared environment, the first texture is drawn to the shared object using the first binding relationship.

[0018] In an exemplary embodiment of the present invention, drawing the first texture to the shared object using the first binding relationship based on the shared environment includes:

[0019] Creating a current texture identifier, and binding the temporary frame buffer and the current texture identifier to obtain a second binding relationship;

[0020] Based on the first binding relationship and the second binding relationship, drawing the first texture to the current texture identifier bound to the temporary frame buffer;

[0021] Based on the shared environment, the first texture is drawn to the shared object according to the current texture identifier.

[0022] In an exemplary embodiment of the present invention, drawing the first texture to the shared object according to the current texture identifier includes:

[0023] creating a data storage area using the shared object;

[0024] Based on the data storage area, the first texture is drawn to the shared object according to the current texture identifier.

[0025] In an exemplary embodiment of the present invention, after drawing the first texture to the shared object according to the current texture identifier, the method further includes:

[0026] Sending the first texture to the rendering thread.

[0027] In an exemplary embodiment of the present invention, after calling the first texture to a second video corresponding to the first video through the shared object, the method further includes:

[0028] The shared context of the first texture is released using the shared object.

[0029] In an exemplary embodiment of the present invention, calling the first texture to a second video corresponding to the first video through the shared object includes:

[0030] Creating an object texture identifier, and binding the shared object and the object texture identifier to obtain a third binding relationship;

[0031] assigning the object texture identifier to the first texture drawn into the shared object according to the third binding relationship;

[0032] When it is monitored that the first texture is drawn in the shared object, the first texture is called to the second video corresponding to the first video using the object texture identifier.

[0033] In an exemplary embodiment of the present invention, the step of performing mixed rendering on the first video and the second video to obtain a mixed video includes:

[0034] When it is determined according to the second video that the first video in the mixed video is transparent, rendering the second video to obtain a mixed video;

[0035] When it is determined according to the second video that the first video in the mixed video is non-transparent, the first video and the second video are mixed and rendered to obtain a mixed video.

[0036] In an exemplary embodiment of the present invention, the step of performing mixed rendering on the first video and the second video to obtain a mixed video includes:

[0037] Obtaining a first vertex position and a first texture pixel value of the first video, and performing a matrix transformation on the first vertex position to obtain a mixed display position;

[0038] The second vertex position and the second texture pixel value of the second video are obtained, and the first video and the second video are mixed and rendered according to the mixed display position, the first texture pixel value, the second vertex position and the second texture pixel value to obtain a mixed video.

[0039] In an exemplary embodiment of the present invention, after obtaining the second vertex position and the second texture pixel value of the second video, the method further includes:

[0040] When the first texture is not called to the second video corresponding to the first video through the shared object, obtaining a default image and image vertex positions and image pixel values of the default image;

[0041] The default image and the second video are mixed and rendered according to the image vertex position, the image pixel value, the second vertex position and the second texture pixel value to obtain a mixed video.

[0042] In an exemplary embodiment of the present invention, the step of performing mixed rendering on the first video and the second video according to the mixed display position, the first texture pixel value, the second vertex position, and the second texture pixel value to obtain a mixed video includes:

[0043] Obtaining a target transparency and a target color value according to the first texture pixel value and the second texture pixel value;

[0044] Obtaining a mixing position and a mixing size of the first video in the second video;

[0045] Based on the mixed display position, the second vertex position, the mixed position and the mixed size, the first video and the second video are mixed and rendered according to the target transparency value and the target color value to obtain a mixed video.

[0046] In an exemplary embodiment of the present invention, obtaining a target transparency and a target color value according to the first texture pixel value and the second texture pixel value includes:

[0047] Performing two-dimensional sampling on the first texture pixel value to obtain a first sampled pixel value, and performing two-dimensional sampling on the second texture pixel value to obtain a second sampled pixel value;

[0048] Performing a difference calculation on the first sampled pixel value and the second sampled pixel value to obtain a target transparency value;

[0049] A target color value is obtained by performing a linear difference calculation on the first sampled pixel value and the second sampled pixel value.

[0050] According to a second aspect of an embodiment of the present invention, there is provided a video rendering apparatus, comprising:

[0051] a texture acquisition module configured to acquire a first texture of a first video and create a shared environment and a shared object of the first video;

[0052] an indirect sharing module, configured to draw the first texture in the shared object based on the shared environment;

[0053] The hybrid rendering module is configured to call the first texture to the second video corresponding to the first video through the shared object, so as to perform hybrid rendering on the first video and the second video to obtain a hybrid video.

[0054] According to a third aspect of an embodiment of the present invention, an electronic device is provided, comprising: a processor and a memory; wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the video rendering method in any of the above exemplary embodiments is implemented.

[0055] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the video rendering method in any of the above exemplary embodiments is implemented.

[0056] As can be seen from the above technical solutions, the video rendering method, video rendering device, computer storage medium, and electronic device in the exemplary embodiments of the present disclosure have at least the following advantages and positive effects:

[0057] In the method and apparatus provided by the exemplary embodiments of the present disclosure, on the one hand, indirect sharing of the first texture of the first video is achieved through a shared environment and a shared object, providing a safe and efficient way to share video textures, avoiding the problem of changes in the first texture due to direct sharing, and preventing the mixed video from producing black screens or flickering in advance; on the other hand, mixed rendering of the first video and the second video is provided, providing a method that supports both video rendering with transparency and mixing of two videos, ensuring the normal display of the color saturation and clarity of the original video, and can also perfectly blend the two videos smoothly and naturally, greatly optimizing the visual effect of the mixed video and enriching the application scenarios of the mixed video.

[0058] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0060] Figure 1 A schematic diagram schematically illustrates a flow chart of a video rendering method in an exemplary embodiment of the present disclosure;

[0061] Figure 2 A schematic flow chart of a method for creating a shared environment for a first video in an exemplary embodiment of the present disclosure is schematically shown;

[0062] Figure 3 Schematically shows a code diagram for creating a new EGLContext in an exemplary embodiment of the present disclosure;

[0063] Figure 4 Schematically illustrates a code diagram for implementing creation of a shared object in an exemplary embodiment of the present disclosure;

[0064] Figure 5 A flowchart schematically illustrating a method for drawing a first texture in a shared object in an exemplary embodiment of the present disclosure is shown;

[0065] Figure 6 A schematic flow chart of a method for drawing a first texture using a first binding relationship in an exemplary embodiment of the present disclosure is schematically shown;

[0066] Figure 7 A schematic flow chart of a method for drawing a first texture according to a current texture identifier in an exemplary embodiment of the present disclosure is schematically shown;

[0067] Figure 8 Schematically illustrates a code diagram for creating a data storage area by using a shared object in an exemplary embodiment of the present disclosure;

[0068] Figure 9 A schematic flow chart schematically illustrates a method for calling a first texture into a second video in an exemplary embodiment of the present disclosure;

[0069] Figure 10 A schematic flow chart schematically illustrates a method for performing mixed rendering on a first video and a second video in an exemplary embodiment of the present disclosure;

[0070] Figure 11 A scene effect diagram schematically showing an original picture of the second video in an exemplary embodiment of the present disclosure;

[0071] Figure 12 A schematic flow chart schematically illustrates a method for further performing mixed rendering on a first video and a second video in an exemplary embodiment of the present disclosure;

[0072] Figure 13 Schematically illustrating a partial code diagram for obtaining a mixed display position and a first texture pixel value in an exemplary embodiment of the present disclosure;

[0073] Figure 14 Schematically illustrates a code diagram for obtaining the second vertex position in an exemplary embodiment of the present disclosure;

[0074] Figure 15 A schematic diagram of a code for obtaining a second texture pixel value in an exemplary embodiment of the present disclosure is schematically shown;

[0075] Figure 16 A schematic flow chart schematically illustrates a further method for performing mixed rendering on a first video and a second video in an exemplary embodiment of the present disclosure;

[0076] Figure 17 A diagram schematically illustrating a method for calculating transparency in an exemplary embodiment of the present disclosure;

[0077] Figure 18 A schematic diagram schematically illustrates a method for calculating target transparency and target color values in an exemplary embodiment of the present disclosure;

[0078] Figure 19 A schematic diagram of a code for a fragment shader performing two-dimensional sampling in an exemplary embodiment of the present disclosure is schematically shown;

[0079] Figure 20 A schematic diagram schematically illustrates a flow chart of a method for mixed rendering of a default image and a second video in an exemplary embodiment of the present disclosure;

[0080] Figure 21 Schematically showing a scene effect diagram of a mixed video obtained by mixed rendering in an exemplary embodiment of the present disclosure;

[0081] Figure 22 A schematic structural diagram of a video rendering device in an exemplary embodiment of the present disclosure is shown schematically;

[0082] Figure 23 Schematically illustrating an electronic device for implementing a video rendering method in an exemplary embodiment of the present disclosure;

[0083] Figure 24 A computer-readable storage medium for implementing a video rendering method in an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0084] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0085] The terms "a", "an", "the" and "said" are used in this specification to indicate the presence of one or more elements / components / etc.; the terms "including" and "having" are used to express open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first" and "second" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0086] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the drawings represent identical or similar parts, and thus repeated descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically separate entities.

[0087] With the continuous development of computer technology, MP4 video rendering technology with transparent channels has become relatively mature. For example, the open source project AlphaPlayer supports rendering MP4 videos with transparent channel information, and VAP supports MP4 video combined with image rendering, which can be used to implement special effects animations. Compared with dynamic image solutions such as Webp and Apng, this method has the advantages of smaller materials and high compression rates. In addition, hardware decoding achieves faster decoding.

[0088] AlphaPlayer only supports playing videos with transparency, so it only plays and renders MP4 videos. VAP supports playing and rendering transparent videos, and also adds the ability to render images and text. However, neither solution supports mixed video and video rendering.

[0089] In addition, existing methods for sharing video screens include transferring the screen of Video A in the form of an image (bitmap) to the corresponding rendering view of another player, such as GLSurfaceView (an OpenGL (Open Graphics Library) class), for image blending rendering. In other words, the screen of Video A is copied by taking a screenshot, then converted into a bitmap for use in rendering the second video. The blending method is to achieve a layer blending effect using texture mapping.

[0090] Among them, OpenGL is the industrial standard in the field of graphics. It is a set of cross-programming language, cross-platform, professional graphics programming (software) interfaces, and is also the underlying graphics library.

[0091] However, this approach requires the GPU (graphics processing unit) to copy each frame of video A, convert it into a bitmap, and then copy it from the CPU (central processing unit) to the GPU to convert it into a texture for rendering. This entire process is very performance-intensive and can easily cause the device to overheat.

[0092] In addition, you can also directly use the SurfaceTexture of video A to transfer the picture data. When video B is rendered, get the image frame from the SurfaceTexture.

[0093] Among them, SurfaceTexture is the core component for rendering on the Android system. It is a combination of Surface and OpenGL ES texture (a texture type of the Android platform, also known as OES texture). It is used to provide Surface for outputting OES texture and is a bridge connecting Surface and OES texture.

[0094] However, this method of directly sharing a SurfaceTexture will have some problems, such as black screen, flickering, etc. Multiple threads accessing the same data source may not be able to obtain video data, resulting in abnormal images.

[0095] In response to the problems existing in the related art, the present disclosure proposes a video rendering method. Figure 1 A flow chart of a video rendering method is shown, as Figure 1 As shown, the video rendering method includes at least the following steps:

[0096] Step S110. Obtain the first texture of the first video, create a shared environment for the first video, and create a shared object for the first video. The shared environment is the rendering environment on which the indirectly shared first texture depends, and the shared object is the object required to expose the first texture to the outside for indirect sharing.

[0097] Step S120: Based on the shared environment, draw the first texture in the shared object.

[0098] Step S130: Call the first texture to the second video corresponding to the first video through the shared object, so as to perform mixed rendering on the first video and the second video to obtain a mixed video.

[0099] In an exemplary embodiment of the present disclosure, on the one hand, indirect sharing of the first texture of the first video is achieved through a shared environment and a shared object, providing a safe and efficient way to share video textures, avoiding the problem of changes in the first texture due to direct sharing, and preventing the mixed video from generating black screens or flickering in advance; on the other hand, mixed rendering is performed on the first video and the second video, providing a method that supports both video rendering with transparency and mixing of two videos, ensuring the normal display of the color saturation and clarity of the original video, and can also perfectly blend the two videos smoothly and naturally, greatly optimizing the visual effect of the mixed video and enriching the application scenarios of the mixed video.

[0100] The following is a detailed description of each step of the video rendering method.

[0101] In step S110, the first texture of the first video is obtained, and a shared environment of the first video and a shared object of the first video are created. The shared environment is a rendering environment on which the indirectly shared first texture depends, and the shared object is an object required to expose the first texture to the outside for indirect sharing.

[0102] In an exemplary embodiment of the present disclosure, the first video may be a video producer, that is, a video to be mixed into other videos. For example, the first video may be a live stream, camera data, etc., which is not particularly limited in this exemplary embodiment.

[0103] For each frame that can be rendered in the first video, a corresponding texture can be obtained as a first texture, that is, Texture. Furthermore, the first texture is represented by textIdA of the integer type.

[0104] Among them, Texture is usually a 2D image that can be used to add details to the object.

[0105] When the first video is rendered, a shared environment and a shared object of the first video may also be created.

[0106] In an alternative embodiment, Figure 2 A flow chart showing a method for creating a shared environment for a first video is shown. Figure 2 As shown, the method may at least include the following steps: in step S210, determining that the context object of the first video is a shared context object, and creating a rendering thread.

[0107] Specifically, the current Context of the first video is obtained as the shared context object sharedContext through EGL14.eglGetCurrentContext().

[0108] Among them, Context stores rendering-related input data, such as internal state information and GL call instructions.

[0109] The Android application model is a component-based application design pattern. Components require a complete Android project environment for operation. Within this environment, system components like Activity (an Android system component) and Service (an Android system component) can function properly. These components cannot be created using the standard Java (a computer programming language) object creation method, where simply clicking "new" creates an instance. Instead, they require their own context, or Context. Context is the core functional class that maintains the proper functioning of components within an Android application.

[0110] The comments in the source code explain that Context provides an interface for global information about the application environment. It is an abstract class whose implementation is provided by the Android system. It allows access to application-specific resources and types, and is a context that governs a number of resources (such as application environment variables).

[0111] Context describes the information of an application environment (i.e., context); Android provides a concrete implementation class of this abstract class; through it, you can obtain the resources and classes of the application (including application-level operations such as starting an Activity, sending a broadcast, and accepting an Intent (intention, media intermediary)).

[0112] Context has four functions: interaction of four major components, including starting Activity, Broadcast (an Android system component), Service, obtaining ContentResolv er, etc.; obtaining system / application resources, including AssetManager, PackageManager, Resources, System Service, and system / application resources such as color, string, drawable, etc.; files, including obtaining cache folders, deleting files, SharedPreference (a lightweight data storage) related, etc.; database (SQLite) related, including opening a database, deleting a database, obtaining a database path, etc.

[0113] Furthermore, a new rendering thread is created using HandlerThread.

[0114] In step S220 , a shared environment is created using a shared context object based on the rendering thread.

[0115] In the created rendering thread, use the shared context object sharedContext to create a new EGLContext in another thread to build a new EGL environment, that is, a shared environment.

[0116] Among them, EGL is the communication interface between OpenGL ES and the native window system (Native Window System).

[0117] Figure 3 A schematic diagram of the code for creating a new EGLContext is shown below. Figure 3 As shown in the figure, the input parameter mEGLDisplay is obtained by calling EGL14.eglGetDisplay(0) to get the default display. Config is obtained by calling EGL14.eglChooseConfig, which contains various properties related to EGL (Android system's drawing API (Application Programming Interface)).

[0118] Therefore, all subsequent initialization, rendering and other operations must be performed in the rendering thread, and the executed operations must rely on the shared environment.

[0119] Furthermore, a new SurfaceTexture object is created as a shared object of the first video.

[0120] Specifically, Figure 4A schematic diagram of the code for creating a shared object is shown in FIG. Figure 4 As shown, the shared object is used to associate a shared context (attachToGLContext), unbind the context (detachFromGLContext), draw shared texture data, and set up monitoring for external use.

[0121] In this exemplary embodiment, by creating a shared environment and shared objects for the first video, environmental support and object support are provided for subsequent operations such as rendering, thereby ensuring the feasibility of video rendering.

[0122] In step S120 , a first texture is drawn in the shared object based on the shared environment.

[0123] In an exemplary embodiment of the present disclosure, after the shared environment and shared object of the first video are created, the first texture may be further drawn according to the shared environment and the shared object.

[0124] In an alternative embodiment, Figure 5 A flow chart of a method for drawing a first texture in a shared object is shown, Figure 5 As shown, the method may at least include the following steps: in step S510, a temporary frame buffer is created, and a first texture and the temporary frame buffer are bound to obtain a first binding relationship.

[0125] Create a temporary frame buffer, and the temporary frame buffer can be represented by frameBuffer.

[0126] Among them, the temporary frame buffer is an FBO (Frame Buffer Object), which is actually a container to which a buffer can be added, and a texture or render buffer object can be added to it.

[0127] FBO itself cannot be used for rendering. It can only be used as a rendering target after adding a texture or render buffer. It only provides three attachments: color attachment, depth attachment, and template attachment.

[0128] Furthermore, the temporary frame buffer is bound to the first texture to obtain a first binding relationship between the first texture and the temporary frame buffer.

[0129] Binding the first texture and the temporary framebuffer is an OpenGL API operation, namely GLES20.glFramebufferTexture2D.

[0130] Specifically, it can be achieved through the following code:

[0131] GLES20.glFramebufferTexture2D(

[0132] GLES20.GL_FRAMEBUFFER,

[0133] GLES20.GL_COLOR_ATTACHMENT0,

[0134] GLES20.GL_TEXTURE_2D,textureId,0 )

[0136] In step S520 , based on the shared environment, the first texture is drawn to the shared object using the first binding relationship.

[0137] In an alternative embodiment, Figure 6 A flow chart of a method for drawing a first texture using a first binding relationship is shown. Figure 6 As shown, the method may at least include the following steps: in step S610, a current texture identifier is created, and the temporary frame buffer and the current texture identifier are bound to obtain a second binding relationship.

[0138] At the same time, a new texture is created and represented by the current texture identifier textTempId.

[0139] Specifically, the creation method can be:

[0140] GLES20.glGenTextures(count,texture,0)

[0141] Furthermore, the temporary frame buffer is bound to the new texture textTempId through GLES20.glFramebufferTexture2D to obtain a second binding relationship.

[0142] Specifically, you can implement binding through the following code:

[0143] GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER,frameBuffer)

[0144] GLES20.glFramebufferTexture2D(

[0145] GLES20.GL_FRAMEBUFFER,GLES20.GL_COLOR_ATTACHMENT0,

[0146] GLES20.GL_TEXTURE_2D,textTempId,0 )

[0148] In step S620 , based on the first binding relationship and the second binding relationship, the first texture is drawn into the current texture identifier bound to the temporary frame buffer.

[0149] After generating a first binding relationship between the first texture and the temporary frame buffer and a second binding relationship between the temporary frame buffer and the current texture identifier, the first texture represented by textIdA can be drawn to the current texture identifier textTempId bound to the temporary frame buffer. The specific drawing method can be implemented using GLES20.glDrawArrays.

[0150] In step S630 , based on the shared environment, the first texture is drawn to the shared object according to the current texture identifier.

[0151] Figure 7 A flow chart of a method for drawing a first texture according to a current texture identifier is shown. Figure 7 As shown, the method may at least include the following steps: In step S710, a data storage area is created using a shared object.

[0152] Figure 8 A schematic diagram of the code for creating a data storage area using a shared object is shown. Figure 8 As shown, the shared object SurfaceTexture object is used as an input parameter to create a surface for drawing the picture as a data storage area.

[0153] Figure 8 The specific implementation can be EGL14.eglCreateWindowSurface(this.mEGLDisplay,this.mEGLConfig,surface,new int[]{12344},0).

[0154] Among them, the surface corresponds to a screen buffer. Each window corresponds to a Surface, which stores rendering-related output data, such as color, depth, stencil buffer, etc.

[0155] In step S720 , based on the data storage area, the first texture is drawn to the shared object according to the current texture identifier.

[0156] In a shared environment, the first texture may be drawn into the shared object according to the data in the surface data storage area according to the current texture identifier.

[0157] Since the shared object SurfaceTexture object is passed to the second video associated with the first video for use, each time there is an image update, the second video can be obtained through SurfaceTexture later.

[0158] In this exemplary embodiment, the first texture is drawn according to the shared environment and shared object, which provides an implementation method for exposing the first video to the outside, solves the resource sharing problem of the first video, and the first video sharing method is unique to the Android system and has a series of advantages such as security and efficiency.

[0159] It is worth noting that the first texture can also be copied by off-screen rendering for other purposes.

[0160] In an alternative embodiment, the first texture is sent to a rendering thread.

[0161] After the first texture is drawn to the current texture identifier bound to the temporary frame buffer based on the first binding relationship and the second binding relationship in step S620 , the first texture may be transferred to the rendering thread of the shared context object through the current texture identifier textTempId.

[0162] In this way, the data of the first video can be obtained in the new EGL environment without the need to copy the texture image through glCopyTexImage, providing a more convenient way to share video textures, improving the rendering efficiency of the first video, and avoiding problems such as screen flickering and black screen.

[0163] Even if the second video subsequently renders the copied first texture, another frame buffer object can be created again to perform secondary processing on the image texture of the first video.

[0164] In step S130, the first texture is called to the second video corresponding to the first video through the shared object, so as to perform mixed rendering on the first video and the second video to obtain a mixed video.

[0165] In an exemplary embodiment of the present disclosure, after the first texture is drawn to the shared object through the shared environment, the first texture can be called through the shared object.

[0166] In an alternative embodiment, Figure 9 A flow chart of a method for calling a first texture into a second video is shown, as shown in FIG. Figure 9 As shown, the method may include at least the following steps: in step S910, creating an object texture identifier, and binding the shared object and the object texture identifier to obtain a third binding relationship.

[0167] After creating the shared object of the first video, an object texture identifier may be created and represented by a texture id.

[0168] Specifically, you can create an object texture identifier through the following code:

[0169] int[]texture = new int[1];

[0170] GLES20.glGenTextures(1,texture,0)

[0171] Among them, the texture id is passed to SurfaceTexture as an input parameter.

[0172] Furthermore, the shared object and the object texture identifier may be bound to obtain a third binding relationship between the shared object and the object texture identifier.

[0173] Specifically, GLES20.glBindTexture(int target, int texture) is used to bind the shared object and object texture identifier.

[0174] The first parameter is the texture ID, and the second parameter is the texture type. The bound object texture ID is used to render the video image, making it easier to pass the texture ID to obtain the first texture.

[0175] In step S920 , the object texture identifier is assigned to the first texture drawn into the shared object according to the third binding relationship.

[0176] Since the third binding relationship is to bind the shared object and the object texture identifier, the object texture identifier can be assigned to the first texture that has been drawn into the shared object through the binding relationship for subsequent rendering.

[0177] In step S930 , when it is monitored that the first texture is drawn in the shared object, the first texture is called to the second video corresponding to the first video using the object texture identifier.

[0178] Since the listener is set when the SurfaceTexture is created: this.vSurfaceTexture.setOnFrameAvailableListener(listener), the listener set through the shared object can monitor the first texture drawn in the shared object.

[0179] Furthermore, for the shared object with the first texture drawn that has been shared by the first video, the updateTexImage() method can be called to update the texture, and then the first texture can be called to the second video through the object texture identifier of the shared object.

[0180] Furthermore, the first video and the second video may be mixed and rendered to obtain a corresponding mixed video.

[0181] When the second video renders a frame, the texture of the most recently updated frame of the first video, that is, the first texture, can be obtained through surfaceTexture.attachToGLContext(inttextureId).

[0182] This first texture is bound to the GL_TEXTURE_EXTERNAL_OES texture target, so the first texture is an OES texture.

[0183] Specifically, in GLES20.glBindTexture(target,textId), binding a texture takes two parameters: the target type and the texture ID. GL_TEXTURE_EXTERN AL_OES is a built-in variable that indicates an external OES texture. Android platform textures are OES textures, so they need to be converted to regular 2D textures later.

[0184] The attachToGLContext method can obtain the first texture of the first video, and the texture type is OES texture.

[0185] The attachToGLContext method attaches the SurfaceTexture to the calling thread's current OpenGLES context, creating a new OpenGL ES texture object and filling it with the SurfaceTexture image frame from the last call to SurfaceTexture.detachFromGLContext(). This essentially retrieves the first texture from the first video, which was most recently updated, and applies it to the texture from the second video. The textureId parameter is the texture ID generated in the second video's rendering thread, corresponding to the location of the GL_TEXTURE_EXTERNAL_OES texture. This allows the data from the first video to be rendered in the second video.

[0186] Figure 10 A flow chart of a method for performing mixed rendering of a first video and a second video is shown. Figure 10As shown, the method may at least include the following steps: in step S1010, when it is determined according to the second video that the first video in the mixed video is transparent, the second video is rendered to obtain the mixed video.

[0187] Since the second video is actually rendered first, with the first texture of the first video inserted during the rendering process, the G value (the value in the G channel of the RGB (Red, Green, Blue) color system) of the right half of the second video can be obtained during the rendering process of the second video. The G value of the right half of the second video is used to indicate whether the first video is transparent in the mixed video.

[0188] Figure 11 The scene effect diagram of the original picture of the second video is shown, as shown in FIG. Figure 11 As shown, the second video is divided into two parts, left and right.

[0189] The left half contains the RGB values of the second video, the R channel of the right half stores the transparency information Alpha2 of the second video, and the G channel is used to store the transparency information Alpha1 of the first video. The transparency information of the first video can also be stored using the B channel, which is not specifically limited in this exemplary embodiment.

[0190] Based on this, the transparency information of the first video stored in the G channel or R channel of the right half of the second video is obtained. When the transparency information is 0, it indicates that the first video in the mixed video is transparent. Therefore, the first video cannot be seen in the mixed video. In this case, the mixed video can be obtained by rendering only the second video.

[0191] In step S1020, when it is determined according to the second video that the first video in the mixed video is non-transparent, the first video and the second video are mixed and rendered to obtain the mixed video.

[0192] Since the second video is actually rendered first, but the first texture of the first video is inserted during the rendering process, the G value of the right half of the second video can be obtained first during the rendering process of the second video. The G value of the right half of the second video is used to indicate whether the first video is transparent in the mixed video.

[0193] like Figure 11 As shown, the second video is divided into two parts, the left half contains the RGB values of the second video, the R channel of the right half stores the transparency information Alpha2 of the second video, and the G channel is used to store the transparency information Alpha1 of the first video. The transparency information of the first video can also be stored using the B channel, which is not specifically limited in this exemplary embodiment.

[0194] Based on this, the transparency information of the first video stored in the G channel or R channel of the right half of the second video is obtained. When the transparency information is not 0, it indicates that the first video in the mixed video is not transparent. Therefore, the first video can be seen in the mixed video. In this case, the first video and the second video need to be mixed and rendered to obtain the mixed video in this case.

[0195] In an alternative embodiment, Figure 12 A flow chart of a method for further performing mixed rendering on the first video and the second video is shown. Figure 12 As shown, the method may at least include the following steps: in step S1210, a first vertex position and a first texture pixel value of a first video are obtained, and a matrix transformation is performed on the first vertex position to obtain a mixed display position.

[0196] After obtaining the first texture of the first video in a shared manner, the first texture needs to be bound during rendering. Furthermore, the first vertex position and the first texture pixel value of the first texture of each frame of the first video are obtained.

[0197] Specifically, the first vertex position is obtained by converting the physical coordinates of the first video rendered within the range of the second video into the first vertex position of the OpengGL coordinate system, thereby determining the display position of the first video.

[0198] It is worth noting that, for the texture coordinates, the entire area of the first video is taken, that is, the complete 1*1 texture coordinate range.

[0199] Specifically, it is determined by passing position information to the vertex shader. The position information includes the values of x, y, z, and w. The vertex coordinates are a four-dimensional vector, where x, y, and z are the three common axes, and w represents the scaling factor. The scaling is done by dividing x, y, and z by w. Under normal circumstances, w can be 1.0.

[0200] The vertex shader program is a source code / executable file used to describe the model transformation, view transformation, projection transformation, and lighting processing that needs to be performed on the vertex.

[0201] The inputs of the vertex shader are: attributes modified with attributes, which can pass vertex data, texture coordinates, etc.; attributes modified with uniforms, which can pass transformation matrices, etc.

[0202] Common outputs of the vertex shader are: gl_Position, which outputs the transformed vertex data; gl_PointSize, which sets the point size.

[0203] The business processing performed in the vertex shader includes the calculation of matrix transformation, calculation of lighting formula to generate vertex color, generation / transformation of texture coordinates, etc.

[0204] In order to ensure that the first vertex position to be displayed in the first video perfectly matches the second video, the first vertex position may be further subjected to matrix transformation to obtain a mixed display position.

[0205] Specifically, the matrix transformation for the first vertex position is implemented using the Android system's GLES20.glUniformMatrix4fv(vertexMatrixHandler,1,false,matrix,0) function. Matrix is the external matrix calculated for moving or scaling the first video, and vertexMatrixHandler is the handle (reference) to the vertex shader's matrix variable. This handle represents an int reference to the matrix passed to the fragment shader, effectively finding an associated object to help pass the matrix.

[0206] The fragment shader program is a fragment shader program source code / executable file used to describe operations (such as color mixing) performed on fragments.

[0207] The inputs to the fragment shader are: texture coordinates generated for each fragment by interpolation during the rasterization stage; attributes modified with uniforms, which can pass data needed in the fragment shader; attributes modified with sampler2D, which pass texture objects, actually pass the corresponding texture unit index number.

[0208] Common output of the fragment shader: gl_FragColor outputs the color of each fragment.

[0209] The fragment shader's business processing includes calculating color; obtaining texels; filling color values into pixels; it can be used to fill the color of each pixel in an image / video. For example, adding a filter to a video actually modifies the color fill of each pixel in the video.

[0210] Since the external input matrix is a four-dimensional vector, and each first vertex position has a corresponding (x, y, z, w), multiplying the four-dimensional vector of the external input matrix and the four-dimensional vector of the first vertex position can obtain the corresponding mixed display position, thereby achieving the purpose of transformation.

[0211] The first texture pixel value is obtained by sampling the first texture using samplerExternalOES in the fragment shader to determine the RGB color value of each pixel. Since the first texture pixel value is the ARGB color value of the first texture, in addition to sampling the first texture of the first video to obtain the RGB color value, the transparency can also be obtained by sampling the G channel in the second video through the fragment shader.

[0212] Figure 13 FIG. 4 shows a schematic diagram of a portion of the code for obtaining the mixed display position and the first texture pixel value. Figure 13 As shown, the code indicates that obtaining the mixed display position and the first texture pixel value can be implemented through the vertex shader and fragment shader of the first video.

[0213] Among them, a_Position is the coordinate position of the texture passed in from the outside, u_Matrix is the passed in matrix, which is used for view transformation (rotation, scaling, movement, etc.), gl_Position outputs the transformed vertex data, and Gl_FragColor is the color of the final A texture. The transparency of this part is 1.0, and the RGB value is the RGB of the original video.

[0214] In step S1220, the second vertex position and the second texture pixel value of the second video are obtained, and the first video and the second video are mixed and rendered according to the mixed display position, the first texture pixel value, the second vertex position and the second texture pixel value to obtain a mixed video.

[0215] The rendering process for the second video is the same as for the first video, requiring the corresponding second vertex position and second texture pixel value to be obtained first. Furthermore, the position depends on the size of the container in which the video is played and its position on the screen. For example, when displaying full screen, the canvas is the entire screen size; when displaying half screen, which is the normal screen height, it can be top-aligned, centered, or bottom-aligned.

[0216] Figure 14 A schematic diagram of the code for obtaining the second vertex position is shown, such as Figure 14 As shown, the acquisition of the second vertex position can be implemented through a vertex shader, so the code is the acquisition logic of the vertex shader.

[0217] RGBColor obtains the color value of the left half of the second video.

[0218] alphaColor is the color corresponding to the right video area of the second video, and the R value is obtained as the transparency of the rendering.

[0219] gl_FragColor is the actual RGBA color output.

[0220] Figure 15 A schematic diagram of the code for obtaining the second texture pixel value is shown in FIG. Figure 15 As shown, the acquisition of the second texture pixel value can be implemented through a vertex shader. Therefore, the code is the logic of the vertex shader to obtain the second texture pixel value.

[0221] Among them, v_TexCoordinateRGB is the texture coordinate of the left half of the second video (used for RGB shading).

[0222] v_TexCoordinateAlpha is the texture coordinate of the right half of the second video (used for alpha shading).

[0223] v_TexCoordinateSrc is the texture coordinate of the first video (the 1*1 original first video texture area).

[0224] a_Position is externally input and is dynamically determined based on the position and size of the second video rendering, that is, the position and display size of the second video on the screen.

[0225] gl_Position is a built-in variable of the vertex shader, which represents the final coordinate position of the output pixel.

[0226] In an alternative embodiment, Figure 16 A further flow chart of a method for performing mixed rendering on the first video and the second video is shown. Figure 16 As shown, the method may at least include the following steps: in step S1610, a target transparency and a target color value are obtained according to the first texture pixel value and the second texture pixel value.

[0227] Figure 17 A schematic diagram showing how transparency is calculated is shown in FIG. Figure 17 As shown in the figure, this formula describes how to calculate transparency when two layers are blended. For example, if Image A covers Image B, when the human eye looks at Image B, if Image B is completely transparent, only Image A is visible. However, if Image B has a certain degree of transparency, then while viewing Image B, the human eye can also see the partially transparent Image A. The transparency value will have a certain degree of permeability due to the different transparency of the two layers.

[0228] In an alternative embodiment, Figure 18 A schematic diagram of a method for calculating target transparency and target color values is shown, as shown in FIG. Figure 18As shown, the method may at least include the following steps: in step S1810, performing two-dimensional sampling on the first texture pixel value to obtain a first sampled pixel value, and performing two-dimensional sampling on the second texture pixel value to obtain a second sampled pixel value.

[0229] Figure 19 A schematic diagram of the code for two-dimensional sampling of the fragment shader is shown, as shown in Figure 19 As shown in the figure, src represents the RGBA value of the first video obtained by 2D (Two Dimension) sampling of the texture of the first video, which is used as the first sampled pixel value; left represents the four-dimensional vector obtained by 2D sampling of the left texture of the second video, and the four values represent the values of the original RGBA channels (red, green, blue, alpha); right represents the 2D sampling of the right texture of the second video, and also obtains the RGBA values of the right part. Therefore, it can be determined that the four-dimensional vector obtained by 2D sampling of the left texture of the second video and the RGBA values of the right part obtained by 2D sampling of the right texture of the second video are the second sampled pixel values.

[0230] In step S1820 , a difference calculation is performed on the first sampled pixel value and the second sampled pixel value to obtain a target transparency value.

[0231] It is worth noting that, before additional processing is performed on the first video and the second video, the original alpha value (alpha) is basically 1, that is, non-transparent.

[0232] alpha1 is the transparency a on the left side of the second video.

[0233] At this time, alpha2 is the product of the value of the G channel on the right side of the second video and the transparency value of the first video.

[0234] The target transparency value can be obtained by performing a difference calculation on the transparency of the mixed portion of the first video and the second video.

[0235] The target transparency value is calculated as shown in the formula: alpha=1.0-(1.0-alpha1)*(1.0-alpha2).

[0236] Among them, alpha is the target transparency value.

[0237] In step S1830 , a linear difference calculation is performed on the first sampled pixel value and the second sampled pixel value to obtain a target color value.

[0238] When calculating the target color value, it is necessary to mix the color value RGB2 of the left half of the second sampled pixel value of the second video and the color value RGB1 of the first sampled pixel value of the first video.

[0239] Specifically, you can use the fragment shader's mix function to perform linear interpolation. The first two parameters in the mix function are RGB1 and RGB2, respectively, and the third parameter is the linear interpolation coefficient, which can use alpha2. After performing linear interpolation with the mix function, you can obtain the target color value of the mixed part.

[0240] In this exemplary embodiment, the target transparency value and target color value can be obtained through two-dimensional sampling and calculation, which provides data support and basis for the rendering of the mixed video, and ensures the mixing effect of the first video and the second video and the rendering effect of the mixed video.

[0241] In step S1620, the mixing position and mixing size of the first video in the second video are obtained.

[0242] In with Figure 15 In the associated code, the mixing position of the first video in the second video can also be obtained through the vertex shader. The mixing position represents the display position of the first video in the second video in the mixed video.

[0243] In addition, you can also obtain the blending size. This blending size indicates the position range of the first video to be displayed in the second video after blending. For example, the blending size can be to display the original size of the first video in the second video, or to cut out a portion of the first video and display it in the second video.

[0244] In step S1630 , based on the mixed display position, the second vertex position, the mixed position, and the mixed size, the first video and the second video are mixed and rendered according to the target transparency value and the target color value to obtain a mixed video.

[0245] After determining the transformed first video, the display position of the first video in the second video, the display position of the second video and the size of the first video in the second video according to the mixed display position, the second vertex position, the mixed position and the mixed size, the first video and the second video can be mixed and rendered according to the target transparency value and the target color value to obtain a mixed video.

[0246] To output the final mixed video, the RGBA values of the texture pixels of the mixed video may be output via gl_FragColor=vec4(RGB,alpha).

[0247] Among them, gl_FragColor is a built-in variable of the shader, which represents the aRGB value of the output pixel

[0248] This variable can transmit the target pixel value and the target transparency value to the GPU rendering pipeline for shading to obtain a mixed video.

[0249] It is worth noting that the other parts of the second video that are not mixed with the first video can be rendered according to the original second video.

[0250] In this exemplary embodiment, a corresponding mixed video can be obtained by mixed rendering of the first video and the second video, providing a mixing method for two videos with an excellent fusion method. The texture boundaries of the two videos are smooth and natural, and the color saturation and clarity are normal. Therefore, the perfect mixing of the two videos is supported, and the viewing experience of the mixed video is improved.

[0251] It is worth noting that when calling the first texture to the second video through a shared object, it may not be called normally due to network problems. In this case, placeholder rendering of ordinary images is also supported.

[0252] In an alternative embodiment, Figure 20 A flow chart showing a method for mixed rendering of a default image and a second video is shown. Figure 20 As shown, the method may include at least the following steps: in step S2010, when the first texture is not called to the second video corresponding to the first video through the shared object, a default image and the image vertex position and image pixel value of the default image are obtained.

[0253] If the first texture cannot be transferred to the second video through the shared object in step S130, a pre-set default image is obtained. Furthermore, the vertex positions and pixel values of the default image can also be obtained. In this case, no matrix transformation of the vertex positions is required.

[0254] In step S2020, the default image and the second video are mixed and rendered according to the image vertex position, the image pixel value, the second vertex position and the second texture pixel value to obtain a mixed video.

[0255] After obtaining the image vertex position and image pixel value of the default image, the default image and the second video can be mixed and rendered according to the image vertex position, image pixel value, second vertex position and second texture pixel value in accordance with step S1220 to obtain a mixed video. This exemplary embodiment does not specifically limit this.

[0256] It is worth noting that, since the first texture is called into the second texture by using a shared object, the shared object needs to be untied after drawing a frame of the first texture.

[0257] In an optional embodiment, the shared context of the first texture is released using the shared object.

[0258] Specifically, the first texture drawn and the shared context are unbound by calling SurfaceTexture.detachFromGLContext() on the shared object. This allows for secure texture sharing.

[0259] Figure 21 The scene effect diagram of the mixed video obtained by hybrid rendering is shown in FIG. Figure 21 As shown, in Figure 11 In the original picture of the second video shown, the white area in the middle of the left side is a TV.

[0260] However, in the mixed video created by mixing the first video with the second video, the first video can be displayed on the TV while the original second video is playing. The final mixed video effect is that the person in the second video is sitting in front of the TV watching TV, while the TV is playing the first video shared externally.

[0261] It is worth noting that the user only sees the left portion of the second video; the right portion is used to read transparency information and is not actually rendered on the screen. The user only sees the portion of the first video that is visible on the TV.

[0262] To summarize, when sharing video textures, if two videos, A and B (where A is the video producer and B is the video consumer that needs to be mixed with A), are played by two different players and have different rendering views, then they are in different rendering threads and contexts, and video data cannot be read from each other. This is because OpenGL operations determine whether the context associated with the thread issuing the GL instruction is consistent with the context of the target data.

[0263] Video frames cannot be directly passed to another party for use like Java objects. To share video data, you need to use the same OpenGL context or use an indirect method to share.

[0264] Directly sharing the Context can share some resources, but if the video provider does not want the caller to change its original context, it is inconvenient to directly share the Context to achieve video resource sharing.

[0265] Therefore, in the present disclosure, this is achieved through indirect sharing. That is, when video A is rendered, a new EGLContext is created through the context of video A to start a rendering thread, and the texture data of the video A rendering thread is shared with the new rendering thread.

[0266] When drawing video A, a framebuffer is created and a newly created texture is bound for drawing. In other words, this texture data is managed through an FBO, rendered off-screen to a SurfaceTexture bound to a new EGLContext, and then the SurfaceTexture is exposed to the outside world to achieve screen data sharing.

[0267] In the mixed video rendering part, video rendering actually decodes each frame of video data, and each frame corresponds to a 2D image. Each image is then drawn on the screen. Conventional MP4 video rendering does not support transparency and only displays RGB color values, namely the values of the three primary colors red, green, and blue, which form the colorful video image.

[0268] Video A can be a regular video or a live broadcast, and the picture color only has RGB channel information.

[0269] Video B requires special processing to make it into a video divided into left and right halves. It can be generated by calculation using tools, or it can be produced by designers using art tools.

[0270] The vertex coordinates and texture coordinates required for video rendering are determined by the video size and the desired display position. When rendering video B, the RGB values of the left half of the texture are read, followed by the R values of the right half to obtain the ARGB values of video B. Video A uses its original texture for display, and the RGB values of video A are added to the Alpha1 to obtain the ARGB values of video A.

[0271] Then, specify the rendering area size as half of the original video to achieve mixed rendering of textures with alpha values.

[0272] Video B is split into two halves, mainly to facilitate the calculation of the vertex coordinate positions of the video. The R and G channels store transparency information separately because the original R channel has been occupied by the alpha information of video B. Therefore, the transparency information of video A cannot be saved and must be stored in another channel, so it can also be placed in the B channel.

[0273] It is worth noting that when the first texture of the first video is not input, there is no need to perform video texture sharing. It is only necessary to convert video B into a 2D texture and render it.

[0274] In this disclosure, the original video, or the second video, is split into two halves: the left side is a color video with RGB channels, and the right side is an off-white video or other color video. For example, if the R channel is not 0 and the other channels are 0, it is a red video; if the G channel is not 0 and the other channels are 0, it is a green video; and if the B channel is not 0 and the other channels are 0, it is a blue video. Furthermore, the R channel is used to store the alpha value required for video rendering.

[0275] Furthermore, it solves the problem of sharing video resources, such as image frames, shaders, and textures. A currently playing video, such as a live video stream, can transmit real-time video frame data to another currently playing MP4 video, B, allowing A to be rendered synchronously within B. This solves the problem of sharing video resources across different OpenGL environments, essentially sharing resources across different rendering threads and contexts.

[0276] In addition, it solves the problems of texture blending rendering and secondary processing. Specifically, it supports perfect blending of two images, and supports secondary processing of shared video images, such as rotation, scaling, transformation, and movement. It can also realize multiple processing of images through FBO, and solves the problem of edge blending and position alignment between the original MP4 image and the live broadcast image. Specifically, it uses OpenGL to implement custom vertex shaders and fragment shaders, such as vertex coordinate transformation and texture pixel color blending.

[0277] Furthermore, when the acquisition of the video picture data to be mixed fails and mixing cannot be performed, placeholder rendering of ordinary pictures is also supported.

[0278] Furthermore, it also solves problems such as screen flickering, black screen and gaps in mixed boundaries.

[0279] It is worth noting that the present disclosure is completely executed in the GPU, without interacting with the CPU or involving the CPU, thus saving rendering performance, being very friendly to hardware facilities such as devices, and being extremely practical.

[0280] The video rendering method in the exemplary embodiment of the present disclosure, on the one hand, realizes the indirect sharing of the first texture of the first video through a shared environment and a shared object, provides a safe and efficient way to share video textures, avoids the problem of the first texture being changed due to direct sharing, and prevents the mixed video from generating a black screen or flickering in advance; on the other hand, the first video and the second video are mixed and rendered, providing a method that supports both video rendering with transparency and mixing of two videos, ensuring the normal display of the color saturation and clarity of the original video, and can also perfectly blend the two videos smoothly and naturally, greatly optimizing the visual effect of the mixed video, enriching the application scenarios of the mixed video, and also improving the product competitiveness in the application scenarios, and to a certain extent increasing the product's viewing experience and revenue.

[0281] In addition, in an exemplary embodiment of the present disclosure, a video rendering device is also provided. Figure 22 A schematic diagram of the structure of a video rendering device is shown in FIG. Figure 22 As shown, the video rendering device 2200 may include: a texture acquisition module 2210, an indirect sharing module 2220 and a hybrid rendering module 2230. Among them:

[0282] The texture acquisition module 2210 is configured to acquire a first texture of a first video and create a shared environment and a shared object of the first video;

[0283] an indirect sharing module 2220 configured to draw the first texture in the shared object based on the shared environment;

[0284] The mixed rendering module 2230 is configured to call the first texture to the second video corresponding to the first video through the shared object, so as to perform mixed rendering on the first video and the second video to obtain a mixed video.

[0285] In an exemplary embodiment of the present invention, the step of creating a shared environment and a shared object for the first video includes:

[0286] Creating a sharing environment for the first video;

[0287] A shared object for the first video is created.

[0288] In an exemplary embodiment of the present invention, creating a sharing environment for the first video includes:

[0289] Determine that the context object of the first video is a shared context object, and create a rendering thread;

[0290] Based on the rendering thread, a shared environment is created using the shared context object.

[0291] In an exemplary embodiment of the present invention, drawing the first texture in the shared object based on the shared environment includes:

[0292] Creating a temporary frame buffer, and binding the first texture and the temporary frame buffer to obtain a first binding relationship;

[0293] Based on the shared environment, the first texture is drawn to the shared object using the first binding relationship.

[0294] In an exemplary embodiment of the present invention, drawing the first texture to the shared object using the first binding relationship based on the shared environment includes:

[0295] Creating a current texture identifier, and binding the temporary frame buffer and the current texture identifier to obtain a second binding relationship;

[0296] Based on the first binding relationship and the second binding relationship, drawing the first texture to the current texture identifier bound to the temporary frame buffer;

[0297] Based on the shared environment, the first texture is drawn to the shared object according to the current texture identifier.

[0298] In an exemplary embodiment of the present invention, drawing the first texture to the shared object according to the current texture identifier includes:

[0299] creating a data storage area using the shared object;

[0300] Based on the data storage area, the first texture is drawn to the shared object according to the current texture identifier.

[0301] In an exemplary embodiment of the present invention, after drawing the first texture to the shared object according to the current texture identifier, the method further includes:

[0302] Sending the first texture to the rendering thread.

[0303] In an exemplary embodiment of the present invention, after calling the first texture to a second video corresponding to the first video through the shared object, the method further includes:

[0304] The shared context of the first texture is released using the shared object.

[0305] In an exemplary embodiment of the present invention, calling the first texture to a second video corresponding to the first video through the shared object includes:

[0306] Creating an object texture identifier, and binding the shared object and the object texture identifier to obtain a third binding relationship;

[0307] assigning the object texture identifier to the first texture drawn into the shared object according to the third binding relationship;

[0308] When it is monitored that the first texture is drawn in the shared object, the first texture is called to the second video corresponding to the first video using the object texture identifier.

[0309] In an exemplary embodiment of the present invention, the step of performing mixed rendering on the first video and the second video to obtain a mixed video includes:

[0310] When it is determined according to the second video that the first video in the mixed video is transparent, rendering the second video to obtain a mixed video;

[0311] When it is determined according to the second video that the first video in the mixed video is non-transparent, the first video and the second video are mixed and rendered to obtain a mixed video.

[0312] In an exemplary embodiment of the present invention, the step of performing mixed rendering on the first video and the second video to obtain a mixed video includes:

[0313] Obtaining a first vertex position and a first texture pixel value of the first video, and performing a matrix transformation on the first vertex position to obtain a mixed display position;

[0314] The second vertex position and the second texture pixel value of the second video are obtained, and the first video and the second video are mixed and rendered according to the mixed display position, the first texture pixel value, the second vertex position and the second texture pixel value to obtain a mixed video.

[0315] In an exemplary embodiment of the present invention, after obtaining the second vertex position and the second texture pixel value of the second video, the method further includes:

[0316] When the first texture is not called to the second video corresponding to the first video through the shared object, obtaining a default image and image vertex positions and image pixel values of the default image;

[0317] The default image and the second video are mixed and rendered according to the image vertex position, the image pixel value, the second vertex position and the second texture pixel value to obtain a mixed video.

[0318] In an exemplary embodiment of the present invention, the step of performing mixed rendering on the first video and the second video according to the mixed display position, the first texture pixel value, the second vertex position, and the second texture pixel value to obtain a mixed video includes:

[0319] Obtaining a target transparency and a target color value according to the first texture pixel value and the second texture pixel value;

[0320] Obtaining a mixing position and a mixing size of the first video in the second video;

[0321] Based on the mixed display position, the second vertex position, the mixed position and the mixed size, the first video and the second video are mixed and rendered according to the target transparency value and the target color value to obtain a mixed video.

[0322] In an exemplary embodiment of the present invention, obtaining a target transparency and a target color value according to the first texture pixel value and the second texture pixel value includes:

[0323] Performing two-dimensional sampling on the first texture pixel value to obtain a first sampled pixel value, and performing two-dimensional sampling on the second texture pixel value to obtain a second sampled pixel value;

[0324] Performing a difference calculation on the first sampled pixel value and the second sampled pixel value to obtain a target transparency value;

[0325] A target color value is obtained by performing a linear difference calculation on the first sampled pixel value and the second sampled pixel value.

[0326] The specific details of the above-mentioned video rendering device 2200 have been described in detail in the corresponding video rendering method, and therefore will not be repeated here.

[0327] It should be noted that although several modules or units of the video rendering device 2200 are mentioned in the detailed description above, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0328] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0329] Refer to the following Figure 23 23. An electronic device 2300 according to this embodiment of the present invention is described. Figure 23The electronic device 2300 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0330] like Figure 23 As shown, electronic device 2300 is implemented as a general-purpose computing device. Components of electronic device 2300 may include, but are not limited to, the at least one processing unit 2310, the at least one storage unit 2320, a bus 2330 connecting various system components (including storage unit 2320 and processing unit 2310), and a display unit 2340.

[0331] The storage unit stores program codes, which can be executed by the processing unit 2310, so that the processing unit 2310 performs the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification.

[0332] The storage unit 2320 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 2321 and / or a cache memory unit 2322 , and may further include a read-only memory unit (ROM) 2323 .

[0333] The storage unit 2320 may also include a program / utility 2324 having a set (at least one) of program modules 2325, such program modules 2325 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0334] Bus 2330 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0335] The electronic device 2300 can also communicate with one or more external devices 2500 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 2300, and / or any device that enables the electronic device 2300 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 2350. Furthermore, the electronic device 2300 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 2360. As shown, the network adapter 2360 communicates with other modules of the electronic device 2300 via a bus 2330. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 2300, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0336] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0337] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, storing a program product capable of implementing the aforementioned methods of this specification. In some possible embodiments, various aspects of the present invention may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section of this specification.

[0338] refer to Figure 24 As shown, a program product 2400 for implementing the above method according to an embodiment of the present invention is described. The program product 2400 may be a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0339] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0340] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0341] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0342] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and the like, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0343] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

Claims

1. A video rendering method, characterized in that: The method comprises: Obtaining a first texture of a first video, creating a shared environment for the first video, and creating a shared object for the first video, wherein the shared environment is a rendering environment on which the indirectly shared first texture depends, and the shared object is an object required to expose the first texture to the outside for indirect sharing; Drawing the first texture in the shared object based on the shared environment; The first texture is called to the second video corresponding to the first video through the shared object, so as to perform mixed rendering on the first video and the second video to obtain a mixed video; the first video and the second video are two videos.

2. The video rendering method according to claim 1, wherein: The step of creating a shared environment for the first video includes: Determine that the context object of the first video is a shared context object, and create a rendering thread; Based on the rendering thread, a shared environment is created using the shared context object.

3. The video rendering method according to claim 2, wherein: Drawing the first texture in the shared object based on the shared environment includes: Creating a temporary frame buffer, and binding the first texture and the temporary frame buffer to obtain a first binding relationship; Based on the shared environment, the first texture is drawn to the shared object using the first binding relationship.

4. The video rendering method according to claim 3, wherein: Drawing the first texture to the shared object by using the first binding relationship based on the shared environment includes: Creating a current texture identifier, and binding the temporary frame buffer and the current texture identifier to obtain a second binding relationship; Based on the first binding relationship and the second binding relationship, drawing the first texture to the current texture identifier bound to the temporary frame buffer; Based on the shared environment, the first texture is drawn to the shared object according to the current texture identifier.

5. The video rendering method according to claim 4, characterized in that: Drawing the first texture to the shared object according to the current texture identifier includes: creating a data storage area using the shared object; Based on the data storage area, the first texture is drawn to the shared object according to the current texture identifier.

6. The video rendering method according to claim 4, characterized in that: After drawing the first texture to the shared object according to the current texture identifier, the method further includes: Sending the first texture to the rendering thread.

7. The video rendering method according to claim 4, wherein: After calling the first texture to a second video corresponding to the first video through the shared object, the method further includes: The shared context of the first texture is released using the shared object.

8. The video rendering method according to claim 1, wherein: The calling the first texture to the second video corresponding to the first video through the shared object includes: Creating an object texture identifier, and binding the shared object and the object texture identifier to obtain a third binding relationship; assigning the object texture identifier to the first texture drawn into the shared object according to the third binding relationship; When it is monitored that the first texture is drawn in the shared object, the first texture is called to the second video corresponding to the first video using the object texture identifier.

9. The video rendering method according to claim 1, wherein: The performing mixed rendering on the first video and the second video to obtain a mixed video includes: When it is determined according to the second video that the first video in the mixed video is transparent, rendering the second video to obtain a mixed video; When it is determined according to the second video that the first video in the mixed video is non-transparent, the first video and the second video are mixed and rendered to obtain a mixed video.

10. The video rendering method according to claim 9, characterized in that: The performing mixed rendering on the first video and the second video to obtain a mixed video includes: Obtaining a first vertex position and a first texture pixel value of the first video, and performing a matrix transformation on the first vertex position to obtain a mixed display position; The second vertex position and the second texture pixel value of the second video are obtained, and the first video and the second video are mixed and rendered according to the mixed display position, the first texture pixel value, the second vertex position and the second texture pixel value to obtain a mixed video.

11. The video rendering method according to claim 10, wherein: After obtaining the second vertex position and the second texture pixel value of the second video, the method further includes: When the first texture is not called to the second video corresponding to the first video through the shared object, obtaining a default image and image vertex positions and image pixel values of the default image; The default image and the second video are mixed and rendered according to the image vertex position, the image pixel value, the second vertex position and the second texture pixel value to obtain a mixed video.

12. The video rendering method according to claim 10, wherein: The step of performing mixed rendering on the first video and the second video according to the mixed display position, the first texture pixel value, the second vertex position, and the second texture pixel value to obtain a mixed video includes: Obtaining a target transparency and a target color value according to the first texture pixel value and the second texture pixel value; Obtaining a mixing position and a mixing size of the first video in the second video; Based on the mixed display position, the second vertex position, the mixed position and the mixed size, the first video and the second video are mixed and rendered according to the target transparency value and the target color value to obtain a mixed video.

13. The video rendering method according to claim 12, wherein: Obtaining a target transparency and a target color value according to the first texture pixel value and the second texture pixel value includes: Performing two-dimensional sampling on the first texture pixel value to obtain a first sampled pixel value, and performing two-dimensional sampling on the second texture pixel value to obtain a second sampled pixel value; Performing a difference calculation on the first sampled pixel value and the second sampled pixel value to obtain a target transparency value; A target color value is obtained by performing a linear difference calculation on the first sampled pixel value and the second sampled pixel value.

14. A video rendering device, characterized in that: include: a texture acquisition module configured to acquire a first texture of a first video, create a sharing environment for the first video, and create a shared object for the first video, wherein the sharing environment is a rendering environment on which the indirectly shared first texture depends, and the shared object is an object required to expose the first texture to the outside for indirect sharing; an indirect sharing module, configured to draw the first texture in the shared object based on the shared environment; A mixed rendering module is configured to call the first texture to the second video corresponding to the first video through the shared object, so as to perform mixed rendering on the first video and the second video to obtain a mixed video; the first video and the second video are two videos.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video rendering method according to any one of claims 1 to 13 is implemented.

16. An electronic device, characterized in that: include: processor; a memory for storing executable instructions of the processor; The processor is configured to execute the video rendering method according to any one of claims 1 to 13 by executing the executable instructions.

Images