Video processing method, device, electronic device and readable storage medium

Abstract

The present application discloses a video processing method, device, electronic device and readable storage medium. The method includes, when receiving a transparency processing request, attaching the captured current frame of the video to be processed as a texture feature to a target container of the same canvas size as the video to be displayed; the canvas supports transparency information and does not process the pixel color in advance. According to the position of the pixel points of the current frame in the target container, the pixel color of each pixel point corresponding to the video to be processed is mapped; for each pixel point of the current frame, the transparency information of the current pixel point is determined according to the correlation between the transparency elimination range, the pixel color of the current pixel point and the corresponding background color; according to the transparency information of each pixel point of each frame, the transparency operation of the video to be processed is performed pixel by pixel, so as to achieve low-cost and convenient elimination of the video background color on the basis of low impact on system performance and compatibility.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video processing method, device, electronic device and readable storage medium. Background Art

[0002] With the gradual application and popularization of media playback functions on the web platform, videos have been widely used in web page decoration and content due to their high compression rate and ease of use. However, since web video playback does not support transparency, that is, the web platform does not support videos with transparency, this results in the inability of certain video formats to be well combined with web content. In order to ensure the perfect combination of video and web content, it is necessary to specify the background color or propose video transparency for videos with a solid color background, so that the content behind the video material is not blocked. For example, if the rain effect is added to the video, after removing the background, the dynamic effect of rain can be presented on a background image, thereby expanding the scope of video use, improving the effect of video use, and enriching the performance of web page content.

[0003] In order to solve the above problems, the related technology usually uses dynamic pictures or uses the drawing API (Application Programming Interface) drawing method to replace the video. For the dynamic picture solution: use dynamic pictures that support transparent formats instead of videos, that is, dynamic picture materials that support transparency such as webp format, APNG (Animated Portable Network Graphics) format, etc. Although it is easy to use and has low rendering pressure, the rendering pressure is not much different from that of videos. However, it is precisely because these formats support transparency channels that dynamic pictures are difficult to compress, and the file occupies a relatively large space. The same content can often reach 3-5 times the size of the video format, which puts a lot of memory pressure on the page. Using more dynamic pictures consumes a lot of performance of the entire system. In addition, these formats are not widely supported compared to video formats, and their compatibility is not good. For the drawing method using the drawing API (Application Programming Interface): you can use the webgl (Web Graphics Library, 3D drawing protocol), canvas2d and other drawing APIs that come with the web platform to reproduce the content in a drawing way. Although it has interactive capabilities that videos and animated images lack, the content can be changed according to the script, and it is easy to adjust some styles of the content without the need for designers to modify the exported files. However, this method has high rendering pressure and high development costs. Many effects that are easy to create on the designer side cannot be easily realized. In addition, it is a time-consuming and labor-intensive solution that cannot achieve better results when no interaction is required.

[0004] In view of this, how to remove the background color of the video at a low cost and conveniently on the basis of low impact on system performance and compatibility is a technical problem that technicians in the relevant field need to solve. Summary of the invention

[0005] The present application provides a video processing method, device, electronic device and readable storage medium, which can achieve low-cost and convenient removal of video background color on the basis of compatibility and low impact on system performance.

[0006] To solve the above technical problems, the embodiments of the present invention provide the following technical solutions:

[0007] An embodiment of the present invention provides a video processing method, including:

[0008] When a transparency processing request is received, for each frame of the video to be processed, the captured current frame is attached as a texture feature to a target container of a canvas having the same size as the to-be-displayed size of the video to be processed; the canvas supports transparency information and does not process pixel colors in advance;

[0009] Mapping the pixel colors of the pixels in the current frame to obtain the corresponding pixel colors in the video to be processed;

[0010] For each pixel of the current frame, determining transparency information of the current pixel according to a transparency elimination range, a correlation between a pixel color of the current pixel and a corresponding background color;

[0011] According to the transparency information of each pixel point of each frame, the corresponding frame of the video to be processed is processed for transparency, so as to output a picture with a transparency color.

[0012] Optionally, before determining the transparency information of the current pixel point according to the correlation between the transparency elimination range, the pixel color of the current pixel point and the corresponding background color, the method further includes:

[0013] Determining background-pixel color correlation parameters according to the application scenario and transparency requirement information of the video to be processed;

[0014] Calculating a correlation value between the pixel color of the current pixel and the corresponding background color according to the background-pixel color correlation parameter;

[0015] The background-pixel color correlation parameter includes a tolerance and a color weight, and the tolerance is determined according to the background removal effect and the transparency removal range.

[0016] Optionally, determining the background-pixel color correlation parameter according to the application scenario and transparency requirement information of the video to be processed includes:

[0017] Obtaining video fine-tuning requirements for the video to be processed;

[0018] If the video fine-tuning requirement is to perform video fine-tuning based on a single color as a dominant color, the color weight corresponding to the dominant color is greater than the color weight of the non-dominant color;

[0019] If the video fine-tuning requirement is to perform video fine-tuning based on brightness distribution, the color weight of each color is determined according to brightness distribution information.

[0020] Optionally, determining the transparency information of the current pixel point according to the correlation between the transparency elimination range, the pixel color of the current pixel point and the corresponding background color includes:

[0021] The correlation calculation formula is called to calculate the correlation value between the pixel color of the current pixel and the corresponding background color. The correlation calculation formula is:

[0022] relation=w r *abs(r–r p ) / t r +w g *abs(g–g p ) / t g +w b *abs(b–b p ) / t b ;

[0023] The transparency relation formula is called to calculate the transparency information of the current pixel point, and the transparency relation formula is:

[0024] opacity=clamp(relation,min,max)

[0025] In the formula, relation is the correlation value, r, g, b are the background color values ​​of the current pixel, r p , g p 、b p is the pixel color value of the current pixel, abs represents the absolute value, w r is the weight of red color interpolation, w g is the weight of green color interpolation, w b is the weight of blue color interpolation, t r is the tolerance of red color interpolation, t g is the tolerance of green color interpolation, t b is the tolerance of blue color interpolation; opacity is the transparency information; clamp is used to limit the relation within min and max, where min is the minimum value of the relation and max is the maximum value of the relation.

[0026] Optionally, determining the transparency information of the current pixel point according to the correlation between the transparency elimination range, the pixel color of the current pixel point and the corresponding background color includes:

[0027] The correlation calculation formula is called to calculate the correlation value between the pixel color of the current pixel and the corresponding background color. The correlation calculation formula is:

[0028] relation=(abs(r–r p )+abs(g–g p )+abs(b–bp )) / t / 3;

[0029] The transparency relation formula is called to calculate the transparency information of the current pixel point, and the transparency relation formula is:

[0030] opacity=clamp(relation,min,max)

[0031] In the formula, relation is the correlation value, r, g, b are the background color values ​​of the current pixel, r p , g p 、b p is the pixel color value of the current pixel, abs represents the absolute value, t is the tolerance; opacity is the transparency information, clamp is used to limit the relation within min and max, min is the minimum value of the relation, and max is the maximum value of the relation.

[0032] Optionally, attaching the captured current frame as a texture feature to a target container of a canvas having the same size as the to-be-processed video to be displayed includes:

[0033] Call canvas2d api to capture the current frame to the initial canvas;

[0034] According to the to-be-processed video's display size, generate a webgl canvas with the same size as the to-be-processed video's display size, open an alpha channel of the webgl canvas, and simultaneously close a premultiplication setting function of the webgl canvas;

[0035] In response to a target container drawing instruction, draw the target container in the webgl canvas;

[0036] The current frame picture in the initial canvas is used as the texture feature of the target container and attached to the target container.

[0037] Another aspect of the present invention provides a video processing device, including:

[0038] The image preprocessing module is used for, when receiving a transparency processing request, attaching the captured current frame as a texture feature to a target container of a canvas with the same size as the to-be-processed video for each frame of the to-be-processed video; the canvas supports transparency information and does not process pixel colors in advance;

[0039] A pixel color determination module, used for mapping the pixel color of each pixel in the video to be processed according to the position of the pixel of the current frame in the target container;

[0040] A transparency information determination module, used to determine the transparency information of the current pixel point for each pixel point of the current frame according to the transparency elimination range, the correlation between the pixel color of the current pixel point and the corresponding background color;

[0041] The background color removal module is used to perform transparency processing on the corresponding frame of the video to be processed according to the transparency information of each pixel point of each frame, so as to output a picture with transparency color.

[0042] Optionally, it also includes a correlation calculation module, which is used to determine the background-pixel color correlation parameters according to the application scenario and transparency requirement information of the video to be processed; and calculate the correlation value between the pixel color of the current pixel and the corresponding background color according to the background-pixel color correlation parameters; wherein the background-pixel color correlation parameters include tolerance and color weight, and the tolerance is determined according to the background elimination effect and the transparency elimination range.

[0043] An embodiment of the present invention further provides an electronic device, comprising a processor, wherein the processor is configured to implement the steps of the video processing method as described in any of the preceding items when executing a computer program stored in a memory.

[0044] Finally, an embodiment of the present invention further provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the video processing method described in any of the preceding items are implemented.

[0045] The advantage of the technical solution provided by the present application is that each frame of the video being played is captured and processed and placed in a canvas that can output transparent information, and the pixel colors in the video are operated according to the correlation according to a pre-provided background color value, so that the transparency information can be inferred by using the pixel background information, and the video is continuously subjected to transparent processing operations, and the result with transparency is output, so as to achieve the goal of background color removal, and the specified background color is removed for the video with a solid color background. The whole process is not only low-cost and easy to use, but also has good compatibility and high compression rate, and has low performance pressure on the entire system, and also allows the video material to be combined with the content behind it after the transparency is removed, without the background color of the video blocking the content behind it, thereby greatly improving the use scenarios of the video material and improving the dynamic performance of the web page.

[0046] In addition, the embodiments of the present invention also provide corresponding implementation devices, electronic devices and readable storage media for the video processing method, which further make the method more practical, and the devices, electronic devices and readable storage media have corresponding advantages.

[0047] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A schematic diagram of a flow chart of a video processing method provided by an embodiment of the present invention;

[0050] Figure 2 A schematic flow chart of another video processing method provided by an embodiment of the present invention;

[0051] Figure 3 A structural diagram of a specific implementation of a video processing device provided by an embodiment of the present invention;

[0052] Figure 4 A structural diagram of a specific implementation of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0054] The terms "first", "second", "third", "fourth", etc. in the specification and claims of this application and the above drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0055] After introducing the technical solutions of the embodiments of the present invention, various non-limiting implementation methods of the present application are described in detail below.

[0056] See first Figure 1 , Figure 1A flow chart of a video processing method provided for an embodiment of the present invention. The embodiment of the present invention is applicable to a web platform. On a web platform, video playback is generally played through a video tag. This method does not support transparency, and the video itself does not have transparency. That is to say, each pixel has three color information r, g, and b representing the components of red (Red), green (Green), and blue (Blue), and the numerical value corresponding to each color information is an integer from 0 to 255. This embodiment operates the pixel colors in the video according to the correlation based on a specific color, that is, the RGB value of the background color provided in advance. The higher the correlation, the lower the transparency, and the lower the correlation, the higher the transparency. In this way, the transparency information can be inferred using the r, g, and b information. And it is necessary to continuously operate the video according to the current frame rate, and output a result with transparency. The whole process may include the following contents:

[0057] S101: When a transparency processing request is received, for each frame of the video to be processed, the captured current frame is attached as a texture feature to a target container of a canvas having the same size as the to-be-displayed size of the video to be processed.

[0058] In this embodiment, for the video being played, if the background color of the video needs to be removed, that is, the video needs to be processed for transparency, so as to output a video with transparency, a transparency processing request needs to be sent to the web platform, and the web platform receives the transparency processing request and performs a transparency processing operation on the video to be processed according to the technical solution provided in this embodiment. In this step, the video being played will update the content according to a specific frame rate. In the web platform, the upper limit of the frequency of video content update is limited by the browser, generally 60 frames per second. Every time a frame passes, the content on the video is intercepted and drawn as the current frame. In this embodiment, the video tag is no longer used to display the playing video, but the video tag is played in an invisible state, and then any video screen capture method is used to capture the screen to display the video on the canvas. The intercepted content will not be displayed, but is the original data for subsequent operations. Before the current frame is captured, it is drawn on a canvas that supports the output of transparency information. Since the present embodiment needs to calculate the transparency information based on the original pixel color, when the current frame is drawn on the canvas, the original pixel information needs to be maintained, that is, no processing is performed on the original pixel color. The target container refers to the place in the canvas where the current frame is placed. The size of the target container must be the same as the size of the video to be processed. For example, the target container can be a rectangle drawn in the canvas, which has the same size as the size to be displayed.

[0059] S102: Mapping the pixel colors of the pixels in the current frame in the target container to obtain the corresponding pixel colors of the pixels in the video to be processed.

[0060] Since the size of the target container is the same as the display size of the video to be processed, the position of the target container corresponds to the position of the current frame of the video to be processed one-to-one, and the current frame in the target container corresponds to the current frame of the video to be processed one-to-one. Accordingly, the pixel point at a certain position of the target container corresponds to the pixel point at the corresponding position of the current frame of the video to be processed. Therefore, the pixel color of each pixel in the video to be processed can be mapped through the position of the pixel point of the current frame in the target container.

[0061] S103: For each pixel of the current frame, determine transparency information of the current pixel according to a transparency elimination range, a correlation between the pixel color of the current pixel and a corresponding background color.

[0062] The present embodiment realizes the removal of the background color of the video to be processed by performing pixel-by-pixel operations on each frame of the video to be processed, that is, it is necessary to remove the background color of each pixel of each frame, so as to obtain a video with transparency after the background color is finally removed. The transparency removal range refers to the area in the current frame where the background color needs to be removed. The background color of each pixel can be obtained by any image processing algorithm or image processing software. The present application determines the transparency information based on the correlation between the background color and the pixel color. The greater the correlation, the more similar the background color and the pixel color are, and the lower the transparency is. The lower the correlation, the more dissimilar the background color and the pixel color are, and the higher the transparency is.

[0063] S104: performing transparency processing on the corresponding frame of the video to be processed according to the transparency information of each pixel point of each frame, so as to output a picture with a transparency color.

[0064] After determining the transparency information of each pixel of each frame in the previous step, a pixel-by-pixel transparency processing operation is performed on the current frame to obtain the current frame with the background color removed. Each time a frame is captured, the frame is used as the current frame, and pixel-by-pixel transparency processing is performed on the frame according to the above steps. By performing transparency processing on each frame, the final output video to be processed is a transparent picture, or in other words, the final output video to be processed is a video with the background color removed.

[0065] In the technical solution provided by the embodiment of the present invention, each frame of the video being played is captured and processed and placed in a canvas that can output transparent information. The pixel colors in the video are operated according to the correlation based on a pre-provided background color value. In this way, the transparency information can be inferred using the pixel background information, and the video is continuously subjected to transparent processing operations. The result with transparency can be output, thereby achieving the goal of background color removal, and the specified background color is removed for the video with a solid color background. The whole process is not only low-cost and easy to use, but also has good compatibility and high compression rate, and has low performance pressure on the entire system. It also allows the video material to be combined with the content behind it after the transparency is removed, and the content behind it will not be blocked by the background color of the video, thereby greatly improving the use scenarios of the video material and the dynamic performance of the web page.

[0066] In the above embodiment, there is no limitation on how to determine the correlation between the background color and the pixel color. In this embodiment, an optional implementation method for determining the correlation is provided, that is, before determining the transparency information of the current pixel point according to the correlation between the transparency elimination range, the pixel color of the current pixel point and the corresponding background color, the following steps may also be included:

[0067] According to the application scenario and transparency requirement information of the video to be processed, the background-pixel color correlation parameter is determined; according to the background-pixel color correlation parameter, the correlation value between the pixel color of the current pixel and the corresponding background color is calculated; wherein the background-pixel color correlation parameter includes a tolerance and a color weight, and the tolerance is determined according to the background removal effect and the transparency removal range.

[0068] In this embodiment, the application scenario of the video to be processed refers to the actual playback scenario of the video and the specific playback content of the video. The picture parameters of the video are determined according to the application scenario. For example, in order to highlight the red target block in the picture, the proportion of red color needs to be increased, that is, red is used as the dominant color. Transparency requirement information refers to the user's requirements for the background color removal range and transparency of the video, or the background removal effect. In this embodiment, the background removal effect and the transparency removal range are characterized by tolerance, and the picture parameters of the application scenario of the video to be processed are adjusted by color weight. The tolerance limits the range of transparency that can be removed. If t=1, only the colors within the range of background color ±1 can be processed. In some materials, this range may be too small, so that the removal range is too narrow, so it is necessary to set the tolerance to expand the removal range. The larger the tolerance, the better the background removal effect, but the more likely it is to remove some content that should not be removed. Therefore, in actual use, it is necessary to set a suitable tolerance according to the actual situation of the material. For example, the tolerances set for a video to be processed are 5, 10, and 15 respectively. As the tolerance value increases, the degree of refinement of color culling increases.

[0069] Among them, in order to match the application scenario of the video to be processed, the video to be processed can be fine-tuned, and accordingly, the color weight in the background-pixel color correlation parameter is determined according to the different requirements of the video fine-tuning. First, the video fine-tuning requirements of the video to be processed can be obtained; if the video fine-tuning requirements are to fine-tune the video based on a single color as the dominant color, the color weight corresponding to the dominant color is greater than the color weight of the non-dominant color; if the video fine-tuning requirements are to fine-tune the video based on brightness distribution, the color weight of each color is determined according to the brightness distribution information.

[0070] It is understandable that different color components can be processed the same way in general, but if you need to constrain the calculation process from some other aspects, you need to add some other auxiliary. For example, when fine-tuning, you may want the red component to be dominant, so you can increase the weight of the red part, so that more red areas will be removed. In addition, if you need to adjust according to the brightness distribution, since the contributions of r, g, and b components to brightness are different, you need to add corresponding weights according to the brightness calculation formula. Of course, in general, it is sufficient to set it to 1 / 3.

[0071] Furthermore, in order to improve the background color rejection effect, this embodiment also quantitatively represents the correlation between the background color and the pixel color. Based on the above embodiment, the correlation between the background color and the pixel color can be calculated as follows:

[0072] For the background color value [r, g, b], the background color needs to be removed. The correlation between the current color and the background color can be calculated, and then the correlation is converted into transparency. If the pixel color is [rp, gp, bp], p refers to the pixel pixel, then the correlation value between the background color and the pixel color can be calculated according to the relationship (1):

[0073] relation=w r *abs(r–r p ) / t r +w g *abs(g–g p ) / t g +w b *abs(b–b p ) / t b ; (1)

[0074] Among them, wr, wg, and wb are the weights of red, green, and blue color interpolation, respectively. In the simplest case, they can all be set to 1 / 3, but according to some needs, the weights can be set to adjust the results of correlation calculation. abs is the absolute value of the color difference. Whether it is more or less than the original color has nothing to do with the correlation. Only the absolute difference is needed here. tr, tg, and tb values ​​are the tolerances of red, green, and blue, respectively, which can adjust the range of transparency differences when each color produces transparency.

[0075] The above equation 1 is a relatively complex formula. In actual use, in order to more conveniently remove the background color, many quantities can be simplified to obtain a simpler calculation equation, that is, the correlation value can be calculated according to the following equation 2:

[0076] relation=(abs(r–r p )+abs(g–g p )+abs(b–b p )) / t / 3; (2)

[0077] In this formula, it is assumed that the weights of all components are 1 / 3, and the tolerance is unified into a variable t. This simplified formula may be more convenient in use.

[0078] Based on the above embodiment, an optional implementation process of determining the transparency information of the current pixel point according to the correlation between the transparency rejection range, the pixel color of the current pixel point and the corresponding background color may include:

[0079] Call the correlation calculation formula to calculate the correlation value between the pixel color of the current pixel and the corresponding background color. The correlation calculation formula is:

[0080] relation=w r *abs(r–r p ) / t r +w g *abs(g–g p ) / t g +w b *abs(b–b p ) / t b ;

[0081] Call the transparency relationship formula to calculate the transparency information of the current pixel. The transparency relationship formula is:

[0082] opacity=clamp(relation,min,max)

[0083] In the formula, relation is the correlation value, r, g, b are the background color values ​​of the current pixel, r p , g p 、b p is the pixel color value of the current pixel, abs represents the absolute value, w r is the weight of red color interpolation, w g is the weight of green color interpolation, w b is the weight of blue color interpolation, t r is the tolerance of red color interpolation, t g is the tolerance of green color interpolation, t b is the tolerance of blue color interpolation; opacity is the transparency information, and clamp is used to limit the relation within min and max, where min is the minimum value of the relation and max is the maximum value of the relation.

[0084] Another optional implementation process of determining the transparency information of the current pixel point according to the transparency rejection range, the correlation between the pixel color of the current pixel point and the corresponding background color may include:

[0085] Call the correlation calculation formula to calculate the correlation value between the pixel color of the current pixel and the corresponding background color. The correlation calculation formula is:

[0086] relation=(abs(r–r p )+abs(g–g p )+abs(b–b p )) / t / 3;

[0087] Call the transparency relationship formula to calculate the transparency information of the current pixel. The transparency relationship formula is:

[0088] opacity=clamp(relation,min,max)

[0089] In the formula, relation is the correlation value, r, g, b are the background color values ​​of the current pixel, r p , g p 、b p is the pixel color value of the current pixel, abs represents the absolute value, t is the tolerance; opacity is the transparency information, clamp is used to limit the relation within min and max, min is the minimum value of the relation, and max is the maximum value of the relation.

[0090] After determining the correlation value, we can get a positive natural number result relation. Using this value, we can get the transparency. Since the range of transparency is 0 to 1, we need to restrict the relation value once, that is, we can calculate the transparency information according to equation 3:

[0091] opacity=clamp(relation,min,max)(3)

[0092] The clamp function can limit a value to the range of min and max. Generally speaking, min is 0 and max is 1, which corresponds to the range of transparency. The final output color value with transparency is:

[0093] color=rgba(r p , g p , b p , opacity).

[0094] The above embodiment does not limit how to perform S101. In order to improve the compatibility of the entire background removal operation and reduce its impact on performance, this embodiment also provides an optional implementation of this step, which may include the following content:

[0095] Call canvas2d api to capture the current frame to the initial canvas;

[0096] According to the display size of the video to be processed, generate a webgl canvas with the same size as the display size, turn on the alpha channel of the webgl canvas, and turn off the premultiplication setting function of the webgl canvas;

[0097] Respond to the target container drawing instruction and draw the target container in the webgl canvas;

[0098] The current frame in the initial canvas is used as the texture feature of the target container and attached to the target container.

[0099] In this embodiment, when the video is played, the content will be updated according to a specific frame rate, while in the web platform, the upper limit of the frequency of content update is limited by the browser, generally 60 frames per second. Therefore, the API in canvas2d can be used to intercept the content on the video and draw it on the canvas every frame, as the original information for subsequent algorithm operations. For this reason, the video tag is no longer used to explicitly play the video, but the video tag is allowed to play in an invisible state, and then the canvas2d API is used to intercept the picture to display the video on the canvas. The intercepted content will not be displayed explicitly, but will be temporarily stored as the original data for subsequent algorithm operations. The canvas2d API in the web has good performance when operating when drawing images, but the performance of pixel-by-pixel operations is very low. Webgl is a 3D drawing protocol on the web platform, which allows front-end developers of the web platform to use graphics acceleration hardware to efficiently render images. Therefore, this embodiment needs to use the webgl API, which has very good compatibility and uses graphics hardware acceleration when operating pixel by pixel, and has very good performance. It is necessary to generate a webgl canvas of the same size as the video to be displayed. In this webgl canvas, the alpha channel, that is, transparency, needs to be turned on so that a final picture containing transparency information can be output. This setting can be set through the alpha configuration item when the webgl canvas is initialized. Just set it to true. In addition, the default configuration of webgl will pre-multiply transparency, that is, multiply the RGB value with the alpha in advance. Since the original pixel color is needed here to infer transparency, the pre-multiplication setting must be turned off. You can set the configuration item premultipliedAlpha of the initial webgl canvas to false to turn it off. After that, draw a rectangle of the same size as the canvas in the webgl canvas. Then attach the screen captured in the canvas2d api to this rectangle as the texture of the rectangle.

[0100] In order to make the technical personnel in the relevant field more clearly understand the technical solution of the present application, the present application is also combined with Figure 2 An illustrative example is given, which may include:

[0101] For the video being played, when transparency processing is required, call canvas2dapi to capture the current frame to the initial canvas.

[0102] According to the display size of the video to be processed, a webgl canvas with the same size as the display size is generated, and the alpha channel of the webgl canvas is turned on, while the premultiplication setting function of the webgl canvas is turned off.

[0103] Draw a rectangle of the same size as the size to be displayed in the webgl canvas, and attach the current frame in the initial canvas as a texture feature to the rectangle.

[0104] Call the pixel shader that runs the operation once on each pixel to perform the following background color culling operation: The pixel shader maps the pixel color of the corresponding video screen according to the position of the current pixel on the rectangle. Pass the value of the background color corresponding to the current pixel to the pixel shader. For example, if it is a black background, the corresponding background color value is [0, 0, 0]. These three numbers represent the amount of red, green and blue color components, ranging from 0 to 255. After obtaining the pixel color and background color values ​​of the current pixel, the correlation value between the pixel color and the background color of the current pixel is calculated according to the above relationship 2. According to the correlation value, call the above relationship 3 to calculate the transparency value of the current pixel. According to color=rgba(r p , g p , b p , opacity) outputs a color value with transparency. Through the above process, the background color of the video can be removed and a result with transparency can be output.

[0105] It should also be noted that when the background color is completely black, the additive color mode of layer overlay can be used to achieve the purpose of background removal.

[0106] After obtaining the pixel color and background color value of each pixel point through the pixel shader, this embodiment can efficiently and batch determine the corresponding transparency in the pixel shader, and then output a result with transparency, thereby improving the efficiency of background color culling.

[0107] It should be noted that there is no strict order of execution between the steps in this application. As long as they comply with the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 1-Figure 2 This is just a schematic and does not mean that this is the only execution order.

[0108] The embodiment of the present invention also provides a corresponding device for the video processing method, which further makes the method more practical. The device can be described from the perspective of functional modules and hardware. The video processing device provided by the embodiment of the present invention is introduced below. The video processing device described below and the video processing method described above can be referred to each other.

[0109] From the perspective of functional modules, see Figure 3 , Figure 3 A structural diagram of a video processing device provided in an embodiment of the present invention in a specific implementation manner, the device may include:

[0110] The image preprocessing module 301 is used to, when receiving a transparency processing request, attach the captured current frame image as a texture feature to a target container of a canvas with the same size as the to-be-processed video for each frame of the to-be-processed video; the canvas supports transparency information and does not process the pixel color in advance.

[0111] The pixel color determination module 302 is used to map and obtain the pixel color of each pixel in the video to be processed according to the position of the pixel of the current frame in the target container.

[0112] The transparency information determination module 303 is used to determine the transparency information of the current pixel for each pixel of the current frame according to the transparency elimination range, the correlation between the pixel color of the current pixel and the corresponding background color.

[0113] The background color removal module 304 is used to perform transparency processing on the corresponding frame of the video to be processed according to the transparency information of each pixel point of each frame, so as to output a picture with transparency color.

[0114] Optionally, in some implementations of the present embodiment, the above-mentioned device may also include a correlation calculation module, for example, for determining a background-pixel color correlation parameter based on the application scenario and transparency requirement information of the video to be processed; and calculating the correlation value between the pixel color of the current pixel and the corresponding background color based on the background-pixel color correlation parameter; wherein the background-pixel color correlation parameter includes a tolerance and a color weight, and the tolerance is determined based on the background elimination effect and the transparency elimination range.

[0115] As an optional implementation of the above embodiment, the above correlation calculation module can also be used to: obtain the video fine-tuning requirement of the video to be processed; if the video fine-tuning requirement is to perform video fine-tuning based on a single color as the dominant color, then the color weight corresponding to the dominant color is greater than the color weight of the non-dominant color; if the video fine-tuning requirement is to perform video fine-tuning based on brightness distribution, then the color weight of each color is determined according to the brightness distribution information.

[0116] Optionally, in some other implementations of this embodiment, the transparency information determination module 303 may be further used to:

[0117] Call the correlation calculation formula to calculate the correlation value between the pixel color of the current pixel and the corresponding background color. The correlation calculation formula is:

[0118] relation=w r *abs(r–r p ) / t r +w g *abs(g–g p ) / tg +w b *abs(b–b p ) / t b ;

[0119] Call the transparency relationship formula to calculate the transparency information of the current pixel. The transparency relationship formula is:

[0120] opacity=clamp(relation,min,max)

[0121] In the formula, relation is the correlation value, r, g, b are the background color values ​​of the current pixel, r p , g p 、b p is the pixel color value of the current pixel, abs represents the absolute value, w r is the weight of red color interpolation, w g is the weight of green color interpolation, w b is the weight of blue color interpolation, t r is the tolerance of red color interpolation, t g is the tolerance of green color interpolation, t b is the tolerance of blue color interpolation; opacity is the transparency information, and clamp is used to limit the relation within min and max, where min is the minimum value of the relation and max is the maximum value of the relation.

[0122] Optionally, in some other implementations of this embodiment, the transparency information determination module 303 may be further used to:

[0123] Call the correlation calculation formula to calculate the correlation value between the pixel color of the current pixel and the corresponding background color. The correlation calculation formula is:

[0124] relation=(abs(r–r p )+abs(g–g p )+abs(b–b p )) / t / 3;

[0125] Call the transparency relationship formula to calculate the transparency information of the current pixel. The transparency relationship formula is:

[0126] opacity=clamp(relation,min,max)

[0127] In the formula, relation is the correlation value, r, g, b are the background color values ​​of the current pixel, r p , g p 、b pis the pixel color value of the current pixel, abs represents the absolute value, t is the tolerance; opacity is the transparency information, clamp is used to limit the relation within min and max, min is the minimum value of the relation, and max is the maximum value of the relation.

[0128] As an optional implementation of this embodiment, the above-mentioned picture preprocessing module 301 can be further used to: call the canvas2d API to capture the current frame picture to the initial canvas; generate a webgl canvas with the same size as the display size of the video to be processed, and turn on the alpha channel of the webgl canvas, and turn off the premultiplication setting function of the webgl canvas; respond to the target container drawing instruction, and draw the target container in the webgl canvas; use the current frame picture in the initial canvas as the texture feature of the target container and attach it to the target container.

[0129] The functions of the functional modules of the video processing device described in the embodiment of the present invention can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0130] It can be seen from the above that the embodiment of the present invention can achieve low-cost and convenient removal of video background color on the basis of compatibility and low impact on system performance.

[0131] The video processing device mentioned above is described from the perspective of functional modules. Furthermore, the present application also provides an electronic device, which is described from the perspective of hardware. Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application in one implementation manner. Figure 4 As shown, the electronic device includes a memory 40 for storing a computer program; and a processor 41 for implementing the steps of the video processing method mentioned in any of the above embodiments when executing the computer program.

[0132] Among them, the processor 41 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 41 may also be a controller, a microcontroller, a microprocessor or other data processing chip. The processor 41 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 41 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 41 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 41 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0133] The memory 40 may include one or more computer-readable storage media, which may be non-transitory. The memory 40 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. The memory 40 may be an internal storage unit of an electronic device in some embodiments, such as a hard disk of a server. The memory 40 may also be an external storage device of an electronic device in other embodiments, such as a plug-in hard disk equipped on a server, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Further, the memory 40 may also include both an internal storage unit of an electronic device and an external storage device. The memory 40 may not only be used to store application software and various types of data installed in the electronic device, such as: the code of the program for executing the vulnerability processing method, etc., but may also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 40 is at least used to store the following computer program 401, wherein, after the computer program is loaded and executed by the processor 41, the relevant steps of the video processing method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 40 may also include an operating system 402 and data 403, and the storage method may be temporary storage or permanent storage. The operating system 402 may include Windows, Unix, Linux, etc. The data 403 may include but is not limited to data corresponding to the video processing result.

[0134] In some embodiments, the electronic device may further include a display screen 42, an input / output interface 43, a communication interface 44 or a network interface, a power supply 45 and a communication bus 46. Among them, the display screen 42 and the input / output interface 43 such as a keyboard belong to the user interface, and the optional user interface may also include a standard wired interface, a wireless interface, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface. The communication interface 44 may optionally include a wired interface and / or a wireless interface, such as a WI-FI interface, a Bluetooth interface, etc., which is usually used to establish a communication connection between the electronic device and other electronic devices. The communication bus 46 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0135] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than those shown in the figure, for example, it may also include a sensor 47 for realizing various functions.

[0136] The functions of the functional modules of the electronic device described in the embodiment of the present invention can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0137] It can be seen from the above that the embodiment of the present invention can achieve low-cost and convenient removal of video background color on the basis of compatibility and low impact on system performance.

[0138] It is understandable that if the video processing method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, removable disk, CD-ROM, disk or optical disk, etc. Various media that can store program codes.

[0139] Based on this, an embodiment of the present invention further provides a readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the video processing method described in any of the above embodiments are performed.

[0140] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the hardware disclosed in the embodiments, including devices and electronic devices, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0141] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0142] The above is a detailed introduction to a video processing method, device, electronic device and readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A video processing method, characterized in that: include: When a transparency processing request is received, for each frame of the video to be processed, the captured current frame is attached as a texture feature to a target container of a canvas having the same size as the to-be-displayed size of the video to be processed; the canvas supports transparency information and does not process pixel colors in advance; Mapping the pixel colors of the pixels in the current frame to obtain the corresponding pixel colors in the video to be processed; For each pixel point of the current frame, the correlation value between the pixel color of the current pixel point and the corresponding background color is calculated: relation=w r *abs(r–r p ) / t r +w g *abs(g–g p ) / t g +w b *abs(b–b p ) / t b ; or, relation = (abs(r–r p )+abs(g–g p )+abs(b–b p )) / t / 3; The transparency information of the current pixel is calculated as follows: opacity=clamp(relation, min, max); relation is the correlation value, r, g, b are the background color values, r p , g p 、b p is the pixel color value, abs means absolute value, w r 、w g 、w b are the weights of color interpolation for red, green, and blue, t is the tolerance, and t r ,t g ,t b are the color interpolation tolerances for red, green, and blue, respectively; opacity is the transparency information, and clamp limits the relation to min and max, where min and max are the minimum and maximum values ​​of the relation, respectively; According to the transparency information of each pixel point of each frame, the corresponding frame of the video to be processed is processed for transparency, so as to output a picture with a transparency color.

2. The video processing method according to claim 1, characterized in that: Also includes: Determining background-pixel color correlation parameters according to the application scenario and transparency requirement information of the video to be processed; Calculating the correlation value between the pixel color of the current pixel and the corresponding background color according to the background-pixel color correlation parameter; The background-pixel color correlation parameter includes a tolerance and a color weight, and the tolerance is determined according to the background removal effect and the transparency removal range.

3. The video processing method according to claim 2, characterized in that: Determining the background-pixel color correlation parameter according to the application scenario and transparency requirement information of the video to be processed includes: Obtaining video fine-tuning requirements for the video to be processed; If the video fine-tuning requirement is to perform video fine-tuning based on a single color as a dominant color, the color weight corresponding to the dominant color is greater than the color weight of the non-dominant color; If the video fine-tuning requirement is to perform video fine-tuning based on brightness distribution, the color weight of each color is determined according to brightness distribution information.

4. The video processing method according to claim 1, characterized in that: The method of attaching the captured current frame as a texture feature to a target container of a canvas having the same size as the to-be-processed video to be displayed includes: Call canvas2d api to capture the current frame to the initial canvas; According to the to-be-processed video's display size, generate a webgl canvas with the same size as the to-be-processed video's display size, open an alpha channel of the webgl canvas, and simultaneously close a premultiplication setting function of the webgl canvas; In response to a target container drawing instruction, draw the target container in the webgl canvas; The current frame picture in the initial canvas is used as the texture feature of the target container and attached to the target container.

5. A video processing device, characterized in that: include: The image preprocessing module is used for, when receiving a transparency processing request, attaching the captured current frame as a texture feature to a target container of a canvas with the same size as the to-be-processed video for each frame of the to-be-processed video; the canvas supports transparency information and does not process pixel colors in advance; A pixel color determination module, used for mapping the pixel color of each pixel in the video to be processed according to the position of the pixel of the current frame in the target container; The transparency information determination module is used to calculate the correlation value between the pixel color of the current pixel and the corresponding background color for each pixel of the current frame: relation=w r *abs(r–r p ) / t r +w g *abs(g–g p ) / t g +w b *abs(b–b p ) / t b ; or, relation = (abs(r–r p )+abs(g–g p )+abs(b–b p )) / t / 3; The transparency information of the current pixel is calculated as follows: opacity=clamp(relation, min, max); relation is the correlation value, r, g, b are the background color values, r p , g p 、b p is the pixel color value, abs means absolute value, w r 、w g 、w b are the weights of color interpolation for red, green, and blue, t is the tolerance, and t r ,t g ,t b are the color interpolation tolerances for red, green, and blue, respectively; opacity is the transparency information, and clamp limits the relation to min and max, where min and max are the minimum and maximum values ​​of the relation, respectively; The background color removal module is used to perform transparency processing on the corresponding frame of the video to be processed according to the transparency information of each pixel point of each frame, so as to output a picture with transparency color.

6. The video processing device according to claim 5, characterized in that: It also includes a correlation calculation module, which is used to determine the background-pixel color correlation parameter according to the application scenario and transparency requirement information of the video to be processed; and calculate the correlation value between the pixel color of the current pixel and the corresponding background color according to the background-pixel color correlation parameter; wherein the background-pixel color correlation parameter includes a tolerance and a color weight, and the tolerance is determined according to the background removal effect and the transparency removal range.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor is used to implement the steps of the video processing method according to any one of claims 1 to 4 when executing a computer program stored in the memory.

8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 4 are implemented.

Images