Bullet screen anti-blocking method, device, equipment and storage medium

Abstract

The present disclosure relates to a bullet screen anti-blocking method, device, equipment and storage medium, and relates to the field of video technology. The method includes: performing text recognition on multiple video frames in a video sequence of a target video, obtaining a binary grayscale image including a first area and a second area for each of the multiple video frames, and the grayscale value corresponding to each pixel in the binary grayscale image is used to indicate whether the pixel belongs to the first area, and the first area is the central area of ​​the text area in the video frame; for each video frame, if the overlap rate between the first area in the binary grayscale image of the video frame and the first area in the binary grayscale image of the previous video frame of the video frame is greater than a preset threshold, the text area in the previous video frame is used as the text area in the video frame; based on the text area in the video frame, a text mask of the video frame is generated, and the text mask is used to render the bullet screen of the target video so that the bullet screen is displayed in an area outside the text area, thereby improving the bullet screen anti-blocking effect.

Description

Technical Field

[0001] The present disclosure relates to the field of video technology, and in particular to a bullet screen anti-blocking method, device, equipment and storage medium. Background Art

[0002] Currently, many streaming videos have the function of displaying bullet screens, and the text in the video is often blocked by the bullet screen, affecting the video viewing experience. Therefore, it is necessary to identify the text area in the video to prevent the bullet screen from blocking the text area. However, when identifying the text area, the identified text area may jitter, resulting in different text areas being identified in multiple video frames with the same text area, which in turn causes the bullet screen to appear and disappear, reducing the anti-blocking effect of the bullet screen. Summary of the invention

[0003] The present disclosure provides a bullet screen anti-blocking method, device, equipment and storage medium, which can improve the stability of text recognition, avoid jitter in the recognized text area, and then generate a text mask based on the stable text area, and render the bullet screen on the video based on the text mask, which can ensure the stability of the bullet screen display, avoid the occasional appearance of the bullet screen, and improve the anti-blocking effect of the bullet screen. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a bullet screen anti-blocking method is provided, the method comprising:

[0005] Performing text recognition on multiple video frames in a video sequence of a target video respectively, and obtaining binary grayscale images of the multiple video frames, wherein the binary grayscale images include a first area and a second area, and the grayscale value corresponding to each pixel in the binary grayscale image is used to indicate whether the pixel belongs to the first area, the first area is a central area of ​​a text area in the video frame, and the second area is an area outside the first area;

[0006] For each of the video frames, if the overlap rate between the first region in the binary grayscale image of the video frame and the first region in the binary grayscale image of the previous video frame of the video frame is greater than a preset threshold, the text region in the previous video frame is used as the text region in the video frame;

[0007] A text mask of the video frame is generated based on a text area in the video frame, and the text mask is used to render the barrage on the target video so that the barrage is displayed in an area outside the text area.

[0008] According to a second aspect of an embodiment of the present disclosure, a barrage anti-blocking device is provided, the device comprising:

[0009] A text recognition unit is configured to perform text recognition on a plurality of video frames in a video sequence of a target video, respectively, to obtain a binary grayscale image of each of the plurality of video frames, wherein the binary grayscale image includes a first region and a second region, and a grayscale value corresponding to each pixel in the binary grayscale image is used to indicate whether the pixel belongs to the first region, the first region is a central region of a text region in the video frame, and the second region is a region outside the first region;

[0010] a text region determining unit configured to, for each of the video frames, if a first region in a binary grayscale image of the video frame and a first region in a binary grayscale image of a previous video frame of the video frame have an overlap ratio greater than a preset threshold, determine the text region in the previous video frame as the text region in the video frame;

[0011] The mask generation unit is configured to generate a text mask of the video frame based on a text area in the video frame, and the text mask is used to render the barrage on the target video so that the barrage is displayed in an area outside the text area.

[0012] In some embodiments, the text recognition unit is configured to perform text recognition on the multiple video frames respectively to obtain text probability maps for the multiple video frames, wherein the text probability map includes probability values ​​corresponding to multiple pixel points respectively, and the probability value represents the probability that the pixel point belongs to the first area; for the text probability map of each video frame, the pixel points with a probability value greater than a probability threshold are assigned a first grayscale value, and the pixel points with a probability value less than or equal to the probability threshold are assigned a second grayscale value to obtain the binary grayscale map, wherein the first grayscale value is used to indicate the first area, and the second grayscale value is used to indicate the second area.

[0013] In some embodiments, the device also includes an overlap rate determination unit, which is configured to determine the intersection and union of pixel points of a first area in the binary grayscale image of the video frame and a first area in the binary grayscale image of the previous video frame; based on the intersection and the union, determine the overlap rate, and the overlap rate is positively correlated with the intersection and negatively correlated with the union.

[0014] In some embodiments, the text area determination unit is further configured to perform area expansion on the first area in the binary grayscale image of the video frame to obtain the text area in the video frame if the overlap rate is less than or equal to the preset threshold.

[0015] In some embodiments, the mask generation unit is configured to use the text area as a non-filled area and the area outside the text area as a filled area to generate the text mask, and the filled area is an area where barrage is allowed to be displayed.

[0016] In some embodiments, the target video is a video with a greater number of barrages than a preset number of barrages; or, the target video is a video with an interaction rate greater than a preset interaction rate; or, the target video is a video with a viewing rate greater than a preset viewing rate; or, the target video is a video with text displayed in the barrage area.

[0017] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0018] one or more processors;

[0019] A memory for storing program codes executable by the processor;

[0020] Among them, the processor is configured to execute the program code to implement the above-mentioned barrage anti-occlusion method.

[0021] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device can execute the above-mentioned barrage anti-obstruction method.

[0022] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned barrage anti-occlusion method when executed by a processor.

[0023] The disclosed embodiment provides a method for preventing barrage from being occluded, which performs text recognition on multiple video frames respectively to obtain binary grayscale images of the multiple video frames, wherein the binary grayscale images indicate the central area of ​​the text area in the video frame; and for each video frame, when the overlap rate between the central area of ​​the text area in the video frame and the central area of ​​the text area in the previous video frame is greater than a preset threshold, it indicates that the video frame has a high probability of being the same as the text area in the previous video frame, so that the text area in the previous video frame is used as the text area in the video frame, which can improve the stability of text recognition and avoid jittering of the recognized text area, and then generate a text mask based on the stable text area, and render the video based on the text mask, which can ensure the stability of the barrage display, avoid the intermittent appearance of the barrage, and improve the anti-occlusion effect of the barrage.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0026] Figure 1 It is a schematic diagram showing an implementation environment according to an exemplary embodiment.

[0027] Figure 2 The present invention is a flowchart of a bullet screen anti-blocking method according to an exemplary embodiment.

[0028] Figure 3 The present invention is a flowchart of another bullet screen anti-blocking method according to an exemplary embodiment.

[0029] Figure 4 The figure is a schematic diagram showing a method of recognizing a text area according to an exemplary embodiment.

[0030] Figure 5 The present invention is a flowchart of another bullet screen anti-blocking method according to an exemplary embodiment.

[0031] Figure 6 It is a schematic diagram of a framework of barrage anti-occlusion according to an exemplary embodiment.

[0032] Figure 7 It is a block diagram of a barrage anti-blocking device according to an exemplary embodiment.

[0033] Figure 8 It is a block diagram of a terminal according to an exemplary embodiment.

[0034] Fig. 9 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0035] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0037] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the videos involved in this disclosure are all obtained with full authorization.

[0038] The barrage anti-blocking method provided in the embodiment of the present disclosure is executed by an electronic device, and the electronic device is provided as a server. Figure 1 This is a schematic diagram of an implementation environment provided by the embodiment of the present disclosure, see Figure 1 , the implementation environment includes: a terminal 101, a first server 102 and a second server 103. In the embodiment of the present disclosure, the terminal 101 is installed with a target application for playing videos, the first server 102 is a background server of the target application, the second server 103 is a server for processing videos, and the second server can be a video-on-demand cloud. The first server 102 is used to process videos with the help of the second server 103. For example, the second server 103 is used for video transcoding.

[0039] The terminal 101 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop, a virtual reality terminal, an augmented reality terminal, a wireless terminal, and a laptop computer. The terminal 101 has a communication function and can access a wired network or a wireless network. The terminal 101 may generally refer to one of a plurality of terminals, and those skilled in the art may know that the number of the above terminals may be more or less. The first server 102 and the second server 103 may be independent physical servers, or a server cluster or a distributed file system composed of a plurality of physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some embodiments, the first server 102 is directly or indirectly connected to the terminal 101 and the second server 103 by wired or wireless communication, which is not limited in the embodiments of the present disclosure. Optionally, the number of the first server 102 and the second server 103 may be more or less, which is not limited in the embodiments of the present disclosure. Of course, the first server 102 may also include other functional servers to provide more comprehensive and diversified services. The first server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or, the first server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or, the first server 102 or the terminal 101 can each independently undertake the computing work, which is not limited in the embodiments of the present disclosure.

[0040] Figure 2 is a flow chart of a bullet screen anti-blocking method according to an exemplary embodiment. Figure 2 As shown, the method is executed by the second server, and the method includes the following steps:

[0041] In step S201, the second server performs text recognition on multiple video frames in the video sequence of the target video to obtain binary grayscale images of the multiple video frames. The binary grayscale image includes a first area and a second area. The grayscale value corresponding to each pixel in the binary grayscale image is used to indicate whether the pixel belongs to the first area. The first area is the central area of ​​the text area in the video frame, and the second area is the area outside the first area.

[0042] In the disclosed embodiment, the target video is a video including text, so as to identify the text area in the video and prevent the bullet screen from covering the text in the text area. The text in the text area can be a video title, a video subtitle, etc.

[0043] In the embodiment of the present disclosure, the binary grayscale image includes multiple pixel points whose grayscale values ​​are the first grayscale value or the second grayscale value, the first grayscale value and the second grayscale value are different values, the pixel points with the first grayscale value belong to the first area, and the pixel points with the second grayscale value belong to the second area.

[0044] In the embodiment of the present disclosure, the central area of ​​the text area is similar in shape to the text area and has the same center point. The area size of the central area is smaller than the area size of the text area, that is, it can be understood that the central area is obtained by reducing the text area according to a preset ratio.

[0045] In step S202, for each video frame, if the overlap rate between the first area in the binary grayscale image of the video frame and the first area in the binary grayscale image of the previous video frame of the video frame is greater than a preset threshold, the second server uses the text area in the previous video frame as the text area in the video frame.

[0046] In the disclosed embodiment, if the overlap rate between the first area in the binary grayscale image of the video frame and the first area in the binary grayscale image of the previous video frame is greater than the preset threshold, it means that the text area in the video frame and the previous video frame is more likely to be the same. If the overlap rate is less than or equal to the preset threshold, it means that the text area in the video frame and the previous video frame is more likely to be different, and the binary grayscale image of the video frame is post-processed to obtain the text area in the video frame. The preset threshold can be set and changed as needed, and is not specifically limited here.

[0047] In step S203, the second server generates a text mask of the video frame based on the text area in the video frame, and the text mask is used to render the bullet screen for the target video so that the bullet screen is displayed in an area outside the text area.

[0048] In the disclosed embodiment, the text mask includes a plurality of pixel points whose pixel values ​​are the first value or the second value, the area to which the pixel points of the first value belong corresponds to the text area, and the area to which the pixel points of the second value belong corresponds to the non-text area. That is, when the text mask is generated based on the text area, the pixel points on the text mask corresponding to the text area are assigned the first value, and the pixel points on the text mask corresponding to the non-text area are assigned the second value to obtain the text mask, and the first value indicates the area where the bullet screen is not allowed to be displayed, and the second value indicates the area where the bullet screen is allowed to be displayed.

[0049] In the embodiment of the present disclosure, when barrage rendering is performed based on a text mask, the barrage display is controlled based on the pixel values ​​of multiple pixels on the text mask, that is, the barrage is only displayed on the area composed of pixel points whose pixel values ​​are the second value, and then the barrage rendering is performed based on the text mask, which can ensure that the barrage is displayed in an area outside the text area.

[0050] An embodiment of the present disclosure provides a method for preventing barrage from being occluded. The method performs text recognition on multiple video frames respectively to obtain binary grayscale images of the multiple video frames, wherein the binary grayscale images indicate the central area of ​​the text area in the video frame. For each video frame, when the overlap rate between the central area of ​​the text area in the video frame and the central area of ​​the text area in the previous video frame is greater than a preset threshold, it indicates that the video frame is highly likely to be the same as the text area in the previous video frame. In this way, the text area in the previous video frame is used as the text area in the video frame, which can improve the stability of text recognition and avoid jittering of the recognized text area. Further, a text mask is generated based on the stable text area, and barrage rendering is performed on the video based on the text mask. This can ensure the stability of the barrage display and avoid the intermittent appearance of the barrage, thereby improving the anti-occlusion effect of the barrage.

[0051] In some embodiments, text recognition is performed on multiple video frames in a video sequence of a target video to obtain binary grayscale images of the multiple video frames, including: text recognition is performed on multiple video frames to obtain text probability maps of the multiple video frames, the text probability map includes probability values ​​corresponding to multiple pixel points, and the probability value represents the probability that the pixel point belongs to the first area; for the text probability map of each video frame, a first grayscale value is assigned to a pixel point with a probability value greater than a probability threshold, and a second grayscale value is assigned to a pixel point with a probability value less than or equal to the probability threshold, to obtain a binary grayscale map, the first grayscale value is used to indicate the first area, and the second grayscale value is used to indicate the second area.

[0052] In the embodiment of the present disclosure, since pixels with large probability values ​​have a greater probability of belonging to the first area, and pixels with small probability values ​​have a greater probability of belonging to the second area, multiple pixels are assigned values ​​based on the probability threshold to divide the first area and the second area, and a binary grayscale image is obtained, thereby improving the accuracy of the determined binary grayscale image.

[0053] In some embodiments, the process of determining the overlap rate includes: determining the intersection and union of pixel points of a first area in a binary grayscale image of a video frame and a first area in a binary grayscale image of a previous video frame; based on the intersection and union, determining the overlap rate, the overlap rate is positively correlated with the intersection and negatively correlated with the union.

[0054] In the embodiment of the present disclosure, the overlap rate is determined based on the intersection and union of the pixel points in the first area, thereby ensuring the accuracy and rationality of the determined overlap rate.

[0055] In some embodiments, the method further includes: if the overlap rate is less than or equal to a preset threshold, performing region expansion on the first region in the binary grayscale image of the video frame to obtain a text region in the video frame.

[0056] In the disclosed embodiment, if the overlap rate is less than a preset threshold, it means that there is a high probability that the video frame is different from the text area in the previous video frame, and then the first area is expanded to obtain the text area, thereby improving the efficiency of obtaining the text area on the basis of improving the rationality of obtaining the text area.

[0057] In some embodiments, a text mask of a video frame is generated based on a text area in the video frame, including: using the text area as a non-filled area and using an area outside the text area as a filled area to generate a text mask, wherein the filled area is an area where barrage is allowed to be displayed.

[0058] In the embodiment of the present disclosure, the text area is used as a non-filled area, and the area outside the text area is used as a filled area allowing the barrage to be displayed, to generate a text mask. In this way, when the barrage is rendered based on the text mask, the barrage will only be displayed in the area outside the text area, and will not be displayed in the text area, thereby effectively achieving the anti-occlusion effect of the barrage.

[0059] In some embodiments, the target video is a video with a greater number of barrages than a preset number of barrages; or, the target video is a video with an interaction rate greater than a preset interaction rate; or, the target video is a video with a viewing rate greater than a preset viewing rate; or, the target video is a video with text displayed in the barrage area.

[0060] In the disclosed embodiment, videos with a large number of bullet screens, a large interaction rate, and a large viewing rate have a wide audience, and thus performing bullet screen anti-blocking on such videos can improve the effectiveness of bullet screen anti-blocking. Since only text displayed in the bullet screen area will be blocked by the bullet screen, only processing videos with text displayed in the bullet screen area can improve the effectiveness of bullet screen anti-blocking.

[0061] The disclosed embodiment provides a method for preventing barrage from being occluded, which performs text recognition on multiple video frames respectively to obtain binary grayscale images of the multiple video frames, wherein the binary grayscale images indicate the central area of ​​the text area in the video frame; and for each video frame, when the overlap rate between the central area of ​​the text area in the video frame and the central area of ​​the text area in the previous video frame is greater than a preset threshold, it indicates that the video frame has a high probability of being the same as the text area in the previous video frame, so that the text area in the previous video frame is used as the text area in the video frame, which can improve the stability of text recognition and avoid jittering of the recognized text area, and then generate a text mask based on the stable text area, and render the video based on the text mask, which can ensure the stability of the barrage display, avoid the intermittent appearance of the barrage, and improve the anti-occlusion effect of the barrage.

[0062] Above Figure 2This is the basic process for anti-blocking of bullet screen. Figure 3 Further elaboration on the process of anti-blocking of bullet screen. Figure 3 , Figure 3 The present invention is a flowchart of a bullet screen anti-blocking method according to an exemplary embodiment. The method is executed by a second server and includes the following steps.

[0063] In step S301, the second server obtains a video sequence of a target video, where the video sequence includes multiple video frames.

[0064] In an embodiment of the present disclosure, the target video is a screened video. In some embodiments, the target video is a video that has been played, then the target video is a video whose number of barrages is greater than a preset number of barrages; or, the target video is a video whose interaction rate is greater than a preset interaction rate, and the interaction rate includes at least one of a like rate, a forwarding rate, a collection rate, etc.; or, the target video is a video whose viewing rate is greater than a preset viewing rate. The target video may also be a video that satisfies at least two of the above. For example, the target video is a video whose number of barrages is greater than a preset number of barrages and whose interaction rate is greater than a preset interaction rate. In this embodiment, videos with a large number of barrages, a large interaction rate, and a large viewing rate have a wider audience, and then performing barrage anti-occlusion on such videos can improve the effectiveness of barrage anti-occlusion.

[0065] In other embodiments, the target video is a video that has not been played. Since the target video is a video containing text, in order to further improve the effectiveness of video processing, the target video is a video in which text is displayed in the bullet screen area. Since only text displayed in the bullet screen area will be blocked by the bullet screen, only processing the video in which text is displayed in the bullet screen area can improve the effectiveness of bullet screen anti-blocking.

[0066] Optionally, when the target video is a video that has been played, the target video may also be a video with text displayed in the barrage area, that is, the target video not only satisfies the requirement of text being displayed in the barrage area, but also the number of barrages, interaction rate and viewing rate satisfy at least one of the preset number of barrages, preset interaction rate and preset viewing rate, so as to further improve the effectiveness of barrage anti-occlusion.

[0067] In the disclosed embodiments, in order to improve the efficiency of text recognition for the target video. In some embodiments, if the frame rate of the target video is greater than the target frame rate and the resolution is greater than the target resolution, the target video is transcoded based on the target frame rate and the target resolution to obtain the video sequence. Optionally, the target frame rate and the target resolution are the minimum frame rate and the minimum resolution supported by the target application for playing the video, respectively, and the video with a lower frame rate and resolution is easy to process, thereby facilitating the improvement of the efficiency of subsequent text recognition. For example, the target frame rate is 30fps and the target resolution is 360p. The number of multiple video frames in the video sequence matches the target frame rate, that is, the number is the product of the target frame rate and the duration of the target video. For example, the target frame rate is 30fps, which means that the video is played at a speed of 30 frames per second, and there is the same playback time interval between multiple video frames. If the duration of the target video is 10 seconds, the number of multiple video frames in the video sequence is 300.

[0068] In step S302, the second server performs text recognition on multiple video frames in the video sequence of the target video respectively to obtain binary grayscale images of the multiple video frames, wherein the binary grayscale image includes a first area and a second area. The grayscale value corresponding to each pixel in the binary grayscale image is used to indicate whether the pixel belongs to the first area. The first area is the central area of ​​the text area in the video frame, and the second area is the area outside the first area.

[0069] In an embodiment of the present disclosure, the second server performs text recognition on multiple video frames in a video sequence of a target video, and the process of obtaining a binary grayscale image of each of the multiple video frames includes the following steps: the second server performs text recognition on multiple video frames, and obtains a text probability map of each of the multiple video frames, wherein the text probability map includes probability values ​​corresponding to multiple pixel points, and the probability value indicates the probability that the pixel point belongs to the first area; for the text probability map of each video frame, the second server assigns a first grayscale value to a pixel point whose probability value is greater than a probability threshold, and assigns a second grayscale value to a pixel point whose probability value is less than or equal to the probability threshold, to obtain a binary grayscale image, wherein the first grayscale value is used to indicate the first area, and the second grayscale value is used to indicate the second area.

[0070] In the embodiment of the present disclosure, the first grayscale value and the second grayscale value are different values, and the first grayscale value and the second grayscale value are different according to different grayscale value algorithms, and then the first area and the second area can be distinguished based on the first grayscale value and the second grayscale value. Optionally, the second grayscale value is 0, and the first grayscale value is a non-zero value, then the second server determines the pixel point with a non-zero grayscale value as belonging to the first area, and determines the pixel point with a grayscale value of 0 as belonging to the second area. Optionally, the first grayscale value and the second grayscale value are any two different non-zero values.

[0071] In the embodiment of the present disclosure, since pixels with large probability values ​​have a greater probability of belonging to the first area, and pixels with small probability values ​​have a greater probability of belonging to the second area, multiple pixels are assigned values ​​based on the probability threshold to divide the first area and the second area, and a binary grayscale image is obtained, thereby improving the accuracy of the determined binary grayscale image.

[0072] In step S303, the second server determines, for each video frame, an overlap ratio between a first region in a binary grayscale image of the video frame and a first region in a binary grayscale image of a previous video frame of the video frame.

[0073] In the disclosed embodiment, the process of determining the overlap rate by the second server includes the following steps: the second server determines the intersection and union of the pixel points of the first area in the binary grayscale image of the video frame and the first area in the binary grayscale image of the previous video frame; the second server determines the overlap rate based on the intersection and the union, and the overlap rate is positively correlated with the intersection and negatively correlated with the union. In the disclosed embodiment, the overlap rate is determined based on the intersection and the union of the pixel points of the first area, which ensures the accuracy and rationality of the determined overlap rate.

[0074] In some embodiments, the second server obtains the pixel identifiers of the pixel points in the first region of the binary grayscale image of the video frame and the pixel identifiers of the pixel points in the first region of the binary grayscale image of the previous video frame, and determines the intersection and union of the pixel points in the first region corresponding to the two video frames based on the pixel identifiers of the multiple pixel points. Optionally, the pixel identifier is a pixel coordinate to improve the efficiency and accuracy of determining the intersection and union.

[0075] Optionally, the second server determines the overlap rate based on the intersection and the union by using the following formula (1).

[0076] iou=inter / (union+1e-6) (1)

[0077] Among them, for any video frame, iou represents the overlap rate between the first area in the binary grayscale image of the video frame and the first area in the binary grayscale image of the previous video frame of the video frame, inter represents the intersection of the pixel points of the first area in the binary grayscale image of the video frame and the first area in the binary grayscale image of the previous video frame, union represents the union of the pixel points of the first area in the binary grayscale image of the video frame and the first area in the binary grayscale image of the previous video frame, and 1e-6 is a scientific notation method, which means 1 times 10 to the negative 6th power.

[0078] In step S304, if the overlap rate is greater than a preset threshold, the second server uses the text region in the previous video frame as the text region in the current video frame.

[0079] It should be noted that if the video frame is the first video frame in the video sequence, the second server performs region expansion on the binary grayscale image of the video frame to obtain the text region in the video frame.

[0080] In the disclosed embodiment, text recognition is performed using a DBNet (Differentiable Binarization Network) model to obtain a binary grayscale image, and then the binary grayscale image is region-expanded to obtain a text region in the video frame.

[0081] Among them, the DBNet model includes a feature pyramid, an adaptive threshold module and a binarization module. The feature pyramid is used to extract features from video frames to obtain image features of video frames. The adaptive threshold module is used to perform adaptive threshold prediction on image features to obtain a threshold feature map, and the threshold feature map includes probability thresholds corresponding to multiple pixel points. The binarization module is used to predict the text probability map corresponding to the image feature. Then, a binary grayscale image is obtained based on the text probability map and the threshold feature map. It should be noted that when determining the binary grayscale image, the probability threshold of multiple pixel points can be a preset fixed value, and multiple pixel points can also correspond to a probability threshold respectively. The probability thresholds corresponding to the multiple pixel points are obtained based on the threshold feature map, and then the binary grayscale image is obtained based on the text probability map and the threshold feature map. For example, see Figure 4 , Figure 4 The present invention is a schematic diagram of identifying a text area according to an exemplary embodiment, and the text area is identified by a DBNet model. A video frame is input into the DBNet model, a binary grayscale image is output by the DBNet model, and finally the DBNet model obtains the text area based on the binary grayscale image.

[0082] Among them, the second server performs regional expansion through a binarization module. In one implementation, the second server obtains the expansion ratio corresponding to the region size from the corresponding relationship between the region size and the expansion ratio based on the region size of the first region. Then, based on the expansion ratio, the expansion range of the central region in the binary grayscale image is determined with the central region as a reference, and the region corresponding to the expansion range is used as the text region, so as to improve the accuracy and effectiveness of regional expansion. In another implementation, the contour of the central region is extracted through cv2 (a computer vision library) to obtain a text region, thereby improving the efficiency of obtaining a text region.

[0083] In step S305, if the overlap rate is less than or equal to a preset threshold, the second server performs region expansion on the first region in the binary grayscale image of the video frame to obtain a text region in the video frame.

[0084] In the embodiment of the present disclosure, the process of the second server expanding the first region is the same as the process of region expansion in step S305, which is not described in detail here. In the embodiment of the present disclosure, if the overlap rate is less than the preset threshold, it means that the video frame has a high probability of being different from the text region in the previous video frame, and then the first region is expanded to obtain the text region, which improves the efficiency of obtaining the text region on the basis of improving the rationality of obtaining the text region.

[0085] In some embodiments, the second server implements the above steps S302-S305 through a text timing stability enhancement module, and the text timing stability enhancement module implements the above steps S302-S305 through the following process.

[0086] For any two adjacent video frames x and video frames y in a video sequence, the text timing stability enhancement module determines the process of the overlap rate between the first area in the binary grayscale image of video frame x and the first area in the binary grayscale image of video frame y, including the following steps: binarize video frame x and video frame y respectively to obtain the binary grayscale images of video frame x and video frame y. Then, by performing a logical "and" operation, the intersection inter of the pixel points in the first area of ​​video frame x and video frame y is obtained, and by performing a logical "or" operation, the union union of the pixel points in the first area of ​​video frame x and video frame y is obtained. Based on the intersection inter and the union union, the overlap rate between the first area in the binary grayscale image of video frame x and the first area in the binary grayscale image of video frame y is obtained by the above formula (1). The process of determining the text area in the video frame by the text timing stability enhancement module includes the following steps: first, the video sequence is binarized into a grayscale image and the text area is initialized, that is, starting from the first video frame F0, the first video frame F0 is input into the DBNet model to obtain the binarized grayscale image P0 of the first video frame F0, and then the binarized grayscale image P0 is post-processed to obtain the text area in the first video frame F0. If the video sequence includes N video frames, N is an integer greater than 1, then for the second video frame to the Nth video frame, the following steps are performed in sequence to determine the text area in the video frame: for video frame F i , input it into the DBNet model, and obtain its binary grayscale image P i , determine the binary grayscale image P i The first region in the previous video frame F i-1 Binarized grayscale image P i-1If the overlap rate is greater than a preset threshold value of 0.9, the previous video frame F i-1 Text area in i-1 As the video frame F i Text area in i ; Otherwise, the binary grayscale image P i Perform post-processing operations to obtain the video frame F i Text area in i .

[0087] In this embodiment, the binary grayscale image of each video frame is used to determine whether the text area has changed relative to the previous video frame. If there is no change, the text timing stability enhancement module uses the text area in the previous video frame as the recognition result of the text area in the video frame for output, otherwise the text area in the video frame is calculated for output. This module is simple and effective, and can greatly improve the timing stability of text recognition. Among them, the text recognition algorithm is not limited to the DBNet model, and can also be replaced by other models or algorithms with text recognition functions.

[0088] In step S306, the second server generates a text mask of the video frame based on the text area in the video frame, and the text mask is used to render the bullet screen for the target video so that the bullet screen is displayed in an area outside the text area.

[0089] In the disclosed embodiment, the process of the second server generating a text mask of a video frame based on a text area in a video frame comprises the following steps: the second server uses the text area as a non-filled area and the area outside the text area as a filled area to generate a text mask, and the filled area is an area where the bullet screen is allowed to be displayed. In the disclosed embodiment, the text area is used as a non-filled area and the area outside the text area is used as an area where the bullet screen is allowed to be displayed to generate a text mask, so that when the bullet screen is rendered based on the text mask, the bullet screen will only be displayed in the area outside the text area, and will not be displayed in the text area, effectively achieving the bullet screen anti-blocking effect.

[0090] It should be noted that when performing text recognition based on the DBNet model, the input is a video sequence. Although the position of the text area in each video frame in the video sequence may remain basically unchanged, the text area recognized by the DBNet model will jitter in each frame, which is unacceptable at the business level. The jittering text area will make the barrage appear and disappear, which greatly affects the viewing experience. Therefore, it is necessary to optimize the stability of text recognition. Through observation and experiments, it is known that although the text area will jitter, the binary grayscale image output in the middle of the DBNet model does not jitter much. Therefore, in the disclosed embodiment, determining whether the text area in the video frame has changed based on the binary grayscale image can effectively improve the timing stability during text recognition, effectively ensure that the video frames with the same text area can recognize the same text area, reduce the jitter of the text area, and thus improve the anti-occlusion effect of the barrage.

[0091] The disclosed embodiment provides a method for preventing barrage from being occluded, which performs text recognition on multiple video frames respectively to obtain binary grayscale images of the multiple video frames, wherein the binary grayscale images indicate the central area of ​​the text area in the video frame; and for each video frame, when the overlap rate between the central area of ​​the text area in the video frame and the central area of ​​the text area in the previous video frame is greater than a preset threshold, it indicates that the video frame has a high probability of being the same as the text area in the previous video frame, so that the text area in the previous video frame is used as the text area in the video frame, which can improve the stability of text recognition and avoid jittering of the recognized text area, and then generate a text mask based on the stable text area, and render the video based on the text mask, which can ensure the stability of the barrage display, avoid the intermittent appearance of the barrage, and improve the anti-occlusion effect of the barrage.

[0092] Above Figure 3 The second server is used as the execution subject for explanation. Figure 5 , taking the interaction between the terminal, the first server and the second server as an example. Figure 5 , Figure 5 It is a flowchart of a bullet screen anti-blocking method according to an exemplary embodiment. The method includes the following steps.

[0093] In step S501, the first server obtains a video and determines whether the video is a target video that needs to be protected from barrage blocking.

[0094] In some embodiments, the first server is a background server of a terminal, a target application for playing videos is installed on the terminal, the terminal is used to play videos, and the first server is used to provide background services for the target application. The second server is a server for processing videos, and the first server processes videos with the help of the second server. For example, the second server can be used to transcode videos, recognize text on videos, and generate text masks, etc.

[0095] In some embodiments, the first server acquires videos in real time, and the first server determines whether the video is the target video each time it acquires a video. In other embodiments, the first server acquires multiple videos at one time, and for each of the multiple videos, step S501 is performed to determine whether the video is the target video.

[0096] In the embodiment of the present disclosure, the video obtained by the first server may be from a video that has been played by the target application, or may be from a video that has been uploaded by the target application but has not been played, which is not specifically limited here.

[0097] In some embodiments, the first server determines whether the video is a target video based on at least one of a preset number of bullet screens, a preset interaction rate, a preset viewing rate, and a text display area. The video may be a video that has been played or a video that has not been played. In the case where the video is a played video, the first server determines whether the video is a target video based on at least one of a preset number of bullet screens, a preset interaction rate, and a preset viewing rate. That is, if the number of bullet screens, the interaction rate, and the viewing rate of the video meet at least one of the preset number of bullet screens, the preset interaction rate, and the preset viewing rate, the first server determines that the video is a target video. Alternatively, if the number of bullet screens, the interaction rate, and the viewing rate of the video meet at least two of the preset number of bullet screens, the preset interaction rate, and the preset viewing rate, the first server determines that the video is a target video. Alternatively, if the number of bullet screens, the interaction rate, and the viewing rate of the video meet the preset number of bullet screens, the preset interaction rate, and the preset viewing rate, respectively, the first server determines that the video is a target video. Optionally, the target video can also be determined based on the text display area, that is, only when the text of the video is displayed in the bullet screen area, the first server determines the target video based on the preset bullet screen number, preset interaction rate and preset viewing rate. If the text is displayed in the non-bullet screen area, the first server directly determines that the video is not the target video. In the case that the video is a video that has not been played, if the text of the video is displayed in the bullet screen area, the first server determines that the video is the target video.

[0098] In step S502, when the video is a target video, the first server sends the target video to the second server.

[0099] In some embodiments, there are multiple target videos, and the first server may send one target video to the second server each time it determines a target video; the first server may also determine a target number of target videos and send the target number of target videos to the second server at the same time, which is not specifically limited here.

[0100] In step S503, the second server receives the target video, transcodes the target video to obtain a plurality of videos at different levels, and transcodes the target video based on the target frame rate and the target resolution to obtain a video sequence of the target video.

[0101] In the embodiment of the present disclosure, at least one of the resolution, frame rate and bit rate of the multiple videos at different gears is different. The second server transcodes the target video based on the preset multiple resolutions, multiple frame rates and multiple bit rates, and the multiple resolutions, multiple frame rates and multiple bit rates are cross-combined to obtain multiple videos at different gears.

[0102] Optionally, if the multiple videos include a video with a target frame rate and a target resolution, there is no need to perform transcoding based on the target frame rate and the target resolution, and a video sequence of the target video can be directly obtained based on the video.

[0103] In step S504, the second server performs text recognition on a plurality of video frames in the video sequence of the target video to obtain respective text regions in the plurality of video frames.

[0104] In the embodiment of the present disclosure, step S504 is the same as step S302 - S305 and will not be described in detail here.

[0105] In step S505, the second server generates a text mask of each video frame based on the text area in the video frame.

[0106] In the embodiment of the present disclosure, step S505 is similar to step S306 and will not be described in detail here.

[0107] In some embodiments, after the second server transcodes the target video to obtain multiple videos and obtains the text mask of the target video, it stores them in the second server, and only sends the video and text mask in response to the terminal's playback request. The execution process is shown in the following steps S506-S509.

[0108] In step S506, the terminal sends a play request of the target video to the first server, where the play request carries the video identifier of the target video and the configuration information of the terminal.

[0109] In the embodiment of the present disclosure, the configuration information includes the resolution, bit rate and frame rate of the video playback supported by the terminal, and the configuration information also includes the terminal identification of the terminal, so as to facilitate returning the video and text mask for the terminal.

[0110] In step S507, the first server receives the play request and forwards the play request to the second server.

[0111] In the disclosed embodiment, the forwarded playback request also carries the server identifier of the first server, thereby facilitating the return of the video and text mask to the first server.

[0112] In step S508, the second server receives the play request, and based on the play request, sends the target gear video and the text masks of the target video frames among the multiple videos obtained by transcoding to the first server.

[0113] In the embodiment of the present disclosure, multiple videos obtained based on the transcoding of the target video and text masks of multiple video frames of the target video are stored correspondingly, and the multiple videos share the text masks of the multiple video frames. The second server searches for the video of the target gear and the text masks of the multiple video frames based on the video identifier and configuration information carried in the playback request.

[0114] In step S509, the first server forwards the video of the target gear and the text masks of the multiple video frames to the terminal.

[0115] In the disclosed embodiment, the above steps S506-S509 are described by taking the example of storing multiple videos and text masks of different gears on the second server. In other embodiments, multiple videos and text masks of different gears are stored on the first server. Accordingly, after executing step S505, the second server sends the multiple videos of different gears and the text masks of the multiple video frames to the first server, and the first server receives and stores the multiple videos and the text masks of the multiple video frames. In response to the playback request of the terminal, the first server sends the video of the target gear and the text masks of the multiple video frames to the terminal.

[0116] In step S510, the terminal receives the target gear video and the text masks of the multiple video frames, plays the target gear video, and performs bullet screen rendering on the target gear video based on the text masks of the multiple video frames so that the bullet screen is displayed in an area outside the text area.

[0117] In the disclosed embodiment, the terminal renders bullet screens for the multiple video frames of the target gear video based on the text masks of the multiple video frames. It should be noted that since the multiple video frames in the video sequence are obtained based on the target frame rate, and the frame rate of the video of the target gear may be different from the target frame rate, the number of video frames of the target gear video does not match the number of multiple video frames in the video sequence. Optionally, for each video frame of the target gear video, the text mask of the video frame with the smallest difference between the broadcast time and the broadcast time of the video frame in the multiple video frames of the video sequence is used as the text mask of the video frame to render bullet screens for the video frame. In this embodiment, the bullet screen is rendered for the video frame based on the text mask of the video frame with the closest playback time, which ensures the accuracy of the determined text mask, and then ensures the accuracy of the bullet screen display; and avoids generating text masks for each gear video separately, improving the efficiency of generating text masks on the basis of ensuring accuracy, and saving resources.

[0118] See also Figure 6 , Figure 6 : This is a schematic diagram of a framework for bullet screen anti-blocking according to an exemplary embodiment. The framework includes a terminal, a first server, and a second server. The first server is used to receive an input video and send a target video in the video to the second server. The second server is an on-demand cloud, which also includes a transcoding module and a text recognition algorithm module. The on-demand cloud schedules the transcoding module to transcode the target video to obtain multiple videos of different gears, and transcodes the target video based on the target frame rate and target resolution to obtain a video sequence. The on-demand cloud schedules the text recognition algorithm module to perform text recognition on multiple video frames in the video sequence to obtain text masks of multiple video frames. Among them, the multiple videos obtained by the transcoding module and the text masks obtained by the text recognition algorithm module are all sent to the on-demand cloud for management. The on-demand cloud then sends multiple videos and text masks of different gears to the first server, which is sent by the first server to the terminal for video playback. Among them, the terminal plays the video through the player and renders the video with bullet screen based on the text mask.

[0119] The bullet screen anti-occlusion framework provided by the embodiment of the present disclosure is the first framework in the industry to apply text for anti-occlusion. The embodiment of the present disclosure proposes for the first time a solution for text bullet screen anti-occlusion, which is not currently available in the video field; and the stability of text recognition is not considered in the target video field, while the method provided by the embodiment of the present disclosure focuses on optimizing the stability of text recognition.

[0120] The disclosed embodiment provides a barrage anti-occlusion method, based on which barrage anti-occlusion is performed on text in a video, thereby improving the stability of text recognition, greatly alleviating the problem of text recognition jitter, and being able to bring a good video and barrage viewing experience to the audience, thereby being able to bring the audience an ultimate viewing experience.

[0121] The disclosed embodiment provides a method for preventing barrage from being occluded, which performs text recognition on multiple video frames respectively to obtain binary grayscale images of the multiple video frames, wherein the binary grayscale images indicate the central area of ​​the text area in the video frame; and for each video frame, when the overlap rate between the central area of ​​the text area in the video frame and the central area of ​​the text area in the previous video frame is greater than a preset threshold, it indicates that the video frame is highly likely to be the same as the text area in the previous video frame, so that the text area in the previous video frame is used as the text area in the video frame, which can improve the stability of text recognition and avoid jittering of the recognized text area, and then generate a text mask based on the stable text area, and render the video based on the text mask, which can ensure the stability of the barrage display and avoid the intermittent appearance of the barrage, thereby improving the anti-occlusion effect of the barrage.

[0122] Figure 7 is a block diagram of a bullet screen anti-blocking device according to an exemplary embodiment. Figure 7 , the device comprises:

[0123] The text recognition unit 701 is configured to perform text recognition on multiple video frames in the video sequence of the target video respectively, and obtain binary grayscale images of the multiple video frames respectively, wherein the binary grayscale image includes a first area and a second area, and the grayscale value corresponding to each pixel in the binary grayscale image is used to indicate whether the pixel belongs to the first area, the first area is the central area of ​​the text area in the video frame, and the second area is the area outside the first area;

[0124] The text region determining unit 702 is configured to, for each video frame, if the overlap rate between the first region in the binary grayscale image of the video frame and the first region in the binary grayscale image of the previous video frame of the video frame is greater than a preset threshold, determine the text region in the previous video frame as the text region in the video frame;

[0125] The mask generation unit 703 is configured to generate a text mask of the video frame based on the text area in the video frame, and the text mask is used to render the bullet screen for the target video so that the bullet screen is displayed in an area outside the text area.

[0126] In some embodiments, the text recognition unit 701 is configured to perform text recognition on multiple video frames respectively to obtain text probability maps for the multiple video frames, wherein the text probability map includes probability values ​​corresponding to multiple pixel points respectively, and the probability value represents the probability that the pixel point belongs to the first area; for the text probability map of each video frame, the pixel points whose probability values ​​are greater than the probability threshold are assigned a first grayscale value, and the pixel points whose probability values ​​are less than or equal to the probability threshold are assigned a second grayscale value to obtain a binary grayscale map, wherein the first grayscale value is used to indicate the first area, and the second grayscale value is used to indicate the second area.

[0127] In some embodiments, the device also includes an overlap rate determination unit, which is configured to determine the intersection and union of pixel points of a first area in a binary grayscale image of a video frame and a first area in a binary grayscale image of a previous video frame; based on the intersection and union, the overlap rate is determined, and the overlap rate is positively correlated with the intersection and negatively correlated with the union.

[0128] In some embodiments, the text region determination unit 702 is further configured to perform region expansion on the first region in the binary grayscale image of the video frame to obtain the text region in the video frame if the overlap rate is less than or equal to a preset threshold.

[0129] In some embodiments, the mask generation unit 703 is configured to use the text area as a non-filled area and the area outside the text area as a filled area to generate a text mask, and the filled area is an area where barrage is allowed to be displayed.

[0130] In some embodiments, the target video is a video with a greater number of barrages than a preset number of barrages; or, the target video is a video with an interaction rate greater than a preset interaction rate; or, the target video is a video with a viewing rate greater than a preset viewing rate; or, the target video is a video with text displayed in the barrage area.

[0131] The disclosed embodiment provides a device for preventing barrage from being blocked, which performs text recognition on a plurality of video frames respectively to obtain binary grayscale images of the plurality of video frames, wherein the binary grayscale images indicate the central area of ​​the text area in the video frame; and for each video frame, when the overlap rate between the central area of ​​the text area in the video frame and the central area of ​​the text area in the previous video frame is greater than a preset threshold, it indicates that the video frame is highly likely to be the same as the text area in the previous video frame, so that the text area in the previous video frame is used as the text area in the video frame, which can improve the stability of text recognition and avoid jittering of the recognized text area, and then generate a text mask based on the stable text area, and perform barrage rendering on the video based on the text mask, which can ensure the stability of barrage display and avoid the intermittent appearance of barrage, thereby improving the anti-blocking effect of barrage.

[0132] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0133] Figure 8 The structure block diagram of a terminal 800 provided by an exemplary embodiment of the present disclosure is shown. The terminal 800 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 800 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0134] Typically, the terminal 800 includes a processor 801 and a memory 802 .

[0135] The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0136] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one program code, which is used to be executed by the processor 801 to implement the bullet screen anti-blocking method provided in the method embodiment of the present disclosure.

[0137] In some embodiments, the terminal 800 may further optionally include: a peripheral device interface 803 and at least one peripheral device. The processor 801, the memory 802 and the peripheral device interface 803 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 803 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807 and a power supply 808.

[0138] The peripheral device interface 803 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 801 and the memory 802. In some embodiments, the processor 801, the memory 802, and the peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 801, the memory 802, and the peripheral device interface 803 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0139] The radio frequency circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 804 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 804 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 804 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0140] The display screen 805 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 805 is a touch display screen, the display screen 805 also has the ability to collect touch signals on the surface or above the surface of the display screen 805. The touch signal can be input to the processor 801 as a control signal for processing. At this time, the display screen 805 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 805 can be one, and the front panel of the terminal 800 is set; in other embodiments, the display screen 805 can be at least two, which are respectively set on different surfaces of the terminal 800 or are folded; in some other embodiments, the display screen 805 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 800. Even, the display screen 805 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 805 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode) and the like.

[0141] The camera assembly 806 is used to capture images or videos. Optionally, the camera assembly 806 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0142] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 801 for processing, or input them into the radio frequency circuit 804 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 800. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 807 may also include a headphone jack.

[0143] The power supply 808 is used to power various components in the terminal 800. The power supply 808 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 808 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0144] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the terminal 800, and the terminal 800 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0145] Fig. 9 It is a structural diagram of a server provided according to an embodiment of the present disclosure. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 901 and one or more memories 902, wherein the memory 902 is used to store executable program codes, and the processor 901 is configured to execute the above executable program codes to implement the bullet screen anti-blocking method provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be repeated here.

[0146] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by a processor of a terminal to complete the above-mentioned bullet screen anti-blocking method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0147] In an exemplary embodiment, a computer program product is also provided, including a computer program, which implements the above-mentioned bullet screen anti-blocking method when executed by a processor. In some embodiments, the computer program product involved in the embodiments of the present disclosure can be deployed on a computer device for execution, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network can constitute a blockchain system.

[0148] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0149] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A bullet screen anti-blocking method, characterized in that: The method comprises: Performing text recognition on multiple video frames in a video sequence of a target video respectively, and obtaining binary grayscale images of the multiple video frames, wherein the binary grayscale images include a first area and a second area, and the grayscale value corresponding to each pixel in the binary grayscale image is used to indicate whether the pixel belongs to the first area, the first area is a central area of ​​a text area in the video frame, and the second area is an area outside the first area; For each of the video frames, if the overlap rate between the first area in the binary grayscale image of the video frame and the first area in the binary grayscale image of the previous video frame of the video frame is greater than a preset threshold, the text area in the previous video frame is used as the text area in the video frame; if the overlap rate is less than or equal to the preset threshold, based on the area size of the first area in the binary grayscale image of the video frame, the expansion ratio corresponding to the area size is obtained from the corresponding relationship between the area size and the expansion ratio; based on the expansion ratio, the expansion range of the central area in the binary grayscale image is determined with the central area as a reference, and the area corresponding to the expansion range is used as the text area in the video frame; A text mask of the video frame is generated based on a text area in the video frame, and the text mask is used to render the barrage on the target video so that the barrage is displayed in an area outside the text area.

2. The barrage anti-blocking method according to claim 1, characterized in that: The step of performing text recognition on a plurality of video frames in a video sequence of a target video to obtain respective binary grayscale images of the plurality of video frames includes: Performing text recognition on the multiple video frames respectively to obtain text probability maps of the multiple video frames, wherein the text probability maps include probability values ​​corresponding to multiple pixel points respectively, and the probability values ​​represent probabilities that the pixel points belong to the first area; For the text probability map of each video frame, the pixel points whose probability values ​​are greater than the probability threshold are assigned a first grayscale value, and the pixel points whose probability values ​​are less than or equal to the probability threshold are assigned a second grayscale value to obtain the binary grayscale map, wherein the first grayscale value is used to indicate the first area, and the second grayscale value is used to indicate the second area.

3. The barrage anti-blocking method according to claim 1, characterized in that: The process of determining the overlap rate includes: Determine the intersection and union of the pixel points of the first area in the binary grayscale image of the video frame and the first area in the binary grayscale image of the previous video frame; Based on the intersection and the union, the overlap ratio is determined, and the overlap ratio is positively correlated with the intersection and negatively correlated with the union.

4. The barrage anti-blocking method according to claim 1, characterized in that: The step of generating a text mask of the video frame based on the text area in the video frame includes: The text area is used as a non-filled area, and the area outside the text area is used as a filled area to generate the text mask, and the filled area is an area where the bullet screen is allowed to be displayed.

5. The barrage anti-blocking method according to claim 1, characterized in that: The target video is a video with a number of bullet screens greater than a preset number of bullet screens; or, The target video is a video whose interaction rate is greater than a preset interaction rate; or, The target video is a video with a viewing rate greater than a preset viewing rate; or, The target video is a video in which text is displayed in the bullet screen area.

6. A barrage anti-blocking device, characterized in that: The device comprises: A text recognition unit is configured to perform text recognition on a plurality of video frames in a video sequence of a target video, respectively, to obtain a binary grayscale image of each of the plurality of video frames, wherein the binary grayscale image includes a first region and a second region, and a grayscale value corresponding to each pixel in the binary grayscale image is used to indicate whether the pixel belongs to the first region, the first region is a central region of a text region in the video frame, and the second region is a region outside the first region; The text area determination unit is configured to, for each of the video frames, if the overlap rate between the first area in the binary grayscale image of the video frame and the first area in the binary grayscale image of the previous video frame of the video frame is greater than a preset threshold, use the text area in the previous video frame as the text area in the video frame; if the overlap rate is less than or equal to the preset threshold, based on the area size of the first area in the binary grayscale image of the video frame, obtain the expansion ratio corresponding to the area size from the corresponding relationship between the area size and the expansion ratio; based on the expansion ratio, with the central area as a reference, determine the expansion range of the central area in the binary grayscale image, and use the area corresponding to the expansion range as the text area in the video frame; The mask generation unit is configured to generate a text mask of the video frame based on a text area in the video frame, and the text mask is used to render the barrage on the target video so that the barrage is displayed in an area outside the text area.

7. The anti-blocking device for barrage according to claim 6, characterized in that: The character recognition unit is configured as follows: Performing text recognition on the multiple video frames respectively to obtain text probability maps of the multiple video frames, wherein the text probability maps include probability values ​​corresponding to multiple pixel points respectively, and the probability values ​​represent probabilities that the pixel points belong to the first area; For the text probability map of each video frame, the pixel points whose probability values ​​are greater than the probability threshold are assigned a first grayscale value, and the pixel points whose probability values ​​are less than or equal to the probability threshold are assigned a second grayscale value to obtain the binary grayscale map, wherein the first grayscale value is used to indicate the first area, and the second grayscale value is used to indicate the second area.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the instructions to implement the barrage anti-occlusion method as described in any one of claims 1 to 5.

9. A computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the barrage anti-obstruction method described in any one of claims 1 to 5.

Images