Danmaku Display Method and Device

Abstract

The present application discloses a method for displaying bullet screens. The method includes: during the process of video playback, performing target object detection processing on the original frame image of the video at a first preset frequency, and when the target object is detected, outputting the display area of the target object in the original frame image; generating a mask picture corresponding to the original frame image according to the display area; obtaining the bullet screen display area corresponding to the original frame image; if there is an overlapping area between the bullet screen display area and the display area of the target object in the original frame image, then synthesizing the mask picture and the bullet screens displayed in the bullet screen display area into a video bullet screen prevention and blocking screen, and displaying the video bullet screen prevention and blocking screen on the bullet screen layer, wherein the bullet screens included in the video bullet screen prevention and blocking screen do not block the target object. The present application can achieve the bullet screen prevention and blocking effect.

Description

Technical Field

[0001] This application relates to the field of video technology, and in particular, to a method and device for displaying bullet screens. Background Art

[0002] When playing a video through a client or website, if the bullet screen function is turned on, the bullet screen content will be played while the video picture is being played. However, during the display of the bullet screen, the bullet screen content on the screen often obscures key content (such as a portrait) in the video picture, resulting in a poor user experience. Summary of the Invention

[0003] In view of this, a method and device for displaying bullet screens, a computer device, and a computer-readable storage medium are provided to solve the problem that the bullet screen content on the screen often obscures key content in the video picture in the prior art.

[0004] This application provides a method for displaying bullet screens, including:

[0005] During the video playback process, the original frame image of the video is subjected to target object detection processing at a first preset frequency, and when the target object is detected, the display area of the target object in the original frame image is output;

[0006] A mask picture corresponding to the original frame image is generated according to the display area;

[0007] The bullet screen display area corresponding to the original frame image is obtained;

[0008] If there is an overlapping area between the bullet screen display area and the display area of the target object in the original frame image, the mask picture is combined with the bullet screens displayed in the bullet screen display area to form a video bullet screen prevention picture, and the video bullet screen prevention picture is displayed on the bullet screen layer, where the bullet screens included in the video bullet screen prevention picture do not obscure the target object.

[0009] Optionally, during the video playback process, the step of subjecting the original frame image of the video to target object detection processing at a first preset frequency and outputting the display area of the target object in the original frame image when the target object is detected includes:

[0010] During the video playback process, it is detected in real time whether the number of bullet screens displayed on the current screen exceeds a preset number;

[0011] If the number of bullet screens displayed on the current screen exceeds the preset number, the original frame image of the video is subjected to target object detection processing at a first preset frequency, and when the target object is detected, the display area of the target object in the original frame image is output.

[0012] Optionally, the method of performing target object detection processing on the original frame image of the video at the first preset frequency and outputting the display area of the target object in the original frame image when the target object is detected includes:

[0013] Obtaining the original frame image of the video at the first preset frequency;

[0014] Performing resolution sampling processing on the obtained original frame image to obtain a frame image with a target resolution;

[0015] Performing target object detection processing on the frame image with the target resolution and outputting when the target object is detected

[0016] Optionally, during the video playback, the method of performing target object detection processing on the original frame image of the video at the first preset frequency and outputting the display area of the target object in the original frame image when the target object is detected includes:

[0017] During the video playback, performing target object detection processing on the original frame image of the video at the second preset frequency;

[0018] After detecting that the original frame image contains the target object, performing target object detection processing on the original frame image of the video at the first preset frequency and outputting the display area of the target object in the original frame image when the target object is detected.

[0019] Optionally, the method further includes:

[0020] Providing a frequency setting interface for the user to set the first preset frequency according to the frequency setting interface.

[0021] Optionally, during the video playback, the method of performing target object detection processing on the original frame image of the video at the first preset frequency and outputting the display area of the target object in the original frame image when the target object is detected includes:

[0022] During the video playback, performing target object detection processing on the original frame image of the video at the first preset frequency through the first thread and outputting the display area of the target object in the original frame image when the target object is detected;

[0023] The method of generating a mask picture corresponding to the original frame image according to the display area includes:

[0024] Generating a mask picture corresponding to the original frame image according to the display area through the second thread.

[0025] Optionally, perform target object detection processing on the original frame image of the video, and when the target object is detected, output the display area of the target object in the original frame image, including:

[0026] Perform target object detection processing on the original frame image of the video through a target object detection model obtained based on a machine learning algorithm, and when the target object is detected, output the display area of the target object in the original frame image.

[0027] Optionally, during the video playback process, perform target object detection processing on the original frame image of the video at a first preset frequency, and when the target object is detected, output the display area of the target object in the original frame image, including:

[0028] During the video playback process, sample the original frame image of the video at a first preset frequency, and upload the sampled original frame image to the server for the server to perform target object detection processing on the sampled original frame image. When the target object is detected, the server outputs the display area of the target object in the original frame image;

[0029] Receive the display area of the target object in the original frame image output by the server.

[0030] This application also provides a bullet screen display device, including:

[0031] An output module for performing target object detection processing on the original frame image of the video at a first preset frequency during the video playback process, and when the target object is detected, outputting the display area of the target object in the original frame image;

[0032] A generation module for generating a mask picture corresponding to the original frame image according to the display area;

[0033] An acquisition module for acquiring the bullet screen display area corresponding to the original frame image;

[0034] A display module for, if there is an overlapping area between the bullet screen display area and the display area of the target object in the original frame image, synthesizing the mask picture with the bullet screens displayed in the bullet screen display area into a video anti-blocking bullet screen picture, and displaying the video anti-blocking bullet screen picture in the bullet screen layer, where the bullet screens included in the video anti-blocking bullet screen picture do not block the target object.

[0035] The present application also provides a bullet screen display system, which includes a host terminal, multiple clients installed on viewer terminals, a video server, and a bullet screen server. The viewer terminal is the terminal device of the viewer user who watches the live broadcast by the host through the host terminal in the host's live broadcast room, where:

[0036] The host terminal is used to upload the live video of the host to the video server;

[0037] The video server is used to forward the live video to the viewer terminals of the viewer users who watch the live video in the host's live broadcast room;

[0038] The bullet screen server is used to store the bullet screens sent by viewer users and forward the stored bullet screens to each client for each client to display;

[0039] Each client is used to, during the playback of the live video, perform target object detection processing on the original frame image of the live video at a first preset frequency, and when the target object is detected, output the display area of the target object in the original frame image; generate a mask picture corresponding to the original frame image according to the display area; obtain the bullet screen display area corresponding to the original frame image; if there is an overlapping area between the bullet screen display area and the display area of the target object in the original frame image, then synthesize the mask picture and the bullet screens displayed in the bullet screen display area into a video anti-blocking bullet screen picture and display the video anti-blocking bullet screen picture in the bullet screen layer, where the bullet screens included in the video anti-blocking bullet screen picture do not block the target object.

[0040] The present application also provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the above method are implemented.

[0041] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0042] In this embodiment, during the video playback process, the original frame images of the video are subjected to target object detection processing at a first preset frequency. When the target object is detected, the display area of the target object in the original frame image is output. A mask picture corresponding to the original frame image is generated according to the display area. The bullet screen display area corresponding to the original frame image is obtained. If there is an overlapping area between the bullet screen display area and the display area of the target object in the original frame image, the mask picture and the bullet screens displayed in the bullet screen display area are synthesized into a video anti-blocking bullet screen picture, and the video anti-blocking bullet screen picture is displayed in the bullet screen layer, where the bullet screens included in the video anti-blocking bullet screen picture do not block the target object. In this application, by displaying the video anti-blocking bullet screen picture during the bullet screen display process, the bullet screens can be prevented from blocking the target object in the video, improving the user experience. At the same time, by detecting and processing the original frame images of the video in the manner of the first preset frequency, it is not necessary to process all the original video frames, which can reduce the occupation of computing resources of the terminal device. In addition, in this application, by implementing real-time calculation in the client to obtain the mask picture, compared with the method of obtaining the mask picture by real-time calculation on the anchor device in the prior art, the consumption of network resources for uploading the mask picture to the video server can be saved. That is to say, the implementation method of this application can save the computing cost, storage cost, and bandwidth cost of the video service provider. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is an environmental schematic diagram of an embodiment of the bullet screen display method according to an embodiment of this application;

[0044] Figure 2 It is an environmental schematic diagram of another embodiment of the bullet screen display method according to an embodiment of this application;

[0045] Figure 3 It is a flowchart of an embodiment of the bullet screen display method described in this application;

[0046] Figure 4 It is a detailed flowchart of the step of, during the video playback process in an embodiment of this application, detecting and processing the original frame images of the video at the first preset frequency and outputting the display area of the target object in the original frame image when the target object is detected;

[0047] Figure 5 It is a detailed flowchart of the step of detecting and processing the original frame images of the video at the first preset frequency and outputting the display area of the target object in the original frame image when the target object is detected in an embodiment of this application;

[0048] Figure 6In an embodiment of the present application, during the video playback process, the original frame image of the video is subjected to target object detection processing at a first preset frequency, and when the target object is detected, the display area of the target object in the original frame image is output; the detailed flowchart of the steps

[0049] Figure 7 The program module diagram of an embodiment of the barrage display device described in the present application;

[0050] Figure 8 The flowchart of the barrage display system implementing the barrage display method;

[0051] Figure 9 The schematic diagram of the mask image in an embodiment of the present application;

[0052] Figure 10 The schematic diagram of the hardware structure of the computer device for executing the barrage display method provided by the embodiment of the present application. Detailed implementation manners

[0053] The advantages of the present application are further elaborated below in conjunction with the accompanying drawings and specific embodiments.

[0054] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of the devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0055] The terms used in the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms "a", "the" and "said" used in the present disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0056] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0057] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order of execution of the steps, but are only used to facilitate the description of the present application and to distinguish each step. Therefore, it should not be construed as a limitation to the present application.

[0058] Figure 1 Fig. shows a schematic diagram of a live application scenario provided by an embodiment of the present application. The application scenario includes: a host 10, a live video server 11, a client 12, an audience 13, and a bullet screen server 14. Among them, the live video server 11, the client 12, and the bullet screen server 14 are located in a wireless network or a wired network. The host 10 sends the live video to the live video server 11 through the host terminal. The audience 13 sends bullet screens through the audience terminal and uploads the sent bullet screens to the bullet screen server. Other audiences can obtain the live video from the live video server 11 and the bullet screens from the bullet screen server 14 through the client or web page installed on their audience terminals, so as to realize the viewing of the live video and the display of the bullet screens.

[0059] In the embodiment of the present application, the live video server 11 and the bullet screen server 14 can be separate servers or a server cluster, and can be local servers or cloud servers. The specific server type may not be limited in the embodiment of the present application. The host terminal and the audience terminal can be a mobile phone, a smart phone, a laptop computer, a desktop computer, a tablet computer, a personal digital assistant (PDA), a media player, a smart TV, a wearable electronic device, etc. The specific type of the terminal device may not be limited in the embodiment of the present application.

[0060] The client 12 can be a computer application (APP) or a Web client, and the Web client can refer to an application developed based on the Web architecture.

[0061] Figure 2 Fig. shows a schematic diagram of an on-demand application scenario provided by an embodiment of the present application. The application scenario includes: a creator 20, an on-demand video server 21, a client 22, an audience 23, and a bullet screen server 24. Among them, the on-demand video server 21, the client 22, and the bullet screen server 24 are located in a wireless network or a wired network. The creator 20 uploads the created video to the on-demand video server 21 through the terminal device. The audience 23 sends bullet screens through the audience terminal and uploads the sent bullet screens to the bullet screen server. Other audiences can obtain the video from the on-demand video server 21 and the bullet screens from the bullet screen server 24 through the client or web page installed on their audience terminals, so as to realize the viewing of the video and the display of the bullet screens.

[0062] In the embodiments of the present application, the on-demand video server 21 and the bullet screen server 24 may be separate servers or server clusters, and may be local servers or cloud servers. The specific server type may not be limited in the embodiments of the present application. The host terminal and the viewer terminal may be mobile phones, smart phones, laptop computers, desktop computers, tablet computers, personal digital assistants (PDAs), media players, smart TVs, wearable electronic devices, etc. The specific type of terminal device may not be limited in the embodiments of the present application.

[0063] In this embodiment, the client 22 may be a computer application (APP) or a Web client, and the Web client may refer to an application developed based on the Web architecture.

[0064] Refer to Figure 3 , which is a schematic flowchart of the bullet screen display method according to an embodiment of the present application. The bullet screen display method of the present application is applied to the client, and the client may be a live video client or a non-live video client, which is not limited in this embodiment. It can be understood that the flowchart in the method embodiment is not used to limit the order of execution steps. As can be seen from the figure, the bullet screen display method provided in this embodiment includes:

[0065] Step S31: During the video playback, perform target object detection processing on the original frame image of the video at a first preset frequency, and when the target object is detected, output the display area of the target object in the original frame image.

[0066] Specifically, the first preset frequency is the time frequency for sampling the original video frames in the video. For example, if the first preset video is 15 frames / s, it means that 15 original frame images in the video need to be subjected to target object detection processing in 1 second.

[0067] In this embodiment, the video may be a live video or an ordinary on-demand video, and the on-demand video may be all types of videos other than the live video. The original video frame is the video frame in the video that has not been preprocessed.

[0068] In an exemplary embodiment, during the process of performing object detection processing on the original frame image of the video, the original frame image can be subjected to object detection processing through an object detection model obtained based on a machine learning algorithm. For example, the original frame image can be input into the object detection model obtained based on the machine learning algorithm, and the object detection model can detect whether the original frame image includes the target object. After detecting the target object, the object detection model outputs the display area of the target object in the original frame image.

[0069] Among them, the target object model is a model obtained by training a data set using a machine learning algorithm. Through this object detection model, the display area of the target object can be output from the original frame image. In one embodiment, the area where the pixels of the detected target object are located can be used as the display area. In another embodiment, the area where the detection frame for framing the target object is located can also be used as the display area.

[0070] In this embodiment, the target object can include various types of objects such as people, animals, and plants. The target object can include only one object or multiple objects. The specific object types and the number of objects included can be determined according to the actual situation and are not limited in this embodiment. For example, if the video is a movie, each person in the movie can be used as the target object. Another example is that if the video is a game live broadcast, each game character in the game can be used as the target object.

[0071] In one embodiment, when playing a live video through a client, object detection processing can be performed on the original video frames of the acquired live video in real time at a first preset frequency, and when the target object is detected, the display area of the target object in the original video frame image is output, so that subsequent steps can be used to perform anti-occlusion processing on the original video frames where the target object is detected, so that the barrage does not occlude the target object in the original video frame.

[0072] In another embodiment, when playing an on-demand video through a client, object detection processing can be performed on the original video frames of the acquired live video in real time at a first preset frequency, and when the target object is detected, the display area of the target object in the original video frame image is output, so that subsequent steps can be used to perform anti-occlusion processing on the original video frames where the target object is detected, so that the barrage does not occlude the target object in the original video frame.

[0073] In an exemplary embodiment, in order to enable the bullet screen display method in the present application to be implemented even when the performance of the terminal device is low, during the video playback, the original frame image of the video is subjected to target object detection processing at a first preset frequency, and when the target object is detected, the display area of the target object in the original frame image is output, including: Step A, during the video playback, the original frame image of the video is sampled at the first preset frequency, and the sampled original frame image is uploaded to the server for the server to perform target object detection processing on the sampled original frame image, and when the target object is detected, the server outputs the display area of the target object in the original frame image; Step B, receive the display area of the target object in the original frame image output by the server.

[0074] Specifically, in the present application, after the original frame image is sampled on the client side, the sampled original frame image is uploaded to the server, and the server performs target object detection processing on the sampled original frame image. When the target object is detected, the server will output the display area of the target object in the original frame image. In this way, the detection processing is implemented through the server, so that the technical solution of the present application can be applied to more low-performance devices and the application scope of the present application is improved.

[0075] In an exemplary embodiment, referring to Figure 4 , during the video playback, the method of performing target object detection processing on the original frame image of the video at the first preset frequency and outputting the display area of the target object in the original frame image when the target object is detected includes: Step S40, during the video playback, real-time detect whether the number of bullet screens displayed on the current screen exceeds a preset number; Step S41, if the number of bullet screens displayed on the current screen exceeds the preset number, then perform target object detection processing on the original frame image of the video at the first preset frequency, and when the target object is detected, output the display area of the target object in the original frame image.

[0076] Specifically, during the video playback, to avoid wasting computing resources by performing target object detection processing on the original frame image of the video even when there are no bullet screens. In this embodiment, during the video playback, it will be detected in real time whether the number of bullet screens displayed on the current screen exceeds a preset number. When it is detected that the number of bullet screens displayed on the current screen exceeds the preset number, the original frame image of the video will be subjected to target object detection processing at a first preset frequency, and when the target object is detected, the display area of the target object in the original frame image will be output. When it is detected that the number of bullet screens displayed on the current screen does not exceed the preset number, the original frame image of the video will not be subjected to target object detection processing at the first preset frequency, and when the target object is detected, the display area of the target object in the original frame image will be output.

[0077] Wherein, the preset number is a preset quantity, which can be set and adjusted according to the actual situation. For example, if the preset number is 2, it means that only when the number of bullet screens displayed on the screen exceeds 2, the original frame image of the video will be subjected to target object detection processing at the first preset frequency, and when the target object is detected, the display area of the target object in the original frame image will be output.

[0078] In an exemplary embodiment, refer to Figure 5 , the performing target object detection processing on the original frame image of the video at the first preset frequency and outputting the display area of the target object in the original frame image when the target object is detected includes:

[0079] Step S50, obtaining the original frame image of the video at the first preset frequency;

[0080] As an example, if the frame rate of the video is 60 and the first preset frequency is 15, then during the video playback, an original frame image can be obtained from the video every 66.67 ms, that is, an original frame image is obtained from the video every 4 frames, that is, the first original frame image, the fifth original frame image, the ninth original frame image,... of the video will be obtained in sequence, where n is an integer greater than or equal to 1.

[0081] Step S51, performing resolution sampling processing on the obtained original frame image to obtain a frame image with a target resolution;

[0082] Specifically, the resolution sampling process is a processing operation for reducing the resolution of the original frame picture. By performing resolution sampling on the original frame image, a frame image with a lower resolution can be obtained. The target resolution is a preset resolution, which can be set and adjusted according to the actual situation. For example, the resolution of the original frame picture is 1080P, and the target resolution is 480P.

[0083] Step S52: Perform target object detection on the frame image with the target resolution, and when the target object is detected, output the display area of the target object in the original frame image.

[0084] In this embodiment, by performing resolution sampling on the original frame picture and performing target detection on the frame image after resolution sampling, the computational complexity in the target detection process can be reduced, and computing resources can be saved.

[0085] In an exemplary embodiment, refer to Figure 6 , during the video playback process, performing target object detection on the original frame image of the video at a first preset frequency, and when the target object is detected, outputting the display area of the target object in the original frame image includes: Step S60: During the video playback process, performing target object detection on the original frame image of the video at a second preset frequency; Step S61: After detecting that the original frame image contains the target object, performing target object detection on the original frame image of the video at the first preset frequency, and when the target object is detected, outputting the display area of the target object in the original frame image.

[0086] Specifically, the second preset frequency is preset and can be set and modified according to the actual situation. For example, the second preset frequency is 10 frames / s, that is, 10 original frame images in the video are obtained per second, and then target image detection is performed on the obtained original frame images. In this embodiment, once it is detected that the original frame image contains the target object, the sampling frequency of the original video frame will be immediately adjusted, so that the second preset frequency is adjusted to the first preset frequency.

[0087] It should be noted that the second preset frequency is less than the first preset frequency.

[0088] In this embodiment, by performing target object detection on the original frame picture at a lower frequency when the target object does not exist in the original frame picture, and performing target object detection on the original frame picture at a higher frequency after the target object is detected, the consumption of computing resources can be reduced on the premise of achieving the anti-blocking bullet screen effect.

[0089] Step S31: Generate a mask picture corresponding to the original frame image according to the display area.

[0090] Specifically, the mask picture is a picture in png format, which contains the contour information of the display area. As Figure 9 shown, the contour area of this mask picture is fully transparent, and other areas are pure black.

[0091] Step S32: Obtain the bullet screen display area corresponding to the original frame image.

[0092] Specifically, the bullet screen display area is the display position of each bullet screen. In this embodiment, since the bullet screens displayed on the screen usually move from right to left, that is to say, each original frame picture in the video has a corresponding bullet screen display area.

[0093] It can be understood that in this embodiment, whenever it is detected that the current original frame picture contains the target object, the display area of the target object in the current original frame picture will be output, and the following steps will be executed: Step 1: Generate a mask picture corresponding to the original frame image according to the display area; Step 2: Obtain the bullet screen display area corresponding to the original frame image; Step 3: If there is an overlapping area between the bullet screen display area and the display area of the target object in the original frame image, then synthesize the mask picture with the bullet screens displayed in the bullet screen display area to form a video anti-blocking bullet screen picture, and display the video anti-blocking bullet screen picture on the bullet screen layer, where the bullet screens included in the video anti-blocking bullet screen picture do not block the target object.

[0094] In this embodiment, the bullet screen display area corresponding to each original frame image can be confirmed according to the display time period of the bullet screen information and the display time of the target original frame image. For example, the bullet screen file of video A includes N pieces of bullet screen information, and the display time period corresponding to bullet screen information 1 is 00:00:05 - 00:00:10; the display time period corresponding to bullet screen information 2 is 00:00:06 - 00:00:11,..., and the display time period corresponding to bullet screen information N is 00:20:00 - 00:20:10. It is possible to determine that the bullet screen information corresponding to the first original frame image is bullet screen information 1 to bullet screen information 5, the bullet screen information corresponding to the second original frame image is bullet screen information 1 to bullet screen information 8,..., and the bullet screen information corresponding to the Nth original frame image is bullet screen information (N - 3) to bullet screen information N. It is possible to determine the bullet screen display area of each bullet screen information in the target frame image according to the quantity and content of the bullet screen information corresponding to the target original frame image.

[0095] Step S33: If there is an overlapping area between the bullet screen display area and the display area of the target object in the original frame image, then synthesize the mask picture with the bullet screens displayed in the bullet screen display area to form a video anti-blocking bullet screen picture, and display the video anti-blocking bullet screen picture on the bullet screen layer, where the bullet screens included in the video anti-blocking bullet screen picture do not block the target object.

[0096] Specifically, the video anti-blocking bullet screen picture is a picture that displays bullet screens and at the same time does not allow the bullet screens to block the target object.

[0097] In this embodiment, when the currently played original video frame contains a target object, and there is an overlapping area between the bullet screen display area corresponding to the currently played original video frame and the display area corresponding to the target object, then the bullet screens to be currently displayed on the bullet screen layer will be synthesized with the mask picture to obtain a video anti-blocking bullet screen picture, and this video anti-blocking bullet screen picture will be displayed on the bullet screen layer. In this way, while the bullet screens are displayed, the target object included in the original video frame will not be blocked by the bullet screens.

[0098] In an implementation manner, when synthesizing the mask picture with the bullet screens displayed in the bullet screen display area to form a video anti-blocking bullet screen picture, the bullet screens located at the corresponding positions of the mask picture can be covered by the mask picture. In this way, the bullet screens will no longer be displayed in the area corresponding to the target object included in the original video frame in the synthesized video anti-blocking bullet screen picture, and the bullet screens will be normally displayed in other areas, thus achieving the effect of anti-blocking bullet screens.

[0099] In another implementation manner, when synthesizing the mask picture with the bullet screens displayed in the bullet screen display area to form a video anti-blocking bullet screen picture, the bullet screens located at the corresponding positions of the mask picture can be covered by the mask picture. At the same time, the covered bullet screens are displayed at other positions, so as to achieve the effect of anti-blocking bullet screens while all the bullet screens are displayed and the target object included in the original frame image is not blocked.

[0100] In this embodiment, during the video playback process, the original frame image of the video is subjected to target object detection processing at a first preset frequency. When the target object is detected, the display area of the target object in the original frame image is output. A mask picture corresponding to the original frame image is generated according to the display area. The bullet screen display area corresponding to the original frame image is obtained. If there is an overlapping area between the bullet screen display area and the display area of the target object in the original frame image, the mask picture is synthesized with the bullet screens displayed in the bullet screen display area to form a video anti-blocking bullet screen picture, and the video anti-blocking bullet screen picture is displayed in the bullet screen layer, where the bullet screens included in the video anti-blocking bullet screen picture do not block the target object. In this application, by displaying the video anti-blocking bullet screen picture during the bullet screen display process, it is possible to avoid the bullet screens from blocking the target object in the video, improving the user experience. At the same time, by detecting and processing the original frame image of the video in the manner of the first preset frequency, it is not necessary to process all the original video frames, which can reduce the occupation of computing resources of the terminal device. In addition, in this application, by implementing real-time calculation in the client to obtain the mask picture, compared with the method of obtaining the mask picture by real-time calculation in the anchor device in the prior art, the consumption of network resources for uploading the mask picture to the video server can be saved. That is to say, the implementation method of this application can save the computing cost, storage cost, and bandwidth cost of the video service provider.

[0101] In an exemplary embodiment, the method further includes:

[0102] Providing a frequency setting interface for the user to set the first preset frequency according to the frequency setting interface.

[0103] In this embodiment, in order to facilitate the user to use the bullet screen display method of this application when using terminal devices with different device performances, in this application, a frequency setting interface can be provided. In this way, the user can set the first preset frequency according to the performance of the device used by himself. In this way, when the user uses a terminal device with high performance, a higher first preset frequency can be set. When the user uses a terminal device with low performance, a generally high first preset frequency can be set.

[0104] It can be understood that the higher the set first preset frequency, the better the anti-blocking bullet screen effect achieved, but at the same time, the more computing resources consumed by the device.

[0105] It should be noted that the maximum value of the first preset frequency is the same as the frame rate of the video.

[0106] In an exemplary embodiment, in order to reduce the calculation latency, during the video playback, the first thread may perform object detection processing on the original frame image of the video at a first preset frequency, and when the object is detected, output the display area of the object in the original frame image, and in parallel, the second thread may generate a mask picture corresponding to the original frame image according to the display area.

[0107] Refer to Figure 7 As shown, it is a program module diagram of an embodiment of the bullet screen display device 70 of the present application.

[0108] In this embodiment, the bullet screen display device 70 includes a series of computer program instructions stored in a memory. When the computer program instructions are executed by a processor, the bullet screen display functions of various embodiments of the present application can be implemented. In some embodiments, based on the specific operations implemented by each part of the computer program instructions, the bullet screen display device 70 can be divided into one or more modules. The specific modules that can be divided are as follows:

[0109] The output module 71 is used to perform object detection processing on the original frame image of the video at a first preset frequency during the video playback, and when the object is detected, output the display area of the object in the original frame image;

[0110] The generation module 72 is used to generate a mask picture corresponding to the original frame image according to the display area;

[0111] The acquisition module 73 is used to acquire the bullet screen display area corresponding to the original frame image;

[0112] The display module 74 is used to, if there is an overlapping area between the bullet screen display area and the display area of the object in the original frame image, synthesize the mask picture and the bullet screens displayed in the bullet screen display area into a video anti-blocking bullet screen picture, and display the video anti-blocking bullet screen picture on the bullet screen layer, where the bullet screens included in the video anti-blocking bullet screen picture do not block the object.

[0113] In an exemplary embodiment, the output module 71 is further used to, during the video playback, detect in real time whether the number of bullet screens displayed on the current screen exceeds a preset number; if the number of bullet screens displayed on the current screen exceeds the preset number, perform object detection processing on the original frame image of the video at a first preset frequency, and when the object is detected, output the display area of the object in the original frame image.

[0114] In an exemplary embodiment, the output module 71 is further configured to obtain the original frame image of the video at a first preset frequency; perform resolution sampling processing on the obtained original frame image to obtain a frame image with a target resolution; perform target object detection processing on the frame image with the target resolution, and when the target object is detected, output the display area of the target object in the original frame image.

[0115] In an exemplary embodiment, the output module 71 is further configured to, during the video playback process, perform target object detection processing on the original frame image of the video at a second preset frequency; after detecting that the original frame image contains the target object, perform target object detection processing on the original frame image of the video at the first preset frequency, and when the target object is detected, output the display area of the target object in the original frame image.

[0116] In an exemplary embodiment, the barrage display device 70 further includes a providing module.

[0117] The providing module is configured to provide a frequency setting interface for the user to set the first preset frequency according to the frequency setting interface.

[0118] In an exemplary embodiment, the output module 71 is further configured to, during the video playback process, perform target object detection processing on the original frame image of the video at the first preset frequency through a first thread, and when the target object is detected, output the display area of the target object in the original frame image;

[0119] The generating module 72 is further configured to generate a mask picture corresponding to the original frame image according to the display area through a second thread.

[0120] In an exemplary embodiment, the output module 71 is further configured to perform target object detection processing on the original frame image of the video through a target object detection model obtained based on a machine learning algorithm, and when the target object is detected, output the display area of the target object in the original frame image.

[0121] Figure 8 It is a schematic flowchart of the barrage display method implemented by the barrage display system in the present application.

[0122] Refer to Figure 8 As shown, the barrage display system includes a host terminal 80, a plurality of clients 810 installed on a viewer terminal 81, a video server 82, and a barrage server 83. Among them, the viewer terminal 81 is a terminal device of a viewer user who watches the host's live broadcast through the host terminal 80 in the host's live broadcast room.

[0123] In this embodiment, when the host conducts a live broadcast through the host terminal 80, the host terminal 80 uploads the host's live video to the video server 82. When the video server 82 receives the live video uploaded by the host terminal 80, it forwards the live video to the viewer terminals 81 of the viewer users who are watching the live video in the host's live broadcast room. The viewer users watching the live video in the live broadcast room can send bullet comments through the viewer terminals 81. After the viewer users send bullet comments through the viewer terminals 81, the viewer terminals 81 upload the bullet comments to the bullet comment server 83. After the bullet comment server 83 receives the bullet comments sent by each viewer terminal 81, it stores the bullet comments sent by the viewer users and forwards the stored bullet comments to each client 810 for each client 810 to display the bullet comments sent by the viewer users while playing the live video. During the playback of the live video by each client 810, the original frame image of the video is subjected to target object detection processing at a first preset frequency, and when the target object is detected, the display area of the target object in the original frame image is output; a mask picture corresponding to the original frame image is generated according to the display area; the bullet comment display area corresponding to the original frame image is obtained; if there is an overlapping area between the bullet comment display area and the display area of the target object in the original frame image, the mask picture and the bullet comments displayed in the bullet comment display area are combined into a video anti-blocking bullet comment picture, and the video anti-blocking bullet comment picture is displayed in the bullet comment layer, wherein the bullet comments included in the video anti-blocking bullet comment picture do not block the target object.

[0124] Figure 10 Schematically shows a hardware architecture diagram of a computer device 8 suitable for implementing the bullet comment display method according to an embodiment of the present application. In this embodiment, the computer device 8 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. For example, it can be a tablet computer, a notebook computer, a desktop computer, a rack-mounted server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. As Figure 10 shown, the computer device 8 at least includes but is not limited to: a memory 111, a processor 121, and a network interface 122 that can be communicatively linked to each other through a system bus. Among them:

[0125] The memory 111 includes at least one type of computer-readable storage medium, which can be volatile or non-volatile. Specifically, the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disc, etc. In some embodiments, the memory 111 can be an internal storage module of the computer device 8, such as the hard disk or memory of the computer device 8. In other embodiments, the memory 111 can also be an external storage device of the computer device 8, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. equipped on the computer device 8. Of course, the memory 111 can also include both the internal storage module and the external storage device of the computer device 8. In this embodiment, the memory 111 is generally used to store the operating system and various application software installed on the computer device 8, such as the program code of the bullet screen display method. In addition, the memory 111 can also be used to temporarily store various data that have been output or will be output.

[0126] In some embodiments, the processor 121 can be a central processing unit (CPU), controller, microcontroller, microprocessor, or other bullet screen display chip. The processor 121 is generally used to control the overall operation of the computer device 8, such as performing control and processing related to data interaction or communication with the computer device 8. In this embodiment, the processor 121 is used to run the program code stored in the memory 111 or process data.

[0127] The network interface 122 may include a wireless network interface or a wired network interface, which is generally used to establish a communication link between the computer device 8 and other computer devices. For example, the network interface 122 is used to connect the computer device 8 to an external terminal through a network, and establish a data transmission channel and a communication link between the computer device 8 and the external terminal. The network may be a wireless or wired network such as an enterprise intranet (Intranet), the Internet, the Global System of Mobile communication (GSM for short), Wideband Code Division Multiple Access (WCDMA for short), a 4G network, a 5G network, Bluetooth, or Wi-Fi.

[0128] It should be noted that Figure 10 only the computer device with components 111 to 122 is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively.

[0129] In this embodiment, the bullet screen display method stored in the memory 111 can be divided into one or more program modules and executed by one or more processors (processor 121 in this embodiment) to complete this application.

[0130] The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the bullet screen display method in the embodiment are implemented.

[0131] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disc, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a Smart Media Card (SMC for short), a Secure Digital (SD for short) card, a Flash Card, etc. Of course, the computer-readable storage medium may also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is generally used to store the operating system installed on the computer device and various application software, such as the program code of the bullet screen display method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various data that have been output or will be output.

[0132] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to at least two network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present application. A person of ordinary skill in the art can understand and implement it without creative effort.

[0133] Through the description of the above embodiments, a person of ordinary skill in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. A person of ordinary skill in the art can understand that all or part of the processes of implementing the above method embodiments can be completed by instructing relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. Among them, the storage medium can be a magnetic disk, an optical disc, a Read-Only Memory (ROM), or a Random Access Memory (RAM), etc.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A bullet screen display method, which is applied to a client, and is characterized in that Including: During the video playback, the original frame image of the video is subjected to target object detection processing at a first preset frequency, and when the target object is detected, the display area of the target object in the original frame image is output; A mask picture corresponding to the original frame image is generated according to the display area; The bullet screen display area corresponding to the original frame image is obtained; If there is an overlapping area between the bullet screen display area and the display area of the target object in the original frame image, the mask picture is synthesized with the bullet screens displayed in the bullet screen display area to form a video bullet screen blocking picture, and the video bullet screen blocking picture is displayed in the bullet screen layer, where the bullet screens included in the video bullet screen blocking picture do not block the target object; Among them, the step of during the video playback, the original frame image of the video is subjected to target object detection processing at a first preset frequency, and when the target object is detected, the display area of the target object in the original frame image is output includes: During the video playback, the original frame image of the video is subjected to target object detection processing at a second preset frequency; After it is detected that the original frame image contains the target object, the original frame image of the video is subjected to target object detection processing at a first preset frequency, and when the target object is detected, the display area of the target object in the original frame image is output; Among them, the step of the original frame image of the video is subjected to target object detection processing at a first preset frequency, and when the target object is detected, the display area of the target object in the original frame image is output includes: The original frame image of the video is obtained at a first preset frequency; The obtained original frame image is subjected to resolution sampling processing to obtain a frame image with a target resolution; The frame image with the target resolution is subjected to target object detection processing, and when the target object is detected, the display area of the target object in the original frame image is output; Among them, the step of during the video playback, the original frame image of the video is subjected to target object detection processing at a first preset frequency, and when the target object is detected, the display area of the target object in the original frame image is output includes: During the video playback, the original frame image of the video is sampled at a first preset frequency, and the sampled original frame image is uploaded to the server for the server to perform target object detection processing on the sampled original frame image, and when the target object is detected, the server outputs the display area of the target object in the original frame image; Receive the display area of the target object in the original frame image output by the server.

2. The bullet screen display method according to claim 1, wherein The step of during the video playback, the original frame image of the video is subjected to target object detection processing at a first preset frequency, and when the target object is detected, the display area of the target object in the original frame image is output includes: During the video playback, it is detected in real time whether the number of bullet screens displayed on the current screen exceeds a preset number; If the number of bullet screens displayed on the current screen exceeds the preset number, the original frame image of the video is subjected to target object detection processing at a first preset frequency, and when the target object is detected, the display area of the target object in the original frame image is output.

3. The bullet screen display method according to claim 1, characterized in that, The method further includes: Providing a frequency setting interface for the user to set the first preset frequency according to the frequency setting interface.

4. The bullet screen display method according to claim 1, wherein, During the video playback, performing target object detection processing on the original frame image of the video at a first preset frequency, and when the target object is detected, outputting the display area of the target object in the original frame image includes: During the video playback, performing target object detection processing on the original frame image of the video at a first preset frequency through a first thread, and when the target object is detected, outputting the display area of the target object in the original frame image; Generating a mask picture corresponding to the original frame image according to the display area includes: Generating a mask picture corresponding to the original frame image according to the display area through a second thread.

5. The bullet screen display method according to claim 1, wherein Performing target object detection processing on the original frame image of the video, and when the target object is detected, outputting the display area of the target object in the original frame image includes: Performing target object detection processing on the original frame image of the video through a target object detection model obtained based on a machine learning algorithm, and when the target object is detected, outputting the display area of the target object in the original frame image.

6. A bullet screen display device, characterized in that, Includes: An output module for performing target object detection processing on the original frame image of the video at a first preset frequency during video playback, and when the target object is detected, outputting the display area of the target object in the original frame image; A generation module for generating a mask picture corresponding to the original frame image according to the display area; An acquisition module for acquiring the bullet screen display area corresponding to the original frame image; A display module for, if there is an overlapping area between the bullet screen display area and the display area of the target object in the original frame image, synthesizing the mask picture and the bullet screens displayed in the bullet screen display area into a video anti-blocking bullet screen picture, and displaying the video anti-blocking bullet screen picture on the bullet screen layer, where the bullet screens included in the video anti-blocking bullet screen picture do not block the target object; Wherein, during the video playback, performing target object detection processing on the original frame image of the video at a first preset frequency, and when the target object is detected, outputting the display area of the target object in the original frame image includes: During the video playback, performing target object detection processing on the original frame image of the video at a second preset frequency; After detecting that the original frame image contains the target object, performing target object detection processing on the original frame image of the video at a first preset frequency, and when the target object is detected, outputting the display area of the target object in the original frame image; Among them, the method of performing target object detection processing on the original frame image of the video at the first preset frequency and outputting the display area of the target object in the original frame image when the target object is detected includes: Obtaining the original frame image of the video at the first preset frequency; Performing resolution sampling processing on the obtained original frame image to obtain a frame image with a target resolution; Performing target object detection processing on the frame image with the target resolution, and outputting the display area of the target object in the original frame image when the target object is detected; Among them, the method of performing target object detection processing on the original frame image of the video at the first preset frequency and outputting the display area of the target object in the original frame image when the target object is detected during video playback includes: During video playback, sampling the original frame image of the video at the first preset frequency, and uploading the sampled original frame image to the server for the server to perform target object detection processing on the sampled original frame image, and when the target object is detected, the server outputs the display area of the target object in the original frame image; Receiving the display area of the target object in the original frame image output by the server.

7. A bullet screen display system, characterized in that, The bullet screen display system includes a host terminal, multiple clients installed on the viewer terminals, a video server, and a bullet screen server. The viewer terminal is the terminal device of the viewer user who watches the live broadcast by the host through the host terminal in the host's live broadcast room. Among them: The host terminal is used to upload the live video of the host to the video server; The video server is used to forward the live video to the viewer terminals of the viewer users who watch the live video in the host's live broadcast room; The bullet screen server is used to store the bullet screens sent by the viewer users and forward the stored bullet screens to each client for each client to display; Each client is used to perform target object detection processing on the original frame image of the live video at the first preset frequency during the live video playback, and output the display area of the target object in the original frame image when the target object is detected; generating a mask image corresponding to the original frame image according to the display area; obtaining the bullet screen display area corresponding to the original frame image; if there is an overlapping area between the bullet screen display area and the display area of the target object in the original frame image, then synthesizing the mask image and the bullet screens displayed in the bullet screen display area into a video anti-blocking bullet screen picture and displaying the video anti-blocking bullet screen picture on the bullet screen layer, where the bullet screens included in the video anti-blocking bullet screen picture do not block the target object; Among them, the method of performing target object detection processing on the original frame image of the video at the first preset frequency and outputting the display area of the target object in the original frame image when the target object is detected during video playback includes: During the video playback process, perform object detection processing on the original frame image of the video at a second preset frequency; After detecting that the original frame image contains the target object, perform object detection processing on the original frame image of the video at a first preset frequency, and when the target object is detected, output the display area of the target object in the original frame image; Among them, the performing object detection processing on the original frame image of the video at a first preset frequency and, when the target object is detected, outputting the display area of the target object in the original frame image includes: Obtain the original frame image of the video at a first preset frequency; Perform resolution sampling processing on the obtained original frame image to obtain a frame image with a target resolution; Perform object detection processing on the frame image with the target resolution, and when the target object is detected, output the display area of the target object in the original frame image; Among them, the performing object detection processing on the original frame image of the video at a first preset frequency during the video playback process and, when the target object is detected, outputting the display area of the target object in the original frame image includes: During the video playback process, sample the original frame image of the video at a first preset frequency, and upload the sampled original frame image to the server for the server to perform object detection processing on the sampled original frame image, and when the target object is detected, the server outputs the display area of the target object in the original frame image; Receive the display area of the target object in the original frame image output by the server.

8. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images