Bullet screen recognition method and related device

Abstract

The embodiments of the present application provide a barrage recognition method and related devices, which are applied to the field of terminal technology. The method includes: obtaining the video frame rate of the target application, and the drawing frequency corresponding to each layer of the target application in the drawing service within a preset time length; judging whether there is a barrage in the video interface of the target application according to the video frame rate and the drawing frequency corresponding to each layer. Based on the fact that the drawing frequency corresponding to the barrage is greater than the video frame rate, the method judges whether there is a drawing corresponding to the barrage in each layer by comparing the video frame rate and the drawing frequency corresponding to each layer, and then can identify whether there is a barrage in the video interface of the target application. The terminal device can adjust the device performance according to whether there is a barrage in the video interface of the target application, which can improve the user experience.

Description

[0001] This application is a divisional application of the patent application “Bullet screen recognition method and related device” with application number 202210621235.7 filed on June 2, 2022. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a bullet screen recognition method and related devices. Background Art

[0003] At present, with the development of terminal technology, more and more terminal devices support video applications, and users' functional requirements for video applications are becoming more diversified, for example, the barrage function in video applications.

[0004] In one implementation, during video playback on a terminal device, a user can control the barrage function to be turned on or off.

[0005] However, when a video including barrage is played, the terminal device cannot recognize whether there is barrage. Summary of the invention

[0006] The embodiment of the present application provides a bullet screen recognition method and related device, which are applied to the field of terminal technology. According to the video frame rate and the drawing frequency corresponding to each layer, it is judged whether there is a bullet screen in the video interface of the target application, and the terminal device can adjust the device performance according to whether there is a bullet screen in the video interface of the target application to improve the user experience.

[0007] In the first aspect, an embodiment of the present application proposes a barrage recognition method, including: obtaining the video frame rate of the target application, and the drawing frequency corresponding to each layer of the target application in the drawing service within a preset time length; based on the video frame rate and the drawing frequency corresponding to each layer, determining whether there is a barrage in the video interface of the target application.

[0008] In this way, based on the fact that the drawing frequency corresponding to the barrage is greater than the video frame rate, this method determines whether there are drawings corresponding to the barrage in each layer by comparing the video frame rate and the drawing frequency corresponding to each layer, and then can identify whether there is a barrage in the video interface of the target application. The terminal device can adjust the device performance according to whether there is a barrage in the video interface of the target application, which can improve the user experience.

[0009] Optionally, judging whether barrage exists in the video interface of the target application based on the video frame rate and the drawing frequencies corresponding to each layer, including: calculating a first number of drawing frame rates that are greater than the video frame rate among the drawing frequencies corresponding to each layer, and when the first number is greater than 0, determining that barrage exists in the video interface of the target application.

[0010] In this way, since the drawing frequency of barrage is greater than the video frame rate, the number of drawing frame rates whose drawing frequency is greater than the video frame rate can be calculated to determine the presence of barrage in the video interface of the target application, and the terminal device can adjust the device performance to improve the user experience.

[0011] Optionally, the drawing frequency corresponding to each layer is recorded as a correspondence between the layer identifier and the drawing frequency corresponding to the layer identifier. The drawing frequency corresponding to any layer identifier is a ratio of the number of drawings to a preset duration. The number of drawings is the number of times any layer identifier is drawn within the preset duration. The number of layers of the target application in the drawing service is one or more layers.

[0012] In this way, by recording the corresponding relationship between the layer identifier and the drawing frequency corresponding to the layer identifier, when the number of layers in the drawing service of the target application is one or more, it is convenient to compare the size relationship between the video frame rate and the drawing frequency corresponding to each layer.

[0013] Optionally, after determining whether there is barrage in the video interface of the target application based on the video frame rate and the drawing frequencies corresponding to each layer, the method further includes: when it is determined that there is no barrage in the video interface of the target application, monitoring whether the drawing frequencies corresponding to each layer of the target application in the drawing service change; when it is monitored that there is a change in the drawing frequencies corresponding to each layer of the target application in the drawing service, determining that there is barrage in the video interface of the target application.

[0014] In this way, by monitoring whether the drawing frequency corresponding to each layer of the target application in the drawing service changes, it is possible to determine whether there is a sudden barrage in the video interface of the target application. This is simple and intuitive, and the terminal device can adjust the device performance according to whether there is a barrage in the video interface of the target application, thereby improving the user experience.

[0015] Optionally, monitoring whether the drawing frequencies corresponding to each layer of the target application in the drawing service have changed includes: recording the drawing frequencies of each layer of the target application in the drawing service according to a preset period; for any record obtained according to the preset period, comparing any record with the drawing frequencies corresponding to each layer obtained within a preset time period; for any layer in the drawing service, when the difference between the drawing frequencies corresponding to any layer in any record and any layer obtained within a preset time period is greater than or equal to a threshold, determining that the drawing frequencies corresponding to each layer of the target application in the drawing service have changed; or, when the difference between the drawing frequencies corresponding to any layer in any record and any layer obtained within a preset time period is less than a threshold, determining that the drawing frequencies corresponding to each layer of the target application in the drawing service have not changed.

[0016] In this way, by calculating the difference between the drawing frequency of any layer in any record and the drawing frequency corresponding to any layer obtained within a preset time period or the drawing frequency in the previous record, it is determined whether the drawing frequencies corresponding to each layer in the drawing service of the target application have changed, thereby judging whether there are sudden barrages on the video interface of the target application, and the terminal device adjusts the device performance to improve the user experience.

[0017] Optionally, the video frame rate of the target application and the corresponding drawing frequencies of each layer of the target application in the drawing service within a preset duration are obtained, including: when the target application starts video playback, the video in the target application is displayed in full screen, the terminal device displaying the video interface of the target application switches between horizontal and vertical screen states, and / or when the video interface of the target application receives a user operation, the video frame rate of the target application and the corresponding drawing frequencies of each layer of the target application in the drawing service within a preset duration are obtained.

[0018] In this way, on the one hand, by obtaining the video frame rate of the target application and the corresponding drawing frequency of each layer of the target application in the drawing service within the preset duration in the above-mentioned scenarios that may trigger the opening or closing of the barrage function, the terminal device can identify whether there is a barrage on the video interface of the target application, and adjust the device performance to improve the user experience. On the other hand, when the video in the target application is displayed in full screen, there will be no other objects in the video interface, reducing the interference caused by other objects when identifying whether there is a barrage on the video interface of the target application.

[0019] Optionally, obtaining the video frame rate of the target application includes: obtaining the video frame rate of the target application in the media codec MediaCodec.

[0020] In this way, by obtaining the video frame rate of the video itself in the target application from the media codec, it is possible to determine whether there is a barrage drawing with a drawing frame rate greater than the video frame rate in the drawing service, thereby identifying whether there is a barrage in the video interface of the target application, and the terminal device adjusts the device performance to improve the user experience.

[0021] Optionally, the method further includes: when it is determined that there are bullet screens on the video interface of the target application, increasing the screen refresh rate; or, when it is determined that there are no bullet screens on the video interface of the target application, reducing the screen refresh rate.

[0022] In this way, the terminal device can adjust the screen refresh rate according to whether there is a bullet screen. When the terminal device recognizes the presence of a bullet screen, it can increase the screen refresh rate accordingly to reduce the lag of the bullet screen scrolling; when the terminal device recognizes the absence of a bullet screen, it can decrease the screen refresh rate accordingly to reduce the processor load and power consumption, and improve the user experience.

[0023] In a second aspect, an embodiment of the present application provides a terminal device, which may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device may be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc.

[0024] The terminal device includes: a touch screen, wherein the touch screen includes a touch panel and a display screen; one or more processors; a memory; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions. When the instructions are executed by the terminal device, the terminal device performs the following steps: obtaining the video frame rate of the target application, and the drawing frequency corresponding to each layer of the target application in the drawing service within a preset time length; judging whether there is a barrage on the video interface of the target application according to the video frame rate and the drawing frequency corresponding to each layer.

[0025] Optionally, judging whether barrage exists in the video interface of the target application based on the video frame rate and the drawing frequencies corresponding to each layer, including: calculating a first number of drawing frame rates that are greater than the video frame rate among the drawing frequencies corresponding to each layer, and when the first number is greater than 0, determining that barrage exists in the video interface of the target application.

[0026] Optionally, the drawing frequency corresponding to each layer is recorded as a correspondence between the layer identifier and the drawing frequency corresponding to the layer identifier. The drawing frequency corresponding to any layer identifier is a ratio of the number of drawings to a preset duration. The number of drawings is the number of times any layer identifier is drawn within the preset duration. The number of layers of the target application in the drawing service is one or more layers.

[0027] Optionally, after determining whether there is barrage in the video interface of the target application based on the video frame rate and the drawing frequencies corresponding to each layer, the method further includes: when it is determined that there is no barrage in the video interface of the target application, monitoring whether the drawing frequencies corresponding to each layer of the target application in the drawing service change; when it is monitored that there is a change in the drawing frequencies corresponding to each layer of the target application in the drawing service, determining that there is barrage in the video interface of the target application.

[0028] Optionally, monitoring whether the drawing frequencies corresponding to each layer of the target application in the drawing service have changed includes: recording the drawing frequencies of each layer of the target application in the drawing service according to a preset period; for any record obtained according to the preset period, comparing any record with the drawing frequencies corresponding to each layer obtained within a preset time period; for any layer in the drawing service, when the difference between the drawing frequencies corresponding to any layer in any record and any layer obtained within a preset time period is greater than or equal to a threshold, determining that the drawing frequencies corresponding to each layer of the target application in the drawing service have changed; or, when the difference between the drawing frequencies corresponding to any layer in any record and any layer obtained within a preset time period is less than a threshold, determining that the drawing frequencies corresponding to each layer of the target application in the drawing service have not changed.

[0029] Optionally, the video frame rate of the target application and the corresponding drawing frequencies of each layer of the target application in the drawing service within a preset duration are obtained, including: when the target application starts video playback, the video in the target application is displayed in full screen, the terminal device displaying the video interface of the target application switches between horizontal and vertical screen states, and / or when the video interface of the target application receives a user operation, the video frame rate of the target application and the corresponding drawing frequencies of each layer of the target application in the drawing service within a preset duration are obtained.

[0030] Optionally, obtaining the video frame rate of the target application includes: obtaining the video frame rate of the target application in the media codec MediaCodec.

[0031] Optionally, the method further includes: when it is determined that there are bullet screens on the video interface of the target application, increasing the screen refresh rate; or, when it is determined that there are no bullet screens on the video interface of the target application, reducing the screen refresh rate.

[0032] In a third aspect, an embodiment of the present application provides a terminal device, including: a processor and a memory, the processor being used to call a program in the memory so that the terminal device executes the method provided in the first aspect above.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a terminal device, the terminal device executes the method provided in the first aspect above.

[0034] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the method provided in the first aspect.

[0035] The beneficial effects of the possible implementation methods of the second to fifth aspects mentioned above can be referred to the beneficial effects brought about by the first aspect mentioned above, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0037] Figure 2 A software structure diagram of a terminal device provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of a scenario applicable to the embodiments of the present application;

[0039] Figure 4 A flowchart of a bullet comment recognition method provided in an embodiment of the present application;

[0040] Figure 5 A flowchart of another bullet comment recognition method provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of the structure of a bullet screen recognition device provided in an embodiment of the present application;

[0042] Figure 7 A schematic diagram of the hardware structure of a bullet screen recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish between the same items or similar items with substantially the same functions and effects. For example, the interface of the first target function and the interface of the second target function are to distinguish between different response interfaces, and their order is not limited. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit them to be different.

[0044] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0045] The embodiment of the present application provides a bullet screen recognition method, which can be applied to electronic devices that support application software such as video applications or live broadcast applications and have bullet screen functions. Electronic devices include terminal devices, which can also be called terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. The terminal device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiment of the present application does not limit the specific technology and specific device form adopted by the terminal device.

[0046] In order to better understand the embodiment of the present application, the structure of the terminal device in the embodiment of the present application is introduced below:

[0047] Figure 1The structure diagram of the terminal device 100 is shown. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0048] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0049] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0050] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0051] The processor 110 may also be provided with a memory for storing instructions and data.

[0052] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuits sound (I2S) interface, a pulse code modulation (PCM) interface, and / or a universal serial bus (USB) interface, etc.

[0053] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL).

[0054] The I2S interface can be used for audio communication.

[0055] The PCM interface can also be used for audio communication to sample, quantize and encode analog signals.

[0056] The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication.

[0057] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 100. For example, in an embodiment of the present application, the processor 110 and the display screen 194 communicate via the DSI interface to display the video and bullet screen involved in any bullet screen recognition method proposed in the embodiment of the present application on the display screen 194.

[0058] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or a data signal.

[0059] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0060] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0061] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0062] The terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information. For example, in an embodiment of the present application, the GPU can be used to display the video and bullet screen involved in any bullet screen recognition method proposed in an embodiment of the present application on the display screen 194.

[0063] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diodes (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1. For example, in an embodiment of the present application, the display screen 194 can be used to display the video and barrage involved in any barrage recognition method proposed in an embodiment of the present application.

[0064] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0065] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals.

[0066] The video codec is used to compress or decompress digital video. The terminal device 100 may support one or more video codecs. In this way, the terminal device 100 can play or record videos in multiple coding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc. For example, in the embodiment of the present application, the video codec can be used to decode the video played by the video application involved in any barrage method proposed in the embodiment of the present application.

[0067] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function.

[0068] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 can include a program storage area and a data storage area.

[0069] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0070] The audio module 170 is used to convert digital audio information into analog audio signal output, and also used to convert analog audio input into digital audio signal. The audio module 170 can also be used to encode and decode audio signals.

[0071] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.

[0072] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the terminal device 100 by inserting the SIM card interface 195 or pulling the SIM card interface 195 out. The terminal device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communications.

[0073] The software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture, etc. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device 100.

[0074] Figure 2 1 is a software structure diagram of the terminal device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the management service layer, the Android runtime (Android runtime) and the system library, and the kernel layer.

[0075] The application layer can include a series of application packages. Figure 3 As shown, the application package may include applications such as camera, calendar, phone, map, game, video, etc.

[0076] Among them, the video application is used to provide video playback function for users.

[0077] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0078] like Figure 3 As shown, the application framework layer may include media codecs, resource managers, notification managers, graphics drawing and graphics rendering, etc.

[0079] The media codec is used to decode the video that the video application needs to play and obtain the video source frame rate and video play / stop information.

[0080] The resource manager provides various resources for applications, such as horizontal and vertical screen switching, icons, pictures, layout files, video files, etc.

[0081] Graphics drawing is used to draw graphics, such as drawing videos and / or bullet comments in video applications.

[0082] Graphics rendering is used to render the drawn graphics. For example, after the graphics rendering completes the drawing of the video and / or bullet screen in the video application, the graphics rendering renders the video and / or bullet screen.

[0083] The window manager service (WMS) is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0084] The package manager is used for program management within the system, such as application installation, uninstallation, and upgrades.

[0085] The management service layer includes barrage recognition, etc.

[0086] Bullet screen recognition is used to identify bullet screens in video applications.

[0087] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0088] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0089] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0090] The system library may include multiple functional modules, such as graphics synthesis, surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0091] Graphics synthesis is used to synthesize one or more rendered views into a display interface, for example, synthesizing the video and / or bullet comments in a video application after drawing and rendering.

[0092] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0093] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0094] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing.

[0095] A 2D graphics engine is a drawing engine for 2D drawings.

[0096] The kernel layer is the layer between hardware and software. The kernel layer can include display drivers, camera drivers, audio drivers, CPU drivers, etc.

[0097] The display driver is used to display an interface obtained through drawing, rendering, synthesis and other processes, such as displaying a video and / or bullet screen interface in a video application.

[0098] The following is an illustrative description of the software and hardware workflow of the terminal device 100 in conjunction with a scenario where an application is started or an interface switch occurs in an application.

[0099] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into raw input events (including touch coordinates, touch strength, timestamp of the touch operation, and other information). The raw input events are stored in the kernel layer. The application framework layer obtains the raw input events from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the video application icon, the video application calls the interface of the application framework layer to start the video application, and then starts the display driver by calling the kernel layer to display the functional interface of the video application.

[0100] With the development of terminal technology, more and more terminal devices support video applications, and users have more and more diverse functional requirements for video applications, such as the bullet screen function in video applications. The technical solution provided in the embodiment of the present application can be applied to video applications that support the bullet screen function, especially to identify the bullet screen during video playback.

[0101] In one implementation, during the video playback process of the terminal device, the user can control the opening or closing of the bullet screen function. For example, the user can turn on or off the bullet screen by clicking the "bullet screen" button. After the bullet screen function is turned on, a rolling bullet screen will appear during the video playback process.

[0102] For example, when the bullet screen is turned on, a 60Hz screen refresh rate is required to meet the mobile display requirements of the bullet screen; when the bullet screen is not turned on, only a 30Hz screen refresh rate is required to meet the smooth display requirements of most videos. Therefore, when the bullet screen function is turned on during video playback, the refresh rate requirement will be higher than when the bullet screen is not turned on because the video interface needs to display the scrolling bullet screen dynamically.

[0103] However, in the above method, the terminal device cannot identify whether there is a bullet screen in the video application, and cannot adjust the device performance according to whether there is a bullet screen, resulting in a poor user experience. For example, the screen refresh rate can be adjusted according to whether there is a bullet screen.

[0104] In view of this, the embodiment of the present application provides a method for identifying bullet screens. The terminal device obtains the video frame rate of the target application and the drawing frequency corresponding to each layer of the target application in the drawing service within a preset time. According to the video frame rate and the drawing frequency corresponding to each layer, it determines whether there is a bullet screen in the video interface of the target application. Based on the fact that the drawing frequency corresponding to the bullet screen is greater than the video frame rate, the method determines whether there is a drawing corresponding to the bullet screen in the layer by comparing the video frame rate and the drawing frequency corresponding to each layer, and then it can be identified whether there is a bullet screen in the target application. The terminal device can adjust the device performance according to whether there is a bullet screen, which can improve the user experience.

[0105] Figure 3 A schematic diagram of a scenario to which the bullet screen recognition method according to an embodiment of the present application is applicable is shown.

[0106] like Figure 3 As shown in a in FIG. 3 , during the video playback process, the bullet screen function is not turned on, and the video being played is displayed in the video playback interface. The user turns on the bullet screen function by clicking the “bullet screen” button 301, and then the video playback interface appears as follows: Figure 3 The barrage 303 is shown in b.

[0107] For example, the terminal device can adjust the device performance by identifying whether there is a bullet screen on the video interface of the target application. Figure 3 When the video played by a in the figure is a movie, a refresh rate of only 30Hz is required to meet the requirement of smooth display. Figure 3 In the scenario shown in b in the figure, when the bullet screen function is turned on during video playback, the refresh rate requirement will be higher than Figure 3 The scene of the non-opened bullet screen as shown in a in the figure. Therefore, the terminal device can adjust the screen refresh rate according to whether there is a bullet screen. When the terminal device recognizes that there is a bullet screen, the screen refresh rate can be increased accordingly; and when the terminal device recognizes that there is no bullet screen, the screen refresh rate can be decreased accordingly.

[0108] The following is a detailed description of the bullet screen recognition method of the embodiment of the present application through a specific embodiment. The following embodiments can be combined with each other or implemented independently, and the same or similar concepts or processes may not be repeated in some embodiments.

[0109] Figure 4 A flowchart of a bullet screen recognition method provided in an embodiment of the present application. Figure 4 As shown, the method may include:

[0110] S401 : Obtain the video frame rate of the target application and the drawing frequencies corresponding to each layer of the target application in the drawing service within a preset duration.

[0111] In the embodiment of the present application, the target application may be an application with a video playback function, such as a video application, a live broadcast application, a music application that can play music clips, etc. The video frame rate may be understood as the frame rate at which the video played in the target application is output to the display screen through the graphics card. Different target applications may correspond to different video frame rates.

[0112] In the embodiment of the present application, the drawing service can be used to draw, render and synthesize the video and / or bullet screen in the target application. The drawing frequency corresponding to each layer of the target application in the drawing service can be the ratio of the number of drawings corresponding to each layer to the preset duration.

[0113] It should be noted that the preset duration can be understood as a certain period of time during the video playback process, for example, it can be set to 100ms. The specific value of the preset duration is not limited in the embodiment of the present application.

[0114] For example, in the drawing service, the drawing in the target application may include video drawing and / or bullet screen drawing. When there is a bullet screen during video playback, the drawing in the target application includes video drawing and bullet screen drawing; when there is no bullet screen during video playback, the drawing in the target application includes video drawing.

[0115] In a possible implementation, when there is no bullet screen during video playback, the layers of the target application in the drawing service include a video layer, and the drawing frequency corresponding to the video layer in the drawing service within a preset duration is obtained.

[0116] In another possible implementation, when there is a bullet screen during video playback, the video drawing and the bullet screen drawing are drawn in the same layer, and the drawing frequency corresponding to the layer in the drawing service within the preset duration is obtained. Among them, the drawing frequency corresponding to the layer is the larger drawing frequency of the video drawing frequency and the bullet screen drawing frequency. Exemplarily, the drawing frequency of the bullet screen can be greater than the video frame rate, and the drawing frequency corresponding to the layer is the drawing frequency of the bullet screen. For example, the drawing frequency corresponding to the video drawing is 30Hz, and the drawing frequency corresponding to the bullet screen drawing is 60Hz, then the drawing frequency corresponding to the layer is 60Hz.

[0117] In another possible implementation, when there is a bullet screen during video playback, the video drawing and the bullet screen drawing are drawn in different layers, and the drawing frequency corresponding to the video layer and the drawing frequency corresponding to the bullet screen layer in the drawing service within a preset duration are obtained.

[0118] S402: Determine whether there is a bullet screen in the video interface of the target application according to the video frame rate and the drawing frequency corresponding to each layer.

[0119] In the embodiments of the present application, barrage can be understood as commentary subtitles that pop up when watching a video. Different videos may correspond to different barrage contents and numbers of barrages, and the sending time of different barrages is different. Barrages can appear at a specific time point in the video, and barrages sent at the same time basically have the same theme.

[0120] For example, when a video is played to the 43rd minute, user 1 sends a barrage. When other users watch the video, they can see the barrage sent by user 1 at the 43rd minute. Among the barrages sent at the 43rd minute, most of the barrages sent by users are comments on the content played around the 43rd minute.

[0121] Exemplarily, the video frame rate and the size relationship of the drawing frequencies corresponding to each layer in the drawing service within a preset time are compared, and whether there is a bullet screen is determined based on the size relationship.

[0122] In one possible implementation, when there is no barrage during video playback, the layers of the target application in the drawing service include a video layer. Then, by comparing the video frame rate and the drawing frequency corresponding to the video layer in the drawing service within a preset duration, it can be determined whether there is a barrage on the video interface of the target application.

[0123] In another possible implementation, when there is barrage during video playback, the video drawing and the barrage drawing are drawn in the same layer. The video frame rate and the corresponding drawing frequency of this layer in the drawing service within a preset duration can be compared to determine whether there is barrage in the video interface of the target application.

[0124] In another possible implementation, when there is barrage during video playback, the video drawing and the barrage drawing are drawn in different layers. The video frame rate and the drawing frequency corresponding to the video layer and the drawing frequency corresponding to the barrage layer in the drawing service within a preset duration can be compared to determine whether there is barrage in the video interface of the target application.

[0125] The barrage recognition method provided in the embodiment of the present application is that the terminal device obtains the video frame rate of the target application and the drawing frequency corresponding to each layer of the target application in the drawing service within a preset time; based on the video frame rate and the drawing frequency corresponding to each layer, it is determined whether there is a barrage in the video interface of the target application. Based on the fact that the drawing frequency corresponding to the barrage is greater than the video frame rate, the method determines whether there is a drawing corresponding to the barrage in each layer by comparing the video frame rate and the drawing frequency corresponding to each layer, and then it can be identified whether there is a barrage in the video interface of the target application. The terminal device can adjust the device performance according to whether there is a barrage in the video interface of the target application, which can improve the user experience.

[0126] Optional, in Figure 4 On the basis of the corresponding embodiments, in the method of the embodiments of the present application:

[0127] In one possible implementation, the video frame rate of the target application and the corresponding drawing frequencies of each layer of the target application in the drawing service within a preset duration are obtained, including: when the target application starts video playback, the video in the target application is displayed in full screen, the terminal device displaying the video interface of the target application switches between horizontal and vertical screen states, and / or when the video interface of the target application receives a user operation, the video frame rate of the target application and the corresponding drawing frequencies of each layer of the target application in the drawing service within a preset duration are obtained.

[0128] In the embodiment of the present application, the target application starting video playback can be understood as starting video playback in the target application. It should be noted that the video can be a live video, a recorded video, a TV series or a movie, etc. For example, starting video playback in a video application or starting live playback in a live application, etc.

[0129] In the embodiment of the present application, full screen display can be understood as displaying the required window as the largest on the screen. The full screen display of the video in the target application can be understood as displaying the video as the largest on the screen when the target application plays the video. For example, when the video is displayed in full screen, the video can cover the entire screen.

[0130] In the embodiment of the present application, the horizontal and vertical screen state switching can be switching the horizontal screen to the vertical screen or switching the vertical screen to the horizontal screen. The horizontal and vertical screen state switching of the terminal device displaying the video interface can be understood as switching the horizontal and vertical screen states of the terminal device when the display interface of the terminal device is a video interface. Among them, the horizontal screen can be full screen.

[0131] In the embodiment of the present application, the video interface receiving a user operation can be understood as when the user operates the video interface by touching the screen or clicking a button, the video interface can receive the user operation. Figure 3 As shown, on the video interface, the user clicks the "bullet screen" button 301. After the video interface receives the user's operation of clicking the "bullet screen" button 301, the bullet screen function is turned on.

[0132] Exemplarily, when the target application starts video playback, the video in the target application is displayed in full screen, the terminal device displaying the video interface switches between horizontal and vertical screen states, the video interface receives user operations and other scenarios, the video frame rate of the target application and the corresponding drawing frequencies of each layer of the target application in the drawing service within a preset duration are obtained.

[0133] Exemplarily, multiple of the above scenarios may include the target application starting video playback and the video being displayed in full screen, or, when the video in the target application is displayed in horizontal full screen, the terminal device switches from a horizontal state to a vertical state, or, when the video in the target application is displayed in full screen, the video interface receives a user operation, or, the terminal device displaying the video interface switches from a vertical non-full screen state to a horizontal full screen state, and so on.

[0134] It is understandable that since the target application turns the barrage function on or off by default, when the target application starts video playback, the video frame rate of the target application and the corresponding drawing frequency of each layer of the target application in the drawing service within the preset duration can be obtained to identify whether there is a barrage on the video interface of the target application. The terminal device can then adjust the device performance to improve the user experience.

[0135] It is understandable that in one possible implementation, the horizontal screen state may be full-screen display, and the vertical screen state may be non-full-screen display. Under the premise that the target application turns on the barrage function, when the video interface is displayed in full screen, barrage will appear on the video interface; when the video interface is displayed in non-full screen, barrage will not appear on the video interface. Therefore, when the terminal device displaying the video interface switches between horizontal and vertical screen states, the barrage situation of the video interface may change. For example, the video interface is a horizontal full-screen display interface and there is a barrage. When the terminal device switches from horizontal to vertical, the video interface switches to a vertical non-full-screen display interface and the barrage disappears. Therefore, when the terminal device displaying the video interface switches between horizontal and vertical screen states, it identifies whether there is a barrage on the video interface of the target application, and the terminal device adjusts the device performance to improve the user experience.

[0136] It is understandable that when the video interface receives a user operation, the operation may be an operation of the user turning on or off the bullet screen. Therefore, when the video interface receives a user operation, it identifies whether there is a bullet screen on the video interface of the target application, and the terminal device adjusts the device performance to improve the user experience.

[0137] It is understandable that when the video in the target application is displayed in full screen, there will be no other objects in the video interface, which reduces the interference caused by other objects when identifying whether there is a barrage on the video interface of the target application.

[0138] In a possible implementation, obtaining the video frame rate of the target application includes: obtaining the video frame rate of the target application in a media codec Media Codec.

[0139] Exemplarily, the media codec can be used to decode the video to be played by the target application to obtain the video frame rate, and then the terminal device can obtain the video frame rate of the target application from the media codec.

[0140] In an embodiment of the present application, by obtaining the video frame rate of the video itself in the target application from the media codec, it is determined whether there is a barrage drawing with a drawing frame rate greater than the video frame rate in the drawing service, thereby identifying whether there is a barrage in the video interface of the target application, and the terminal device adjusts the device performance to improve the user experience.

[0141] In one possible implementation, based on the video frame rate and the drawing frequencies corresponding to each layer, it is determined whether there are barrages in the video interface of the target application, including: calculating a first number of drawing frame rates that are greater than the video frame rate among the drawing frequencies corresponding to each layer; when the first number is greater than 0, it is determined that there are barrages in the video interface of the target application.

[0142] Exemplarily, after obtaining the video frame rate of the target application and the drawing frequency corresponding to each layer of the target application in the drawing service within a preset time, the size relationship between the video frame rate and the drawing frequency corresponding to each layer is compared. In the scenario where the bullet screen function is turned on during video playback, since the bullet screen is displayed in a dynamic scrolling manner, the drawing frequency of the bullet screen is greater than the video frame rate.

[0143] Based on this, the drawing frame rates that are greater than the video frame rate can be obtained from the drawing frequencies corresponding to each layer. When the number of drawing frame rates that are greater than the video frame rate is greater than 0, it is determined that there are barrages in the video interface of the target application; when the number of drawing frame rates that are greater than the video frame rate is not greater than 0, it is determined that there are no barrages in the video interface of the target application.

[0144] In one possible implementation, when there is no barrage during video playback, the layers of the target application in the drawing service include a video layer, and it can be obtained that the drawing frequency corresponding to the video layer is not greater than the video frame rate, then the first number is equal to 0, and it can be determined that there is no barrage on the video interface of the target application.

[0145] In another possible implementation, when there is barrage during video playback, the video drawing and the barrage drawing are drawn in the same layer, and the drawing frequency corresponding to this layer is the drawing frequency of the barrage. Then, it can be obtained that the drawing frequency corresponding to this layer is greater than the video frame rate, and the first number is greater than 0, and then it can be determined that there is barrage in the video interface of the target application.

[0146] In another possible implementation, when there is barrage during video playback, the video drawing and the barrage drawing are drawn in different layers, then it can be obtained that the drawing frequency corresponding to the barrage layer is greater than the video frame rate, and the first number is greater than 0, and then it can be determined that there is barrage in the video interface of the target application.

[0147] In an embodiment of the present application, since the drawing frequency of the barrage is greater than the video frame rate, the presence of barrage in the video interface of the target application can be determined by calculating the number of drawing frame rates whose drawing frequency is greater than the video frame rate. The terminal device can then adjust the device performance to improve the user experience.

[0148] In a possible implementation, the drawing frequency corresponding to each layer is recorded as a correspondence between the layer identifier and the drawing frequency corresponding to the layer identifier. The drawing frequency corresponding to any layer identifier is a ratio of the number of drawings to a preset time length. The number of drawings is the number of times any layer identifier is drawn within the preset time length. The number of layers of the target application in the drawing service is one or more layers.

[0149] In the embodiment of the present application, when recording the drawing frequencies corresponding to each layer, the layer identifier and the drawing frequency corresponding to the layer identifier may be matched one by one. For example, the drawing frame rate corresponding to the video layer is 30 Hz, the drawing frame rate corresponding to the bullet screen layer is 60 Hz, and so on.

[0150] In an embodiment of the present application, the drawing frequency corresponding to any layer identifier can be the ratio of the number of times drawn within a preset time length to the preset time length. For example, the preset time length is 4 seconds, and the number of times the video layer is drawn within 4 seconds is 120 times, then the drawing frame rate corresponding to the video layer is 30Hz.

[0151] Exemplarily, the case where the number of layers of the target application in the drawing service is one layer may include: when there is no barrage during video playback, the layers of the target application in the drawing service include a video layer, or, when there is barrage during video playback, the video drawing and the barrage drawing are drawn in the same layer.

[0152] Exemplarily, the case where the number of layers of the target application in the drawing service is multiple layers may include: when there is barrage during video playback, the video drawing and the barrage drawing are drawn in different layers, then the layers of the target application in the drawing service include the video layer and the barrage layer.

[0153] In the embodiment of the present application, by recording the corresponding relationship between the layer identifier and the drawing frequency corresponding to the layer identifier, when the number of layers in the drawing service of the target application is one or more layers, it is convenient to compare the size relationship between the video frame rate and the drawing frequency corresponding to each layer.

[0154] Exemplarily, based on the above embodiments, Figure 5 A flow chart of another method for identifying bullet comments provided by an embodiment of the present application is shown, and the specific steps may be:

[0155] S501. When the target application starts video playback, the video in the target application is displayed in full screen, the terminal device displaying the video interface switches between horizontal and vertical screen states, and / or when the video interface receives a user operation, the video frame rate of the target application in the media codec Media Codec and the drawing frequency corresponding to each layer of the target application in the drawing service within a preset duration are obtained.

[0156] S502: Determine whether there is a bullet screen in the video interface of the target application according to the video frame rate and the drawing frequency corresponding to each layer.

[0157] Exemplarily, for the specific implementation of step S502, reference may be made to steps S5021-S5022.

[0158] S5021. Calculate a first number of drawing frame rates that are greater than the video frame rate among drawing frequencies corresponding to each layer.

[0159] S5022: When the first number is greater than 0, determine that there is a bullet screen on the video interface of the target application; or, when the first number is not greater than 0, determine that there is no bullet screen on the video interface of the target application.

[0160] S503: When it is determined that there is no bullet screen in the video interface of the target application, monitor whether the drawing frequencies corresponding to each layer in the drawing service of the target application change.

[0161] Exemplarily, when it is determined through the above steps S501-S502 that there is no barrage on the video interface of the target application, it means that there is no continuous barrage on the video interface of the target application. Therefore, it is possible to determine whether there is a sudden barrage on the video interface of the target application by monitoring whether the drawing frequency corresponding to each layer in the drawing service of the target application changes. Among them, continuous barrage can be understood as the barrage always exists and is continuous during the entire video playback process; and sudden barrage can be understood as the barrage appears very few times and intermittently during the entire video playback process, for example, a barrage appears once every 7 minutes, or a barrage appears only once during the entire video playback process, etc.

[0162] Exemplarily, for the specific implementation of step S503, reference may be made to steps S5031-S5033.

[0163] S5031. Record the drawing frequency of each layer of the target application in the drawing service according to a preset period.

[0164] In the embodiment of the present application, the preset period can be understood as a certain time interval. For example, the preset period can be 100 ms. The embodiment of the present application does not make any specific limitation on the specific value of the preset period.

[0165] Exemplarily, the drawing frequency of each layer of the target application in the drawing service is recorded once at a certain time interval.

[0166] S5032: For any record obtained according to a preset period, compare the any record with the drawing frequencies corresponding to each layer obtained within a preset time period.

[0167] In the embodiment of the present application, the drawing frequency of each layer of the target application in the drawing service is recorded once at a certain time interval to form a record.

[0168] For example, since it has been determined in the above steps S501-S502 that there is no bullet screen in the video interface of the target application, the obtained drawing frequencies corresponding to each layer of the target application in the drawing service within the preset duration are the drawing frequencies corresponding to each layer when there is no bullet screen in the video interface of the target application. Therefore, it is possible to determine whether there is a change in the drawing frequencies corresponding to each layer by comparing any record obtained according to the preset period and the drawing frequencies corresponding to each layer obtained within the preset duration.

[0169] S5033. For any layer in the drawing service, when the difference between the drawing frequency corresponding to any layer in any record and that obtained within a preset time period is greater than or equal to a threshold, it is determined that the drawing frequencies corresponding to the layers of the target application in the drawing service have changed; or, when the difference between the drawing frequency corresponding to any layer in any record and that obtained within a preset time period is less than a threshold, it is determined that the drawing frequencies corresponding to the layers of the target application in the drawing service have not changed.

[0170] In the embodiment of the present application, any layer can be understood as any layer in one or more layers when the number of layers of the target application in the drawing service is one or more layers, for example, any layer in the video layer and the bullet screen layer.

[0171] It should be noted that the threshold can be set according to actual conditions or experience. For example, the threshold can be set to 10 Hz. The embodiment of the present application does not make any specific limitation on the specific value of the threshold.

[0172] In one possible implementation, when there is no sudden barrage during video playback, the layers of the target application in the drawing service include a video layer. For the video layer, the difference between any record of the video layer and the drawing frequency corresponding to the video layer obtained within a preset time length is less than a threshold, so that it can be determined that the drawing frequencies corresponding to the layers of the target application in the drawing service have not changed.

[0173] In another possible implementation, when there is a sudden barrage during video playback, and the video drawing and the barrage drawing are drawn in the same layer, then for this layer, when a sudden barrage appears during video playback, the difference between the corresponding drawing frequency of this layer within the preset period and the corresponding drawing frequency of this layer obtained within the preset duration is greater than or equal to the threshold, so that it can be determined that the corresponding drawing frequencies of each layer in the drawing service of the target application have changed.

[0174] In another possible implementation, when there is a sudden barrage during video playback, and the video drawing and the barrage drawing are drawn in different layers, before the sudden barrage appears during video playback, the layers of the target application in the drawing service include the video layer, and when the sudden barrage appears during video playback, the layers of the target application in the drawing service include the video layer and the barrage layer. Since before the sudden barrage appears during video playback, the layers of the target application in the drawing service do not include the barrage layer, the difference between the drawing frequency corresponding to the barrage layer within the preset period and the drawing frequency corresponding to the barrage layer obtained within the preset duration is greater than or equal to the threshold, thereby determining that the drawing frequencies corresponding to the layers of the target application in the drawing service have changed.

[0175] It can also be understood that when there is a sudden barrage during video playback, the video drawing and the barrage drawing are drawn in different layers, before the sudden barrage appears during video playback, the layers of the target application in the drawing service include the video layer, and when the sudden barrage appears during video playback, the layers of the target application in the drawing service include the video layer and the barrage layer, and the number of layers of the target application in the drawing service changes, so that it can be determined that the drawing frequency corresponding to each layer of the target application in the drawing service changes.

[0176] Exemplarily, for any record obtained according to a preset period, it is also possible to determine whether the drawing frequencies corresponding to each layer of the target application in the drawing service have changed by comparing any record with the previous record. Specifically, for any layer in the drawing service, when the difference between the drawing frequencies of any layer in any record and the drawing frequencies in the previous record is greater than or equal to a threshold, it is determined that the drawing frequencies corresponding to each layer of the target application in the drawing service have changed; or, when the difference between the drawing frequencies of any layer in any record and the drawing frequencies in the previous record is less than a threshold, it is determined that the drawing frequencies corresponding to each layer of the target application in the drawing service have not changed.

[0177] In an embodiment of the present application, by calculating the difference between the drawing frequency of any layer in any record and the drawing frequency corresponding to any layer obtained within a preset time period or the drawing frequency in the previous record, it is determined whether the drawing frequencies corresponding to each layer in the drawing service of the target application have changed, thereby judging whether there are sudden bullet screens on the video interface of the target application, and the terminal device adjusts the device performance to improve the user experience.

[0178] S504. When it is detected that the drawing frequencies corresponding to each layer of the target application in the drawing service have changed, it is determined that there are bullet screens in the video interface of the target application; or, when it is detected that the drawing frequencies corresponding to each layer of the target application in the drawing service have not changed, it is determined that there are no bullet screens in the video interface of the target application.

[0179] In the embodiment of the present application, the barrage may be a burst barrage. It should be noted that, whether it is a continuous barrage or a burst barrage, the terminal device processes it as if the barrage exists on the video interface of the target application.

[0180] Exemplarily, when it is determined that barrage exists in the video interface of the target application, monitoring may be stopped.

[0181] In an embodiment of the present application, by monitoring whether the drawing frequencies corresponding to each layer of the target application in the drawing service change, it is determined whether there is a sudden barrage in the video interface of the target application. This is simple and intuitive, and the terminal device can adjust the device performance according to whether there is a barrage in the video interface of the target application, which can improve the user experience.

[0182] S505: When it is determined that there is a bullet screen on the video interface of the target application, increase the screen refresh rate; or, when it is determined that there is no bullet screen on the video interface of the target application, reduce the screen refresh rate.

[0183] In the embodiment of the present application, the screen refresh rate can be understood as the number of times the screen is refreshed per second. The higher the screen refresh rate, the higher the frame rate it can support. For example, if the screen refresh rate of a terminal device is 120Hz, it can support both a frame rate of 60FPS and a frame rate of 90FPS, and the maximum frame rate it can support is 120FPS.

[0184] Exemplarily, the terminal device can adjust the device performance according to whether there is a barrage on the video interface of the target application. Take the screen refresh rate as an example. For example, when the barrage function is turned on during video playback, the played video only needs a refresh rate of 30Hz to meet the demand for smooth display of the picture; when the barrage function is turned on during video playback, the demand for refresh rate will be higher than the scene without the barrage. Therefore, the terminal device can adjust the screen refresh rate according to whether there is a barrage. When the terminal device recognizes the presence of a barrage, the screen refresh rate can be increased accordingly to reduce the freeze of the barrage scrolling screen; and when the terminal device recognizes that there is no barrage, the screen refresh rate can be lowered accordingly to reduce the processor load and power consumption, thereby improving the user experience.

[0185] The barrage recognition method provided in the embodiment of the present application is that the terminal device obtains the video frame rate of the target application and the drawing frequency corresponding to each layer of the target application in the drawing service within a preset time; based on the video frame rate and the drawing frequency corresponding to each layer, it is determined whether there is a barrage on the video interface of the target application; when it is determined that there is no barrage on the video interface of the target application, it is monitored whether the drawing frequency corresponding to each layer of the target application in the drawing service changes; when it is monitored that there is a change in the drawing frequency corresponding to each layer of the target application in the drawing service, it is determined that there is a barrage in the video interface of the target application, or when it is monitored that there is no change in the drawing frequency corresponding to each layer of the target application in the drawing service, it is determined that there is no barrage in the video interface of the target application. The terminal device can adjust the screen refresh rate according to whether there is a barrage to improve the user experience.

[0186] The embodiments described in the above exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of methods consistent with some aspects of the present application as detailed in the appended claims.

[0187] The above has described the bullet screen recognition method of the embodiment of the present application, and the following describes the terminal device for executing the above bullet screen recognition method provided by the embodiment of the present application. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the terminal device provided by the embodiment of the present application can execute the steps in the above bullet screen recognition method.

[0188] Figure 6 FIG. 1 shows a schematic diagram of the structure of a bullet screen recognition device provided by an embodiment of the present application. Figure 6 As shown, the bullet screen recognition device can be a terminal device in an embodiment of the present application. The bullet screen recognition device includes: a touch screen 601, wherein the touch screen includes a touch panel and a display screen; one or more processors 602; a memory 603; multiple applications; and one or more computer programs, wherein one or more computer programs are stored in the memory 603, and one or more computer programs include instructions. When the instructions are executed by the bullet screen recognition device, the bullet screen recognition device performs the following steps: obtain the video frame rate of the target application, and the drawing frequency corresponding to each layer of the target application in the drawing service within a preset time length; according to the video frame rate and the drawing frequency corresponding to each layer, determine whether there is a bullet screen on the video interface of the target application.

[0189] Optionally, judging whether barrage exists in the video interface of the target application based on the video frame rate and the drawing frequencies corresponding to each layer, including: calculating a first number of drawing frame rates that are greater than the video frame rate among the drawing frequencies corresponding to each layer, and when the first number is greater than 0, determining that barrage exists in the video interface of the target application.

[0190] Optionally, the drawing frequency corresponding to each layer is recorded as a correspondence between the layer identifier and the drawing frequency corresponding to the layer identifier. The drawing frequency corresponding to any layer identifier is a ratio of the number of drawings to a preset duration. The number of drawings is the number of times any layer identifier is drawn within the preset duration. The number of layers of the target application in the drawing service is one or more layers.

[0191] Optionally, after determining whether there is barrage in the video interface of the target application based on the video frame rate and the drawing frequencies corresponding to each layer, the method further includes: when it is determined that there is no barrage in the video interface of the target application, monitoring whether the drawing frequencies corresponding to each layer of the target application in the drawing service change; when it is monitored that there is a change in the drawing frequencies corresponding to each layer of the target application in the drawing service, determining that there is barrage in the video interface of the target application.

[0192] Optionally, monitoring whether the drawing frequencies corresponding to each layer of the target application in the drawing service have changed includes: recording the drawing frequencies of each layer of the target application in the drawing service according to a preset period; for any record obtained according to the preset period, comparing any record with the drawing frequencies corresponding to each layer obtained within a preset time period; for any layer in the drawing service, when the difference between the drawing frequencies corresponding to any layer in any record and any layer obtained within a preset time period is greater than or equal to a threshold, determining that the drawing frequencies corresponding to each layer of the target application in the drawing service have changed; or, when the difference between the drawing frequencies corresponding to any layer in any record and any layer obtained within a preset time period is less than a threshold, determining that the drawing frequencies corresponding to each layer of the target application in the drawing service have not changed.

[0193] Optionally, the video frame rate of the target application and the corresponding drawing frequencies of each layer of the target application in the drawing service within a preset duration are obtained, including: when the target application starts video playback, the video in the target application is displayed in full screen, the terminal device displaying the video interface of the target application switches between horizontal and vertical screen states, and / or when the video interface of the target application receives a user operation, the video frame rate of the target application and the corresponding drawing frequencies of each layer of the target application in the drawing service within a preset duration are obtained.

[0194] Optionally, obtaining the video frame rate of the target application includes: obtaining the video frame rate of the target application in the media codec MediaCodec.

[0195] Optionally, the method further includes: when it is determined that there are bullet screens on the video interface of the target application, increasing the screen refresh rate; or, when it is determined that there are no bullet screens on the video interface of the target application, reducing the screen refresh rate.

[0196] Figure 7 A hardware structure diagram of a bullet screen recognition device provided in an embodiment of the present application. Figure 7 , the device includes: a memory 701, a processor 702 and a display screen 704. The device may also include an interface circuit 703, wherein the memory 701, the processor 702, the interface circuit 703 and the display screen 704 can communicate; illustratively, the memory 701, the processor 702, the interface circuit 703 and the display screen 704 can communicate through a communication bus, the memory 701 is used to store computer execution instructions, the execution is controlled by the processor 702, and the display screen 704 is used to perform display, so as to realize the bullet screen recognition method provided in the following embodiments of the present application.

[0197] In possible implementations, the computer-executable instructions in the embodiments of the present application may also be referred to as application code, which is not specifically limited in the embodiments of the present application.

[0198] Optionally, the interface circuit 703 may also include a transmitter and / or a receiver. Optionally, the processor 702 may include one or more CPUs, or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0199] The device of this embodiment can be used to execute the steps executed in the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0200] The present application embodiment also provides a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. For example, the available medium may include a magnetic medium (e.g., a floppy disk, a hard disk or a tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0201] The present application embodiment also provides a computer-readable storage medium. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium that can be accessed by a computer.

[0202] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that is intended to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disks and optical disks as used herein include optical disks, laser disks, optical disks, digital versatile disks (DVD), floppy disks and Blu-ray disks, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0203] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A screen refresh method, applied to an electronic device, characterized in that: include: The target application plays a video, wherein the video playback interface of the target application playing the video includes a first control, wherein the first control is used to turn on or off the bullet screen function, and the target application includes: a video application, a live broadcast application, or a music application that can play music clips; An operation of enabling the barrage function is received, and the barrage function of the target application is enabled; The electronic device is switched from a horizontal screen to a vertical screen or from a vertical screen to a horizontal screen; In a case where at least one drawing frequency corresponding to one or more layers of the target application in the drawing service is greater than a video frame rate of the video, increasing a screen refresh rate; In a case where no drawing frequency among at least one drawing frequency corresponding to one or more layers of the target application in the drawing service is greater than the video frame rate of the video, the screen refresh rate is reduced.

2. The method according to claim 1, characterized in that After receiving the operation of starting the barrage function, it includes: displaying the barrage on the video playback interface.

3. The method according to claim 1 or 2, characterized in that: When bullet screen is displayed on the video playback interface, the screen refresh rate is 60 Hz.

4. The method according to claim 1 or 2, characterized in that: When the barrage function is turned off, the screen refresh rate is 30 Hz.

5. The method according to claim 1, characterized in that The video includes: live video, recorded video, TV series or movie.

6. The method according to claim 1, characterized in that The target application plays a video, including: Display the video playback interface of the target application in full screen.

7. The method according to claim 1, characterized in that At least one drawing frequency corresponding to one or more layers of the target application in the drawing service is greater than the video frame rate of the video, including: At least one of the drawing frequencies of one or more layers of the target application in the drawing service has a drawing frequency that is greater than the video frame rate of the video.

8. The method according to claim 7, characterized in that The drawing frequencies corresponding to the one or more layers are recorded as the corresponding relationship between the layer identifier and the drawing frequency corresponding to the layer identifier. The drawing frequency corresponding to any layer identifier is the ratio of the number of drawings to the preset time length. The number of drawings is the number of times any layer identifier is drawn within the preset time length.

9. The method according to claim 1, characterized in that: The method further comprises: When no drawing frequency among at least one drawing frequency corresponding to one or more layers of the target application in the drawing service is greater than the video frame rate of the video, recording the drawing frequency of one or more layers of the target application in the drawing service according to a preset period; For any record obtained according to the preset period, compare the drawing frequency corresponding to the record and one or more layers obtained within a preset time period; For any layer in the drawing service, when the difference between any record of the layer and the drawing frequency corresponding to the layer obtained within the preset time period is greater than or equal to a threshold, increasing the screen refresh rate; Alternatively, when the difference between the drawing frequency corresponding to any layer in any record and that obtained within the preset time period is less than the threshold, the screen refresh rate is increased.

10. The method according to claim 1, characterized in that In the case where at least one drawing frequency corresponding to one or more layers of the target application in the drawing service is greater than the video frame rate of the video, before increasing the screen refresh rate, the method includes: A video frame rate of the video and at least one drawing frequency corresponding to one or more layers of the target application in a drawing service are obtained.

11. The method according to any one of claims 1, 7, 9 and 10, characterized in that: The drawing service is used to draw, render and / or synthesize the video and / or bullet screen in the target application.

12. The method according to any one of claims 1, 7, 9 and 10, characterized in that: The video frame rate is the frame rate at which the video played in the target application is output to the display screen through the graphics card.

13. The method according to any one of claims 1, 7, 9 and 10, characterized in that: The video frame rate of the video is the video frame rate of the target application in the media codec Media Codec.

14. The method according to claim 13, characterized in that The Media Codec is used to decode the video to obtain the video frame rate.

15. The method according to claim 2, characterized in that The display barrage includes: The barrage is displayed in a scrolling manner, and the barrage is a commentary subtitle.

16. The method according to any one of claims 1, 6-7, 9-10, characterized in that: The target application includes a drawing of the target application, and the drawing of the target application includes a video drawing and / or a bullet screen drawing.

17. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to call a program in the memory so that the electronic device executes the method according to any one of claims 1 to 16.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 16 is implemented.

Images