Display device and display method

Abstract

The invention relates to the technical field of display equipment, and provides display equipment and a display method.The display equipment comprises a displayer and a controller, the controller comprises a graphics processing unit and a central processing unit, and the central processing unit is configured to respond to a scrolling instruction input by a user to obtain first consumed time of an Nth frame in a UI drawing thread; under the condition that the first consumed time is greater than first preset time, calling the graphics processor to detect a background primitive of an Nth frame; obtaining a background primitive of an Nth frame detected by a graphics processor, and marking the position of an animation primitive on the background primitive of the Nth frame to obtain a background rendering graph of the Nth frame; according to the technical scheme, by adding a mechanism that the central processing unit draws the background rendering image, the bandwidth consumption of the graphics processor is reduced, the image drawing time of the graphics processor is shortened, and the possibility that the graphics processor causes frame dropping is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a display method. Background Art

[0002] In order to improve the user experience of using display devices, the current display devices are getting larger and larger. At the same time, in order to provide users with better visual effects, the on-screen display (OSD) adjustment method of display devices is also developing from 1080p to 4K user interface (UI).

[0003] Display devices usually need to execute drawing threads, rendering threads, and synthesis threads to display images. For example, the drawing thread includes binding data and triggering animations. The rendering thread includes: display list analysis, recording graphics processor rendering primitives, generating graphics processor rendering instructions, and waiting for the graphics processor to complete rendering. The synthesis thread includes vertex processing and pixel processing.

[0004] When the user controls the scrolling of the display interface to dynamically refresh the user interface screen, the controller obtains the display primitive information of the display screen, and based on the display primitive information, executes the drawing thread, rendering thread and synthesis thread of the display screen. Among them, the reasons for the frame drop phenomenon in the display interface mainly appear in the drawing thread and rendering thread. For example, when the process of binding data of the current frame takes a long time, it is very easy to cause the rendering thread of the current frame to time out, which in turn causes the picture corresponding to the current frame on the display interface to drop frames. Summary of the invention

[0005] Embodiments of the present application provide a display device and a display method to reduce the probability of frame drops on a display interface when a user scrolls the display interface.

[0006] The present application provides a display device, including:

[0007] a display configured to display a user interface;

[0008] A controller, coupled to the display, includes:

[0009] The graphics processor is configured to: detect a background primitive of the Nth frame;

[0010] The CPU is configured as:

[0011] In response to a scroll instruction input by a user, obtaining a first time consumption of the Nth frame in the UI drawing thread, where the Nth frame is a picture displayed on the user interface in response to the scroll instruction, and N is a positive integer greater than or equal to 1;

[0012] In the case where the first time consumption is greater than the first preset time, calling the graphics processor to detect the background image element of the Nth frame;

[0013] Obtaining the background primitive of the Nth frame detected by the graphics processor, and marking the position of the animation primitive on the background primitive of the Nth frame to obtain a background rendering image of the Nth frame;

[0014] Send the background rendering of frame N to the graphics processor.

[0015] In this way, when the first time consumption is greater than the first preset time, the central processor renders the background image and submits the background rendering image to the graphics processor, which completes the layer synthesis of the display screen. In this way, by adding a mechanism for the central processor to draw the background rendering image, the bandwidth consumption of the graphics processor is reduced, and the image drawing time of the graphics processor is shortened, thereby reducing the possibility of the graphics processor causing frame drops.

[0016] In one implementation, after obtaining the first time consumption of the Nth frame in the UI drawing thread, the central processing unit is further configured to:

[0017] When the first time consumption is less than or equal to the first preset time, obtaining the second time consumption of the Nth frame in the rendering thread;

[0018] When the second time consumption is greater than the second preset time, obtaining the background image element of the N+1th frame;

[0019] Mark the position of the animation primitive on the background primitive of the N+1th frame to obtain the background rendering of the N+1th frame;

[0020] Send the background rendering image of frame N+1 to the graphics processor.

[0021] In this way, when the first time consumption of the Nth frame is not greater than the first preset time, the central processor renders the background image of the N+1th frame, and submits the background rendering image of the N+1th frame to the graphics processor, and the graphics processor completes the layer synthesis of the display screen of the N+1th frame. In this way, when the first time consumption of the Nth frame is not greater than the first preset time, by adding a mechanism for the central processor to draw the background rendering image of the N+1th frame, continuous frame drops in the display screen can be avoided as much as possible.

[0022] In one implementation, before obtaining the first time consumption of the Nth frame in the UI drawing thread in response to the scroll instruction input by the user, the central processor is further configured to:

[0023] When the display manager triggers the Nth frame, the UI drawing thread time, rendering thread time and total time of the composite frame of the Nth frame are recorded;

[0024] When the total time consumption of the synthesized frame is less than or equal to the third preset time, the corresponding frame is marked as a normal frame;

[0025] Determine a first preset time according to the time consumption of the UI drawing thread of all normal frames;

[0026] The second preset time is determined according to the time consumed by the rendering threads of all normal frames.

[0027] In this way, by counting the time consumed by each link of a normal frame, the first preset time and the second preset time are determined in advance, so that when a frame timeout is determined, it is determined whether the timed-out link is in the UI drawing thread or the rendering thread.

[0028] In one implementation, after recording the UI drawing thread time, the rendering thread time and the total time of the synthesized frame for each frame, the central processor is further configured to:

[0029] Marking a response frame whose total time of the synthesized frame is greater than a third preset time as an abnormal frame;

[0030] The picture sequences where abnormal frames and normal frames appear are counted, and according to the picture sequences of abnormal frames and normal frames, the key picture sequence and the frame drop stage where the abnormal frames appear are determined, and the frame drop stage includes the UI drawing thread and / or the rendering thread.

[0031] In this way, by pre-analyzing the frame drop phenomenon of the display interface, a key picture series where abnormal frames appear in the display screen can be determined, that is, pictures where the frame drop phenomenon is likely to occur in the display screen can be determined.

[0032] In one implementation, before obtaining the first time consumption of the Nth frame in the UI drawing thread in response to the scroll instruction input by the user, the central processor is further configured to:

[0033] Get the screen refresh rate of the user interface;

[0034] The third preset time is determined according to the screen refresh rate of the user interface.

[0035] In this way, the third preset time is determined in advance through the screen refresh rate, so as to facilitate the determination of abnormal frames, and further facilitate the determination of the key picture sequence in which the abnormal frames appear.

[0036] In one implementation, the central processing unit is further configured to:

[0037] In response to a scroll instruction input by a user, acquiring a picture sequence corresponding to the scroll instruction;

[0038] When there is a key picture sequence in the picture sequence corresponding to the scroll instruction, and the frame drop phase corresponding to the key picture sequence includes a rendering thread, obtaining background picture elements of the video frame corresponding to the key picture sequence;

[0039] Marking the position of the animation primitive on the background primitive of the video frame corresponding to the key picture sequence to obtain a background picture rendering of the video frame corresponding to the key picture sequence;

[0040] And send the background image rendering of the video frame corresponding to the key picture sequence to the graphics processor.

[0041] In this way, the background image rendering of the key picture sequence is realized by the central processing unit, thereby reducing the bandwidth consumption of the graphics processor and reducing the probability of frame drop in the key picture sequence.

[0042] In one implementation, in the step of calling a graphics processor to detect background primitives of the Nth frame, the graphics processor is configured to:

[0043] Get the rendering primitive information of the Nth frame;

[0044] Determine the display content of the graphics element of the Nth frame according to the rendering graphics element information of the Nth frame;

[0045] The graphic element display content of the N-1th frame is obtained, and the background graphic element of the Nth frame is determined according to the graphic element display content of the Nth frame and the graphic element display content of the N-1th frame.

[0046] In this way, the central processor draws the background rendering image, thereby reducing the bandwidth consumption of the graphics processor and shortening the image drawing time of the graphics processor, thereby reducing the possibility of the graphics processor causing frame drops.

[0047] In one implementation, when the first time consumption is greater than the first preset time, the central processing unit is further configured to:

[0048] Get the CPU load list;

[0049] According to the central processor, a non-display process irrelevant to the screen display control of the display interface is determined, and the non-display process is cleared.

[0050] The embodiment of the present application further provides a display method, which is applied to a display device, wherein the display device includes a display and a controller, and the display is configured to display a user interface; the display method includes:

[0051] In response to a scroll instruction input by the user, obtaining the first time consumption of the Nth frame in the UI drawing thread;

[0052] In the case where the first time consumption is greater than the first preset time, calling the graphics processor to detect the background image element of the Nth frame;

[0053] Obtaining the background primitive of the Nth frame detected by the graphics processor, marking the position of the animation primitive on the background primitive of the Nth frame to obtain the background rendering image of the Nth frame;

[0054] Send the background rendering of frame N to the graphics processor.

[0055] In this way, when the first time consumption is greater than the first preset time, the graphics processor is prevented from implementing background rendering, thereby reducing the bandwidth consumption of the graphics processor and shortening the image drawing time of the graphics processor, thereby reducing the possibility of the graphics processor causing frame drops.

[0056] The present application also provides a display device, including:

[0057] An acquisition module, configured to acquire, in response to a scroll instruction input by a user, a first time consumption of the Nth frame in the UI drawing thread, where the Nth frame is a picture displayed on the user interface in response to the scroll instruction, and N is a positive integer greater than or equal to 1;

[0058] A calling module, configured to call a graphics processor to detect a background image element of the Nth frame when the first time consumption is greater than a first preset time;

[0059] A rendering module, used for obtaining background primitives of the Nth frame detected by the graphics processor, and for marking positions of animation primitives on the background primitives of the Nth frame to obtain a background rendering of the Nth frame;

[0060] The transmission module is used to send the background rendering image of the Nth frame to the graphics processor.

[0061] In this way, when the first time consumption is greater than the first preset time, the graphics processor is prevented from implementing background rendering, thereby reducing the bandwidth consumption of the graphics processor and shortening the image drawing time of the graphics processor, thereby reducing the possibility of the graphics processor causing frame drops. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0064] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0065] Figure 3 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0066] Figure 4 Schematic diagram of the display method provided in some embodiments of the present application Figure 1 ;

[0067] Figure 5 Schematic diagram of input method of scrolling instruction provided in some embodiments of the present application Figure 1 ;

[0068] Figure 6 Schematic diagram of input method of scrolling instruction provided in some embodiments of the present application Figure 2 ;

[0069] Figure 7 A schematic diagram of a first time-consuming form provided for some embodiments of the present application;

[0070] Figure 8 Schematic diagram of the display method provided in some embodiments of the present application Figure 2 ;

[0071] Fig. 9 Schematic diagram of the display method provided in some embodiments of the present application Figure 3 ;

[0072] Fig.10 Schematic diagram of the display method provided in some embodiments of the present application Figure 4 ;

[0073] Fig.11 Schematic diagram of the display method provided in some embodiments of the present application Figure 5 . DETAILED DESCRIPTION

[0074] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0075] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0076] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0077] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0078] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0079] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0080] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 is used to receive operation instructions input by the user and convert the operation instructions into control instructions that can be recognized and responded to by the display device 200. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0081] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0082] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application program for controlling the display device 200 .

[0083] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0084] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0085] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0086] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0087] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0088] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0089] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0090] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0091] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0092] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0093] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0094] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0095] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0096] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface. For example, the operating system can directly control the display device to provide a user interface, or provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0097] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0098] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 3As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the system library layer and the kernel layer.

[0099] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0100] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0101] like Figure 3 As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0102] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.

[0103] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0104] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 3 As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver (Universal Serial Bus: USB), HDMI driver (High Definition Multimedia Interface: HDMI), sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.) and power driver, etc.

[0105] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0106] The display device provided in the embodiment of the present application includes a display and a controller, wherein the display is configured to display a user interface; the controller is coupled to the display and includes: a graphics processor and a central processing unit, wherein the graphics processor is configured to detect a background image element of the Nth frame, and Figure 4 As shown, the CPU is configured to execute S410 to S440.

[0107] S410 , in response to a scroll instruction input by a user, obtaining a first time consumption of the Nth frame in the UI drawing thread.

[0108] The Nth frame is a picture displayed on the user interface in response to the scroll instruction, and N is a positive integer greater than or equal to 1.

[0109] like Figure 5As shown, the scrolling instruction can be implemented by the user sliding on the screen of the display. In this way, the display is provided with a touch component, and the touch component is coupled to the display, and the touch component is configured to obtain the scrolling instruction triggered by the user on the user interface.

[0110] Specifically, the touch component relies on sensors to obtain scrolling instructions input by the user. For example, the touch component uses resistance, electromagnetic induction, infrared technology or ultrasonic technology to detect the position of a stylus or finger on the display, and converts these positions into electronic signals, and finally into coordinate points on the user interface.

[0111] Among them, resistive technology applies voltage on the conductive film. When an object (such as a finger or a pen tip) touches the conductive film, it will change the flow of current at the contact point, and then change the voltage value in the horizontal and vertical directions of the conductive film. The voltage values ​​read in the horizontal and vertical directions can be used to calculate the coordinates of the contact point on the user interface.

[0112] Electromagnetic induction technology uses a writing pen that can emit electromagnetic waves and a receiving coil film arranged horizontally and vertically on a screen that can display a user interface. The writing pen emits electromagnetic waves in a specific way. When the writing pen is close to the receiving coil, the electromagnetic waves are induced on the coil. By calculating the induced electromotive force, the coordinate point on the user interface where the writing pen is located can be obtained.

[0113] Infrared technology forms a scanning grid through infrared receiving and transmitting tubes spread around the display area of ​​the monitor. When an object blocks the infrared light, the coordinates of the blocking object in the display area can be determined through the horizontal and vertical positions of the blocked infrared rays, thereby obtaining the coordinate point on the user interface where the object contacts the display area.

[0114] Ultrasonic technology uses the slow transmission speed of ultrasonic waves. Two ultrasonic receiving devices are set on one side of the screen, and the positioning writing pen is used as an ultrasonic transmitter. When the writing pen moves on the screen surface, the emitted ultrasonic waves are detected by the receiver. The distance between the writing pen and the receiver can be calculated based on the time when the ultrasonic waves are received. The position coordinates of the writing pen are calculated through the three-point positioning principle, thereby determining the coordinate point of the writing pen on the user interface.

[0115] Among them, Figure 6 As shown, the scrolling instruction can be input by the user through the remote controller, for example, by triggering the up direction key or the down direction key of the remote controller.

[0116] After receiving the scrolling instruction input by the user, the controller Figure 8As shown, when the display manager triggers the start of each frame, a unique number is created for each frame, and then the binding data time, animation triggering time, rendering instruction generation time, instruction submission time and graphics processor rendering completion time of each frame are recorded in the order of numbers, so as to calculate the first time consumption of the UI drawing thread and the second time consumption of the drawing thread. Among them, the UI drawing thread includes binding data and triggering animation, and the first time consumption can be a duration value, that is, it represents the duration of the entire process of the UI drawing thread. The first time consumption can also be a set of multiple duration values, that is, including the duration values ​​of each link in the UI drawing thread.

[0117] S420: When the first time consumption is greater than the first preset time, call a graphics processor to detect background graphics elements of the Nth frame.

[0118] Similar to the setting of the first time consumption, when the first time consumption is a duration value, the first preset time is also a duration value. Figure 7 As shown, in the case where the first time consumption is a set including multiple duration values, correspondingly, the first preset time is also a set including multiple duration values.

[0119] Wherein, when the first time consumption is a set including multiple duration values, when any duration value in the first time consumption is greater than the corresponding duration value in the first preset time, the first time consumption is determined to be greater than the first preset time. For example, if the duration of the binding data is greater than the corresponding duration in the first preset time, the binding data is determined to have timed out, and then the UI drawing thread is determined to have timed out.

[0120] When the first time consuming is greater than the first preset time, for example, Figure 8 As shown, the binding data times out, and after generating the image processor rendering instruction, the graphics processor is called to detect the background primitive of the Nth frame. Specifically, as Fig. 9 As shown, the graphics processor is configured to execute S910 to S930.

[0121] S910, obtaining rendering primitive information of the Nth frame.

[0122] In the rendering thread, after the graphics processor renders the primitive information of the Nth frame, it records the display content of the primitive of the Nth frame and the position of the actual content. For example, the display content includes a picture, text or a filling block.

[0123] S920: Determine the primitive display content of the Nth frame according to the rendering primitive information of the Nth frame.

[0124] After obtaining the rendering primitive information of the Nth frame, the picture, text or filling block included in the display content can be determined according to the rendering primitive information of the Nth frame.

[0125] S930, acquiring the graphic element display content of the N-1th frame, and determining the background graphic element of the Nth frame according to the graphic element display content of the Nth frame and the graphic element display content of the N-1th frame.

[0126] After determining the display content of the image element of two consecutive frames, the positions of the same display areas and contents of the two consecutive frames are compared to obtain the analyzed image element results, which include static background image elements and dynamic image elements, and then obtain the background image element of the Nth frame.

[0127] S430, obtaining the background image primitives of the Nth frame detected by the graphics processor, marking the positions of the animation image primitives on the background image primitives of the Nth frame, so as to obtain a background image rendering of the Nth frame.

[0128] The central processing unit calculates the animation movement direction according to the rendering primitive information, and calculates the position of the animation primitive, thereby rendering the dynamic foreground primitive of the background primitive and obtaining the background image rendering image of the Nth frame.

[0129] Specifically, first, obtain the element position information of the animation image. The position information can be determined based on the coordinate system where the animation element is located. For example, the coordinate system takes the upper left corner of the screen of the user interface as the origin, the right is the positive direction of the x-axis, and the downward is the positive direction of the y-axis. Then, determine the rendering relationship and hierarchical relationship between the animation element and the background element, and determine the front and back hierarchical relationship of the background element relative to the animation element according to the position of the animation element. If the animation element is in the foreground, the background element should be behind it. For example, in a racing game, the car (animation element) is above the runway (background element), and the runway needs to be rendered behind the car. Finally, render the background element, for example, including cropping and optimization, and crop the background element according to the visible range of the animation element. If part of the background element is outside the visible range of the animation element, this part can be not rendered to improve rendering efficiency. In addition, it can also include color, lighting adjustment and texture mapping, etc., which will not be repeated here.

[0130] S440, sending the background image rendering of the Nth frame to the graphics processor.

[0131] After the background rendering of the Nth frame is sent to the image processor, the graphics processor completes the layer synthesis of the display screen.

[0132] In some examples, after the graphics processor completes layer synthesis of the display picture, the graphics processor may drive the user interface to display the Nth frame of the picture.

[0133] In some examples, after the graphics processor completes layer synthesis of the display screen, the central processor may also obtain the Nth frame of the screen synthesized by the graphics processor based on the background image rendering of the Nth frame, and drive the user interface to display the Nth frame of the screen.

[0134] In the conventional rendering process, the background primitives are rendered by the graphics processor. However, when the first time consumption is greater than the first preset time, if the background primitives are still rendered by the graphics processor, it will cause the Nth frame to drop. In the technical solution of the present application, when the first time consumption is greater than the first preset time, the central processing unit renders the background image, and submits the background rendering image to the graphics processor, and the graphics processor completes the layer synthesis of the display image. In this way, by adding a mechanism for the central processing unit to draw the background rendering image, the bandwidth consumption of the graphics processor is reduced, and the image drawing time of the graphics processor is shortened, thereby reducing the possibility of the graphics processor causing frame drops.

[0135] When the UI drawing thread of the current frame takes a long time, it is very easy to cause the rendering thread of the current frame to time out. When the rendering thread of the current frame takes a long time, it will not only cause the picture corresponding to the current frame to drop frames, but also easily cause the picture corresponding to the next frame to drop frames. To solve this problem, in one possible implementation, Fig.10 As shown, after obtaining the first time consumption of the UI drawing thread for the Nth frame, the central processing unit is further configured to execute: S1010 to S1040.

[0136] S1010, when the first time consumption of the Nth frame is less than or equal to the first preset time, obtain the second time consumption of the Nth frame in the rendering thread.

[0137] If the first duration is not greater than the first preset time, the second duration of the Nth frame in the rendering thread is. Similar to the first duration, the second duration can be a duration value or a set of multiple duration values. Fig.11 As shown, the second time consumption is a set including multiple duration values, and the set includes the duration of each stage of the drawing thread.

[0138] S1020: When the second time consumption of the Nth frame is greater than the second preset time, obtain the background image element of the N+1th frame.

[0139] When the second time consumption of the Nth frame is greater than the second preset time, it indicates that the graphics processor has a rendering timeout. In this case, if the graphics processor continues to complete the subsequent picture rendering thread, it will cause more serious frame drops in the subsequent pictures.

[0140] In view of this situation, the central processing unit detects and obtains the background primitive of the N+1th frame. For example, the central processing unit obtains the rendering primitive information of the N+1th frame, and determines the primitive display content of the N+1th frame according to the rendering primitive information of the N+1th frame; and obtains the primitive display content of the Nth frame, and determines the background primitive of the N+1th frame according to the primitive display content of the N+1th frame and the primitive display content of the Nth frame.

[0141] Compared with the case where the UI drawing thread times out, when the rendering thread times out, the graphics processor has already started rendering the current frame. In this case, it is not appropriate to interrupt the graphics processor's rendering of the current frame, and considering that the frame drop caused by the graphics processor is likely to cause the next frame to also drop, the central processor obtains the background primitives of the N+1th frame, thereby further reducing the bandwidth consumption of the graphics processor.

[0142] S1030, marking the position of the animation primitive on the background primitive of the N+1th frame to obtain a background rendering image of the N+1th frame.

[0143] The central processing unit calculates the animation motion direction based on the rendered graphics element of the N+1th frame, and calculates the position of the animation graphics element, thereby rendering the dynamic foreground graphics element of the background graphics element.

[0144] S1040, sending the background rendering image of the N+1th frame to the graphics processor.

[0145] After the background rendering of the N+1th frame is sent to the image processor, the graphics processor completes the layer synthesis of the display screen. In this way, the bandwidth consumption of the graphics processor can be greatly reduced, thereby alleviating the phenomenon of frame drops caused by the increased load of the image processor.

[0146] In this way, when the first time consumption of the Nth frame is not greater than the first preset time, the central processor renders the background image of the N+1th frame, and submits the background rendering image of the N+1th frame to the graphics processor, and the graphics processor completes the layer synthesis of the display screen of the N+1th frame. In this way, when the first time consumption of the Nth frame is not greater than the first preset time, by adding a mechanism for the central processor to draw the background rendering image of the N+1th frame, continuous frame drops in the display screen can be avoided as much as possible.

[0147] In some embodiments, before adding a mechanism for the CPU to draw a background rendering image, it is necessary to predetermine the values ​​of the first preset time and the second preset time. In response to a scroll instruction input by a user to control the scrolling of the user interface, before obtaining the first time consumption of the UI drawing thread of the Nth frame, such as Fig.11 As shown, the CPU is further configured to execute S1110 to S1140.

[0148] S1110, when the display manager triggers the Nth frame, record the UI drawing thread time consumption, the rendering thread time consumption and the total time consumption of the synthesized frame of the Nth frame.

[0149] When the display manager triggers each frame, it creates a unique number for each frame, and then records in sequence the time of binding data, triggering animation, starting to generate rendering instructions, submitting instructions, and the time when the graphics processor completes rendering. That is, it records the UI drawing thread time, rendering thread time, and total time of the synthesized frame for the Nth frame.

[0150] S1120: Mark corresponding frames whose total time consumption of synthesizing frames is less than or equal to the third preset time as normal frames.

[0151] The first preset time is related to the screen refresh rate of the display. First, the screen refresh rate of the user interface is obtained, and then the third preset time can be determined according to the screen refresh rate of the user interface. For example, the screen refresh rate of the user interface is 60HZ, that is, the display time of each frame is 16.67ms, that is, the third preset time is 16.67ms.

[0152] After each frame is synthesized, the total time taken to synthesize the frames is used as a reference to determine whether the frame is dropped. If the total time taken to synthesize the frames is less than or equal to 16.67ms, the frame is not dropped and the current frame is marked as a normal frame.

[0153] S1130: Determine a first preset time according to the time consumed by the UI drawing threads of all normal frames.

[0154] After the normal frame is determined, the average time consumption of the UI drawing thread of the normal frame is calculated according to the time consumption of the UI drawing thread of all normal frames, and the average time consumption of the UI drawing thread of the normal frame is used as the first preset time.

[0155] S1140: Determine a second preset time according to the time consumed by the rendering threads of all normal frames.

[0156] According to the time consumption of the rendering threads of all normal frames, the average time consumption of the rendering threads of the normal frames is calculated, and the average time consumption of the rendering threads of the normal frames is used as the second preset time.

[0157] In this way, by counting the time consumed by each link of a normal frame, the first preset time and the second preset time are determined in advance, so that when a frame timeout is determined, it is determined whether the timed-out link is in the UI drawing thread or the rendering thread.

[0158] When users use display devices, there are some commonly used screens. For example, the display device is a TV, and the user often uses the home page interface. The existing home page interface is generally a waterfall flow scene. The display content on the home page interface has certain regularity. These regular display contents have some fixed screens with a high probability of frame drop.

[0159] In order to deal with this situation, in some embodiments, after the step of recording the total time of the synthetic frame of the current frame, the central processing unit is further configured to: mark the current frame as an abnormal frame when the total time of the synthetic frame of a certain frame is greater than the third preset time; count the picture sequences of the abnormal frames and normal frames, and in the actual application process, each frame has a corresponding number, and the number is set continuously, so that the picture where the abnormal frame appears can be determined, the content on the user interface, for example, under the homepage interface, different areas of the waterfall flow scene display different columns, under the control of the scrolling instruction, when the homepage interface is displayed to the popular video column, there are many abnormal frames, at this time, according to the picture sequence of the abnormal frame and the normal frame, the key picture sequence and the frame drop stage where the abnormal frame appears can be determined, that is, the corresponding position of the key picture sequence is the area where the popular video column is located. Among them, the frame drop stage includes the UI drawing thread and / or the rendering thread, that is, when the display interface scrolls to the key picture sequence, the display interface is prone to frame drop phenomenon. In this way, by pre-analyzing the frame drop phenomenon of the display interface, the key picture sequence where the abnormal frame appears in the display picture can be determined, that is, the picture where the frame drop phenomenon is prone to be determined.

[0160] In view of the above-mentioned key picture sequence being prone to frame drops, in some embodiments, after responding to a scroll instruction input by a user to control the scrolling of the user interface, the CPU is further configured to predict the possible frame drops according to the key picture sequence, and add a mechanism for the CPU to render the background image in advance.

[0161] For example, first, obtain the picture sequence corresponding to the scrolling instruction. Then, compare whether there is a key picture sequence in the picture sequence corresponding to the scrolling instruction. If there is a key picture sequence, then when the frame drop stage corresponding to the key picture sequence contains a rendering thread, obtain the background image element of the video frame corresponding to the key picture sequence. Then, mark the position of the animation image element on the background image element of the video frame corresponding to the key picture sequence to obtain the background image rendering of the video frame corresponding to the key picture sequence; finally, send the background image rendering of the video frame corresponding to the key picture sequence to the graphics processor. That is, the background image rendering of the key picture sequence is realized by the central processing unit, thereby reducing the bandwidth consumption of the graphics processor and reducing the probability of frame drop in the key picture sequence.

[0162] In order to reserve more bandwidth during the image rendering process so that the CPU can render the background image, in some embodiments of the present invention, when the first time consumption is greater than the first preset time, the CPU obtains the CPU load list; and according to the load list, determines the non-display process that is not related to the image display control, and then clears the non-display process. Of course, in the actual application process, it is not limited to clearing the non-display process when the first time consumption is greater than the first preset time, and the non-display process of the CPU can also be cleared after obtaining the scroll instruction input by the user.

[0163] The present application also provides a display method, which is applied to a display device, the display device includes a display and a controller, and the display is configured to display a user interface; the display method includes:

[0164] The central processing unit obtains the first time consumption of the Nth frame in the UI drawing thread in response to the scroll instruction input by the user.

[0165] When the first time consumption is greater than the first preset time, the CPU calls the GPU to detect the background image element of the Nth frame.

[0166] The central processing unit obtains the background primitives of the Nth frame detected by the graphics processing unit, and marks the positions of the animation primitives on the background primitives of the Nth frame to obtain the background rendering image of the Nth frame.

[0167] The CPU sends the background rendering image of the Nth frame to the GPU.

[0168] In this way, when the first time consumption is greater than the first preset time, the central processor renders the background image and submits the background rendering image to the graphics processor, which completes the layer synthesis of the display screen. In this way, by adding a mechanism for the central processor to draw the background rendering image, the bandwidth consumption of the graphics processor is reduced, and the image drawing time of the graphics processor is shortened, thereby reducing the possibility of the graphics processor causing frame drops.

[0169] The present application also provides a display device, comprising:

[0170] The acquisition module is used to obtain the first time consumption of the Nth frame in the UI drawing thread in response to the scroll instruction input by the user.

[0171] The calling module is used to call the graphics processor to detect the background image element of the Nth frame when the first time consumption is greater than the first preset time.

[0172] The rendering module is used to obtain the background primitives of the Nth frame detected by the graphics processor, and to mark the positions of the animation primitives on the background primitives of the Nth frame to obtain the background rendering image of the Nth frame.

[0173] The transmission module is used to send the background rendering image of the Nth frame to the graphics processor.

[0174] In this way, when the first time consumption is greater than the first preset time, the graphics processor is prevented from implementing background rendering, thereby reducing the bandwidth consumption of the graphics processor and shortening the image drawing time of the graphics processor, thereby reducing the possibility of the graphics processor causing frame drops.

Claims

1. A display device, characterized in that: include: a display configured to display a user interface; A controller, coupled to the display, comprising: The graphics processor is configured to: detect a background primitive of the Nth frame; The CPU is configured as: In response to a scroll instruction input by a user, obtaining a first time consumption of an Nth frame in a UI drawing thread, wherein the Nth frame is a picture displayed on a user interface in response to the scroll instruction, and N is a positive integer greater than or equal to 1; In the case where the first time consumption is greater than a first preset time, calling the graphics processor to detect background graphics elements of the Nth frame; Acquire the background primitives of the Nth frame detected by the graphics processor, and mark the positions of animation primitives on the background primitives of the Nth frame to obtain a background rendering of the Nth frame; Sending the background rendering image of the Nth frame to the graphics processor.

2. The display device according to claim 1, characterized in that After obtaining the first time consumption of the Nth frame in the UI drawing thread, the central processing unit is further configured to: When the first time consumption is less than or equal to the first preset time, obtaining the second time consumption of the Nth frame in the rendering thread; When the second time consumption is greater than the second preset time, obtaining the background image element of the N+1th frame; Marking the position of the animation primitive on the background primitive of the N+1th frame to obtain the background rendering of the N+1th frame; Sending the background rendering image of the N+1th frame to the graphics processor.

3. The display device according to claim 2, characterized in that Before obtaining the first time consumption of the Nth frame in the UI drawing thread in response to the scroll instruction input by the user, the central processor is further configured to: When the display manager triggers the Nth frame, the UI drawing thread time consumption, the rendering thread time consumption and the total time consumption of the synthesized frame of the Nth frame are recorded; Marking corresponding frames whose total time consumption of synthesized frames is less than or equal to the third preset time as normal frames; Determine the first preset time according to the time consumed by the UI drawing threads of all normal frames; The second preset time is determined according to the time consumed by the rendering threads of all normal frames.

4. The display device according to claim 3, characterized in that After recording the UI drawing thread time consumption, the rendering thread time consumption and the total time consumption of the synthesized frame of each frame, the central processor is further configured to: Marking corresponding frames whose total time of the synthesized frames is greater than the third preset time as abnormal frames; Count the picture sequences where the abnormal frame and the normal frame appear, and determine the key picture sequence and the frame drop stage where the abnormal frame appears according to the picture sequences of the abnormal frame and the normal frame, wherein the frame drop stage includes a UI drawing thread and / or a rendering thread.

5. The display device according to claim 2 or 3, characterized in that: Before obtaining the first time consumption of the Nth frame in the UI drawing thread in response to the scroll instruction input by the user, the central processor is further configured to: Obtaining a screen refresh rate of the user interface; The third preset time is determined according to a screen refresh rate of the user interface.

6. The display device according to claim 4, characterized in that The central processing unit is also configured to: In response to a scroll instruction input by a user, acquiring a picture sequence corresponding to the scroll instruction; When there is a key picture sequence in the picture sequence corresponding to the scroll instruction, and the frame drop phase corresponding to the key picture sequence includes a rendering thread, obtaining background picture elements of the video frame corresponding to the key picture sequence; Marking the position of the animation primitive on the background primitive of the video frame corresponding to the key picture sequence to obtain a background picture rendering of the video frame corresponding to the key picture sequence; And send the background image rendering image of the video frame corresponding to the key picture sequence to the graphics processor.

7. The display device according to claim 1, characterized in that In the step of calling the graphics processor to detect the background primitive of the Nth frame, the graphics processor is configured to: Obtaining rendering primitive information of the Nth frame; Determining the primitive display content of the Nth frame according to the rendering primitive information of the Nth frame; The primitive display content of the N-1th frame is obtained, and the background primitive of the Nth frame is determined according to the primitive display content of the Nth frame and the primitive display content of the N-1th frame.

8. The display device according to claim 1, characterized in that When the first time consumption is greater than a first preset time, the central processor is further configured to: Obtaining a load list of the central processing unit; According to the central processor, a non-display process irrelevant to the screen display control of the display interface is determined, and the non-display process is cleared.

9. A display method, characterized in that: The display method is applied to a display device, the display device includes a display and a controller, the display is configured to display a user interface; The display method comprises: In response to a scroll instruction input by a user, obtaining a first time consumption of an Nth frame in a UI drawing thread, wherein the Nth frame is a picture displayed on a user interface in response to the scroll instruction, and N is a positive integer greater than or equal to 1; In the case where the first time consumption is greater than a first preset time, calling the graphics processor to detect background graphics elements of the Nth frame; Acquire the background primitives of the Nth frame detected by the graphics processor, and mark the positions of animation primitives on the background primitives of the Nth frame to obtain a background rendering of the Nth frame; Sending the background rendering image of the Nth frame to the graphics processor.

10. A display device, characterized in that: include: An acquisition module, configured to acquire, in response to a scroll instruction input by a user, a first time consumption of an Nth frame in a UI drawing thread, wherein the Nth frame is a picture displayed on a user interface in response to the scroll instruction, and N is a positive integer greater than or equal to 1; A calling module, configured to call the graphics processor to detect background graphics elements of the Nth frame when the first time consumption is greater than a first preset time; A rendering module, used for acquiring the background primitives of the Nth frame detected by the graphics processor, and for marking the positions of animation primitives on the background primitives of the Nth frame to obtain a background rendering of the Nth frame; The transmission module is used to send the background rendering image of the Nth frame to the graphics processor.

Images