Application program control method, and device and computer-readable storage medium

Abstract

Provided in the embodiments of the present description are an application program control method, and a device and a computer-readable storage medium. The method can comprise: within a rendering time of the current frame data of a target application program, recording, by means of a counter, the number of writing instances of a Vulkan module writing data into a command cache region of the Vulkan module, so as to obtain the total number of writing instances of the Vulkan module writing into the command cache region within the rendering time of the current frame data; determining the current rendering pressure of the target application program on the basis of the total number of writing instances; and controlling at least one of a performance resource and a display resolution of the target application program on the basis of the current rendering pressure.

Description

Application control method, device, and computer-readable storage medium

[0001] Cross-references

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on October 16, 2023, with application number 202311333595.8 and titled “Application Control Method, Device and Computer-readable Storage Medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of application interface rendering technology, and in particular to a method, device, and computer-readable storage medium for controlling an application program. Background Art

[0004] With the advancement of hardware technology and the increasing complexity of application interfaces (such as game applications), traditional graphics application programming interfaces (APIs) can no longer meet the needs of developers. Therefore, a lower-level, more efficient, and more scalable graphics API is needed to support the display requirements of application interfaces. Against this background, Vulkan came into being.

[0005] The Vulkan graphics API has been widely adopted by game developers due to its asynchronous command submission model. This allows game applications to submit multiple commands to the GPU for processing simultaneously, without having to wait for previous commands to complete. However, this also means that game applications developed based on this API consume more GPU computing power, and the GPU itself is the largest heat source in the hardware composition of electronic devices. Therefore, when applications developed based on this graphics API run on electronic devices, they will experience poor performance issues such as poor frame rates and excessive heat.

[0006] Currently, related technologies rely on collecting the phone's temperature to release performance resources, which is energy-inefficient and results in inefficient performance release. Alternatively, they identify application threads and prioritize them to improve frame rates. This, however, places high demands on the CPU itself and results in minimal performance gains. Furthermore, these solutions are not suitable for applications developed using the Vulkan graphics API.

[0007] It can be seen that there is currently a lack of effective performance improvement solutions for applications developed using the Vulkan graphics API.

[0008] Summary of the Invention

[0009] The present invention provides a method, device and computer-readable storage medium for controlling an application program.

[0010] In a first aspect, a method for controlling an application is provided, the method comprising: recording, by a counter, the number of times a Vulkan module writes data to a command buffer of the Vulkan module during the rendering time of current frame data of a target application, to obtain a total number of times the Vulkan module writes data to the command buffer during the rendering time of the current frame data; determining a current rendering pressure of the target application based on the total number of writes; and controlling at least one of performance resources and display resolution of the target application based on the current rendering pressure.

[0011] In a second aspect, a control device for an application is provided, the device comprising: a counting module, for recording the number of times a Vulkan module writes data to a command buffer of the Vulkan module through a counter during the rendering time of current frame data of a target application, and obtaining the total number of times the Vulkan module writes data to the command buffer during the rendering time of the current frame data; a load determination module, for determining a current rendering pressure of the target application based on the total number of writes; and a performance control module, for controlling at least one of performance resources and display resolution of the target application based on the current rendering pressure.

[0012] According to a third aspect, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method described in the first aspect.

[0013] In a fourth aspect, a computer-readable storage medium is provided. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0015] FIG1 is a flow chart of a method for controlling an application program provided in one embodiment of this specification.

[0016] FIG2 is a schematic diagram of an implementation architecture of a control method for an application program provided in an embodiment of this specification.

[0017] FIG3 is a detailed flowchart of a method for controlling an application program provided by an embodiment of this specification.

[0018] FIG4 is a schematic diagram of the structure of a control device for an application program provided by an embodiment of this specification.

[0019] FIG5 is a schematic structural diagram of an electronic device according to an embodiment of the present specification. DETAILED DESCRIPTION

[0020] In order to help those skilled in the art better understand the technical solutions in the embodiments of this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0021] The terms "first," "second," and the like in this specification and claims are used to distinguish similar objects and are not intended to describe a particular order or precedence. It should be understood that such terms are interchangeable where appropriate, so that the embodiments of this specification can be implemented in orders other than those illustrated or described herein. Furthermore, the term "and / or" in this specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the connected objects are in an "or" relationship.

[0022] In the past, graphics APIs were primarily based on fixed-function pipelines, which limited the flexibility and performance optimization of graphics rendering. With advancements in hardware technology and the increasing complexity of graphics applications, traditional graphics APIs are no longer able to meet the needs of developers. Consequently, a lower-level, more efficient, and more extensible graphics API is needed to support the demands of modern graphics applications. Against this backdrop, Vulkan emerged. Reflecting the shortcomings of traditional graphics APIs and recognizing the needs of modern graphics applications, and the need for a lower-level, more efficient, and more extensible graphics API, Vulkan has become a vital tool for modern graphics application development. Consequently, a growing number of manufacturers are developing graphics applications based on the Vulkan graphics API. However, mainstream manufacturers currently either rely on temperature monitoring of electronic devices to free up performance resources, which is energy-inefficient and results in inadequate performance. Alternatively, they identify application threads and prioritize them to improve frame rates, which places high demands on the CPU and results in insignificant performance gains. Furthermore, these solutions are not suitable for applications developed using the Vulkan graphics API.

[0023] To address the current lack of effective performance enhancements for applications developed using the Vulkan graphics API, this specification proposes a method, device, and computer-readable storage medium for controlling an application. The method can be executed by an electronic device or software installed in an electronic device. The electronic device includes, but is not limited to, any of the following intelligent devices: smartphones, personal computers (PCs), laptops, tablets, e-readers, internet-connected televisions, wearable devices, and the like.

[0024] This specification provides a method for controlling an application, which aims to collect the actual load of the target application at every moment, and assign performance resources to the target application and / or adjust the display resolution according to different actual loads. Because, taking the target application as a game application as an example, it is impossible for any scene and operation to have a constant load on the electronic device throughout the game. By identifying the actual load and then releasing performance resources in real time according to the actual load, it is possible to ensure that the performance is continuously optimized while improving the energy efficiency ratio. In addition, based on the identified actual load, the display resolution of each picture is dynamically adjusted to enhance the image quality of the main object in the picture, and maintain or reduce the image quality of non-main objects (edges). This can save some energy consumption while greatly improving the viewing experience and smoothness of the entire game.

[0025] A method for controlling an application program provided by an embodiment of this specification will be described in detail below with reference to the accompanying drawings.

[0026] As shown in FIG1 , an embodiment of this specification provides a method for controlling an application program, which may include:

[0027] Step 101: During the rendering time of the current frame data of the target application, a counter is used to record the number of times a Vulkan module writes data to a command buffer of the Vulkan module, thereby obtaining a total number of times the Vulkan module writes data to the command buffer during the rendering time of the current frame data.

[0028] The target application may be any graphics application. Alternatively, the target application may be a graphics application with relatively high display performance requirements, such as a game application, a drawing application, or other application with high graphics rendering frequency and complex rendering.

[0029] The current frame data of the target application refers to the frame data that the target application currently needs to synthesize and display. Typically, the second interface (such as notifyVKCmdCountFlagNeeded) of the display synthesis system (generally referring to SurfaceFlinger) continuously reads each frame data of the target application, then synthesizes the framebuffer and saves it to the frame buffer. Finally, the screen of the electronic device reads the framebuffer from the frame buffer and displays it.

[0030] Optionally, before step 101, the first interface in the display composition system may be used to identify whether the graphics API used by the target application is Vulkan. If so, step 101 is performed again; otherwise, the process ends. The first interface may be a newly added interface.

[0031] Specifically, the Vulkan module can call the method SurfaceFlingerBlastBufferQueue in the display synthesis system - SurfaceFlinger, and add an interface (first interface) in this method to monitor the integrity and cycle of the current frame data in real time, and before rendering the current frame data, determine whether the drawing API used by the target application is Vulkan. Among them, the main function of SurfaceFlingerBlastBufferQueue is to process the graphics buffer queue, and its specific function is to add the graphics buffer to the queue, and take it out of the queue and display it on the screen when needed. Generally speaking, when an application or system needs to display a new graphical interface, it will add the corresponding graphics buffer area to the BufferQueue queue, and take it out and display it on the screen when needed.

[0032] The first interface may include but is not limited to notifyVKCmdCountFlagIfNeeded added to the SurfaceFlingerBlastBufferQueue method of SurfaceFlinger.

[0033] The counter is pre-inserted into the Vulkan module. Optionally, the counter is cleared before the Vulkan module starts rendering the current frame data to ensure that the current actual load can be correctly collected.

[0034] A counter (vk_messager) inserted into the Vulkan module is used to record the number of times a series of commands used to perform rendering and computing tasks are submitted to the Vulkan buffer in parallel in a multi-threaded environment. Through algorithmic filtering and refreshing, each frame of data is accurately and real-timely counted. Specifically, the counter inserted into the Vulkan module is used to record the number of times the Vulkan module writes data to the Vulkan module's command buffer (vulkan count). This write count can be represented by an integer.

[0035] The counter of this graphics API inserted into the Vulkan module is implemented by counting through statistical queries (occlusion queries). The main function logic of the code is used to query the number of times a specific event occurs and count it.

[0036] It can be understood that the total number of writes corresponding to the current frame data can reflect the rendering pressure of the current frame data or the current actual load of the target application. The core of the control method of an application provided in an embodiment of this specification is to use the number of writes to the command buffer of the Vulkan module - the actual submission number (vulkan count) to approximate the real-time rendering pressure of the target application. If this value is recorded with a counter, the different rendering pressures at different times during the operation of the target application can be identified. The larger the value at a certain moment, the greater the rendering pressure of the target application at this point in time.

[0037] Because the rendering pressure corresponding to each frame of data is different, the number of threads used by the Vulkan graphics API to perform rendering and computing tasks is different, and the sum of the number of writes to its own buffer (total writes) is also different. Taking the target application as a game application as an example, for the flying scene in the game, the rendering pressure when the flying object just takes off may be relatively low. Under this rendering pressure, there are N1 computing tasks. The Vulkan module needs to write N1 times to its own command buffer (Command Buffer). Accordingly, the counter counts N1 times; then, as the scene changes, the number of thread tasks in the synthesis process will also change. In response to this change, the counter continuously adds 1 to the integer data with a base value of N1 to increase the count value until the rendering of this frame is completed (dequenbuff writes to quenbuff). For example, assuming the frame rate is 60 frames, it takes 16ms.

[0038] Optionally, after step 101 and before the following step 102, the method may further include: recording the total number of write times in the command buffer area.

[0039] Furthermore, after step 101 and before step 102, the method may further include: encrypting the total write count recorded in the command buffer using encryption code in the Vulkan module to ensure the security of the total write count. Specifically, the total write count may be encrypted by code (verifySocket) pre-implanted in the Vulkan module. Optionally, the display composition system (rendering layer) may query the total write count using a specified switch (vkCmdCount).

[0040] Step 102: Determine the current rendering pressure of the target application according to the total number of writes.

[0041] As an example, step 102 may include: using the total number of write times as the current rendering pressure value of the target application.

[0042] As another example, the total number of writes may be converted into a numerical value and used as the current rendering pressure value of the target application. The numerical conversion may include, but is not limited to, proportional conversion (e.g., conversion to a value between 0 and 1), level conversion (e.g., conversion to a matching level), and the like.

[0043] It should be noted that, no matter whether the total number of writes itself or the converted value of the total number of writes is used to represent the current rendering pressure value of the target application, the current rendering pressure value of the target application is always positively correlated with the number of writes.

[0044] Step 103: Control at least one of the performance resources and display resolution of the target application according to the current rendering pressure.

[0045] It should be noted that, in addition to performance resources and display resolution, other performance indicators of the target application may also be controlled according to the current rendering pressure, and this specification does not impose any restrictions on this.

[0046] As an example, step 103 may include but is not limited to at least one of the following two methods:

[0047] 1) determining a performance resource usage policy for the target application based on the current rendering pressure value, and controlling the target application to run according to the performance resource usage policy (or, releasing performance resources to the target application according to the performance resource usage policy); wherein the performance resources include but are not limited to at least one of a CPU and a GPU.

[0048] 2) Determine a display resolution adjustment strategy for different areas of the display screen corresponding to the current frame data according to the current rendering pressure value, and control the target application to display the screen corresponding to the current frame data according to the display resolution adjustment strategy.

[0049] The above two methods are described one by one below with examples.

[0050] With respect to the first method, determining the performance resource usage strategy of the target application based on the current rendering pressure value may include: first reporting the current rendering pressure value to a performance central control via the Vulkan module, so that the performance central control determines the performance resource usage strategy of the target application based on the current rendering pressure value. The performance central control refers to a performance control module in an electronic device.

[0051] Specifically, the Vulkan module can package the current rendering pressure value into a what type and report it to the performance control center through a socket, so that the performance control center determines the performance resource usage strategy of the target application according to the current rendering pressure value.

[0052] Optionally, the Vulkan module can report the current rendering pressure value to the performance central control according to the reporting numerical interval, wherein the report is made when the current rendering pressure value reaches the upper limit of the reporting numerical interval. That is to say, after obtaining the current rendering pressure value, the current rendering pressure value is interval-matched, and a suitable reporting interval is set for reporting. For example, 1 to 5 can be used as a limited interval. If the current rendering pressure value falls within this interval, it is reported only when the current rendering pressure value is equal to 5. The next time the current rendering pressure value falls within this interval but is not equal to 5, it will not be reported. This can reduce the number of reports, reduce system overhead, and save energy consumption.

[0053] Furthermore, determining the performance resource usage strategy of the target application according to the current rendering pressure value may include:

[0054] When the current rendering pressure value is greater than or equal to a second threshold, determining that the performance resource usage policy of the target application is a first resource usage policy, wherein the first resource usage policy includes: controlling at least one of a CPU and a GPU to be at a first frequency point;

[0055] When the current rendering pressure value is less than or equal to the first threshold, determining that the performance resource usage policy of the target application is a second resource usage policy, wherein the second resource usage policy includes: controlling at least one of the CPU and the GPU to be at a second frequency point lower than the first frequency point;

[0056] The first threshold is less than or equal to the second threshold.

[0057] The significance of the above two resource usage strategies is that: when the current rendering pressure value is large, more CPU or GPU resources are released to the target application, such as making the CPU or GPU at the highest frequency point; and when the current rendering pressure value is small, fewer CPU or GPU resources are released to the target application, such as making the CPU or GPU at two-thirds of the highest frequency point, so as to ensure continuous optimal performance while improving energy efficiency.

[0058] Of course, in addition to the first resource usage strategy and the second resource usage strategy mentioned above, those skilled in the art can also expand other resource usage strategies based on the above technical concepts, and are not limited to these two.

[0059] For the second method, determining the display resolution adjustment strategy for different areas in the display screen corresponding to the current frame data based on the current rendering pressure value may include: obtaining the current rendering pressure value through the extended function of the Vulkan module, and determining the display resolution adjustment strategy for different areas in the display screen corresponding to the current frame data based on the current rendering pressure value.

[0060] Furthermore, determining, according to the current rendering pressure value, a display resolution adjustment strategy for different areas in the display screen corresponding to the current frame data may include:

[0061] When the current rendering pressure value is less than or equal to a first threshold, determining that a display resolution adjustment strategy for different areas in the display screen corresponding to the current frame data is a first display resolution adjustment strategy, wherein the first display resolution adjustment strategy includes: controlling the display resolution of the area where the main object is located in the screen corresponding to the current frame data to increase, and / or controlling the display resolution of the area surrounding the main object to remain unchanged;

[0062] When the current rendering pressure value is greater than or equal to a second threshold, determining that the display resolution adjustment strategy for different areas in the display screen corresponding to the current frame data is a second display resolution adjustment strategy, wherein the second display resolution adjustment strategy includes: controlling the display resolution of the area surrounding the main object in the screen corresponding to the current frame data to be lowered, and controlling the display resolution of the area where the main object is located to remain unchanged;

[0063] The first threshold is less than or equal to the second threshold.

[0064] The main object refers to the main display object in the display screen, such as the central character in the game screen.

[0065] The significance of the above two display resolution adjustment strategies is that: when the current rendering pressure value is large (high rendering pressure), the display resolution of the central character remains unchanged, and the display resolution of the edge scene is appropriately reduced; and when the current rendering pressure value is small (low rendering pressure), the display resolution of the central character is appropriately increased, and the display resolution of the edge scene remains unchanged, so as to save some energy consumption while greatly improving the viewing experience and smoothness of the entire game.

[0066] In specific implementation, the display synthesis system can pass the current rendering pressure value to the Vulkan core API through the interface for Vulkan extension to read the data in the buffer queue, construct rendering instructions for the rendering channel, capture the resolution under different display screens, and then match the current rendering pressure value through the ResolutionDynamic function to dynamically change the display resolution of different areas in the display screen corresponding to the current frame data.

[0067] Of course, in addition to the first display resolution adjustment strategy and the second display resolution adjustment strategy, those skilled in the art can also develop other display resolution adjustment strategies based on the above technical concept, and are not limited to these two.

[0068] An embodiment of the present specification provides a method for controlling an application. During the rendering time of the current frame data of the target application, a counter is used to record the number of times a Vulkan module writes data to a command buffer of the Vulkan module, thereby obtaining the total number of writes by the Vulkan module to the command buffer during the rendering time of the current frame data. Based on the total number of writes, the current actual rendering pressure (equivalent to the actual load) of the target application can be accurately calculated. Then, based on the current actual rendering pressure of the target application, at least one of the performance resources and display resolution of the target application can be dynamically controlled, thereby improving the performance of the target application developed using Vulkan.

[0069] The following uses SurfaceFlinger as an example of a display synthesis system to introduce a possible implementation architecture of a control method for an application provided in an embodiment of this specification.

[0070] As shown in Figure 2, a possible implementation architecture of a control method for an application provided in an embodiment of this specification may include: SurfaceFlingerBlastBufferQueue 21, a counter 22 inserted into the Vulkan module, an extended function 23 of the Vulkan module, a performance central control 24, a performance adjustment module 25 and a ResolutionDynamic function 26.

[0071] SurfaceFlingerBlastBufferQueue 21 is a method in SurfaceFlinger. Its main function is to process the graphics buffer queue, add the graphics buffer to the queue, and take it out of the queue and display it on the screen when needed. Specifically, when an application or system needs to display a new graphics interface, it adds the corresponding graphics buffer to the BufferQueue queue, and takes it out and displays it on the screen when needed. An interface (notifyVKCmdCountFlagIfNeeded) is added to SurfaceFlingerBlastBufferQueue 21 to identify the graphics API and rendering process used by the target application.

[0072] The counter 22 inserted into the Vulkan module is used to record the number of times a series of commands for executing rendering and computing tasks are submitted in parallel in a multi-threaded environment to the Vulkan buffer area, and through algorithm filtering and refreshing, accurate and real-time counting of each frame of data is achieved.

[0073] The Vulkan module's extension function 23 extends the functionality of the Vulkan core API. Vulkan extensions are implemented by adding additional commands, functions, and data structures to the Vulkan core API. The extension primarily adds new graphics functionality: the ResolutionDynamic function 26.

[0074] The performance central control 24 is used to receive the vulkan count (the total number of writes) uploaded by framewors, and send different performance resource usage strategies to the performance adjustment module 25 according to the current rendering pressure value.

[0075] The performance adjustment module 25 is configured to release performance resources to the target application according to the issued performance resource usage policy.

[0076] The ResolutionDynamic function 26 is used to receive the vulkan count (the total number of writes) to dynamically improve the display resolution of the display image to improve the user experience while optimizing performance.

[0077] Based on the implementation architecture shown in FIG2 , as shown in FIG3 , an application control method provided in an embodiment of this specification may specifically include:

[0078] Step 300, start.

[0079] Step 301: Add a first interface to the display synthesis system.

[0080] Specifically, by adding a first interface (such as notifyVKCmdCountFlagIfNeeded) to SurfaceFlingerBlastBufferQueue of SurfaceFlinger (a display composition system), the graphics API and rendering process used by the target application can be identified.

[0081] Among them, SurfaceFlingerBlastBufferQueue is a method in SurfaceFlinger, whose main function is to process the graphics buffer queue, add the graphics buffer to the queue, and take it out of the queue and display it on the screen when needed. Specifically, when an application or system needs to display a new graphics interface, it will add the corresponding graphics buffer to the BufferQueue queue, and take it out and display it on the screen when needed.

[0082] Step 302 , identifying whether the graphics API used by the target application is Vulkan through the first interface; if yes, executing step 303 , otherwise executing step 309 .

[0083] Step 303: Read the frame data of the target application through the second interface in the display composition system.

[0084] Step 304: During the rendering time of the current frame data of the target application, a counter is used to record the number of times the Vulkan module writes data to the command buffer of the Vulkan module, thereby obtaining the total number of times the Vulkan module writes data to the command buffer during the rendering time of the current frame data.

[0085] Step 305: Report the total number of write times to the performance control center.

[0086] That is, the total number of write times is reported as the current rendering pressure value.

[0087] Optionally, the total number of write times is matched with a reporting value interval, and reporting is performed when the total number of write times reaches an upper limit of the reporting value interval.

[0088] Specifically, the total number of write times is packaged into the what data format and then reported to the frameworks in the electronic device through the socket. The frameworks parse the data and transmit it to the performance control center in sections. The performance control center provides different levels of performance resources according to different numerical intervals.

[0089] Step 306: The performance central control determines a performance resource usage strategy for the target application based on the total number of writes.

[0090] In step 307 , the extended function of the Vulkan module transmits the total number of write times to the core of the Vulkan module.

[0091] That is, the data is obtained through the extended function of Vulkan and passed to the core of the Vulkan module.

[0092] Step 308: The core of the Vulkan module determines and adjusts the display resolution adjustment strategy of the target application based on the total number of write times.

[0093] Specifically, the display resolution of different areas of the display screen is dynamically adjusted through the ResolutionDynamic function. The main implementation principle of the ResolutionDynamic function is to match the rendering detail data with the size of the collected vulkan count value, to readjust the rendering detail data, focus on the main object screen, and dynamically enhance or reduce the rendering details of the main screen or the edge screen.

[0094] Step 309, end.

[0095] When the control method for an application provided in an embodiment of this specification is applied to a gaming application, first, the game interface freezes less and the frame rate experience improves. Second, the display resolution of the game screen is dynamically refreshed. When the rendering pressure is high, the display resolution of the center screen is enhanced, while the display resolution of the edge screen is reduced more. When the rendering pressure is low, the display resolution of the center screen is greatly enhanced, and the display resolution of the edge screen is reduced less. In addition, by adjusting the performance resource release strategy (resource utilization strategy), the frame rate can be kept full while the user is playing high-load games, and the device heat is significantly reduced, greatly improving the energy efficiency ratio. By dynamically adjusting the display resolution, the user's gaming visual experience can be enhanced while improving performance.

[0096] The above describes a method for controlling an application program provided in this specification. Corresponding to the above method, this specification also provides a device for controlling an application program, which is briefly described below.

[0097] FIG4 is a schematic diagram of a control device 400 of an application program according to an embodiment of the present disclosure. Referring to FIG4 , in a software implementation, the control device 400 of the application program may include: a counting module 401 , a load determination module 402 , and a performance control module 403 .

[0098] The counting module 401 is configured to record, by a counter, the number of times the Vulkan module writes data to the command buffer of the Vulkan module during the rendering time of the current frame data of the target application, thereby obtaining a total number of times the Vulkan module writes data to the command buffer during the rendering time of the current frame data.

[0099] The target application may be any graphics application. Alternatively, the target application may be a graphics application with relatively high display performance requirements, such as a game application, a drawing application, or other application with high graphics rendering frequency and complex rendering.

[0100] The current frame data of the target application refers to the frame data that the target application currently needs to synthesize and display. Typically, the second interface (such as notifyVKCmdCountFlagNeeded) of the display synthesis system (generally referring to SurfaceFlinger) continuously reads each frame data of the target application, then synthesizes the framebuffer and saves it to the frame buffer. Finally, the screen of the electronic device reads the framebuffer from the frame buffer and displays it.

[0101] Optionally, the apparatus may further include a determination module configured to, before triggering the counting module 401, identify whether the graphics API used by the target application is Vulkan through a first interface in the display composition system, and if so, trigger the counting module 401; otherwise, do not trigger the counting module 401. The first interface may be a newly added interface.

[0102] Specifically, the Vulkan module can call the method SurfaceFlingerBlastBufferQueue in the display synthesis system - SurfaceFlinger, and add an interface (first interface) in this method to monitor the integrity and cycle of the current frame data in real time, and before rendering the current frame data, determine whether the drawing API used by the target application is Vulkan. Among them, the main function of SurfaceFlingerBlastBufferQueue is to process the graphics buffer queue, and its specific function is to add the graphics buffer to the queue, and take it out of the queue and display it on the screen when needed. Generally speaking, when an application or system needs to display a new graphical interface, it will add the corresponding graphics buffer area to the BufferQueue queue, and take it out and display it on the screen when needed.

[0103] The first interface may include but is not limited to notifyVKCmdCountFlagIfNeeded added to the SurfaceFlingerBlastBufferQueue method of SurfaceFlinger.

[0104] The counter is pre-inserted into the Vulkan module. Optionally, the device may further include a clearing module configured to clear the counter before triggering the counting module 401 to ensure that the current actual load can be correctly collected.

[0105] Optionally, the device may further include: a writing module, configured to record the total number of write times in the command buffer area.

[0106] Furthermore, the apparatus may further include an encryption module configured to encrypt the total write count recorded in the command buffer using encryption code in the Vulkan module to ensure the security of the total write count. Specifically, the total write count may be encrypted using code (verifySocket) pre-implanted in the Vulkan module. Optionally, the display composition system (rendering layer) may query the total write count using a specified switch (vkCmdCount).

[0107] The load determination module 402 is configured to determine the current rendering pressure of the target application according to the total number of write times.

[0108] As an example, the load determination module 402 may use the total number of write times as the current rendering pressure value of the target application.

[0109] As another example, the load determination module 402 may also convert the total number of writes into a numerical value and use it as the current rendering pressure value of the target application. The numerical conversion may include, but is not limited to, proportional conversion (e.g., conversion to a value between 0 and 1), level conversion (e.g., conversion to a matching level), etc.

[0110] It should be noted that, no matter whether the total number of writes itself or the converted value of the total number of writes is used to represent the current rendering pressure value of the target application, the current rendering pressure value of the target application is always positively correlated with the number of writes.

[0111] The performance control module 403 is configured to control at least one of the performance resources and the display resolution of the target application according to the current rendering pressure.

[0112] It should be noted that, in addition to performance resources and display resolution, other performance indicators of the target application may also be controlled according to the current rendering pressure, and this specification does not impose any restrictions on this.

[0113] As an example, the performance control module 403 may be used to:

[0114] 1) determining a performance resource usage policy for the target application based on the current rendering pressure value, and controlling the target application to run according to the performance resource usage policy (or, releasing performance resources to the target application according to the performance resource usage policy); wherein the performance resources include but are not limited to at least one of a CPU and a GPU; and / or

[0115] 2) Determine a display resolution adjustment strategy for different areas of the display screen corresponding to the current frame data according to the current rendering pressure value, and control the target application to display the screen corresponding to the current frame data according to the display resolution adjustment strategy.

[0116] The above two methods are described one by one below with examples.

[0117] For the first method, the performance control module 403 may be configured to first report the current rendering pressure value to the performance control center through the Vulkan module, so that the performance control center determines the performance resource usage strategy of the target application based on the current rendering pressure value. The performance control center refers to a performance control module in an electronic device.

[0118] Specifically, the Vulkan module can package the current rendering pressure value into a what type and report it to the performance control center through a socket, so that the performance control center determines the performance resource usage strategy of the target application according to the current rendering pressure value.

[0119] Optionally, the Vulkan module can report the current rendering pressure value to the performance central control according to the reporting numerical interval, wherein the report is made when the current rendering pressure value reaches the upper limit of the reporting numerical interval. That is to say, after obtaining the current rendering pressure value, the current rendering pressure value is interval-matched, and a suitable reporting interval is set for reporting. For example, 1 to 5 can be used as a limited interval. If the current rendering pressure value falls within this interval, it is reported only when the current rendering pressure value is equal to 5. The next time the current rendering pressure value falls within this interval but is not equal to 5, it will not be reported. This can reduce the number of reports, reduce system overhead, and save energy consumption.

[0120] Furthermore, the performance control module 403 may be used to:

[0121] When the current rendering pressure value is greater than or equal to a second threshold, determining that the performance resource usage policy of the target application is a first resource usage policy, wherein the first resource usage policy includes: controlling at least one of a CPU and a GPU to be at a first frequency point;

[0122] When the current rendering pressure value is less than or equal to the first threshold, determining that the performance resource usage policy of the target application is a second resource usage policy, wherein the second resource usage policy includes: controlling at least one of the CPU and the GPU to be at a second frequency point lower than the first frequency point;

[0123] The first threshold is less than or equal to the second threshold.

[0124] The significance of the above two resource usage strategies is that: when the current rendering pressure value is large, more CPU or GPU resources are released to the target application, such as making the CPU or GPU at the highest frequency point; and when the current rendering pressure value is small, fewer CPU or GPU resources are released to the target application, such as making the CPU or GPU at two-thirds of the highest frequency point, so as to ensure continuous optimal performance while improving energy efficiency.

[0125] Of course, in addition to the first resource usage strategy and the second resource usage strategy mentioned above, those skilled in the art can also expand other resource usage strategies based on the above technical concepts, and are not limited to these two.

[0126] For the second method, the performance control module 403 can be used to: obtain the current rendering pressure value through the extended function of the Vulkan module, and determine the display resolution adjustment strategy for different areas in the display screen corresponding to the current frame data based on the current rendering pressure value.

[0127] Furthermore, the performance control module 403 may be used to:

[0128] When the current rendering pressure value is less than or equal to a first threshold, determining that a display resolution adjustment strategy for different areas in the display screen corresponding to the current frame data is a first display resolution adjustment strategy, wherein the first display resolution adjustment strategy includes: controlling the display resolution of the area where the main object is located in the screen corresponding to the current frame data to increase, and / or controlling the display resolution of the area surrounding the main object to remain unchanged;

[0129] When the current rendering pressure value is greater than or equal to a second threshold, determining that the display resolution adjustment strategy for different areas in the display screen corresponding to the current frame data is a second display resolution adjustment strategy, wherein the second display resolution adjustment strategy includes: controlling the display resolution of the area surrounding the main object in the screen corresponding to the current frame data to be lowered, and controlling the display resolution of the area where the main object is located to remain unchanged;

[0130] The first threshold is less than or equal to the second threshold.

[0131] The main object refers to the main display object in the display screen, such as the central character in the game screen.

[0132] The significance of the above two display resolution adjustment strategies is that: when the current rendering pressure value is large (high rendering pressure), the display resolution of the central character remains unchanged, and the display resolution of the edge scene is appropriately reduced; and when the current rendering pressure value is small (low rendering pressure), the display resolution of the central character is appropriately increased, and the display resolution of the edge scene remains unchanged, so as to save some energy consumption while greatly improving the viewing experience and smoothness of the entire game.

[0133] In specific implementation, the display synthesis system can pass the current rendering pressure value to the Vulkan core API through the interface for Vulkan extension to read the data in the buffer queue, construct rendering instructions for the rendering channel, capture the resolution under different display screens, and then match the current rendering pressure value through the ResolutionDynamic function to dynamically change the display resolution of different areas in the display screen corresponding to the current frame data.

[0134] Of course, in addition to the first display resolution adjustment strategy and the second display resolution adjustment strategy, those skilled in the art can also develop other display resolution adjustment strategies based on the above technical concept, and are not limited to these two.

[0135] An embodiment of the present specification provides an application control device. Since it can record the number of times a Vulkan module writes data to a command buffer of the Vulkan module through a counter during the rendering time of the current frame data of the target application, the total number of times the Vulkan module writes data to the command buffer during the rendering time of the current frame data can be obtained, thereby accurately calculating the current actual rendering pressure (equivalent to the actual load) of the target application based on the total number of writes. Then, at least one of the performance resources and display resolution of the target application can be dynamically controlled based on the current actual rendering pressure of the target application, thereby improving the performance of the target application developed using Vulkan.

[0136] The control device 400 of the application provided in the embodiment of this specification can also execute the method of Figure 1, and realize the functions of the embodiment shown in Figure 1, and achieve the same technical effects. The embodiments of this specification are not described in detail here.

[0137] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0138] FIG5 is a schematic diagram of the structure of an electronic device 500 according to an embodiment of the present specification. Referring to FIG5 , at the hardware level, the electronic device 500 includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include internal memory, such as high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device. Of course, the electronic device may also include hardware required for other services.

[0139] The processor, network interface, and memory can be interconnected via an internal bus, such as an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. These buses can be categorized as address buses, data buses, and control buses. For ease of illustration, FIG5 shows only one bidirectional arrow, but this does not imply that there is only one bus or only one type of bus.

[0140] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0141] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming the control device of the application at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0142] During the rendering time of current frame data of the target application, recording, by a counter, the number of times a Vulkan module writes data to a command buffer of the Vulkan module, to obtain a total number of times the Vulkan module writes data to the command buffer during the rendering time of the current frame data;

[0143] determining a current rendering pressure of the target application according to the total number of writes;

[0144] At least one of performance resources and display resolution of the target application is controlled according to the current rendering pressure.

[0145] The method executed by the control device of the application disclosed in the embodiment shown in FIG4 of this specification can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or by software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this specification can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of this specification can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0146] The electronic device can also execute the methods of Figures 1 to 3 and implement the functions of the control device of the application in the embodiments shown in Figures 1 to 3, which will not be described in detail in this specification.

[0147] Of course, in addition to software implementation, the electronic device in this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0148] The embodiments of this specification also provide a computer-readable storage medium, which stores one or more programs. The one or more programs include instructions. When the instructions are executed by a portable electronic device including multiple target applications, the portable electronic device can execute the method of the embodiments shown in Figures 1 to 3, and specifically is used to perform the following operations:

[0149] During the rendering time of current frame data of the target application, recording, by a counter, the number of times a Vulkan module writes data to a command buffer of the Vulkan module, to obtain a total number of times the Vulkan module writes data to the command buffer during the rendering time of the current frame data;

[0150] determining a current rendering pressure of the target application according to the total number of writes;

[0151] At least one of performance resources and display resolution of the target application is controlled according to the current rendering pressure.

[0152] In short, the above description is only a preferred embodiment of this specification and is not intended to limit the scope of protection of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification shall be included in the scope of protection of this specification.

[0153] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0154] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0155] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0156] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

Claims

1. A method for controlling an application, comprising: During the rendering time of the current frame data of the target application, the number of times the Vulkan module writes data to the command buffer of the Vulkan module is recorded by a counter, so as to obtain the total number of times the Vulkan module writes data to the command buffer during the rendering time of the current frame data; Determine the current rendering pressure of the target application according to the total number of writes; At least one of the performance resources and the display resolution of the target application is controlled according to the current rendering pressure.

2. The method according to claim 1, wherein: Before determining the current rendering pressure of the target application according to the total number of write times, the method further includes: The total number of write times is recorded in the command buffer area.

3. The method according to claim 2, further comprising: The total number of write times recorded in the command buffer is encrypted by the encryption code in the Vulkan module.

4. The method according to claim 1, wherein: The counter is pre-inserted into the Vulkan module.

5. The method according to any one of claims 1 to 4, wherein: The determining the current rendering pressure of the target application according to the total number of write times includes: The total number of write times is used as the current rendering pressure value of the target application.

6. The method according to claim 5, wherein: The controlling at least one of the performance resources and the display resolution of the target application according to the current rendering pressure includes: Determine a performance resource usage policy of the target application according to the current rendering pressure value, and control the target application to run according to the performance resource usage policy; and / or, A display resolution adjustment strategy for different areas in the display screen corresponding to the current frame data is determined according to the current rendering pressure value, and the target application is controlled to display the screen corresponding to the current frame data according to the display resolution adjustment strategy.

7. The method according to claim 6, wherein: The determining the performance resource usage strategy of the target application according to the current rendering pressure value includes: The current rendering pressure value is reported to the performance central control through the Vulkan module, so that the performance central control determines the performance resource usage strategy of the target application according to the current rendering pressure value.

8. The method according to claim 7, wherein: The reporting the current rendering pressure value to the performance central control through the Vulkan module includes: The current rendering pressure value is reported to the performance central control through the Vulkan module according to the reporting value interval, wherein the reporting is performed when the current rendering pressure value reaches the upper limit of the reporting value interval.

9. The method according to any one of claims 6 to 8, wherein: The determining the performance resource usage strategy of the target application according to the current rendering pressure value includes: When the current rendering pressure value is greater than or equal to the second threshold, determining the performance resource usage strategy of the target application to be a first resource usage strategy, wherein the first resource usage strategy includes: controlling at least one of the CPU and the GPU to be at a first frequency point; When the current rendering pressure value is less than or equal to the first threshold, determining the performance resource usage policy of the target application to be a second resource usage policy, wherein the second resource usage policy includes: controlling at least one of the CPU and the GPU to be at a second frequency point lower than the first frequency point; The first threshold is less than or equal to the second threshold.

10. The method according to claim 6, wherein: The step of determining, according to the current rendering pressure value, a display resolution adjustment strategy for different areas in the display screen corresponding to the current frame data includes: The current rendering pressure value is obtained through the extended function of the Vulkan module, and the display resolution adjustment strategy of different areas in the display screen corresponding to the current frame data is determined according to the current rendering pressure value.

11. The method according to claim 6 or 10, wherein: The step of determining, according to the current rendering pressure value, a display resolution adjustment strategy for different areas in the display screen corresponding to the current frame data includes: When the current rendering pressure value is less than or equal to the first threshold, determining that the display resolution adjustment strategy for different areas in the display screen corresponding to the current frame data is a first display resolution adjustment strategy, wherein the first display resolution adjustment strategy includes: controlling the display resolution of the area where the main object is located in the screen corresponding to the current frame data to increase, and / or controlling the display resolution of the surrounding area of ​​the main object to remain unchanged; When the current rendering pressure value is greater than or equal to the second threshold, determining that the display resolution adjustment strategy for different areas in the display screen corresponding to the current frame data is a second display resolution adjustment strategy, wherein the second display resolution adjustment strategy includes: controlling the display resolution of the surrounding area of ​​the main object in the screen corresponding to the current frame data to be reduced, and controlling the display resolution of the area where the main object is located to remain unchanged; The first threshold is less than or equal to the second threshold.

12. The method according to claim 1, before recording the number of times the Vulkan module writes data to the command buffer of the Vulkan module by a counter, the method further comprises: Identify, through a first interface in a display composition system, whether a drawing API used by the target application is Vulkan; If the drawing API used by the target application is Vulkan, the step of recording the number of times the Vulkan module writes data to the command buffer of the Vulkan module through a counter is performed.

13. The method according to claim 1, further comprising: Before the Vulkan module starts rendering the current frame data, the counter is cleared.

14. An electronic device comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 13. 15 . A computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to claim 1 .

Images