An image processing method, electronic device, medium, and product

Abstract

This application relates to the field of image processing technology, and discloses an image processing method, electronic device, medium, and product. In this method, during the process of drawing an image on a display screen, the electronic device can obtain the visual effect drawing instructions and background drawing instructions corresponding to the image to be drawn, and determine the maximum number of instructions that the GPU can draw per unit time. When drawing the image, if the total number of visual effect drawing instructions and background drawing instructions corresponding to the image exceeds the maximum number of instructions of the GPU, i.e., the GPU cannot complete drawing the image per unit time, the electronic device can reduce the visual effect drawing instructions so that the total number of visual effect drawing instructions and background drawing instructions corresponding to the image is less than or equal to the maximum number of instructions that the GPU can draw per unit time, thereby avoiding frame dropping. It can be understood that when the GPU draws the image based on the target visual effect drawing instructions, the total power consumption of the CPU and GPU is less than or equal to the maximum power consumption of the electronic device.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, electronic device, medium, and product. Background Technology

[0002] During the process of rendering and displaying images in an electronic device, the central processing unit (CPU) sends visual effect rendering instructions (hereinafter referred to as visual effect rendering instructions) corresponding to each frame of the image to the graphics processing unit (GPU), and executes the visual effect rendering instructions to render images with visual effects. For example, executing visual effect rendering instructions to blur the background area and brighten the subject area will render an image with a blurred background area and a bright subject area. Specifically, the CPU in the electronic device can obtain the visual effect rendering instructions corresponding to each frame of the image and transmit them to the GPU for image rendering.

[0003] It's understandable that the maximum number of instructions a GPU can execute per unit of time is fixed. Therefore, when the number of visual rendering instructions corresponding to a certain frame in the display exceeds the maximum number of instructions the GPU can execute per unit of time, the GPU will be unable to complete all the visual rendering instructions corresponding to that frame within that unit of time. This will cause the GPU to render the next frame before finishing rendering that frame, resulting in dropped frames in the display. Summary of the Invention

[0004] To address the issue of dropped frames in displayed images, embodiments of this application provide an image processing method, electronic device, medium, and product, including:

[0005] In a first aspect, embodiments of this application provide an image processing method applied to an electronic device, characterized in that it includes: acquiring a first drawing instruction for a first image, the first drawing instruction including a first number of first-type drawing instructions and a second number of second-type drawing instructions, wherein the first-type drawing instructions are used to draw background elements of the first image, and the second-type drawing instructions are used to draw visual effect elements of the first image; corresponding to a time when the duration of drawing the first image based on the first drawing instruction is greater than a preset time, changing the first drawing instruction to a second drawing instruction, the second drawing instruction including a first number of first-type drawing instructions and a third number of second-type drawing instructions, wherein the duration of drawing the first image based on the second drawing instruction is less than or equal to the preset time.

[0006] Based on the above scheme, when drawing an image, if the total number of visual effect drawing instructions and background drawing instructions corresponding to the image is greater than the maximum number of instructions, that is, the graphics processor cannot draw the image in a unit of time, the electronic device can reduce the visual effect drawing instructions so that the total number of visual effect drawing instructions and background drawing instructions corresponding to the image is less than or equal to the maximum number of instructions that the graphics processor can draw in a unit of time, so as to avoid frame dropping.

[0007] It is understandable that when an electronic device detects a user switching the display screen, or a signal from an application switching the display screen, the GPU needs to draw a first image, and the GPU needs to execute the first drawing instructions corresponding to the first image. The first drawing instructions refer to the visual effects drawing instructions and background drawing instructions contained in the first image.

[0008] The background drawing instruction can be a first type of drawing instruction used to draw the background elements of the first image, and the number of background drawing instructions is a first quantity. The background elements can be other image elements that the GPU needs to draw besides the visual effect elements when drawing the image.

[0009] Visual effects rendering instructions can be classified as second-class rendering instructions, used to render the visual effects elements of the first image, and the number of visual effects rendering instructions is the second number. It can be understood that when the GPU renders the first image, it needs to execute all the visual effects rendering instructions and background rendering instructions contained in the first image in order to complete the rendering of the first image.

[0010] Understandably, in some possible circumstances, an electronic device can change the first rendering instruction to a second rendering instruction. The second rendering instruction includes a first number of first-type rendering instructions and a third number of second-type rendering instructions. That is, the electronic device can adjust the visual rendering instructions so that the second rendering instruction is less than or equal to the maximum number of instructions the GPU can execute per unit time. Therefore, the time it takes for the GPU to render the first image according to the second rendering instruction is less than or equal to a preset duration.

[0011] Based on the above scheme, if the GPU cannot complete the first rendering instruction corresponding to the first image within a unit of time, causing the GPU to render the first image according to the first rendering instruction to take longer than the preset time, resulting in frame drops, the electronic device can adjust the visual rendering instruction corresponding to the first image to obtain the second rendering instruction corresponding to the first image. When the GPU renders the first image according to the second rendering instruction, the time required is less than or equal to the preset time, thus avoiding frame drops.

[0012] In one possible implementation of the first aspect above, the visual effect elements include at least one of the following: blur, brighten, shadow, and pixel expansion.

[0013] In one possible implementation of the first aspect described above, the preset duration is determined by the clock frequency and cycle number of the graphics processing in the electronic device.

[0014] For example, electronic devices can determine the preset duration based on the graphics processor's clock speed and the number of cycles of the wallpaper.

[0015] In one possible implementation of the first aspect above, corresponding to a duration for drawing the first image based on the first drawing instruction being longer than a preset duration, changing the first drawing instruction to a second drawing instruction includes:

[0016] The first drawing instruction is input into the first model. If the duration of drawing the first image based on the first drawing instruction is longer than a preset duration, the first model outputs the second drawing instruction.

[0017] In one possible implementation of the first aspect above, the first model is trained by: acquiring sample data, wherein the sample data includes sample visual effect rendering instructions and sample background rendering instructions for sample images; inputting the sample data into a second model, and using the second model to predict the predicted visual effect rendering instructions and predicted frame generation time corresponding to the sample data, wherein the predicted frame generation time is the time required for the graphics processor to execute the predicted visual effect rendering instructions and sample background rendering instructions to render the image; if the predicted frame generation time is greater than the label frame generation time, adjusting the parameters of the second model to obtain a second model with adjusted parameters; inputting the sample data into the second model with adjusted parameters until the second model with adjusted parameters predicts that the predicted frame generation time corresponding to the sample data is less than or equal to the label frame generation time; and using the second model with adjusted parameters as the first model.

[0018] Understandably, in some possible circumstances, an electronic device can predict, based on a first model, whether the time required for the GPU to execute the first drawing instruction of the first image is greater than a preset time, and if the time required for the GPU to execute the first drawing instruction of the first image is greater than the preset time, the electronic device can change the first drawing instruction to a second drawing instruction.

[0019] Specifically, the electronic device can acquire sample visual effect rendering instructions and sample background rendering instructions from sample images as sample data. The electronic device can input training data into a second model and perform visual effect instruction prediction and frame generation time prediction based on the sample visual effect rendering instructions and sample background rendering instructions to obtain predicted visual effect rendering instructions and predicted frame generation times. It can be understood that if the predicted frame generation time is greater than the label frame generation time (the generation time of a single frame image), the electronic device can adjust the parameters of the second model (these parameters can be the sample visual effect rendering instructions corresponding to the image), and based on the adjusted second model, re-perform visual effect instruction prediction and frame generation time prediction based on the sample visual effect rendering instructions and sample background rendering instructions until the predicted frame generation time is less than or equal to the label frame generation time, thus obtaining predicted visual effect rendering instructions and predicted frame generation times. Furthermore, the electronic device can use the adjusted second model as the first model. It can be understood that when the GPU renders the image based on the image's background rendering instructions and the adjusted visual effect rendering instructions, no frame dropping will occur.

[0020] Based on the above scheme, electronic devices can obtain adjusted visual rendering instructions, namely the third number of second-type rendering instructions, by training a model. Furthermore, when the GPU renders the first image based on the first number of first-type rendering instructions and the third number of second-type rendering instructions, the time required is less than or equal to the preset time. In other words, the GPU can execute all the rendering instructions corresponding to the first image within a unit of time, thereby avoiding frame dropping.

[0021] In one possible implementation of the first aspect described above, the clock speed and cycle count of the graphics processor are obtained by any of the following methods: obtaining the clock speed and cycle count of the graphics processor from a hardware information table stored in the electronic device; obtaining the clock speed of the graphics processor from a hardware information table stored in the electronic device, and determining the cycle count of the graphics processor based on the clock speed of the graphics processor; obtaining the clock speed of the graphics processor from a hardware information table stored in the electronic device, and determining the cycle count of the graphics processor based on the clock speed of the graphics processor and a first constraint coefficient.

[0022] In one possible implementation of the first aspect described above, the sample visual effect rendering instruction is obtained by decomposing the visual effects of the sample image.

[0023] In one possible implementation of the first aspect above, the visual effects of the sample image are decomposed in the following way: the instructions corresponding to each visual effect element of the sample image are obtained by the detection device during the process of drawing the sample image by the electronic device, and the sample visual effect drawing instructions are obtained.

[0024] In a second aspect, this application provides an electronic device, comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for executing the image processing method mentioned in the first aspect or any one of the first aspects of this application.

[0025] Thirdly, this application provides a readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the image processing method mentioned in the first aspect or any one of the first aspects of this application.

[0026] Fourthly, embodiments of this application provide a computer program product, which includes computer instructions. When executed by an electronic device, the electronic device executes the computer program code of the image processing method mentioned in the first aspect or any one of the first aspects of this application. Attached Figure Description

[0027] Figure 1 According to some embodiments of this application, a schematic diagram of an application scenario for GPU rendering of display screens is shown;

[0028] Figure 2 According to some embodiments of this application, a schematic flowchart of an image processing method is shown;

[0029] Figure 3A According to some embodiments of this application, a flowchart of a CPU load limit model and a flowchart of a GPU load limit model are shown.

[0030] Figure 3B According to some embodiments of this application, a schematic diagram of an electronic device acquiring training data is shown;

[0031] Figure 3C According to some embodiments of this application, a flowchart illustrating the training of a visual effects parameter model and the application of the visual effects parameter model for an electronic device is shown.

[0032] Figure 4 According to some embodiments of this application, a schematic diagram of an application scenario for a detection device to acquire training data is shown;

[0033] Figure 5 According to some embodiments of this application, a schematic diagram of the structure of an image processing system is shown;

[0034] Figure 6 According to some embodiments of this application, a schematic diagram of the hardware structure of an electronic device is shown. Detailed Implementation

[0035] The illustrative embodiments of this application include, but are not limited to, an image processing method, an electronic device, a medium, and a product.

[0036] It is understood that the image processing method mentioned in the embodiments of this application can be applied to any electronic device with image processing capabilities. The electronic device can be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. Furthermore, the electronic device can be a mobile phone, smart TV, wearable device, tablet computer, computer with wireless transceiver capabilities, virtual reality (VR) terminal device, augmented reality (AR) terminal device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, and other similar devices.

[0037] It is understood that the image processing method mentioned in the embodiments of this application can be applied to scenarios where electronic devices draw display screens. For example, scenarios where switching interfaces are drawn, scenarios where dynamic wallpapers are drawn, scenarios where animation effects are drawn, etc.

[0038] The image processing method mentioned in the embodiments of this application will be described below in the context of a specific scenario of drawing and switching interfaces. In the embodiments of this application, the image displayed on the screen can be referred to as the interface.

[0039] Figure 1 A schematic diagram of interface switching on a mobile phone 100 is shown.

[0040] like Figure 1 As shown, in some specific implementations, the phone displays 100... Figure 1 In the case of the application interface 101 shown in (a), if a user pulls down the notification bar (e.g., long-pressing the upper right corner of the application interface 101 and moving it down), the mobile phone 100 can... Figure 1 The application interface 101 shown in (a) switches to Figure 1 The transition interface 102 shown in (b) is shown in the middle.

[0041] It is understandable that the drawing instructions for each frame of the interface can include one or more. For example, when the phone 100 detects the user's pull-down notification bar operation, the CPU will send the visual effect drawing instructions (an example of the second type of drawing instructions) corresponding to the transition interface 102 to the GPU to draw the transition interface 102 and display it. For example, the visual effect drawing instructions can include drawing a blurred area, drawing the first shadow within the blurred area, drawing the second shadow within the blurred area, drawing the third shadow within the blurred area, etc., that is, the visual effect drawing instructions are used to draw the visual effect elements of the interface. Among them, the blurred area can be a rectangular area, and the coordinates of the four vertices of the rectangular area can be (x1, y1), (x2, y1), (x1, y2), and (x2, y2). Specifically, (x1, y1) can refer to the coordinates of the upper left corner of the rectangular blurred area in the device coordinate system, (x2, y1) can refer to the coordinates of the upper right corner of the rectangular blurred area in the device coordinate system, (x1, y2) can refer to the coordinates of the lower left corner of the rectangular blurred area in the device coordinate system, and (x2, y2) can refer to the coordinates of the lower right corner of the rectangular blurred area in the device coordinate system. The device coordinate system can be a coordinate system established with the lower left corner of the mobile phone 100's display screen as the origin, the horizontal direction from left to right as the X direction, and the vertical direction from bottom to top as the Y direction.

[0042] It's understandable that the drawing instructions for each frame of the interface can also include background drawing instructions (an example of the first type of drawing instructions). Specifically, background drawing instructions are other drawing instructions besides visual effects drawing instructions during the image drawing process. For example... Figure 1 The transition interface 102 shown in (b) includes background drawing instructions for drawing the interface area of ​​the transition interface 102, drawing icons, and the text corresponding to the icons. In other words, background drawing instructions are used to draw the background elements of the interface. It can be understood that visual effects drawing instructions and background drawing instructions are all the drawing instructions that the GPU needs to draw when drawing images.

[0043] If the total number of drawing instructions N (N is a positive integer) required by the GPU is greater than the maximum number of instructions M (M is a positive integer) that the GPU can execute per unit time (e.g., the time corresponding to drawing one frame of an image), the GPU will not be able to complete all the drawing instructions corresponding to the transition interface 102 within the unit time. This will cause the GPU to draw the next interface before completing the transition interface 102. Specifically, after the GPU executes the background drawing instructions and some visual effects drawing instructions (e.g., drawing a blurred area, drawing the first shadow within the blurred area, and drawing the second shadow within the blurred area), it will draw the interface as shown in the image. Figure 1The notification center interface 103 shown in (c) is such that the visual rendering instruction for drawing a third shadow in the blurred area corresponding to the transition interface 102 cannot be completed, thus the transition interface 102 cannot be displayed. Upon detecting a user pulling down the notification bar, the phone 100 will... Figure 1 The application interface 101 shown in (a) directly switches to Figure 1 The notification center interface 103 shown in (c) causes the transition interface 102 to be missing, resulting in frame dropping.

[0044] Therefore, in some implementations, developers can set a relatively large maximum number of instructions (M) that the GPU can execute per unit time when developing a GPU. However, a large maximum number of instructions (M) can lead to higher power consumption in electronic devices during GPU image rendering.

[0045] In other words, drawing images based on the above image processing methods cannot reduce the power consumption of electronic devices while avoiding frame dropping.

[0046] Therefore, this application provides an image processing method. In this method, during the rendering of an image on a display screen, the electronic device can acquire visual effect rendering instructions and background rendering instructions corresponding to the image to be rendered, and determine the maximum number of instructions that the GPU can render per unit time under a preset power consumption threshold constraint of the electronic device. When rendering the image, if the total number of visual effect rendering instructions and background rendering instructions corresponding to the image is greater than the maximum number of instructions, or the execution time of the visual effect rendering instructions and background rendering instructions corresponding to the image is greater than a preset time (i.e., per unit time), meaning the GPU cannot finish rendering the image per unit time, the electronic device can reduce the number of visual effect rendering instructions, making the total number of visual effect rendering instructions and background rendering instructions corresponding to the image less than or equal to the maximum number of instructions that the GPU can render per unit time, to avoid frame dropping. It can be understood that since visual effect rendering instructions are rendering instructions for visual effects such as blurring, brightening, and shadows, and do not contain image elements such as text and icons, adjusting the visual effect rendering instructions will not reduce the display of information such as text and icons in the image. The adjusted visual effect rendering instructions can be called the target visual effect rendering instructions. When the GPU renders an image based on the target visual rendering instructions, the total power consumption of the CPU and GPU is less than or equal to the maximum power consumption of the electronic device.

[0047] For example, as mentioned earlier, the visual effect drawing instruction for drawing a third shadow in the blurred area corresponding to the transition interface 102 cannot be completed. Therefore, when the user pulls down the notification bar, the phone 100 will... Figure 1 The application interface 101 shown in (a) directly switches to Figure 1The notification center interface 103 shown in (c) causes the transition interface 102 to be missing, resulting in frame drops. Therefore, the electronic device can reduce the number of visual effect rendering instructions so that the total number of visual effect rendering instructions and background rendering instructions corresponding to the image is less than the maximum number of instructions that the GPU can execute per unit time, thus obtaining the target visual effect rendering instructions. For example, the target visual effect rendering instructions may include drawing a blurred area, drawing a first shadow within the blurred area, and drawing a second shadow within the blurred area, that is, deleting the visual effect rendering instruction of drawing a third shadow within the blurred area. It can be understood that, according to the target visual effect rendering instructions, the mobile phone 100 can complete the rendering of the transition interface 102. Furthermore, when the GPU renders the transition interface 102, the total power consumption of the CPU and GPU is less than or equal to the maximum power consumption of the electronic device.

[0048] The image processing methods mentioned in the embodiments of this application are described in detail below. For example... Figure 2 The diagram illustrates a flowchart of an image processing method, which can be executed by an electronic device, such as the aforementioned mobile phone 100. Specifically, the image processing method may include:

[0049] S201: The electronic device can acquire a first drawing instruction for a first image. The first drawing instruction includes a first number of first-type drawing instructions and a second number of second-type drawing instructions, wherein the first-type drawing instructions are used to draw background elements of the first image, and the second-type drawing instructions are used to draw visual effect elements of the first image.

[0050] It is understandable that when an electronic device detects a user switching the display screen, or a signal from an application switching the display screen, the GPU needs to draw the first image, and the GPU needs to execute the first drawing instructions corresponding to the first image. The first drawing instructions refer to the visual effects drawing instructions and background drawing instructions contained in the first image.

[0051] The background drawing instruction can be a first-type drawing instruction used to draw the background elements of the first image, and the number of background drawing instructions is a first quantity. The background elements can be other image elements that the GPU needs to draw besides the visual effect elements that need to be drawn when drawing an image.

[0052] Visual effects rendering instructions can be classified as second-type rendering instructions, used to render the visual effects elements of the first image, and the number of visual effects rendering instructions is the second maximum. These visual effects elements can include blurring, brightening, shadows, pixel expansion, etc. It can be understood that when the GPU renders the first image, it needs to execute all the visual effects rendering instructions and background rendering instructions contained in the first image to complete the rendering.

[0053] Furthermore, visual effects rendering instructions can include visual effects parameters as shown in Table 1:

[0054] Table 1

[0055]

[0056]

[0057] As shown in Table 1, the drawing instructions for creating blurred visual effects include parameters such as radius and grayscale blur parameters. The radius can refer to the radius of the blurred area, and the grayscale blur parameter can refer to the degree of blur. Other visual effect drawing instructions include parameters detailed in Table 1, which will not be elaborated upon here.

[0058] S202: When the electronic device predicts that the time required for the GPU to execute the first drawing instruction of the first image is greater than a preset time, the first drawing instruction is changed to a second drawing instruction. The second drawing instruction includes a first number of first-type drawing instructions and a third number of second-type drawing instructions. The time required to draw the first image based on the second drawing instruction is less than or equal to the preset time.

[0059] As we understand it, since monitors display images at a fixed refresh rate, which refers to the number of frames of an image that the monitor can update per second. For example, a monitor with a refresh rate of 60Hz can display 60 frames per second, while a monitor with a refresh rate of 120Hz can display 120 frames per second. Therefore, in order for the monitor to display images correctly without frame drops, the GPU needs to complete the image rendering within a unit of time. This unit of time is the time corresponding to rendering one frame of an image, that is, the generation time of a single frame, which can also be called the preset duration. In some specific implementations, electronic devices can determine the preset duration based on the GPU's clock speed and cycle count (such as the preset cycle count mentioned below). For example, the ratio of the GPU's clock speed to the cycle count can be used as the preset duration.

[0060] It is understandable that different electronic devices contain different GPU models, therefore, the maximum number of instructions a GPU can execute per unit of time varies. Electronic devices can obtain the GPU's hardware information table from information stored in memory. Based on the hardware information table, the electronic device can obtain information such as the GPU's clock speed, preset cycle count, and maximum power consumption. Specifically, the electronic device can directly obtain the GPU's clock speed and preset cycle count from memory. Alternatively, the electronic device can use the data relationship between the GPU's clock speed and preset cycle count, employing the clock speed to perform mathematical calculations to obtain the GPU's preset cycle count. The electronic device can also obtain the GPU's clock speed and preset cycle count under actual testing conditions through testing equipment; the specific methods are described in detail below. Based on the obtained correspondence between the GPU's clock speed and preset cycle count, the electronic device can construct a GPU load limit model. Based on the load limit model, the maximum number of instructions the GPU can execute per unit of time can be determined. For example, in the GPU load limit model, if the GPU's clock speed is 260MHz and the corresponding preset cycle count is 260 million, and the GPU can execute one instruction within one cycle count, then the corresponding number of instructions is 260 million.

[0061] In some embodiments, the electronic device may, based on a trained model (an example of the first model), change the first drawing instruction to a second drawing instruction when the duration of drawing the first image based on the first drawing instruction is longer than a preset duration (i.e., the number of instructions in the first drawing instruction is greater than the maximum number of instructions that the electronic device can draw within the preset duration). That is, the second number of second-type drawing instructions is adjusted to a third number of second-type drawing instructions, wherein the third number is less than the second number.

[0062] The second rendering instruction refers to the background rendering instruction and the predicted visual effect rendering instruction corresponding to the image. The prediction frame generation time is the time required for the GPU to execute the second rendering instruction, and the prediction frame generation time is less than the preset duration.

[0063] It is understandable that when the duration of drawing the first image based on the first drawing instruction is less than or equal to the preset duration, the second drawing instruction output by the trained model is the same as the first drawing instruction, that is, the second number of the second type of drawing instructions is maintained, which means that the third number is equal to the second number.

[0064] In some embodiments, the training process of the model may include:

[0065] It is understandable that electronic devices can acquire sample visual effect rendering instructions and sample background rendering instructions from sample images as sample data.

[0066] Electronic devices can train a visual effects parameter model based on sample data to obtain the visual effects parameter model (example of the first model). Specifically, the electronic device can input the training data into the visual effects parameter model and predict visual effects instructions and frame generation times based on sample visual effects rendering instructions and sample background rendering instructions to obtain predicted visual effects rendering instructions and predicted frame generation times. It can be understood that if the predicted frame generation time is greater than the label frame generation time, the electronic device can adjust the parameters of the visual effects parameter model and, based on the adjusted visual effects parameter model (example of the second model), re-predict visual effects instructions and frame generation times based on sample visual effects rendering instructions and sample background rendering instructions until the predicted frame generation time is less than or equal to the label frame generation time. It can be understood that when the GPU renders the image based on the image's background rendering instructions and the adjusted visual effects rendering instructions, no frame dropping will occur.

[0067] It is understood that the visual effect parameter model can be a neural network model, such as a convolutional neural network model, a recurrent neural network model, a long short-term memory network model, etc., and the embodiments of this application do not specifically limit it.

[0068] It is understandable that electronic devices can obtain hardware capability information tables from information stored in memory. These tables may include information such as the identifier, clock speed, preset maximum number of instructions that can be executed per unit time (hereinafter referred to as preset cycle count), and maximum power consumption of different CPU models. They may also include information such as the identifier, clock speed, preset cycle count, and maximum power consumption of different GPU models. Therefore, based on the clock speed and preset cycle count of different CPU models in the hardware capability information table, the electronic device can construct a load limit model describing the relationship between the clock speed and cycle count of different CPU models. Similarly, based on the clock speed and preset cycle count of different GPU models in the hardware capability information table, the electronic device can construct a load limit model describing the relationship between the clock speed and cycle count of different GPU models. It is understandable that, according to the load limit model, the electronic device can obtain the maximum number of instructions that the GPU can execute per unit time.

[0069] In this embodiment of the application, when drawing an image, if the total number of visual effect drawing instructions and background drawing instructions corresponding to the image is greater than the maximum number of instructions, that is, the graphics processor cannot draw the image in a unit of time, the electronic device can reduce the visual effect drawing instructions so that the total number of visual effect drawing instructions and background drawing instructions corresponding to the image is less than or equal to the maximum number of instructions that the graphics processor can draw in a unit of time, so as to avoid frame dropping.

[0070] The following section uses the construction of a GPU load limit model as an example to introduce the process of constructing CPU and GPU load limit models.

[0071] For example, such as Figure 3A As shown, the electronic device can obtain the preset cycle counts of GPUs with clock speeds of 260MHz, 300MHz, 380MHz, 420MHz, 480MHz, and 680MHz respectively from the information stored in the memory. Therefore, the electronic device can establish a load limit model relating the clock speed of different GPU models to the preset cycle counts.

[0072] In some optional implementations, the higher the GPU's clock speed, the higher the GPU's preset cycle count. The relationship between the GPU's clock speed and preset cycle count is that the GPU's preset cycle count increases as the clock speed increases.

[0073] It is understandable that, in some alternative implementations, the hardware capability information table can store a load limit model describing the relationship between the clock speed and preset cycle count of different GPU models in graphical form; it can also store a load limit model describing the relationship between the clock speed and estimated cycle count of different GPU models in graphical form; or it can store a load limit model describing the relationship between the clock speed and measured cycle count of different GPU models in graphical form. Thus, electronic devices can directly obtain the GPU's load limit model from the information stored in memory.

[0074] Understandably, the hardware information table can also record the correspondence between the clock speed and preset cycle number of different GPU models in the form of a list, bar chart, pie chart, or bar graph, and there are no restrictions here.

[0075] However, due to factors such as device aging and environmental conditions, the actual number of cycles of a GPU may be less than the preset number of cycles recorded in the hardware capability information table during practical applications. Therefore, in some alternative implementations, the electronic device can construct a load limit model relating the clock speed and estimated number of cycles of different GPU models based on the clock speed, preset number of cycles, and constraint percentage of different GPU models in the hardware capability information table.

[0076] Specifically, the electronic device can use the product of the preset number of cycles for different GPU models and the first constraint coefficient as the estimated number of cycles for the GPU. This allows for the construction of a load limit model relating the clock speed and estimated number of cycles for different GPU models. The first constraint coefficient can be 95%. It is understood that the first constraint coefficient can be set according to the user's specific needs. The 95% listed in the embodiments of this application is only one optional implementation and does not represent the only implementation.

[0077] In some alternative implementations, the higher the GPU's clock speed, the higher the estimated number of cycles. The relationship between the GPU's clock speed and the estimated number of cycles is that the estimated number of cycles increases with the increase of the clock speed.

[0078] To improve the accuracy of GPU load limit models, electronic devices can also construct load limit models based on the GPU's clock speed and measured cycle count, establishing a relationship between the clock speed and measured cycle count for different GPU models.

[0079] In some specific implementations, electronic devices can obtain the number of cycles of the GPU during multiple image rendering processes, and determine the actual number of cycles of the GPU based on the number of cycles during multiple image rendering processes.

[0080] For example, an electronic device can obtain multiple cycles when frame drops occur during the rendering of different images by the GPU, and use any one of these multiple cycles as the actual number of cycles, or the average of the multiple cycles as the actual number of cycles, or the product of any one of the multiple cycles and a second constraint coefficient as the actual number of cycles, or the result of the average of the multiple cycles and the second constraint coefficient as the actual number of cycles. This application embodiment does not limit the specific implementation. The second constraint coefficient can be 95%. It is understood that the second constraint coefficient can be set according to the user's specific needs. The 95% listed in this application embodiment is only one optional implementation method and does not represent the only implementation method.

[0081] In some alternative implementations, the higher the GPU clock speed, the higher the GPU's measured cycle count. The relationship between the GPU clock speed and the measured cycle count is that the GPU's measured cycle count increases with the increase of the clock speed.

[0082] To balance the performance and power consumption of electronic devices, devices can also impose comprehensive performance and power consumption constraints on GPUs and CPUs.

[0083] In some alternative implementations, the electronic device can determine the maximum number of instructions the GPU can execute per unit time under performance constraints based on the number of cycles corresponding to the GPU in the load-limiting model. For example, the product of the GPU's preset instruction count, estimated instruction count, or actual instruction count and the constraint percentage (498) can be used as the maximum number of instructions the GPU can execute per unit time under performance constraints, i.e., the overall GPU performance baseline. Furthermore, the electronic device can obtain the GPU's maximum cycle count (375) when the electronic device's power consumption is at maximum, and use this as the GPU's cycle count under power constraints, i.e., the overall GPU power consumption baseline.

[0084] Similarly, electronic devices can multiply the CPU's preset instruction count, estimated instruction count, or actual instruction count by the constraint percentage, 134e. 5 This represents the CPU's instruction count under performance constraints, i.e., the overall CPU performance baseline. Furthermore, it indicates the maximum number of instructions the CPU can execute per unit time when the electronic device's power consumption is at its maximum. 5 , which is the maximum number of instructions that a CPU can execute per unit of time under power consumption constraints, i.e., the total baseline of CPU power consumption.

[0085] By considering different models and versions of graphics processing and the extreme load under the current hardware settings, a differentiated load limit model can be obtained.

[0086] The aforementioned constraint percentage can be 90%. It is understood that the constraint percentage can be set according to the user's specific needs. The 90% listed in the embodiments of this application is only an optional implementation method and does not represent the only implementation method.

[0087] It is understandable that during the image rendering process, the GPU needs to execute not only visual rendering instructions, but also storage instructions, instructions for communicating with other electronic devices, and voice instructions. The number of cycles the GPU executes for storage instructions, instructions for communicating with other electronic devices, and voice instructions can be called the base workload, while the number of cycles the GPU executes for visual rendering instructions can be called the effects workload.

[0088] Similarly, during the image rendering process, the CPU needs to execute not only visual rendering instructions, but also storage instructions, instructions for communicating with other electronic devices, and voice instructions. The number of storage instructions, instructions for communicating with other electronic devices, and voice instructions executed by the CPU can be called the base workload, while the number of visual rendering instructions executed by the CPU can be called the effects workload.

[0089] In some specific implementation methods, such as Figure 4 As shown, electronic device 100 can be connected to detection device 200. Thus, detection device 200 can obtain the following data during the process of electronic device 100 rendering an image under performance and power constraints: GPU performance base load is 187, power base load is 155, visual performance load is 311, and visual power load is 202. Similarly, detection device 200 can also obtain the following data during the process of electronic device 100 rendering an image under performance and power constraints: CPU performance base load is 98e. 5 The power consumption base load is 55e. 5 The visual effects performance load is 36e. 5 The visual power consumption load is 44e. 5 .

[0090] The following section introduces the methods for obtaining training data.

[0091] like Figure 3B As shown, the electronic device 100 can be connected to the detection device 200, which can display the sample visual effect rendering instructions, sample background rendering instructions, and label frame generation time of the sample image. It can be understood that the electronic device can use the sample visual effect rendering instructions and sample background rendering instructions of the sample image acquired by the detection device 200 as training data.

[0092] It is understandable that, in order to obtain the sample visual effect rendering instructions and sample background rendering instructions for the sample image, the visual effects of the sample image can be decomposed. Visual effects such as shadows, brightening, blurring, and pixel expansion can be extracted from the sample image, thereby determining the sample visual effect rendering instructions corresponding to each visual effect. Here, the sample visual effect rendering instructions can refer to instructions for rendering visual effect elements. For example, as mentioned earlier, for... Figure 1 The transition interface 102 shown in (b) can include drawing a blurred area, drawing a first shadow within the blurred area, drawing a second shadow within the blurred area, and drawing a third shadow within the blurred area. The blurred area can be a rectangular area, and the coordinates of the four vertices of the rectangular area can be (x1, y1), (x2, y1), (x1, y2), and (x2, y2).

[0093] The sample background drawing instruction can refer to the instruction of the electronic device 100 to draw background elements. For example, for Figure 1 The transition interface 102 shown in (b) may include drawing icon 1, drawing icon 2, and drawing icon 3 on the application interface 101.

[0094] To facilitate intuitive user experience, the testing device 200 can count the number of sample visual effect rendering commands and sample background rendering commands, and... Figure 3B The visual effects rendering instructions are represented in the form of a bar chart, where the height of the bar chart represents the number of rendering instructions. By distinguishing between visual effects rendering instructions and background rendering instructions, visual effects rendering instructions can be analyzed separately, enabling a highly accurate visual effects load assessment mechanism.

[0095] In some specific implementations, continue as follows Figure 4 As shown, the detection device 200 can display the sample visual effect drawing instructions, sample background drawing instructions, and the time required to draw the image, i.e., the tag frame generation time, executed by the electronic device 100 during the image drawing process under performance or power consumption constraints.

[0096] In some specific implementations, taking a display showing 60 frames per second as an example, the generation time for a single frame is 16.6ms. If the GPU's frame generation time for the first frame is longer than the generation time for a single frame, then frame dropping has occurred, which can be called first-look visual effect frame dropping. If the GPU continuously renders two frames, and the generation time for both frames is greater than twice the generation time for a single frame (this twice-times single-frame generation time could be 33ms), then frame dropping has occurred, which can be called visual effect frame dropping. If the GPU continuously renders six frames, and the generation time for all six frames is greater than six times the generation time for a single frame (this six times-times single-frame generation time could be 100ms), then frame dropping has occurred, which can be called general frame dropping.

[0097] It is understandable that the detection device 200 can determine whether frame dropping has occurred based on the detected tag frame generation time. For example, assuming a display shows 60 frames per second, if the tag frame generation time is greater than 16.6ms, the detection device 200 can determine that first-look visual frame dropping has occurred. If the tag frame generation time is greater than 33ms, the detection device 200 can determine that visual frame dropping has occurred. If the tag frame generation time is greater than 100ms, the detection device 100 can determine that general frame dropping has occurred.

[0098] The training process of the visual effects parameter model is described below.

[0099] In some methods of pre-training visual effects parameter models, such as Figure 3C As shown, the electronic device can acquire training data, which may include sample visual effect rendering instructions and sample background rendering instructions. Next, the electronic device can input the training data into a visual effect parameter model. Using the visual effect parameter model, based on the sample visual effect rendering instructions and sample background rendering instructions, it performs visual effect instruction prediction and frame generation time prediction to obtain the predicted visual effect rendering instructions and predicted frame generation times. Furthermore, the electronic device can determine whether the pre-trained visual effect parameter model is well-trained based on the relationship between the predicted frame generation time and the label frame generation time (the generation time of a single frame image).

[0100] Specifically, when the predicted frame generation time is greater than the label frame generation time, the electronic device can adjust the parameters of the visual effect parameter model, and based on the adjusted visual effect parameter model, re-predict the visual effect command and the frame generation time based on the sample visual effect rendering command and the sample background rendering command, until the predicted frame generation time is less than or equal to the label frame generation time, thus obtaining the trained visual effect parameter model.

[0101] The visual effects parameter model can be a neural network model, such as a convolutional neural network model, a recurrent neural network model, a long short-term memory network model, etc., and the embodiments of this application do not specifically limit it.

[0102] It is understandable that the image processing methods mentioned above can be applied to image processing systems in electronic devices. In some alternative implementations, the image processing system may also include, for example... Figure 5 The structure shown.

[0103] like Figure 5 The diagram shows a schematic representation of an image processing system. The image processing methods mentioned in the embodiments of this application will be described below in conjunction with this image processing system.

[0104] It is understood that the image processing system may include a hardware modeling module 501, a performance simulation model 502, a visual effect parameter optimization module 503, a rendering process 504, an XPU management module 505, and a visual effect configuration module 506.

[0105] The hardware modeling module 501 can be used to read the hardware capability information table and construct load limit models for different CPU and GPU models based on the CPU and GPU clock speeds and preset cycle counts in the hardware capability information table. It can also be used to read the hardware capability information table and construct load limit models for different CPU and GPU models based on the CPU and GPU clock speeds and measured cycle counts in the hardware capability information table. Furthermore, it can also be used to construct load limit models for different CPU and GPU models based on the CPU and GPU clock speeds, preset cycle counts, and the maximum number of instructions that the CPU and GPU can execute per unit time under performance and power consumption constraints in the hardware capability information table.

[0106] The performance simulation model 502 can be used to simulate the sample visual effect rendering instructions, sample background rendering instructions, and frame generation time required to render the image executed by the electronic device 100 under performance and power consumption constraints. Furthermore, it can be used to decompose the visual effect rendering instructions and background rendering instructions, and identify the key visual effect rendering instructions that affect the load of the electronic device.

[0107] The visual effects parameter optimization module 503 is used to optimize the visual effects rendering instructions of the image to be rendered if, during the GPU's image rendering process, the total number of visual effects rendering instructions and background rendering instructions for the graphics exceeds the maximum number of instructions the GPU can execute per unit time, or if the duration of the GPU executing visual effects rendering instructions and background rendering instructions exceeds a preset duration. For example, it can reduce the number of visual effects rendering instructions and compare the duration of rendering the image to be rendered by the electronic device based on the reduced visual effects rendering instructions with the preset duration. If the duration is less than or equal to the preset duration, the image is submitted to the graphics processor for rendering. This ensures that frame dropping is avoided while reducing the power consumption of the electronic device.

[0108] It is understandable that the specific implementation methods of the hardware modeling module 501, the performance simulation model 502, and the visual effect parameter optimization module 503 can be referred to the above text. To avoid repetition, they will not be repeated here.

[0109] The rendering process 504 can be used to manage the execution operations of the electronic device. These operations can be related to tasks involving drawing images, such as display rendering, script execution, and event handling. Specifically, the rendering process 504 can obtain visual effect drawing instructions and background drawing instructions submitted by the XPU management module 505, and can also submit the obtained visual effect drawing instructions and background drawing instructions to the performance simulation module 502 and the visual effect parameter optimization module 503 for use.

[0110] The XPU management module 505 may include a CPU management submodule and a GPU management submodule. The CPU management submodule can be used to obtain background drawing instructions and transmit them to the GPU management submodule.

[0111] The visual effects configuration module 506 can be used to obtain target visual effects rendering instructions from the visual effects parameter optimization module 503 and transmit them to the GPU management submodule. Thus, the GPU management submodule can be used to render the image to be rendered based on the target visual effects rendering instructions and the background rendering instructions.

[0112] It is understood that, in this embodiment of the application, the image processing method described above can be executed by an electronic device. The hardware structure of the electronic device is described below.

[0113] Figure 6 A schematic diagram of the hardware structure of an electronic device is shown. It is understood that the structure illustrated in this embodiment of the invention does not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0114] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0115] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The processor can be used to execute the image processing methods mentioned in the embodiments of this application.

[0116] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0117] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0118] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0119] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. The GPIO interface can be configured via software. The USB interface 130 is a USB-compliant interface, specifically a Mini USB interface, a MicroUSB interface, a USB Type-C interface, etc.

[0120] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0121] The charging management module 140 is used to receive charging input from the charger. The power management module 141 is used to connect the battery 142, and the charging management module 140 is connected to the processor 110.

[0122] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0123] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 150 can provide solutions for wireless communication applications in electronic devices, including 2G / 3G / 4G / 5G. The modem processor can include a modulator and a demodulator. Wireless communication module 160 can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0124] In some embodiments, antenna 1 of the electronic device is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling the electronic device to communicate with networks and other devices via wireless communication technology.

[0125] Electronic devices implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0126] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include one or N displays 194, where N is a positive integer greater than 1.

[0127] Electronic devices can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0128] The ISP (Internet Service Provider) processes data fed back from camera 193. Camera 193 is used to capture still images or videos. The digital signal processor (DSP) processes digital signals, including digital image signals and other digital signals. For example, when the electronic device selects a frequency, the DSP performs Fourier transforms on the frequency energy. The video codec is used to compress or decompress digital video. The electronic device can support one or more video codecs. The NPU (Neural Processing Unit) is a neural network (NN) computing processor that, by borrowing from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, rapidly processes input information and can continuously learn. The NPU enables intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0129] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0130] Internal memory 121 can be used to store executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in internal memory 121 and / or instructions stored in memory disposed within the processor.

[0131] Electronic devices can implement audio functions such as music playback and recording through audio modules 170, speakers 170A, receivers 170B, microphones 170C, headphone jacks 170D, and application processors.

[0132] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0133] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. Electronic devices can listen to music or make hands-free calls through the speaker 170A. The receiver 170B, also known as a "handpiece," is used to convert audio electrical signals into sound signals. When an electronic device answers a phone call or voice message, it can be held close to the user's ear. The microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. The headphone jack 170D is used to connect wired headphones.

[0134] The pressure sensor 180A senses pressure signals and converts them into electrical signals. The gyroscope sensor 180B determines the motion posture of the electronic device. The barometric pressure sensor 180C measures air pressure. The magnetic sensor 180D includes a Hall effect sensor. The accelerometer sensor 180E detects the magnitude of acceleration in various directions (typically three axes) of the electronic device. The distance sensor 180F measures distance; the electronic device can measure distance using infrared or laser. The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a photodetector, such as a photodiode. The ambient light sensor 180L senses ambient light intensity. The fingerprint sensor 180H collects fingerprints. The temperature sensor 180J detects temperature. The touch sensor 180K is also called a "touch device." The bone conduction sensor 180M acquires vibration signals.

[0135] Buttons 190 include a power button, volume buttons, etc. A motor 191 can generate vibration feedback. An indicator 192 can be an indicator light, used to indicate charging status, battery level changes, and also to indicate messages, missed calls, notifications, etc. A SIM card interface 195 is used to connect a SIM card.

[0136] It is understood that the structures illustrated in this application do not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0137] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0138] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0139] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0140] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0141] Although this application has been illustrated and described with reference to certain embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. An image processing method applied to electronic devices, characterized in that, include: Obtain a first drawing instruction for a first image, the first drawing instruction including a first number of first-type drawing instructions and a second number of second-type drawing instructions, wherein the first-type drawing instructions are used to draw the background elements of the first image, and the second-type drawing instructions are used to draw the visual effect elements of the first image; If the duration of drawing the first image based on the first drawing instruction is greater than a preset duration, the first drawing instruction is changed to a second drawing instruction. The second drawing instruction includes the first number of first-type drawing instructions and the third number of second-type drawing instructions, wherein the duration of drawing the first image based on the second drawing instruction is less than or equal to the preset duration.

2. The method according to claim 1, characterized in that, The visual effect elements include at least one of the following: Blur, brighten, shadow, pixel expansion.

3. The method according to claim 1, characterized in that, The preset duration is determined by the clock speed and cycle number of the graphics processor in the electronic device.

4. The method according to claim 1, characterized in that, The step of changing the first drawing instruction to a second drawing instruction when the duration of drawing the first image based on the first drawing instruction is greater than a preset duration includes: The first drawing instruction is input into the first model. If the duration of drawing the first image based on the first drawing instruction is greater than a preset duration, the first model outputs the second drawing instruction.

5. The method according to claim 4, characterized in that, The first model was trained in the following way: Acquire sample data, wherein the sample data includes sample visual effect drawing instructions and sample background drawing instructions for sample images; The sample data is input into the second model, and the second model is used to predict the predicted visual effect rendering instructions and the predicted frame generation time corresponding to the sample data. The predicted frame generation time is the time required for the graphics processor to execute the predicted visual effect rendering instructions and the sample background rendering instructions to render the image. If the predicted frame generation time is longer than the tag frame generation time, the parameters of the second model are adjusted to obtain the second model with adjusted parameters. The sample data is input into the second model after the parameters are adjusted, until the second model after the parameters are adjusted predicts that the generation time of the prediction frame corresponding to the sample data is less than or equal to the generation time of the label frame. The second model with the adjusted parameters is used as the first model.

6. The method according to claim 3, characterized in that, The clock speed and cycle number of the graphics processor can be obtained by any of the following methods: Obtain the clock speed and cycle count of the graphics processor from the hardware information table stored in the electronic device; The clock speed of the graphics processor is obtained from the hardware information table stored in the electronic device, and the number of cycles of the graphics processor is determined based on the clock speed of the graphics processor. The clock speed of the graphics processor is obtained from the hardware information table stored in the electronic device, and the number of cycles of the graphics processor is determined based on the clock speed of the graphics processor and the first constraint coefficient.

7. The method according to claim 5, characterized in that, The sample visual effect rendering instructions are obtained by decomposing the visual effects of the sample image.

8. The method according to claim 7, characterized in that, The visual effects of the sample images are decomposed in the following way: The sampling visual effect drawing instructions are obtained by acquiring the instructions corresponding to each visual effect element of the sample image during the process of the electronic device drawing the sample image.

9. An electronic device, characterized in that, Includes: a memory for storing instructions executed by one or more processors of the electronic device. And a processor, which is one or more processors of the electronic device, for performing the image processing method according to any one of claims 1 to 8.

10. A readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the image processing method according to any one of claims 1 to 8.

11. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by an electronic device, enable the electronic device to perform the image processing method as described in any one of claims 1 to 8.

Images