Method and electronic device for assisting application development

Abstract

One embodiment of the invention provides an electronic device. The electronic device includes a memory configured to store a program. The electronic device also includes a central processing unit (CPU) configured to read the program to execute the assistance module and the application program. The assistance module is configured to receive a query from the application about an image processing function. The assistance module is further configured to send a response message to the application. The response message indicates processing information of an image processing function performed by a specific hardware component of the electronic device.

Description

[0001] Cross-referencing

[0002] This application claims priority to U.S. Provisional Application Serial No. 63 / 690,836, filed on 2024 / 09 / 05, the entire contents of which are incorporated herein by reference. [Technical Field]

[0003] This disclosure relates to application development, and in particular to an assist module for image processing functions. [Background Technology]

[0004] With the rise of short videos and online streaming, numerous related applications have been developed. These applications typically offer image processing capabilities. To achieve the best user experience, it's crucial to optimize power consumption and performance when devices execute these application functions. Generally, using hardware components to perform this function consumes less power than using a central processing unit (CPU) to run software. However, different platforms (such as mobile phones) have different hardware configurations. Therefore, for the same function, some platforms may be able to perform it using hardware components, while others may not. Because applications and their developers cannot know whether a particular platform can perform a function using hardware components, developers often simply configure their applications to perform the function using software. This results in higher power consumption.

[0005] Therefore, a solution or method is needed to solve the above problems. [Summary of the Invention]

[0006] One embodiment of this disclosure provides an electronic device. The electronic device includes a memory configured to store a program. The electronic device also includes a central processing unit (CPU) configured to read the program to execute an assistance module and an application. The assistance module is configured to receive queries from the application regarding image processing functions. The assistance module is further configured to send a response message to the application. The response message indicates processing information of the image processing function performed by a specific hardware component of the electronic device.

[0007] In some embodiments, processing information indicates whether the image processing function can be performed by a specific hardware component. This specific hardware component is a digital signal processor, image signal processor, image scaler, encoder, or monitor. In some embodiments, the assistance module is configured to consult a table stored in memory to determine whether the image processing function can be performed by a specific hardware component of the electronic device.

[0008] In some embodiments, when a response message indicates that the image processing function can be performed by a specific hardware component of the electronic device, the application is configured to perform the image processing function using that specific hardware component. When a response message indicates that the image processing function cannot be performed by a specific hardware component of the electronic device, the application is further configured to perform the image processing function using the CPU or GPU.

[0009] In some embodiments, the assistance module is further configured to receive setting parameters for an image processing function from an application. The processing information further includes a quality metric. The quality metric indicates the result of a specific hardware component performing the image processing function based on the setting parameters. The specific hardware component is a digital signal processor, image signal processor, image scaler, encoder, or monitor.

[0010] In some embodiments, the application is configured to perform image processing functions using specific hardware components of the electronic device, rather than the CPU and GPU, in response to determining that quality metrics meet requirements. The application is further configured to adjust setting parameters and transmit the adjusted settings parameters back to the assistance module, or to perform image processing functions using the CPU or GPU in response to determining that quality metrics do not meet requirements.

[0011] In some embodiments, the assistance module is further configured to consult a table stored in memory to determine quality metrics. In some embodiments, the assistance module is further configured to determine quality metrics by running a simulation based on set parameters.

[0012] In some embodiments, quality metrics include processing time, power consumption, peak signal-to-noise ratio, or combinations thereof. In some embodiments, image processing functions are edge enhancement, image scaling, image rotation, super-resolution, or transcoding.

[0013] One embodiment of this disclosure provides a method for assisting application development. The method includes an operation in which a central processing unit (CPU) of an electronic device reads a program stored in the memory of the electronic device to execute an assisting module and an application. The method further includes an operation in which the assisting module receives a query from the application regarding image processing functions. The method further includes an operation in which the assisting module sends a response message to the application. The response message indicates processing information of the image processing function performed by a specific hardware component of the electronic device.

[0014] The application development method and electronic device of the present invention enable the application to understand the support of the hardware processing module of the electronic device for the application's image processing function, thereby helping the application to select CPU, GPU or other suitable hardware processing modules to execute the image processing function, while ensuring a balance between the execution effect of the application's image processing function and the performance of the electronic device. [Attached Image Description]

[0015] A more complete understanding of this disclosure can be achieved by reading the following detailed description and examples, and by referring to the accompanying drawings, in which:

[0016] Figure 1 This is a block diagram of an electronic device according to an embodiment of the present disclosure;

[0017] Figure 2 This is a flowchart of a method for assisting application development according to an embodiment of the present disclosure;

[0018] Figure 3 This is a flowchart of a method for assisting application development according to an embodiment of the present disclosure;

[0019] Figure 4 This is a flowchart of a method for assisting application development according to an embodiment of the present disclosure.

Detailed Implementation Methods

[0020] Detailed description

[0021] The following description is intended to illustrate the general principles of this disclosure and should not be considered limiting. The scope of this disclosure is best determined by the appended claims.

[0022] Figure 1 This is a block diagram of an electronic device 100 according to an embodiment of the present disclosure. For example, the electronic device 100 may be a mobile device, a wearable device, a wireless communication device, or a computing device. In some embodiments, the electronic device 100 may be implemented as a smartphone, tablet computer, or laptop computer. The electronic device 100 includes a central processing unit (CPU) 110, a memory 120, a graphics processing unit (GPU) 130, and an image and media processing unit 140.

[0023] CPU 110 controls the operation of electronic device 100 and provides the processing power required to execute operating systems, programs, software, modules, applications, and functions of electronic device 100. In some embodiments, CPU 110 may be implemented as an electronic component comprising transistors, diodes, capacitors, resistors, or inductors. These components are configured and arranged to perform methods consistent with embodiments of this disclosure. In other words, CPU 110 is a dedicated machine specifically configured to perform a particular task consistent with embodiments of this disclosure.

[0024] Memory 120 stores data and instructions required by CPU 110. Memory 120 may include non-volatile memory, such as read-only memory (ROM) and flash memory. Memory 120 may also include volatile memory, such as dynamic random access memory (DRAM) and static random access memory (SRAM). In some embodiments, memory 120 stores program 121, such as computer-readable instructions. Program 121 can be read by CPU 110. When program 121 is read and executed by CPU 110, program 121 causes CPU 110 to execute (or implement) assist module 111 and application program 112, and perform methods consistent with embodiments of this disclosure. Assist module 111 and application program 112 are software modules.

[0025] In some embodiments, application 112 has image processing functions. For example, image processing functions may include edge enhancement, image scaling, image rotation, super-resolution, or transcoding. Edge enhancement functions are configured to improve the sharpness or quality of an image or video. Image scaling functions are configured to resize an image. Super-resolution functions are configured to enhance the details of an image or video or increase its resolution. Transcoding functions are configured to transcode an image or video or increase its transcoding speed. Electronic device 100 may use software or hardware to perform (or implement) image processing functions. Implementing image processing functions using software may mean that electronic device 100 uses CPU 110 or GPU 130 to execute software to implement image processing functions. Implementing image processing functions using hardware may mean that electronic device 100 uses specific hardware components of electronic device 100 to perform image processing functions. These hardware components may or may not cooperate with CPU 110 or GPU 130 to implement image processing functions. In some embodiments, the specific hardware component used to perform the image processing function (i.e., the hardware component in electronic device 100 other than CPU 110 or GPU 130) does not execute instructions from CPU 110. In some embodiments, the hardware component is configured to perform a fixed operation. Specifically, the hardware component may initially be in a sleep mode or idle mode. Then, the hardware component is woken up to receive a task. Because the hardware component is performing a fixed operation, it processes the task without instructions from CPU 110. After the hardware component completes the task, it returns the result and re-enters sleep mode or idle mode. For example, the hardware component is an image and media processing unit 140, an encoder, or a monitor. In some embodiments, the image and media processing unit includes a digital signal processor, an image signal processor, and an image scaler.

[0026] Typically, performing image processing functions using specific hardware components can achieve lower power consumption than implementing the same function using software running on CPU 110 or GPU 130. However, different platforms (e.g., mobile phones) have different hardware configurations. Therefore, application 112 or its developer cannot determine whether the image processing function can be implemented using specific hardware components, or the performance of those components. The assistance module 111 in this embodiment allows application 112 or its developer to know whether the image processing function can be implemented using specific hardware components. Furthermore, the assistance module 111 allows application 112 or its developer to know the result of the image processing function performed by the specific hardware components. Therefore, application 112 can determine how to perform the image processing function.

[0027] Figure 2 This is a flowchart of a method 20 for assisting application development according to an embodiment of this disclosure. In operation 21, CPU 110 reads program 121 stored in memory 120 to execute assistance module 111 and application 112. In operation 22, assistance module 111 receives a query from application 112 regarding an image processing function. The query indicates the image processing function of application 112. For example, the query is a message that includes the name of the image processing function or an identifier indicating the image processing function. In operation 23, assistance module 111 sends a response message to application 112. The response message indicates processing information of the image processing function (i.e., the image processing function indicated by the query) performed by a specific hardware component of electronic device 100. In other words, the processing information is processing information of the image processing function associated with a specific hardware component. For example, the processing information may indicate whether the image processing function can be performed by specific hardware information, and / or the result of the specific hardware component performing the image processing function. In some embodiments, the specific hardware component may be a digital signal processor, an image signal processor, an image scaler, an encoder, or a monitor. In some embodiments, specific hardware components may differ from CPU 110 and GPU 130, and may work in conjunction with CPU 110 and GPU 130 to implement image processing functions.

[0028] In some embodiments, the assistance module 111 consults (looks up) a table stored in memory 120 to determine whether an image processing function can be performed by a particular hardware component. Memory 120 stores information in tabular form regarding whether each image processing function can be performed by a particular hardware component. For example, rows of the table represent image processing functions, and columns represent indicators indicating whether an image processing function can be performed by a particular hardware component. In some embodiments, the table may further record which hardware components are capable of performing the image processing functions. For example, rows of the table represent image processing functions, and columns represent hardware components capable of performing the corresponding image processing functions. The table may differ in different electronic devices. In some embodiments, the developer of the assistance module 111 collects data from different electronic devices (e.g., through experimentation) to construct the table.

[0029] Figure 3 This is a flowchart of a method 30 for assisting application development according to an embodiment of the present disclosure. In operation 31, CPU 110 reads program 121 stored in memory 120 to execute assist module 111 and application 112. In operation 32, assist module 111 receives a query from application 112 regarding image processing functions. In operation 33, assist module 111 sends a response message to application 112. The response message indicates processing information of an image processing function performed by a specific hardware component of the electronic device. Operations 31-33 may be similar to operations 21-23.

[0030] In operation 34, application 112 determines, based on processing information, whether the image processing function can be performed by a specific hardware component of electronic device 100. When the processing information indicates that the image processing function can be performed by the specific hardware component, application 112 performs operation 35. When the processing information indicates that the image processing function cannot be performed by the hardware component, application 112 performs operation 36. In operation 35, application 112 performs the image processing function using the specific hardware component (e.g., image and media processing unit 140, encoder, or monitor). In operation 36, application 112 performs the image processing function using a CPU or GPU (executing a software program, such as program 121).

[0031] Figure 4This is a flowchart of a method 40 for assisting application development according to an embodiment of the present disclosure. In operation 41, CPU 110 reads program 121 stored in memory 120 to execute assistance module 111 and application 112. In operation 42, assistance module 111 receives a query from application 112 regarding an image processing function. In operation 43, assistance module 111 sends a response message to application 112. The response message indicates processing information of the image processing function performed by a specific hardware component of the electronic device. Operations 41-43 may be similar to operations 21-23. In operation 44, application 112 determines whether the image processing function can be performed by the specific hardware component based on the response message. When the processing information indicates that the image processing function can be performed by the specific hardware component, application 112 performs operation 45. When the processing information indicates that the image processing function cannot be performed by the specific hardware component, application 112 performs operation 49. In operation 49, application 112 performs the image processing function using CPU or GPU (executing a software program, such as program 121).

[0032] In operation 45, the assistance module 111 receives setting parameters for an image processing function (i.e., an image processing function indicated by a query) from the application 112. These setting parameters are parameters required for a specific hardware component to perform the image processing function. For example, when the image processing function is image scaling, the setting parameters might be the original size and the target size, such as 720P and 1080P. When the image processing function is edge enhancement, the setting parameters might be the target sharpness level.

[0033] In operation 46, the assistant module 111 sends a response message to the application 112. This response message includes processing information such as a quality index. This quality index indicates the result of a specific hardware component performing an image processing function based on set parameters. In some embodiments, the quality index includes processing time, power consumption, peak signal-to-noise ratio (PSNR), or a combination thereof. Specifically, processing time is the time consumed by the specific hardware component to complete the image processing function based on the set parameters. Power consumption is the power consumed by the specific hardware component to perform the image processing function based on the set parameters. PSNR is the peak signal-to-noise ratio of the image generated by the specific hardware component through performing the image processing function based on the set parameters.

[0034] It is important to note that in operation 46, the specific hardware component has not yet performed image processing functions. The quality metrics transmitted by the assistance module 111 are quality metrics estimated by the assistance module 111. For example, when the image processing function is image scaling, and the settings are 720P and 1080P, the quality metrics might be the processing time and power consumption of the specific hardware component in converting a 720P image to a 1080P image. When the image processing function is edge enhancement, and the settings are target sharpness, the quality metrics might be the processing time, power consumption, and PSNR of the specific hardware component in converting the image to the target sharpness.

[0035] In some embodiments, the assistance module consults a table stored in memory 120 to determine quality metrics. The table records the quality metrics, each corresponding to an image processing function and a setting parameter (or a set of setting parameters). For example, Table 1 shows a portion of the table contents stored in memory 120.

[0036]

[0037]

[0038] The table may differ across different electronic devices. In some embodiments, the developer of the assist module 111 constructs the table by collecting data from different electronic devices (e.g., through experimentation).

[0039] In other embodiments, the assistance module 111 determines quality metrics by running a simulation based on set parameters. The assistance module 111 may use simulation or algorithms based on set parameters to determine the quality metrics. In some embodiments, the assistance module 111 may also activate specific hardware components before determining the quality metrics.

[0040] In operation 47, application 112 determines whether the quality metric meets the requirements. In some embodiments, application 112 determines whether the quality metric meets the requirements by comparing the quality metric to a threshold. When the quality metric meets the requirements (e.g., the quality metric is above or below the threshold), the application performs operation 48. In operation 48, application 112 performs image processing functions using specific hardware components. When the quality metric does not meet the requirements, operation 45 is re-executed. In other words, the application can adjust the setting parameters to different values ​​and transmit the adjusted setting parameters to the assistance module again. In other embodiments, when the quality metric does not meet the requirements, the application performs operation 49.

[0041] Embodiments of this disclosure allow applications or developers to understand whether image processing functions can be implemented using specific hardware components, and the results of performing image processing functions using specific hardware components. Applications can then use specific hardware components to perform image processing functions where possible. Therefore, power consumption and performance are improved, resulting in a better user experience.

[0042] Although this disclosure has been described by way of example and preferred embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. Rather, this disclosure is intended to cover various modifications and similar arrangements. Therefore, the scope of the appended claims should be given the broadest interpretation to cover all such modifications and similar arrangements.

Claims

1. An electronic device, comprising: a memory configured to store a program; a central processing unit (CPU) configured to read the program to execute an assistance module and an application program; wherein the assistance module is configured to: receive a query from the application program regarding an image processing function; and send a response message to the application program, wherein the response message indicates processing information of the image processing function performed by a specific hardware component of the electronic device.

2. The electronic device of claim 1, wherein the processing information indicates whether the image processing function can be performed by the specific hardware component, wherein the specific hardware component is a digital signal processor, an image signal processor, an image scaler, an encoder, or a monitor.

3. The electronic device of claim 2, wherein the assistance module is configured to consult a table stored in the memory to determine whether the image processing function can be performed by the specific hardware component.

4. The electronic device of claim 2, wherein the application program is configured to: perform the image processing function using the specific hardware component when the processing information indicates that the image processing function can be performed by the specific hardware component; and perform the image processing function using the CPU or a GPU when the response message indicates that the image processing function cannot be performed by the specific hardware component.

5. The electronic device of claim 1, wherein the assistance module is further configured to: receive setting parameters of the image processing function from the application program; wherein the processing information further comprises a quality indicator, wherein the quality indicator indicates a result of the specific hardware component performing the image processing function based on the setting parameters, wherein the specific hardware component is a digital signal processor, an image signal processor, an image scaler, an encoder, or a monitor.

6. The electronic device of claim 5, wherein the application program is configured to: perform the image processing function using the specific hardware component in response to determining that the quality indicator meets a requirement; and adjust the setting parameters and transmit the adjusted setting parameters to the assistance module again or perform the image processing function using the CPU or the GPU in response to determining that the quality indicator does not meet the requirement.

7. The electronic device of claim 5, wherein the assistance module is further configured to: consult a table stored in the memory to determine the quality indicator.

8. The electronic device of claim 5, wherein the assistance module is further configured to: determine the quality indicator by running a simulation based on the setting parameters.

9. The electronic device of claim 5, wherein the quality indicator comprises a processing time, a power consumption, a peak signal-to-noise ratio, or a combination thereof.

10. The electronic device of claim 1, wherein the image processing function is edge enhancement, image scaling, image rotation, super-resolution, or transcoding.

11. A method for assisting application development, comprising: reading, by a central processing unit (CPU) of an electronic device, a program stored in a memory of the electronic device to execute an assistance module and an application program; receiving, by the assistance module, a query from the application regarding an image processing function; and sending, by the assistance module, a response message to the application, wherein the response message indicates processing information of the image processing function performed by a specific hardware component of the electronic device.

12. The method of claim 11, wherein the processing information indicates whether the image processing function can be performed by the specific hardware component, wherein the specific hardware component is a digital signal processor, an image signal processor, an image scaler, an encoder, or a monitor.

13. The method of claim 12, further comprising: consulting, by the assistance module, a table stored in the memory to determine whether the image processing function can be performed by the specific hardware component.

14. The method of claim 12, further comprising: performing, by the application, the image processing function using the specific hardware component when the response message indicates that the image processing function can be performed by the specific hardware component; and performing, by the application, the image processing function using the CPU or the GPU when the response message indicates that the image processing function cannot be performed by the specific hardware component.

15. The method of claim 11, further comprising: receiving, by the assistance module, setting parameters of the image processing function from the application; wherein the processing information further comprises a quality indicator, wherein the quality indicator indicates a result of the specific hardware component performing the image processing function based on the setting parameters, wherein the specific hardware component is a digital signal processor, an image signal processor, an image scaler, an encoder, or a monitor.

16. The method of claim 15, further comprising: performing, by the application, the image processing function using the specific hardware component in response to determining that the quality indicator meets a requirement; and adjusting, by the application, the setting parameters and transmitting the adjusted setting parameters to the assistance module again, or performing, by the application, the image processing function using the CPU or the GPU in response to determining that the quality indicator does not meet the requirement.

17. The method of claim 15, further comprising: consulting, by the assistance module, a table stored in the memory to determine the quality indicator.

18. The method of claim 15, further comprising: running, by the assistance module, a simulation based on the setting parameters to determine the quality indicator.

19. The method of claim 15, wherein the quality indicator comprises a processing time, a power consumption, a peak signal to noise ratio, or a combination thereof.

20. The method of claim 11, wherein the image processing function is edge enhancement, resizing the image, rotating the image, super-resolution, or transcoding.

Images