Image processing method and electronic equipment

Abstract

The invention discloses an image processing method and electronic equipment, relates to the technical field of image processing, and can reduce the time delay of interface display while ensuring the interface display effect. According to the method and the device, after the GPU finishes image rendering, the GPU can store the parameter information of the image obtained after rendering in the first storage space; the NPU can read the parameter information from the first storage space, perform post-processing related calculation according to the parameter information and the parameter target to obtain optimized parameter information, and store the optimized parameter information in the second storage space; and the DSS can read the optimized parameter information from the second storage space, and performs parameter adjustment on the image obtained after rendering according to the optimized parameter information. Therefore, the powerful computing power of the NPU and the display effect processing power of the DSS can be fully utilized, and the time delay of interface display is reduced while the interface display effect is ensured.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and electronic equipment. Background Art

[0002] With the rapid and diversified development of device screen technology and display technology, the interface display effect of electronic devices is getting better and better, and users' requirements for interface display effect are getting higher and higher. Especially in scenes with rich interface elements such as game scenes, video scenes, augmented reality (AR), virtual reality (VR), etc., users have higher and higher requirements for interface display brightness, resolution, etc.

[0003] In order to provide users with better interface display effects, after the graphics processing unit (GPU) completes the rendering of the image to be displayed, the rendered image can be post-processed, including post-processing related calculations such as display brightness and resolution, and performing image post-processing based on the calculation results. However, the amount of calculation for post-processing such as display brightness and resolution is relatively large, and the GPU is also quite heavy in performing mathematical and geometric calculations related to image rendering. Therefore, if the GPU is responsible for image rendering and image post-processing, it will increase the processing load of the GPU, increase the time required for the image processing process, and cause a large delay in the interface display. Summary of the invention

[0004] The present application provides an image processing method and an electronic device, which can provide a low-cost, low-latency, high-performance image processing solution, while ensuring the interface display effect and reducing the delay of the interface display.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, an image processing method is provided, which is applied to an electronic device, wherein the electronic device includes a neural-network processing unit (NPU) and a display service system (DSS), and the method includes: the NPU obtains first parameter information corresponding to a first image from a first storage space, wherein the first parameter information is used to represent one or more of the following image parameters of the first image: resolution, brightness, and color gamut; the NPU performs post-processing calculation according to the first parameter information and a parameter target to obtain second parameter information; the NPU stores the second parameter information in a second storage space; the DSS obtains the second parameter information from the second storage space, and obtains the first image from a third storage space; the DSS adjusts the image parameters of the first image according to the second parameter information to obtain the second image.

[0007] Among them, resolution is used to characterize the amount of information in an image; brightness is used to characterize the degree of brightness of pixels on an image; and color gamut (also called color space) represents the coverage of colors. In the present application, the first parameter information can be used to represent the above-mentioned one parameter of the first image, or can be used to represent any of the above-mentioned multiple parameters of the first image, without specific limitation.

[0008] Among them, the first image can be any one of the following: game image frame, video image frame, framing image, picture, this application does not make specific limitations, depending on the specific usage scenario.

[0009] As an example, the electronic device further includes an on-chip memory, and the first storage space, the second storage space and the third storage space are located in the on-chip memory.

[0010] In the solution provided by the first aspect above, when performing image processing, the NPU can read the first parameter information from the first storage space, perform post-processing related calculations based on the first parameter information and the parameter target, obtain the optimized parameter information (i.e., the second parameter information), and store the second parameter information in the second storage space; the DSS can read the second parameter information from the second storage space, read the rendered image from the third storage space, and adjust the parameters of the rendered image based on the second parameter information to obtain the second image.

[0011] Based on this, the electronic device can make full use of the powerful computing power of the NPU and the display effect processing capability of the DSS by having the NPU perform post-processing related calculations for local brightening, super-resolution display, HDR, etc., and the DSS adjusts the image-related parameters according to the calculation results of the NPU, so as to reduce the interface display delay while ensuring the interface display effect. For example, compared with the solution in which the GPU is responsible for image rendering and post-processing (including post-processing related calculations and adjustment of image-related parameters according to the calculation results), the solution provided in the first aspect can greatly reduce the GPU load, reduce the image processing delay and ensure the interface display effect.

[0012] In addition, since the solution can be completed based on the existing hardware of the electronic device, no additional cost will be added. For example, compared with the solution of performing post-processing related calculations by a separate graphics chip and adjusting image-related parameters based on the calculation results, the cost required for image processing can be greatly reduced.

[0013] As a possible implementation, the first parameter information includes resolution information of the first image, and the parameter target includes a resolution target; the NPU performs post-processing calculation according to the first parameter information and the parameter target to obtain the second parameter information, including: the NPU obtains a pixel expansion coefficient according to the resolution information and the resolution target based on a preset resolution algorithm, and the second parameter information includes a pixel expansion coefficient; the DSS adjusts the image parameters of the first image according to the second parameter information to obtain the second image, including: the DSS performs pixel expansion on the second image according to the pixel expansion coefficient to obtain the second image. For example, the preset resolution algorithm may include but is not limited to a super-resolution algorithm, such as a super-resolution sharpening (FidelityFX super resolution, FSR) algorithm. In this way, based on the existing hardware of the electronic device, the powerful computing power of the NPU and the display effect processing capability of the DSS can be fully utilized to reduce the delay of the interface display while ensuring a higher resolution of the interface.

[0014] As a possible implementation method, the first parameter information includes brightness information of pixels in the first image, and the parameter target includes a brightness target; the NPU performs post-processing calculations based on the first parameter information and the parameter target to obtain the second parameter information, including: the NPU identifies the dark areas of pixels in the first image, and performs pixel brightening calculations based on the brightness target to obtain a pixel brightening coefficient for the dark areas of pixels; the DSS adjusts the image parameters of the first image based on the second parameter information to obtain the second image, including: the DSS performs pixel brightening for the dark areas of pixels based on the pixel brightening coefficient to obtain the second image. In this way, based on the existing hardware of the electronic device, the powerful computing power of the NPU and the display effect processing capabilities of the DSS can be fully utilized to reduce the delay of the interface display while ensuring the brightness balance of the interface.

[0015] As a possible implementation method, the first parameter information includes the color gamut information of the pixels in the first image, and the parameter target includes a high dynamic range imaging (HDR) target; the NPU performs post-processing calculations based on the first parameter information and the parameter target to obtain the second parameter information, including: the NPU performs dynamic range calculations based on the color gamut information to obtain an HDR expansion coefficient for the pixels in the first image; the DSS adjusts the image parameters of the first image based on the second parameter information to obtain the second image, including: the DSS performs HDR pixel expansion on the first image based on the HDR expansion coefficient to obtain the second image. In this way, based on the existing hardware of the electronic device, the powerful computing power of the NPU and the display effect processing capability of the DSS can be fully utilized to reduce the delay of the interface display while ensuring a high dynamic range of the interface.

[0016] As a possible implementation, after the NPU stores the second parameter information in the second storage space, the method further includes: the NPU sets a first mark, wherein the first mark is used to indicate that the NPU has stored the second parameter information in the second storage space; the DSS obtains the second parameter information from the second storage space specifically as follows: the DSS obtains the second parameter information from the second storage space according to the first mark. For example, the NPU may set a first mark (such as acquireFence) based on a fence synchronization mechanism, and the DSS may obtain the second parameter information from the second storage space based on the fence synchronization mechanism when it recognizes that a new first mark has been added. Based on this, data reading and writing synchronization can be achieved to further reduce the delay between data writing by the NPU and data reading by the DSS during image processing, thereby improving the efficiency of image processing.

[0017] As a possible implementation, after the DSS obtains the second parameter information from the second storage space, the method further includes: the DSS sets a second mark, wherein the second mark is used to indicate that the DSS has obtained the second parameter information from the second storage space; and the NPU deletes the second parameter information in the second storage space according to the second mark. Based on this, data reading and writing synchronization can be achieved, which facilitates the NPU to timely clean up the second parameter information that the DSS has read to reduce the load of the second storage space.

[0018] It should be noted that after the DSS obtains the second parameter information from the second storage space, the NPU deletes the second parameter information in the second storage space only as an example and is not required. In some possible embodiments, after the DSS obtains the second parameter information from the second storage space, the NPU may also directly store the image parameters corresponding to the next frame of the image obtained by post-processing calculation into the second storage space.

[0019] As a possible implementation, the electronic device further includes a GPU, and the method further includes: the GPU renders the image to be displayed to obtain a first image; the GPU stores the first image in a third storage space, and stores the first parameter information corresponding to the first image in the first storage space. In this way, it is convenient for the subsequent NPU to perform post-processing related calculations based on the first parameter information in the first storage space, and it is convenient for the subsequent DSS to perform corresponding parameter adjustments on the first image in the third storage space based on the post-processing calculation results, so as to obtain a better interface display effect.

[0020] Of course, the present application does not limit the entity responsible for image rendering and providing the first parameter information. In some embodiments, the entity responsible for rendering the first image and providing the first parameter information of the first image may also be other modules or units, depending on the specific device function, structure and usage scenario.

[0021] As a possible implementation, after the GPU stores the first parameter information corresponding to the first image in the first storage space, the method further includes: the GPU sets a third mark, wherein the third mark is used to indicate that the GPU has stored the first parameter information in the first storage space; the NPU obtains the first parameter information corresponding to the first image from the first storage space specifically as follows: the NPU obtains the first parameter information from the first storage space according to the third mark. For example, the GPU may set a third mark (such as acquireFence) based on a fence synchronization mechanism, and the NPU may obtain the first parameter information from the first storage space based on the fence synchronization mechanism when identifying the addition of a third mark. Based on this, data reading and writing synchronization can be achieved to further reduce the delay between GPU data writing and NPU data reading during image processing, thereby improving the efficiency of image processing.

[0022] As a possible implementation, after the NPU obtains the first parameter information from the first storage space, the method further includes: the NPU sets a fourth flag, wherein the fourth flag is used to indicate that the NPU has obtained the first parameter information from the first storage space; and the GPU deletes the first parameter information in the first storage space according to the fourth flag. Based on this, data reading and writing synchronization can be achieved, which facilitates the GPU to timely clean up the first parameter information that the NPU has read to reduce the load of the first storage space.

[0023] It should be noted that after the NPU obtains the first parameter information from the first storage space, the GPU deletes the first parameter information in the first storage space only as an example and is not required. In some possible embodiments, after the NPU obtains the first parameter information from the first storage space, the GPU may also directly store the image parameters corresponding to the next frame of the rendered image into the first storage space.

[0024] In a second aspect, an electronic device is provided, which includes: an NPU, a DSS, a first storage space, a second storage space and a third storage space; the NPU is used to: obtain first parameter information corresponding to a first image from the first storage space, wherein the first parameter information is used to represent one or more of the following image parameters of the first image: resolution, brightness, and color gamut; perform post-processing calculations based on the first parameter information and parameter targets to obtain second parameter information; and, store the second parameter information in the second storage space; the DSS is used to: obtain second parameter information from the second storage space, obtain the first image from the third storage space, and adjust the image parameters of the first image based on the second parameter information to obtain the second image.

[0025] Among them, the first image can be any one of the following: game image frame, video image frame, framing image, picture, this application does not make specific limitations, depending on the specific usage scenario.

[0026] As an example, the electronic device further includes an on-chip memory, and the first storage space, the second storage space and the third storage space are located in the on-chip memory.

[0027] In the solution provided in the second aspect above, when performing image processing, the NPU can read the first parameter information from the first storage space, perform post-processing related calculations based on the first parameter information and the parameter target, obtain the optimized parameter information (i.e., the second parameter information), and store the second parameter information in the second storage space; the DSS can read the second parameter information from the second storage space, read the rendered image from the third storage space, and adjust the parameters of the rendered image based on the second parameter information to obtain the second image. Based on this, the electronic device can make full use of the powerful computing power of the NPU and the display effect processing capability of the DSS by having the NPU perform post-processing related calculations for local brightening, super-resolution display, HDR, etc., and the DSS adjusts the image related parameters based on the calculation results of the NPU, thereby ensuring the interface display effect while reducing the interface display delay. In addition, since the solution can be completed based on the existing hardware of the electronic device, it will not increase additional costs.

[0028] As a possible implementation method, the first parameter information includes resolution information of the first image, and the parameter target includes a resolution target; the NPU is specifically used for: obtaining a pixel expansion coefficient according to the resolution information and the resolution target based on a preset resolution algorithm, and the second parameter information includes a pixel expansion coefficient; the DSS is specifically used for: performing pixel expansion on the second image according to the pixel expansion coefficient to obtain the second image. For example, the preset resolution algorithm may include but is not limited to a super resolution algorithm, such as an FSR algorithm. In this way, based on the existing hardware of the electronic device, the powerful computing power of the NPU and the display effect processing capability of the DSS can be fully utilized to reduce the delay of the interface display while ensuring a higher resolution of the interface.

[0029] As a possible implementation method, the first parameter information includes the brightness information of the pixels in the first image, and the parameter target includes the brightness target; the NPU is specifically used to: identify the dark areas of the pixels in the first image, and perform pixel brightening calculations according to the brightness target to obtain the pixel brightening coefficient for the dark areas; the DSS is specifically used to: perform pixel brightening for the dark areas according to the pixel brightening coefficient to obtain the second image. In this way, based on the existing hardware of the electronic device, the powerful computing power of the NPU and the display effect processing capabilities of the DSS can be fully utilized to ensure the brightness balance of the interface while reducing the delay of the interface display.

[0030] As a possible implementation method, the first parameter information includes the color gamut information of the pixels in the first image, and the parameter target includes the HDR target; the NPU is specifically used to: perform dynamic range calculation according to the color gamut information to obtain the HDR expansion coefficient for the pixels in the first image; the DSS is specifically used to: perform HDR pixel expansion on the first image according to the HDR expansion coefficient to obtain the second image. In this way, based on the existing hardware of the electronic device, the powerful computing power of the NPU and the display effect processing capability of the DSS can be fully utilized to reduce the delay of the interface display while ensuring a high dynamic range of the interface.

[0031] As a possible implementation, after the NPU stores the second parameter information in the second storage space, the NPU is further used to: set a first mark, wherein the first mark is used to indicate that the NPU has stored the second parameter information in the second storage space; and the DSS is specifically used to: obtain the second parameter information from the second storage space according to the first mark. Based on this, data reading and writing synchronization can be achieved to further reduce the delay between data writing of the NPU and data reading of the DSS during the image processing process, thereby improving the efficiency of image processing.

[0032] As a possible implementation, after the DSS obtains the second parameter information from the second storage space, the DSS is further used to: set a second mark, wherein the second mark is used to indicate that the DSS has obtained the second parameter information from the second storage space; and the NPU is further used to: delete the second parameter information in the second storage space according to the second mark. Based on this, data reading and writing synchronization can be achieved, which facilitates the NPU to timely clean up the second parameter information that the DSS has read to reduce the load of the second storage space.

[0033] As a possible implementation, the electronic device further includes a GPU, which is used to: render the image to be displayed to obtain a first image, store the first image in a third storage space, and store first parameter information corresponding to the first image in the first storage space. In this way, it is convenient for the subsequent NPU to perform post-processing related calculations based on the first parameter information in the first storage space, and it is convenient for the subsequent DSS to perform corresponding parameter adjustments on the first image in the third storage space based on the post-processing calculation results, so as to obtain a better interface display effect.

[0034] As a possible implementation, after the GPU stores the first parameter information corresponding to the first image in the first storage space, the GPU is further used to: set a third mark, wherein the third mark is used to indicate that the GPU has stored the first parameter information in the first storage space; the NPU is specifically used to: obtain the first parameter information from the first storage space according to the third mark. Based on this, data reading and writing synchronization can be achieved to further reduce the delay between GPU data writing and NPU data reading during image processing, thereby improving the efficiency of image processing.

[0035] As a possible implementation, after the NPU obtains the first parameter information from the first storage space, the NPU is further used to: set a fourth flag, wherein the fourth flag is used to indicate that the NPU has obtained the first parameter information from the first storage space; the GPU is further used to: delete the first parameter information in the first storage space according to the fourth flag. Based on this, data reading and writing synchronization can be achieved, which facilitates the GPU to timely clean up the first parameter information that the NPU has read to reduce the load of the first storage space.

[0036] In a third aspect, an electronic device is provided, comprising: a display screen for displaying images, such as displaying a second image; a memory for storing computer program instructions; and a processor for executing the computer program instructions to support the electronic device to implement a method in any possible implementation manner of the first aspect.

[0037] According to a fourth aspect, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method in any possible implementation of the first aspect is implemented.

[0038] According to a fifth aspect, a computer program product comprising instructions is provided. When the computer program product is run on a computer, the computer implements the method in any possible implementation of the first aspect.

[0039] In a sixth aspect, a chip system is provided, the chip system comprising a processing circuit and a storage medium, wherein the storage medium stores computer program instructions; when the computer program instructions are executed by the processor, the method in any possible implementation of the first aspect is implemented. The chip system may be composed of a chip, or may include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of an image processing architecture provided in an embodiment of the present application;

[0042] Figure 3 A flow chart of an image processing method provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of a resolution post-processing process provided in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of a brightness post-processing process provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of an HDR post-processing process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0047] In the following, the terms "first", "second", etc. are only used to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. For example, if the described object is a "field", the ordinal number before the "field" in the "first field" and the "second field" does not limit the position or order between the "fields", and the "first" and "second" do not limit whether the "fields" they modify are in the same message, nor do they limit the order of the "first field" and the "second field". For another example, if the described object is a "level", the ordinal number before the "level" in the "first level" and the "second level" does not limit the priority between the "levels". For another example, the number of described objects is not limited by the ordinal number, and can be one or more. Taking the "first device" as an example, the number of "devices" can be one or more. In addition, the objects modified by different prefixes can be the same or different. For example, if the described object is a "device", the "first device" and the "second device" can be the same type of device or different types of devices. For another example, if the described object is "information", the "first information" and the "second information" can be information of the same content or information of different contents. In summary, the use of prefixes such as ordinal numbers to distinguish the described objects in the embodiments of the present application does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and no unnecessary limitation should be constituted due to the use of such prefixes.

[0048] Furthermore, in the embodiments of the present application, "connection" may be a direct connection or an indirect connection; in addition, it may refer to an electrical connection or a communication connection; for example, the connection between two electrical components A and B may refer to a direct connection between A and B, or may refer to an indirect connection between A and B through other electrical components or connecting media, or may refer to an indirect connection between A and B through other communication devices or communication media, as long as communication between A and B can be achieved.

[0049] As mentioned above, in order to provide users with better interface display effects, the GPU can be responsible for image rendering and post-processing, including post-processing related calculations such as display brightness, resolution, HDR, and adjustment of image-related parameters based on the calculation results. However, both rendering tasks and post-processing related calculation tasks are relatively heavy, which will bring a large load on the GPU in terms of calculation and data transmission, which will not only affect whether the post-processing related calculations can be completed smoothly, but also affect the processing efficiency of the rendering tasks, thus increasing the time required for the image processing process, causing a large delay in the interface display, and unable to guarantee the final display effect of the interface.

[0050] In order to solve the above problems, in some embodiments, considering the problem of heavy GPU processing load, after the GPU completes image rendering, a chip with higher processing power (such as an independent graphics chip) can perform post-processing related calculations and adjust image related parameters based on the calculation results. However, in this solution, the introduction of an independent graphics chip will greatly increase the cost of the electronic device.

[0051] In other embodiments, considering that the NPU also has relevant computing capabilities, after the GPU completes image rendering, the neural-network processing unit (NPU) can perform post-processing related calculations, and then the GPU adjusts the image-related parameters according to the calculation results of the NPU. However, in this solution, the NPU returns the calculation results to the GPU, and the GPU adjusts the image-related parameters. Since the GPU also needs to process related tasks such as image rendering in parallel, it may affect the timeliness of adjusting the image-related parameters according to the calculation results of the NPU. Therefore, there may be a problem of large interface display delay, and it will also add additional processing load to the GPU.

[0052] In other embodiments, considering the display effect processing capability of the display server system (DSS), after the GPU completes the image rendering, the DSS can perform post-processing related calculations and adjust the image related parameters according to the calculation results. However, due to the limitation of the computing power of the DSS, the post-processing calculations that the DSS can support are usually limited, such as usually only being able to perform the calculation of some display indicators (such as only being able to perform arbitrary resolution scaling (ARSR)), so the final interface display effect will be relatively limited.

[0053] In order to solve one or more problems such as large delay, large GPU processing load, high cost, and limited interface display effect during the interface display process in the above-mentioned embodiments, the embodiment of the present application provides an image processing method, which takes into account the computing power of the NPU and the display effect processing capability of the DSS, and provides an image processing solution in which the NPU performs post-processing related calculations after the GPU completes image rendering, and the DSS adjusts image-related parameters according to the calculation results of the NPU. This solution can make full use of the powerful computing power of the NPU and the display effect processing capability of the DSS, while ensuring the interface display effect, reducing the delay of the interface display; in addition, since the solution can be completed based on the existing hardware of the electronic device, it will not increase additional costs.

[0054] For example, in the embodiment of the present application, after the GPU completes image rendering, the GPU can cache the rendered image in a preset cache area, and at the same time, the GPU can send parameter information (such as first parameter information) of the rendered image to the NPU, and the NPU performs post-processing related calculations based on the parameter target and the parameter information of the image to obtain optimized parameter information (such as second parameter information). Afterwards, the DSS can obtain the rendered image from the preset cache area, and adjust the relevant parameters of the image according to the optimized parameter information before displaying it.

[0055] Exemplarily, based on the image processing method described in the present application, the image can be efficiently and reliably post-processed in aspects such as local brightening, super-resolution display, and high dynamic range imaging (HDR) to ensure the effect of interface display. Among them, local brightening refers to brightening the dark areas of pixels in the image; super-resolution display refers to displaying the image at a high resolution (such as higher than a preset resolution); HDR refers to providing more dynamic range and image details to better reflect the visual effects in the real environment. Taking the game scene as an example, we know that in the game scene, the GPU of the electronic device is heavily loaded with the rendering of the game interface. In order to achieve a high-quality display effect of the game interface, it is usually necessary to compensate for the display effect or enhance the display effect through an effect algorithm (such as a post-processing algorithm) after the GPU completes the interface rendering. Based on the image processing method provided in the embodiment of the present application, display effect compensation or display effect enhancement can be completed efficiently and reliably.

[0056] Among them, the image processing method described in the embodiment of the present application can be applied to but not limited to the interface display process, especially in scenes with rich interface elements and / or involving image rendering, such as game scenes, video scenes, AR, VR, etc. The image described in the embodiment of the present application may include but is not limited to image frames, pictures, viewfinder images, interface layers corresponding to interface data in the video stream, etc., and the embodiment of the present application does not make specific limitations.

[0057] The electronic device described in the embodiment of the present application may include but is not limited to the device that can provide image rendering and image post-processing in the above-mentioned scenarios, such as the electronic device may include but is not limited to smart phones, netbooks, tablet computers, handwriting tablets, smart watches, smart bracelets, phone watches, smart cameras, PDAs, car computers, personal computers (personal computers, PCs), personal digital assistants (personal digital assistants, PDAs), portable multimedia players (portable multimedia players, PMPs), AR devices / VR devices, smart TVs, projection devices, or somatosensory game consoles in human-computer interaction scenarios, etc. Alternatively, the electronic device may also be a device of other types or structures, which is not limited in the embodiment of the present application.

[0058] As an example, see Figure 1 , Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0059] like Figure 1As shown, the electronic device may include a processor 110, a memory (including an external memory interface 120 and an internal memory 121), a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, etc. Among them, the sensor module 180 may include but is not limited to a touch sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, an ambient light sensor, a bone conduction sensor, etc.

[0060] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0061] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a flight controller, a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU), a display server system (DSS), etc. Among them, different processing units may be independent devices or integrated into one or more processors.

[0062] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0063] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0064] The charging management module 140 is used to receive charging input from the charger. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the display screen 194, the camera 193 and the wireless communication module 160.

[0065] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0066] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0067] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to electronic devices. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0068] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0069] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as WiFi networks), Bluetooth BT, global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to electronic devices. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0070] In some embodiments, the antenna 1 of the electronic device is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).

[0071] In some embodiments of the present application, the electronic device can exchange information with other devices through the wireless communication module 160 based on wireless communication technology, such as receiving image data from an application server or other devices.

[0072] The electronic device realizes the display function through GPU, NPU, DSS, display screen 194, and application processor.

[0073] The GPU is a microprocessor for image processing. The GPU can be used to perform mathematical and geometric calculations, for image rendering, etc. The processor 10 may include one or more GPUs that execute program instructions to generate or change display information.

[0074] Due to its high performance and low energy consumption, NPU is widely used in deep neural network calculations in fields such as artificial intelligence and machine learning. Among them, NPU can simulate neurons and synaptic structures through circuits to perform deep neural network calculations.

[0075] The DSS may be used to perform display-related processing of the image, such as related adjustments of brightness, resolution, HDR, etc. After completing the display-related processing of the image, the DSS may send the image to the display screen 194 for subsequent display.

[0076] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0077] In an embodiment of the present application, the GPU can be used to perform image rendering, etc., the NPU can perform post-processing related calculations based on the parameter information (such as the first parameter information) and parameter targets of the image obtained after rendering to obtain optimized parameter information (such as the second parameter information), the DSS can adjust the relevant parameters of the image based on the optimized parameter information, and the display screen 194 can be used to display the image after the relevant parameters are adjusted.

[0078] The electronic device can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0079] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function.

[0080] Internal memory 121 may be used to store computer executable program code. For example, computer programs may include operating system programs and application programs. The operating system may include but is not limited to OS and other operating systems. Among them, the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, at least one application required for a function, etc. The data storage area may store data created during the use of the electronic device (such as rendered images, image parameter information (such as first parameter information), optimized parameter information (such as second parameter information), parameter adjusted images, etc.). In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0081] In an embodiment of the present application, the storage data area of ​​the internal memory 121 can be used to store data such as rendered images created during image processing by the electronic device, image parameter information (such as first parameter information), optimized parameter information (such as second parameter information), and images after parameter adjustment.

[0082] The electronic device can realize audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C and the application processor, etc. For example, music playing, recording, etc. The specific working principle and function of the hardware such as the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the button 190, the motor 191 and the indicator 192 can be referred to the introduction in the conventional technology, and the embodiments of the present application will not be repeated.

[0083] It is understandable that this application Figure 1 The illustrated structure does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0084] For ease of understanding, the following examples are Figure 2 The image processing architecture shown is used as an example to introduce the image processing method provided by the embodiment of the present application. Figure 2As shown, the image processing architecture may include a GPU, an NPU, a DSS, a memory and a display screen. The memory includes a first storage space, a second storage space and a third storage space.

[0085] like Figure 2 As shown, first, after acquiring the image obtained after drawing, the GPU can render the image. After the GPU completes the image rendering, the GPU can cache the rendered image (such as the first image) in the third storage space of the memory, and store the parameter information of the rendered image (such as the first parameter information) in the first storage space. Among them, the first storage space can accept the access of the GPU (such as read / write / clear, etc.), and can also accept the access of the NPU (such as read / clear, etc.). Exemplarily, the GPU can obtain the image obtained after drawing from the CPU frame buffer (CPU frame buffer, CFB).

[0086] Then, if Figure 2 As shown, the NPU can read the first parameter information from the first storage space, perform post-processing related calculations based on the first parameter information and the parameter target, obtain optimized parameter information (such as the second parameter information), and store the second parameter information in the second storage space. The second storage space can be accessed by the NPU (such as read / write / clear, etc.) and can also be accessed by the DSS (such as read / clear, etc.).

[0087] Then, if Figure 2 As shown, the DSS can read the second parameter information from the second storage space, read the rendered image from the third storage space, and adjust the parameters of the rendered image according to the second parameter information to obtain the second image.

[0088] Finally, if Figure 2 As shown, the DSS may send the obtained second image to the display screen for subsequent display.

[0089] In some embodiments, Figure 2 As shown, when the first image flows between the GPU and the DSS, the first parameter information flows between the GPU and the NPU, and the second parameter information flows between the NPU and the DSS, the fence synchronization mechanism can be used to achieve data reading and writing synchronization to reduce the delay between data writing and data reading in each processing link during the image processing process and improve the efficiency of image processing.

[0090] The fence synchronization mechanism is mainly used to avoid the read / write conflict / confusion problem when multiple processes access the same storage space. The fence synchronization mechanism can avoid the above read / write conflict / confusion problem by coordinating the access of multiple processes to the same storage space. For example, the fence synchronization mechanism can identify whether the data is cached, whether it is used, etc. through the fence tag of the data.

[0091] For example, after the GPU caches the first parameter information to the first storage space, a third mark (such as acquireFence) can be inserted into the thread to indicate that the first parameter information caching is complete; when the NPU recognizes that the third mark is newly added to the thread, the NPU can read the first parameter information from the first storage space cache; after the NPU completes reading the first parameter information from the first storage space cache, the NPU can insert a fourth mark (such as releaseFence) in the thread to indicate that the first parameter information reading is complete; when the GPU recognizes that the second mark is newly added to the thread, the first parameter information in the first storage space can be cleaned up and stored in the image parameters of the next frame image after rendering. Regarding the process of the second parameter information flowing between the NPU and the DSS, the above-mentioned process of the first parameter information flowing between the GPU and the NPU can also be referred to.

[0092] In some embodiments, after the first parameter information is marked as the fourth mark (such as releaseFence), the first storage space may clear the first parameter information to avoid storage load and processing load caused by redundant information.

[0093] For a detailed introduction to the fence synchronization mechanism, please refer to conventional technologies and will not be elaborated in detail in the embodiments of this application.

[0094] It should be noted that Figure 2 This is only an example of an image processing architecture. In actual applications, it can be adaptively adjusted according to actual usage requests, and the embodiments of this application do not make specific limitations. For example, if the interface to be displayed includes multiple layers, the GPU can also synthesize the multiple layers through SurfaceFlinger after completing image rendering, and then store the synthesized image in the third storage space of the image memory.

[0095] The following will be combined Figure 2 Taking the image processing architecture shown as an example, the image processing method provided in the embodiment of the present application is specifically introduced in combination with the accompanying drawings.

[0096] For example, please refer to Figure 3 , Figure 3 FIG. 1 shows a flow chart of an image processing method provided by an embodiment of the present application. Figure 3As shown, an image processing method provided in an embodiment of the present application may include the following S301-S308:

[0097] S301: GPU renders the image to be displayed to obtain a first image.

[0098] The rendering stage is mainly used to render the image to be displayed in one or more aspects such as brightness, saturation, etc. Brightness is used to represent the brightness of pixels on the image, and saturation is used to represent the number of color types on the image.

[0099] In some embodiments, when rendering an image to be displayed, the electronic device may render the image to be displayed by executing a DrawFrame command.

[0100] In some embodiments, before the GPU renders the image to be displayed, the electronic device (such as the CPU of the electronic device) will also draw the image to be displayed. Among them, the drawing stage may include the following processes: measure → layout → draw. The measurement is used to obtain and save the size of each View and ViewGrop corresponding to the image according to the setting of the control properties in the interface layout file and code corresponding to the image. The layout is used to determine the display position of each control on the image based on the information obtained by the measurement. Drawing is used to draw the corresponding layer on the canvas (canvas) after determining the display position of each control, and construct a drawing instruction. Exemplarily, after completing the drawing of the image to be displayed, the electronic device can also cache the image obtained after drawing in a preset cache space, such as CFB.

[0101] For the relevant introduction of image rendering, as well as the relevant introduction of the processing process before image rendering (such as drawing, etc.), reference can be made to conventional technology, and the embodiments of this application will not be repeated here.

[0102] S302: The GPU caches the first image in the third storage space, and caches the first parameter information corresponding to the first image in the first storage space.

[0103] The first parameter information is used to represent but is not limited to any one or more of the following image parameters of the first image: resolution, brightness, color gamut, etc.

[0104] Among them, resolution is used to characterize the amount of information in an image. For example, resolution can be represented by the number of pixels per inch on the image (such as pixels per inch (PPI)). Brightness is used to characterize the brightness of pixels on an image. For example, the brightness of a pixel can be represented by a grayscale value. For example, the grayscale value can be represented by [0,255], where the closer the grayscale value is to 0, the lower the brightness, and the closer the grayscale value is to 255, the higher the brightness. The color gamut (also called color space) represents the coverage of colors. The format of the color gamut is such as sRGB, and the color gamut can be represented by the number of bits.

[0105] As an example, the first storage space may be a storage space of an on-chip memory, such as Ion memory 1. The first storage space may be a preset storage space for caching image parameters.

[0106] Exemplarily, the GPU can cache the first parameter information corresponding to the first image in the first storage space based on the fence synchronization mechanism. For example, after completing the caching of the first parameter information in the first storage space, the GPU can insert a third mark (such as acquireFence) in the thread to indicate that the caching of the first parameter information is complete, which is used to ensure that the NPU will not have a read-write conflict / confusion problem with the GPU, and can also enable the NPU to read the first parameter information from the first storage space in a timely manner according to the third mark, thereby reducing the delay between the GPU writing data to the first storage space and the NPU obtaining data from the first storage space during the image processing process, thereby improving the efficiency of image processing.

[0107] As an example, the third storage space may be a storage space of an on-chip memory, such as Ion memory 0. The third storage space may be a preset storage space for caching a buffer queue (BufferQueue). For example, after completing the rendering of the image to be displayed, the GPU may send the first image to the buffer queue by calling a cache function, such as eglSwapBuffersWithDamageKHR function.

[0108] S303: The NPU obtains first parameter information from the first storage space.

[0109] As an example, the NPU may acquire the first parameter information from the first storage space when recognizing that a first flag (such as acquireFence) is newly added to the thread based on the fence synchronization mechanism.

[0110] It can be understood that the addition of a first mark in the thread indicates that the GPU has completed the caching of the first parameter information. Therefore, the NPU obtaining the first parameter information from the first storage space at this time does not conflict with the reading and writing of the GPU. The NPU can also read the first parameter information from the first storage space in a timely manner after the GPU completes the caching of the first parameter information, thereby reducing the delay of each processing link in the image processing process and improving the efficiency of image processing.

[0111] In some embodiments, after the NPU obtains the first parameter information from the first storage space, the NPU may insert a fourth marker (such as releaseFence) in the thread to indicate that the first parameter information has been read. When the GPU recognizes that a fourth marker (such as releaseFence) is newly added to the thread, the first parameter information in the first storage space may be deleted. In this way, it is convenient for the GPU to store the image parameters of the next frame of the rendered image, and it is also possible to avoid the storage load and processing load caused by redundant information.

[0112] S304: The NPU performs post-processing calculation according to the first parameter information and the parameter target to obtain second parameter information.

[0113] Among them, the parameter target may include but is not limited to target values ​​for one or more parameters such as resolution and brightness. The parameter target can be pre-set in the NPU, or pre-set in a path accessible to the NPU. Among them, the parameter target may be related to the display requirements of the application described in the first image, or may be related to the image processing capability and / or display capability of the electronic device, etc., or may be set based on empirical values, which is not limited in the embodiments of the present application.

[0114] In some embodiments, the NPU may use corresponding post-processing algorithms to perform post-processing calculations for different types of image parameters to obtain second parameter information. The following will introduce the post-processing calculation process of the NPU for different types of image parameters with specific examples:

[0115] Example 1: The first parameter information includes resolution information of the first image. In this case, the NPU may perform post-processing calculation according to the resolution information and the resolution target to obtain the second parameter information, wherein the second parameter information includes a pixel expansion coefficient.

[0116] For example, please refer to Figure 4 , Figure 4 Taking the first parameter information including resolution information as an example, a schematic diagram of a resolution post-processing process provided by an embodiment of the present application is shown.

[0117] like Figure 4As shown, first, after acquiring the image obtained after drawing, the GPU can render the image. After the GPU completes the image rendering, the GPU can cache the rendered image (such as the first image) in the third storage space of the memory, and store the resolution information of the rendered image in the first storage space. Among them, the format of the resolution information is such as RGB888, and the resolution information is such as 480*1080. Then, the NPU can read the above resolution information from the first storage space, perform post-processing related calculations based on the resolution information and the resolution target, and obtain the pixel expansion coefficient. Among them, the pixel expansion coefficient is such as [w 1 ,w 2 ,w 3 ……,w n ], n is the number of pixels in the first image, i∈[1,n], w i =1 / 2-1 / 4*feature i (direction, length), feature i (direction, length) is the fitting feature corresponding to the ith pixel, and the fitting feature is related to the resolution information of the ith pixel and its distance from the resolution target (which can be expressed by direction and length).

[0118] As an example, Figure 4 As shown, the NPU can perform post-processing calculations based on the resolution information and the resolution target to obtain the pixel expansion coefficient based on a super-resolution algorithm, such as the FidelityFX super resolution (FSR) algorithm. The FSR algorithm is mainly used to detect the edge of the first image and try to maintain the original edge while enlarging the image, thereby achieving super-resolution.

[0119] Exemplarily, the FSR algorithm can achieve super-resolution of the image by adding pixels to the pixels of the image, that is, by interpolation. For example, super-resolution can be achieved by edge adaptive spatial upsampling (EASU). Among them, EASU mainly includes the following three steps: pixel sampling, interpolation calculation, and output restriction. Interpolation calculation is a key step in adding pixels to the pixels of the first image. The interpolation method may include but is not limited to the nearest neighbor sample interpolation, bilinear interpolation, bicubic interpolation, and lanczos interpolation. For the specific introduction of each interpolation method and the FSR algorithm, please refer to conventional technology, which will not be repeated here.

[0120] In some embodiments, in order to avoid blur or ghost errors in the first image pixels, the FSR algorithm can also optimize each pixel through a sharpening step to ensure the final image quality.

[0121] For example, the FSR algorithm can achieve image sharpening through robust contrast adaptive sharpening (RCAS). Among them, RCAS is mainly used to adjust the color by using the local contrast near the pixel point to eliminate the blurring of details caused by anti-aliasing, image stretching and other operations. Among them, contrast is used to characterize the difference between the dark and bright pixels on the image, such as the difference between the maximum grayscale value and the minimum grayscale value of the pixels in the local area of ​​the image.

[0122] As another example, the NPU can perform post-processing calculations based on resolution information and resolution targets based on an artificial neural network, such as an AI deep learning method, to obtain a pixel expansion coefficient. Exemplarily, the NPU can perform post-processing calculations based on resolution information and resolution targets based on a deep learning super sampling (DLSS) method to obtain a pixel expansion coefficient. In addition, the NPU can also perform image sharpening based on the DLSS method to optimize each pixel to ensure the final image quality. For a specific introduction to AI deep learning methods for extending resolution such as the DLSS method, please refer to conventional techniques, which will not be elaborated here.

[0123] Example 2: The first parameter information includes brightness information of pixels in the first image. In this case, the NPU can perform post-processing calculation based on the brightness information and the brightness target to obtain the second parameter information, where the second parameter information includes a pixel brightening coefficient.

[0124] For example, please refer to Figure 5 , Figure 5 Taking the case where the first parameter information includes brightness information as an example, a schematic diagram of a brightness post-processing process provided by an embodiment of the present application is shown.

[0125] like Figure 5 As shown, first, after acquiring the image obtained after drawing, the GPU can render the image. After the GPU completes image rendering, the GPU can cache the rendered image (such as the first image) in the third storage space of the memory, and store the brightness information of the pixels in the rendered image in the first storage space. Among them, the brightness information is such as the pixel channel RGB and channel filter coefficients α, β, γ of the first image. Then, the NPU can read the above brightness information from the first storage space, and perform post-processing related calculations based on the brightness information and the brightness target, such as Figure 5 As shown, the pixel scanning algorithm is used to identify pixels in the first image whose brightness values ​​are lower than a preset threshold (hereinafter referred to as pixel dark areas), and then the pixel brightening calculation is performed according to the brightness target to obtain the pixel brightening coefficient for the pixel dark area. The pixel brightening coefficient is such as the adjusted channel filter coefficients α', β', and γ'.

[0126] Example 3: The first parameter information includes color gamut information of pixels in the first image. In this case, the NPU can perform post-processing calculations based on the color gamut information and the HDR target to obtain the second parameter information, where the second parameter information includes the HDR expansion coefficient.

[0127] For example, please refer to Figure 6 , Figure 6 Taking the first parameter information including color gamut information as an example, a schematic diagram of an HDR post-processing process provided by an embodiment of the present application is shown.

[0128] like Figure 6 As shown, first, after acquiring the image obtained after drawing, the GPU can render the image. After the GPU completes image rendering, the GPU can cache the rendered image (such as the first image) in the third storage space of the memory, and store the color gamut information of the pixels in the rendered image in the first storage space. Among them, the color gamut information is such as the bit depth (bit) of the first image, such as 4 bits (i.e., 4 bits each for R, G, and B). Then, the NPU can read the above color gamut information from the first storage space, and perform relevant post-processing calculations based on the color gamut information and the HDR target, such as Figure 6 The dynamic range is calculated as shown, and the HDR expansion coefficient for each pixel is obtained. Among them, the HDR expansion coefficient is 2 (representing that the bit depth is expanded to 8 bits).

[0129] S305: The NPU caches the second parameter information in the second storage space.

[0130] As an example, the second storage space may be a storage space of an on-chip memory, such as Ion memory 2. The second storage space may be a preset storage space for caching parameter information (such as second parameter information) after post-processing calculation.

[0131] Exemplarily, the NPU can cache the second parameter information in the second storage space based on the fence synchronization mechanism. For example, after completing the caching of the second parameter information in the second storage space, the NPU can insert a first mark (such as acquireFence) in the thread to indicate that the caching of the second parameter information is completed, which is used to ensure that the DSS will not have a read-write conflict / confusion problem with the NPU, and can also enable the DSS to read the second parameter information from the second storage space in a timely manner according to the first mark, thereby reducing the delay between the NPU writing data to the second storage space and the DSS obtaining data from the second storage space during the image processing process, thereby improving the efficiency of image processing.

[0132] For example, Figure 4 As shown, after performing post-processing calculation based on the FSR algorithm and the resolution information and the resolution target to obtain the pixel expansion coefficient, the NPU can store the pixel expansion coefficient in the second storage space. Figure 5 As shown, after performing post-processing calculations according to the brightness target to obtain the pixel brightening coefficient for the dark area of ​​the pixel, the NPU can store the pixel brightening coefficient in the second storage space. Figure 6 As shown, after performing dynamic range calculation according to the color gamut information and the HDR target to obtain the HDR expansion coefficient for each pixel, the NPU can store the HDR expansion coefficient in the second storage space.

[0133] S306: The DSS obtains the first image from the third storage space, and obtains the second parameter information from the second storage space.

[0134] As an example, the DSS may acquire the second parameter information from the second storage space when it is recognized that a first flag (such as acquireFence) is newly added to the thread based on the fence synchronization mechanism.

[0135] It can be understood that the addition of the first mark in the thread indicates that the NPU has completed the caching of the second parameter information. Therefore, the DSS obtaining the second parameter information from the first storage space at this time will not conflict with the reading and writing of the NPU. The DSS can also read the second parameter information from the second storage space in a timely manner after the NPU completes the caching of the second parameter information, thereby reducing the delay of each processing link in the image processing process and improving the efficiency of image processing.

[0136] In some embodiments, after the DSS obtains the second parameter information from the second storage space, the DSS may insert a second mark (such as releaseFence) in the thread to indicate that the second parameter information has been read. When the DSS recognizes that a second mark (such as releaseFence) is newly added to the thread, the second parameter information in the second storage space may be deleted. In this way, the NPU can store the calculated second parameter information of the next frame of the image conveniently, and the storage load and processing load caused by redundant information can be avoided.

[0137] S307: The DSS adjusts the image parameters of the first image according to the second parameter information to obtain a second image.

[0138] Take the second parameter information including the pixel expansion coefficient as an example, Figure 4 As shown, after obtaining the pixel expansion coefficient from the second storage space, the DSS can perform corresponding pixel expansion according to the pixel expansion coefficient to obtain a second image that meets the resolution target. The more pixels the second image contains after pixel expansion, the clearer the image becomes, which can bring a better viewing experience to the user.

[0139] Or, taking the second parameter information including the pixel brightening coefficient as an example, Figure 5 As shown, after obtaining the pixel brightening coefficient from the second storage space, DSS can perform corresponding pixel dark area brightening according to the pixel brightening coefficient to obtain a second image that meets the brightness target. After the pixel dark area brightening, the brightness of each area of ​​the second image is more uniform, and the display effect presented to the user is also better, which not only facilitates the user to accurately obtain the information to be presented by the image, but also brings a better viewing experience to the user.

[0140] Alternatively, taking the second parameter information including the HDR expansion coefficient as an example, Figure 6 As shown, after obtaining the HDR expansion coefficient from the second storage space, the DSS can perform corresponding HDR pixel expansion according to the HDR expansion coefficient to obtain a second image that meets the HDR target. The second image after HDR pixel expansion can provide more dynamic range and image details to better reflect the visual effect of the first image, which can bring a better viewing experience to the user.

[0141] S308: The DSS sends the second image to the display screen for display.

[0142] It should be noted that the post-processing calculations described in the above embodiments of the present application, such as the FSR algorithm, pixel scanning algorithm, pixel brightening calculation, dynamic range calculation, etc., are only examples of several post-processing calculation methods. The embodiments of the present application are not limited to the specific algorithms used for performing local brightening, super-resolution display, HDR and other related post-processing calculations, and can be adjusted with reference to conventional technologies depending on the specific device capabilities, scene requirements, etc.

[0143] It can be understood that based on the image processing method provided in the embodiment of the present application, when the electronic device performs image processing, the NPU can perform post-processing related calculations for local brightening, super-resolution display, HDR, etc., and the DSS can adjust the image-related parameters according to the calculation results of the NPU. Exemplarily, after the GPU completes the image rendering, the GPU can cache the rendered image (such as the first image) in the third storage space of the memory, and store the parameter information of the rendered image (such as the first parameter information) in the first storage space; the NPU can read the first parameter information from the first storage space, perform post-processing related calculations according to the first parameter information and the parameter target, obtain the optimized parameter information (such as the second parameter information), and store the second parameter information in the second storage space; the DSS can read the second parameter information from the second storage space, read the rendered image from the third storage space, and adjust the parameters of the rendered image according to the second parameter information to obtain the second image. In this way, the powerful computing power of the NPU and the display effect processing capability of the DSS can be fully utilized to reduce the interface display delay while ensuring the interface display effect. For example, compared to the solution where the GPU is responsible for image rendering and post-processing (including post-processing related calculations and adjustment of image-related parameters based on calculation results), the GPU load can be greatly reduced, as well as the latency of the image processing process and the interface display effect can be guaranteed. For another example, compared to the solution where the DSS is responsible for post-processing (including post-processing related calculations and adjustment of image-related parameters based on calculation results), more image parameter post-processing optimization can be achieved to ensure the interface display effect.

[0144] In addition, since the solution can be completed based on the existing hardware of the electronic device, no additional cost will be added. For example, compared with the solution of performing post-processing related calculations by a separate graphics chip and adjusting image-related parameters based on the calculation results, the cost required for image processing can be greatly reduced.

[0145] In addition, based on the image processing method provided in the embodiment of the present application, the GPU, NPU and DSS can realize data reading and writing synchronization based on the fence synchronization mechanism to further reduce the delay between data writing and data reading in each processing link during the image processing process and improve the efficiency of image processing.

[0146] It should be understood that the various schemes of the embodiments of the present application can be used in reasonable combination, and the explanations or descriptions of the various terms appearing in the embodiments can be mutually referenced or explained in the various embodiments, without limitation.

[0147] It should also be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0148] It is understandable that, in order to implement the functions of any of the above-mentioned embodiments, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0149] The embodiments of the present application can divide the electronic device into functional modules. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0150] It should also be understood that each module in the electronic device can be implemented in software and / or hardware form, without specific limitation. In other words, the electronic device is presented in the form of functional modules. The "module" here can refer to a specific application integrated circuit ASIC, a circuit, a processor and a memory that executes one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions.

[0151] In an optional manner, when data transmission is implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is implemented in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disk (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0152] The steps of the method or algorithm described in conjunction with the embodiments of the present application can be implemented in hardware or by executing software instructions by a processor. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM) memory, register, hard disk, mobile hard disk, compact disc read-only memory (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). In addition, the ASIC can be located in an electronic device. Of course, the processor and the storage medium can also exist as discrete components.

[0153] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device includes a neural network processing unit NPU and a display service system DSS, the method includes: The NPU acquires first parameter information corresponding to the first image from the first storage space, where the first parameter information is used to represent one or more of the following image parameters of the first image: resolution, brightness, and color gamut; The NPU performs post-processing calculation according to the first parameter information and the parameter target to obtain second parameter information; The NPU stores the second parameter information in a second storage space; The DSS acquires the second parameter information from the second storage space, and acquires the first image from a third storage space; The DSS adjusts image parameters of the first image according to the second parameter information to obtain a second image.

2. The method according to claim 1, characterized in that The first parameter information includes resolution information of the first image, and the parameter target includes a resolution target; The NPU performs post-processing calculation according to the first parameter information and the parameter target to obtain second parameter information, including: The NPU obtains a pixel expansion coefficient according to the resolution information and the resolution target based on a preset resolution algorithm, and the second parameter information includes the pixel expansion coefficient; The DSS adjusts the image parameters of the first image according to the second parameter information to obtain a second image, including: The DSS performs pixel expansion on the second image according to the pixel expansion coefficient to obtain the second image.

3. The method according to claim 1 or 2, characterized in that: The first parameter information includes brightness information of pixels in the first image, and the parameter target includes a brightness target; The NPU performs post-processing calculation according to the first parameter information and the parameter target to obtain second parameter information, including: The NPU identifies a dark pixel area in the first image, and performs a pixel brightening calculation according to the brightness target to obtain a pixel brightening coefficient for the dark pixel area; The DSS adjusts the image parameters of the first image according to the second parameter information to obtain a second image, including: The DSS performs pixel brightening on the dark pixel area according to the pixel brightening coefficient to obtain the second image.

4. The method according to any one of claims 1 to 3, characterized in that The first parameter information includes color gamut information of pixels in the first image, and the parameter target includes a high dynamic range imaging HDR target; The NPU performs post-processing calculation according to the first parameter information and the parameter target to obtain second parameter information, including: The NPU performs a dynamic range calculation according to the color gamut information to obtain an HDR expansion coefficient for a pixel in the first image; The DSS adjusts the image parameters of the first image according to the second parameter information to obtain a second image, including: The DSS performs HDR pixel expansion on the first image according to the HDR expansion coefficient to obtain the second image.

5. The method according to any one of claims 1 to 4, characterized in that After the NPU stores the second parameter information in a second storage space, the method further includes: The NPU sets a first mark, where the first mark is used to indicate that the NPU has stored the second parameter information in the second storage space; Specifically, the DSS acquires the second parameter information from the second storage space: the DSS acquires the second parameter information from the second storage space according to the first mark.

6. The method according to any one of claims 1 to 5, characterized in that After the DSS acquires the second parameter information from the second storage space, the method further includes: The DSS sets a second flag, where the second flag is used to indicate that the DSS has acquired the second parameter information from the second storage space; The NPU deletes the second parameter information in the second storage space according to the second mark.

7. The method according to any one of claims 1 to 6, characterized in that The electronic device further includes a graphics processing unit (GPU), and the method further includes: The GPU renders the image to be displayed to obtain the first image; The GPU stores the first image in the third storage space, and stores the first parameter information corresponding to the first image in the first storage space.

8. The method according to claim 7, characterized in that After the GPU stores the first parameter information corresponding to the first image in the first storage space, the method further includes: The GPU sets a third mark, where the third mark is used to indicate that the GPU has stored the first parameter information in the first storage space; The NPU obtaining the first parameter information corresponding to the first image from the first storage space specifically includes: the NPU obtaining the first parameter information from the first storage space according to the third mark.

9. The method according to claim 7 or 8, characterized in that: After the NPU acquires the first parameter information from the first storage space, the method further includes: The NPU sets a fourth flag, where the fourth flag is used to indicate that the NPU has obtained the first parameter information from the first storage space; The GPU deletes the first parameter information in the first storage space according to the fourth mark.

10. The method according to any one of claims 1 to 9, characterized in that The electronic device further comprises an on-chip memory, wherein the first storage space, the second storage space and the third storage space are located in the on-chip memory.

11. The method according to any one of claims 1 to 10, characterized in that The first image is any one of the following: a game image frame, a video image frame, a viewfinder image, and a picture.

12. An electronic device, characterized in that: The electronic device comprises: A display screen, used for displaying images; a memory for storing computer program instructions; A processor, configured to execute the computer program instructions to support the electronic device to implement the method as described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processing circuit, the method according to any one of claims 1 to 11 is implemented.

14. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 11.

15. A chip system, characterized in that: The chip system includes a processing circuit and a storage medium, wherein the storage medium stores computer program instructions; when the computer program instructions are executed by the processing circuit, the method as described in any one of claims 1 to 11 is implemented.

Images