Image processing method and related device

Abstract

The invention provides an image processing method and a related device, the method is applied to an image processing system of electronic equipment, in the image processing system, the number of rendering buffer areas in an SRAM (Static Random Access Memory) is two, and a frame buffer area in charge of storing an image to be sent and displayed is set in a PSRAM. Thus, in the process of rendering the same frame of user interface, only the image data corresponding to the 1 / n frames of user interface can be cached in the SRAM at the same time, the memory occupation amount of the SRAM in the image rendering and sending display process of the electronic equipment can be effectively reduced, the memory resource of the SRAM is saved, and the stability of system operation is improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method and related devices. Background Art

[0002] With the accelerated development of new-generation information technologies such as virtual reality, the application scenarios of wearable devices such as smart watches and smart bracelets are becoming more and more extensive. The storage space of wearable devices includes static random-access memory (SRAM), which has a fast reading rate but a small storage capacity. In the process of rendering and displaying images on wearable devices, a large amount of SRAM storage space will be occupied, and the operation process of wearable devices is prone to jamming, which damages the user experience.

[0003] Therefore, how to reduce the SRAM memory usage in the process of rendering and displaying images is an urgent problem to be solved. Summary of the invention

[0004] The purpose of the present application is to provide an image processing method and related devices, which are applied to an image processing system of an electronic device, in which the number of rendering buffers in the SRAM is two, and the frame buffer responsible for storing the image to be displayed is set in the PSRAM. In this way, in the process of rendering the same frame of the user interface, only the image data corresponding to 1 / n frames of the user interface can be cached in the SRAM at the same time, which can effectively reduce the memory usage of the SRAM of the electronic device during the image rendering and display process, save the memory resources of the SRAM, and improve the stability of the system operation.

[0005] The above-mentioned and other objects are achieved by the features of the independent claims. Further implementations are reflected in the dependent claims, the description and the drawings.

[0006] In a first aspect, the present application provides an image processing method, which is applied to an image processing system, wherein the image processing system includes a static random access memory SRAM and a pseudo-static random access memory PSRAM, wherein the SRAM includes a first rendering buffer and a second rendering buffer, and the PSRAM includes at least one frame buffer. The method includes: rendering a first bitmap data, and caching the obtained first image data into the first rendering buffer; in the process of copying the first image data from the first rendering buffer to the at least one frame buffer, rendering the second bitmap data and caching the obtained second image data into the second rendering buffer, wherein the first image data and the second image data both correspond to 1 / n of the first picture, where n is an integer greater than 1.

[0007] In this method, there are two rendering buffers in SRAM, namely the first rendering buffer and the second rendering buffer. For any frame of the first screen that needs to be rendered, the two rendering buffers can work together, and each rendering buffer is only responsible for storing 1 / n of a frame of the user interface at a time. In addition, the frame buffer responsible for storing the image to be displayed is set in PSRAM. When the two rendering buffers work together to render the image, the image data in the rendering buffer will also be copied to the frame buffer synchronously. When the image data corresponding to the first screen is completely copied to the frame buffer, the image data in the frame buffer will be output to the display screen. In this way, for any frame of the user interface, the rendering buffer only needs to store 1 / n (n is an integer greater than 1) of the image data corresponding to the frame of the user interface each time, and the frame buffer is moved out to PSRAM. Therefore, in the process of image rendering and display, the memory occupation of the electronic device for the rendering buffer and the frame buffer on the SRAM is significantly reduced, which can effectively save the memory resources of the SRAM and improve the stability of the system operation.

[0008] In addition, when the image data in the rendering buffer is subsequently copied to the frame buffer, due to the low PSRAM read rate, the two rendering buffers work together. While the image data in one rendering buffer is copied to the frame buffer, the other bitmap data that has not been rendered is rendered, and the newly obtained image data is synchronously stored in another rendering buffer. This can avoid the negative impact caused by the low PSRAM read rate as much as possible, and ensure the image rendering efficiency while reducing the occupancy of SRAM memory.

[0009] Optionally, the above n can be any integer from 2 to 10.

[0010] Optionally, the image processing system can be applied to wearable devices, such as smart bracelets and smart watches. Optionally, the image processing system can also be applied to any type of embedded device including SRAM and display screen.

[0011] In combination with the first aspect, in a possible implementation, the at least one frame buffer includes a first frame buffer and a second frame buffer, and the first image data and the second image data are both cached in the first frame buffer. The method further includes: in the process of outputting the complete image data corresponding to the first picture in the first frame buffer to the display screen, rendering the second picture, and caching the obtained third image data in the second frame buffer, and the second picture is a picture displayed after the first picture.

[0012] It is understandable that, since the reading speed of PSRAM is slower than that of SRAM, and in this embodiment, since there are two frame buffers, the first frame buffer and the second frame buffer, the electronic device can render the next frame of the user interface (i.e., the second picture) while displaying a frame of the rendered user interface (i.e., the first picture), and store the third image data obtained by rendering the second picture in another frame buffer to wait for display. In this way, the influence of the slow reading and writing speed of PSRAM 22 on the screen frame rate can be minimized, and the occupancy of the SRAM memory can be reduced while ensuring the smoothness of the screen picture.

[0013] Optionally, the image data corresponding to the first picture in the first frame buffer can be output to the display screen under the control of the TE signal. The TE signal can be used to control the system to output the first picture and render the second picture at the same time. Similarly, when rendering the second picture, the first rendering buffer and the second rendering buffer can still work together, and each rendering buffer is only responsible for storing 1 / n of the second picture each time. When the two rendering buffers work together to render the image, the image data in the rendering buffer will also be synchronously copied to the second frame buffer. When the image data corresponding to the second picture is completely copied to the frame buffer, the image data in the frame buffer will be output to the display screen (under the control of the next TE signal).

[0014] In combination with the first aspect, in a possible implementation, before rendering the second bitmap data and caching the obtained second image data in the second rendering buffer, the method further includes: switching the rendering buffer to be worked on from the first rendering buffer to the second rendering buffer; in the case where fourth image data is cached in the second rendering buffer, clearing the fourth image data in the second rendering buffer, the fourth image data corresponding to 1 / n of a third picture, and the third picture is a picture displayed before the first picture.

[0015] It is understandable that after each rendering of an image and storing the obtained image in the above-mentioned first rendering buffer or the above-mentioned second rendering buffer, the system will switch the rendering buffer to be worked on to another rendering buffer to store the new image data obtained by subsequent rendering in the rendering buffer to be worked on. However, in the process of historical rendering and sending other pictures for display, the above-mentioned two buffers still store the image data corresponding to the historically displayed pictures (such as the third picture), and this part of the image data still occupies the memory of SRAM. Therefore, in this embodiment, after switching the rendering buffer to be worked, the system will first delete the image data historically cached in the rendering buffer to be worked, and then cache the image data corresponding to the picture to be displayed (i.e., the first picture) in the rendering buffer to be worked, so as to achieve the purpose of reducing the amount of SRAM memory occupied.

[0016] In combination with the first aspect, in a possible implementation, before caching the obtained third image data into the above-mentioned second frame buffer, the above-mentioned method also includes: switching the frame buffer to be worked from the above-mentioned first frame buffer to the above-mentioned second frame buffer; when the fifth image data is cached in the above-mentioned second frame buffer, clearing the above-mentioned fifth image data in the above-mentioned second frame buffer, the above-mentioned fifth image data is the complete image data corresponding to the third picture, and the above-mentioned third picture is the picture displayed before the above-mentioned first picture.

[0017] Similarly, after each time the image data stored in the first frame buffer or the second frame buffer is sent for display, the image data stored in the first frame buffer or the second frame buffer still occupies the memory of the PSRAM. Therefore, in this embodiment, when the system's working frame buffer is switched to another frame buffer, the system first deletes the image data cached in the working frame buffer, and then caches the image data corresponding to the picture to be displayed (i.e., the first picture) in the working frame buffer, so as to reduce the amount of system memory occupied.

[0018] In combination with the first aspect, in a possible implementation, the image processing system further includes a main control module, the SRAM is included in the main control module, and the PSRAM is located outside the main control module.

[0019] In the present application method, the main control module can also be called the main microcontroller unit (MCU), which is the main controller in the embedded system, usually a small computer system integrating multiple functional modules such as processor, memory, input and output interface, timer, ADC, DAC, etc. In terms of physical location, the SRAM located in the main MCU can be called "on-chip memory", and the PSRAM independent of the main MCU can be called "off-chip memory". It is understandable that since the main control MCU is responsible for most of the processing work in the electronic device, the effective allocation of on-chip memory is more important during the operation of the electronic device. In this embodiment, the frame buffer for sending and displaying is placed in the PSRAM as an off-chip memory, which can effectively reduce the occupation of on-chip memory during image rendering and sending and displaying, and the saved on-chip memory can be applied to other processes, which can effectively improve the stability of the system.

[0020] In the second aspect, the present application provides an image processing system, which includes a rendering module, a display driver module, a static random access memory SRAM and a pseudo-static random access memory PSRAM, the SRAM includes a first rendering buffer and a second rendering buffer, and the PSRAM includes at least one frame buffer, wherein: the above-mentioned rendering module is used to render the first bitmap data, and cache the obtained first image data in the above-mentioned first rendering buffer; the above-mentioned rendering module is also used to render the second bitmap data and cache the obtained second image data in the above-mentioned second rendering buffer during the process of copying the above-mentioned first image data from the above-mentioned first rendering buffer to the above-mentioned at least one frame buffer, and the above-mentioned first image data and the above-mentioned second image data both correspond to 1 / n of the first picture.

[0021] In combination with the second aspect, in a possible implementation, the at least one frame buffer includes a first frame buffer and a second frame buffer, the first image data and the second image data are both cached in the first frame buffer, and the rendering module is also used to render the second picture in the process of outputting the complete image data corresponding to the first picture in the first frame buffer to the display screen, and cache the obtained third image data in the second frame buffer, and the second picture is a picture displayed after the first picture.

[0022] In a third aspect, the present application provides an electronic device, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method in the first aspect or any possible implementation of the first aspect.

[0023] In a fourth aspect, the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions so that the electronic device executes a method as in the first aspect or any possible implementation of the first aspect.

[0024] In a fifth aspect, the present application provides a computer-readable storage medium, comprising instructions, which, when executed on an electronic device, enable the electronic device to execute a method in the first aspect or any possible implementation of the first aspect.

[0025] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute a method in the first aspect or any possible implementation of the first aspect.

[0026] The beneficial effects of the devices and products involved in the second to sixth aspects of the present application can be specifically referred to the aforementioned description of the beneficial effects of the image rendering method in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 An architecture diagram of an image processing system provided in an embodiment of the present application;

[0028] Figure 2 An architecture diagram of an image processing system provided in an embodiment of the present application;

[0029] Figure 3 An interactive flow chart of an image processing method provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of an image rendering scene provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of a process of displaying a rendered user interface and synchronously rendering the next frame of the user interface provided in an embodiment of the present application;

[0032] Figure 6 A schematic diagram of an image rendering scene provided in an embodiment of the present application;

[0033] Figure 7 A schematic diagram of a process of displaying a rendered user interface and synchronously rendering the next frame of the user interface provided in an embodiment of the present application;

[0034] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0036] Since the embodiments of the present application involve the application of neural networks, for ease of understanding, the relevant terms involved in the embodiments of the present application are first introduced below.

[0037] (1) Static random access memory and pseudo-static random access memory

[0038] Static random-access memory (SRAM) is a type of random access memory, which is characterized by the fact that as long as the power is on, the internally stored data can be kept constant without periodic refresh. The "static" characteristic of SRAM means that it does not need to refresh the data regularly, so it has higher performance and stability. However, SRAM has a low integration level and high power consumption. SRAM of the same capacity requires a larger volume, which makes it more expensive. Due to the high speed and stability of SRAM, it is often used in memories that require fast access, such as cache and register files. SRAM can be used as a cache between the CPU and the main memory, reducing the access delay to the main memory and improving system performance. However, due to the size of wearable devices and the high cost of SRAM, the amount of memory that can be provided by SRAM contained in wearable devices is generally small. When the memory of the SRAM in a wearable device is occupied by a large amount, the operation of the wearable device will be stuck.

[0039] Pseudo static random access memory (PSRAM) is a storage technology that combines the characteristics of dynamic random access memory DRAM and SRAM. PSRAM has the advantages of simple interface and fast random read and write speed similar to SRAM, while also having the high density characteristics of DRAM. It adopts the 1T1C (one transistor and one capacitor) architecture. Compared with SRAM, PSRAM is smaller in size, consumes less power, and has a larger capacity, but is not as fast as SRAM in reading speed.

[0040] (2) Render buffer and frame buffer

[0041] A renderbuffer is a 2D image buffer allocated by an application. A renderbuffer can be used to allocate and store color, depth, or stencil values, and to store rendered images.

[0042] A framebuffer is a container object that contains a set of image views that represent actual image resources, which can be color attachments, depth attachments, or template attachments. The framebuffer is used to store images output by the renderbuffer, which can be called by upper-layer applications and then output to the device's display screen for display.

[0043] In the present application, the rendering buffer can be placed in SRAM, the frame buffer can be placed in PSRAM, and the number of SRAM rendering buffers can be 2. When rendering an image, for any frame of image, the electronic device can render and draw the frame of image n times. In this process, each rendering buffer can work alternately, and any rendering buffer only stores 1 / n of the frame image each time to reduce the occupation of SRAM memory by image data during the rendering process. Specifically, when a rendering buffer stores 1 / n of the rendered image, the electronic device will copy 1 / n of the image in the rendering buffer to the frame buffer and wait for it to be displayed. While copying, the electronic device will continue to render the part of the frame image that has not been rendered, and store the rendered 1 / n in another rendering buffer, and so on, until the last 1 / n of the frame image is rendered.

[0044] Optionally, in the present application, the number of frame buffers in the PSRAM can also be 2. When sending and displaying images, for any two frames of images that need to be displayed continuously, after the previous frame of the image is rendered and copied as a whole to one of the above two frame buffers, the electronic device can send and display the frame image, and at the same time render and copy the next frame image to the other frame buffer, and do this alternately to minimize the frame interval between the two frames of images when they are sent and displayed, and further improve the smoothness of the electronic device display screen. For details, please refer to the subsequent embodiments.

[0045] (3)TE signal

[0046] The image frame will have a tearing effect (TE) during display. The TE signal is a signal generated by the chip to prevent the tearing problem when the image is refreshed during display. When the next frame is ready to refresh, the chip generates a TE signal. After the chip sends the TE signal, the UI thread will start rendering the next frame of the user interface and display the previous frame of the user interface.

[0047] The storage space of wearable devices can be divided into SRAM and PSRAM. SRAM has a high read rate but a small storage space, while PSRAM has a slow read rate but a large storage space. In existing wearable devices, the rendering buffer used to store rendered images and the frame buffer used to store images to be displayed are both stored in SRAM. However, for such devices, due to the size limitation of the device, the amount of SRAM memory it contains is very limited. When the electronic device renders and displays the image, the image data contained in the rendering buffer and frame buffer in the SRAM will take up a large amount of memory space, which can easily cause the operation of the wearable device to become stuck, damaging the user experience. Please refer to Figure 1 .

[0048] Figure 1 FIG. 1 shows an architecture diagram of an image processing system included in an electronic device. Figure 1 As shown, the image processing system includes a multimedia management module 10, a main control module 11, a pseudo-static random access memory 12 (hereinafter referred to as "PSRAM 12"), a display control module 13 and a display screen 14. Among them:

[0049] The multimedia management module 10 may also be referred to as an embedded multi-media card (EMMC), which is an embedded flash memory that integrates a controller, a memory, and an interface, and is mainly used for data storage and transmission in mobile devices and embedded systems, and can be used to store various data types such as applications, audio, video, and pictures (including bitmap data). In electronic devices, the multimedia management module 10 is usually used as a built-in memory, which can provide fast data storage and reading capabilities, and support fast startup and application response of wearable devices.

[0050] The main control module 11 is a main controller in an electronic device that integrates multiple functional modules such as a processor, a memory, and an input / output interface. It is responsible for controlling the coordinated work of various components in the electronic device to ensure the stability and reliability of the entire system. The main control module 11 may include a UI engine 111 and a static random access memory 112 (hereinafter referred to as SRAM 112); further, the SRAM 112 may also include a rendering buffer 112a and a frame buffer 112b. Among them, the UI engine 111 is used to create and manage the user interface displayed by the electronic device. The rendering buffer 112a is used to store the image data obtained after the UI engine 111 renders the interface, and the frame buffer 112b is used to store the image data copied from the rendering buffer 112a, so as to output the interface corresponding to the image data to the display screen.

[0051] The pseudo-static random access memory 12 (hereinafter referred to as PSRAM 12) is a memory independent of the main control module 11, so in this application, the PSRAM 12 can also be called an off-chip memory, and the corresponding SRAM 112 can also be called an on-chip memory. The PSRAM 12 can include a media resource buffer 121, which can be used to cache the bitmap data loaded by the UI engine from the multimedia management module 10.

[0052] The display control module 13, also known as the display controller (LCD-TFT Display Controller, LTDC), can be used to control and manage the working process of the display screen 14. Specifically, the display control module 13 can send image data to the display screen 14 under the control of the main control module 11 to achieve continuous screen refresh.

[0053] Specifically, during the entire process of the electronic device rendering and displaying an image, the UI engine 111 will first load the bitmap data corresponding to the user interface from the multimedia management module 10, and further send it to the media resource buffer 121 in the PSRAM 12 for caching. After the UI engine 111 receives the rendering instruction, the UI engine 111 will read the bitmap data from the media resource buffer 121 again, and the bitmap data corresponds to a complete frame of the user interface (i.e., a full screen image subsequently displayed on the display screen 14).

[0054] Afterwards, the UI engine starts to render the bitmap data. After the bitmap data corresponding to a complete frame of the user interface is rendered, the UI engine stores the rendered image data in the rendering buffer 112a. Similarly, the image data stored in the rendering buffer 112a also corresponds to a complete frame of the user interface.

[0055] Next, the image data in the rendering buffer 112a will be copied to the frame buffer 112b to be displayed. Similarly, the image data stored in the frame buffer 112b also corresponds to a complete frame of the user interface. It can be understood that at this time, the rendering buffer 112a and the frame buffer 112b in the SRAM 112 both store image data corresponding to a complete frame of the user interface.

[0056] Finally, the image data in the frame buffer 112b will be further transmitted to the display control module 13. The display control module 13 will send a driving instruction to the display screen 14 to refresh the display screen 14 and output the user interface corresponding to the above image data to the display screen 14.

[0057] Combined with the above description, it can be known that in the process of rendering and displaying an image by an electronic device, a complete frame of image will be stored in both the rendering buffer 112a and the frame buffer 112b in the SRAM 112, so the rendering buffer 112a and the frame buffer 112b will occupy a large amount of memory resources of the SRAM 112. Especially for wearable devices, due to the limitation of the device size, the memory space that the SRAM in the wearable device can provide is only about 2.5MB, but in the process of rendering and displaying an image by the device, the rendering buffer and the frame buffer need to occupy about 1.24MB of memory space, which almost occupies half of the total memory space that the SRAM can provide. In this case, the available memory of the system is significantly reduced, which is likely to cause the electronic device to freeze, or even cause the system of the electronic device to crash, causing serious damage to the user experience.

[0058] In view of the problems existing in the above-mentioned image processing system, the present application provides an image processing system and an image processing method applied to an electronic device, in which the number of rendering buffers in the SRAM is increased to two, and for any frame of the user interface that needs to be rendered, the two rendering buffers can work together, and each rendering buffer is only responsible for storing 1 / n of a frame of the user interface at a time. In addition, the frame buffer responsible for storing the image to be displayed is set in the PSRAM, and when the two rendering buffers work together to render the image, the electronic device will also synchronously copy the image data in the rendering buffer to the frame buffer, and when the image data corresponding to a complete frame of the user interface is completely copied to the frame buffer, the electronic device can issue a display instruction to output the image data in the frame buffer to the display screen. In this way, for any frame of the user interface, the rendering buffer only needs to store 1 / n (n is an integer greater than 1) of the image data corresponding to the frame of the user interface each time, and the frame buffer is moved out to the PSRAM, so in the process of image rendering and display, the memory occupation of the electronic device for the rendering buffer and the frame buffer to the SRAM is significantly reduced, which can effectively save the memory resources of the SRAM and improve the stability of the system operation.

[0059] First combine Figure 2 An image processing system provided in an embodiment of the present application is introduced.

[0060] like Figure 2As shown, the image processing system includes a multimedia management module 20, a main control module 21, a pseudo-static random access memory 22 (hereinafter referred to as "PSRAM 22"), a display control module 23 and a display screen 24. It should be understood that the division of the various modules in the above image processing system is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. For example, the above modules can be separately established processing elements, or they can be integrated into the same chip for implementation. In addition, they can also be stored in the storage module of the controller in the form of program code, and a certain processing element of the processor can call and execute the functions of the above modules. In addition, the modules can be integrated together or implemented independently.

[0061] The specific functions of the multimedia management module 20, the main control module 21 and the display control module 23 can be referred to in the above description. Figure 1 The related descriptions of the multimedia management module 10, the main control module 11 and the display control module 13 are not repeated here.

[0062] It should be noted that, in the embodiment of the present application, the static random access memory 212 (hereinafter referred to as SRAM 212) contained in the main control module 21 may include two rendering buffers, namely Figure 2 , and the rendering buffer 212a and rendering buffer 212b shown in . In addition, the SRAM 212 may not include a frame buffer. The rendering buffer 212a and the rendering buffer 212b may work together alternately, alternately caching the image data obtained by rendering the same frame of the user interface, and each rendering buffer only renders 1 / n of a frame of the user interface at a time; while one of the rendering buffers caches (if the rendering buffer stores historical cached image data, it is cleared first) the image data obtained by the latest rendering of the 1 / n frame of the user interface, the other rendering buffer can synchronously copy the image data corresponding to the historically cached 1 / n frame of the user interface to the frame buffer in the PSRAM 22, and so on and so forth, until a complete frame of the user interface is rendered. It is understandable that when the image data in the rendering buffer is subsequently copied to the frame buffer, due to the low PSRAM reading rate, the two rendering buffers work together. While the image data in one rendering buffer is copied to the frame buffer, the other part of the bitmap data that has not been rendered is rendered, and the newly obtained image data is synchronously stored in another rendering buffer. This can avoid the negative impact caused by the low PSRAM reading rate as much as possible, and ensure the image rendering efficiency while reducing the occupancy of SRAM memory.

[0063] There may be two frame buffers in PSRAM 22, namely Figure 222a and 222b shown in the figure. The frame buffer 222a and the frame buffer 222b are responsible for storing the image data copied from the rendering buffer 212a and the rendering buffer 212b, and further outputting them to the display screen 24 for the user to browse. Accordingly, in the embodiment of the present application, the frame buffer 222a and the frame buffer 222b can also work together: when one of the frame buffers successfully obtains the image data corresponding to a complete frame of the user interface from the rendering buffer 212a and the rendering buffer 212b, the image data corresponding to the complete frame of the user interface can be output to the display screen 24 for the user to browse under the control of the display control module 23; at the same time, the UI engine will render the next frame of the user interface n times in succession, and the image data obtained by each rendering will also be alternately stored in the rendering buffer 212a and the rendering buffer 212b according to the aforementioned method. That is to say, when the electronic device sends a frame of the user interface cached in one of the frame buffers for display, the electronic device is also synchronously rendering the next frame of the user interface, and the image data corresponding to the frame of the user interface is copied from the rendering buffer 212a and the rendering buffer 212b to the other rendering buffer. In this way, the influence of the slow reading and writing speed of PSRAM 22 on the screen frame rate can be reduced as much as possible, and the occupancy of SRAM memory can be reduced under the premise of ensuring the smoothness of the screen picture.

[0064] Similarly, during the entire process of the electronic device rendering and displaying an image, the UI engine 211 will first load the bitmap data corresponding to the user interface from the multimedia management module 20, and further send it to the media resource buffer 221 in the PSRAM 22 for caching. After the UI engine 211 receives the rendering instruction, the UI engine 211 will read the bitmap data from the media resource buffer 221 again, and the bitmap data corresponds to a complete frame of the user interface (i.e., a full screen image subsequently displayed on the display screen 24).

[0065] After that, the UI engine 211 starts to render the bitmap data in n times. Optionally, the specific value of n can be set when the electronic device leaves the factory. Optionally, the value of n can be any integer between 2 and 10.

[0066] Take n as 2 as an example for explanation. For the bitmap data corresponding to any frame of user interface, the UI engine 211 can render the bitmap data twice in succession. It is assumed here that the user interface UI1 corresponds to two parts, bitmap1 and bitmap2. When rendering the bitmap data corresponding to the user interface UI1, the UI engine 211 can first render the bitmap data bitmap1, and store the image data pic1 obtained after rendering (the image data pic1 only corresponds to the user interface UI1 of 1 / 2 frame) in the rendering buffer 212a. Afterwards, the main control module 21 will further copy the image data pic1 to the frame buffer 222a. While copying the image data pic1 to the frame buffer 222a, the UI engine will synchronously render the bitmap data bitmap2, and store the image data pic2 obtained after rendering (the image data pic2 also corresponds to the user interface UI1 of 1 / 2 frame) in the rendering buffer 212b. Similarly, the main control module 21 will further copy the image data pic1 to the frame buffer 222a. It is understandable that at this time, the corresponding complete image data has been stored in the frame buffer 222a.

[0067] When the user interface UI1 is displayed, the image data corresponding to the user interface UI1 stored in the frame buffer 222a (i.e., the image data pic1 and the image data pic2) will be further transmitted to the display control module 23. The display control module 23 will send a driving instruction to the display screen 24 to make the display screen 24 refresh the display screen and output the user interface corresponding to the above image data to the display screen 24.

[0068] If there is another frame of user interface UI2 to be displayed after displaying user interface UI1, then while outputting the image data corresponding to user interface UI1 in frame buffer 222a to display screen 24, UI engine 211 can synchronously render the bitmap data of user interface UI2 in the aforementioned manner, and the obtained image data will also be transmitted to the frame buffer in PSRAM 22 via the rendering buffer in SRAM 212 in the aforementioned manner.

[0069] It should be noted that, since the rendering buffer 212a and the rendering buffer 212b have already stored part of the image data corresponding to the user interface UI1, after the bitmap data of the user interface UI2 is rendered and the corresponding image data is obtained, before the image data is stored in the rendering buffer 212a or the rendering buffer 212b, the electronic device will first delete the existing image data in the rendering buffer 212a or the rendering buffer 212b, and then cache the image data corresponding to the user interface UI1 to the rendering buffer 212a or the rendering buffer 212b, so as to reduce the memory usage of the SRAM 212. Similarly, since the image data corresponding to the user interface UI1 has been cached in the frame buffer, and this part of the image data is currently being sent for display, the image data corresponding to the user interface UI2 cached in the rendering buffer 212a and the rendering buffer 212b can be copied to the frame buffer 222b. After the user interface UI1 is sent for display, the image data corresponding to the user interface UI2 stored in the frame buffer 222b will be output to the display screen 24 under the control of the display control module 23, and the display screen 24 can display the user interface UI2.

[0070] It is understandable that the reading speed of PSRAM 22 is slower than that of SRAM. Figure 2 In the image processing system shown, since there are two frame buffers, the electronic device can render the next frame of the user interface while displaying a frame of the user interface that has been rendered, and store the rendered image data in another frame buffer to wait for display. In this way, the influence of the slow reading and writing rate of the PSRAM 22 on the screen frame rate can be minimized, and the occupation of the SRAM memory can be reduced under the premise of ensuring the smoothness of the screen picture.

[0071] It can be understood that, in the process of rendering the same frame of the user interface, at the same time, only one of the rendering buffers 212a and 212b caches the image data corresponding to the 1 / n frame of the user interface, and because the frame buffer has been moved to the PSRAM 22 outside the main control module 21, during the process of rendering and displaying the image, only the image data corresponding to the 1 / n frame of the user interface may exist in the SRAM 212, and Figure 1 The SRAM 112 in the image processing system shown in the figure caches at least two complete sets of image data corresponding to the same user interface (including all the image data corresponding to one frame of the user interface stored in the rendering buffer 112a and all the image data corresponding to one frame of the user interface stored in the frame buffer 112b). Therefore, the image processing system provided by the present application can effectively reduce the memory usage of the SRAM during the image rendering process, thereby improving the overall performance of the electronic device system.

[0072] Combined with the above description, it can be seen that in this application, when the above n is larger, the memory usage of SRAM can be effectively reduced during the image rendering process; but when the above n is larger, the UI engine 211 renders the same frame of user interface more times, and the computing power required for the main control module 21 is also greater. Therefore, in the embodiment of the present application, the specific value of the above n can be determined based on the requirements of the electronic device for both memory and computing power. Optionally, the specific value of n can be any integer from 2 to 10.

[0073] In addition, in a possible implementation, there may be only one frame buffer in the PSRAM 22. In this case, the rendering buffer 212a and the rendering buffer 212b can still work together alternately during the rendering phase, that is, the image data obtained by rendering the same frame of the user interface are cached alternately, and each rendering buffer only renders 1 / n of a frame of the user interface at a time. However, during the display phase, after the electronic device successfully renders a frame of the user interface and caches the corresponding image data in the frame buffer, the electronic device can clear the data in the frame buffer after displaying the frame of the user interface, and then start rendering the bitmap data corresponding to the next frame of the user interface, and cache the rendered image data in the above-mentioned frame buffer.

[0074] Combining the above Figure 2 The image processing system described in the following will introduce an image processing method provided by the present application. Specifically, the image processing method can be applied to a system deployed with Figure 2 The electronic device of the image processing system shown in the figure may be a wearable device, a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, and a vehicle-mounted device. The embodiment of the present application does not impose any special restrictions on the specific type of the electronic device. Figure 3 As shown, Figure 3 The hardware terminal module in can be included in Figure 2 In the main control module 21 shown in FIG. , the display module and the rendering module can be included in Figure 2 The image processing method may include but is not limited to the following steps:

[0075] S101: The hardware interrupt module receives a first TE signal.

[0076] The first TE signal can be Figure 2 The main control module 21 generates and sends it to the hardware interrupt module.

[0077] After receiving the first TE signal, the hardware interrupt module can determine whether a rendering / sending / copying operation is currently occurring in the system based on the interrupt trigger function TECallBack. If there is no rendering / sending / copying operation in the system, the hardware interrupt module executes the subsequent step S102, i.e., sending a rendering instruction to the rendering module.

[0078] S102: The hardware interrupt module sends a first rendering instruction to the rendering module.

[0079] S103: The rendering module renders the first part of the first bitmap, and stores the first image data obtained by rendering in a first rendering buffer.

[0080] After the hardware interrupt module sends the first rendering instruction to the rendering module, the rendering module can respond to the instruction and start drawing the first user interface. The first bitmap is the bitmap data corresponding to the first user interface. Combined with the above description, it can be seen that the first bitmap can be cached in the media resource buffer of the PSRAM. After the rendering module receives the first rendering instruction, the rendering module obtains the first bitmap from the media resource buffer.

[0081] Specifically, in the process of rendering the first bitmap, the rendering module may draw a whole screen page n times starting from the top area of ​​the first user interface, that is, draw 1 / n of the first bitmap each time. It is understandable that the first part of the first bitmap corresponds to the top 1 / n of the first user interface, and the first image data is the top 1 / n of the first user interface. Specific rendering operations may include but are not limited to operations such as merging invalid areas (i.e., areas that need to be redrawn) and calculating the position of the rendering area.

[0082] After obtaining the above-mentioned first image data, the rendering module can cache the first image data in the first rendering buffer. It can be understood that, since there are two rendering buffers in the SRAM in the application embodiment, before executing step S103, the main control module can determine the rendering buffer to be worked as the above-mentioned first rendering buffer; after the switch is successful, if other image data is currently stored in the first rendering buffer (for example, the image data cached in the first rendering buffer when the rendering module renders the previous frame of the user interface), the image data already stored in the first rendering buffer will be cleared before the above-mentioned first image data is cached in the first rendering buffer. Only after the existing image data in the first rendering buffer is cleared, the rendering module will cache the above-mentioned first image data in the first rendering buffer.

[0083] S104: The rendering module sends the first image data to the display driving module.

[0084] S105: The display driving module copies the first image data to the first frame buffer.

[0085] After rendering the first part of the first bitmap, the rendering module may send the rendered first image data to the display driver module, and the display driver module caches the first image data in the first frame buffer. Specifically, the display driver module may include a graphics accelerator DMA2D, and the display driver module may copy the first image data to the first frame buffer via DMA2D.

[0086] Combining the above Figure 2 It can be seen from the relevant description that while caching the first image data to the first frame buffer, the rendering module will continue to render the next 1 / n unrendered bitmap data in the first bitmap. That is, while executing steps S105-S106, the electronic device is also synchronously executing subsequent steps S106-S107.

[0087] S106: The rendering module switches the rendering buffer to be worked on to the second rendering buffer.

[0088] It is understandable that, since the first rendering buffer area has already stored the above-mentioned first image data, and the above-mentioned first image data is in the process of being copied to the above-mentioned first frame buffer at this time, the rendering module can switch the rendering buffer to be worked from the above-mentioned first rendering buffer to the above-mentioned second rendering buffer to cache the image data obtained in the subsequent rendering process.

[0089] Similarly, after the rendering buffer to be worked is switched to the second rendering buffer, if other image data has been stored in the second rendering buffer (for example, image data cached in the second rendering buffer when the rendering module rendered the previous frame of the user interface), the rendering module can clear the image data already stored in the first rendering buffer. Only after the existing image data in the first rendering buffer is cleared, the rendering module will start to render the subsequent part of the first bitmap.

[0090] S107: The rendering module renders the second part of the first bitmap, and stores the rendered second image data in a second rendering buffer.

[0091] It can be understood that the second part of the first image corresponds to another 1 / n continuous with the top 1 / n in the first user interface, and the second image data is another 1 / n image connected to the top 1 / n in the first user interface. The rendering module will render the second part of the first image, and the rendered second image data will be cached in the second rendering buffer.

[0092] S108: The rendering module sends the second image data to the display driving module.

[0093] S109: The display driving module copies the second image data to the first frame buffer.

[0094] After rendering the second part of the first bitmap, the rendering module may send the rendered first image data to the display driver module, and the display driver module caches the first image data in the first frame buffer.

[0095] It should be noted that in the present application, after each time the display module completes drawing 1 / n of the first bitmap, it will copy the rendered image data from the rendering buffer that caches the image data to the corresponding frame buffer, and at the same time switch the next rendering buffer to draw the next area until the first bitmap is completely rendered. That is to say, after the rendering module successfully renders the last 1 / n part of the first bitmap, while copying the image data corresponding to the last 1 / n part of the first bitmap stored in the rendering buffer to the first frame buffer, because the first bitmap has been fully rendered, the rendering module can no longer perform concurrent rendering operations until the central control module receives the next TE signal and sends the next rendering instruction to the rendering module (which can be understood as the electronic device starting to prepare for the rendering and display of the next frame of the user interface). The rendering module will continue to render the next frame of the user interface in accordance with the aforementioned rendering method.

[0096] For example, when the value of n is 2, it indicates that the second part of the first bitmap is the last 1 / 2 of the first bitmap. Therefore, when the rendering module caches the second image data cached in the second rendering buffer into the first frame buffer via the display driver module, the rendering module does not concurrently render other bitmap data. Optionally, after copying the second image data to the first frame buffer, the rendering module may switch the rendering buffer from the second rendering buffer to the first rendering buffer, and clear the first image data stored in the first rendering buffer, so that the first rendering buffer can cache subsequent image data (for example, image data obtained after the rendering module renders the next frame of the user interface).

[0097] Correspondingly, when the value of the above n is greater than 2, it means that the second part of the above first image is not the data of the last 1 / n part of the first image. In the process that the rendering module caches the second image data cached in the second rendering buffer into the first frame buffer via the display driver module, the rendering module will continue to concurrently render the subsequent 1 / n part of the data that has not been rendered in the above first image, and so on; until the rendering module completes the rendering of the last 1 / n part of the data of the above first image, and caches the image data corresponding to the rendered last 1 / n part of the data into the first frame buffer value, the main control module will continue to send the TE signal, that is, when n is greater than 2, after executing step S109, the electronic device still needs to continue to render the part that has not been rendered in the first image in the above manner, until the rendering of the first image corresponding to the first user interface is completed, the electronic device will execute the subsequent step S110 to start the rendering and display operation of the next frame of the user interface.

[0098] S110: The hardware interrupt module receives a second TE signal.

[0099] Similarly, the second TE signal can be Figure 2 The main control module 21 in the system generates and sends the signal to the hardware interrupt module. The second TE signal is used to prompt the system to start rendering the next frame of the user interface.

[0100] S111: The hardware interrupt module switches the frame buffer to be worked on to the second frame buffer.

[0101] Combined with the above description, it can be known that in order to ensure the display frame rate of the electronic device, while the display driver module outputs the first user interface rendered in the first frame buffer to the display screen for display, the rendering module will synchronously render the next frame user interface (i.e., the second user interface), and will cache the rendered image data in the frame buffer in the same manner as in the above description and wait for display. Therefore, before formally rendering the bitmap data corresponding to the second interface, the hardware interrupt module will first switch the frame buffer to be worked from the first frame buffer to the above-mentioned second frame buffer, so as to provide for the subsequent caching of the image data corresponding to the second user interface. Similarly, after switching the frame buffer to be worked to the above-mentioned second frame buffer, if the second frame buffer already has image data stored therein (for example, image data corresponding to another user interface displayed before the first user interface), the hardware interrupt module can first clear the image data stored therein so as to subsequently store the image data corresponding to the second user interface.

[0102] S112: The hardware interrupt module sends a second rendering instruction to the rendering module.

[0103] S113: The hardware interrupt module sends a first display sending instruction to the display driver module.

[0104] After receiving the second TE signal, if there is no rendering / image sending / image copying operation in the system, the hardware interrupt module can send a second rendering instruction to the rendering module, and synchronously send a first display sending instruction to the display driver module, so that the electronic device executes subsequent steps S111-S121. Specifically, in the second rendering instruction and the first display sending instruction sent synchronously, the rendering instruction is used to start the rendering module to render the next frame of the user interface (i.e., the second user interface), and the first display sending instruction is used to start the display driver module to output the first interface that has been rendered and stored in the first frame buffer to the display screen.

[0105] S114: The display driving module outputs the first user interface to the display screen.

[0106] The display driver module outputs the first user interface image data (including the aforementioned first image data and second image data) stored in the first frame buffer to the display screen through the LTDC, so that the first user interface is displayed on the display screen for the user to browse. In this process, the rendering module synchronously renders the bitmap corresponding to the second user interface, and caches the rendered image data in the above-mentioned second frame buffer with the help of the display driver module. That is to say, when the electronic device executes step S114, the electronic device also synchronously executes the subsequent steps S115-step S121.

[0107] S115: The rendering module renders the first part of the second bitmap, and stores the third image data obtained by rendering in the first rendering buffer.

[0108] S116: The rendering module sends the third image data to the display driving module.

[0109] S117: The display driving module copies the third image data to the second frame buffer.

[0110] S118: The rendering module switches the rendering buffer to be worked on to the second rendering buffer.

[0111] S119: The rendering module renders the second part of the second bitmap, and stores fourth image data obtained by rendering in a second rendering buffer.

[0112] S120: The rendering module sends the fourth image data to the display driving module.

[0113] S121: The display driving module copies the fourth image data to the second frame buffer.

[0114] The second bitmap is the bitmap data corresponding to the second user interface. Similarly, the second bitmap can be cached in the media resource buffer of the PSRAM. After the rendering module receives the second rendering instruction, the rendering module obtains the second bitmap from the media resource buffer.

[0115] Similarly, when rendering the second bitmap, the display module will copy the rendered image data from the rendering buffer that caches the image data to the second frame buffer each time after drawing 1 / n of the second bitmap, and switch to the next rendering buffer to draw the next area, until the second bitmap is completely rendered, and the second buffer caches the completed image data corresponding to the second user interface. For details, please refer to the relevant instructions for the aforementioned steps S103-step S109, which will not be repeated here. After the rendering module copies the image data corresponding to the last 1 / n part of the second bitmap stored in the rendering buffer to the first frame buffer, because the second bitmap has been fully rendered, the rendering module can no longer perform concurrent rendering operations until the central control module receives the next TE signal, and then the electronic device will continue to render the next frame of the user interface, and so on.

[0116] Figure 4 and Figure 5 It shows the change process of the data in SRAM and PSRAM when the electronic device renders and displays the picture when the value of the aforementioned n is 2.

[0117] First combine Figure 4 The rendering process of the electronic device is described. Figure 4 As shown, user interface 401 is a user interface that needs to be rendered and displayed by the electronic device in the next frame, and its corresponding bitmap data may include two parts, namely, bitmap data 401a and bitmap data 401b. Figure 4 The PSRAM and SRAM shown in the figure may be the PSRAM 22 and SRAM 212 in the above description, wherein rb1 and rb2 are two rendering buffers included in the SRAM, and fb1 and fb2 are two frame buffers included in the PSRAM.

[0118] Combined with the above description, it can be known that when the value of n is 2, for any frame of the user interface, the electronic device will render the user interface twice from top to bottom, rendering 1 / 2 of the user interface each time. After receiving the TE signal TE1 for starting to render the user interface 401, the electronic device can Figure 4 The three stages shown in FIG. 4 are used to render the bitmap data corresponding to the user interface 401 .

[0119] like Figure 4As shown in (A) in FIG. 1 , in the first stage, the electronic device first renders the bitmap data 401a, and the rendered image data 401A is first cached in the rendering buffer rb1 in the SRAM. Figure 4 It can be seen from (A) that at this time, only 1 / 2 frame of image data is stored in the SRAM, that is, the image data 401A in the rendering buffer rb1.

[0120] Afterwards, if Figure 4 As shown in (B), in the second stage, the electronic device will render the bitmap data 401b, and the rendered image data 401B will be cached in rb2 in the rendering buffer in the SRAM. At the same time, the electronic device will copy the image data 401A stored in the rendering buffer rb1 to the frame buffer fb1 in the PSRAM. Combined with the above description, it can be seen that after the electronic device renders the bitmap data 401b, the electronic device will switch the working rendering buffer back to the rendering buffer rb1, and at the same time clear the image data 401A already stored in the rendering buffer rb1. Therefore, in the second stage, only 1 / 2 frame of image data, that is, the image data 401B in the rendering buffer rb2, is stored in the SRAM.

[0121] Afterwards, if Figure 4 As shown in (C), in the third stage, combined with the above description, it can be seen that since the electronic device has completed rendering the user interface 401 in this stage, the electronic device will only copy the image data 401B stored in the rendering buffer rb2 to the frame buffer fb1 in the PSRAM. At this time, the frame buffer fb1 has cached the complete image data corresponding to the user interface 401. And in the third stage, only 1 / 2 frame of image data, that is, the image data 401B in the rendering buffer rb2, is stored in the SRAM.

[0122] Combined with the above description, it can be seen that during the entire process of rendering the user interface 401, the SRAM of the electronic device only needs to cache at most 1 / 2 frame of image data. Figure 1 The rendering method used by the image processing system shown in the figure can significantly reduce the amount of SRAM memory occupied by the electronic device in the embodiment of the present application, and its occupied amount is only Figure 1 1 / 4 of the SRAM memory usage.

[0123] exist Figure 4 On the basis of Figure 5 It shows the specific process of the electronic device displaying the rendered user interface 401 and synchronously rendering the next frame of user interface 501.

[0124] like Figure 5As shown, user interface 501 is the next frame of user interface after user interface 401, and its corresponding bitmap data may also include two parts, namely, bitmap data 501a and bitmap data 501b.

[0125] After receiving the TE signal TE2 for starting the rendering of the user interface 501, the electronic device may first switch the frame buffer to be worked from the frame buffer fb1 to the frame buffer fb2; if other image data has been cached in the frame buffer fb2, this part of the image data is cleared first. Figure 5 The bitmap data corresponding to the user interface 501 is rendered in the three stages shown in , and the user interface 401 is synchronously output to the display screen of the electronic device during the rendering process. It should be noted in advance that the moment when the electronic device starts rendering the user interface 501 and the moment when the electronic device starts displaying the user interface 401 can be the same, but the total time length of the electronic device rendering the user interface 501 and the total time length of the electronic device displaying the user interface 401 can be the same or different. For ease of explanation, Figure 5 The description is made by taking the example that the total time duration for the electronic device to render the user interface 501 is the same as the total time duration for the electronic device to display the user interface 401 .

[0126] like Figure 5 As shown in (A), in the first stage, the electronic device first renders the bitmap data 501a, and the rendered image data 501A is first cached in the rendering buffer rb1 in the SRAM; at the same time, the electronic device starts to render the image data in the frame buffer fb1 (i.e., the image data 401A and image data 401B corresponding to the user interface 401). After the bitmap data 501a is rendered, the electronic device switches the rendering buffer to be worked on to the rendering buffer rb2, and at the same time clears the image data 402B currently stored in the rendering buffer rb2. Figure 5 It can be seen from (A) that at this time, only 1 / 2 frame of image data is stored in the SRAM, that is, the image data 501A in the rendering buffer rb1.

[0127] Afterwards, if Figure 5As shown in (B), in the second stage, the electronic device will render the bitmap data 401b, and the rendered image data 401B will be cached in rb2 in the rendering buffer in the SRAM. At the same time, the electronic device will copy the image data 501A stored in the rendering buffer rb1 to the frame buffer fb2 in the PSRAM. In combination with the above description, it can be seen that after the electronic device renders the bitmap data 501b, the electronic device will switch the working rendering buffer back to the rendering buffer rb1, and at the same time clear the image data 501A already stored in the rendering buffer rb1. Therefore, in the second stage, only 1 / 2 frame of image data, that is, the image data 501B in the rendering buffer rb2, is stored in the SRAM. It can be understood that in this stage, the electronic device is still in the process of sending the image data in the frame buffer fb1 for display.

[0128] Afterwards, if Figure 5 As shown in (C), we come to the third stage. Combined with the above description, since the electronic device has completed rendering the user interface 501 in this stage, the electronic device will only copy the image data 501B stored in the rendering buffer rb2 to the frame buffer fb2 in the PSRAM. At this time, the frame buffer fb2 has cached the complete image data corresponding to the user interface 501. And in the third stage, the SRAM also only stores 1 / 2 frame of image data, that is, the image data 501B in the rendering buffer rb2. In addition, in this stage, the electronic device has completed the display of the image data in the frame buffer fb1, and the user interface 401 is successfully displayed on the display screen of the electronic device.

[0129] According to the above description, during the whole process of rendering the user interface 501 and displaying the user interface 401, the SRAM of the electronic device still only needs to cache at most 1 / 2 frame of image data, and the SRAM memory occupied by the electronic device can be greatly reduced.

[0130] Similarly, if subsequent electronic devices receive other TE signals, the electronic devices can still follow Figure 5 The process shown continues to render the next frame of the user interface and display the user interface 501, and so on, which will not be repeated here.

[0131] In the embodiment of the present application, the rendering buffer rb1 may be referred to as the "first rendering buffer", rb2 may be referred to as the "second rendering buffer", the frame buffer fb1 may be referred to as the "first frame buffer", the frame buffer fb2 may also be referred to as the "second frame buffer", and the frame buffer fb1 and the frame buffer fb2 may be collectively referred to as "at least one frame buffer". In the process of rendering and displaying the user interface 401, the bitmap data 401a may be referred to as the "first bitmap data", the image data 401A may be referred to as the "first image data", the bitmap data 401b may be referred to as the "second bitmap data", the image data 401B may be referred to as the "second image data", the user interface 401 may be referred to as the "first screen", the user interface 501 may be referred to as the "second screen", and the image data 501A and the image data 501B may be collectively referred to as the "third image data".

[0132] Figure 6 and Figure 7 It shows the change process of the data in SRAM and PSRAM when the electronic device renders and displays the picture when the value of the aforementioned n is 3.

[0133] First combine Figure 6 The rendering process of the electronic device is described. Figure 6 As shown, user interface 601 is a user interface that needs to be rendered and displayed by the electronic device in the next frame, and its corresponding bitmap data may include three parts, namely, bitmap data 601a, bitmap data 601b and bitmap data 601c. Figure 6 The PSRAM and SRAM shown in the figure may be the PSRAM 22 and SRAM 212 in the above description, wherein rb3 and rb4 are two rendering buffers included in the SRAM, and fb3 and fb4 are two frame buffers included in the PSRAM.

[0134] Combined with the above description, it can be known that when the value of n is 3, for any frame of the user interface, the electronic device will render the user interface three times from top to bottom, rendering 1 / 3 of the user interface each time. After receiving the TE signal TE3 for starting to render the user interface 601, the electronic device can Figure 6 The three stages shown in FIG. 6 are used to render the bitmap data corresponding to the user interface 601 .

[0135] like Figure 6 As shown in (A), in the first stage, the electronic device first renders the bitmap data 601a, and the rendered image data 601A is first cached in the rendering buffer rb3 in the SRAM. Figure 6It can be seen from (A) that at this time, only 1 / 3 frame of image data is stored in the SRAM, that is, the image data 601A in the rendering buffer rb3.

[0136] Afterwards, if Figure 6 As shown in (B), in the second stage, the electronic device will render the bitmap data 601b, and the rendered image data 601B will be cached in rb4 in the rendering buffer in the SRAM. At the same time, the electronic device will copy the image data 601A stored in the rendering buffer rb3 to the frame buffer fb3 in the PSRAM. Combined with the above description, it can be seen that after the electronic device renders the bitmap data 601b, the electronic device will switch the working rendering buffer back to the rendering buffer rb3, and at the same time clear the image data 601A already stored in the rendering buffer rb3. Therefore, in the second stage, only 1 / 3 of the frame of image data, that is, the image data 601B in the rendering buffer rb4, is stored in the SRAM.

[0137] Afterwards, if Figure 6 As shown in (C), in the third stage, the electronic device will render the bitmap data 601c, and the rendered image data 601C will be cached in the rendering buffer rb3 in the SRAM. At the same time, the electronic device will copy the image data 601B stored in the rendering buffer rb4 to the frame buffer fb3 in the PSRAM. Similarly, after the electronic device renders the bitmap data 601c, the electronic device will switch the working rendering buffer back to the rendering buffer rb4, and at the same time clear the image data 601B already stored in the rendering buffer rb4. Therefore, in the third stage, only 1 / 3 of the frame of image data, that is, the image data 601C in the rendering buffer rb3, is stored in the SRAM.

[0138] Next, if Figure 6 As shown in (D), in the fourth stage, combined with the above description, it can be seen that since the electronic device has completed rendering the user interface 601 in this stage, the electronic device will only copy the image data 601C stored in the rendering buffer rb3 to the frame buffer fb3 in the PSRAM. At this time, the frame buffer fb3 has cached the complete image data corresponding to the user interface 601. And in the fourth stage, only 1 / 3 of the frame image data is stored in the SRAM, that is, the image data 601C in the rendering buffer rb3.

[0139] Combined with the above description, it can be seen that during the entire process of rendering the user interface 601, the SRAM of the electronic device only needs to cache at most 1 / 3 of the image data of the frame. Figure 1The rendering method used by the image processing system shown in the figure can significantly reduce the amount of SRAM memory occupied by the electronic device in the embodiment of the present application, and its occupied amount is only Figure 1 1 / 6 of the SRAM memory usage.

[0140] exist Figure 6 On the basis of Figure 7 It shows the specific process of the electronic device displaying the rendered user interface 601 and synchronously rendering the next frame of user interface 701.

[0141] like Figure 7 As shown, user interface 701 is the next frame user interface after user interface 401, and its corresponding bitmap data may also include three parts, namely, bitmap data 701a, bitmap data 701b and bitmap data 701c.

[0142] After receiving the TE signal TE4 for starting the rendering of the user interface 701, the electronic device may first switch the frame buffer to be worked from the frame buffer fb3 to the frame buffer fb4; if other image data has been cached in the frame buffer fb4, this part of the image data is cleared first. Figure 7 The bitmap data corresponding to the user interface 701 is rendered in the four stages shown in , and the user interface 401 is synchronously output to the display screen of the electronic device during the rendering process. Similarly, the moment when the electronic device starts rendering the user interface 701 can be the same as the moment when the electronic device starts displaying the user interface 601, but the total time for the electronic device to render the user interface 701 and the total time for the electronic device to display the user interface 601 can be the same or different. For ease of explanation, Figure 7 The description is made by taking the example that the total time duration for the electronic device to render the user interface 701 is the same as the total time duration for the electronic device to display the user interface 601 .

[0143] like Figure 7 As shown in (A), in the first stage, the electronic device first renders the bitmap data 701a, and the rendered image data 701A is first cached in the rendering buffer rb4 in the SRAM; at the same time, the electronic device begins to render the image data in the frame buffer fb3 (i.e., the image data 601A, image data 601B, and image data 601C corresponding to the user interface 601). After the bitmap data 701a is rendered, the electronic device switches the rendering buffer to be worked on to the rendering buffer rb3, and at the same time clears the image data 602C currently stored in the rendering buffer rb3. Figure 7 It can be seen from (A) that at this time, only 1 / 3 frame of image data is stored in the SRAM, that is, the image data 701A in rb4 in the rendering buffer.

[0144] Afterwards, if Figure 7 As shown in (B), in the second stage, the electronic device will render the bitmap data 701b, and the rendered image data 701B will be cached in rb3 in the rendering buffer in the SRAM. At the same time, the electronic device will copy the image data 701A stored in the rendering buffer rb4 to the frame buffer fb4 in the PSRAM. In combination with the above description, it can be seen that after the electronic device renders the bitmap data 701b, the electronic device will switch the working rendering buffer back to the rendering buffer rb4, and at the same time clear the image data 701A already stored in the rendering buffer rb4. Therefore, in the second stage, only 1 / 3 of the frame of image data is stored in the SRAM, that is, the image data 701B in rb3 in the rendering buffer. It can be understood that in this stage, the electronic device is still in the process of sending the image data in the frame buffer fb3 for display.

[0145] Afterwards, if Figure 7 As shown in (C), in the third stage, the electronic device will render the bitmap data 701c, and the rendered image data 701C will be cached in rb4 in the rendering buffer in the SRAM. At the same time, the electronic device will copy the image data 701B stored in the rendering buffer rb3 to the frame buffer fb4 in the PSRAM. In combination with the above description, it can be seen that after the electronic device renders the bitmap data 701c, the electronic device will switch the working rendering buffer back to the rendering buffer rb3, and at the same time clear the image data 701B already stored in the rendering buffer rb3. Therefore, in the second stage, only 1 / 3 of the frame of image data is stored in the SRAM, that is, the image data 701C in rb4 in the rendering buffer. It can be understood that in this stage, the electronic device is still in the process of sending the image data in the frame buffer fb3 for display.

[0146] Afterwards, if Figure 7 As shown in (D), in the fourth stage, combined with the above description, it can be seen that since the electronic device has completed the rendering of the user interface 701 in this stage, the electronic device will only copy the image data 701C stored in the rendering buffer rb4 to the frame buffer fb4 in the PSRAM. At this time, the frame buffer fb4 has cached the complete image data corresponding to the user interface 701. And in the third stage, only 1 / 3 of the frame image data is stored in the SRAM, that is, the image data 701C in the rendering buffer rb4. In addition, in this stage, the electronic device has completed the display of the image data in the frame buffer fb3, and the user interface 601 is successfully displayed on the display screen of the electronic device.

[0147] Combined with the above description, it can be known that during the entire process of rendering the user interface 701 and displaying the user interface 601, the SRAM of the electronic device still only needs to cache at most 1 / 3 of the image data of the frame, and the SRAM memory occupied by the electronic device can be greatly reduced.

[0148] Similarly, if subsequent electronic devices receive other TE signals, the electronic devices can still follow Figure 7 The process shown continues to render the next frame of the user interface and display the user interface 701, and so on, which will not be repeated here.

[0149] In the present application, the electronic device only renders 1 / n of a frame each time and caches the rendered image data in a rendering buffer. Figure 4-Figure 7 It can be seen that the larger the above n is, the lower the amount of SRAM memory that can be effectively reduced during image rendering. For example, when the above n is 4, the electronic device only needs to cache 1 / 4 frame of image data in the SRAM during rendering and displaying the image. However, when n is larger, the electronic device renders the same frame of user interface more times, and the computing power consumed by the electronic device is also greater. Therefore, in an embodiment of the present application, the specific value of the above n can be determined based on the electronic device's requirements for memory and computing power. Optionally, the specific value of n can be any integer from 2 to 10.

[0150] In the embodiment of the present application, the rendering buffer rb3 may be referred to as the "first rendering buffer", rb4 may be referred to as the "second rendering buffer", the frame buffer fb3 may be referred to as the "first frame buffer", the frame buffer fb4 may also be referred to as the "second frame buffer", and the frame buffer fb3 and the frame buffer fb4 may be collectively referred to as "at least one frame buffer". In the process of rendering and displaying the user interface 601, the user interface 601 may be referred to as the "first screen", the user interface 701 may be referred to as the "second screen", and the image data 701A and the image data 701C may be collectively referred to as the "third image data".

[0151] Next, a schematic diagram of the structure of the electronic device 100 provided in an embodiment of the present application is introduced.

[0152] like Figure 8As shown, the electronic device 100 can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA) or a dedicated camera (such as a SLR camera, a card camera), etc. The electronic device 100 can also be the second electronic device in the aforementioned description, and the present application does not impose any limitation on the specific type of the electronic device.

[0153] The electronic device 100 may include a processor 110 , a memory 122 , a universal serial bus (USB) interface 130 , a charging management module 140 , a power management module 141 , a battery 142 , a camera 193 , a display screen 194 , and the like.

[0154] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0155] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0156] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0157] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0158] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0159] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0160] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the memory 122, the external memory, the display screen 194, the camera 193, etc. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0161] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0162] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0163] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0164] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image or video signal. The ISP outputs the digital image or video signal to the DSP for processing. The DSP converts the digital image or video signal into an image or video signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0165] The memory 122 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 122. The memory 122 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image or video playback function, etc.). The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.). In addition, the memory 122 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0166] In the present application, the processor 110 also includes a static random access memory SRAM 1101, and the SRAM 1101 may include two rendering buffers, namely, a rendering buffer 1101a and a rendering buffer 1101b. During the image rendering process of the electronic device 100, for any frame of the picture, the electronic device can render and draw the frame of the picture n times. In this process, the rendering buffer 1101a and the rendering buffer 1101b can work alternately, and any rendering buffer only stores 1 / n of the frame image each time to reduce the image data occupying the SRAM memory during the rendering process. Specifically, the rendering buffer 1101a and the rendering buffer 1101b can be the aforementioned Figure 2 The rendering buffer 212a and the rendering buffer 212b in the embodiment may also be the aforementioned Figure 3 The first render buffer and the second render buffer in .

[0167] In addition, in the present application, the memory 122 includes a pseudo-static random access memory PSRAM 1211 independent of the processor 110. The PSRAM 1211 may include two frame buffers, namely, a frame buffer 1211a and a frame buffer 1211b. When displaying an image, for any two frames of images that need to be displayed continuously, after the previous frame of the image is rendered and copied as a whole to one of the two frame buffers, the electronic device can display the frame image, and at the same time render the next frame image and copy it to the other frame buffer, and perform this alternating operation to minimize the frame interval between the two frames of the image when displaying them.

[0168] There are one or more processors 110 and a memory 122 in the electronic device 100; the memory 122 is coupled to the one or more processors 110, and the memory 122 is used to store computer program code, which includes computer instructions. The one or more processors call the computer instructions to enable the electronic device 100 to execute the image processing method provided in the embodiment of the present application.

[0169] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0170] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0171] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. An image processing method, characterized in that: The method is applied to an image processing system, the image processing system includes a static random access memory SRAM and a pseudo static random access memory PSRAM, the SRAM includes a first rendering buffer and a second rendering buffer, the PSRAM includes at least one frame buffer, and the method includes: Rendering the first image data, and caching the obtained first image data into the first rendering buffer; In the process of copying the first image data from the first rendering buffer to the at least one frame buffer, second bitmap data is rendered and the obtained second image data is cached in the second rendering buffer, and the first image data and the second image data both correspond to 1 / n of the first picture, where n is an integer greater than 1.

2. The method according to claim 1, characterized in that The at least one frame buffer includes a first frame buffer and a second frame buffer, the first image data and the second image data are both cached in the first frame buffer, and the method further includes: In the process of outputting the complete image data corresponding to the first picture in the first frame buffer to the display screen, the second picture is rendered and the obtained third image data is cached in the second frame buffer, and the second picture is displayed after the first picture.

3. The method according to claim 1 or 2, characterized in that: Before rendering the second bitmap data and caching the obtained second image data into the second rendering buffer, the method further includes: Switching the rendering buffer to be worked on from the first rendering buffer to the second rendering buffer; In the case where fourth image data is cached in the second rendering buffer, the fourth image data in the second rendering buffer is cleared, the fourth image data corresponds to 1 / n of a third picture, and the third picture is a picture displayed before the first picture.

4. The method according to claim 2 or 3, characterized in that: Before caching the obtained third image data into the second frame buffer, the method further includes: Switching the frame buffer to be worked from the first frame buffer to the second frame buffer; In the case where fifth image data is cached in the second frame buffer, the fifth image data in the second frame buffer is cleared, the fifth image data being complete image data corresponding to a third picture, and the third picture being a picture displayed before the first picture.

5. The method according to any one of claims 1 to 4, characterized in that: The image processing system further comprises a main control module, the SRAM is contained in the main control module, and the PSRAM is located outside the main control module.

6. An image processing system, characterized in that: The system includes a rendering module, a display driver module, a static random access memory SRAM and a pseudo static random access memory PSRAM, the SRAM includes a first rendering buffer and a second rendering buffer, the PSRAM includes at least one frame buffer, wherein: The rendering module is used to render the first first image data, and cache the obtained first image data into the first rendering buffer; The rendering module is also used to render second bitmap data and cache the obtained second image data into the second rendering buffer during the process of copying the first image data from the first rendering buffer to the at least one frame buffer, and the first image data and the second image data both correspond to 1 / n of the first picture.

7. The system according to claim 6, characterized in that The at least one frame buffer includes a first frame buffer and a second frame buffer, the first image data and the second image data are both cached in the first frame buffer, The rendering module is also used to render a second picture while outputting the complete image data corresponding to the first picture in the first frame buffer to the display screen, and cache the obtained third image data in the second frame buffer, and the second picture is the picture displayed after the first picture.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors, a memory and a display screen; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 5.

9. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method as described in any one of claims 1-5.

10. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 5.

Images