Image processing methods and related devices

Abstract

The application provides an image processing method and related device, the method is applied to the image processing system of an electronic device, in the image processing system, the number of rendering buffers in the SRAM is two, and the frame buffer responsible for storing the image to be displayed is set in the PSRAM. In this way, in the process of rendering the same frame user interface, the SRAM can only cache the image data corresponding to 1 / n frame user interface at the same time, which can effectively reduce the memory occupation of the SRAM in the process of image rendering and display, save the memory resources of the SRAM, and improve the stability of system operation.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to image processing methods and related devices. Background Technology

[0002] With the accelerated development of next-generation information technologies such as virtual reality, wearable devices, such as smartwatches and smart bracelets, are finding increasingly widespread applications. Wearable devices utilize static random-access memory (SRAM) for storage. While SRAM offers fast read speeds, its storage capacity is relatively small. During image rendering and display processes, SRAM storage is heavily consumed, making the device prone to stuttering and negatively impacting the user experience.

[0003] Therefore, how to reduce the SRAM memory usage during image rendering and display is an urgent problem to be solved. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method and related apparatus. This method is applied to an image processing system of an electronic device. In this system, there are two rendering buffers in the SRAM, and the frame buffer responsible for storing the image to be displayed is located in the PSRAM. Thus, during the rendering of the same user interface frame, the SRAM can cache only the image data corresponding to 1 / n frames of the user interface at any given time. This effectively reduces the memory usage of the electronic device in the SRAM during image rendering and display, saves SRAM memory resources, and improves the stability of system operation.

[0005] The aforementioned and other objectives will be achieved through the features described in the independent claims. Further implementations are illustrated in the dependent claims, the specification, and the drawings.

[0006] In a first aspect, this application provides an image processing method applied to an image processing system, the image processing system including a static random access memory (SRAM) and a pseudo static random access memory (PSRAM), the SRAM including a first rendering buffer and a second rendering buffer, the PSRAM including at least one frame buffer, the method including: rendering first bitmap data, and caching the obtained first image data in the first rendering buffer; during the process of copying the first image data from the first rendering buffer to the at least one frame buffer, rendering second bitmap data and caching the obtained second image data in the second rendering buffer, wherein the first image data and the second image data both correspond to 1 / n of a first frame, where n is an integer greater than 1.

[0007] In this method, there are two rendering buffers in SRAM: the first rendering buffer and the second rendering buffer. For any frame of the first screen to be rendered, the two rendering buffers can work together. Each rendering buffer is only responsible for storing 1 / n of the user interface frame at a time. In addition, the frame buffer responsible for storing the image to be displayed is set in PSRAM. When the two rendering buffers work together to render the image, the image data in the rendering buffer is also copied to the frame buffer synchronously. When the image data corresponding to the first screen is completely copied to the frame buffer, the image data in the frame buffer is output to the display screen. In this way, for any frame of the user interface, the rendering buffer only needs to store 1 / n (n is an integer greater than 1) of the image data corresponding to that frame of the user interface at a time, and the frame buffer is moved out to PSRAM. Therefore, during the image rendering and display process, the memory usage of the rendering buffer and frame buffer in SRAM by the electronic device is significantly reduced, which can effectively save SRAM memory resources and improve the stability of system operation.

[0008] Furthermore, when copying image data from the render buffer to the frame buffer, due to the low PSRAM read rate, the two render buffers work together. While copying image data from one render buffer to the frame buffer, the remaining bitmap data that could not be rendered is rendered, and the newly obtained image data is synchronously stored in the other render buffer. This can minimize the negative impact of the low PSRAM read rate, reduce the occupation of SRAM memory, and ensure image rendering efficiency.

[0009] Optionally, n can be any integer from 2 to 10.

[0010] Optionally, the image processing system described above can be applied to wearable devices, such as smart bracelets and smartwatches. Alternatively, the image processing system can also be applied to any type of embedded device that includes SRAM and a display screen.

[0011] In conjunction with the first aspect, in one possible implementation, the at least one frame buffer includes a first frame buffer and a second frame buffer, the first image data and the second image data are both cached in the first frame buffer, and the method further includes: during the process of outputting the complete image data corresponding to the first screen in the first frame buffer to the display screen, rendering the second screen, and caching the resulting third image data in the second frame buffer, wherein the second screen is a screen displayed after the first screen.

[0012] Understandably, PSRAM has a slower read speed than SRAM. However, in this embodiment, because there are two frame buffers, a first frame buffer and a second frame buffer, the electronic device can render the next frame of the user interface (i.e., the second frame) while simultaneously displaying a rendered first frame (i.e., the first screen), and store the resulting third image data in another frame buffer for later display. This minimizes the impact of the slow PSRAM read / write speed on the screen frame rate, reducing SRAM memory usage while ensuring smooth screen display.

[0013] Optionally, the image data corresponding to the first frame in the first frame buffer can be output to the display screen under the control of the TE signal. This TE signal can be used to control the system to output the first frame and simultaneously render the second frame. Similarly, when rendering the second frame, the first and second rendering buffers can still work together. Each rendering buffer is only responsible for storing 1 / n of the second frame at a time. When the two rendering buffers are rendering the image together, the image data in the rendering buffers will also be synchronously copied to the second frame buffer. When the image data corresponding to the second frame is completely copied to the frame buffer, the image data in the frame buffer will be output to the display screen (under the control of the next TE signal).

[0014] In conjunction with the first aspect, in one possible implementation, before rendering the second bitmap data and caching the resulting second image data in the second rendering buffer, the method further includes: switching the rendering buffer to be worked from the first rendering buffer to the second rendering buffer; if the second rendering buffer contains cached fourth image data, clearing the fourth image data in the second rendering buffer, the fourth image data corresponding to 1 / n of the third screen, the third screen being the screen displayed before the first screen.

[0015] Understandably, after each image rendering and storage of the resulting image in either the first or second rendering buffer, the system switches the pending rendering buffer to another buffer to store newly rendered image data. However, during historical rendering and display of other screens, the two buffers still store image data corresponding to previously displayed screens (e.g., the third screen), which still occupies SRAM memory. Therefore, in this embodiment, after switching the pending rendering buffer, the system first deletes the historically cached image data in the pending rendering buffer, and then caches the image data corresponding to the screen to be displayed (i.e., the first screen) in the pending rendering buffer to reduce SRAM memory usage.

[0016] In conjunction with the first aspect, in one possible implementation, before caching the obtained third image data into the second frame buffer, the method further includes: switching the frame buffer to be worked from the first frame buffer to the second frame buffer; if the second frame buffer contains cached fifth image data, clearing the fifth image data from the second frame buffer, wherein the fifth image data is complete image data corresponding to the third screen, and the third screen is a screen displayed before the first screen.

[0017] Similarly, after each display of the image data stored in the first or second frame buffer, the image data stored in the first or second frame buffer still occupies PSRAM memory. Therefore, in this embodiment, when the system switches to another frame buffer, the system first deletes the historical cached image data of the frame buffer to be displayed, and then caches the image data corresponding to the screen to be displayed (i.e., the first screen) into the frame buffer to be displayed, thereby reducing the system memory usage.

[0018] In conjunction with the first aspect, in one possible implementation, the image processing system further includes a main control module, wherein the SRAM is contained within the main control module and the PSRAM is located outside the main control module.

[0019] In this application, the main control module can also be called the main microcontroller unit (MCU). The main MCU is the main controller in the embedded system, typically a small computer system integrating multiple functional modules such as a processor, memory, input / output interfaces, timers, ADCs, and DACs. Physically, the SRAM located within the main MCU can be called "on-chip memory," while the PSRAM, independent of the main MCU, can be called "off-chip memory." Understandably, since the main control MCU is responsible for most of the processing work in the electronic device, the efficient allocation of on-chip memory is crucial during the device's operation. In this embodiment, the frame buffer used for display is placed in the PSRAM, which is off-chip memory. This effectively reduces the on-chip memory usage during image rendering and display, and the saved on-chip memory can be used in other processes, effectively improving system stability.

[0020] Secondly, this application provides an image processing system, which includes a rendering module, a display driver module, a static random access memory (SRAM), and a pseudo static random access memory (PSRAM). The SRAM includes a first rendering buffer and a second rendering buffer, and the PSRAM includes at least one frame buffer. The rendering module is used to render first bitmap data and cache the obtained first image data in the first rendering buffer. The rendering module is also used to render second bitmap data and cache the obtained second image data in the second rendering buffer during the process of copying the first image data from the first rendering buffer to the at least one frame buffer. The first image data and the second image data both correspond to 1 / n of the first frame.

[0021] In conjunction with the second aspect, in one possible implementation, the at least one frame buffer includes a first frame buffer and a second frame buffer. The first image data and the second image data are both cached in the first frame buffer. The rendering module is further configured to render the second image while outputting the complete image data corresponding to the first image in the first frame buffer to the display screen, and cache the resulting third image data in the second frame buffer. The second image is the image displayed after the first image.

[0022] Thirdly, this application provides an electronic device comprising: one or more processors and a memory; the memory is coupled to the one or more processors and is used to store computer program code, the computer program code including computer instructions, wherein the one or more processors invoke the computer instructions to cause the electronic device to perform the method of the first aspect or any possible implementation thereof.

[0023] Fourthly, this application provides a chip system applied to an electronic device, the chip system including one or more processors, the processors being configured to invoke computer instructions to cause the electronic device to perform methods as described in the first aspect or any possible implementation thereof.

[0024] Fifthly, this application provides a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in the first aspect or any possible implementation thereof.

[0025] In a sixth aspect, this application provides a computer program product containing instructions that, when the computer program product is run on an electronic device, cause the electronic device to perform the method as described in the first aspect or any possible implementation thereof.

[0026] For details regarding the beneficial effects of the apparatus and products involved in aspects two through six of this application, please refer to the foregoing description of the beneficial effects of the image rendering method in aspect one, which will not be repeated here. Attached Figure Description

[0027] Figure 1 An architecture diagram of an image processing system provided in this application embodiment;

[0028] Figure 2 An architecture diagram of an image processing system provided in this application embodiment;

[0029] Figure 3 An interactive flowchart of an image processing method provided in an embodiment of this application;

[0030] Figure 4 A scene illustration of image rendering provided in an embodiment of this application;

[0031] Figure 5 This is a schematic diagram illustrating a process of displaying a rendered user interface and simultaneously rendering the next frame of the user interface, as provided in an embodiment of this application.

[0032] Figure 6 A scene illustration of image rendering provided in an embodiment of this application;

[0033] Figure 7 This is a schematic diagram illustrating a process of displaying a rendered user interface and simultaneously rendering the next frame of the user interface, as provided in an embodiment of this application.

[0034] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0036] Since the embodiments of this application involve the application of neural networks, for ease of understanding, the relevant terms involved in the embodiments of this application will be introduced below.

[0037] (1) Static Random Access Memory and Pseudo-Static Random Access Memory

[0038] Static random-access memory (SRAM) is a type of random access memory characterized by its ability to retain stored data as long as it is powered on, without requiring periodic refresh. The "static" nature of SRAM means it doesn't need to be refreshed periodically, resulting in high performance and stability. However, SRAM has lower integration density, higher power consumption, and requires a larger volume for the same capacity, leading to higher costs. Due to its high speed and stability, SRAM is commonly used in memory applications requiring fast access, such as caches and register files. SRAM can act as a cache between the CPU and main memory, reducing access latency to main memory and improving system performance. However, due to the size limitations of wearable devices and the high cost of SRAM, the amount of SRAM provided in wearable devices is generally small. When the SRAM in a wearable device is heavily used, the device's operation can become sluggish.

[0039] Pseudo-static random access memory (PSRAM) is a storage technology that combines the characteristics of dynamic random access memory (DRAM) and SRAM. PSRAM offers advantages similar to SRAM, such as a simple interface and fast random read / write speeds, while also possessing the high-density characteristics of DRAM. It uses a 1T1C (one transistor and one capacitor) architecture. Compared to SRAM, PSRAM is smaller, consumes less power, and has a larger capacity, but its read speed is slower than SRAM.

[0040] (2) Render buffer and frame buffer

[0041] A render buffer is a 2D image buffer allocated by the application. It can be used to allocate and store color, depth, or stencil values, and to store rendered images.

[0042] A framebuffer is a container object that contains a set of image views representing actual image resources, which can be color attachments, depth attachments, or stencil attachments. The framebuffer stores images output from the render buffer, which can then be accessed by upper-layer applications and displayed on the device's screen.

[0043] In this application, the render buffer can be placed in SRAM, and the frame buffer can be placed in PSRAM, with a maximum of two SRAM render buffers. When rendering an image, for any given frame, the electronic device can render and draw it in n steps. During this process, each render buffer can work alternately, with each render buffer storing only 1 / n of the frame image at a time, thus reducing the amount of image data occupying SRAM memory during rendering. Specifically, after one render buffer stores 1 / n of the rendered image, the electronic device copies that 1 / n image from the render buffer to the frame buffer for display. Simultaneously, the electronic device continues rendering the unrendered portion of the frame image and stores the rendered 1 / n in another render buffer. This process is repeated alternately until the last 1 / n of the frame image is rendered.

[0044] Optionally, in this application, the number of frame buffers in the PSRAM can also be two. When displaying images, for any two frames that need to be displayed consecutively, after the previous frame is rendered and copied to one of the two frame buffers, the electronic device can display that frame while simultaneously rendering the next frame and copying it to the other frame buffer. This process is repeated alternately to minimize the frame interval when displaying two frames, further improving the smoothness of the electronic device's display. For details, please refer to the following embodiments.

[0045] (3)TE signal

[0046] Image frames can exhibit a tearing effect (TE) during display. The TE signal is a chip-generated signal used to prevent screen tearing during refresh. The chip generates a TE signal when it's ready to refresh the next frame. Only after the chip sends the TE signal will the UI thread begin rendering the next frame of the user interface and display the rendered user interface from the previous frame.

[0047] Wearable device storage can be divided into SRAM and PSRAM. SRAM has a high read speed but small storage space, while PSRAM has a slow read speed but large storage space. In existing wearable devices, the rendering buffer used to store rendered images and the frame buffer used to store images to be displayed are both located in SRAM. However, due to the size limitations of such devices, the amount of SRAM memory they contain is very limited. When the electronic device renders and displays images, the image data contained in the rendering buffer and frame buffer in SRAM can occupy a large amount of memory space, which can easily cause the wearable device to lag and impair the user experience. Please refer to [link to relevant documentation]. Figure 1 .

[0048] Figure 1 An architecture diagram of an image processing system included in an electronic device is shown. Figure 1 As shown, the image processing system includes a multimedia management module 10, a main control module 11, a pseudo-static random access memory 12 (hereinafter referred to as "PSRAM 12"), a display control module 13, and a display screen 14. Wherein:

[0049] The multimedia management module 10, also known as an embedded multimedia card (EMMC), is an embedded flash memory that integrates a controller, memory, and interface. It is primarily used for data storage and transmission in mobile devices and embedded systems, and can store various data types such as applications, audio, video, and images (including bitmap data). In electronic devices, the multimedia management module 10 is typically used as built-in memory, providing fast data storage and retrieval capabilities, supporting rapid startup and application response in wearable devices.

[0050] The main control module 11 is the main controller in the electronic device, integrating multiple functional modules such as processor, memory, and input / output interfaces. It is responsible for controlling the coordinated operation of various components in the electronic device, ensuring the stability and reliability of the entire system. The main control module 11 may include a UI engine 111 and a static random access memory 112 (hereinafter referred to as SRAM 112). Furthermore, the SRAM 112 may simultaneously include a rendering buffer 112a and a frame buffer 112b. The UI engine 111 is used to create and manage the user interface displayed by the electronic device. The rendering buffer 112a is used to store the image data obtained after the UI engine 111 renders the interface, and the frame buffer 112b is used to store the image data copied from the rendering buffer 112a, in order to output the interface corresponding to the image data to the display screen.

[0051] The pseudo-static random access memory 12 (hereinafter referred to as PSRAM 12) is a memory independent of the main control module 11. Therefore, in this application, PSRAM 12 can also be referred to as off-chip memory, and the corresponding SRAM 112 can also be referred to as on-chip memory. PSRAM 12 may contain a media resource cache 121, which can be used to cache bitmap data loaded by the UI engine from the multimedia management module 10.

[0052] The display control module 13, also known as the LCD-TFT Display Controller (LTDC), is used to control and manage the operation of the display screen 14. Specifically, under the control of the main control module 11, the display control module 13 can send image data to the display screen 14 to achieve continuous screen refresh.

[0053] Specifically, during the entire process of rendering and displaying an image on the electronic device, the UI engine 111 first loads the bitmap data corresponding to the user interface from the multimedia management module 10, and then sends it to the media resource cache 121 in the PSRAM 12 for caching. Once the UI engine 111 receives the rendering command, it will again read the bitmap data from the media resource cache 121. This bitmap data corresponds to a complete frame of the user interface (i.e., a full screen of the image subsequently displayed on the display screen 14).

[0054] Next, the UI engine begins rendering the bitmap data. After rendering the bitmap data corresponding to a complete frame of the user interface, the UI engine stores the rendered image data in the rendering buffer 112a. Similarly, the image data stored in the rendering buffer 112a at this time also corresponds to a complete frame of the user interface.

[0055] Next, the image data in render buffer 112a will be copied to frame buffer 112b to await display. Similarly, the image data stored in frame buffer 112b at this time also corresponds to a complete user interface frame. Therefore, it can be understood that at this time, both render buffer 112a and frame buffer 112b in SRAM 112 store image data corresponding to a complete user interface frame.

[0056] Finally, the image data in frame buffer 112b is further transmitted to display control module 13. Display control module 13 sends a drive command to display screen 14, causing display screen 14 to refresh the display screen and output the user interface corresponding to the above image data to display screen 14.

[0057] As explained above, during the rendering and display of images in an electronic device, both the rendering buffer 112a and the frame buffer 112b in SRAM 112 store a complete frame of image. Therefore, the rendering buffer 112a and frame buffer 112b consume a significant amount of SRAM 112 memory resources. This is especially true for wearable devices, where the SRAM available for rendering and display is limited to approximately 2.5MB due to size constraints. However, the rendering buffer and frame buffer alone require about 1.24MB of memory during image rendering and display, almost half of the total available SRAM. In this situation, the system's available memory is significantly reduced, which can easily cause the electronic device to lag or even crash, severely impacting the user experience.

[0058] To address the problems existing in the aforementioned image processing systems, this application provides an image processing system and method for electronic devices. In this image processing system, the number of rendering buffers in SRAM is increased to two. For any frame of the user interface to be rendered, the two rendering buffers can work together. Each rendering buffer is only responsible for storing 1 / n of the user interface frame at a time. In addition, the frame buffer responsible for storing the image to be displayed is set in PSRAM. When the two rendering buffers work together to render the image, the electronic device also synchronously copies the image data in the rendering buffer to the frame buffer. When the image data corresponding to a complete frame of the user interface is completely copied to the frame buffer, the electronic device can issue a display command to output the image data in the frame buffer to the display screen. In this way, for any frame of the user interface, the rendering buffer only needs to store 1 / n (n is an integer greater than 1) of the image data corresponding to that frame of the user interface at a time, and the frame buffer is moved out to PSRAM. Therefore, during the image rendering and display process, the memory occupation of the rendering buffer and frame buffer in SRAM by the electronic device is significantly reduced, which can effectively save SRAM memory resources and improve the stability of system operation.

[0059] First, combine Figure 2 This application introduces an image processing system provided in its embodiments.

[0060] like Figure 2As shown, the image processing system includes a multimedia management module 20, a main control module 21, a pseudo-static random access memory 22 (hereinafter referred to as "PSRAM 22"), a display control module 23, and a display screen 24. It should be understood that the division of the various modules in the image processing system is merely a logical functional division; in actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. For example, each module can be a separate processing element, or it can be integrated into the same chip. Alternatively, it can be stored as program code in the controller's storage module, and its functions can be called and executed by a processing element of the processor. Furthermore, the modules can be integrated together or implemented independently.

[0061] The specific functions of the multimedia management module 20, the main control module 21, and the display control module 23 can be referred to the aforementioned descriptions. Figure 1 The relevant descriptions of the multimedia management module 10, the main control module 11, and the display control module 13 are not repeated here.

[0062] It should be noted that, in this embodiment of the application, the static random access memory 212 (hereinafter referred to as SRAM 212) included in the main control module 21 may contain two rendering buffers, namely Figure 2 The rendering buffers 212a and 212b are shown in the diagram. Furthermore, SRAM 212 may not contain a frame buffer. Rendering buffers 212a and 212b can work collaboratively and alternately, caching image data rendered for the same frame of the user interface. Each rendering buffer renders only 1 / n of a user interface frame at a time. While one rendering buffer caches (and clears any previously cached image data) the image data of the most recently rendered 1 / n frame of the user interface, the other rendering buffer can simultaneously copy the previously cached image data corresponding to the 1 / n frames of the user interface to the frame buffer in PSRAM 22. This alternation continues until a complete frame of the user interface is rendered. Understandably, when copying image data from the render buffer to the frame buffer, the PSRAM read rate is low. Therefore, by having the two render buffers work together, while copying image data from one render buffer to the frame buffer, the other part of the bitmap data that could not be rendered is rendered, and the newly obtained image data is synchronously stored in the other render buffer, the negative impact of the low PSRAM read rate can be avoided as much as possible, reducing the occupation of SRAM memory while ensuring the rendering efficiency of the image.

[0063] There can be two frame buffers in PSRAM 22, namely Figure 2The image shows frame buffers 222a and 222b. Frame buffers 222a and 222b are responsible for storing image data copied from rendering buffers 212a and 212b, and further outputting it to the display screen 24 for user browsing. Accordingly, in this embodiment, frame buffers 222a and 222b can also work together: after one of the frame buffers successfully obtains a complete frame of user interface image data from rendering buffers 212a and 212b, this complete frame of user interface image data can be output to the display screen 24 for user browsing under the control of the display control module 23; at the same time, the UI engine will render the next frame of user interface n times, and the image data obtained from each rendering will also be alternately stored in rendering buffers 212a and 212b according to the aforementioned method. That is to say, while the electronic device is sending a frame of user interface cached in one of the frame buffers to the display, the electronic device is also simultaneously rendering the next frame of user interface and copying the image data corresponding to the frame of user interface from rendering buffers 212a and 212b to the other rendering buffer. This minimizes the impact of the slow PSRAM 22 read / write speed on the screen frame rate, reducing the usage of SRAM memory while ensuring smooth screen display.

[0064] Similarly, during the entire process of rendering and displaying an image on the electronic device, the UI engine 211 first loads the bitmap data corresponding to the user interface from the multimedia management module 20, and then sends it to the media resource cache 221 in the PSRAM 22 for caching. Only after the UI engine 211 receives the rendering command will it read the bitmap data again from the media resource cache 221. This bitmap data corresponds to a complete frame of the user interface (i.e., a full screen of the image subsequently displayed on the display screen 24).

[0065] Then, UI engine 211 begins rendering the bitmap data in n steps. Optionally, the specific value of n can be a value set at the factory when the electronic device is manufactured. Optionally, the value of n can be any integer between 2 and 10.

[0066] Taking an n=2 frame as an example, the UI engine 211 can render the bitmap data for any given frame of the user interface twice. Assuming that the user interface UI1 consists of two parts, bitmap1 and bitmap2, when rendering the bitmap data for UI1, the UI engine 211 can first render bitmap1 and store the resulting image data pic1 (which corresponds to only half a frame of the user interface UI1) in the rendering buffer 212a. Then, the main control module 21 will further copy the image data pic1 to the frame buffer 222a. Simultaneously, while copying pic1 to the frame buffer 222a, the UI engine will synchronously render bitmap2 and store the resulting image data pic2 (which also corresponds to half a frame of the user interface UI1) in the rendering buffer 212b. Similarly, the main control module 21 will also further copy the image data pic1 to the frame buffer 222a. Understandably, the frame buffer 222a already contains the corresponding complete image data at this point.

[0067] When the user interface UI1 is displayed, the image data (i.e., image data pic1 and image data pic2) corresponding to the user interface UI1 stored in the frame buffer 222a are further transmitted to the display control module 23. The display control module 23 sends a drive command to the display screen 24, causing the display screen 24 to refresh the display screen and output the user interface corresponding to the above image data to the display screen 24.

[0068] If there is another user interface UI2 to be displayed after displaying user interface UI1, then while outputting the image data corresponding to user interface UI1 in frame buffer 222a to display screen 24, UI engine 211 can simultaneously render the bitmap data of user interface UI2 in the aforementioned manner, and the resulting image data will also be transferred to the frame buffer in PSRAM 22 via the rendering buffer in SRAM 212 in the aforementioned manner.

[0069] It should be noted that, since both rendering buffers 212a and 212b already store some image data corresponding to the user interface UI1, after rendering the bitmap data of the user interface UI2 and obtaining the corresponding image data, the electronic device will first delete the existing image data in rendering buffers 212a or 212b before storing the image data in rendering buffers 212a or 212b, and then cache the image data corresponding to the user interface UI1 in rendering buffers 212a or 212b, in order to reduce the memory usage of SRAM 212. Similarly, since the frame buffer already caches the image data corresponding to the user interface UI1, and this part of the image data is currently being sent for display, the image data corresponding to the user interface UI2 cached in rendering buffers 212a and 212b can be copied to frame buffer 222b. After the user interface UI1 is displayed, the image data corresponding to the user interface UI2 stored in frame buffer 222b will be output to the display screen 24 under the control of the display control module 23, and the display screen 24 can then display the user interface UI2.

[0070] Understandably, PSRAM 22 has a slower read speed compared to SRAM, while... Figure 2 In the image processing system shown, because there are two frame buffers, the electronic device can render the next frame of the user interface while simultaneously sending the already rendered frame to the display, and store the rendered image data in the other frame buffer for later display. This minimizes the impact of the slow PSRAM 22 read / write speed on the screen frame rate, reducing SRAM memory usage while ensuring smooth screen display.

[0071] Understandably, during the rendering of the same frame of the user interface, at any given time, only one of the rendering buffers 212a and 212b contains the image data corresponding to 1 / n frames of the user interface. Furthermore, since the frame buffers have been moved to the PSRAM 22 outside the main control module 21, during image rendering and display, the SRAM 212 may only contain the image data corresponding to 1 / n frames of the user interface. Figure 1 The image processing system shown has at least two complete sets of image data corresponding to the same user interface cached in SRAM 112 (including all image data corresponding to one frame of the user interface stored in rendering buffer 112a and all image data corresponding to one frame of the user interface stored in frame buffer 112b). Therefore, the image processing system provided in this application can effectively reduce the memory usage of SRAM during image rendering, thereby improving the overall performance of the electronic device system.

[0072] As can be seen from the above description, in this application, the larger the value of n, the lower the memory usage of SRAM during image rendering; however, the larger the value of n, the more times the UI engine 211 renders the same frame of the user interface, and the greater the computing power requirement of the main control module 21. Therefore, in this embodiment, the specific value of n can be determined based on the electronic device's requirements for both memory and computing power. Optionally, the specific value of n can be any integer from 2 to 10.

[0073] Furthermore, in one possible implementation, PSRAM 22 may contain only one frame buffer. In this case, during the rendering phase, render buffers 212a and 212b can still work together alternately, that is, alternately cache the image data rendered for the same frame of the user interface, with each render buffer rendering only 1 / n of one frame of the user interface at a time. However, during the display phase, after the electronic device successfully renders a frame of the user interface and caches the corresponding image data in the frame buffer, the electronic device can clear the data in the frame buffer after displaying this frame of the user interface, and then start rendering the bitmap data corresponding to the next frame of the user interface, caching the rendered image data into the aforementioned frame buffer.

[0074] In combination with the above Figure 2 Following the description of the image processing system, this application introduces an image processing method. Specifically, this image processing method can be applied to systems deployed with... Figure 2 The image processing system shown is an electronic device that can be a wearable device, mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, and vehicle-mounted device. This application embodiment does not impose any special limitations on the specific type of this electronic device. Figure 3 As shown, Figure 3 The hardware terminal module can be included in Figure 2 The main control module 21 shown is included, while the display module and rendering module can be contained within it. Figure 2 In the UI engine 211 shown, the image processing method may include, but is not limited to, the following steps:

[0075] S101: The hardware interrupt module receives the first TE signal.

[0076] The aforementioned first TE signal can be Figure 2 The main control module 21 generates and distributes the interrupt to the hardware interrupt module.

[0077] After receiving the first TE signal, the hardware interrupt module can determine whether an image rendering / image sending / copying operation is currently occurring in the system based on the interrupt trigger function TECallBack. If no image rendering / image sending / copying operation is currently occurring in the system, the hardware interrupt module will execute the subsequent step S102, that is, send a rendering command to the rendering module.

[0078] S102: The hardware interrupt module sends the first rendering instruction to the rendering module.

[0079] S103: The rendering module renders the first part of the first image and stores the rendered first image data in the first rendering buffer.

[0080] After the hardware interrupt module sends the first rendering command to the rendering module, the rendering module can respond to the command and begin drawing the first user interface. The aforementioned first bitmap is the bitmap data corresponding to the first user interface. As can be seen from the foregoing description, the first bitmap can be cached in the media resource buffer of PSRAM. After the rendering module receives the aforementioned first rendering command, the rendering module then retrieves the aforementioned first bitmap from the aforementioned media resource buffer.

[0081] Specifically, during the rendering of the first image, the rendering module can draw the entire screen page n times, starting from the top area of ​​the first user interface, that is, draw 1 / n of the first image each time. Understandably, the first part of the first image corresponds to the top 1 / n of the first user interface, and the first image data is the top 1 / n of the first user interface. Specific rendering operations may include, but are not limited to, merging invalid areas (i.e., areas that need to be redrawn) and calculating the position of the rendering area.

[0082] After obtaining the first image data, the rendering module can cache the first image data in the first rendering buffer. It is understood that, since there are two rendering buffers in the SRAM in this embodiment, the main control module can determine the first rendering buffer as the rendering buffer to be used before executing step S103. After a successful switch, if the first rendering buffer already stores other image data (e.g., image data cached in the first rendering buffer when the rendering module rendered the previous frame of the user interface), the image data already stored in the first rendering buffer will be cleared before caching the first image data into the first rendering buffer. Only after the existing image data in the first rendering buffer is cleared will the rendering module cache the first image data into the first rendering buffer.

[0083] S104: The rendering module sends the first image data to the display driver module.

[0084] S105: The display driver module copies the first image data to the first frame buffer.

[0085] After rendering the first part of the first image, the rendering module can send the rendered first image data to the display driver module, which then caches the first image data in the first frame buffer. Specifically, the display driver module may include a graphics accelerator DMA2D, which can copy the first image data to the first frame buffer.

[0086] In combination with the above Figure 2 As can be seen from the relevant description, while caching the first image data into the first frame buffer, the rendering module continues to render the next 1 / n unrendered bitmap data in the first image. That is to say, while executing steps S105-S106, the electronic device is also synchronously executing subsequent steps S106-S107.

[0087] S106: The rendering module switches the rendering buffer to the second rendering buffer.

[0088] Understandably, since the first rendering buffer area already stores the first image data, and the first image data is being copied to the first frame buffer, the rendering module can switch the rendering buffer to be used from the first rendering buffer to the second rendering buffer to cache the image data obtained in the subsequent rendering process.

[0089] Similarly, after switching the rendering buffer to the second rendering buffer, if the second rendering buffer already contains other image data (e.g., image data cached in the second rendering buffer when the rendering module rendered the previous frame of the user interface), the rendering module can clear the image data already stored in the first rendering buffer. Only after the existing image data in the first rendering buffer is cleared will the rendering module begin rendering the subsequent parts of the first image.

[0090] S107: The rendering module renders the second part of the first image and stores the rendered second image data in the second rendering buffer.

[0091] Understandably, the second part of the first image corresponds to the other 1 / n consecutively connected to the top 1 / n in the first user interface, and the second image data is the other 1 / n image connected to the top 1 / n in the first user interface. The rendering module will render the second part of the first image, and the rendered second image data will be cached in the second rendering buffer.

[0092] S108: The rendering module sends the second image data to the display driver module.

[0093] S109: The display driver module copies the second image data to the first frame buffer.

[0094] After the second part of the first image above is rendered, the rendering module can send the rendered first image data to the display driver module, and the display driver module can cache the first image data in the first frame buffer.

[0095] It should be noted that in this application, after the display module completes drawing 1 / n of the first image, it copies the rendered image data from the rendering buffer that caches the image data to the corresponding frame buffer, and simultaneously switches to the next rendering buffer to draw the next area, until the first image is completely rendered. That is to say, when the rendering module successfully renders the last 1 / n of the first image, while copying the image data corresponding to the last 1 / n of the first image stored in the rendering buffer to the first frame buffer, because the first image has been completely rendered, the rendering module can stop concurrent rendering operations until the central control module receives the next TE signal and sends the next rendering instruction to the rendering module (which can be understood as the electronic device starting to prepare for the rendering and display of the next frame of the user interface). Only then will the rendering module continue to render the next frame of the user interface according to the aforementioned rendering method.

[0096] For example, when n is 2, it means that the second part of the first bitmap is the last half of the first bitmap. Therefore, during the process of the rendering module caching the second image data cached in the second rendering buffer to the first frame buffer via the display driver module, the rendering module does not concurrently render other bitmap data. Optionally, after copying the second image data to the first frame buffer, the rendering module can switch the rendering buffer from the second rendering buffer to the first rendering buffer and clear the first image data stored in the first rendering buffer so that the first rendering buffer can cache subsequent image data (e.g., image data obtained after the rendering module renders the next frame of the user interface).

[0097] Correspondingly, when the value of n is greater than 2, it means that the second part of the first image is not the data of the last 1 / n part of the first image. In the process of the rendering module caching the second image data cached in the second rendering buffer to the first frame buffer through the display driver module, the rendering module will continue to concurrently render the data of the subsequent 1 / n part of the first image that has not yet been rendered, and so on. Until the rendering module finishes rendering the data of the last 1 / n part of the first image and caches the image data corresponding to the data of the last 1 / n part to the first frame buffer, the main control module will continue to send the TE signal. That is, when n is greater than 2, after executing step S109, the electronic device still needs to continue to render the part of the first image that has not yet been rendered in the first image in the aforementioned manner until the first image corresponding to the first user interface is rendered. Only then will the electronic device execute the subsequent step S110 and start the rendering and display operation of the next frame of the user interface.

[0098] S110: The hardware interrupt module receives the second TE signal.

[0099] Similarly, the second TE signal mentioned above can be Figure 2 The main control module 21 generates and sends the signal to the hardware interrupt module. The second TE signal is used to prompt the system to start rendering the next frame of the user interface.

[0100] S111: The hardware interrupt module switches the frame buffer to be used as the second frame buffer.

[0101] As explained above, to ensure the display frame rate of the electronic device, while the display driver module outputs the first user interface rendered in the first frame buffer to the display screen, the rendering module simultaneously renders the next frame user interface (i.e., the second user interface) and caches the rendered image data in the frame buffer in the same manner as described above, awaiting display. Therefore, before officially rendering the bitmap data corresponding to the second interface, the hardware interrupt module switches the frame buffer to be used from the first frame buffer to the aforementioned second frame buffer, so that the image data corresponding to the second user interface can be cached subsequently. Similarly, after switching the frame buffer to be used to the aforementioned second frame buffer, if the second frame buffer already stores image data (e.g., image data corresponding to another user interface displayed before the first user interface), the hardware interrupt module can clear the stored image data to store the image data corresponding to the second user interface later.

[0102] S112: The hardware interrupt module sends the second rendering instruction to the rendering module.

[0103] S113: The hardware interrupt module sends the first display command to the display driver module.

[0104] Upon receiving the second TE signal, if there are currently no rendering / image sending / copying operations in the system, the hardware interrupt module can send a second rendering instruction to the rendering module and simultaneously send a first display sending instruction to the display driver module, so that the electronic device can execute subsequent steps S111-S121. Specifically, in the simultaneously sent second rendering instruction and first display sending instruction, the rendering instruction is used to start the rendering module to render the next frame of the user interface (i.e., the second user interface), while the first display sending instruction is used to start the display driver module to output the first interface, which has been rendered and stored in the first frame buffer, to the display screen.

[0105] S114: The display driver module outputs the first user interface to the display screen.

[0106] The display driver module outputs the first user interface image data (including the aforementioned first image data and second image data) stored in the first frame buffer to the display screen via LTDC, so that the first user interface is displayed on the display screen for the user to browse. During this process, the rendering module simultaneously renders the bitmap corresponding to the second user interface and caches the rendered image data in the aforementioned second frame buffer with the help of the display driver module. That is to say, when the electronic device executes step S114, the electronic device is also simultaneously executing subsequent steps S115-S121.

[0107] S115: The rendering module renders the first part of the second bitmap and stores the rendered third image data in the first rendering buffer.

[0108] S116: The rendering module sends the third image data to the display driver module.

[0109] S117: The display driver module copies the third image data to the second frame buffer.

[0110] S118: The rendering module switches the rendering buffer to the second rendering buffer.

[0111] S119: The rendering module renders the second part of the second bitmap and stores the rendered fourth image data in the second rendering buffer.

[0112] S120: The rendering module sends the fourth image data to the display driver module.

[0113] S121: The display driver module copies the fourth image data to the second frame buffer.

[0114] The second bitmap mentioned above refers to the bitmap data corresponding to the second user interface. Similarly, the second bitmap can be cached in the media resource cache of PSRAM. After the rendering module receives the second rendering instruction, the rendering module retrieves the second bitmap from the media resource cache.

[0115] Similarly, when rendering the second bitmap, the display module copies the rendered image data from the rendering buffer that caches the image data to the second frame buffer after each 1 / n of the second bitmap is drawn. Simultaneously, it switches to the next rendering buffer to draw the next area, until the second bitmap is completely rendered. The second buffer then caches the completed image data corresponding to the second user interface. For details, please refer to the explanations of steps S103-S109 above; they will not be repeated here. After the rendering module copies the image data corresponding to the last 1 / n of the second bitmap stored in the rendering buffer to the first frame buffer, since the second bitmap has been fully rendered, the rendering module can stop concurrent rendering operations until the central control module receives the next TE signal. Only then will the electronic device continue rendering the next frame of the user interface, and so on.

[0116] Figure 4 and Figure 5 The diagram illustrates the changes in data in SRAM and PSRAM when the electronic device renders and displays the image, provided that the value of n is 2.

[0117] First, combine Figure 4 Describe the rendering process for electronic devices. For example... Figure 4 As shown, the user interface 401 is the user interface that the electronic device needs to render and display in the next frame. Its corresponding bitmap data can contain two parts, namely bitmap data 401a and bitmap data 401b. Figure 4 The PSRAM and SRAM shown can be PSRAM 22 and SRAM 212 as described above, where rb1 and rb2 are the two rendering buffers contained in the SRAM, and fb1 and fb2 are the two frame buffers contained in the PSRAM.

[0118] As explained above, when n is 2, for any given frame of the user interface, the electronic device will render the user interface twice, sequentially from top to bottom, rendering half of the user interface each time. After receiving the TE signal TE1 used to initiate the rendering of the user interface 401, the electronic device can proceed according to... Figure 4 The three stages shown render the bitmap data corresponding to the user interface 401.

[0119] like Figure 4As shown in (A), in the first stage, the electronic device first renders the bitmap data 401a, and the rendered image data 401A is first cached in the rendering buffer rb1 in SRAM. Figure 4 As can be seen in (A), at this time, only 1 / 2 frame of image data is stored in the SRAM, that is, image data 401A in the rendering buffer rb1.

[0120] After that, as Figure 4 As shown in (B), in the second stage, the electronic device renders the bitmap data 401b, and the rendered image data 401B is cached in rb2 of the rendering buffer in SRAM. Simultaneously, the electronic device copies the image data 401A stored in the rendering buffer rb1 to the frame buffer fb1 in PSRAM. As explained above, after rendering the bitmap data 401b, the electronic device switches the working rendering buffer back to the rendering buffer rb1 and simultaneously clears the image data 401A already stored in the rendering buffer rb1. Therefore, in the second stage, only half a frame of image data, i.e., the image data 401B in the rendering buffer rb2, is stored in the SRAM.

[0121] After that, as Figure 4 As shown in (C), in the third stage, as explained above, since the electronic device has already completed rendering the user interface 401 in this stage, it only copies the image data 401B stored in the rendering buffer rb2 to the frame buffer fb1 in the PSRAM. At this time, the frame buffer fb1 has already cached the complete image data corresponding to the user interface 401. Furthermore, in the third stage, only half a frame of image data, namely the image data 401B in the rendering buffer rb2, is stored in the SRAM.

[0122] As can be seen from the foregoing explanation, during the entire rendering process of the user interface 401, the electronic device's SRAM only needs to cache a maximum of 1 / 2 frame of image data, compared to the aforementioned... Figure 1 The rendering method used in the image processing system shown in this application embodiment can significantly reduce the SRAM memory usage of the electronic device, to only a fraction of the required amount. Figure 1 It accounts for 1 / 4 of the SRAM memory usage.

[0123] exist Figure 4 On this basis, Figure 5 The diagram illustrates the specific process by which an electronic device sends the rendered user interface 401 to the display and simultaneously renders the next frame of user interface 501.

[0124] like Figure 5As shown, user interface 501 is the next frame of user interface after user interface 401, and its corresponding bitmap data can also contain two parts, namely bitmap data 501a and bitmap data 501b.

[0125] Upon receiving the TE signal TE2, used to initiate the rendering of the user interface 501, the electronic device can first switch the frame buffer to be used from frame buffer fb1 to frame buffer fb2; if other image data is already buffered in frame buffer fb2, that image data will be cleared first. Afterwards, the electronic device will proceed according to... Figure 5 The three stages shown render the bitmap data corresponding to the user interface 501, and simultaneously output the user interface 401 to the display screen of the electronic device during the rendering process. It should be noted beforehand that the moment the electronic device begins rendering the user interface 501 and the moment the electronic device begins displaying the user interface 401 can be the same, but the total duration of rendering the user interface 501 and the total duration of displaying the user interface 401 can be the same or different. For ease of explanation, Figure 5 The following example illustrates the point by taking the total time for the electronic device to render the user interface 501 as the same as the total time for the electronic device to display the user interface 401.

[0126] like Figure 5 As shown in (A), in the first stage, the electronic device first renders the bitmap data 501a. The rendered image data 501A is initially cached in the rendering buffer rb1 in SRAM. Simultaneously, the electronic device begins processing the image data (i.e., image data 401A and image data 401B corresponding to the user interface 401) in the frame buffer fb1. After rendering the bitmap data 501a is complete, the electronic device switches the rendering buffer to be used to rendering buffer rb2 and simultaneously clears the image data 402B currently stored in rendering buffer rb2. Figure 5 As can be seen in (A), at this time, only 1 / 2 frame of image data is stored in the SRAM, that is, image data 501A in the rendering buffer rb1.

[0127] After that, as Figure 5As shown in (B), in the second stage, the electronic device renders the bitmap data 401b, and the rendered image data 401B is cached in the rendering buffer rb2 in SRAM. Simultaneously, the electronic device copies the image data 501A stored in the rendering buffer rb1 to the frame buffer fb2 in PSRAM. As explained above, after rendering the bitmap data 501b, the electronic device switches the working rendering buffer back to the rendering buffer rb1 and simultaneously clears the image data 501A already stored in the rendering buffer rb1. Therefore, in the second stage, only half a frame of image data, i.e., the image data 501B in the rendering buffer rb2, is stored in SRAM. Understandably, in this stage, the electronic device is still in the process of sending the image data from the frame buffer fb1 to the display.

[0128] After that, as Figure 5 As shown in (C), in the third stage, as explained above, since the electronic device has already completed rendering the user interface 501 in this stage, it only copies the image data 501B stored in the rendering buffer rb2 to the frame buffer fb2 in the PSRAM. At this time, the frame buffer fb2 has already cached the complete image data corresponding to the user interface 501. Furthermore, only half a frame of image data, i.e., the image data 501B in the rendering buffer rb2, is stored in the SRAM in the third stage. In addition, in this stage, the electronic device has completed sending the image data in the frame buffer fb1 to the display, and the user interface 401 is successfully displayed on the screen of the electronic device.

[0129] As can be seen from the foregoing description, during the entire process of rendering the user interface 501 and displaying the user interface 401, the SRAM of the electronic device still only needs to cache a maximum of 1 / 2 frame of image data, and the SRAM memory usage of the electronic device can be greatly reduced.

[0130] Similarly, if the electronic device receives other TE signals subsequently, the electronic device can still proceed as planned. Figure 5 The process shown continues to render the next frame of the user interface and send the user interface 501 to the display, and so on. It will not be described in detail here.

[0131] In this embodiment, rendering buffer rb1 can be referred to as the "first rendering buffer," rb2 can be referred to as the "second rendering buffer," frame buffer fb1 can be referred to as the "first frame buffer," and frame buffer fb2 can also be referred to as the "second frame buffer." Frame buffer fb1 and frame buffer fb2 can be collectively referred to as "at least one frame buffer." During the rendering and display process of user interface 401, bitmap data 401a can be referred to as "first bitmap data," image data 401A can be referred to as "first image data," bitmap data 401b can be referred to as "second bitmap data," image data 401B can be referred to as "second image data," user interface 401 can be referred to as "first screen," user interface 501 can be referred to as "second screen," and image data 501A and image data 501B can be collectively referred to as "third image data."

[0132] Figure 6 and Figure 7 The diagram illustrates the changes in data in SRAM and PSRAM when the electronic device renders and displays the image, with the aforementioned value of n being 3.

[0133] First, combine Figure 6 Describe the rendering process for electronic devices. For example... Figure 6 As shown, the user interface 601 is the user interface that the electronic device needs to render and display in the next frame. Its corresponding bitmap data can contain three parts, namely bitmap data 601a, bitmap data 601b and bitmap data 601c. Figure 6 The PSRAM and SRAM shown can be PSRAM 22 and SRAM 212 as described above, where rb3 and rb4 are the two rendering buffers contained in the SRAM, and fb3 and fb4 are the two frame buffers contained in the PSRAM.

[0134] As explained above, when n is 3, for any given frame of the user interface, the electronic device will render the user interface three times sequentially from top to bottom, rendering 1 / 3 of the user interface each time. After receiving the TE signal TE3 used to initiate the rendering of the user interface 601, the electronic device can proceed according to... Figure 6 The three stages shown render the bitmap data corresponding to the user interface 601.

[0135] like Figure 6 As shown in (A), in the first stage, the electronic device first renders the bitmap data 601a, and the rendered image data 601A is first cached in the rendering buffer rb3 in SRAM. Figure 6As can be seen in (A), at this time only 1 / 3 frame of image data is stored in the SRAM, that is, image data 601A in the rendering buffer rb3.

[0136] After that, as Figure 6 As shown in (B), in the second stage, the electronic device renders the bitmap data 601b, and the rendered image data 601B is cached in rb4 of the rendering buffer in SRAM. Simultaneously, the electronic device copies the image data 601A stored in the rendering buffer rb3 to the frame buffer fb3 in PSRAM. As explained above, after rendering the bitmap data 601b, the electronic device switches the working rendering buffer back to rendering buffer rb3 and simultaneously clears the image data 601A already stored in rendering buffer rb3. Therefore, in the second stage, only 1 / 3 of a frame of image data, i.e., the image data 601B in rendering buffer rb4, is stored in SRAM.

[0137] After that, as Figure 6 As shown in (C), in the third stage, the electronic device renders the bitmap data 601c, and the rendered image data 601C is cached in the rendering buffer rb3 in SRAM. Simultaneously, the electronic device copies the image data 601B stored in the rendering buffer rb4 to the frame buffer fb3 in PSRAM. Similarly, after rendering the bitmap data 601c, the electronic device switches the working rendering buffer back to the rendering buffer rb4 and simultaneously clears the image data 601B already stored in the rendering buffer rb4. Therefore, in the third stage, only 1 / 3 of the frame's image data, i.e., the image data 601C in the rendering buffer rb3, is stored in the SRAM.

[0138] Next, as Figure 6 As shown in (D), in the fourth stage, as explained above, since the electronic device has already completed rendering the user interface 601 in this stage, it only copies the image data 601C stored in the rendering buffer rb3 to the frame buffer fb3 in the PSRAM. At this time, the frame buffer fb3 has already cached the complete image data corresponding to the user interface 601. Furthermore, in the fourth stage, only 1 / 3 of the frame's image data, namely the image data 601C in the rendering buffer rb3, is stored in the SRAM.

[0139] As can be seen from the foregoing explanation, during the entire rendering process of the user interface 601, the electronic device's SRAM only needs to cache a maximum of 1 / 3 frame of image data, compared to the aforementioned... Figure 1The rendering method used in the image processing system shown in this application embodiment can significantly reduce the SRAM memory usage of the electronic device, to only a fraction of the required amount. Figure 1 It accounts for 1 / 6 of the SRAM memory usage.

[0140] exist Figure 6 On this basis, Figure 7 The diagram illustrates the specific process by which an electronic device sends the rendered user interface 601 to the display and simultaneously renders the next frame of user interface 701.

[0141] like Figure 7 As shown, user interface 701 is the next frame of user interface after user interface 401, and its corresponding bitmap data can also contain three parts, namely bitmap data 701a, bitmap data 701b and bitmap data 701c.

[0142] Upon receiving the TE signal TE4, used to initiate the rendering of the user interface 701, the electronic device can first switch the frame buffer to be used from frame buffer fb3 to frame buffer fb4; if other image data is already buffered in frame buffer fb4, then that image data is cleared first. Afterwards, the electronic device will proceed according to... Figure 7 The four stages shown render the bitmap data corresponding to user interface 701, and simultaneously output user interface 401 to the display screen of the electronic device during the rendering process. Similarly, the moment when the electronic device starts rendering user interface 701 and the moment when the electronic device starts displaying user interface 601 can be the same, but the total time for the electronic device to render user interface 701 and the total time for the electronic device to display user interface 601 can be the same or different. For ease of explanation, Figure 7 The following example illustrates the process by taking the total time for the electronic device to render the user interface 701 as the same as the total time for the electronic device to display the user interface 601.

[0143] like Figure 7 As shown in (A), in the first stage, the electronic device first renders the bitmap data 701a. The rendered image data 701A is initially cached in the rendering buffer rb4 in SRAM. Simultaneously, the electronic device begins processing the image data (i.e., image data 601A, image data 601B, and image data 601C corresponding to the user interface 601) in the frame buffer fb3. After rendering the bitmap data 701a is complete, the electronic device switches the rendering buffer to rb3 and simultaneously clears the image data 602C currently stored in the rendering buffer rb3. Figure 7 As can be seen in (A), at this time only 1 / 3 frame of image data is stored in the SRAM, that is, image data 701A in rb4 of the rendering buffer.

[0144] After that, as Figure 7 As shown in (B), in the second stage, the electronic device renders the bitmap data 701b, and the rendered image data 701B is cached in rb3 of the rendering buffer in SRAM. Simultaneously, the electronic device copies the image data 701A stored in the rendering buffer rb4 to the frame buffer fb4 in PSRAM. As explained above, after rendering the bitmap data 701b, the electronic device switches the rendering buffer back to rb4 and simultaneously clears the image data 701A already stored in rb4. Therefore, in the second stage, only 1 / 3 of the frame's image data is stored in SRAM, namely the image data 701B in rb3 of the rendering buffer. Understandably, in this stage, the electronic device is still in the process of sending the image data from frame buffer fb3 to the display.

[0145] After that, as Figure 7 As shown in (C), in the third stage, the electronic device renders the bitmap data 701c, and the rendered image data 701C is cached in rb4 of the rendering buffer in SRAM. Simultaneously, the electronic device copies the image data 701B stored in the rendering buffer rb3 to the frame buffer fb4 in PSRAM. As explained above, after rendering the bitmap data 701c, the electronic device switches the rendering buffer back to rendering buffer rb3 and simultaneously clears the image data 701B already stored in rendering buffer rb3. Therefore, in the second stage, only 1 / 3 of the frame's image data is stored in SRAM, namely the image data 701C in rendering buffer rb4. Understandably, in this stage, the electronic device is still in the process of sending the image data from frame buffer fb3 to the display.

[0146] After that, as Figure 7 As shown in (D), in the fourth stage, as explained above, since the electronic device has already completed rendering the user interface 701 in this stage, it only copies the image data 701C stored in the rendering buffer rb4 to the frame buffer fb4 in the PSRAM. At this time, the frame buffer fb4 has already cached the complete image data corresponding to the user interface 701. Furthermore, in the third stage, only 1 / 3 of the frame's image data, namely the image data 701C in the rendering buffer rb4, was stored in the SRAM. In addition, in this stage, the electronic device has completed sending the image data in the frame buffer fb3 to the display, and the user interface 601 is successfully displayed on the electronic device's screen.

[0147] As can be seen from the foregoing description, during the entire process of rendering the user interface 701 and displaying the user interface 601, the SRAM of the electronic device still only needs to cache a maximum of 1 / 3 frame of image data, and the SRAM memory usage of the electronic device can be greatly reduced.

[0148] Similarly, if the electronic device receives other TE signals subsequently, the electronic device can still proceed as planned. Figure 7 The process shown continues to render the next frame of the user interface and send the user interface 701 to the display, and so on. It will not be described in detail here.

[0149] In this application, the electronic device renders only 1 / n of a frame at a time and caches the rendered image data in a rendering buffer. Combined with... Figures 4-7 It is known that the larger the value of n, the less memory is needed in the SRAM during image rendering. For example, when n is 4, the SRAM only needs to cache a maximum of 1 / 4 frame of image data during the rendering and display process. However, the larger the value of n, the more times the electronic device renders the same frame of the user interface, and the greater the computing power consumed by the electronic device. Therefore, in this embodiment, the specific value of n can be determined based on the electronic device's requirements for both memory and computing power. Optionally, the specific value of n can be any integer from 2 to 10.

[0150] In this embodiment, the rendering buffer rb3 can be referred to as the "first rendering buffer," rb4 can be referred to as the "second rendering buffer," the frame buffer fb3 can be referred to as the "first frame buffer," and the frame buffer fb4 can also be referred to as the "second frame buffer." Frame buffers fb3 and fb4 can be collectively referred to as "at least one frame buffer." During the rendering and display process of the user interface 601, the user interface 601 can be referred to as the "first screen," the user interface 701 can be referred to as the "second screen," and the image data 701A to image data 701C can be collectively referred to as "third image data."

[0151] The following is a schematic diagram of the structure of the electronic device 100 provided in the embodiments of this application.

[0152] like Figure 8As shown, the electronic device 100 can be a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), or dedicated camera (e.g., SLR camera, point-and-shoot camera), etc. The electronic device 100 can also be the second electronic device described above. This application does not limit the specific type of the electronic device.

[0153] Electronic device 100 may include processor 110, memory 122, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, camera 193, display screen 194, etc.

[0154] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0155] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0156] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0157] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0158] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0159] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0160] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, memory 122, external memory, display 194, camera 193, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0161] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0162] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0163] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0164] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image or video signal. The ISP outputs the digital image or video signal to a DSP for processing. The DSP converts the digital image or video signal into a standard RGB, YUV, or other format image or video signal. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0165] The memory 122 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 122. The memory 122 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image or video playback, etc.). The data storage area may store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.). Furthermore, the memory 122 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0166] In this application, the processor 110 further includes a static random access memory (SRAM) 1101, and the SRAM 1101 may contain two rendering buffers, namely rendering buffer 1101a and rendering buffer 1101b. During the image rendering process of the electronic device 100, for any given frame, the electronic device can render and draw that frame n times. In this process, rendering buffer 1101a and rendering buffer 1101b can work alternately, with each rendering buffer storing only 1 / n of the image in each frame, thereby reducing the image data's occupation of SRAM memory during rendering. Specifically, rendering buffer 1101a and rendering buffer 1101b can be respectively the aforementioned... Figure 2 The rendering buffers 212a and 212b in the above-mentioned... Figure 3 The first rendering buffer and the second rendering buffer in the middle.

[0167] Furthermore, in this application, memory 122 includes a pseudo-static random access memory (PSRAM) 1211 independent of processor 110. PSRAM 1211 may contain two frame buffers, namely frame buffer 1211a and frame buffer 1211b. When displaying images, for any two frames of images that need to be displayed consecutively, after the previous frame image is rendered and copied entirely to one of the two frame buffers, the electronic device can display that frame image, while simultaneously rendering the next frame image and copying it to the other frame buffer, and so on alternately, in order to minimize the frame interval when displaying two frames of images.

[0168] The electronic device 100 has one or more processors 110 and a memory 122; the memory 122 is coupled to the one or more processors 110, and the memory 122 is used to store computer program code, which includes computer instructions. The one or more processors call the computer instructions to cause the electronic device 100 to execute the image processing method provided in the embodiments of this application.

[0169] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0170] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0171] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An image processing method, characterized by, The method is applied to an image processing system, the image processing system comprising a static random access memory (SRAM) and a pseudo static random access memory (PSRAM), the SRAM comprising a first rendering buffer and a second rendering buffer, the PSRAM comprising at least one frame buffer, the method comprising: rendering first bitmap data and caching resultant first image data in the first rendering buffer; in the process of copying the first image data from the first rendering buffer to the at least one frame buffer, rendering second bitmap data and caching resultant second image data in the second rendering buffer, the first image data and the second image data both corresponding to 1 / n of a first picture, the n being an integer greater than 1.

2. The method of claim 1, wherein, The at least one frame buffer comprises a first frame buffer and a second frame buffer, the first image data and the second image data both being cached in the first frame buffer, the method further comprising: in the process of outputting complete image data corresponding to the first picture in the first frame buffer to a display screen, rendering a second picture and caching resultant third image data in the second frame buffer, the second picture being a picture displayed after the first picture.

3. The method according to claim 1 or 2, characterized in that, Before the rendering of the second bitmap data and the caching of the resultant second image data in the second rendering buffer, the method further comprising: switching the rendering buffer to be worked on from the first rendering buffer to the second rendering buffer; in the case that the second rendering buffer caches fourth image data, clearing the fourth image data in the second rendering buffer, the fourth image data corresponding to 1 / n of a third picture, the third picture being a picture displayed before the first picture.

4. The method of claim 2, wherein, Before the caching of the resultant third image data in the second frame buffer, the method further comprising: switching the frame buffer to be worked on from the first frame buffer to the second frame buffer; in the case that the second frame buffer caches fifth image data, clearing the fifth image data in the second frame buffer, the fifth image data being complete image data corresponding to a third picture, the third picture being a picture displayed before the first picture.

5. The method according to any one of claims 1 to 4, characterized in that, The image processing system further comprises a master module, the SRAM being comprised in the master module, and the PSRAM being located outside the master module.

6. An image processing system, characterized by The system comprises a rendering module, a display driving module, a static random access memory (SRAM) and a pseudo static random access memory (PSRAM), the SRAM comprising a first rendering buffer and a second rendering buffer, the PSRAM comprising at least one frame buffer, wherein: the rendering module is configured to render first bitmap data and cache resultant first image data in the first rendering buffer; The rendering module is further configured to render second bitmap data and cache resulting second image data corresponding to 1 / n of the first picture into the second rendering buffer in the process of copying the first image data from the first rendering buffer to the at least one frame buffer.

7. The system of claim 6, wherein, The at least one frame buffer comprises a first frame buffer and a second frame buffer, and the first image data and the second image data are both cached in the first frame buffer, The rendering module is further configured to render a second picture and cache resulting third image data corresponding to a complete picture of the second picture into the second frame buffer in the process of outputting the complete picture of the first picture corresponding to the first picture in the first frame buffer to a display screen, the second picture being a picture displayed after the first picture.

8. An electronic device, comprising: The electronic device comprises one or more processors, a memory and a display screen; The memory is coupled to the one or more processors, and the memory is configured to store computer program code comprising computer instructions, and the one or more processors are configured to invoke the computer instructions to cause the electronic device to perform the method of any one of claims 1-5.

9. A chip system, characterized by The chip system is applied to an electronic device, and the chip system comprises one or more processors configured to invoke computer instructions to cause the electronic device to perform the method of any one of claims 1-5.

10. A computer-readable storage medium comprising instructions, characterized in that, The instructions, when executed on an electronic device, cause the electronic device to perform the method of any one of claims 1-5.

Images