Image processing method and device

Abstract

The invention provides an image processing method and device which are applied to electronic equipment, the electronic equipment runs an application program, and the image processing method comprises the following steps: binding a first frame buffer; drawing a main scene of a first image in the application program to obtain image data of the main scene, and storing the image data of the main scene in the first frame buffer; switching the bound frame buffer to a second frame buffer, wherein the second frame buffer is different from the first frame buffer; drawing a user interface of the first image to obtain image data of the user interface, and storing the image data of the user interface in a second frame buffer; a first image is displayed, the first image being generated based on the image data of the main scene and the image data of the user interface.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and device. Background Art

[0002] Each frame of an application may include a main scene and a user interface (UI). The main scene may be the main screen of the frame, and the UI may be used to control the main screen. The UI of a frame may include icons and text, etc., and the UIs of different images of the same application may be fixed. Taking a game application as an example, the main scene of a frame of a game application may be the game screen, and the UI of the frame is used to control the angle of the game screen, the movement of characters in the game screen, etc., and different frames may share the same UI.

[0003] The UI of a frame image is on the main scene of the frame image, so that there are overlapping positions and intersection positions between the UI and the main scene. When processing the image, the processing effect of the overlapping positions and the intersection positions is poor, which reduces the image quality. Summary of the invention

[0004] The present application provides an image processing method and device for solving the problem of poor processing effect of overlapping positions and boundary positions when processing images, so as to improve image quality. Some embodiments of the present application provide the following technical solutions:

[0005] In a first aspect, the present application provides an image processing method, which is applied to an electronic device, and the electronic device runs an application. The image processing method includes: binding a first frame buffer; drawing the main scene of the first image in the application to obtain image data of the main scene, and the image data of the main scene is stored in the first frame buffer; switching the bound frame buffer to a second frame buffer, and the second frame buffer is different from the first frame buffer; drawing the user interface of the first image to obtain image data of the user interface, and the image data of the user interface is stored in the second frame buffer; sending and displaying the first image, and the first image is generated based on the image data of the main scene and the image data of the user interface. In this embodiment, the electronic device binds the first frame buffer when drawing the main scene of the first image, and the image data of the main scene is stored in the first frame buffer; binds the second frame buffer when drawing the user interface of the first image, and the image data of the user interface is stored in the second frame buffer. The first frame buffer is different from the second frame buffer, so the image data of the main scene of the first image and the image data of the user interface of the first image are stored in different frame buffers, thereby realizing the separation of the image data of the main scene and the image data of the user interface. Then, when processing the first image, the electronic device can process the image data of the main scene and the image data of the user interface separately, thereby reducing the mutual influence between the main scene and the user interface, so that the processing effect of the main scene and the processing effect of the user interface are better, and the problem of poor processing effect of the overlapping position and the boundary position when processing the first image is solved, thereby improving the image quality.

[0006] In some examples, the first frame buffer may be FB0, which is the default frame buffer of the electronic device. When rendering a frame of an image, the electronic device is bound to FB0 by default. The second frame buffer may be a frame buffer other than FB0. In some examples, after the electronic device completes drawing the main scene and the user interface of the first image, the image data of the main scene and the image data of the user interface are stored in FB0. When displaying the first image, the electronic device reads the image data of the main scene and the image data of the user interface from FB0 to complete the display of the first image.

[0007] In a possible implementation, the image processing method further includes: copying the image data of the main scene to a first resource created for the main scene; and / or, copying the image data of the user interface to a second resource created for the user interface, the second resource being different from the first resource. The first resource is different from the first frame buffer and the second frame buffer, and the second resource is also different from the first frame buffer and the second frame buffer, so that the image data of the main scene can be stored in the first frame buffer and the first resource, and / or, the image data of the user interface can be stored in the second frame buffer and the second resource. The electronic device can process the image data of the main scene in the first resource to reduce damage to the image data of the main scene in the first frame buffer. In addition, the electronic device can process the image data of the user interface in the second resource to reduce damage to the image data of the user interface in the second frame buffer.

[0008] In one possible implementation, copying the image data of the main scene to the first resource created for the main scene includes: each time a pixel value of a pixel in the main scene is read from the first frame buffer, the pixel value of the pixel is copied to a position corresponding to the pixel in the texture resource, the color information of the main scene includes the pixel value of each pixel in the main scene, the image data of the main scene includes the color information of the main scene, and the texture resource is the first resource.

[0009] In this embodiment, each time the electronic device reads the pixel value of a pixel in the main scene from the first frame buffer, it copies the pixel value of the pixel to the position corresponding to the pixel in the texture resource. During the copying process, the electronic device does not need to switch frame buffers, thereby eliminating the process of transferring the image data stored in the first frame buffer to an external storage unit, reducing the power consumption of the electronic device and improving flexibility.

[0010] In some examples, the first resource can be created based on the attributes of the main scene, such as based on the pixel format of the main scene and the size of the main scene, so that the first resource can accurately store the image data of the main scene. In this embodiment, the electronic device can determine whether the first resource has been created by the value of the identification bit. If the first resource is not created, the first resource is created before copying the image data of the main scene; if the first resource is created, the electronic device can further determine whether the first resource needs to be changed. If the first resource needs to be changed, the electronic device can recreate the first resource and copy the image data of the main scene to the recreated first resource. If the attributes of the main scene change after the electronic device creates the first resource, the electronic device needs to change the first resource. The second resource can be created based on the attributes of the user interface, and the electronic device also needs to determine whether the second resource is created or needs to be changed. The process can refer to the processing of the first resource and will not be explained here.

[0011] Taking the first resource as an example, the electronic device establishes an empty texture resource, and the attributes of the empty texture resource may include the identifier of the texture resource, the storage format of the texture resource, and the size of the texture resource. Among them, the identifier of the texture resource may be the ID of the texture resource, so as to find the texture resource through the ID of the texture resource. The storage format of the texture resource may indicate the dimension of the data stored in the texture resource, and the storage format of the texture resource may be the same as the pixel format of the current frame. For example, if the pixel format of the current frame is RGBA, then the storage format of the texture resource may also be RGBA. The size of the texture resource may indicate the size of the texture resource, and the size of the texture resource is the same as the size of the current frame. Thus, the empty texture resource established by the electronic device is equivalent to establishing an empty canvas, and the empty canvas is the same as the canvas of the current frame, so as to facilitate writing the image data of the main scene on the empty canvas, thereby completing the replication of the image data of the main scene on the texture resource.

[0012] In one possible implementation, each time a pixel value of a pixel in the main scene is read from the first frame buffer, the pixel value of the pixel is copied to a position in the texture resource corresponding to the pixel, including: calling a DrawCall function, the DrawCall function is used to read the pixel value of the pixel from the first frame buffer, and specifies an identifier of the texture resource, so as to copy the pixel value of the pixel to a position in the texture resource corresponding to the pixel through the DrawCall function.

[0013] In some examples, the central processing unit (CPU) of the electronic device calls the DrawCall function to notify the graphics processing unit (GPU) to read the pixel value of the pixel point from the first frame buffer, and copy the read pixel value to the position corresponding to the pixel point in the texture resource. The copying operation is simple and flexible.

[0014] In a possible implementation, after drawing the main scene of the first image in the application, before switching the bound frame buffer to the second frame buffer, the image processing method further includes: switching the bound frame buffer from the first frame buffer to the third frame buffer, the third frame buffer being different from the second frame buffer and the first frame buffer; copying the image data of the main scene to the first resource created for the main scene includes: copying the image data of the main scene to the third frame buffer, the third frame buffer being the first resource. In this embodiment, the electronic device can switch to the third frame buffer, copy the image data of the main scene to the third frame buffer, so that the image data of the main scene is stored in both the first frame buffer and the third frame buffer, so that the electronic device can process the image data in one of the two frame buffers, and can also store the original data of the main scene in the other frame buffer, thereby reducing the damage to the original data of the main scene while processing the image data of the main scene.

[0015] In a possible implementation, copying the image data of the main scene to the third frame buffer includes: setting the attribute of the first frame buffer as a read attribute, setting the attribute of the third frame buffer as a write attribute; reading the color information of the main scene from the first frame buffer, and copying the color information of the main scene to the third frame buffer, wherein the image data of the main scene includes the color information of the main scene. The format of the texture resource of the third frame buffer is the same as the format of the texture resource of the first frame buffer, and the copying of the color information of the main scene is controlled by setting the attributes of the third frame buffer and the first frame buffer, and the copying operation is relatively simple.

[0016] In a possible implementation, setting the attribute of the first frame buffer as a read attribute, setting the attribute of the third frame buffer as a write attribute, and reading the color information of the main scene from the first frame buffer, and copying the color information of the main scene to the third frame buffer include: calling the glBlitFramebuffer() function, the glBlitFramebuffer() function is used to set the attribute of the first frame buffer as a read attribute, set the attribute of the third frame buffer as a write attribute, and specify the identifier of the color attachment in the glBlitFramebuffer() function, the identifier of the color attachment is used to point to the color information of the main scene; responding to the glBlitFramebuffer() function, reading the color information of the main scene from the first frame buffer, and copying the color information of the main scene to the third frame buffer. In some examples, the CPU calls the glBlitFramebuffer() function, sets the attributes of the third frame buffer and the first frame buffer in the glBlitFramebuffer() function, and specifies the identifier of the color attachment; then the CPU notifies the GPU to read the color information of the main scene from the first frame buffer and copy the color information of the main scene to the third frame buffer.

[0017] In one possible implementation, copying the image data of the main scene to the third frame buffer includes: rasterizing the image data of the main scene in the first frame buffer, sampling the rasterized image data to obtain a sampling result of the image data; writing the sampling result of the image data into the third frame buffer, and using the sampling result of the image data to reconstruct the main scene.

[0018] In one possible implementation, switching the bound frame buffer to the second frame buffer includes: switching the bound frame buffer from the third frame buffer to the first frame buffer, and then switching the bound frame buffer from the first frame buffer to the second frame buffer, thereby achieving separation of image data of the main scene and image data of the user interface while reducing disruption to the existing rendering logic; or, switching the bound frame buffer to the second frame buffer includes: switching the bound frame buffer from the third frame buffer to the second frame buffer, thereby reducing frame buffer switching and reducing power consumption.

[0019] In a possible implementation, the first image is sent for display, and the first image is generated based on the image data of the main scene and the image data of the user interface, including: after intercepting the display command issued by the application, calling the super-resolution algorithm to process the image data of the main scene to obtain the first image data, and the resolution corresponding to the first image data is greater than the resolution corresponding to the image data of the main scene; sending the first image, the first image is generated based on the first image data and the image data of the user interface, and the resolution corresponding to the image data of the user interface is the same as the resolution corresponding to the first image data. In this embodiment, when the electronic device sends the first image for display, the image data of the user interface in the second frame buffer is reused, and the calling of the super-resolution algorithm to process the image data of the user interface is omitted, which saves time to a certain extent.

[0020] In some examples, the resolution corresponding to the image data of the user interface in the second frame buffer is the same as the resolution corresponding to the image data of the main scene, and the electronic device can call a super-resolution algorithm to process the image data of the main scene and the image data of the user interface respectively, and then send them for display based on the processed image data. In some examples, the display command can be an eglswapbuffers() function, the CPU intercepts the eglswapbuffers() function, and then notifies the GPU to call a super-resolution algorithm to process the image data of the main scene.

[0021] In a possible implementation, displaying a first image, wherein the first image is generated based on the image data of the main scene and the image data of the user interface, comprises: after intercepting a display command issued by an application, calling an interpolation algorithm to process the image data of the main scene to obtain second image data, wherein the second image data and the image data of the main scene are stored in different texture resources; displaying the first image, wherein the first image is generated based on the image data of the main scene and the image data of the user interface; displaying the second image, wherein the second image is generated based on the second image data and the image data of the user interface. The display order of the first image and the second image may be determined based on the interpolation algorithm. The texture resources used to store the second image data and the image data of the main scene may be in the same frame buffer or in different frame buffers.

[0022] In some examples, the electronic device may call an interpolation algorithm to process the image data of the user interface to obtain third image data, the third image data and the image data of the user interface are stored in different texture resources, and the second image is generated based on the second image data and the third image data. The texture resources used to store the third image data and the image data of the user interface may be in the same frame buffer or in different frame buffers.

[0023] In a second aspect, the present application provides an electronic device, which includes: one or more processors and a memory; the memory is used to store computer program code, the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the above-mentioned image processing method.

[0024] In a third aspect, the present application provides a computer-readable storage medium, which is used to store a computer program, and the computer program implements the above-mentioned image processing method when executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of calling a frame insertion algorithm to generate an image provided in an embodiment of the present application;

[0026] Figure 2 and Figure 3 Another schematic diagram of calling a frame insertion algorithm to generate an image provided by an embodiment of the present application;

[0027] Figure 4 A hardware structure diagram of an electronic device provided in an embodiment of the present application;

[0028] Figure 5 A flowchart of an image processing method provided in an embodiment of the present application;

[0029] Figure 6 Another flow chart of the image processing method provided in the embodiment of the present application;

[0030] Figure 7 Another flow chart of the image processing method provided in the embodiment of the present application;

[0031] Figure 8 Another flowchart of the image processing method provided in the embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0033] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0034] The multiple involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first", "second", etc. are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0035] The resolution of display screens of electronic devices is getting higher and higher, and the quality and scene complexity of applications rendered on electronic devices are also getting higher and higher, which makes the memory usage and power consumption of electronic devices increase accordingly. As a result, when running applications, electronic devices are prone to heating, frame loss, etc. due to resource and power consumption limitations, thus affecting the user experience.

[0036] In some embodiments of the present application, when an electronic device runs an application, the electronic device can apply super-resolution, interpolation and other algorithms to the image rendering of the application. Among them, the interpolation algorithm can be based on the image data of two frames of images in the application, and generate a new image frame by interpolation; the super-resolution algorithm can be to process low-resolution image data to generate high-resolution image data to improve the resolution of a frame of image. By calling the interpolation algorithm or the super-resolution algorithm, the electronic device can reduce the phenomenon of heating, frame loss, etc., thereby improving the user experience. Among them, the image data can also be called texture data.

[0037] A frame of image may include a main scene and a UI. The drawing process of a frame of image may be: the electronic device may first draw the main scene and then draw the UI to obtain the image data of the frame of image. The image data of the frame of image may include the image data of the main scene and the image data of the UI. The image data of the frame of image may be stored in a frame buffer (FB). Because the UI is located above the main scene, there are overlapping positions and boundary positions between the UI and the main scene. Because the main scene of a frame of image is the main screen of the frame of image, the image data of the main scene and the image data of the UI are integrated into one FB, making it difficult for the electronic device to separate the image data of the main scene and the image data of the UI. Therefore, when processing the overlapping position of the UI and the main scene, the interpolation algorithm and the super-resolution algorithm mainly process according to the processing method of the main scene, resulting in poor processing effect of the UI, thereby reducing the image quality. For example, at the overlapping position of the UI and the main scene, the UI with semi-transparency covers part of the pixels in the main scene, so that the pixels of the UI and the main scene at the overlapping position are mixed together. When processing according to the processing method of the main scene, it is difficult to distinguish the UI and the main scene, resulting in poor processing effect. One side of the junction is the UI and the other side is the main scene, so the pixel values ​​on both sides of the junction are greatly different, which makes the processing effect of the junction poor and reduces the image quality.

[0038] Some embodiments of the present application provide an image processing method, which binds a first FB when drawing a main scene to store the image data of the main scene in the first FB, and after drawing the main scene, stores the image data of the main scene in the resources created for the main scene, and the resources created for the main scene are resources (i.e., first resources) for storing the image data of the main scene. When drawing a UI, bind a second FB, which is different from the first FB, to store the image data of the UI in the second FB, and complete the separation of the image data of the main scene and the image data of the UI. When calling the interpolation algorithm and the super-resolution algorithm, the electronic device can process the main scene and the UI respectively, and after completing the processing of the main scene and the UI, generate a frame of image based on the processed main scene and the processed UI. Because the electronic device processes the main scene and the UI respectively, the main scene and the UI will not affect each other, so that the processing effect of the main scene and the processing effect of the UI are better, and correspondingly, the quality of the generated image is also improved.

[0039] Take the interpolation algorithm as an example to illustrate. Figure 1 An optional process of an electronic device calling an interpolation algorithm to generate an image is shown. Figure 1 In the example, the first frame image and the second frame image are two frame images in the application currently running on the electronic device. The second frame image and the first frame image may be two adjacent frame images. The electronic device calls the interpolation algorithm to interpolate the first frame image and the second frame image to generate the third frame image. The display order of the first frame image to the third frame image is: the first frame image, the third frame image, and the second frame image. Therefore, the electronic device also changes the display order of the images (also called the display order) when generating the third frame image. Because the electronic device does not separate the main scene and the UI, the interpolation algorithm processes the overlapping and boundary positions of the UI and the main scene in the same way as the main scene. As shown in Figure 1 In the example, the UIs of the first and second frames include text, which overlaps with the white background of the main scene. The interpolation algorithm processes the text according to the white background, making the text in the generated third frame blurred, thereby reducing the image quality.

[0040] Figure 2 and Figure 3 Another optional process of an electronic device calling a frame insertion algorithm to generate an image is shown. Figure 2In the process, the electronic device separates the main scene and the UI of the first frame image and the second frame image to obtain the main scene of the first frame image and the UI of the first frame image, and separates the main scene and the UI of the second frame image to obtain the main scene of the second frame image and the UI of the second frame image. After completing the separation of the main scene and the UI of the two frames, the electronic device calls the interpolation algorithm to interpolate the main scene of the first frame image and the main scene of the second frame image to generate a new main scene, and calls the interpolation algorithm to interpolate the UI of the first frame image and the UI of the second frame image to generate a new UI; then the electronic device generates a third frame image based on the new main scene and the new UI, for example, copying the image data of the new main scene and the image data of the new UI to the same FB to generate the third frame image. The electronic device reads the image data from the FB to complete the display of the third frame image.

[0041] Because the new main scene is obtained by the electronic device interpolating the main scene of the first frame image and the main scene of the second frame image, and the new UI is obtained by the electronic device interpolating the UI of the first frame image and the UI of the second frame image, the mutual influence of the main scene and the UI is avoided, and the quality of the new main scene and the new UI is improved. Figure 3 The text in the new UI is clearer, which improves the image quality of the third frame.

[0042] In some embodiments of the present application, the above-mentioned image processing method can be applied to electronic devices, and the electronic devices can be mobile phones, tablet computers, desktops, laptops, notebook computers, ultra-mobile personal computers (Ultra-mobile Personal Computer, UMPC), handheld computers, netbooks, personal digital assistants (Personal Digital Assistant, PDA), wearable electronic devices, smart watches and other devices. The present application does not impose any special restrictions on the specific forms of the above-mentioned electronic devices.

[0043] In this embodiment, an optional hardware structure of the electronic device is as follows: Figure 4As shown, the electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, an antenna 1, an antenna 2, a mobile communication module, a wireless communication module, an audio module, a speaker, a receiver, a microphone, an earphone interface, a sensor module, a button, a motor, an indicator, a camera, a display screen, and a subscriber identity module (SIM) card interface, etc. The sensor module may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0044] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0045] The processor may include one or more processing units, for example: the processor may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. For example, in the present application, the processor can separate the image data of the main scene and the image data of the UI for a frame of image, and after obtaining the image data of the main scene and the image data of the UI, the processor calls an interpolation algorithm and a super-resolution algorithm to process at least the image data of the main scene. In addition, the processor can also reuse the separated image data of the UI, for example, the processor calls the interpolation algorithm to perform extrapolation or interpolation processing on the image data of the main scene of the two frames of images to obtain the image data of the new main scene, and then reuses the image data of the UI of one of the two frames of images, and generates a new image based on the image data of the new main scene and the reused image data of the UI.

[0046] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0047] A memory may also be provided in the processor for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. The memory may store instructions or data that the processor has just used or is cyclically used. If the processor needs to use the instruction or data again, it may be directly called from the memory, thereby avoiding repeated access, reducing the waiting time of the processor, and thus improving the efficiency of the system.

[0048] Electronic devices implement display functions through GPU, display screen, and application processor. GPU is a microprocessor for image processing, connecting the display screen and application processor. GPU is used to perform mathematical and geometric calculations for image rendering. The processor may include one or more GPUs, which execute program instructions to generate or change display information.

[0049] The display screen is used to display images, videos, etc. The display screen includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oled, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens, where N is a positive integer greater than 1.

[0050] A series of graphical user interfaces (GUIs) can be displayed on the display screen of an electronic device, and these GUIs are the main screens of the electronic device. Generally speaking, the size of the display screen of an electronic device is fixed, and only limited controls can be displayed on the display screen of the electronic device. A control is a GUI element, which is a software component included in an application that controls all data processed by the application and interactive operations on these data. Users can interact with the controls through direct manipulation to read or edit relevant information of the application. Generally speaking, controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. For example, in an embodiment of the present application, the display screen can display images included in the application.

[0051] The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor through the external memory interface to implement data storage functions. For example, files such as music and images can be saved in the external memory card.

[0052] The internal memory can be used to store computer executable program codes, which include instructions. The processor executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory. For example, in this embodiment, the processor can separate the image data of the main scene and the image data of the UI for the image in the application by executing the instructions stored in the internal memory. The internal memory may include a storage program area and a storage data area. Among them, the storage program area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The storage data area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory, and / or the instructions stored in the memory provided in the processor.

[0053] In an embodiment of the present application, the electronic device completes the rendering of a frame of image through the CPU and the GPU, wherein the rendering process of a frame of image includes: binding a first FB, drawing the main scene to obtain image data of the main scene and storing the image data of the main scene in the first FB; binding a second FB, drawing the UI to obtain image data of the UI and storing the image data of the UI in the second FB, the second FB is different from the first FB, and realizes the separation of the image data of the main scene and the image data of the UI.

[0054] In addition, after drawing the main scene, the GPU may store the image data of the main scene in the resources created for the main scene. In some examples, the GPU stores the color information of the main scene in the resources created for the main scene. The image data of the main scene includes the color information of the main scene. The attributes of the resources created for the main scene are determined based on the attributes of the color information of the main scene, so that the resources can store the color information of the main scene. The GPU may call an interpolation algorithm or a super-resolution algorithm to process the color information of the main scene to obtain new color information. The color information of the new image is used to generate a new image. In this way, when the GPU generates a new image, it can also ensure that the image data of the main scene in the first FB is not destroyed, so as to ensure that the image data of the main scene can be normally displayed. The display may be to display the main scene on a display screen.

[0055] The image processing method provided by the embodiment of the present application is described below with reference to the flowchart. Figure 5 As shown, the image processing method may include the following steps:

[0056] S101. The CPU responds to the rendering command sent by the application and starts rendering the current frame. The CPU responds to the rendering command and instructs the GPU to start rendering the current frame. During the rendering of the current frame, the GPU binds the first FB to store the image data of the current frame in the first FB. In some examples, the first FB may be FB0. When the GPU displays the current frame, the GPU reads the image data of the current frame from FB0. In some examples, the first FB may be an FB other than FB0. Before the GPU displays the current frame, the GPU copies the image data of the current frame in the first FB to FB0, and then reads the image data of the current frame from FB0 to complete the display of the current frame.

[0057] S102, the GPU draws the main scene of the current frame to obtain the image data of the main scene. Because the GPU is bound to the first FB, when the GPU draws the main scene, the image data of the main scene is drawn to (i.e., written to) the first FB, and the image data of the main scene is stored by the first FB. Because the image data of the current frame is read from FB0 when sending the display, the first FB is preferably FB0, so that when drawing the main scene of the current frame, the image data of the main scene is written to FB0, which reduces the link of copying the image data to FB0 compared to writing the image data of the main scene to other FBs.

[0058] S103, after the GPU finishes drawing the main scene of the current frame, it switches the bound FB from the first FB to the second FB. The second FB is different from the first FB, and the switching of the bound FB from the first FB to the second FB mainly involves switching the FB corresponding to the current frame, so that the FB used to store the image data of the current frame changes from the first FB to the second FB.

[0059] S104, the GPU draws the UI of the current frame to obtain the image data of the UI. Because the FB bound to the GPU is switched from the first FB to the second FB, the image data of the UI is drawn to the second FB during the drawing of the UI of the current frame, so that the first FB can store the image data of the main scene of the current frame, and the second FB can store the image data of the UI of the current frame, thereby storing the image data of the main scene of the current frame and the image data of the UI of the current frame respectively through two different FBs, thereby realizing the separation of the image data of the main scene and the image data of the UI.

[0060] It can be understood that: in the existing rendering process, the GPU starts to draw the UI after drawing the main scene, so as to draw the image data of the main scene and the image data of the UI into the same FB. In the embodiment of the present application, in order to separate the image data of the UI from the image data of the main scene, the GPU needs to bind different FBs when drawing the UI and the main scene, so as to draw the image data of the UI and the image data of the main scene into different FBs, so as to separate the image data of the two. Therefore, the GPU changes the existing rendering process, switches the bound FB after drawing the main scene, and then draws the UI.

[0061] In some examples, after intercepting the main scene drawing end instruction sent by the application, the CPU determines that the main scene drawing is completed. In some examples, the CPU can call the glBindFramebuffer() function, which indicates that FB switching is to be performed. The CPU sends an FB switching instruction to the GPU, and the GPU switches the bound FB from the first FB to the second FB, so that the image data of the UI is drawn to the second FB during the UI drawing process, and after the UI drawing is completed, the GPU completes the separation of the image data of the UI and the image data of the main scene, so that the GPU can smoothly extract / separate the image data of the main scene and the image data of the UI.

[0062] S105 . After the GPU has finished drawing the UI of the current frame and the image data of the main scene and the image data of the UI are written into FB0 , the GPU reads the image data of the main scene and the image data of the UI from FB0 to complete the display of the current frame.

[0063] In an embodiment of the present application, after drawing the UI of the current frame, the CPU can intercept the eglswapbuffers() function issued by the application. The eglswapbuffers() function can be used to instruct the display of the current frame. The CPU can send a display instruction to the GPU, and the GPU displays the current frame. However, in an embodiment of the present application, because the GPU draws the image data of the main scene and the image data of the UI into different FBs respectively, the image data of the main scene and the image data of the UI are not stored in the same FB, and when displaying the current frame, FB0 needs to store the image data of the main scene and the image data of the UI at the same time. Therefore, after receiving the display instruction, the GPU needs to write the image data of the main scene and the image data of the UI in FB0, and then read the image data of the main scene and the image data of the UI from FB0 to complete the display of the current frame. Therefore, the GPU completes the display of the current frame by Figure 5 The image processing method shown modifies the existing rendering process so that the separation of the image data of the main scene and the image data of the UI is completed through the modified rendering process, and the current frame can be displayed normally, thereby completing the separation of the image data of the main scene and the image data of the UI without the application being aware of it.

[0064] In some embodiments, after the CPU intercepts the eglswapbuffers() function, it triggers the GPU to do two things: one is to perform special processing on the image data of the main scene and / or the image data of the UI, and the other is to send the image data for display based on the processed image data. Among them, the special processing may include the GPU calling one of the super-resolution algorithm and the interpolation algorithm. When the GPU calls the super-resolution algorithm or the interpolation algorithm, the GPU's processing is slightly different, which is explained below with reference to the accompanying drawings.

[0065] Figure 6 Another flow chart of the image processing method provided by the embodiment of the present application is shown. In the image processing method, the GPU calls the super-resolution algorithm to process the image data of the main scene, and completes the conversion of the image data of the main scene from low resolution to high resolution. The process is as follows:

[0066] S201. The CPU responds to a rendering command issued by an application program and starts rendering a current frame.

[0067] S202: The GPU draws the main scene of the current frame to obtain low-resolution image data of the main scene. The low-resolution image data of the main scene may be image data of the main scene at a low resolution. The low-resolution image data of the main scene is drawn into a first FB, and before drawing the main scene of the current frame, the GPU binds the first FB.

[0068] S203 . After drawing the main scene of the current frame, the GPU switches the bound FB from the first FB to the second FB to complete the FB binding change.

[0069] S204, the GPU draws the UI of the current frame to obtain high-resolution image data of the UI. The high-resolution image data of the UI can be image data of the UI at a high resolution, where the high resolution is greater than the low resolution. This embodiment does not limit the values ​​of the high resolution and the low resolution. Because the FB bound to the GPU switches from the first FB to the second FB, the high-resolution image data of the UI is drawn to the second FB during the drawing of the UI of the current frame, thereby storing the low-resolution image data of the main scene and the high-resolution image data of the UI respectively through two different FBs, thereby achieving the separation of the low-resolution image data of the main scene and the high-resolution image data of the UI.

[0070] S205. After drawing the UI of the current frame, the GPU calls a super-resolution algorithm to process the low-resolution image data of the main scene to obtain the high-resolution image data of the main scene, thereby completing the conversion of the image data of the main scene from low resolution to high resolution. The resolution corresponding to the high-resolution image data of the main scene is the same as the resolution corresponding to the high-resolution image data of the UI.

[0071] S206 . After writing the high-resolution image data of the main scene and the high-resolution image data of the UI into FB0 , the GPU reads the high-resolution image data of the main scene and the high-resolution image data of the UI from FB0 to complete the display of the current frame.

[0072] exist Figure 6In the image processing method shown, after the CPU intercepts the eglswapbuffers() function, it notifies the GPU to process the low-resolution image data of the main scene. The GPU can call the super-resolution algorithm to convert the low-resolution image data of the main scene from low resolution to high resolution, and then write the high-resolution image data of the main scene into FB0, and write the high-resolution image data of the UI into FB0 to reuse the high-resolution image data of the UI. After receiving the display transmission instruction received by the CPU, the GPU reads the high-resolution image data of the main scene and the high-resolution image data of the UI from FB0 to complete the display transmission of the current frame. After separating the low-resolution image data of the main scene and the high-resolution image data of the UI, the GPU can call the super-resolution algorithm to process the low-resolution image data of the main scene separately, avoid mutual influence between the main scene and the UI, improve the accuracy of the high-resolution image data of the main scene and the high-resolution image data of the UI, and prevent the processing effect of the overlapping position and the boundary position between the main scene and the UI from deteriorating, thereby improving the quality of the current frame.

[0073] In some examples, the GPU may draw the UI of the current frame at a low resolution to obtain low-resolution image data of the UI; after the CPU intercepts the eglswapbuffers() function, it notifies the GPU to process the low-resolution image data of the UI, and the GPU may call a super-resolution algorithm to convert the low-resolution image data of the UI from low resolution to high resolution, which may also avoid mutual influence between the main scene and the UI to improve the accuracy of the high-resolution image data of the main scene and the high-resolution image data of the UI.

[0074] Figure 7 Another flow chart of the image processing method provided by the embodiment of the present application is shown. In the image processing method, the GPU calls the interpolation algorithm to process the image data of the main scene and the image data of the UI respectively. The process is as follows:

[0075] S301. The CPU responds to a rendering command sent by an application program and starts rendering the current frame.

[0076] S302: The GPU draws the main scene of the current frame to obtain image data of the main scene. The image data of the main scene is drawn into the first FB. Before drawing the main scene of the current frame, the GPU binds to the first FB.

[0077] S303: After drawing the main scene of the current frame, the GPU switches the bound FB from the first FB to the second FB.

[0078] S304: The GPU draws the UI of the current frame to obtain image data of the UI. Because the FB bound to the GPU is switched from the first FB to the second FB, the image data of the UI is drawn into the second FB during the process of drawing the UI of the current frame.

[0079] S305 . After drawing the UI of the current frame, the GPU calls the interpolation algorithm to process the image data of the main scene and the image data of the UI respectively to obtain new image data of the main scene and new image data of the UI.

[0080] In some examples, the GPU can call the interpolation algorithm to process only the image data of the main scene to obtain the new image data of the main scene, and then the GPU reuses the image data of the UI. Usually, the interpolation algorithm interpolates or extrapolates two adjacent frames of images. In the case of interpolation, the GPU can reuse the image data of the UI of any one of the two frames; in the case of extrapolation, the GPU can reuse the image data of the UI of the last frame of the two frames. This embodiment does not limit the reuse of the image data of the UI.

[0081] S306 . After writing the image data of the main scene and the image data of the UI into FB0 , the GPU reads the image data of the main scene and the image data of the UI from FB0 , and completes the display of the current frame.

[0082] S307 . After writing the image data of the new main scene and the image data of the new UI in FB0 , the GPU reads the image data of the new main scene and the image data of the new UI from FB0 , and completes the display of the new image frame.

[0083] It can be understood that the interpolation algorithm can include an interpolation frame algorithm and an external interpolation frame algorithm. The interpolation frame algorithm is based on two adjacent frames to generate an intermediate frame between the two adjacent frames, and the external interpolation frame algorithm is based on two adjacent frames to generate a subsequent frame between the two adjacent frames. When the GPU calls the interpolation algorithm to process the image data of the main scene and / or the image data of the UI, the GPU can obtain a new image frame (i.e., a new image frame), and the GPU completes the display of the current frame and the new image frame. That is, after the CPU intercepts the eglswapbuffers() function, it triggers the GPU to perform the following operations:

[0084] 1. Call the interpolation algorithm to process the image data of the main scene and / or the image data of the UI to generate new image data of the main scene and / or the image data of the UI; 2. Complete the display of the current frame and the new image frame. The order of displaying the current frame and the new image frame is based on the interpolation algorithm, which will not be elaborated. For example, after the CPU intercepts the eglswapbuffers() function, it notifies the GPU to call the interpolation algorithm to process the image data of the main scene and / or the image data of the UI to generate new image data of the main scene and / or the image data of the UI; then the CPU sends two display instructions to the GPU, and under the action of the two display instructions, the GPU completes the display of the current frame and the new image frame.

[0085] In some examples, the GPU may call a super-resolution algorithm and an interpolation algorithm to process the image data of the main scene and / or the image data of the UI. One method may be: the GPU calls an interpolation algorithm to process the low-resolution image data of the main scene to obtain low-resolution image data of a new main scene, and then the GPU calls a super-resolution algorithm to process the low-resolution image data of the new main scene to obtain high-resolution image data of the new main scene. The GPU may refer to this method to process the low-resolution image data of the UI to reduce heating and improve the user experience. In addition, the GPU may obtain high-resolution image data of the UI when drawing the UI, and reuse the high-resolution image data of the UI as high-resolution image data of the new UI. For other methods, this embodiment will not list them one by one.

[0086] Figure 8 Another flowchart of the image processing method provided in the embodiment of the present application is shown, which may include the following steps:

[0087] S401: The CPU responds to a rendering command issued by an application program and starts rendering the current frame.

[0088] S402: The GPU draws the main scene of the current frame to obtain image data of the main scene. The image data of the main scene is drawn into the first FB. Before drawing the main scene of the current frame, the GPU binds to the first FB.

[0089] S403, the CPU determines whether the main scene drawing is completed, if it is completed, executes step S404, if it is not completed, returns to execute step S402.

[0090] S404. The CPU determines whether to create a resource for storing the image data of the main scene or whether the resource needs to be changed. If yes, execute step S405; if no, execute step S406.

[0091] The CPU can determine whether to create a resource for storing the image data of the main scene by the value of the flag bit. For example, if the value of the flag bit is 1, the CPU determines that the GPU has created a resource for storing the image data of the main scene. If the value of the flag bit is 0, the CPU determines that the GPU has not created a resource for storing the image data of the main scene. If the GPU has not created a resource for storing the image data of the main scene after the main scene is drawn, the CPU notifies the GPU to create the resource after the main scene is drawn, so as to copy the image data of the main scene to the resource, so that the image data of the main scene is stored in both the first FB and the resource.

[0092] If the CPU determines that the GPU has created a resource for storing the image data of the main scene after the drawing of the main scene is completed, the CPU determines whether the resource needs to be changed. The resource for storing the image data of the main scene is created based on the attributes of the main scene (such as the pixel format of the main scene and the size of the main scene). Before drawing the main scene, the GPU creates the image data for storing the main scene, but when the main scene starts to be drawn, the attributes of the main scene change. Then, after the drawing of the main scene is completed, the CPU determines that the resource for storing the image data of the main scene that has been created needs to be changed, and the CPU notifies the GPU to recreate the resource. The GPU needs to change the resource for storing the image data of the main scene according to the attributes of the changed main scene so that the resource can accurately store the image data of the main scene.

[0093] S405. The GPU obtains resources for storing image data of the main scene, and executes step S406.

[0094] S406: The GPU copies the image data of the main scene to the resource.

[0095] S407 : The GPU switches the bound FB from the first FB to the second FB.

[0096] S408: The GPU draws the UI of the current frame to obtain image data of the UI. Because the FB bound to the GPU is switched from the first FB to the second FB, the image data of the UI is drawn to the second FB during the process of drawing the UI of the current frame.

[0097] S409, the CPU determines whether the current frame drawing is completed. If it is completed, step S410 is executed. If it is not completed, the CPU returns to step S408. After the UI drawing of the current frame is completed, the current frame drawing is completed. In some examples, after the CPU intercepts the UI drawing end instruction sent by the application, it determines that the GPU has completed the UI drawing.

[0098] S410, GPU performs special processing on the image data of the main scene and / or the image data of the UI. In step S410, the GPU may perform special processing on the image data of the main scene in the resource for storing the image data of the main scene, so as to ensure that the image data of the main scene is stored in the first FB when the image data of the main scene is specially processed (that is, the original data of the main scene of the current frame can also be stored), thereby avoiding damage to the original data of the main scene.

[0099] One point that needs to be explained here is that before the GPU performs special processing on the image data of the UI, it can copy the image data of the UI to the resource used to store the image data (i.e., the second resource) to avoid damage to the image data of the UI. The resource used to store the image data of the UI can be created before the special processing is performed on the image data of the UI, or it can be created together with the resource used to store the image data of the main scene. After the current frame is drawn, the CPU determines whether the resource for storing the image data of the UI has been created or whether the resource for storing the image data of the UI needs to be changed. If it has not been created or needs to be changed, the GPU is notified to recreate it to obtain the resource for storing the image data of the UI. The method of copying the image data of the UI in the resource can refer to the method of copying the image data of the main scene, which will not be explained here.

[0100] In addition, after the image data of the main scene and / or the image data of the UI are specially processed, the processing results of the main scene and the processing results of the UI can be stored in different texture resources respectively. The texture resource storing the processing results of the main scene is different from the texture resource storing the image data of the main scene, and the two texture resources can belong to the same FB or different FBs; the texture resource storing the processing results of the UI is different from the texture resource storing the image data of the UI, and the two texture resources can belong to the same FB or different FBs.

[0101] S411, after the GPU writes the image data of the main scene and the image data of the UI in FB0, it reads the image data of the main scene and the image data of the UI from FB0 to complete the display of the current frame. Among them, when the special processing is one of the super-resolution algorithm and the interpolation algorithm, step S410 and step S411 are slightly different. For details, please refer to the above Figure 6 and Figure 7 If the special processing is a super-resolution algorithm, step S410 and step S411 can refer to the above Figure 6 Steps S205 and S206 in the process shown in the figure; when the special processing is the interpolation algorithm, steps S410 and S411 can refer to the above Figure 7 Steps S305 to S307 in the illustrated process are not described in detail here.

[0102] In this embodiment, the GPU may obtain a resource for storing the image data of the main scene and copy the image data of the main scene to the resource in the following manner:

[0103] In the first method, the GPU creates an empty texture resource, which can be a resource for storing the image data of the main scene. The attributes of the empty texture resource include the identifier of the texture resource, the storage format of the texture resource, and the size of the texture resource. Among them, the identifier of the texture resource can be the ID of the texture resource, so as to find the texture resource through the ID of the texture resource. The storage format of the texture resource can indicate the dimension of the data stored in the texture resource. The storage format of the texture resource can be the same as the pixel format of the current frame. For example, if the pixel format of the current frame is RGBA, then the storage format of the texture resource can also be RGBA. The size of the texture resource can indicate the size of the texture resource, and the size of the texture resource is the same as the size of the current frame. Therefore, the empty texture resource created by the GPU is equivalent to creating an empty canvas, and the empty canvas is the same as the canvas of the current frame, so as to write the image data of the main scene on the empty canvas, thereby completing the copying of the image data of the main scene on the texture resource.

[0104] For example, the CPU may call a DrawCall function, which may be used to notify the GPU to read the image data of the main scene from the first FB, and specify the identifier of the resource used to store the image data of the main scene, so as to copy the image data of the main scene in the resource. In some examples, the DrawCall function may be used to read the color information of the main scene from the first FB, and specify the identifier of the resource used to store the image data of the main scene as the identifier of the above-mentioned empty texture resource, and the image data of the main scene includes the color information of the main scene.

[0105] Among them, the DrawCall function can take the pixel value of each pixel in the main scene as input, and copy (duplicate) the pixel value of each pixel to the texture resource, and the color information of the main scene includes the pixel value of each pixel. In an embodiment of the present application, the DrawCall function can use the Framebuffer_Fetch technology, and the Framebuffer_Fetch technology allows the pixel value of the pixel currently being processed to be read in the fragment shader. There are two interfaces for reading pixel values ​​in the Framebuffer_Fetch technology, namely GL_ARM_shader_framebuffer_fetch and GL_EXT_shader_framebuffer_fetch, and different GPU support may be slightly different. Take GL_EXT_shader_framebuffer_fetch as an example: in the DrawCall function, declare that the pixel value of each pixel in the main scene is used as input, and declare that the pixel value of each pixel will be copied to the empty texture resource created by the GPU. In this way, the CPU sends a notification to the GPU based on the DrawCall function. After receiving the notification, the GPU reads the pixel value of each pixel in the main scene from the first FB, and copies the pixel values ​​of these pixel points to the texture resource. For example, each time the pixel value of a pixel point is read, the pixel value of the pixel point is copied to the position corresponding to the pixel point in the texture resource, thereby completing the copying of the color information of the main scene.

[0106] In the first way, the timing when the CPU calls the DrawCall function to notify the GPU to copy the color information of the main scene can be when the GPU finishes drawing the main scene. For example, in the existing rendering process, after the GPU finishes drawing the main scene, the CPU can intercept the instruction to draw the UI issued by the application to notify the GPU to draw the UI. However, in an embodiment of the present application, after the CPU intercepts the instruction to draw the UI issued by the application, it suspends the issuance of the instruction. The CPU will call the DrawCall function to notify the GPU to copy the color information of the main scene, and after the GPU completes the copying of the color information of the main scene, it will issue the instruction to draw the UI to the GPU. This is an example of the GPU copying the color information of the main scene. The embodiment of the present application does not limit the timing when the GPU copies the color information of the main scene.

[0107] The second method is that after the GPU finishes drawing the main scene, the GPU switches the bound FB from the first FB to the third FB, which is different from the first FB and the second FB. The CPU calls the glBlitFramebuffer() function, which notifies the GPU that the attribute of the first FB is a read attribute (GL_READ_FRAMEBUFFER) and the attribute of the third FB is a write attribute (GL_DRAW_FRAMEBUFFER), and specifies the identifier of the color attachment in the function. The identifier of the color attachment can point to the color information of the main scene, such as glBlitFramebuffer(GL_COLOR_BUFFER_BIT), so as to read the color information of the main scene from the first FB through the glBlitFramebuffer() function, and copy the color information of the main scene to the third FB to complete the copying of the color information of the main scene. When copying the color information of the main scene through the glBlitFrameBuffer() function, the format of the texture resources of the third FB is the same as that of the texture resources of the first FB, which reduces flexibility. The format mainly includes the storage format of the texture resources and the size of the texture resources.

[0108] After completing the copying of the color information of the main scene, the GPU switches the bound FB. In some examples, the GPU can switch the bound FB from the third FB to the first FB, and then switch the bound FB from the first FB to the second FB, so that the image data of the main scene and the image data of the UI can be separated without disrupting the existing rendering logic. In some examples, the GPU can switch the bound FB from the third FB to the second FB, reducing FB switching and reducing power consumption.

[0109] In the second method, the GPU changes FB at least twice. Before switching the bound FB from one FB to another, the GPU needs to transfer the data of the previous FB from the inside of the GPU to the external storage unit. The external storage unit is a storage unit outside the GPU, which requires a large bandwidth during the data transmission process. However, the bandwidth of the electronic device is limited, which leads to an increase in the power consumption of the electronic device during the data transmission process to the external storage unit. Therefore, in the second method, the number of GPU binding changes is greater than the number of GPU binding changes in the first method. Compared with this method, the power consumption of the electronic device in the first method is smaller.

[0110] The third method is that after the GPU finishes drawing the main scene, the GPU switches the bound FB from the first FB to the third FB, which is different from the first FB and the second FB. The GPU uses the image data in the first FB as input and copies the image data to the third FB. After the GPU rasterizes the image data in the first FB, it samples the rasterized image data and writes the sampling result to the third FB. The sampling result of the texture resource can reconstruct the main scene, thereby completing the copying of the image data of the main scene. Because it involves rasterization and sampling of image data, power consumption is increased.

[0111] After completing the copying of the image data of the main scene, the GPU switches the bound FB. In some examples, the GPU can switch the bound FB from the third FB to the first FB, and then switch the bound FB from the first FB to the second FB, so that the image data of the main scene and the image data of the UI can be separated without disrupting the existing rendering logic. In some examples, the GPU can switch the bound FB from the third FB to the second FB, reducing FB switching and reducing power consumption.

[0112] In addition, an embodiment of the present application also provides an electronic device, which includes: one or more processors and a memory; the memory is used to store computer program code, and the computer program code includes computer instructions. When one or more processors execute the computer instructions, the electronic device executes the above-mentioned image processing method.

[0113] An embodiment of the present application also provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed, the above-mentioned image processing method is implemented.

Claims

1. An image processing method, characterized in that: Applied in an electronic device, the electronic device runs an application program, and the image processing method includes: Bind the first framebuffer; Draw a main scene of a first image in the application program to obtain image data of the main scene, and store the image data of the main scene in a first frame buffer; Switching the bound frame buffer to a second frame buffer, the second frame buffer being different from the first frame buffer; Draw a user interface of the first image to obtain image data of the user interface, and store the image data of the user interface in a second frame buffer; The first image is displayed, where the first image is generated based on image data of the main scene and image data of the user interface.

2. The image processing method according to claim 1, characterized in that: The image processing method further comprises: copying the image data of the main scene to a first resource created for the main scene; and / or The image data of the user interface is copied to a second resource created for the user interface, the second resource being different from the first resource.

3. The image processing method according to claim 2, characterized in that: The copying of the image data of the main scene to the first resource created for the main scene comprises: Each time a pixel value of a pixel in the main scene is read from the first frame buffer, the pixel value of the pixel is copied to a position corresponding to the pixel in the texture resource, the color information of the main scene includes the pixel value of each pixel in the main scene, the image data of the main scene includes the color information of the main scene, and the texture resource is the first resource.

4. The image processing method according to claim 3, characterized in that: The step of reading a pixel value of a pixel point in the main scene from the first frame buffer and copying the pixel value of the pixel point to a position corresponding to the pixel point in the texture resource comprises: The DrawCall function is called, and the DrawCall function is used to read the pixel value of the pixel point from the first frame buffer, and specifies the identifier of the texture resource, so as to copy the pixel value of the pixel point to the position corresponding to the pixel point in the texture resource through the DrawCall function.

5. The image processing method according to claim 2, characterized in that: After drawing the main scene of the first image in the application and before switching the bound frame buffer to the second frame buffer, the image processing method further includes: Switching the bound frame buffer from the first frame buffer to a third frame buffer, the third frame buffer being different from the second frame buffer and the first frame buffer; The copying of the image data of the main scene to the first resource created for the main scene includes: copying the image data of the main scene to the third frame buffer, where the third frame buffer is the first resource.

6. The image processing method according to claim 5, characterized in that: The copying of the image data of the main scene to the third frame buffer comprises: Setting the attribute of the first frame buffer to a read attribute, and setting the attribute of the third frame buffer to a write attribute; The color information of the main scene is read from the first frame buffer, and the color information of the main scene is copied to the third frame buffer, wherein the image data of the main scene includes the color information of the main scene.

7. The image processing method according to claim 6, characterized in that: The step of setting the attribute of the first frame buffer as a read attribute, setting the attribute of the third frame buffer as a write attribute, and reading the color information of the main scene from the first frame buffer and copying the color information of the main scene to the third frame buffer comprises: Calling a glBlitFramebuffer() function, wherein the glBlitFramebuffer() function is used to set the attribute of the first frame buffer as a read attribute, and to set the attribute of the third frame buffer as a write attribute, and the identifier of a color attachment is specified in the glBlitFramebuffer() function, and the identifier of the color attachment is used to point to the color information of the main scene; In response to the glBlitFramebuffer() function, the color information of the main scene is read from the first frame buffer, and the color information of the main scene is copied to the third frame buffer.

8. The image processing method according to claim 4, characterized in that: The copying of the image data of the main scene to the third frame buffer comprises: After performing rasterization processing on the image data of the main scene in the first frame buffer, sampling is performed on the rasterized image data to obtain a sampling result of the image data; The sampling result of the image data is written into the third frame buffer, and the sampling result of the image data is used to reconstruct the main scene.

9. The image processing method according to any one of claims 4 to 8, characterized in that: The switching of the bound frame buffer to the second frame buffer includes: switching the bound frame buffer from the third frame buffer to the first frame buffer, and then switching the bound frame buffer from the first frame buffer to the second frame buffer; or, The switching the bound frame buffer to the second frame buffer includes: switching the bound frame buffer from the third frame buffer to the second frame buffer.

10. The image processing method according to any one of claims 1 to 9, characterized in that: The sending and displaying of the first image, wherein the first image is generated based on the image data of the main scene and the image data of the user interface, comprises: After intercepting the display command sent by the application, calling a super-resolution algorithm to process the image data of the main scene to obtain first image data, wherein the resolution corresponding to the first image data is greater than the resolution corresponding to the image data of the main scene; The first image is sent for display, where the first image is generated based on the first image data and the image data of the user interface, and the resolution corresponding to the image data of the user interface is the same as the resolution corresponding to the first image data.

11. The image processing method according to any one of claims 1 to 9, characterized in that: The sending and displaying of the first image, wherein the first image is generated based on the image data of the main scene and the image data of the user interface, comprises: After intercepting the display command sent by the application, calling the interpolation algorithm to process the image data of the main scene to obtain second image data, wherein the second image data and the image data of the main scene are stored in different texture resources; Sending and displaying the first image, where the first image is generated based on image data of the main scene and image data of the user interface; A second image is displayed, where the second image is generated based on the second image data and the image data of the user interface.

12. An electronic device, characterized in that: The electronic device comprises: one or more processors and memory; The memory is used to store computer program codes, which include computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the image processing method as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed, the image processing method according to any one of claims 1 to 11 is implemented.

Images