Image processing method and electronic device

Abstract

An image processing method and an electronic device, which relate to the technical field of image processing. The method comprises: after an electronic device receives an event of launching a first window and before same receives an event of closing the first window, the first window being a rounded-corner window: the electronic device compositing rounded corners of the first window, and performing processing to obtain a first image frame; and the electronic device not compositing rounded corners of the first window, and performing processing to obtain a second image frame, wherein the first image frame and the second image frame are two different image frames, and each of the first image frame and the second image frame comprises a rounded-corner effect of the first window. In this way, when a rounded-corner window is launched, an electronic device can achieve a rounded-corner effect for some image frames without performing composition, thereby reducing the load of a GPU and improving the usability performance of the electronic device.

Description

Image processing method and electronic device TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, and in particular, to an image processing method and an electronic device. BACKGROUND

[0002] In electronic devices such as mobile phones and tablets, the ability to support multitasking is usually supported. Multitasking refers to running multiple applications in the foreground, so that the user can use multiple applications at the same time. A typical multitasking scenario includes: the electronic device further displays a video window while displaying an application interface, so that the user can watch a video through the video window while using the application. Hereinafter, this typical multitasking scenario is referred to as a video window scenario for convenience.

[0003] In the above video window scenario, the outer contour of the video window is usually a rounded rectangle, and accordingly, in the processing of each frame of image, the electronic device also needs to obtain the rounded effect of the rounded rectangle through a graphics processing unit (GPU), which will increase the load of the GPU and greatly increase the possibility of the electronic device overheating, affecting the use performance of the electronic device.

[0004] SUMMARY

[0005] The present application provides an image processing method and an electronic device, which can reduce the possibility of the electronic device overheating in a multitasking scenario including a rounded window (such as the above video window scenario), and improve the use performance of the electronic device.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides an image processing method applied to an electronic device, the method comprising: receiving an event of starting a first window, the first window being a non-full-screen rounded window, such as a video window 1021 in a picture-in-picture form in FIG. 1 below or a window 2031 in a floating window form in FIG. 2 below. After receiving the event of starting the first window, a first image is displayed, and the first image includes the first window, such as the image frame 404 in FIG. 4 below.

[0008] The first image includes a first layer and a second layer. The first layer has first to-be-displayed content of the first window drawn therein, and the second layer has second to-be-displayed content of other regions except the first window drawn therein. That is, the first layer is a layer for drawing to-be-displayed content of a rounded window, that is, a rounded layer in the following description. For example, the first layer is a video window layer, and correspondingly, the first to-be-displayed content is video content. The second layer is a layer for drawing to-be-displayed content except the rounded window, that is, another layer in the following description. For example, the second layer is a desktop layer, a wallpaper layer, or the like, and correspondingly, the second to-be-displayed content includes desktop icons, wallpaper, or the like.

[0009] In addition, the second layer also has a transparent rounded rectangular region drawn therein. The transparent rounded rectangular region is located at the same position as the display region of the first window, that is, the transparent rounded rectangular region is the same as the display region of the first window. For example, the transparent rounded rectangular regions 4021 and 4022 in FIG. 4 are the same as the display region of the video window in the image frame 404. In the first image, the second layer is superimposed on the upper layer of the first layer. In this way, the first window can be seen through the transparent rounded rectangular region, and thus the first window can be displayed in the first image.

[0010] In summary, by using the solution provided in the present application, after the rounded window is started, the electronic device can not use a GPU to synthesize a rounded effect, but can draw a transparent rounded rectangular region and superimpose the transparent rounded rectangular region on to-be-displayed content of the rounded window to realize the rounded effect. In this way, the load of the GPU can be reduced, and the use performance of the electronic device can be improved.

[0011] In a possible design of the first aspect, before the first image is displayed, the method further includes: generating the first image. In the process of generating the first image, the rounded effect of the first window is not synthesized. That is, the GPU is not called to synthesize the rounded effect.

[0012] In another possible design of the first aspect, in the process of generating the first image, the rounded effect of the first window is not synthesized, including: if a continuous time length during which a display region of the first window in the display screen of the electronic device remains unchanged reaches a first time length (referred to as time length 1 in the following description), it is indicated that the first window remains stable for a long time, such as not moving or not scaling. In this case, in the process of generating the first image, the rounded effect of the first window is not synthesized.

[0013] In practice, in the case of instability of the first window, the rounded corner effect is achieved by replacing the synthesized rounded corner region with a transparent rounded corner region, which usually results in poor effect due to delay. For details, refer to the related description of FIG. 9B below. Based on this, the electronic device can achieve the rounded corner effect by replacing the synthesized rounded corner region with a transparent rounded corner region only in the case of stability of the first window, which can reduce the GPU load while ensuring the rounded corner effect.

[0014] In another possible design of the first aspect, before or after displaying the first image, the method further includes: generating a second image, and displaying the second image, the second image including the first window. In the process of generating the second image, the rounded corner effect of the first window is synthesized.

[0015] That is, after the first window is started, the rounded corner effect is still achieved by using the GPU synthesis in the process of processing some image frames.

[0016] In another possible design of the first aspect, the second image includes a third layer and a fourth layer, the third layer having third to-be-displayed content of the first window drawn therein, and the fourth layer having fourth to-be-displayed content of other regions except the first window drawn therein. That is, the third layer is a layer for drawing to-be-displayed content of the rounded corner window, and the fourth layer is a layer for drawing to-be-displayed content except the rounded corner window. In addition, the fourth layer does not draw the transparent rounded corner region, and the fourth layer is superimposed below the third layer.

[0017] As can be seen, in the case of achieving the rounded corner effect by synthesis, the electronic device does not need to draw the transparent rounded corner region, and needs to superimpose the layer for drawing to-be-displayed content of the rounded corner window in the upper layer, so that the rounded corner window is visible.

[0018] In another possible design of the first aspect, in the process of generating the second image, the rounded corner effect of the first window is synthesized, including: if a continuous time length during which a display area of the first window in the display screen of the electronic device remains unchanged does not reach a first time length, the rounded corner effect of the first window is synthesized in the process of generating the second image.

[0019] That is, if the first window is unstable, the electronic device achieves the rounded corner effect by using the GPU synthesis, so as to ensure the rounded corner effect.

[0020] In a possible design of the first aspect, the generating the first image comprises: drawing or refreshing the first to-be-displayed content in the first layer. It can be understood that if the first window displays video content, the image content can be refreshed into the first layer by refreshing, without the need of drawing. The second to-be-displayed content and the transparent rounded rectangular region are drawn in the second layer. The second layer is superimposed on the upper layer of the first layer to generate the first image. Thus, the rounded effect can be achieved without compositing.

[0021] In a possible design of the first aspect, the drawing the transparent rounded rectangular region in the second layer comprises: based on the layer attribute of the first layer, such as the position, size, rounded radius of the first window, and the like, the transparent rounded rectangular region is drawn in the second layer, so that the position of the transparent rounded rectangular region is the same as that of the display region of the first window. In this way, it can be ensured that the content in the first window can be seen through the transparent rounded rectangular region.

[0022] In a possible design of the first aspect, the drawing the transparent rounded rectangular region in the second layer comprises: if the first indication information (indication 1 in the following) exists, the transparent rounded rectangular region is drawn in the second layer, and the first indication information indicates to draw the transparent rounded rectangular region.

[0023] That is, the electronic device can indicate to draw the transparent rounded rectangular region through the first indication information, so that the electronic device can explicitly indicate to draw the transparent rounded rectangular region through the first indication information.

[0024] In a possible design of the first aspect, the method further comprises:

[0025] The first indication information is generated after receiving the event of starting the first window. In this way, after the rounded window is started, the electronic device can achieve the rounded effect through the transparent rounded rectangular region.

[0026] Alternatively, the first indication information is generated after receiving the event of starting the first window, and the display region of the first window in the display screen of the electronic device remains unchanged for a continuous time length reaching a first time length. In this way, after the rounded window is started and the rounded window remains stable, the electronic device can achieve the rounded effect through the transparent rounded rectangular region.

[0027] In a possible design of the first aspect, before the second layer is superimposed on the upper layer of the first layer to generate the first image, the method further comprises: canceling the rounded effect of the first window, so that the rounded effect of the first window is not composited in the process of generating the first image.

[0028] It can be understood that the rounded corner effect is usually implemented in a synthetic manner. If the rounded corner effect is not implemented in the synthetic manner, the electronic device can cancel the action of synthesizing the rounded corner effect, so as to ensure that the rounded corner effect is not implemented in the synthetic manner.

[0029] In a possible design of the first aspect, before the second layer is superimposed on the upper layer of the first layer, the method further includes: obtaining a first superimposition order (hereinafter referred to as a superimposition order) in which the first layer is on the upper layer of the second layer. The first superimposition order is adjusted to obtain a second superimposition order (hereinafter referred to as an adjusted superimposition order) in which the second layer is on the upper layer of the first layer. The superimposition of the second layer on the upper layer of the first layer includes superimposing the second layer on the upper layer of the first layer based on the second superimposition order.

[0030] It can be understood that when the rounded corner effect is usually implemented in the synthetic manner, the determined superimposition order is the first order. If the rounded corner effect is not implemented in the synthetic manner, the electronic device can adjust the superimposition order, so as to ensure that the first layer is superimposed on the lower layer.

[0031] In a possible design of the first aspect, the electronic device includes an image synthesizer SurfaceFlinger, a drawing module libhwui, a hardware synthesizer HWC, and a graphics processor GPU.

[0032] After receiving the event of starting the first window, the method further includes: the image synthesizer listening to the existence of the rounded corner window, and sending a first notification message (hereinafter referred to as notification 1) to the drawing module. The first notification message includes first indication information and layer attributes of the first layer.

[0033] The generation of the first image includes: the drawing module drawing the second to-be-displayed content on the second layer. The drawing module draws a transparent rounded corner rectangular region on the second layer based on the first notification message. For example, the drawing module determines to draw the transparent rounded corner rectangular region based on the first indication information, and determines the position, size, rounded corner radius, and the like of the transparent rounded corner rectangular region based on the layer attributes. After the transparent rounded corner rectangular region is drawn, the drawing module sends second indication information (hereinafter referred to as indication information 2) to the image synthesizer. After receiving the second indication information, the image synthesizer does not call the GPU to synthesize the rounded corner effect, and calls the HWC to superimpose the second layer on the upper layer of the first layer, to generate the first image. The first to-be-displayed content in the first layer is drawn by the drawing module or refreshed by the image synthesizer.

[0034] In a possible design of the first aspect, the image synthesizer listening to the existence of the rounded corner window, and sending the first notification message to the drawing module, includes:

[0035] The image synthesizer sends a first notification message to the drawing module after listening to the existence of the round-corner window and the first window remaining unchanged in the display area of the display screen of the electronic device for a continuous time period reaching a first time period.

[0036] In a second aspect, the present application also provides an electronic device, which includes a display screen, a memory and one or more processors. The display screen, the memory and the processors are coupled. The memory is configured to store computer program codes, and the computer program codes include computer instructions, which, when executed by the processors, cause the electronic device to perform the method in the first aspect and any possible design thereof.

[0037] In a third aspect, the present application provides a chip system, which is applied to an electronic device including a display screen and a memory. The chip system includes one or more interface circuits and one or more processors. The interface circuits and the processors are interconnected through lines. The interface circuits are configured to receive signals from the memory of the electronic device and send signals to the processors. The signals include computer instructions stored in the memory. When the processors execute the computer instructions, the electronic device performs the method in the first aspect and any possible design thereof.

[0038] In a fourth aspect, the present application provides a computer storage medium, which includes computer instructions. When the computer instructions are run on an electronic device, the electronic device performs the method in the first aspect and any possible design thereof.

[0039] In a fifth aspect, the present application provides a computer program product. When the computer program product is run on a computer, the computer performs the method in the first aspect and any possible design thereof.

[0040] It can be understood that the electronic device of the second aspect, the chip system of the third aspect, the computer storage medium of the fourth aspect and the computer program product of the fifth aspect provided above can achieve the beneficial effects of the first aspect and any possible design thereof, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0041] FIG. 1 is a schematic diagram of an application scenario of an embodiment of the present application;

[0042] FIG. 2 is a schematic diagram of another application scenario of an embodiment of the present application;

[0043] FIG. 3 is a principle diagram of an image processing process provided by an embodiment of the present application;

[0044] FIG. 4 is a principle diagram of another image processing process provided by an embodiment of the present application;

[0045] FIG. 5 is a hardware structure diagram of an electronic device provided by an embodiment of the present application;

[0046] FIG. 6 is a software architecture diagram of an electronic device according to an embodiment of the present application;

[0047] FIG. 7 is a flowchart of an image processing method according to an embodiment of the present application;

[0048] FIG. 8 is a timing interaction diagram of an image processing method according to an embodiment of the present application;

[0049] FIG. 9A is a diagram illustrating adjustment of a video small window according to an embodiment of the present application;

[0050] FIG. 9B is a diagram illustrating a rounded corner effect according to an embodiment of the present application;

[0051] FIG. 10 is a flowchart of another image processing method according to an embodiment of the present application;

[0052] FIG. 11 is a timing interaction diagram of another image processing method according to an embodiment of the present application;

[0053] FIG. 12 is a diagram illustrating closing of a video small window according to an embodiment of the present application;

[0054] FIG. 13 is a flowchart of still another image processing method according to an embodiment of the present application;

[0055] FIG. 14 is a timing interaction diagram of still another image processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing the specific embodiments and are not intended to be limiting on the present application. As used in the specification and the appended claims of the present application, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that "at least one" and "one or more" refer to one or two or more (including two) in the following embodiments of the present application. The term "and / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships; for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects.

[0057] Reference in the specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiment, although it can. The terms "including," "comprising," "having" and variations thereof are meant to encompass the items listed thereafter and equivalents thereof as well as additional items. The terms "connected" and "coupled" are not restricted to direct connections or couplings but include indirect connections or couplings through another component or components. The terms "first," "second," and "third" are used to describe various elements, but can not be used to imply relative importance or to imply a number of such elements. The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations. The term "example" is used in conjunction with a description of a component, aspect, or feature to indicate that the component, aspect, or feature is among a number of possible implementations. It is not intended to imply that the described implementation is preferred or advantageous over other possible implementations.

[0058] In the embodiments of the present application, the words "exemplary" and "for example" are used to mean serving as an example, instance, or illustration. Any implementation described herein as "exemplary" or as an "example" is not necessarily to be construed as preferred or advantageous over other implementations. The terms "exemplary" and "for example" are intended to be used to describe one of a number of possible implementations, as opposed to the preferred or advantageous implementation. It is intended that the terms "exemplary" and "for example" be used to describe a possible implementation, not to imply a preferred or advantageous implementation.

[0059] The embodiments of the present application provide an image processing method, which can be applied to a multi-task operation scene including a rounded corner window, and an image frame including a rounded corner effect of the rounded corner window is obtained.

[0060] The rounded corner window can be a video window, i.e., a video window scene. Taking a mobile phone as an example, referring to FIG. 1, the mobile phone can display an interface 101, which is an application interface of a video player, and a video 1011 is being played in the interface 101. In response to a user's up-swipe operation from the bottom of the interface 101 (as indicated by an arrow in the interface 101), the mobile phone can return to the desktop and display an interface 102, which is the desktop of the mobile phone and includes a video window 1021. The video 1011 is being played in the video window 1021. The video window 1021 is a rounded corner window. The image processing method provided by the embodiments of the present application can be used to obtain an image frame of the interface 102, and the image frame includes the rounded corner effect of the video window 1021.

[0061] The rounded-corner window can be an application floating window. Still taking the electronic device as a mobile phone as an example, referring to FIG. 2, the mobile phone can display an interface 201, which is a desktop of the mobile phone. In response to a left-swiping-and-pausing operation of a user from a right edge of the interface 201 to the left (as indicated by an arrow in the interface 201), the mobile phone can display an interface 202, which is also a desktop of the mobile phone, and the interface 202 includes a pop-up window 2021. The pop-up window 2021 includes application icons of a plurality of shortcut applications, such as an application icon 2022 of a favorite space, an application icon 2023 of an electronic mailbox, an application icon 2024 of a note, and the like. In response to a click operation of a user on an application icon of any shortcut application, the mobile phone can further display an application interface of the any shortcut application in the form of a floating window. Taking the electronic mailbox as an example, in response to a click operation of a user on the application icon 2023 of the electronic mailbox in the interface 202, the mobile phone can display an interface 203. The interface 203 is still a desktop of the mobile phone, and the interface 203 includes an application floating window 2031 of the electronic mailbox, and the application floating window 2031 displays an application interface of the electronic mailbox. The application floating window 2031 is a rounded-corner window. The image processing method provided in the embodiments of the present application can be used to process an image frame of the interface 203, and the image frame includes the rounded-corner effect of the application floating window 2031.

[0062] Of course, the above-mentioned FIG. 1 and FIG. 2 only show two typical rounded-corner windows, and the present application is not limited thereto. Hereinafter, the video window scenario is mainly taken as an example to describe the present application.

[0063] In some schemes, the rounded-corner effect of the rounded-corner window is obtained by the GPU in the electronic device through composition (such as performing rounded-corner clipping). Specifically, the electronic device can obtain (such as by drawing, refreshing, or the like) layer content of a plurality of layers, the plurality of layers including a layer of the rounded-corner window (hereinafter referred to as a rounded-corner layer), perform rounded-corner clipping on the rounded-corner layer by the GPU, superimpose the clipped rounded-corner layer and other layers in the plurality of layers, and obtain an image frame.

[0064] Taking the electronic device as a mobile phone, the multiple layers including the video window layer 301, the desktop layer 302 and the wallpaper layer 303 shown in FIG. 3, and the video window layer 301 as a round-corner layer as an example, the mobile phone can draw desktop icons in the desktop layer 302, draw wallpaper in the wallpaper layer 303, and refresh video content in the video window layer 301. It should be noted that the mobile phone can decode a video file through a codec component (such as media codec) to obtain video frames, and then refresh the video frames into the video window layer 301, without the need to obtain the content in the video window layer 301 through drawing. The mobile phone can perform synthesis processing on the video window layer through a GPU to obtain a video window layer 304 with a round-corner effect. Then, the mobile phone superimposes the video window layer 304 with the round-corner effect, the desktop layer 302 with the desktop icons drawn thereon, and the wallpaper layer 303 with the desktop wallpaper drawn thereon in the order from top to bottom, that is, superimposes the video window layer 304 with the round-corner effect on the uppermost layer, to obtain an image frame 305. The image frame 305 includes the video window 3031 with the round-corner effect.

[0065] In the above scheme, the electronic device completes the synthesis of the round-corner effect through the GPU, which increases the load of the GPU in the image processing process, greatly increases the possibility of heating of the electronic device, and affects the use performance of the electronic device.

[0066] Therefore, in the embodiments of the present application, the electronic device can further draw a transparent round-corner rectangular region (imagined as a hole) in other layers after the layer content of the other layers is drawn. The other layers include multiple layers in which content needs to be drawn in the image frame, except for the round-corner layer, such as the desktop layer 303 and the wallpaper layer 304 in FIG. 3. The round-corner rectangular region matches the position, size and round-corner radius of the round-corner window. The electronic device superimposes the other layers with the transparent round-corner rectangular region drawn thereon on the round-corner layer, that is, the round-corner layer is located at the bottom layer, to obtain an image frame. In the image frame, the content of the bottom round-corner layer is visible through the transparent round-corner rectangular region in the other layers superimposed on the upper layer, so that the round-corner effect is presented.

[0067] Still taking the electronic device is a mobile phone, the plurality of layers include the video window layer 401, the desktop layer 402 and the wallpaper layer 403 shown in FIG. 4, and the video window layer 401 is a round-corner layer as an example, the desktop layer 402 and the wallpaper layer 403 are other layers. The mobile phone can draw a transparent round-corner rectangular region 4021 in the desktop layer 402 on which desktop icons are drawn, and draw a transparent round-corner rectangular region 4022 in the wallpaper layer 403 on which a desktop wallpaper is drawn. The mobile phone superimposes the desktop layer 402, the wallpaper layer 403 and the video window layer 401 in the order from top to bottom, that is, the video window layer 401 is at the bottom layer, and an image frame 404 can be obtained, which is the same as the image frame 305 in the foregoing FIG. 3 and has a round-corner effect.

[0068] Therefore, in the embodiments of the present application, the electronic device can replace the GPU synthesis manner by drawing a transparent round-corner rectangular region and superimposing a round-corner layer on the bottom layer (hereinafter referred to as a hole digging manner), to realize the presentation of the round-corner effect. In this way, the round-corner effect does not need to be realized by GPU synthesis, which can reduce the load of the GPU in the image processing process, reduce the possibility of the electronic device from generating a heating phenomenon, and improve the use performance of the electronic device, such as making the multi-task operation can be smoothly responded.

[0069] At this point, it should be noted that the mobile phone usually also draws a transparent round-corner rectangular region in other layers through a GPU. In practice, the power consumption of the GPU for drawing a transparent round-corner rectangular region is much lower than that of the GPU for synthesizing a round-corner effect. Therefore, the electronic device can reduce the load of the GPU by drawing a transparent round-corner rectangular region and superimposing a round-corner layer on the bottom layer.

[0070] Exemplarily, the electronic device described above can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device and / or a smart city device, etc. The specific form of the electronic device is not specially limited in the embodiments of the present application.

[0071] Referring to FIG. 5, a hardware structure diagram of an electronic device is shown. As shown in FIG. 5, taking the electronic device as a mobile phone for example, the electronic device can include a processor 210, an external storage interface 220, an internal storage (referred to as memory) 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headset jack 270D, a sensor module 280, a key 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.

[0072] It can be understood that the structure shown in the embodiment does not constitute a specific limitation on the mobile phone. In other embodiments, the mobile phone can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0073] The processor 210 can include one or more processing units, such as: an application processor (AP), a modem processor, a central processing unit (CPU), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or integrated in one or more processors.

[0074] The charging management module 240 is configured to receive charging input from a charger. The power management module 241 is configured to connect the battery 242 and the charging management module 240 to the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to power the processor 210, the internal storage 221, the display screen 294, the camera 293, and the wireless communication module 260, etc. The power management module 241 can also be configured to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), etc.

[0075] The wireless communication function of a mobile phone can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor, and baseband processor. Mobile communication module 250 can provide wireless communication solutions for mobile phones, including 2G / 3G / 4G / 5G. Wireless communication module 260 can provide wireless communication solutions for mobile phones, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), Bluetooth Low Energy (BLE), ultra-wideband (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0076] The mobile phone can achieve display functions through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0077] Mobile phones can achieve camera functions through the camera 293, ISP, video codec, GPU, display 294, application processor (AP), neural network processor (NPU), etc.

[0078] Mobile phones can perform audio functions, such as music playback and recording, through an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, and an application processor.

[0079] The software system of the aforementioned electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered architecture based on Android. TM Taking a system as an example, this section illustrates the software structure of an electronic device. A layered architecture divides the software system of an electronic device into several layers, each with a clear role and function, and the layers communicate with each other through software interfaces.

[0080] Referring to FIG. 6, the software and hardware architecture of the electronic device can include an applications (APP) layer, an application framework layer, a Native layer, a kernel layer, and a hardware layer, for example, in the case of a mobile phone.

[0081] The applications in the applications layer can include a launcher, an email, a video player, a memo, an alarm, a settings, a game application, and the like. The mobile phone can run a plurality of the above applications in the foreground, and one of the applications can include a rounded window.

[0082] For example, in the scenario shown in FIG. 1 (a specific video window scenario), the mobile phone can run a launcher and a video player in the foreground, and the video player can include a rounded window.

[0083] For example, in the scenario shown in FIG. 2, the mobile phone can run a launcher and an email application in the foreground, and the email application can include a rounded window.

[0084] The application framework layer provides an application programming interface (API) and a programming framework for the applications layer.

[0085] The application framework layer can include a plurality of system servers, such as a window management service (WMS) and a display management service (DMS), and the like.

[0086] The WMS can be used for window management, window animation management, surface management, and as a transit station for an input system. The DMS can be used for refreshing of an application view, management of a frame buffer, and as a passage for dual-screen display to implement switching of a display screen.

[0087] The application framework layer can include a view system API and a graphics system API. The view system API can be used for layout calculation of an application and execution of drawing instructions, etc. For example, the layout calculation includes calculation of positions of various icons in a desktop. Further, the view system API includes classes such as view, view group, surface view, etc. The graphics system API can be used for setting of related attributes of graphics. Further, the graphics system API includes classes such as canvas, render node, paint, etc.

[0088] The native layer provides various services for upper layers (such as the application framework layer). For example, the native layer includes a surface flinger, a drawing module (libhwui), a capability library of a skia (an open source 2D graphics library) API (libskia), an API support module of OpenGL (libGLES_XX), a layer management module (libgui), etc. It should be noted that the surface flinger can also be referred to as a surface drawer, a surface drawing module, an image composition processing service, etc., which are not specifically limited in the present application.

[0089] The surface flinger can be used for refresh rate control, control of image composition, etc. For example, the surface flinger can assign a composition task to a hardware composer (HWC) or a GPU by decision.

[0090] In some embodiments, the surface flinger can be used to monitor a round-corner window, such as monitoring whether there is a round-corner window, and controlling the switch to be turned on or off based on the monitoring result.

[0091] In a specific implementation, the surface flinger can turn on the switch when it is monitored that there is a round-corner window, such as when it is monitored that there is a video window. The surface flinger can turn off the switch when it is monitored that there is no round-corner window.

[0092] In another specific implementation, SurfaceFlinger can turn on the switch when it is detected that there is a rounded corner window and the rounded corner window is stable. SurfaceFlinger can turn off the switch when it is detected that there is a rounded corner window but the rounded corner window is not stable (e.g., the position and size of the rounded corner window are changing) or when it is detected that there is no rounded corner window.

[0093] In some embodiments, SurfaceFlinger can be configured to cancel the rounded corner composition when the switch is turned on, and to superimpose the rounded corner layer on the bottom layer during the composition. SurfaceFlinger can be configured to perform the rounded corner composition by calling GPU and superimpose the rounded corner layer on the top layer when the switch is turned off and there is a rounded corner window.

[0094] Further, a monitoring service can be started in SurfaceFlinger to monitor the rounded corner window.

[0095] In some embodiments, libhwui can be configured to draw images. libhwui can also be configured to achieve the rounded corner effect by digging holes when the switch is turned on. When the switch is turned off, libhwui can be configured to normally draw the content of the layer without drawing the transparent rounded corner rectangle region.

[0096] For example, SurfaceFlinger can send notification 1 to libhwui to indicate that the rounded corner effect is achieved by drawing when the switch is turned on. libhwui can be configured to draw the transparent rounded corner rectangle region during the drawing of other layers after receiving notification 1.

[0097] For another example, SurfaceFlinger can send notification 2 to libhwui to indicate that the rounded corner effect is not achieved by drawing when the switch is turned off. libhwui can be configured to not draw the transparent rounded corner rectangle region during the drawing of other layers after receiving notification 2.

[0098] It can be understood that each application can complete drawing through a corresponding render thread (RenderThread). RenderThread is achieved by calling the value in libhwui. That is, each application has a corresponding libhwui to achieve the drawing of the application. For example, a desktop application has libhwui 1, a wallpaper application has libhwui 2, and a video player has libhwui 3.

[0099] Hereinafter, for simplifying the description, the action of the RenderThread is represented as the action of the libhwui, and the drawing of the RenderThread is represented as the drawing of the libhwui.

[0100] The libskia can be used to submit the drawing instruction of the libhwui to the GPU to complete the drawing processing. The libGLES_XX can be used to convert the drawing instruction submitted by the libskia into the GLES format, so as to facilitate the GPU to execute. The libgui can be used for the rotation of the layer and the layer attribute, and the libhwui can submit the layer and the layer attribute to the SurfaceFlinger through the libgui.

[0101] The kernel layer includes the driver for driving the hardware work, such as the display driver and the GPU driver. The display driver can drive the display screen to display the image, such as the image frame including the round-corner window. The GPU driver can drive the GPU to work, such as driving the GPU to perform the image composition.

[0102] The image processing method provided by the embodiment of the present application can be executed in the electronic device with the above-mentioned hardware and software structure. Hereinafter, except for the timing interaction diagrams of FIG. 8, FIG. 11 and FIG. 14, the execution subject is the electronic device, and the image processing method provided by the embodiment of the present application is described, and the execution subject will not be described one by one hereinafter.

[0103] The image processing method provided by the embodiment of the present application is described below in three cases.

[0104] Case 1: the round-corner window is not started; case 2: the round-corner window is started; and case 3: the round-corner window is closed.

[0105] It should be noted that in the process of one-time operation (such as from power-on to power-off), only case 1 can occur, or any one of the following various combinations can occur: the combination of case 1 and case 2; the combination of case 1, case 2 and case 3.

[0106] Case 1: the round-corner window is not started. For example, when the interfaces 101, 201 and 202 are displayed, the round-corner window has not been started, and accordingly, when the image frames are processed, the round-corner effect does not need to be processed.

[0107] For example, the round-corner window is started in response to the event 1. The event 1 is used to trigger the start of the round-corner window, such as the event 1 being the up-slip operation event of the user from the bottom to the top of the video playing interface, or the event 1 being the click operation event of the user on any shortcut application icon in the front pop-up window 2021. That is, before the event 1 is received, it belongs to case 1 of not starting the round-corner window.

[0108] In case 1, during the processing of each image frame, no round-corner window is detected, it can be determined that the image processing process does not include implementing the round-corner effect, and the GPU does not need to implement the round-corner effect by composition, and there is no problem of increasing the load of the GPU due to implementing the round-corner effect. Correspondingly, for case 1, the image frame can be obtained through image drawing and image composition processing. Among them, neither the image drawing process nor the image composition process includes the step of implementing the round-corner effect, such as the image drawing process does not include the step of drawing a transparent round-corner rectangular region, and the image composition process does not include the step of synthesizing the round-corner effect.

[0109] For the specific implementation of the image processing process in case 1, please refer to the description in the related art, which is not limited herein.

[0110] Case 2, start the round-corner window. For example, the video window 1021 in the aforementioned interface 102 is a round-corner window, and the floating window 2031 in the interface 203 is a round-corner window, which all belong to the case of starting the round-corner window.

[0111] For example, after receiving event 1, the round-corner window can be started.

[0112] In case 2, since the round-corner window is started, the image processing process includes implementing the round-corner effect. The round-corner effect can be implemented by the hole digging method instead of the GPU composition method, thereby reducing the load of the GPU and improving the use performance.

[0113] In some embodiments, referring to FIG. 7, the image processing process includes:

[0114] S701, after receiving event 1, during the process of processing an image frame, it is detected that there is a round-corner window.

[0115] For event 1, please refer to the related description in the foregoing, which will not be repeated here.

[0116] Among them, an image frame can be any image frame displayed after starting the round-corner window in response to event 1 and before closing the round-corner window (as described below in response to event 3). For example, when event 1 is received, the image frame displayed in the aforementioned interface 101, then an image frame can be an image frame in the interface 102.

[0117] Event 1 is used to trigger the start of the round-corner window, and in response to event 1, the round-corner window is started, so after event 1, it can be detected that there is a round-corner window.

[0118] It can be understood that if the round-corner window is started, a layer corresponding to the round-corner window, i.e., the round-corner layer in the foregoing, will be created for drawing (or refreshing) the content displayed in the round-corner window.

[0119] The obvious difference between the rounded-corner layer and other layers is that the rounded-corner layer has a corner radius attribute. Based on this, in one specific implementation, it can be determined whether there is a rounded-corner window by identifying whether a layer included in an image frame (e.g., a layer participating in composition) has a corner radius attribute. If any layer included in the image frame has a corner radius attribute, it is determined that there is a rounded-corner window. If none of the layers included in the image frame has a corner radius attribute, it is determined that there is no rounded-corner window.

[0120] Further, the phone can also record a white list, which includes the identifiers (e.g., names) of layers that can be used to draw specific content. The specific content includes content that can be presented in a rounded-corner window. For example, the specific content includes video content, interface content of an email, etc. The layers included in the image frame can be matched with the white list to determine whether there is a rounded-corner window.

[0121] The layer attributes of the layers included in the image frame can be obtained, which include a name attribute and a corner radius attribute. If the layer identifier is in the white list, it indicates that the layer can be used to draw (or refresh) content displayed in the rounded-corner window, i.e., it can be a rounded-corner layer. Moreover, if it has a corner radius attribute, it is determined that there is a rounded-corner window. If the layer identifier is not in the white list, it indicates that the layer is not used to draw (or refresh) content displayed in the rounded-corner window, i.e., it is not a rounded-corner layer, and it is determined that there is no rounded-corner window.

[0122] Taking an example in which the white list includes the name of a layer that refreshes video content, when displaying the aforementioned interface 101, in response to event 1, a video window layer that is used to refresh the video content in the video window 1021 shown in the interface 102 can be created, i.e., the video window layer is a layer that draws video content. The video window layer is matched with the white list, it is determined that the name attribute of the video window layer is in the white list, and it is determined that the video window layer has a corner radius attribute, and it is determined that there is a rounded-corner window.

[0123] In this way, it can be accurately determined whether there is a rounded-corner window through both the white list and the layer attributes.

[0124] It should be noted that one image frame includes one or more layers, such as a video window layer, a desktop layer, a wallpaper layer, etc. If one of the layers has a corner radius attribute or is in the white list and has a corner radius attribute, it is determined that there is a rounded-corner window.

[0125] S702, draw / refresh multiple layers. When drawing other layers, a transparent rounded-corner rectangular region is drawn.

[0126] The multiple layers refer to layers included in the image frame, and the other layers refer to layers in the image frame other than the rounded corner layer, such as the desktop layer and the wallpaper layer in FIG. 4.

[0127] For the rounded corner layer, the content displayed in the rounded corner window is drawn / refreshed in the rounded corner layer, that is, the layer content of the rounded corner layer is the content displayed in the rounded corner window, such as the video content in the video window. It should be noted that, in general, the layer content of the rounded corner layer can be obtained by drawing in the rounded corner layer, and for video content, the layer content of the rounded corner layer can also be obtained by decoding and refreshing. Hereinafter, the way of refreshing the rounded corner layer (such as the video window layer) will be mainly taken as an example for illustration.

[0128] After listening to the existence of the rounded corner window, for the other layers, not only the corresponding layer content is drawn in the other layers, such as the desktop icons are drawn in the desktop layer and the wallpaper is drawn in the wallpaper layer, but also a transparent rounded rectangle area is drawn in the other layers.

[0129] In some embodiments, the transparent rounded rectangle area can be drawn according to the layer attributes of the rounded corner layer, so that the rounded rectangle area is consistent with the rounded corner window, such as the same position, size, etc. After the image frame is obtained by superimposing the subsequent layers, the complete rounded corner window and the content presented therein can be presented through the transparent rounded rectangle area.

[0130] The layer attributes of the rounded corner layer include the position information of the rounded corner window in the display screen, the size of the rounded corner window, the radius of the rounded corner window, etc. so that the position, size, and radius of the rounded corner rectangle area in each other layer can be calculated based on the layer attributes of the rounded corner layer, thereby making the rounded rectangle area consistent with the rounded corner window.

[0131] Further, the position of the rounded rectangle area in each other layer can also be calculated in combination with the state information of the mobile phone. The state information includes the horizontal screen state and the vertical screen state, or the state information includes the rotation angle of the display screen. It can be understood that the rotation angle of the display screen can be changed to switch between the horizontal screen state and the vertical screen state. That is, the rotation angle corresponds to the horizontal screen state and the vertical screen state, such as when the rotation angle is less than an angle threshold, the vertical screen state is in, and when the rotation angle is greater than the angle threshold, the horizontal screen state is switched to. When in the horizontal screen state and the vertical screen state, the display direction of the interface in the display screen changes, and accordingly, the position of the rounded rectangle area can also change. The position of the rounded rectangle area in each other layer is calculated in combination with the state information, so that the calculated position matches the state of the mobile phone.

[0132] Further, the position of each other layer in the display screen can be calculated based on the length-width information of the display screen, the zooming scale information of each other layer, and the like. On this basis, the position of the rounded rectangular region in each other layer is calculated based on the layer attribute of the rounded layer, so that the position of the rounded rectangular region matches the position of the other layers in the display screen.

[0133] In S703, the rounded layer is superimposed on the bottommost layer to obtain an image frame, and the rounded window in the image frame has a rounded effect.

[0134] In S703, the rounded synthesis can be cancelled, and accordingly, the GPU synthesis for obtaining the rounded effect is not needed, which can reduce the load of the GPU.

[0135] Further, the rounded layer is superimposed on the bottommost layer, which is equivalent to changing the superimposition order from being superimposed on the uppermost layer to being superimposed on the bottommost layer. In this way, the rounded effect can be presented through the boundary of the rounded rectangular region.

[0136] In S704, the image frame is displayed, so that the rounded effect can be presented.

[0137] The image processing process of case 2 will be further introduced below in combination with the hardware and software structure of a mobile phone and taking the scenario shown in FIG. 4 as an example, i.e., the rounded layer is a video small window layer, and the other layers are a desktop layer and a wallpaper layer. Specifically, as shown in FIG. 8, the image processing process includes the following steps.

[0138] In S801, in response to the booting initialization, the surfaceflinger starts to monitor the rounded window.

[0139] When the mobile phone is booted, the surfaceflinger is also initialized. When the surfaceflinger is initialized, the monitoring of the rounded window can be started, so that the starting and closing of the rounded window after the booting can be monitored.

[0140] In some embodiments, the surfaceflinger can start to monitor the rounded window by starting a monitoring service.

[0141] After the surfaceflinger starts to monitor the rounded window, the surfaceflinger can monitor the existence of the rounded window after event 1.

[0142] In S802, in response to event 1, the desktop application sends a creation request of the desktop layer to the WMS.

[0143] It should be understood that in the scenario of FIG. 4, event 1 is an upward swiping operation event of the user from the bottom of the video playing interface.

[0144] In response to event 1, the desktop application determines that the desktop content needs to be displayed, sends a creation request of the desktop layer to the WMS for requesting creation of the desktop layer for drawing the desktop content (e.g. desktop icons).

[0145] S803, in response to event 1, the wallpaper application sends a creation request of the wallpaper layer to the WMS.

[0146] In response to event 1, the wallpaper application determines that the wallpaper needs to be displayed, sends a creation request of the wallpaper layer to the WMS for requesting creation of the wallpaper layer for drawing the wallpaper.

[0147] S804, in response to event 1, the video player sends a creation request of the video window layer to the WMS.

[0148] In response to event 1, the video player determines that the video needs to be played in the video window, sends a creation request of the video window layer to the WMS for requesting creation of the video window layer for refreshing the video content.

[0149] S805, the WMS creates the desktop layer, the wallpaper layer and the video window layer, and adds layer attributes to each layer.

[0150] It can be understood that the WMS can add layer attributes based on the creation request of each application, such as adding attributes of position, size, etc. to the video window layer, and also adding a rounded corner attribute.

[0151] It should be noted that the WMS can create each layer synchronously in the surfaceflinger and add layer attributes.

[0152] Through the above S802-S805, each layer included in an image frame can be created.

[0153] After sending the corresponding layer creation request, the desktop application, the wallpaper application and the video player can initialize the RenderThread for setting the corresponding layer, as shown in S806 and S809.

[0154] S806, the desktop application initializes the RenderThread for setting the desktop layer.

[0155] For example, the desktop application can initialize the RenderThread of the desktop application to set the drawing environment parameters of the desktop layer. For example, the RenderThread sets the drawing environment parameters through a SurfaceControl, which is a value held by the WMS, libhwui, etc. corresponding to a layer in the surfaceflinger and can be used as a unique identifier of the layer to interact with the layer in the surfaceflinger, including setting the layer attributes. Based on this, the libhwui1 can identify the layer content of the desktop layer to be drawn when the RenderThread sets the drawing environment parameters of the desktop layer.

[0156] After detecting that the RenderThread sets the desktop layer, the libhwui1 can also perform the following S807 and S808 so that the subsequent libhwui1 determines whether to dig a hole and whether to return an indication of completing the hole digging.

[0157] S807, the libhwui1 registers a listening callback of the desktop layer to the surfaceflinger.

[0158] For example, the libhwui1 can register the listening callback to the surfaceflinger through a binder (a cross-process communication mechanism).

[0159] By registering the listening callback, the libhwui1 can subsequently obtain an indication of whether to dig a hole.

[0160] Further, in the case of receiving the indication of digging a hole, through the listening callback, the libhwui1 can also obtain the layer attributes of the rounded corner layer, the state information of the mobile phone, etc. to draw a rounded rectangular region consistent with the rounded corner window.

[0161] S808, the surfaceflinger saves a callback function in the layer attributes of the desktop layer.

[0162] For the role of the callback function saved in the layer attributes of the desktop layer, please refer to the description of S814 below, which will not be described in detail here.

[0163] S809, the wallpaper application initializes the RenderThread to set the desktop layer.

[0164] Exemplarily, the wallpaper application can also initialize the RenderThread of the wallpaper application to set the drawing environment parameters of the wallpaper layer. Based on this, the libhwui2 can identify the layer content of the wallpaper layer to be drawn when the RenderThread sets the drawing environment parameters of the wallpaper layer.

[0165] After detecting that the RenderThread sets the desktop layer, the libhwui2 can also perform the following S810 and S811 so that the subsequent libhwui2 determines whether to dig a hole.

[0166] S810, the libhwui2 registers a listening callback of the desktop layer to the surfaceflinger.

[0167] It can be understood that registering a listening callback means registering a listener and a callback function.

[0168] Through the listening callback, the libhwui2 can obtain an indication of whether to dig a hole.

[0169] Further, in the case of receiving the indication of digging a hole, through the listening callback, the libhwui2 can also obtain the layer attribute of the rounded corner layer, the state information of the mobile phone, etc., for drawing a rounded rectangular area consistent with the rounded window.

[0170] S811, the surfaceflinger saves the callback function in the layer attribute of the desktop layer.

[0171] For the role of the callback function saved in the layer attribute of the wallpaper layer, please refer to the description of S815 below, which will not be described in detail here.

[0172] At this point, the following points need to be explained:

[0173] First, the execution order of S806-S808 and S809-S811 is not limited to that shown in FIG. 8. Exemplarily, the execution order of S806 and S809 is not sequential, and S809 can be executed first, and S806 can be executed later, or S806 and S809 can be executed simultaneously, and the corresponding libhwui triggers the execution of the subsequent steps after detecting the setting of the corresponding layer.

[0174] Second, S806-S811 only shows the process of setting the desktop layer and the wallpaper layer and triggering the subsequent registration of the listening callback. In practice, the video player can also initialize the RenderThread to set the video small window layer.

[0175] In a specific implementation, after detecting the setting of the video window layer, the libhwui 3 of the video player can also trigger the registration of the monitoring callback, but the surfaceflinger can identify that the video window layer is a rounded corner layer, so even if the monitoring callback of the video window layer is registered, the surfaceflinger will not feed back the indication of whether to dig holes to the libhwui 3, so that the libhwui will not draw the transparent rounded rectangular area in the video window layer.

[0176] In another specific implementation, after detecting the setting of the video window layer, the libhwui 3 of the video player can identify that the video window layer is a rounded corner layer, and can not trigger the registration of the monitoring callback. In this way, the surfaceflinger will not feed back the indication of whether to dig holes to the linbhwui 3, so that the libhwui will not draw the transparent rounded rectangular area in the video window layer. Hereinafter, this implementation will be mainly described.

[0177] Thirdly, after the libhwui registers the monitoring callback of a certain layer to the surfaceflinger, the libhwui does not need to register again when detecting the setting of the certain layer again. That is, during the use of the mobile phone from booting to shutting down, the libhwui only registers the monitoring callback of the certain layer to the surfaceflinger once. Therefore, the above S807-S808 and S810-S811 may not be executed in the flow shown in FIG. 8, but the monitoring callback may be registered before the flow in FIG. 8, such as when the desktop is entered for the first time after booting, the monitoring callback of the desktop layer and the monitoring callback of the wallpaper layer are registered. In short, in the flow of FIG. 8, the above S807-S808 and S810-S811 are not necessary, and are most likely not present.

[0178] S812, the surfaceflinger listens to the start of the rounded corner layer based on the layer attribute of the video window layer.

[0179] The surfaceflinger listens to whether the rounded corner layer is included in the layers participating in the synthesis in the synthesis information detection stage of each frame. For example, the surfaceflinger detects that the name of the video window layer is in the whitelist, and the video window layer has a rounded corner attribute, so it is listened that there is a rounded corner window.

[0180] S813, the surfaceflinger generates a notification message 1, and the notification message 1 includes the layer attribute of the video window layer and the indication information 1.

[0181] The indication information 1 is used to indicate the hole digging, that is, to open the round corner effect by drawing a transparent round corner rectangle region.

[0182] The layer attribute of the video small window layer in the notification message 1 can be used to calculate the position, size, round corner radius and the like of the round corner rectangle region.

[0183] S814, the surfaceflinger distributes the notification message 1 to the libhwui 1 based on the callback function in the layer attribute of the desktop layer.

[0184] Since the callback function is saved in the layer attribute of the desktop layer, it indicates that the desktop layer registers the listening callback, and the surfaceflinger can distribute the notification message 1 to the libhwui 1 which draws the desktop layer.

[0185] S815, the surfaceflinger distributes the notification message 1 to the libhwui 2 based on the callback function in the layer attribute of the wallpaper layer.

[0186] Since the callback function is saved in the layer attribute of the wallpaper layer, it indicates that the wallpaper layer registers the listening callback, and the surfaceflinger can distribute the notification message 1 to the libhwui 2 which draws the wallpaper layer.

[0187] That is, the surfaceflinger can distribute the notification message 1 to the libhwui corresponding to each layer which registers the listening callback, to notify the libhwui to draw a transparent round corner rectangle region in the layer drawing process.

[0188] It can be understood that when the application needs to draw a new image frame, the content to be drawn can be sent to the libhwui, so that the libhwui can draw the content in the corresponding layer. Details are shown in S816 and S819.

[0189] S816, the desktop application sends the content 1 to be drawn to the libhwui 1.

[0190] For example, the content 1 is a desktop icon.

[0191] S817, the libhwui 1 draws the content 1 in the desktop layer, and draws a round corner rectangle region in the desktop layer according to the notification message 1.

[0192] In a normal case, even if there is a round corner window in the image frame, the drawing stage mainly completes the drawing of the layer content, and does not need to realize the round corner effect in the drawing stage.

[0193] In the embodiment of the present application, after receiving the content 1 to be drawn, the libhwui 1 can draw the content 1 in the desktop layer. In addition, the libhwui 1 also receives the notification message 1, and then, after the libhwui 1 finishes drawing the content 1 in the desktop layer, the libhwui 1 can determine to draw the transparent rounded rectangular region based on the indication information 1 in the notification message 1, and calculate the position, size, and rounded radius of the rounded rectangular region based on the layer attribute of the video window layer in the notification message 1, and draw the rounded rectangular region.

[0194] Of course, the libhwui 1 can also calculate the position, size, and rounded radius of the rounded rectangular region based on more information, which can be carried in the notification message 1, such as the state information of the mobile phone, the scaling ratio of the desktop layer, etc., or the information can also be obtained by the libhwui 1 from other modules, such as the length and width information of the display screen obtained from the memory. The present application does not make specific limitation thereto.

[0195] S818, the libhwui 1 feeds back the indication information 2 after drawing is completed.

[0196] The indication information 2 is used to indicate that the hole digging is completed.

[0197] The drawing completion includes completion of drawing the content 1 and the transparent rounded rectangular region.

[0198] For example, the libhwui 1 can feed back the indication information 2 indicating that the hole digging is completed to the surfaceflinger through the surfacecontrol.

[0199] S819, the desktop application sends the content 2 to be drawn to the libhwui 2.

[0200] For example, the content 2 is a wallpaper.

[0201] S820, the libhwui 2 draws the content 2 in the wallpaper layer, and draws the rounded rectangular region in the wallpaper layer according to the notification message 1.

[0202] As described above in S817, the libhwui 2 can determine to draw the transparent rounded rectangular region based on the indication information 1 in the notification message 1, and calculate the position, size, and rounded radius of the rounded rectangular region based on the layer attribute of the video window layer in the notification message 1, and draw the rounded rectangular region.

[0203] S821, the libhwui 2 feeds back the indication information 2 after drawing is completed.

[0204] The drawing completion includes completion of drawing the content 2 and the transparent rounded rectangular region.

[0205] Exemplarily, libhwui 2 can feed back the indication information 2 of completing digging holes to surfaceflinger through surfacecontrol.

[0206] At this point, the following points need to be explained:

[0207] First, generally, each application can send the content to be drawn to the corresponding libhwui to realize drawing. Based on the foregoing description, the video content in the video window can be obtained by decoding and refreshing, and the video player can realize drawing of the video content in the video window layer without sending the video content to the corresponding libhwui 3.

[0208] Second, if other layers are static, that is, the layer content is the same as the previous image frame, the corresponding application will not repeatedly send the content to be drawn. Correspondingly, libhwui also does not need to repeatedly draw the layer content.

[0209] S822A, surfaceflinger cancels the round corner composition after receiving the indication information 2 corresponding to the desktop layer and the wallpaper layer.

[0210] In the scenario shown in FIG. 4, after event 1, the image frame includes the desktop layer, the wallpaper layer and the video window layer, and the video window layer is a round corner layer, and there is no need to draw a transparent round corner rectangular area. Therefore, after surfaceflinger receives the indication information 2 corresponding to the desktop layer and the wallpaper layer, it is indicated that the indication information 2 corresponding to all other layers is received, that is, all other layers have completed digging holes. Surfaceflinger can execute a set of processes cooperating with the transparent round corner rectangular area, including canceling the round corner composition in S822A, refreshing the video window layer, and S823-S825, to realize the round corner effect through the round corner rectangular area.

[0211] Exemplarily, after surfaceflinger receives the indication information 2 of libhwui 1, it can save the indication information 2 to the layer attribute of the desktop layer, and after receiving the indication information 2 of libhwui 2, it can save the indication information 2 to the layer attribute of the wallpaper layer. Surfaceflinger can compare the layer in which the indication information 2 is saved with other layers in the image frame, and if the layer attribute of other layers all saves the indication information 2, it is indicated that the indication information 2 corresponding to all other layers is received.

[0212] In general, the rounded corner effect is achieved by composition. In the embodiment of the present application, after receiving the indication information 2 corresponding to all other layers, it is indicated that the rounded corner effect can be achieved by a transparent rounded rectangular area, and the surfaceflinger cancels the rounded corner composition.

[0213] S822B, the surfaceflinger refreshes the video content of the video window layer.

[0214] The surfaceflinger can refresh the layer content, i.e., the video content, of the video window layer, so that the video window layer can present the latest video frame. For example, the codec component can send the video frame to the surfaceflinger after decoding the latest video frame, and the surfaceflinger can refresh the video window layer. It should be noted that the timing of the surfaceflinger refreshing the video window layer is not limited to that shown in FIG. 8. For example, the codec component can decode a video frame according to the needs of the video player and send it to the surfaceflinger, and the surfaceflinger can refresh the video window layer after receiving the video frame. Of course, in a plurality of consecutive image frames, the video content displayed in the video window can be static, such as after pausing the play, the video content does not change and is static. In this case, the surfaceflinger does not need to refresh the video window layer every frame. That is, the action of refreshing the video window layer is not necessary every frame.

[0215] S823, the surfaceflinger obtains the superimposition order of the desktop layer, the wallpaper layer, and the video window layer, wherein the video window layer is located at the topmost layer.

[0216] S824, the surfaceflinger adjusts the video window layer from the topmost layer to the bottommost layer to obtain an adjusted superimposition order.

[0217] In this way, the surfaceflinger can achieve superimposing the video window layer at the bottommost layer, so that the rounded corner effect can be achieved by the outline of the rounded rectangular area.

[0218] S825, the surfaceflinger composites the desktop layer, the wallpaper layer, and the video window layer according to the adjusted superimposition order to obtain an image frame.

[0219] Since the rounded corner composition is canceled, the surfaceflinger does not call the GPU to crop the video window layer to achieve the rounded corner effect, thereby reducing the load of the GPU. For example, the surfaceflinger can call the HWC to complete layer superimposition to obtain an image frame.

[0220] In the above S816-S825, the case that all other layers have completed the hole digging is described. In practice, it is also possible that at least one of the plurality of other layers has not completed the hole digging, and accordingly, the libhwui does not feed back the indication information 2 corresponding to the at least one layer to the surfaceflinger. The surfaceflinger can determine that the indication information 2 corresponding to all other layers has not been received, and accordingly, the rounded corner synthesis is not cancelled. Subsequently, the surfaceflinger can not perform S824, and in S825, the surfaceflinger synthesizes the desktop layer, the wallpaper layer and the video window layer in the unadjusted superposition order to obtain an image frame, wherein the surfaceflinger calls the GPU to complete the rounded corner synthesis, so that the video window in the obtained image frame has a rounded corner effect.

[0221] S826, the surfaceflinger sends the image frame to the display screen.

[0222] S827, the display screen displays the image frame.

[0223] For example, the display screen displays the next image frame after each refresh.

[0224] The above embodiments of FIG. 7 and FIG. 8 describe the processing of one image frame after receiving the event 1. In practice, the processing of each image frame can be processed in a similar manner before the rounded corner window is closed, so that the rounded corner effect can be realized in each image frame. That is, the mobile phone can listen for the existence of the rounded corner window during the processing of each image frame, and if the existence of the rounded corner window is listened, a transparent rounded corner rectangular region is drawn when each other layer is drawn.

[0225] In a specific implementation, in the flow shown in FIG. 8, S812-S827 can be executed in a loop, so that the load of the GPU can be reduced in each image frame. After the existence of the rounded corner rectangular region is listened in S812, the surfaceflinger can open the switch and generate the notification message 1. After the existence of the rounded corner rectangular region is listened again in S812 in the subsequent execution, the switch state can be queried, and if the switch state is queried to be in the open state, S813-S815 can be omitted, and the notification message 1 does not need to be repeatedly generated and distributed. Accordingly, after the libhwui receives the notification message 1, the notification message 1 can be saved, and after the libhwui receives the content to be drawn, the libhwui can draw the transparent rounded corner rectangular region based on the saved notification message 1. In this way, the communication between the surfaceflinger and the libhwui can be reduced.

[0226] The foregoing introduction about case 2 illustrates the processing procedure of each image frame after the rounded corner window is started. Further, after the rounded corner window is started, the position, size, rounded corner radius, etc. of the rounded corner window can be adjusted in response to event 2. Event 2 is used to trigger the mobile phone to adjust the rounded corner window. For example, event 2 can be a user's drag operation event on the rounded corner window, or event 2 can be a user's zoom operation (such as a double-click operation) event on the rounded corner window, etc.

[0227] Taking the drag operation event on the rounded corner window as an example, after the video window is started, the interface 901 shown in FIG. 9A can be displayed, and the interface 901 includes the video window 9011. In response to the user's drag operation of the video window 9011 in the interface 901 along the arrow direction in the interface 901, the position of the video window 9011 can be adjusted, and after the adjustment, the interface 902 can be displayed. The interface 2 also includes the video window 9011. Different from the interface 901, the position of the video window 9011 in the interface 902 has changed.

[0228] In some embodiments, for the above-mentioned adjustment of the rounded corner window, the rounded corner effect can still be achieved by the hole digging method for each image frame, so as to reduce the load of the GPU in the processing procedure of each image frame.

[0229] Taking the scenario shown in FIG. 9A as an example, in response to the user's drag operation of the video window 9011 in the interface 901 along the arrow direction in the interface 901, the video player can send an update request of the layer attribute to the WMS, and the WMS can update the position attribute of the video window layer and synchronously send it to the surfaceflinger in response to the update request. After receiving the updated position attribute, the surfaceflinger can generate a new notification 1, carry the updated layer attribute of the video window layer and an indication 1 in the notification 1, and distribute it to the libhwui 1 and the libhwui 2, so that the libhwui 1 and the libhwui 2 can draw the transparent rounded corner rectangle region based on the updated layer attribute of the video window layer.

[0230] In other embodiments, for the above-mentioned adjustment of the rounded corner window, in the adjustment process (referred to as case 21), such as the process of adjusting the position of the video window 9011 in the interface 901 to the position of the video window 9011 in the interface 902, the rounded corner effect is achieved by the GPU for the rounded corner synthesis, and after the adjustment is completed and is stable (referred to as case 22), such as after the adjustment to the position of the video window 9011 in the interface 902, the rounded corner effect is achieved by the hole digging method.

[0231] In actual implementation, in the process of adjusting the rounded window, if the rounded effect is realized by means of hole digging, there may be a delay, usually 1 frame of delay. For example, event 2 is received when the kth frame of image is displayed. After surfaceflinger receives the updated layer attribute (such as the position attribute) of the rounded window, it sends the updated layer attribute to libhwui through notification 1. Before libhwui receives the updated layer attribute, it may have been performing the drawing of the k+1th frame of image, that is, libhwui may draw the rounded rectangular region based on the un-updated layer attribute, that is, the layer attribute used in hole digging has a delay, which is referred to as hole digging delay. Subsequently, surfaceflinger may use the updated layer attribute for composition when compositing, such as calculating the position of the rounded window for composition based on the updated position attribute. In this way, the rounded rectangular region and the rounded window may not be consistent, such as different positions, thereby causing problems such as that the rounded rectangular region is black and the content in the rounded window is not fully displayed.

[0232] Still taking the example of dragging the video widget 9011 from the position of the video widget 9011 in the interface 901 shown in FIG. 9A in the direction indicated by the arrow, in the process of dragging, if the hole digging manner is used, the interface 911 shown in FIG. 9B may be displayed, part of the region in the rounded rectangular region 9111 in the interface 911 is black (that is, no pixel is filled), and only the part of the person is displayed in the content in the video widget 9011, and there is a loss.

[0233] Based on the problems in the process of adjusting the rounded window described above, in this embodiment, in the process of adjusting the rounded window, the GPU is still used to realize the rounded effect by means of rounded composition, which can avoid the above problems caused by hole digging delay.

[0234] That is, in the same scene with a rounded window, in some cases, the hole digging manner can be used to realize the rounded effect and reduce the load of the GPU, and in other cases, the GPU can be used to realize the rounded effect by means of rounded composition to avoid problems such as that the rounded rectangular region is black and the content in the rounded window is not fully displayed.

[0235] Referring to FIG. 10, the process of image processing includes:

[0236] S1001, after receiving event 1, in the process of processing a frame of image, it is listened that there is a rounded window.

[0237] It should be noted that after event 1, until the rounded window is closed, it can be listened that there is a rounded window.

[0238] S1002, detecting whether the continuous time length that the rounded-corner window keeps unchanged reaches time length 1. If not, S1003 is executed; if yes, S1006 is executed.

[0239] If it is detected that the continuous time length that the rounded-corner window keeps unchanged does not reach time length 1, such as not reaching 1s, 2s, etc., it indicates that the rounded-corner window is in an unstable state. Exemplarily, the rounded-corner window being in the unstable state includes adjusting the rounded-corner window, such as adjusting the position, size, etc. of the rounded-corner window, i.e. the above-mentioned case 21. Further exemplarily, the rounded-corner window being in the unstable state includes just starting the rounded-corner window. Still further exemplarily, the rounded-corner window being in the unstable state includes just ending the adjustment of the rounded-corner window, such as just ending the movement of the position of the rounded-corner window. For this case, the following S1003-S1005 can be executed to realize the rounded-corner effect by means of GPU synthesis.

[0240] If it is detected that the continuous time length that the rounded-corner window keeps unchanged reaches time length 1, it indicates that the rounded-corner window is in a stable state. Exemplarily, the rounded-corner window being in the stable state includes not receiving the event 2 after starting the rounded-corner window. Further exemplarily, the rounded-corner window being in the stable state includes keeping unchanged for a long time after adjusting the rounded-corner window, i.e. the above-mentioned case 21. For this case, the following S1006-S1008 can be executed to realize the rounded-corner effect by means of hole digging.

[0241] S1003, drawing / refreshing multiple layers. Among them, when drawing other layers, the transparent rounded-corner rectangular region is not drawn.

[0242] Since the rounded-corner effect is realized by means of GPU synthesis, the transparent rounded-corner rectangular region does not need to be drawn.

[0243] S1004, synthesizing multiple layers, superimposing the rounded-corner layer on the uppermost layer by means of GPU synthesis of the rounded-corner effect, obtaining an image frame, and the rounded-corner window in the image frame having the rounded-corner effect.

[0244] That is, the rounded-corner effect is synthesized by means of GPU to ensure the timeliness of realizing the rounded-corner effect. And the rounded-corner layer is superimposed on the uppermost layer to ensure that the content in the rounded-corner window is visible.

[0245] S1005, displaying the image frame. Thus, the rounded-corner effect can be presented.

[0246] After receiving the event 1, until the rounded-corner window is closed, the continuous time length that the rounded-corner window keeps unchanged does not reach time length 1, then the steps of the above-mentioned S1003-S1005 can be adopted to complete the processing of each image frame to ensure the timeliness of realizing the rounded-corner effect.

[0247] ​S1006, draw / refresh multiple layers. Wherein, when drawing other layers, draw a transparent rounded rectangle region.

[0248] After the rounded window change ends and reaches time length 1, when S1006 is executed, the transparent rounded rectangle region can be drawn based on the layer attribute of the rounded layer at the end of the change, so that the rounded rectangle region is consistent with the changed rounded window.

[0249] S1007, when synthesizing multiple layers, superimposing the rounded layer on the bottommost layer to obtain an image frame, and the rounded window in the image frame has a rounded effect.

[0250] S1008, display the image frame. Thus, the rounded effect can be presented.

[0251] The above S1006-S1008 can refer to the description of S702-S704 in the foregoing, which will not be repeated here.

[0252] After receiving event 1, until the rounded window is closed, the continuous time length that the rounded window remains unchanged reaches time length 1, then the steps of S1006-S1008 described above can be used to complete the processing of each image frame, so as to reduce the load of GPU.

[0253] Next, some typical scenarios are substituted into the above flowchart of FIG. 10 to further introduce the flowchart of FIG. 10:

[0254] Scenario 1, start the rounded window at t1, and then trigger the adjustment of the rounded window at t2 (the time interval between t1 and t2 is less than time length 1), and the adjustment ends at t3, and then there is no adjustment.

[0255] Between t1-t2: execute S1001, can listen to the existence of the rounded window; execute S1002, can detect that the continuous time length that the rounded window remains unchanged does not reach time length 1, and then S1003-S1005 will be executed, and the rounded effect is synthesized by GPU. That is, after starting the rounded window, if the rounded window is adjusted for a short time, the rounded effect will be synthesized by GPU.

[0256] Between t2-t4, t4

[0257] After t4: S1001 is executed, it can be listened that the rounded window exists; S1002 is executed, it can be detected that the continuous time length of the rounded window keeping unchanged reaches time length 1, and then S1006-S1008 will be executed to realize the rounded effect by drawing the transparent rounded rectangular area. That is, after the adjustment of the rounded window is ended and the interval time length 1, the rounded effect is realized by drawing the transparent rounded rectangular area.

[0258] Scenario 2, the rounded window is started at t1, the adjustment of the rounded window is triggered at t5 (the time interval between t1 and t5 is greater than or equal to time length 1), and the adjustment is ended at t6, and then it is not adjusted all the time.

[0259] Between t1-t7, t7

[0260] Between t7-t5: S1001 is executed, it can be listened that the rounded window exists; S1002 is executed, it can be detected that the continuous time length of the rounded window keeping unchanged reaches time length 1, and then S1006-S1008 will be executed to realize the rounded effect by drawing the transparent rounded rectangular area. That is, after the rounded window is started, if the rounded window is not adjusted for a long time, the rounded effect can be realized by drawing the transparent rounded rectangular area.

[0261] Between t5-t8, t8

[0262] After t8: S1001 is executed, it can be detected that the rounded window exists; S1002 is executed, it can be detected that the continuous time length of the rounded window remaining unchanged reaches time length 1, and then S1006-S1008 are executed, the rounded effect is realized by drawing a transparent rounded rectangular area. That is, after the adjustment of the rounded window is ended and the interval time length 1 is reached, the rounded effect is realized by drawing a transparent rounded rectangular area.

[0263] Of course, after adjusting the rounded window once, the rounded window can be adjusted again. Or, after starting the rounded window, the rounded window can not be adjusted at all. Here, it will not be described one by one.

[0264] Next, the image processing flow of FIG. 10 is further introduced in combination with the hardware and software structure of the mobile phone and still taking the scenario shown in FIG. 4 as an example, that is, the rounded layer is a video window layer, and the other layers are a desktop layer and a wallpaper layer.

[0265] Specifically, as shown in FIG. 11, after S812, S1101 is further included:

[0266] S1101: The surfaceflinger detects whether the continuous time length of the video window layer remaining unchanged reaches time length 1. If yes, S813 is executed; if no, S1102 is executed.

[0267] Wherein, the video window layer remaining unchanged indicates that the video window remains unchanged.

[0268] After the surfaceflinger listens to the start of the rounded window, the surfaceflinger can start timing, and then the surfaceflinger re-timing after receiving the layer attribute of the video window layer updated each time. In this way, when the timing reaches time length 1, it indicates that the continuous time length of the video window layer remaining unchanged reaches time length 1.

[0269] In a specific implementation mode, in response to event 2, the video player can send an attribute update request of the video window layer to the WMS. After receiving the attribute update request, the WMS can update the layer attribute of the video window layer. The WMS can also synchronize the updated layer information to the surfaceflinger, so that the surfaceflinger can receive the updated layer information, thereby controlling to start timing again.

[0270] If the continuous time length of the video window layer remaining unchanged is detected to reach time length 1, indicating that the round-corner window is in a stable state, the foregoing S813-S827 can be executed, which will not be described herein. It should be noted that if the round-corner window has changed, in S813, the notification 1 generated by the surfaceflinger carries the layer attribute of the video window layer after the last update, so that the libhwui draws a transparent round-corner rectangular region consistent with the video window at the time when the change ends.

[0271] If the continuous time length of the video window layer remaining unchanged is detected to not reach time length 1, indicating that the round-corner window is not stable, the following S1102-S1110 can be executed.

[0272] S1102, the surfaceflinger generates a notification message 2, and the notification message 2 includes indication information 3.

[0273] The indication information 3 is used to indicate no hole digging, that is, to close the round-corner effect achieved by drawing a transparent round-corner rectangular region.

[0274] S1103, the surfaceflinger distributes the notification message 2 to the libhwui 1 based on the callback function in the layer attribute of the desktop layer.

[0275] S1104, the surfaceflinger distributes the notification message 2 to the libhwui 2 based on the callback function in the layer attribute of the wallpaper layer.

[0276] It can be understood that when an application needs to draw a new image frame, the libhwui can be sent the content to be drawn, so that the libhwui can draw the content in the corresponding layer. Details are shown in S1105 and S1108.

[0277] S1105, the desktop application sends the content 3 to be drawn to the libhwui 1.

[0278] S1106, the libhwui 1 draws the content 3 in the desktop layer, and according to the notification message 2, does not draw a round-corner rectangular region in the desktop layer.

[0279] S1107, after the drawing is completed, the libhwui 1 feeds back indication information 4.

[0280] The indication information 4 is used to indicate that the hole digging is not completed.

[0281] S1108, the wallpaper application sends the content 4 to be drawn to the libhwui 2.

[0282] S1109, libhwui 2 draws content 4 in the wallpaper layer, and does not draw the rounded rectangular region in the wallpaper layer according to the notification message 2.

[0283] S1110, libhwui 2 feeds back the indication information 4 after the drawing is completed.

[0284] The principles and specific implementations of S1103-S1110 are similar to S814-S821, and details can be referred to the description of S814-S821, which will not be repeated here.

[0285] S1111A, surfaceflinger does not cancel the rounded corner synthesis after receiving the indication information 4 of the desktop layer and the wallpaper layer.

[0286] Since the continuous time length of the video window layer remains unchanged does not reach the time length 1, it is indicated that the GPU synthesis mode should be used to realize the rounded corner effect, and then surfaceflinger can not cancel the rounded corner synthesis after receiving the indication information 4 without digging holes.

[0287] S1111B, surfaceflinger refreshes the video content of the video window layer.

[0288] S1112, surfaceflinger obtains the superimposition order of the desktop layer, the wallpaper layer and the video window layer, wherein the video window layer is located at the uppermost layer.

[0289] Using the GPU synthesis mode, the video window layer should be superimposed on the uppermost layer to ensure that the video window is visible. Accordingly, surfaceflinger can not adjust the video window layer to the bottom layer after determining the superimposition order.

[0290] In a specific implementation, after the continuous time length of the video window layer remaining unchanged is detected to reach the time length 1 in S1101, the surfaceflinger can open the switch, and after the continuous time length of the video window layer remaining unchanged is detected not to reach the time length 1, the surfaceflinger can close the switch. Subsequently, the surfaceflinger can query the switch state to determine whether to cancel the rounded corner synthesis and adjust the superposition order. When the surfaceflinger queries that the switch state is to close the switch, it is determined not to cancel the rounded corner synthesis and not to adjust the superposition order, as shown in S1111 and S1112. When the surfaceflinger queries that the switch state is to open the switch and receives an indication 2 that all other layers complete the hole digging, it is determined to cancel the rounded corner synthesis and adjust the superposition order, as shown in S822-S824. When the surfaceflinger queries that the switch state is to open the switch and receives an indication 2 that part of the other layers complete the hole digging, it is determined not to cancel the rounded corner synthesis and not to adjust the superposition order, so that the rounded corner effect can be realized in the case that part of the other layers do not complete the hole digging.

[0291] S1113, the surfaceflinger calls the GPU to synthesize the rounded corner effect, and synthesizes the desktop layer, the wallpaper layer and the video window layer according to the superposition order to obtain an image frame.

[0292] Since the rounded corner synthesis is not cancelled, the surfaceflinger can call the GPU to complete the rounded corner synthesis in the synthesis stage, and the video window in the obtained image frame also has the rounded corner effect.

[0293] S1114, the surfaceflinger sends the image frame to the display screen.

[0294] S1115, the display screen displays the image frame.

[0295] The embodiments of FIGS. 10 and 11 above illustrate the processing process of one image frame after receiving the event 1. In practice, the processing of each image frame can be processed in a similar manner before the rounded corner window is closed, so that the rounded corner effect can be realized in each image frame. That is, in the process of processing each image frame, it can be listened whether there is a rounded corner window and whether the continuous time length of the rounded corner window remaining unchanged reaches the time length 1. If it is listened that there is a rounded corner window and the continuous time length of the rounded corner window remaining unchanged reaches the time length 1, a transparent rounded corner rectangular region is drawn when each other layer is drawn.

[0296] In a specific implementation, in the flow shown in FIG. 11, S812-S1115 can be executed in a loop, so that in each image frame, if it is detected that the round-corner window exists and the continuous time length that the round-corner window remains unchanged reaches time length 1, the load of the GPU can be reduced, and if it is detected that the round-corner window exists but the continuous time length that the round-corner window remains unchanged does not reach time length 1, problems such as that the round-corner rectangular region is black and the content in the round-corner window is not fully displayed due to delay can be avoided. Similarly, in a case where it is detected in S812 that the round-corner rectangular region exists and it is detected in S1101 that the continuous time length that the video window layer remains unchanged reaches time length 1, the surfaceflinger can query the switch state, and if it is found that the switch state is on, S813-S815 can be omitted, and it is not necessary to repeatedly generate and distribute notification 1. Conversely, in a case where it is detected in S812 that the round-corner rectangular region exists but it is detected in S1101 that the continuous time length that the video window layer remains unchanged does not reach time length 1, the surfaceflinger can query the switch state, and if it is found that the switch state is off, S1102-S1104 can be omitted, and it is not necessary to repeatedly generate and distribute notification message 2. In this way, the communication between the surfaceflinger and the libhwui can also be reduced.

[0297] In addition to the round-corner synthesis described above, some other effects such as shading and blur effects also need to be synthesized by the GPU. Accordingly, in the synthesis stage, the GPU is called (for example, the surfaceflinger in a mobile phone) to complete the synthesis of other effects.

[0298] To address this situation, in some embodiments, if it is detected that the round-corner window exists, it can be further detected (for example, by the surfaceflinger) whether other effects are synthesized by the GPU. If it is detected that other effects are not synthesized by the GPU, it indicates that in the synthesis stage, the GPU is mainly called to synthesize the round-corner effect, and the round-corner effect can be implemented in a hole-digging manner, for example, S1006-S1008 are executed, and in S1007, the GPU is not called but the HWC can be called to complete layer superposition. In this way, for a scenario in which other effects are not synthesized by the GPU, the round-corner effect is implemented in a hole-digging manner, so that in the synthesis stage, the GPU is not called at all to perform synthesis processing, and the load of the GPU is obviously reduced.

[0299] In addition, if it is detected that other effects are to be GPU-composited, it indicates that not only the rounded corner effect is to be GPU-composited, but also other effects are to be GPU-composited in the composition stage. In this case, even if the rounded corner effect is implemented in the way of digging holes, the other effects are still to be GPU-composited in the composition stage, and the effect of reducing the GPU load is not obvious. Therefore, the rounded corner effect can still be implemented in the way of GPU-composition, such as performing S1003-S1005 and invoking the GPU-composition of other effects in S1004.

[0300] For the specific implementation of the rounded corner effect implemented in the way of digging holes or GPU-composition in this embodiment, refer to the related description in FIG. 11, which will not be repeated here. For example, the rounded corner effect is implemented in the way of digging holes, which is implemented by S813-S827 in FIG. 11, and the rounded corner effect is implemented in the way of GPU-composition, which is implemented by S1102-S1115 in FIG. 11.

[0301] In actual implementation, the embodiments in the above case 2 can also be combined. For example, in the case that the existence of the rounded corner window is monitored, the duration of the unchanged rounded corner window exceeds duration 1, and other effects are not GPU-composited, the rounded corner effect can be implemented in the way of digging holes.

[0302] Case 3, closing the rounded corner window.

[0303] In response to event 3, the rounded corner window can be closed. Event 3 is used to trigger the closing of the rounded corner window. Event 3 can be a user trigger operation event on the close control in the rounded corner window, or event 3 can be a user operation event of dragging the rounded corner window to the edge of the display screen, or event 3 can be an operation event of exiting the rounded corner window, and the present application does not make specific limitations thereto.

[0304] Taking the case that event 3 is a user trigger operation event on the close control in the rounded corner window, and the close control is “X” 12012 in the video window 12011 in the interface 1201 shown in FIG. 12 as an example, in response to the clicking operation of the user on “X” 12012, the video window 12011 can be closed and the interface 1202 shown in FIG. 12 is displayed. The interface 1202 no longer includes the video window 12011.

[0305] After the rounded corner window is closed, the image processing process no longer includes implementing the rounded corner effect. Referring to FIG. 13, the image processing process includes:

[0306] S1301, after receiving event 3, in the processing process of an image frame, it is monitored that there is no rounded corner window.

[0307] It should be noted that the listening is always on after the start, and the result of the existence or non-existence of the rounded corner window can be listened to for each image frame. After receiving event 3, the rounded corner window is closed, and then the non-existence of the rounded corner window is listened to.

[0308] S1302, draw / refresh multiple layers.

[0309] For example, in the scenario shown in FIG. 4, after the video window layer is closed, the desktop icon can be drawn in the desktop layer, and the wallpaper can be drawn in the wallpaper layer, without further refreshing the layer content of the video window layer.

[0310] S1303, synthesize multiple layers to obtain an image frame, and the image frame does not have a rounded corner effect.

[0311] For the non-existence of the rounded corner window, the conventional image processing flow can be followed, that is, the conventional drawing and rendering are completed for image processing. For example, in S1302, the transparent rounded rectangular region can not be drawn, and in S1303, the rounded corner can not be synthesized.

[0312] S1304, display the image frame.

[0313] Next, the image processing flow in case 3 is further introduced in combination with the hardware and software structure of the mobile phone and taking the scenario shown in FIG. 12 as an example, that is, the rounded corner layer is the video window layer, and the other layers are the desktop layer and the wallpaper layer.

[0314] Specifically, as shown in FIG. 14, the image processing process includes:

[0315] S1401, the video player sends a destruction request of the video window layer to the WMS in response to event 3.

[0316] S1402, the WMS destroys the video window layer.

[0317] For example, the WMS can destroy the video window layer by releasing the address of the video window layer.

[0318] It should be noted that the WMS can destroy the video window layer in the surfaceflinger synchronously by destroying the video window layer. In this way, the surfaceflinger can listen to the result of S1403 as follows.

[0319] S1403, the surfaceflinger listens to the non-existence of the rounded corner window.

[0320] After the video window layer is destroyed, the surfaceflinger can listen to the non-existence of the rounded corner window before the video window (or other rounded corner windows) is started again.

[0321] S1404, the surfaceflinger detects whether the hole digging is started. If yes, S1405-S1407 are executed; if no, S1405-S1407 are not executed.

[0322] For example, the surfaceflinger can query the switch state. If the switch is turned on, it indicates that the hole digging is started. If the switch is turned off, it indicates that the hole digging is not started.

[0323] If the hole digging is started, the hole digging needs to be stopped. The surfaceflinger can execute the following S1405-S1407 to stop the hole digging.

[0324] If the hole digging is not started, the hole digging does not need to be stopped. The surfaceflinger can not execute the following S1405-S1407.

[0325] S1405, the surfaceflinger generates a notification message 2, and the notification message 2 includes indication information 3.

[0326] The indication information 3 is used to indicate that the hole digging is not started, i.e., the round corner effect is not realized by drawing a transparent round corner rectangle region.

[0327] S1406, the surfaceflinger distributes the notification message 2 to the libhwui 1 based on the callback function in the layer attribute of the desktop layer.

[0328] S1407, the surfaceflinger distributes the notification message 2 to the libhwui 2 based on the callback function in the layer attribute of the wallpaper layer.

[0329] S1408, the desktop application sends content 5 to be drawn to the libhwui 1.

[0330] S1409, the libhwui 1 draws the content 5 in the desktop layer, and according to the notification message 2, does not draw a round corner rectangle region in the desktop layer.

[0331] It should be noted that for the case of no in S1404, the surfaceflinger does not generate the notification message 2 through S1405 and send the notification message 2 to the libhwui 1 through S1406, but the libhwui 1 saves the notification message 2, and the libhwui 1 can not draw the round corner rectangle region according to the saved notification message 2.

[0332] Exemplarily, before the round-corner window is closed, the surfaceflinger detects that the continuous time length that the round-corner window remains unchanged does not reach the time length 1, and then the hole digging can be started, for example, a switch is opened, and a notification message 2 is generated and sent to the libhwui 1. The libhwui 1 can save the notification message 2, and before receiving the notification 1, the libhwui 1 does not draw the round-corner rectangular area based on the saved notification message 2.

[0333] S1410, the libhwui 1 feeds back the indication information 4 after the drawing is completed.

[0334] S1411, the wallpaper application sends the content 6 to be drawn to the libhwui 2.

[0335] S1412, the libhwui 2 draws the content 6 in the wallpaper layer, and does not draw the round-corner rectangular area in the wallpaper layer according to the notification message 2.

[0336] Similarly, for the case of if no detected in S1404, the surfaceflinger does not generate the notification message 2 through S1405 and send the notification message 2 to the libhwui 2 through S1407, but the libhwui 2 saves the notification message 2, and the libhwui 2 can not draw the round-corner rectangular area according to the saved notification message 2.

[0337] S1413, the libhwui 2 feeds back the indication information 4 after the drawing is completed.

[0338] By using the above S1403-S1413, after the round-corner window is closed, the libhwui can also not draw the round-corner rectangular area based on the notification message 2, and feed back the indication information 4 to the surfaceflinger.

[0339] The principle and specific implementation of the above S1405-S1413 are similar to the foregoing S1102-S1110, and specific reference can be made to the description of S1102-S1110, which will not be repeated here.

[0340] S1414, the surfaceflinger obtains the superposition order of the desktop layer, the wallpaper layer and the video window layer after receiving the indication information 4 of the desktop layer and the wallpaper layer, wherein the video window layer is located at the uppermost layer.

[0341] By using the GPU synthesis mode, the video window layer should be superimposed on the uppermost layer to ensure that the video window is visible. Correspondingly, the surfaceflinger can not adjust the video window layer to the bottom layer after determining the superposition order.

[0342] In a specific implementation, in the case that the opening of the hole is detected in S1404, the surfaceflinger can close the switch. Subsequently, the surfaceflinger can determine whether to cancel the rounded corner composition and adjust the superposition order by querying the switch state. In this case, the surfaceflinger queries that the switch state is the closed switch, and determines not to cancel the rounded corner composition and not to adjust the superposition order.

[0343] In addition, since it is detected that there is no rounded corner window, it is indicated that the GPU composition of the rounded corner will not be called at all, and thus the surfaceflinger can not cancel the rounded corner composition after receiving the indication information 4 that there is no hole. In addition, since the video window layer has been destroyed, the surfaceflinger also does not need to refresh the video content in the video window layer.

[0344] S1415, the surfaceflinger composites the desktop layer and the wallpaper layer according to the superposition order to obtain an image frame, and the image frame does not include the video window.

[0345] Since there is no rounded corner window such as the video window, the surfaceflinger will not call the GPU to complete the rounded corner composition in the composition stage.

[0346] S1416, the surfaceflinger sends the image frame to the display screen.

[0347] S1417, the display screen displays the image frame.

[0348] The above embodiments of FIG. 13 and FIG. 14 illustrate the processing of one image frame after receiving the event 3. In practice, after the rounded corner window is closed, the processing of each image frame can be processed in a similar manner before the rounded corner window is started again, so that the transparent rounded corner rectangular region is not drawn and the superposition order between the layers is not changed in the processing of each image frame.

[0349] In a specific implementation, in the flow shown in FIG. 14, S1403-S1417 can be executed in a loop, so that in each image frame, in the case that it is detected that there is no rounded corner window, the transparent rounded corner rectangular region is not drawn and the superposition order between the layers is not changed. In addition, in the case that it is detected that there is no rounded corner rectangular region in S1403 and it is detected that the hole is not opened in S1404, S1405-S1407 can be omitted, and the notification message 2 does not need to be repeatedly generated and distributed. In this way, the communication between the surfaceflinger and the libhwui can also be reduced.

[0350] It should be noted that in the foregoing description of the case 2 and the case 3, the drawing operation is performed by libhwui. In practice, some layers can be drawn by other modules, such as surfaceview. For this case, after the layers are created, the other modules also need to register a callback to the surfaceflinger to establish a path between the other modules and the surfaceflinger. Subsequently, the notification 1 and the notification 2 are transmitted through the path to control the other modules to dig holes, and the other modules can feed back an indication of whether the digging of the holes is completed to the surfaceflinger through the path. Details are not described herein.

[0351] The embodiment of the present application further provides an electronic device, which can include a memory and one or more processors (such as CPU, GPU, NPU, etc.). The memory and the processor are coupled. The memory is configured to store computer program codes, which include computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps performed by the device in the above method embodiments.

[0352] The embodiment of the present application further provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected through a circuit. For example, the interface circuit can be configured to receive signals from other devices (such as the memory of the electronic device). For another example, the interface circuit can be configured to send signals to other devices (such as the processor). For example, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which are not limited in the embodiment of the present application.

[0353] The embodiment of the present application further provides a computer storage medium, which stores computer instructions. When the computer instructions are run on the electronic device, the electronic device can perform the above related method steps to implement the image processing method in the above embodiments.

[0354] The embodiment of the present application further provides a computer program product, which can make the computer perform the above related steps to implement the image processing method in the above embodiments when the computer program product is run on the computer.

[0355] In addition, the embodiment of the present application further provides a device, which can be a chip, a component or a module. The device can include a processor and a memory connected to each other. The memory is configured to store computer execution instructions. When the device is running, the processor can execute the computer execution instructions stored in the memory to make the chip perform the image processing method in the above method embodiments.

[0356] Among them, the electronic device, computer storage medium, computer program product or chip provided by the embodiment are used for executing the corresponding method provided above, so the beneficial effects achieved by them can refer to the beneficial effects of the corresponding method provided above, which will not be repeated here.

[0357] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional module is taken as an example, and in actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0358] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other ways. For example, the apparatus embodiment described above is only schematic, for example, the division of the module or unit is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0359] The unit described as a separate component can or can not be physically separated, and the component shown as a unit can be one physical unit or multiple physical units, that is, it can be located in one place or distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0360] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0361] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application essentially or in other words the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The software product is stored in a storage medium, and includes a plurality of instructions for causing an apparatus (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0362] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. An image processing method, characterized by, The method is applied to an electronic device, and the method comprises: receiving an event of starting a first window, the first window being a non-full-screen round-corner window; after receiving the event of starting the first window, displaying a first image; wherein the first image comprises a first layer and a second layer, the second layer being superimposed on an upper layer of the first layer, the first layer having first to-be-displayed content of the first window drawn therein, and the second layer having second to-be-displayed content of other regions except the first window and a transparent round-corner rectangular region drawn therein, the transparent round-corner rectangular region having a same position as a display region of the first window.

2. The method of claim 1, wherein, Before the first image is displayed, the method further comprises: generating the first image; wherein during the generation of the first image, a round-corner effect of the first window is not synthesized.

3. The method of claim 2, wherein, During the generation of the first image, the round-corner effect of the first window is not synthesized, comprising: if a continuous time length during which the display region of the first window in the display screen of the electronic device remains unchanged reaches a first time length, during the generation of the first image, the round-corner effect of the first window is not synthesized.

4. The method according to claim 2 or 3, characterized in that, Before or after the first image is displayed, the method further comprises: generating a second image, and displaying the second image, the second image comprising the first window; wherein during the generation of the second image, the round-corner effect of the first window is synthesized.

5. The method of claim 4, wherein, The second image comprises a third layer and a fourth layer, the fourth layer being superimposed on an upper layer of the third layer, the third layer having third to-be-displayed content of the first window drawn therein, and the fourth layer having fourth to-be-displayed content of other regions except the first window drawn therein.

6. The method according to claim 4 or 5, characterized in that, During the generation of the second image, the round-corner effect of the first window is synthesized, comprising: if a continuous time length during which the display region of the first window in the display screen of the electronic device remains unchanged does not reach the first time length, during the generation of the second image, the round-corner effect of the first window is synthesized.

7. The method according to any one of claims 2-6, characterized in that, The generation of the first image comprises: drawing or refreshing the first to-be-displayed content in the first layer; drawing the second to-be-displayed content in the second layer, and drawing the transparent round-corner rectangular region in the second layer based on layer attributes of the first layer, so that the transparent round-corner rectangular region has a same position as the display region of the first window; superimposing the second layer on the upper layer of the first layer to generate the first image.

8. The method of claim 7, wherein, The layer attributes comprise a position, a size of the first layer, and a round-corner radius of the first window.

9. The method according to claim 7 or 8, characterized in that, The drawing of the transparent round-corner rectangular region in the second layer comprises: if there is first indication information, the transparent round-corner rectangular region is drawn in the second layer, the first indication information indicating that the transparent round-corner rectangular region is drawn.

10. The method of claim 9, wherein, The method further comprises: generating the first indication information after receiving the event of starting the first window; or After receiving the event of starting the first window, and a continuous time length of a display area of the first window in a display screen of the electronic device remaining unchanged reaches a first time length, the first indication information is generated.

11. The method according to any one of claims 7-10, characterized in that, Before the superimposing the second layer on an upper layer of the first layer, and generating the first image, the method further comprises: canceling a rounded corner effect of the first window, so that the rounded corner effect of the first window is not synthesized in the process of generating the first image.

12. The method according to any one of claims 7-11, characterized in that, Before the superimposing the second layer on an upper layer of the first layer, the method further comprises: obtaining a first superimposition order, in which the first layer is located on an upper layer of the second layer; adjusting the first superimposition order to obtain a second superimposition order, in which the second layer is located on an upper layer of the first layer; the superimposing the second layer on an upper layer of the first layer comprises: superimposing the second layer on an upper layer of the first layer based on the second superimposition order.

13. The method according to any one of claims 2-12, characterized in that, The electronic device comprises an image synthesizer, a drawing module, a hardware synthesizer (HWC), and a graphics processing unit (GPU); After receiving the event of starting the first window, the method further comprises: The image synthesizer listens to the existence of the rounded corner window, and sends a first notification message to the drawing module, the first notification message comprising first indication information and layer attributes of the first layer; The generating the first image comprises: The drawing module draws the second to-be-displayed content on the second layer; The drawing module draws the transparent rounded corner rectangular region on the second layer based on the first notification message; After drawing the transparent rounded corner rectangular region, the drawing module sends second indication information to the image synthesizer; After receiving the second indication information, the image synthesizer does not call the GPU to synthesize the rounded corner effect, and calls the HWC to superimpose the second layer on an upper layer of the first layer to generate the first image. The first to-be-displayed content in the first layer is drawn by the drawing module or refreshed by the image synthesizer.

14. The method of claim 13, wherein, The image synthesizer listens to the existence of the rounded corner window, and sends a first notification message to the drawing module, comprising: The image synthesizer listens to the existence of the rounded corner window, and sends a first notification message to the drawing module after a continuous time length of a display area of the first window in a display screen of the electronic device remaining unchanged reaches a first time length.

15. An electronic device, comprising: comprises: a display screen, one or more processors, and one or more memories; the one or more processors are coupled with the display screen and the one or more memories; the one or more memories are used to store computer program codes, the computer program codes comprising computer instructions, when the one or more processors execute the computer instructions, causing the electronic device to perform the method of any one of claims 1-14.

16. A computer readable storage medium having stored thereon computer instructions, wherein, When the computer instructions are run on an electronic device, the electronic device is caused to perform the method of any one of claims 1-14.

17. A computer program product comprising computer instructions, characterized in that, When the computer program product is run on a computer, the computer is caused to perform the method of any one of claims 1-14.

Images