Image processing method and electronic device

Abstract

An image processing method and an electronic device, relating to the technical field of image processing. The method comprises: running a first application, the first application comprising a first window. Before the display duration of the first window reaches a first duration, first layer content of a first target layer associated with the first window is generated, and window content of the first window is displayed on the basis of the first layer content; and when the display duration of the first window reaches the first duration, second layer content of the first target layer is generated, the value of blue channels of pixels in the second layer content is enhanced to obtain third layer content, and the window content of the first window is displayed on the basis of the third layer content. In this way, after a window is displayed for a long time, the effect of eye protection can be achieved by enhancing the blue color of window content.

Description

An image processing method and electronic device

[0001] This application claims priority to Chinese Patent Application No. 202410941811.5, filed on July 12, 2024, entitled "An Image Processing Method and Electronic Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, and in particular to an image processing method and electronic device. Background Technology

[0003] Prolonged daily use of electronic devices such as mobile phones and tablets can damage the eyes and affect vision. At the same time, with improved living standards and increased health awareness, people are paying more and more attention to eye protection. Therefore, there is an urgent need for a solution that can protect the eyes while using electronic devices. Summary of the Invention

[0004] This application provides an image processing method and electronic device that can enhance the value of the blue channel of pixels in an image by using an eye protection algorithm during the processing of a frame of an image, thereby achieving eye protection.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, this application provides an image processing method, comprising: running a first application (such as application 1, application 2, etc. below), wherein the first application includes a first window (such as window A, window B, etc. below).

[0007] Before the first window's display duration reaches the first duration (duration 1 in the following text), the content of the first layer of the first target layer associated with the first window is generated, and the window content of the first window is displayed based on the content of the first layer. In other words, when the first window's display duration is short, it can be rendered, composited, and then the window content can be displayed normally without performing eye protection processing.

[0008] After the first window has been displayed for its designated duration, the second layer content of the first target layer is generated. The blue channel values ​​of the pixels in the second layer content are enhanced (this can also be referred to as blue enhancement below), resulting in the third layer content. The window content of the first window is then displayed based on the third layer content. In other words, after the first window has been displayed for a relatively long time and the target layer content has been obtained, the blue color in the layer content needs to be enhanced. Consequently, the blue color in the final displayed window content will be enhanced, thus achieving eye protection.

[0009] In summary, by adopting the solution of this application, the blue color in the window can be enhanced after the window has been displayed for a long time, thereby improving the user's viewing comfort when looking at the window and achieving eye protection.

[0010] It should be noted that the target layer associated with a window refers to the layer used to draw the window content to be displayed in that window. For example, if the window is a video playback window, the target layers include the video content layer, the subtitle layer, and so on. As another example, if the window is a text and image (such as a novel) reading window, the target layers include the text layer, the illustration layer, and so on.

[0011] In one possible design approach of the first aspect, enhancing the blue channel value of pixels in the second layer content includes: enhancing the blue channel value of pixels in the second layer content during the image compositing stage or the image rendering stage.

[0012] In one possible design approach of the first aspect, enhancing the blue channel value of pixels in the second layer content during the image compositing or image rendering stage includes:

[0013] When the first window is a first-type window (target window 1 in the following text), the blue channel value of pixels in the second layer content is enhanced during the image compositing stage. At least one target layer associated with the first-type window can obtain its layer content without rendering. For example, at least one target layer is a video content layer, which can bypass the rendering stage and directly transmit the data decoded by the video codec in the display memory (graphic buffer) to the image compositor, and then the image compositor refreshes the data to the layer. That is, the first-type window can include a video playback window. Clearly, if at least one target layer of the first-type window does not obtain its layer content through rendering, eye protection cannot be achieved during the image rendering stage. Meanwhile, each image processing involves an image compositing process; therefore, for the first-type window, blue can be enhanced during the image compositing stage to achieve eye protection. This ensures that the blue in the video content layer can be enhanced, achieving an eye-protecting effect.

[0014] When the first window is a type II window, the blue channel value of pixels in the second layer content is enhanced during the image rendering stage. All target layers associated with the type II window (such as target window 2 in the text below) undergo rendering to obtain layer content. For example, if the type II window includes a text / image (e.g., a novel) reading window, enhancing the blue channel during the image compositing stage for type II windows can achieve eye protection and ensure the desired effect. Furthermore, the power consumption during image rendering is typically lower than that during image compositing. Consequently, implementing eye protection during image rendering has a smaller impact on device power consumption and is more conducive to ensuring device performance.

[0015] It should be noted that in the following text, the first type of window is a video window, and the second type of window is a text and image reading window.

[0016] In one possible design approach of the first aspect, before enhancing the blue channel values ​​of pixels in the second layer content during the image compositing stage, the method further includes: if the first window is a video window, setting the GPU compositing flag of the first target layer (in the force client composition mode described below) to a first identifier (identifier 1 described below). Image compositing is then determined to be performed using the GPU based on the first identifier. It should be noted that if the GPU compositing flag of one layer is set to the first identifier, it can be determined that image compositing will be performed using the GPU.

[0017] Accordingly, the blue channel values ​​of pixels in the second layer content are enhanced during the image compositing stage, including: enhancing the blue channel values ​​of pixels in the second layer content during the GPU-executed image compositing stage. This leverages the powerful image processing capabilities of the GPU to achieve eye protection.

[0018] In one possible design approach of the first aspect, after generating the second layer content of the first target layer, the method further includes: if the first window is not a video playback window or a text and image reading window, not enhancing the blue channel values ​​of the pixels in the second layer content, and displaying the window content of the first window based on the second layer content.

[0019] Typically, eye strain is most pronounced in scenarios involving prolonged video playback or reading through an image / text reader, where eye protection is often necessary. Outside of these situations, eye protection is generally not a requirement. Furthermore, implementing eye protection would increase power consumption. Therefore, for scenarios where eye protection is not a necessity, it can be omitted. This ensures that the eye protection effect matches the required level of eye protection while simultaneously reducing device power consumption.

[0020] In one possible design approach of the first aspect, an eye-protection solution is implemented during the image rendering stage, and its specific implementation is as follows:

[0021] Generate the second layer content of the first target layer associated with the first window, including: rendering the second layer content in the first target layer, that is, obtaining the second layer content through rendering.

[0022] In the image rendering stage, the blue channel values ​​of pixels in the second layer are enhanced to obtain the third layer content. This includes: obtaining a first screenshot of the second layer content during image rendering; fusing the first screenshot with a first shader to obtain the third layer content; and the first shader including an algorithm to enhance the blue channel values ​​(eye-protection algorithm). In other words, the rendering stage adds screenshotting and fusion steps to achieve eye protection. The screenshot provides the object on which the eye-protection algorithm is executed. The fusion process uses this object as input to the eye-protection algorithm, and running the algorithm yields the eye-protection output, which is the third layer content.

[0023] Displaying the content of the first window based on the content of the third layer involves: compositing the content of multiple layers to obtain and display the content of the first window, where the content of the multiple layers includes the content of the third layer. In other words, after implementing eye protection during the rendering stage, further image compositing (including layer overlay) can be performed to obtain and display the window content. This allows the first window to display an enhanced blue image, thus achieving eye protection.

[0024] In one possible design approach of the first aspect, an eye-protection solution is implemented during the image rendering stage, and its specific implementation is as follows:

[0025] Generate the second layer content of the first target layer associated with the first window, including rendering and / or refreshing the second layer content in the first target layer. Specifically, the video content layer needs to be refreshed to obtain its layer content, while other target layers can be rendered to obtain their layer content.

[0026] In the image compositing stage, the blue channel values ​​of pixels in the second layer content are enhanced to obtain the third layer content. This includes: obtaining a first screenshot of the second layer content in the image compositing stage; correcting the first screenshot according to the resolution of the first display screen (the screen displaying the first window); and fusing the corrected first screenshot with a second shader to obtain the third layer content. The second shader includes an algorithm to enhance the blue channel values. It should be noted that each frame of video content is decoded from the video source using a video codec and then used as the layer content of the video content layer (a first target layer), rather than being rendered according to the buffer size (i.e., resolution) of the video content layer. Therefore, the resolution of the video content is highly likely to be mismatched with the resolution of the first display screen used for display. Therefore, after obtaining the screenshot of the video content, the screenshot is first corrected according to the resolution of the first display screen so that the corrected screenshot matches the resolution of the first display screen. Then, it is fused with the second shader to obtain the layer content that corresponds exactly to the resolution of the first display screen.

[0027] The first window's content is displayed based on the content of the third layer. This includes overlaying the content of multiple layers to obtain and display the content of the first window, where the content of the multiple layers includes the content of the third layer. In other words, after achieving eye protection during the compositing stage, it is necessary to further refine the image by overlaying layers during the compositing stage to obtain and display the content of the first window. This allows the enhanced blue tone of the image to be displayed in the first window, thus achieving eye protection.

[0028] In one possible design approach of the first aspect, before enhancing the blue channel values ​​of pixels in the second layer content, the method further includes: filtering invalid layers in the first target layer to obtain a filtered first target layer; wherein, invalid layers include at least one of the following: layers for drawing bullet screen content, layers with a display frequency greater than a first frequency (frequency 1 as described below), and layers with a buffer size greater than the resolution of the first display screen, where the first display screen is the display screen that displays the first window. After filtering out invalid layers, some layers that consume a lot of power for eye protection can be filtered out, narrowing the range of eye-protection layers and improving the targeting of eye protection. Correspondingly, enhancing the blue channel values ​​of pixels in the second layer content includes: enhancing the blue channel values ​​of pixels in the second layer content of the filtered first target layer.

[0029] In one possible design approach of the first aspect, enhancing the blue channel value of pixels in the second layer content includes: enhancing the blue channel value of pixels in the second layer content based on the resolution of the first display screen, the blur radius of the first target layer (as recorded in the configuration table below), and the buffer size of the first target layer. Here, the blur radius indicates the sampling specification when enhancing the blue channel value, and the first display screen is the display screen showing the first window. This allows for targeted enhancement of the blue channel for different first target layers, improving the eye-friendly design.

[0030] In one possible design approach of the first aspect, the blur radius is positively correlated with the enhancement level of the blue channel value. That is, the larger the blur radius, the higher the blue enhancement level, resulting in greater eye protection. A first ratio is also positively correlated with the enhancement level of the blue channel value; this first ratio is the ratio of the buffer size of the first target layer to the resolution of the first display screen. That is, the higher the first ratio, the higher the blue enhancement level, resulting in greater eye protection.

[0031] In one possible design approach of the first aspect, the above method is applied to an electronic device, and the method further includes: correcting the blur radius based on the ambient light intensity of the environment in which the electronic device is located, so that the blur radius matches the current ambient light intensity. Correspondingly, based on the resolution of the first display screen, the blur radius of the first target layer, and the buffer size of the first target layer, the value of the blue channel in the content of the second layer is enhanced, including: enhancing the value of the blue channel of pixels in the content of the second layer based on the resolution of the first display screen, the corrected blur radius, and the buffer size of the first target layer. This can improve the reasonableness of eye protection.

[0032] In one possible design approach of the first aspect, the blur radius of the first target layer is corrected based on the ambient light intensity of the environment in which the electronic device is located. This includes: 1) When the ambient light intensity is a first intensity, the corrected blur radius is the first radius; 2) When the ambient light intensity is a second intensity, the corrected blur radius is the second radius. Wherein, the first intensity is lower than the second intensity, and the first radius is smaller than the second radius. That is, the higher the ambient light intensity, the smaller the corrected blur radius.

[0033] The higher the ambient light, the brighter the environment, and the more significantly the image sharpness is affected. In this case, the blur radius can be reduced to decrease the impact of eye protection on image sharpness. Conversely, the lower the ambient light, the darker the environment, and the less the image sharpness is affected. In this case, the blur radius can be increased to better protect the eyes.

[0034] In one possible design approach of the first aspect, the above method is applied to an electronic device, which includes a display engine service, an image synthesizer, and a view root implementation.

[0035] In the case where the first window is a video playback window, the above-mentioned enhancement of the blue channel values ​​of pixels in the second layer content during the image compositing stage includes:

[0036] When the display engine service detects that the first window is a video playback window, it sends a first notification to the image compositor (including layer whitelists and algorithm parameters as described below). In response to this first notification, the image compositor controls the enhancement of the blue channel values ​​of pixels in the second layer content during the image compositing stage. In other words, for a video playback window, the display engine service notifies the image compositor, thereby enabling eye-friendly features to be implemented during the image compositing stage. For example, the image compositor can set the GPU compositing flag of the first target layer to a first identifier and incorporate eye-friendly shaders (such as a second shader) during the compositing stage, thus controlling the GPU to perform image compositing and implementing eye-friendly features during the compositing stage.

[0037] When the first window is a text / image reading window, the blue channel value of pixels in the second layer content is enhanced during the image rendering stage. This includes: when the display engine service detects that the first window is a text / image reading window, the display engine service sends a second notification (including layer whitelists and algorithm parameters as described below) to the view root implementation. In response to the second notification, the view root implementation controls the enhancement of the blue channel value of pixels in the second layer content during the image rendering stage. In other words, for a text / image reading window, the display service engine notifies the image root implementation, thereby allowing the image root implementation to control eye protection during the image rendering stage. For example, the image root implementation can incorporate eye-protection shaders (such as the first shader) during the rendering stage, thus controlling eye protection during the rendering stage.

[0038] In one possible design approach of the first aspect, after running the first application, the method further includes: the view root implementation registering a listener for the first window with the display engine service; and the second type of window includes the window for which the view root implementation has registered the listener. By registering the listener, cross-process communication between the display engine service and the view root implementation can be achieved, enabling the display engine service to provide feedback on the listening result of the first window to the view root implementation. Correspondingly, the display engine service sends a second notification to the view root implementation, including: the display engine service sending a second notification to the view root implementation that registered the listener for the first window. This notifies the view root implementation that registered the listener.

[0039] Understandably, the view root implementations of multiple applications (such as multiple novel reading applications) can register listeners for their respective windows with the display engine service. The display engine service can then maintain a table mapping view root implementations to windows based on these registrations. Subsequently, after a window is detected, the registered view root implementation can be determined by looking up the table.

[0040] Secondly, this application provides an image processing method, comprising: displaying a first interface (as shown in Figure 18 below), the first interface including a video playback window (video playback window 1801 in Figure 18) and an image and text reading window (image and text reading window 1802 in Figure 18). Generating the fourth layer content of a second target layer associated with the video playback window, enhancing the blue channel values ​​of pixels in the fourth layer content during an image compositing stage to obtain a fifth layer content, and displaying the window content of the video playback window based on the fifth layer content. Generating the sixth layer content of a third target layer associated with the image and text reading window, enhancing the blue channel values ​​of pixels in the sixth layer content during an image rendering stage to obtain a seventh layer content, and displaying the window content of the image and text reading window based on the seventh layer content.

[0041] In summary, by adopting this application, for scenarios where a video playback window and an image / text reading window are running in the foreground, eye protection can be implemented for the video playback window during the image compositing stage. This ensures that the blue color in the video content layer is enhanced, achieving an eye-protecting effect. Furthermore, implementing eye protection for the image / text reading window during the image compositing stage not only ensures eye protection but also reduces the impact on device power consumption, further enhancing device performance.

[0042] In one possible design approach of the second aspect, enhancing the blue channel values ​​of pixels in the fourth layer content during the image compositing stage includes: enhancing the blue channel values ​​of pixels in the fourth layer content after the display duration of the video playback window has reached a first duration. Enhancing the blue channel values ​​of pixels in the fifth layer content during the image rendering stage includes: enhancing the blue channel values ​​of pixels in the fifth layer content after the display duration of the text and image reading window has reached a first duration.

[0043] In one possible design approach of the second aspect, generating the sixth layer content of the third target layer associated with the text and image reading window includes: rendering the sixth layer content in the third target layer. Enhancing the blue channel values ​​of pixels in the sixth layer content during the image rendering stage to obtain the seventh layer content includes: obtaining a second screenshot of the sixth layer content during the image rendering stage, fusing the second screenshot with a third shader to obtain the seventh layer content, wherein the third shader includes an algorithm for enhancing the blue channel values. Displaying the window content of the text and image reading window based on the seventh layer content includes: compositing the layer content of multiple layers to obtain and display the window content of the text and image reading window, wherein the layer content of the multiple layers includes the seventh layer content.

[0044] In one possible design approach of the second aspect, generating the fourth layer content of the second target layer associated with the video playback window includes: rendering and / or refreshing the fourth layer content in the second target layer. Enhancing the blue channel values ​​of pixels in the fourth layer content during the image compositing stage to obtain the fifth layer content includes: obtaining a third screenshot of the fourth layer content during the image compositing stage, correcting the third screenshot according to the resolution of the second display screen, and fusing the corrected third screenshot with a fourth shader to obtain the fifth layer content, wherein the fourth shader includes an algorithm for enhancing the blue channel values. Displaying the window content of the video playback window based on the fifth layer content includes: overlaying the layer content of multiple layers to obtain the window content of the video playback window, wherein the layer content of the multiple layers includes the fifth layer content.

[0045] In one possible design approach of the second aspect, before enhancing the blue channel values ​​of pixels in the fourth layer content during the image compositing stage, the method further includes: filtering invalid layers in the second target layer to obtain a filtered second target layer. The invalid layers include at least one of the following: a layer for drawing bullet screen content, a layer with a display frequency greater than a first frequency, and a layer with a buffer size greater than the resolution of a second display screen, where the second display screen is a display screen showing a video playback window. Enhancing the blue channel values ​​of pixels in the fourth layer content during the image compositing stage includes: during the image compositing stage, enhancing the blue channel values ​​of pixels in the fourth layer content of the filtered second target layer.

[0046] In one possible design approach of the second aspect, enhancing the blue channel values ​​of pixels in the fourth layer content during the image compositing stage includes: enhancing the blue channel values ​​of pixels in the fourth layer content based on the resolution of the second display screen, the blur radius of the second target layer, and the buffer size of the second target layer. Here, the blur radius indicates the sampling specifications when enhancing the blue channel values, and the second display screen is a display screen showing the video playback window.

[0047] In one possible design approach of the second aspect, the above method is applied to an electronic device, and the method further includes: correcting the blur radius based on the ambient light of the environment in which the electronic device is located. Enhancing the blue channel value of pixels in the content of a fourth layer based on the resolution of the second display screen, the blur radius of the second target layer, and the buffer size of the second target layer includes: enhancing the blue channel value of pixels in the content of the fourth layer based on the resolution of the second display screen, the corrected blur radius, and the buffer size of the second target layer.

[0048] It should be noted that the descriptions of the design methods in the second aspect above, such as filtering invalid layers, enhancing the parameters of the blue channel, and correcting the blur radius, are mainly based on the video playback window. In practice, the same applies to the text and image reading window, so they will not be elaborated here.

[0049] In addition, regarding the effects of the various possible design methods in the second aspect mentioned above, please refer to the relevant explanations in the first aspect and its possible design methods above, which will not be repeated here.

[0050] Thirdly, this application also provides an electronic device including a display screen, a memory, and one or more processors. The display screen, memory, and processors are coupled. The memory stores computer program code, which includes computer instructions. When executed by the processor, the computer instructions cause the electronic device to perform the methods of the first aspect, the second aspect, and any possible design thereof.

[0051] Fourthly, this application provides a chip system applied to an electronic device including a display screen and a memory; the chip system includes one or more interface circuits and one or more processors; the interface circuits and processors are interconnected via lines; the interface circuits are used to receive signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs a method as described in the first aspect, the second aspect, and any possible design of the present invention.

[0052] Fifthly, this application provides a computer storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform a method as described in the first aspect, the second aspect, and any possible design thereof.

[0053] Sixthly, this application provides a computer program product that, when run on a computer, causes the computer to perform a method as described in the first aspect, the second aspect, and any possible design thereof.

[0054] Understandably, the beneficial effects that the electronic device of the third aspect, the chip system of the fourth aspect, the computer storage medium of the fifth aspect, and the computer program product of the sixth aspect can achieve can be referred to the beneficial effects of the first aspect, the second aspect, and any possible design of them, which will not be repeated here. Attached Figure Description

[0055] Figure 1 is a hardware structure diagram of a mobile phone provided in an embodiment of this application;

[0056] Figure 2 is a software architecture diagram of a mobile phone provided in an embodiment of this application;

[0057] Figure 3 is a basic principle diagram of image processing provided in an embodiment of this application;

[0058] Figure 4 is a sampling principle diagram of fuzzy processing provided in an embodiment of this application;

[0059] Figure 5 is a flowchart of an eye protection algorithm provided in an embodiment of this application;

[0060] Figure 6 is a flowchart of another eye protection algorithm provided in an embodiment of this application;

[0061] Figure 7 is one of the effect diagrams of the image processing method provided in the embodiment of this application;

[0062] Figure 8 is one of the timing interaction diagrams of the image processing method in Scheme 1 provided in the embodiments of this application;

[0063] Figure 9 is one of the schematic diagrams of object invocation in the image processing method of Scheme 1 provided in the embodiments of this application;

[0064] Figure 10 is a second schematic diagram of object invocation in the image processing method of Scheme 1 provided in the embodiments of this application;

[0065] Figure 11 is a second timing interaction diagram of the image processing method in Scheme 1 provided in the embodiments of this application;

[0066] Figure 12 is a second example of the effect diagram of the image processing method provided in the embodiment of this application;

[0067] Figure 13 is one of the timing interaction diagrams of the image processing method in Scheme 2 provided in the embodiments of this application;

[0068] Figure 14 is one of the schematic diagrams of object invocation in the image processing method of Scheme 2 provided in the embodiments of this application;

[0069] Figure 15 is a second schematic diagram of object invocation in the image processing method of Scheme 2 provided in the embodiments of this application;

[0070] Figure 16 is a second timing interaction diagram of the image processing method in Scheme 2 provided in the embodiments of this application;

[0071] Figure 17 is one of the effect diagrams of combining Scheme 1 and Scheme 2 provided in the embodiments of this application;

[0072] Figure 18 is a second example of the combined effect of Scheme 1 and Scheme 2 provided in the embodiments of this application. Detailed Implementation

[0073] The technical solutions of the embodiments of this application are described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to limit the application. As used in the specification and appended claims of this application, the singular expressions "a," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one or more (including two). The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0074] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0075] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0076] The image processing method provided in this application can be applied to scenarios requiring eye protection when using electronic devices such as mobile phones and tablets (hereinafter referred to as eye protection scenarios). Taking a mobile phone as an example, eye protection scenarios can include scenarios where users spend a long time watching videos / binge-watching dramas on their phones (hereinafter referred to as video playback scenarios) or reading novels on their phones for extended periods (hereinafter referred to as reading scenarios). Specifically, in eye protection scenarios, during the processing of a single frame of an image, the electronic device can enhance the value of the blue channel of pixels in the image (hereinafter referred to as blue enhancement) through an eye protection algorithm, thereby achieving an eye protection effect.

[0077] For example, the aforementioned electronic devices may be mobile phones, tablets, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices, and / or smart city devices. This application does not impose any special limitations on the specific form of the electronic device.

[0078] Referring to Figure 1, which is a hardware structure diagram of an electronic device. As shown in Figure 1, taking a mobile phone as an example, the electronic device may include a processor 210, internal memory (RAM) 221, charging management module 240, power management module 241, battery 242, antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, audio module 270, display screen 294, and subscriber identification module (SIM) card interface 295, etc.

[0079] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the mobile phone. In other embodiments, the mobile phone may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0080] The processor 210 may include one or more processing units, such as an application processor (AP), a modem processor, a central processing unit (CPU), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0081] Internal memory 221 can be used to store computer executable program code, which includes instructions. Processor 210 performs various functions or data processing of the mobile phone by executing the instructions stored in internal memory 221.

[0082] In some embodiments, the processor 210 can execute an image processing method by running instructions stored in the internal memory 221.

[0083] The charging management module 240 receives charging input from the charger. The power management module 241 connects the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to power the processor 210, internal memory 221, display screen 294, etc.

[0084] The wireless communication function of a mobile phone can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor, and baseband processor.

[0085] The mobile phone can achieve display functions through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0086] In some embodiments, during the processing of a single frame of an image, the mobile phone can perform image rendering and image compositing using the GPU.

[0087] Furthermore, eye-protection algorithms can be implemented using shaders, which are small programs that run on the GPU and are used to handle specific tasks in image rendering and image compositing. Therefore, in practical implementation, the GPU can execute the shader of the eye-protection algorithm during image rendering or image compositing to achieve the eye-protection effect.

[0088] In some embodiments, after executing the image processing method proposed in this application, the display screen 294 can display bluish text and images to achieve an eye-protection effect.

[0089] Mobile phones can implement audio functions through audio modules 270, such as speakers, receivers, microphones, headphone jacks, and application processors. Examples include music playback and recording.

[0090] The software system of the aforementioned electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered architecture based on Android. TM Taking a system as an example, this section illustrates the software structure of an electronic device. A layered architecture divides the software system of an electronic device into several layers, each with a clear role and function, and the layers communicate with each other through software interfaces.

[0091] Referring to Figure 2, taking a mobile phone as an example, the hardware and software architecture of an electronic device can include an application (APP) layer, an application framework (framework) layer, a native layer, and a kernel layer, etc.

[0092] The application layer can accommodate applications such as video players, readers, email, memos, alarm clocks, settings, and calling.

[0093] During the operation of the aforementioned video player, reader, and other applications, the mobile phone can execute the image processing method provided in the embodiments of this application to achieve an eye-protection effect.

[0094] The application framework layer provides the application programming interface (API) and programming framework for the application layer.

[0095] The application framework layer can include various system services, such as Activity Manager Service (AMS) and Display Engine Service (DES).

[0096] The Activity Management Service can be used for starting, switching, and scheduling system components (such as activities, services, content providers, and broadcast receivers), as well as managing and scheduling application processes.

[0097] The display engine service can be used to monitor whether the conditions for turning the eye protection switch on or off are met (i.e., monitoring the switching conditions), such as monitoring window switching or the temperature of electronic devices. It should be noted that windows and activities are usually correlated; an activity creates a window. Therefore, monitoring window switching can also be understood as monitoring activity switching.

[0098] The display engine service can also control the eye protection switch to be turned on and off based on the monitoring results. When the eye protection switch is turned on, the phone will execute the eye protection algorithm to achieve the eye protection effect; when the eye protection switch is turned off, the phone will not execute the eye protection algorithm.

[0099] For example, if the display engine service detects that the video player or reader window has been running in the foreground for more than a preset time (duration 1 and duration 2 below), such as 5 minutes, the eye protection switch can be turned on, so that the eye protection effect can be achieved when watching videos or reading for a long time.

[0100] For another example, the display engine service can turn off the eye protection switch when the temperature of an electronic device is too high, thereby reducing the power consumption of the electronic device by turning off the eye protection switch when the temperature is too high.

[0101] Furthermore, the display engine service may include a window switching listener module and a callback management module.

[0102] The listening module can be used to listen for switching conditions, mainly window listening. After the switching conditions are met, it will send feedback to other modules (such as the image compositor (SurfaceFlinger, SF) and the view root implementation (ViewRootImpl, VRI)) to enable or disable eye protection.

[0103] The callback management module can be used by other modules to register listener callbacks. This is especially useful for modules that originally did not have a cross-process (binder) communication link with the display engine service, such as the view root implementation. By registering listener callbacks, cross-process communication can be achieved between the display engine service and these modules, such as the display engine service providing feedback to these modules based on the listener results.

[0104] In some embodiments, the mobile phone provides an eye protection setting. In response to the user's operation of opening the setting, the mobile phone will listen through the display engine service to see if the conditions for turning the eye protection switch on or off are met, and control the eye protection switch to be turned on or off based on the listening result.

[0105] The application framework layer may also include the ViewRootImpl (VRI) and view modules.

[0106] Each window corresponds to a VRI instance, which is responsible for managing and drawing the window content. For example, the VRI instance can be used to traverse the view hierarchy, also known as doing traversal, thereby enabling drawing and event dispatching.

[0107] The view module can be used to implement the rendering effects of each component in the window. For example, the view module can execute rendering effects.

[0108] The native layer provides various services to the upper layers (such as the application framework layer). For example, the native layer includes an image compositor (SurfaceFlinger, SF), a hardware-accelerated UI (HWUI), and a graphics library. It should be noted that SurfaceFlinger can also be translated as surface thrower, surface drawing module, image compositing service, etc., and this application does not specifically limit it in this way.

[0109] SurfaceFlinger can be used for refresh rate control, image compositing control, and more. For example, SurfaceFlinger can make decisions to assign compositing tasks to Hardware Composer (HWC) or the GPU.

[0110] Normally, HWC primarily performs simple compositing processes such as layer overlay. However, for compositing tasks involving special effects like rounded corners or blurring, SurfaceFlinger can assign the compositing task to the GPU to complete the compositing of these special effects.

[0111] In some embodiments, SurfaceFlinger can force the synthesis of each frame of image to be completed by the GPU, so that the GPU can execute eye protection algorithms during the image synthesis process to achieve an eye protection effect.

[0112] HWUI can be used for image rendering. For example, it can be used with the Skia graphics library (an open-source 2D graphics library) to implement graphics rendering. Specifically, HWUI can translate the application's drawing logic into GPU-executable rendering instructions, such as GL instructions, and send them to the GPU for execution.

[0113] In some embodiments, such as Scheme 2 below, HWUI can also send the eye protection algorithm to the GPU, so that the GPU can execute the eye protection algorithm during the image rendering process to achieve the eye protection effect.

[0114] Furthermore, HWUI can invoke render nodes and render pipelines (such as the Skia Pipeline, a mechanism for rendering using the Skia graphics library). The render node performs image filtering, while the Skia Pipeline performs image frame rendering.

[0115] The graphics library can provide drawing and manipulation of 2D and 3D graphics in the application. For example, the graphics library may include at least one of the following: Open Graphics Library (OpenGL), OpenGL for Embedded Systems (OpenGL ES), Vulkan, etc.

[0116] At this point, it should be noted that when an application is running, it can call services and functional modules provided in the underlying layers, such as the application framework layer and the local layer. For example, the application can call VRI to perform drawing and call HWUI to perform rendering. After the call, the resources occupied by these services and functional modules belong to the application process.

[0117] The kernel layer includes drivers that drive the hardware, such as display drivers and GPU drivers. Display drivers enable the screen to display images, such as images with eye-friendly features. GPU drivers enable the GPU to perform tasks, such as image rendering and image compositing.

[0118] The image processing method provided in this application embodiment can be executed in an electronic device having the above-described hardware and software structure.

[0119] Before detailing the image processing method provided in the embodiments of this application, the basic process of image processing will be briefly introduced below in conjunction with the hardware and software configuration of the electronic device:

[0120] Referring to Figure 3, on the application side, a blank layer can be created, and then drawing and rendering can be performed. The rendering instructions are executed through the GPU to draw content in the blank layer, resulting in a layer with content.

[0121] For example, an application process can obtain the drawing logic through VRI to implement image drawing.

[0122] For example, an application process can use HWUI to parse drawing logic and generate GPU-executable rendering instructions, such as GL instructions, to perform image rendering. For instance, HWUI can use a rendering pipeline (such as Skia Pipeline) to generate rendering instructions.

[0123] For example, an application can submit rendering instructions to the GPU through HWUI. The GPU executes the rendering instructions, and after passing through the geometry stage, rasterization stage, fragment shader, etc., the image content is drawn into the memory corresponding to the layer, thus obtaining a layer with content.

[0124] Understandably, multiple applications can adopt the above application-side process to draw content in a blank layer and obtain a layer with content.

[0125] Referring again to Figure 3, on the display system side, SurfaceFlinger can decide whether to use GPU compositing or HWC compositing. If HWC compositing is chosen, it can overlay multiple application-specific layers containing content to obtain an image frame, which is then sent to the display screen. If GPU compositing is chosen, the GPU can perform compositing processing, such as applying rounded corners and blurring effects, and overlay the layers to obtain an image frame, which is then sent to the display screen via HWC.

[0126] For example, the SurfaceFlinger Process (SF) can identify visible layers through SF (a primary object in the SF process).

[0127] For example, the SF process can output the information required for compositing to a rendering engine (such as the Skia Render Engine) or HWC via hardware output and output layer for image compositing. Specifically, if the decision uses GPU compositing, the output is sent to the rendering engine; if the decision uses HWC compositing, the output is sent to the HWC. It should be noted that the output is virtual to the screen; typically, each screen has a corresponding output.

[0128] Additionally, if GPU compositing is used, the SF process can also execute the create runtime effect shader flow through the rendering engine to create the shaders required for compositing and encapsulate them into compositing instructions (such as GL instructions) and send them to the GPU. Finally, the GPU executes the compositing instructions, performing image compositing through the geometry stage, rasterization stage, fragment shader, and so on.

[0129] This application will exemplarily illustrate the specific implementation of the eye protection algorithm in the following embodiments:

[0130] Electronic devices can perform blurring and assign the highest weight to the blue channel, that is, the weight of the blue channel is higher than that of the red and green channels, to sample the blue channel and enhance the blue color of each pixel in the image.

[0131] For example, if the number of sampling points is n (n≥2, n is an integer), then for any pixel in the image to be processed, the electronic device can calculate the color of the pixel (R', G', B') after blurring processing: (k1*(R+R1+R2+…+Rn) / (n+1), k2*(G+G1+G2+…+Gn) / (n+1), k3*(B+B1+B2+…+Bn) / (n+1)).

[0132] Among them, k1, k2 and k3 are the weights of the red (R) channel, green (G) channel and blue (B) channel respectively. k1, k2 and k3 all belong to (0,1), and k3>k1, k3>k2, that is, the blue channel has the highest weight.

[0133] Furthermore, k1, k2, and k3 can be constants or variables. In one specific implementation, k1, k2, and k3 are related to the resolution of the source data and the display resolution. The source data can be a video source or an image source; for example, in a video playback scenario, the source data is the video source.

[0134] For example, the weights (k1, k2, k3) are related to the ratio of the source data resolution to the display resolution (denoted as p). It's understandable that blurring reduces sharpness and affects visual experience. Furthermore, a larger p indicates a larger ratio of the source data resolution to the display resolution, meaning less detail is lost when displaying the image; conversely, a larger p indicates a smaller ratio, meaning more detail is lost. Based on this, a larger p allows electronic devices to set a larger k3, thus achieving a greater eye-protection effect, while correspondingly, k1 and k2 become smaller. Conversely, a smaller p allows electronic devices to set a smaller k3, thus ensuring greater image sharpness, while correspondingly, k1 and k2 become larger. In this way, eye protection can be achieved while maintaining image sharpness. For example, k3 = m * p, where m is a constant; a larger p results in a larger k3.

[0135] Where R, R1, R2...Rn are the color values ​​of the red channel of the pixel, the 1st sampling point, the 2nd sampling point...the nth sampling point, respectively; G, G1, G2...Gn are the color values ​​of the green channel of the pixel, the 1st sampling point, the 2nd sampling point...the nth sampling point, respectively; and B, B1, B2...Bn are the color values ​​of the blue channel of the pixel, the 1st sampling point, the 2nd sampling point...the nth sampling point, respectively.

[0136] Electronic devices can sample using convolution. The convolution kernel can be in the form of 15*15, 9*9, 3*3, etc., that is, the blur radius (denoted as rad) is (15 / 2)+1=8, (9 / 2)+1=5, (3 / 2)+1=2 respectively.

[0137] The blur radius can be a constant, or it can be a variable matched to different eye-protection scenarios. For example, in a scenario where video is played by player A, the blur radius is rad1, while in a scenario where video is played by player B, the blur radius is rad2, which is different from rad1. As another example, in a scenario where video is played by a video player, the blur radius is rad3, while in a scenario where a book is read on an e-reader, the blur radius is rad4, which is different from rad3.

[0138] Furthermore, electronic devices can perform continuous sampling or interval (skip) sampling. Continuous sampling means sampling at every point within the blur radius, while interval sampling means sampling at least one pixel interval within the blur radius. It can be understood that interval sampling reduces the number of samples, thereby reducing the computational load.

[0139] Taking a blur radius of 5 as an example, as shown in Figure 4, for pixel A in image 301, the electronic device can take pixel A as the center and sample 36 pixels (pixels filled with black in the figure) in a 9*9 area around it, i.e., n=36, and calculate the color (R', G', B') of pixel A after blurring.

[0140] In some embodiments, the electronic device can perform the above-mentioned fuzzing processing in Linear space or LMS (Long, Medium, Short) space, thereby simplifying the calculation. Here, in LMS, L represents Long wavelength, M represents Medium wavelength, and S represents Short wavelength, corresponding to three types of light waves that cone cells are sensitive to.

[0141] In one specific implementation, the electronic device performs the aforementioned blurring process in the Linear space. In this implementation, referring to Figure 5, the eye-protection algorithm includes the following steps:

[0142] Step 1: Convert RGB to Linear color space.

[0143] Electronic devices convert the RGB colors of each pixel to the Linear color space to obtain the color in the Linear color space. This process can be simply referred to as Gamma.

[0144] For example, an electronic device squares the color values ​​of each channel in the RGB color space to obtain the color in the Linear color space.

[0145] For example, an electronic device can convert the RGB color (R, G, B) of pixel A in Figure 5 to the Linear color space to obtain the color (x, y, z) of pixel A = (R... 2 G 2 B 2 ).

[0146] Furthermore, electronic devices can first normalize the RGB colors and then convert them to the Linear color space. For example, by converting the RGB colors (R, G, B) of pixel A in Figure 5 to the Linear color space, the electronic device can obtain the color of pixel A as (x, y, z) = ((R / 255)). 2 (G / 255) 2 (B / 255) 2 ).

[0147] Step 2, blurring.

[0148] Electronic devices perform blurring on each pixel in Linear space to obtain the blurred color.

[0149] For details, please refer to the previous explanation of blurring. You only need to replace the color values ​​of the R channel, G channel, and B channel with the corresponding color values ​​in the Linear color space. For example, (R', G', B')=(k1*(R+R1+R2+…+Rn) / (n+1), k2*(G+G1+G2+…+Gn) / (n+1), k3*(B+B1+B2+…+Bn) / (n+1)) can be replaced with (R', G', B')=(k1*(x+x1+x2+…+xn) / (n+1), k2*(y+y1+y2+…+yn) / (n+1), k3*(z+z1+z2+…+zn) / (n+1)). Where x, x1, x2...xn are the color values ​​of the pixel, the 1st sampling point, the 2nd sampling point...the nth sampling point in the first channel (e.g., the x channel) of the Linear space, respectively; y, y1, y2...yn are the color values ​​of the pixel, the 1st sampling point, the 2nd sampling point...the nth sampling point in the second channel (e.g., the y channel) of the Linear space, respectively; and z, z1, z2...zn are the color values ​​of the pixel, the 1st sampling point, the 2nd sampling point...the nth sampling point in the third channel (e.g., the z channel) of the Linear space, respectively.

[0150] For example, the electronic device samples the 36 black-filled pixels surrounding pixel A in Figure 5, performs blurring processing, and calculates the blurred color (R', G', B') of pixel A in Linear space.

[0151] Step 3: Convert Linear to RGB space.

[0152] Since blurring is performed in Linear color space, electronic devices need to convert the image back to RGB color before they can display the RGB color image. This process can be simply referred to as deGamma.

[0153] For example, an electronic device takes the square root (sqrt) of the color values ​​of each channel after blurring to obtain RGB colors.

[0154] For example, an electronic device can convert the blurred color (R', G', B') of pixel A in Figure 5 to RGB space to obtain the RGB color (R", G", B") of pixel A = (sqrtR', sqrtG', sqrtB').

[0155] In another specific implementation, the electronic device performs the aforementioned blurring process in the LMS space. In this implementation, referring to Figure 6, the eye protection algorithm includes the following steps:

[0156] Step 4: Convert RGB to LMS color space.

[0157] Electronic devices convert the RGB colors of each pixel to the LMS color space to obtain the color in the LMS space.

[0158] Electronic devices can convert RGB colors to the LMS color space using matrix multiplication. For example, RGB colors can be viewed as a 1x3 matrix, which is then multiplied by a 3x3 transformation matrix (denoted as M). 3*3 Then, a 1*3 matrix can be obtained, which can be converted to the color values ​​of the L, M and S channels in the LMS space respectively.

[0159] For example, if the RGB color of pixel A in Figure 6 is (R, G, B), then the color after conversion to LMS space is (L, M, S) = (R, G, B) * M. 3*3 .

[0160] For example, M 3*3 = [m11, m12, m13; m21, m22, m23; m31, m32, m33], where mij represents the value in the i-th row and j-th column of the matrix. The color can then be converted to the LMS color space using the following formula:

[0161] L = m11*R + m21*G + m31*B;

[0162] M = m1² * R + m2² * G + m3² * B;

[0163] S = m13*R + m23*G + m33*B.

[0164] Furthermore, during the conversion, red can be reduced while blue and / or green can be increased, thereby further enhancing the eye protection effect. For example, in the conversion matrix, m33 > m11, m22 > m11, so that blue and green can be increased as much as possible while red is reduced.

[0165] For example, M 3*3 One possible example is [0.201, 0.022882, 0.0; 0.735, 0.845, 0.010; 0.044, 0.092, 0.899], with corresponding L = 0.201*R + 0.735*G + 0.044*B, M = 0.022882*R + 0.845*G + 0.092*B, and S = 0.0*R + 0.010*G + 0.899*B.

[0166] Step 5, blurring.

[0167] Electronic devices perform blurring on each pixel in the LMS space to obtain the blurred color.

[0168] For details, please refer to the previous explanation of blurring. You only need to replace the color values ​​of the R channel, G channel, and B channel with the corresponding color values ​​in the LMS space. For example, (R', G', B')=(k1*(R+R1+R2+…+Rn) / (n+1), k2*(G+G1+G2+…+Gn) / (n+1), k3*(B+B1+B2+…+Bn) / (n+1)) can be replaced with (R', G', B')=(k1*(L+L1+L2+…+Ln) / (n+1), k2*(M+M1+M2+…+Mn) / (n+1), k3*(S+S1+S2+…+Sn) / (n+1)). Where L, L1, L2...Ln are the color values ​​of the pixel, the 1st sampling point, the 2nd sampling point...the nth sampling point in the LMS space, respectively; M, M1, M2...Mn are the color values ​​of the pixel, the 1st sampling point, the 2nd sampling point...the nth sampling point in the LMS space, respectively; and S, S1, S2...Sn are the color values ​​of the pixel, the 1st sampling point, the 2nd sampling point...the nth sampling point in the LMS space, respectively.

[0169] For example, the electronic device samples the 36 black-filled pixels surrounding pixel A in Figure 6, performs blurring processing, and calculates the blurred color (R', G', B') of pixel A in the LMS space.

[0170] Step 6: Convert LMS space to RGB.

[0171] Since the blurring process is performed in LMS space, the electronic device needs to convert it back to RGB color before it can display the RGB color image.

[0172] Similar to step 4, electronic devices can convert LMS color space back to RGB color using matrix multiplication. For example, the color (R', G', B') in LMS color space can be viewed as a 1x3 matrix, which is then multiplied by a 3x3 transformation matrix (denoted as N). 3*3 If we use this method, we can obtain a 1*3 matrix, which gives us the RGB colors (R′″, G′″, B′″).

[0173] For example, the electronic device converts the blurred color (R', G', B') of pixel A in Figure 6 to RGB space, and obtains the RGB color of pixel A as (R′″, G′″, B′″) = (R', G', B') * N. 3*3 .

[0174] For example, N 3*3= [4.028, -0.123, 0.001; 3.153, 1.123, 0.010; 0.092, 0.075, 1.200], and correspondingly, R′″ = 4.028*R' – 3.153*G' + 0.092*B', G′″ = -0.123*R' + 1.123*G' - 0.075*B', B′″ = 0.001*R' - 0.010*G' + 1.200*B'.

[0175] It should be noted that the two specific implementation methods shown in Figures 5 and 6 above can be combined. For example, an electronic device can first convert RGB to Linear space, then from Linear space to LMS space, perform blurring in LMS space, then convert back to Linear space, and finally convert back to RGB space.

[0176] Electronic devices can be configured with the aforementioned eye-protection algorithms. Subsequently, during image processing (such as image rendering or image compositing), the electronic devices can use the eye-protection algorithms to enhance blue to achieve an eye-protection effect.

[0177] Electronic devices can implement the aforementioned eye-protection algorithms as shaders. When the GPU in an electronic device performs image rendering or image compositing, it can execute shaders, thus enhancing the blue color in the processed image. Ultimately, the electronic device can display the image with enhanced blue, achieving an eye-protection effect.

[0178] The following sections will detail the proposed solution by using an eye-protection algorithm to enhance blue colors during image rendering and image compositing.

[0179] Option 1 involves using an eye-protection algorithm to enhance blue colors during the image synthesis process.

[0180] As discussed earlier regarding image compositing, electronic devices typically utilize the GPU for image compositing in scenarios requiring rounded corners or blurring effects. In scenarios primarily involving layer overlay, HWC (Hardware-Wide-Chip) can handle the compositing. Therefore, in Solution 1, the electronic device can be configured to use the GPU for image compositing, essentially forcing it to do so, and implementing an eye-protection algorithm during the compositing process. This achieves an eye-protection effect.

[0181] In some embodiments, the electronic device primarily applies eye-protection algorithms to behaviors that are particularly harmful to the eyes, achieving an eye-protection effect. Based on this, the electronic device can detect whether an eye-protection need exists. If an eye-protection need exists, the electronic device is configured to use the GPU for image composition. If no eye-protection need exists, the electronic device is not configured to use the GPU for image composition. This way, eye-protection needs can be addressed specifically, while reducing the GPU load when no eye-protection need exists.

[0182] For example, the process by which an electronic device detects whether there is a current need for eye protection includes:

[0183] The electronic device detects the duration of continuous activity in the foreground window. If the duration exceeds duration 1 (e.g., 5 minutes, 10 minutes), the electronic device can further determine if the window is a target window. The target window is a window relevant to eye protection scenarios, such as a video playback window or a text / image reading window (e.g., a novel). If the window is a target window, it indicates an eye protection need. Note that, to distinguish it from the target window in Scheme 2 below, the target window in Scheme 1 can be denoted as Target Window 1.

[0184] Conversely, if the duration of continuous activity of the foreground window does not exceed duration 1, or if the foreground window is not the target window 1, the electronic device can detect that there is no eye protection requirement.

[0185] In practice, multiple windows may be displayed simultaneously on the foreground of an electronic device. If some windows are not the target window 1, then there is no need to implement eye protection for those windows. For example, if there is a status bar window (used to display battery level, network signal, etc.) and a video playback window in the foreground, and the status bar window is not the target window 1, but the video playback window is the target window 1, then eye protection can be implemented only for the video playback window, without needing to implement eye protection for the status bar window.

[0186] Furthermore, when electronic devices use GPUs to perform image compositing, they can perform eye protection algorithms on the layer associated with the target window 1 (hereinafter referred to as the target layer), but not on other layers outside the target layer, thereby achieving an eye protection effect for the target window 1.

[0187] For example, an electronic device can set the target layer to use GPU to perform image compositing, while other layers are not set to use GPU to perform image compositing. When the GPU performs image compositing later, it can execute an eye-protection algorithm for the eye-protection layer that is set to use GPU to perform image compositing.

[0188] It should be noted that as long as one layer is set to use GPU to perform image compositing, the GPU will perform image compositing on all layers involved in the compositing. However, for layers that are not set to use GPU to perform image compositing, the GPU will not perform eye protection algorithms.

[0189] Additionally, it's important to note that not all blank layers require GPU rendering as shown in Figure 3 to become content-rich layers. For example, in a video playback scenario, after creating a blank layer, the video file can be decoded by a MediaCodec to obtain video frames. These frames can then be refreshed onto the blank layer using a SurfaceFlinger, resulting in a content-rich layer. This process does not require GPU rendering. In contrast, image compositing typically involves processing each frame of an image.

[0190] In other words, compared to image rendering, each frame of an image involves an image compositing process. Therefore, the first approach can be applied to the processing of each frame of an image, especially in video playback scenarios, where forced GPU compositing can achieve an eye-friendly effect.

[0191] Taking a video playback scenario as an example, as shown in Figure 7, before adopting Scheme 1, the electronic device can display interface 701, where the video images, such as people 7011 and objects 7012, are displayed normally. After adopting Scheme 2, the electronic device can display interface 702, where the edges of the video images, such as people 7011 and objects 7012, show a bluish effect, indicated by a gray shading in the figure, thus achieving an eye-protection effect.

[0192] It should be noted that the eye protection effect diagrams presented in this article, such as Figure 7 and Figures 12, 17 and 18 below, which show the eye protection effect represented by gray shading, are illustrative and do not limit the actual eye protection effect.

[0193] The following section provides a more detailed description of Solution 1, taking into account the hardware and software components of the electronic device. Specifically, as shown in Figure 8, Solution 1 includes the following steps:

[0194] S801, Application 1 receives the startup event of window A.

[0195] Application 1 can be any application in an electronic device, such as a video player, reader, caller, alarm clock, etc.

[0196] Window A can be any window displayed during the operation of application 1. For example, if application 1 is a video player, then window A could be the homepage window, the video playback window, etc.

[0197] The startup event is used to trigger the display of window A in the foreground.

[0198] For example, if window A is the homepage window of the video player, then the startup event is the event that starts the video player, such as the click event of the video player icon on the desktop.

[0199] For example, if window A is a video playback window, then the start event can be a video playback event, such as a click event on the video cover.

[0200] S802, Application 1 sends Creation Request 1 to the Activity Manager. Creation Request 1 is used to request the creation of Activity A corresponding to Window A.

[0201] After receiving the launch event of window A, application 1 needs to create an activity corresponding to window A. This activity can be used to manage window A, such as controlling the foreground / background switching of window A, operating the window, etc.

[0202] S803, Activity Manager creates Activity A.

[0203] S804. After switching to the foreground via the Activity A management window A, the Activity Manager sends a notification 1 to the Display Service Engine that window A has switched to the foreground.

[0204] In one specific implementation, the Activity Manager executes the onResume() method for Activity A, indicating that Window A has switched to the foreground, and then sends a notification to the Display Engine Service that Window A has switched to the foreground.

[0205] S805. After window A has been continuously active in the foreground for more than duration 1, the service engine checks whether window A is the target window 1. If so, execute S806.

[0206] For example, after receiving notification 1, the display engine service can time window A to obtain the duration of window A's continuous activity in the foreground.

[0207] In some embodiments, the display engine service can match the identifier of window A with a window whitelist, which records the identifier of target window 1. For example, the window whitelist records the identifiers of windows such as video playback windows and image / text reading windows. If the identifier of window A is in the window whitelist, it indicates that window A is target window 1; if the identifier of window A is not in the window whitelist, it indicates that window A is not target window 1.

[0208] For example, notification 1 may include an identifier for activity A, such as a name. The display service engine can determine the identifier of window A based on the identifier of activity A in notification 1. Those skilled in the art will understand that the identifier of activity A and the identifier of window A correspond; therefore, the display service engine can determine the identifier of window A through the identifier of activity A.

[0209] Of course, in actual implementation, notification 1 can also directly carry the identifier of window A, showing that after receiving notification 1, the service engine can match the identifier of window A in notification 1 with the window whitelist. This application embodiment does not specifically limit this.

[0210] If window A remains active in the foreground for more than duration 1, and window A is the target window 1, it indicates that some eye-straining windows are running in the foreground for a long time, and there is a need for eye protection. S807 can be executed further to achieve eye protection.

[0211] If window A remains active in the foreground for more than duration 1, but window A is not the target window 1, then there is no need for eye protection, and the display service engine does not need to perform subsequent eye protection processing for window A.

[0212] Additionally, if the duration of continuous activity of window A in the foreground does not exceed duration 1, the display engine service can continue timing for window A until the duration of continuous activity in the foreground exceeds duration 1, at which point it will further check whether window A is the target window 1. Alternatively, if the duration of continuous activity in the foreground still does not exceed duration 1 even after window A exits its foreground operation, the timing will end.

[0213] In one specific implementation, similar to when window A enters the foreground, the Activity Manager can send a notification 2 indicating that activity A has exited the foreground when the onPause() method is executed for activity A. In this way, the display engine service can determine that window A has exited the foreground based on notification 2.

[0214] Furthermore, if the duration of continuous activity of window A in the foreground does not exceed duration 1, or if window A is not the target window 1, the electronic device can process and display the window content of window A according to the normal processing flow.

[0215] For example, the layer content of the target layer associated with window A can be obtained through rendering and / or refreshing. It should be noted that if the target layer is a video content layer or other layer whose content does not require rendering, then refreshing may be necessary to obtain the layer content. Then, by compositing multiple layers (including the target layer), the window content of window A can be obtained and displayed.

[0216] It should be noted that after the electronic device is powered on and the display engine service is initialized, the display service engine can identify whether each window A has an eye protection requirement through the above S804-S805.

[0217] S806, The display engine service sends the layer whitelist of window A and the algorithm parameters of the eye protection algorithm to the image synthesizer. The algorithm parameters include indication information 1 and blur radius.

[0218] For ease of explanation, the information carrying the layer whitelist of the aforementioned window A and the algorithm parameters of the eye protection algorithm can be referred to as the first notification.

[0219] The layer whitelist includes the layer identifier of the target layer associated with window A, such as the layer name. It can be understood that the target layer associated with window A is the layer used to draw the content of window A.

[0220] For example, if window A is a video playback window, then the target layers in the layer whitelist include the video content layer, the bullet screen layer, the subtitle layer, etc.

[0221] For example, if window A is a text and image reading window, then the target layers in the layer whitelist include text layers, illustration layers, etc.

[0222] Among them, instruction information 1 is used to indicate that eye protection is turned on.

[0223] The fuzzy radius is used to indicate the sampling size, such as 15*15, 9*9, etc.

[0224] In practice, a configuration table can be set in the display engine service, which records the layer whitelist and blur radius corresponding to each target window 1. In this way, the display engine service can obtain the matching layer whitelist and blur radius based on window A.

[0225] In one specific implementation, the display service engine can establish a configuration table based on window identifiers and layer identifiers. Typically, the identifier of target window 1 and the layer identifier of the layer used to draw the window content of target window 1 share the same keyword. For example, the identifiers of a video playback window and the layers used to draw the window content of the video playback window, such as layers for video content, bullet comments, and subtitles, usually contain the keyword "video". Therefore, in the configuration table, windows and layers carrying the keyword "video" can be configured as a corresponding whitelist of target windows 1 and layers.

[0226] Furthermore, the display engine service can dynamically adjust the blur radius based on ambient light. Brighter ambient light significantly impacts image sharpness, and in this case, the display engine service can reduce the blur radius (found in the configuration table) to decrease the effect of the eye-protection algorithm on image sharpness. Conversely, darker ambient light has a relatively smaller impact on image sharpness, and in this case, the display engine service can increase the blur radius (found in the configuration table) to improve eye protection.

[0227] Of course, if the fuzzy radius is a constant, such as fixed at 9, then the fuzzy radius may not be included in the algorithm parameters.

[0228] S807, the image synthesizer sets the GPU compositing flag of the target layer to flag 1 based on the layer whitelist and algorithm parameters, and adds a blur radius to the layer properties of the target layer. The target layer is a layer included in the layer whitelist.

[0229] It is understandable that after receiving the layers with content drawn by each foreground application (each application has a corresponding window), the image compositer can decide to output the layers with visible content for image compositing.

[0230] On one hand, the image compositer can filter out target layers from layers with visible content based on a layer whitelist. Layers with visible content include target layers associated with window A, such as layer A1 and layer A2, as well as layers of other windows (such as the status bar window, other windows of application 1, and windows of other applications running in the foreground at the same time), such as layer B1 and layer C1. The layer whitelist only includes target layers associated with window A, such as layer A1 and layer A2 mentioned above.

[0231] The image manager matches layers with visible content against a layer whitelist. If a layer with visible content (such as layer A1 and layer A2 above) is in the whitelist, it can be selected as the target layer. If a layer with visible content (such as layer B1 and layer C1 above) is not in the whitelist, it can be selected as not a target layer. In this way, the image manager can determine the target layer for which the eye protection algorithm needs to be performed.

[0232] Furthermore, the image synthesizer can also filter out invalid layers by combining layer attributes, and perform eye protection algorithms only on valid target layers, thereby narrowing the range of eye protection layers and further improving the targeting of eye protection.

[0233] Invalid layers include at least one of the following: bullet screen layers, layers with a display frequency greater than frequency 1, and layers with a resolution greater than the screen resolution, which will be explained below.

[0234] First, the bullet screen layer.

[0235] The bullet comment layer displays the bullet comment content itself. Users typically focus on the video content and subtitles, not the bullet comment content. Therefore, the bullet comment layer usually doesn't require eye-protection algorithms. Furthermore, the display frequency of bullet comment content is usually high, potentially reaching 60Hz. Consequently, even higher frequencies are needed to execute eye-protection algorithms, resulting in high power consumption. Therefore, the image synthesizer can filter out the bullet comment layer, improving the targeting of eye protection while reducing power consumption.

[0236] In one specific implementation, the image compositor can filter out layers with a transparent background attribute, thereby filtering out the bullet comment layer. It's understood that the background of the bullet comment layer consists of transparent pixels to avoid obscuring the video content behind it; therefore, the image compositor can accurately filter out the bullet comment layer based on its background attribute.

[0237] It should be noted that, especially in the aforementioned implementation where the display service engine builds a configuration table based on window and layer identifiers, the bullet screen layer is typically in the layer whitelist of the video playback window. Therefore, the image compositer can correct the target layer determined based on the configuration table by filtering the bullet screen layer.

[0238] Second, display layers with a frequency greater than 1.

[0239] A higher display frequency for a layer indicates that the GPU needs to execute eye-protection algorithms at a higher frequency to obtain and display the image, resulting in higher power consumption. Based on this, the image synthesizer can filter out layers with display frequencies greater than 1 (such as 48Hz or 60Hz) in their layer attributes to reduce power consumption.

[0240] It's understandable that filtering out layers with display frequencies greater than the display frequency plus 1 can also filter out bullet screen layers.

[0241] Third, layers with a resolution greater than the display resolution.

[0242] The layer's resolution is equivalent to the layer's buffer size, which reflects the resolution of the data source, such as the video source. For a layer containing video content, its buffer size must be large enough to store the video content corresponding to the video source's resolution; this buffer size is equal to the video source's resolution. Furthermore, a higher layer resolution indicates a higher data source resolution, meaning the GPU needs to process more pixels and incur greater computational load when executing eye-protection algorithms.

[0243] Here, display resolution refers to the resolution of the display screen to be used. For example, if the display screen is small, the resolution is low; if the display screen is large, the resolution is high.

[0244] It's understandable that if the layer's resolution is greater than the display resolution, it means that after applying an eye-protection algorithm to the layer, further compression is needed to match the display resolution. This wastes some resources used for the eye-protection algorithm. For example, if the layer's resolution is 4K (4096*2160 pixels) and the display resolution is 2K (2048*1080 pixels), the GPU needs to apply the eye-protection algorithm to the 4096*2160 pixels and then compress it to 2048*1080 pixels before displaying it. This means approximately (4096*2160 - 2048*1080) pixels of resources are wasted.

[0245] Therefore, the image compositor can filter out layers whose resolution is greater than the display resolution, thus avoiding resource waste. For example, if the display resolution is 2k, then layers with a resolution higher than 2k should be filtered out.

[0246] On the other hand, image compositors can set layer properties for a target layer (or a valid target layer, the same below). For example, an image compositor can perform layer setting to set layer properties.

[0247] Setting layer properties includes setting the GPU composition flag to flag 1, thereby instructing the GPU to perform image composition. For example, flag 1 is true. For instance, the image compositor can perform layer setting to set the force client composition mode of the target layer to true, thus setting the GPU composition flag to flag 1.

[0248] After setting the flag to 1, the image compositor can decide whether to use the GPU to perform image compositing. Specifically, among the visible layers with content, if one layer has a GPU compositing flag set to 1, the image compositor can decide whether to use the GPU to perform image compositing.

[0249] Additionally, setting layer properties also includes setting the blur radius, which is used to determine the specifications of the blur processing later.

[0250] S808. When at least one layer is set with identifier 1, the image compositor generates an eye-friendly shader for the target layer based on the layer attributes and display resolution of the target layer.

[0251] The layer properties of the target layer include the target layer's resolution and blur radius.

[0252] Regarding display resolution, please refer to the relevant explanation in the S808 section above, which will not be repeated here.

[0253] The eye-protection shader includes an eye-protection algorithm to enhance the blue color of pixels in the layer content of the target layer.

[0254] The image synthesizer can calculate the weights for blurring the target layer using both the target layer's resolution and the display resolution. For example, k3 = m * p, where p is the ratio of the target layer's resolution to the display resolution. See the previous section on weight settings in eye-protection algorithms for details; they will not be repeated here. It should be noted that in some High Dynamic Range (HDR) scenes, the target layer's resolution is usually higher. Accordingly, when calculating the weights, the resolution corresponding to the HDR effect will be used, thus ensuring that the calculated weights match the HDR effect.

[0255] Of course, if the weight is a constant, the image synthesizer may also calculate the blur weights without based on the target layer's resolution and the display resolution.

[0256] The image synthesizer can input the weights and blur radius of the target layer into the eye protection algorithm to generate an eye protection shader for the target layer.

[0257] S809, The image compositor sends compositing instruction 1 to the GPU. Compositing instruction 1 includes eye protection instructions based on the target layer generated by the eye protection shader.

[0258] Among them, the compositing instruction 1 includes an eye-protection instruction for the target layer, which can instruct the GPU to perform an eye-protection algorithm on the target layer, such as performing an eye-protection algorithm on the content in the target layer to enhance blue. In other words, an eye-protection instruction is incorporated into the compositing instructions for each frame of the image.

[0259] Furthermore, the eye-protection instructions include a screenshot (makeSnapshot) instruction and a blending instruction. The screenshot instruction directs the GPU to take a screenshot of the original content of the target layer (i.e., content without the added eye-protection effect). The blending instruction directs the GPU to merge the screenshot with the eye-protection shader, allowing each pixel of the screenshot to be used as input to the eye-protection algorithm, resulting in an eye-protection output that enhances the blue channel values ​​of the pixels, thus achieving an eye-protection effect on the original content.

[0260] It should be noted that the video content is obtained by decoding the video source using a video codec, and then used as the layer content of the video content layer (a target layer). It is not rendered according to the buffer size (i.e., resolution) of the video content layer. Therefore, the resolution of the video content is very likely to be mismatched with the resolution of the display screen. Therefore, after obtaining a screenshot of the video content, the screenshot should be corrected according to the resolution of the display screen so that the corrected screenshot is the same as the resolution of the display screen. Then, it should be merged with the eye-care shader to obtain the layer content that corresponds exactly to the resolution of the display screen. This will not cause resource waste, nor will it result in layer content with a resolution lower than the resolution of the display screen.

[0261] The S810 and GPU execute synthesis instruction 1 to generate image 1.

[0262] In this way, when the GPU executes compositing instruction 1, it can perform an eye-protection algorithm on the target layer based on the target layer's eye-protection shader, thereby enhancing the blue color in the layer.

[0263] In the process of image synthesis, the GPU can use eye-protection algorithms in the eye-protection shader (such as the blurring algorithm mentioned above) to enhance the blue channel value of the pixels in the target layer. Then, through layer overlay, multiple layers (including the target layer of window A and the layers of other windows) are superimposed to obtain the window content of window A.

[0264] It should be noted that S807-S810 above pertains to the image compositing process. In practice, image processing steps such as rendering / refreshing can be performed before this to generate a layer with content. For details, please refer to the relevant explanation in Figure 3 above, which will not be repeated here.

[0265] S811, GPU sends image 1 to the display screen.

[0266] For example, the GPU can send image 1 to the display screen via HWC.

[0267] S812, The display screen shows image 1.

[0268] Image 1 includes window A, where the blue color in the window content of window A is enhanced.

[0269] Subsequently, during the processing of each frame of the image, steps S808-S812 can be executed repeatedly, enhancing the blue color in window A within each frame. It should be noted that in S808, after generating the eye-protection shader, the image synthesizer can store the shader in a cache. Subsequent executions of S808 can reuse the cached shader, eliminating the need to generate it for each frame and reducing the computational load of image processing. It should also be noted that in a scheme based on dynamically adjusting the blur radius according to ambient light, a new eye-protection shader can be generated for each frame based on the adjusted blur radius.

[0270] Furthermore, the process in Figure 8 above primarily uses window A as an example to illustrate the specific implementation of the eye-protection effect. In practice, a similar process can be used to achieve the eye-protection effect after starting any window in the foreground.

[0271] Understandably, after an electronic device sets the target layer of window A to use GPU image compositing, if window A exits its foreground (e.g., runs in the background or is closed), the target layer of window A and its related data are released. Consequently, the electronic device can no longer execute the eye-protection algorithm on the target layer of window A. In other words, after window A exits its foreground, the display engine service can cancel the execution of the eye-protection algorithm on the target layer of window A without sending a notification to the image compositor to disable eye protection.

[0272] The following section, in conjunction with the object invocation process, further details the specific implementation of S804-S809 in the process shown in Figure 8 above.

[0273] The above-mentioned S806 can be seen in ① of Figures 9 and 10. After recognizing the eye protection needs, the Display Engine Service (DES) in the system server process can send the layer whitelist and algorithm parameters to SF (a major object in the SF process) in the SF process.

[0274] The above S807 can be seen in ② of Figure 9. In the SF process, SF can perform layer setting based on the layer whitelist and algorithm parameters, set the GPU composition flag of the target layer to flag 1, and add a blur radius in the layer properties of the target layer.

[0275] Furthermore, see ② in Figure 10:

[0276] In the SF process, SF can output to the hardware, namely the virtual object on the display screen, the synchronization layer whitelist and algorithm parameters, so that the output can obtain the parameters needed for the processing of the next frame image.

[0277] Upon refreshing the next frame, SF can execute a commit process and a present process. In the commit process, SF can locate the output to be displayed, i.e., the display screen to be shown. For that output, SF can execute the present process. Through the present process, the corresponding output can determine the content to be displayed.

[0278] Next, Output can perform a preparation process, based on the layer whitelist and algorithm parameters obtained in the previous synchronization, to filter out the target layer and perform Layer Setting on the target layer, setting the GPU composition flag of the target layer to flag 1, and adding a blur radius to the layer properties of the target layer.

[0279] The process of filtering target layers includes: Output can traverse each layer (i.e., visible layers with content) output by Outputlayer, match the layer with the layer whitelist, and if the layer whitelist includes the layer, it indicates that the layer is the target layer.

[0280] The process of performing Layer Setting for the target layer includes: after determining that a certain layer is the target layer, Output can set the force client composition mode of the layer to true, thereby setting the GPU composition flag to flag 1.

[0281] The above S808 can be seen in ③-⑥ of Figure 9:

[0282] In Figure 9, ③, the SF process (such as Output) can decide to switch to GPU compositing based on identifier 1, that is, decide to use the GPU to perform image compositing.

[0283] Furthermore, referring to ③ in Figure 10, Output can determine the compositing method (updateCompositionState). By querying the compositing identifiers of each Layer output by OutputLayer, if a Layer's force client composition mode is set to true, it can decide to use the GPU to perform image compositing. Next, Output can execute the finish frame process, indicating that the necessary steps for a frame have been completed.

[0284] In Figure 9, ④, OutputLayer can output the algorithm parameters and buffer size (i.e., the layer resolution) of each layer. In Figure 9, ⑤, Output can output the display resolution. In Figure 9, ⑥, the rendering engine (such as Skia Render Engine) can generate an eye-friendly shader based on the above algorithm parameters, layer resolution, and display resolution.

[0285] Further, referring to ④-⑥ in Figure 10: The Render Engine can execute the Draw layer internal process, create a runtime effect shader, and wrap the eye-protection algorithm (an object that the SF process can call) into a shader object. The shader needs to bind some external parameters, such as the blur radius and blur weights, to achieve the eye-protection effect. Specifically, the process of binding external parameters includes: The Render Engine calculates the weights, such as calculating the blur weights in the eye-protection algorithm object based on the layer resolution and display resolution, including the aforementioned k1, k2, and k3. The Render Engine object initializes the eye-protection algorithm, such as by executing the buildRuntimeEffectShader process. The Render Engine binds the blur radius and weights to the eye-protection algorithm, such as by executing the createRuntimeEffectShader process. Finally, the eye-protection algorithm is parsed, such as by executing the build shader process, and wrapped into an eye-protection shader with bound external parameters.

[0286] The above-mentioned S809 can be seen in ⑦ of Figure 9. The Render Engine can submit the eye-protection shader to the GPU so that the algorithm takes effect during the compositing stage.

[0287] Furthermore, referring to ⑦ in Figure 10, the Render Engine can execute a Flush process to submit the eye-friendly shader to the GPU.

[0288] The S810 mentioned above can be seen in Figure 9, ⑧. The GPU can execute the eye-protection shader, thereby incorporating the eye-protection algorithm into the compositing stage to achieve the eye-protection effect.

[0289] Furthermore, after eye protection is enabled, the electronic device can also detect whether the conditions for disabling eye protection are met. If the conditions for disabling eye protection are met, the electronic device can terminate the execution of the eye protection algorithm via the GPU during image synthesis.

[0290] In one specific implementation, the condition for disabling eye protection includes the electronic device's temperature exceeding a temperature threshold. It's understood that increased temperature in electronic devices typically leads to performance degradation. Simultaneously, GPU execution of eye protection algorithms increases power consumption. Therefore, once the temperature threshold is exceeded, the electronic device can cease using the GPU to execute eye protection algorithms during image synthesis to achieve the eye protection effect, essentially canceling GPU-based image synthesis.

[0291] It should be noted that during the processing of a single frame of an image, the electronic device itself may need to use the GPU to perform image compositing, such as using the GPU to compose rounded corners. Therefore, ending the execution of eye protection algorithms through the GPU during image compositing does not equate to not using the GPU for image compositing.

[0292] Referring to Figure 11, Scheme 1 may also include the following steps:

[0293] S1101. The display engine service checks whether the conditions for exiting eye protection are met.

[0294] S1102. After detecting that the conditions for exiting eye protection are met, the display engine service sends instruction information 2 to the image synthesizer.

[0295] Among them, instruction message 2 indicates that eye protection should be turned off.

[0296] S1103, The image synthesizer sets the GPU compositing flag of the target layer to identifier 2 based on indication information 2.

[0297] Here, flag 2 indicates that image compositing is not performed using the GPU. For example, setting the force client composition mode of the target layer to false will set the GPU compositing flag of the target layer to flag 2.

[0298] If the GPU compositing flag is set to 2 for all layers output by the image compositor, then the default image compositing method, i.e., HWC compositing, can be used to complete the image compositing. See details in S1104-S1107 below:

[0299] S1104. When the GPU composition flags of all layers are flag 2, the image compositor sends composition instruction 2 to HWC. Composition instruction 2 does not include the eye protection shader.

[0300] S1105, HWC executes compositing instruction 2 to compose image 2.

[0301] For example, HWC overlays the layers to obtain image 2. During this process, HWC does not perform eye protection algorithms and does not enhance blues.

[0302] S1106, HWC sends image 2 to the display screen.

[0303] S1107, The display screen shows image 2.

[0304] For example, even if window A is not brought to the foreground, image 2 also includes window A, but the blue in window A is not enhanced, thus failing to achieve eye protection.

[0305] Option 2 involves using an eye-protection algorithm to enhance blue colors during image rendering.

[0306] Electronic devices can perform image rendering processing using the GPU. Based on this, electronic devices can implement eye-protection algorithms during the image rendering process on the GPU, thereby achieving an eye-protection effect.

[0307] In addition, the power consumption of GPUs performing image rendering is usually lower than that of performing image compositing. Consequently, performing eye protection algorithms during GPU image rendering has a smaller impact on GPU power consumption, which helps to ensure the performance of electronic devices.

[0308] Similar to the aforementioned Solution 1, in some embodiments, the electronic device primarily executes eye-protection algorithms for behaviors that are particularly harmful to the eyes, achieving an eye-protection effect. Based on this, the electronic device can detect whether an eye-protection need exists. If an eye-protection need exists, the electronic device executes the eye-protection algorithm during the GPU's image rendering process. If no eye-protection need exists, the electronic device does not execute the eye-protection algorithm during the GPU's image rendering process. This way, it can both specifically meet eye-protection needs and reduce the GPU load when there is no need for eye-protection.

[0309] For example, an electronic device can have a target window running in the foreground, and if the target window remains active in the foreground for more than duration 1, an eye protection requirement is detected. Similarly, to distinguish it from the target window in Scheme 1, the target window in Scheme 2 can be referred to as target window 2.

[0310] In reality, multiple windows may be displayed simultaneously on the foreground of an electronic device, and some of these windows are not the target window 2, so there is no need to implement eye protection for those windows.

[0311] Furthermore, when electronic devices use GPUs to perform image synthesis, they can apply eye protection algorithms to the layers involved in the target window 2 (hereinafter referred to as the target layer).

[0312] For details regarding the parts of Scheme 2 that were not explained in detail above, please refer to the previous explanation of Scheme 1; they will not be repeated here.

[0313] It should be noted that, as explained above, in scenarios involving video playback, the image compositor (SurfaceFlinger) can refresh the video frames onto a blank layer to obtain a layer with content; this process does not require GPU rendering. However, in scenarios other than video playback, such as reading through a reader, GPU participation in image rendering is typically required to obtain a layer with content. Therefore, Scheme 2 is primarily suitable for scenarios requiring GPU participation in image rendering; correspondingly, the target window 2 in Scheme 2 can differ from the target window 1 in Scheme 1.

[0314] Typically, target window 1 in Scheme 1 may include a video playback window and a text and image reading window; target window 2 in Scheme 2 may include a text and image reading window, but usually does not include a video playback window.

[0315] Taking a reading scenario as an example, as shown in Figure 12, before adopting Scheme 2, the electronic device can display interface 1201, and the text in interface 1201, such as "ABC", is displayed normally. After adopting Scheme 2, the electronic device can display interface 1202, and the text in interface 1202, such as "ABC", is surrounded by a bluish effect, which is represented by a gray shadow in the figure, thereby achieving an eye-protection effect.

[0316] The following section provides a more detailed description of Scheme Two, taking into account the hardware and software components of the electronic device. Specifically, as shown in Figure 13, Scheme Two includes the following steps:

[0317] S1301, Application 2 receives the startup event of window B.

[0318] Application 2 can be any application in the electronic device, and window B can be any window displayed during the operation of application 2.

[0319] S1302, Application 2 sends Creation Request 2 to Activity Manager. Creation Request 2 is used to request the creation of Activity B corresponding to Window B.

[0320] S1303, Activity Manager creates Activity B.

[0321] S1304. After switching to the foreground via the activity management window B, the activity manager sends a notification 3 to the display service engine that window B has switched to the foreground.

[0322] S1305. After window B has been continuously active in the foreground for more than duration 1, the service engine checks whether window B is the target window 2. If so, execute S1306.

[0323] It should be noted that the target window 2 in S1305 may be different from the target window 2 in S805 mentioned earlier. For example, the target window 2 in S1305 usually does not include the video playback window.

[0324] S1306, The display engine service sends the layer whitelist of window B and the algorithm parameters of the eye protection algorithm to the view root implementation. The algorithm parameters include indication information 1 and blur radius.

[0325] For ease of explanation, the information carrying the layer whitelist of the aforementioned window B and the algorithm parameters of the eye protection algorithm can be referred to as the second notification.

[0326] It's important to note that there was originally no cross-process (binder) communication link between the display engine service and the view root implementation. Therefore, application 2 can register a listener with the display engine service through the view root implementation, carrying the window identifier of target window 2 (e.g., window B) during registration. Subsequently, after the display engine service detects that the continuous activity of target window 2 exceeds duration 1, it can send back the layer whitelist and algorithm parameters to the view root implementation. This enables cross-process communication between the display engine service and the view root implementation.

[0327] Subsequently, after the lifecycle of target window 2 ends, such as after `doDie`, the view root implementation can unregister with the display engine service. In this way, the display engine service will no longer continue listening to target window 2.

[0328] In Option 2, if the eye-protection algorithm is executed during image rendering, then the algorithm should be integrated into the rendering process. Accordingly, the display engine service can send a layer whitelist and algorithm parameters to the view root implementation, allowing the view root implementation to integrate the eye-protection algorithm into the rendering process.

[0329] Understandably, a view root implementation corresponds to a window, and typically one window corresponds to one instance of a view root implementation. For example, window B corresponds to one instance of a view root implementation (let's call it view root implementation B). Correspondingly, the display service engine can send layer whitelists and algorithm parameters to the view root implementations that have registered listeners for the corresponding windows. For instance, if view root implementation B registers a listener for window B with the display engine service, then the layer whitelist and algorithm parameters are sent to view root implementation B.

[0330] For the parts not detailed in S1301-S1306 above, please refer to the explanations of S801-S806 above, which will not be repeated here.

[0331] S1307, The view root implements an eye-protection shader for the target layer based on the layer whitelist and algorithm parameters. The target layer is a layer included in the layer whitelist.

[0332] Unlike Scheme 1, Scheme 2 does not require mandatory GPU compositing, therefore it does not require setting the GPU compositing flag of the target layer to flag 1.

[0333] For example, the view root implementation can calculate the weights of the blur processing, bind the weights and blur radius to the eye protection algorithm, and generate eye protection shaders for each target layer.

[0334] For the parts not detailed in S1307 above, please refer to the explanation in S808 above, which will not be repeated here.

[0335] S1308, The view root implements setting the eye-friendly shader of the target layer to the hardware-accelerated image rendering component.

[0336] In this way, the hardware-accelerated image rendering component can subsequently generate rendering instructions that achieve eye-protection effects based on the eye-protection shader.

[0337] S1309, The view root implements sending drawing logic to the hardware-accelerated image rendering component 1.

[0338] After each frame refresh, the view root implementation can send drawing logic 1, allowing the GPU to render the new frame image.

[0339] S1310, the hardware-accelerated image rendering component processes the rendering logic 1 to obtain rendering instruction 1, which includes eye protection instructions for the target layer generated based on the eye protection shader.

[0340] For example, the hardware-accelerated image rendering component can generate a Skia-wrapped rendering instruction 1 from the drawing logic 1, which can be executed by the GPU.

[0341] Among them, rendering instruction 1 includes an eye-protection instruction for the target layer, which instructs the GPU to perform an eye-protection algorithm on the target layer, such as enhancing the blue color of the content drawn on the target layer. In other words, an eye-protection instruction is incorporated into the rendering instructions for each frame of the image.

[0342] Furthermore, the eye-protection instructions include a screenshot (makeSnapshot) instruction and a blending instruction. The screenshot instruction directs the GPU to take a screenshot of the original content rendered from the target layer (i.e., the content without eye-protection effects). The blending instruction directs the GPU to blend the screenshot with the eye-protection shader, allowing each pixel of the screenshot to be used as input to the eye-protection algorithm, resulting in an eye-protection output that enhances the blue channel values ​​of the pixels, thus achieving an eye-protection effect on the original content.

[0343] S1311, The hardware-accelerated image rendering component sends rendering instructions to the GPU.

[0344] S1312, GPU executes rendering instruction 1, draws each layer, and blends the eye-protection shader into the target layer.

[0345] Specifically, for the target layer, the GPU executes an eye-protection instruction to capture the original content of the target layer, and then integrates an eye-protection shader based on the original content of the target layer to achieve an eye-protection effect.

[0346] In this way, the GPU can enhance the blue in the target layer during the image rendering process.

[0347] At this point, image rendering is complete. Subsequent steps such as image compositing and display can be performed to ultimately obtain image 3 and send it to the display screen.

[0348] S1313, The display screen shows image 3.

[0349] Image 3 includes window B, where the blue color is enhanced.

[0350] Subsequently, during the processing of each frame of the image, the above S1309-S1312 can be executed repeatedly, so that the blue in window B in each frame of the image is enhanced.

[0351] Furthermore, the process in Figure 13 above primarily uses window B as an example to illustrate the specific implementation of the eye-protection effect. In practice, a similar process can be used to achieve the eye-protection effect after starting any window in the foreground.

[0352] Understandably, if window B exits its foreground operation (e.g., by running in the background or being closed), the target layer of window B and its related data are released. Consequently, the electronic device can no longer execute the eye-protection algorithm on the target layer of window B. In other words, after window B exits its foreground operation, the display engine service can cancel the execution of the eye-protection algorithm on the target layer of window B without sending a notification to the view root implementation to disable eye protection.

[0353] The following section will further explain the specific implementation of the process shown in Figure 13 above, taking into account the object invocation process.

[0354] Referring to (1) in Figure 14, prior to S1306, the view root implementation in the application process can register a listener with the display engine service in the system service process. For example, after a window is created, such as after executing setView, the view root implementation corresponding to that window can register a listener with the display engine service.

[0355] Furthermore, referring to (1) in Figure 15, the view root implementation can register a listener with the display engine service through an eye protection algorithm (an object of an eye protection algorithm that can be called by the application process). By registering a listener through an eye protection algorithm, the parameters required by the eye protection algorithm can be specified, so that the display engine service can provide feedback on parameters such as the blur radius after recognizing the existence of an eye protection requirement.

[0356] The above S1304 and S1305 can be seen in Figure 15 (8). After the on Resume() method is executed, making the window run in the foreground, the display engine service can listen for window switching. In this case, the display engine service can perform checks on window whitelist, temperature, time and other information to detect whether the eye protection requirements are met.

[0357] The above S1306 can be seen in (2) of Figures 14 and 15. After the display engine service detects that the continuous activity time of the target window 2 in the foreground exceeds the duration 1, it can send a feedback (callback) to the view root implementation of the layer whitelist and algorithm parameters corresponding to the target window 2.

[0358] The above S1307 can be seen in (3) of Figure 14. After receiving the layer whitelist and algorithm parameters, the view root implementation can generate the eye-protection shader of the target layer.

[0359] Furthermore, referring to (3) in Figure 15, the view root implementation can transmit the blur radius to the view, the view can receive the radius, and the view root implementation can set the eye protection shader through the view, such as by executing the apply effect to set the eye protection shader.

[0360] The above S1308 can be seen in (4) of Figure 14. The view root implementation can send the eye protection shader to the HWUI in the application process through the view.

[0361] For example, the view root implementation can manipulate the root node (DecorView, i.e., the view in Figure 14) to encapsulate the eye-protection shader as a runtime shader. By executing `setRenderEffect(createRuntimeShaderEffect(eye-protection shader))`, the eye-protection shader is set to the corresponding HWUI render node. For instance, the view can execute a render effect to parse the eye-protection shader into an image filter, which is then used as a filter for the render node in preparation for rendering.

[0362] The above-mentioned S1310 can be seen in (5) of Figure 14. The rendering pipeline in HWUI (such as Skia Pipeline) can traverse to each rendering node based on the view root and encapsulate the rendering instructions of each rendering node. Among them, if the rendering node has the effect of eye-protection shader, HWUI can also encapsulate eye-protection instructions, such as screenshot instructions and blending instructions.

[0363] The above S1311 can be seen in (6) of Figure 14. The rendering pipeline in HWUI can send rendering instructions to the GPU in the order of traversal, so that the algorithm takes effect during the rendering stage. For example, the rendering pipeline can execute the Flush process to send the rendering instructions to the GPU in sequence.

[0364] The above S1312 can be seen in (7) of Figure 14. The GPU can execute the eye-protection shader, thereby integrating the eye-protection algorithm into the rendering stage to achieve the eye-protection effect.

[0365] Similar to Option 1, after enabling eye protection, the electronic device can also detect whether the conditions for disabling eye protection are met. If the conditions for disabling eye protection are met, the electronic device can terminate the execution of the eye protection algorithm through the GPU during image rendering.

[0366] Referring to Figure 16, Scheme 2 may also include the following steps:

[0367] S1601. The display engine service checks whether the conditions for exiting eye protection are met.

[0368] S1602. After detecting that the conditions for exiting eye protection are met, the display engine service sends instruction information 2 to the view root implementation.

[0369] Among them, instruction message 2 indicates that eye protection should be turned off.

[0370] S1603, The view root implements the cancellation of the eye-protection shader set in the hardware-accelerated image rendering component based on instruction information 2.

[0371] For example, the view root implementation can send a cancellation command for the eye-care shader to the corresponding rendering node in the hardware-accelerated image rendering component, so that the corresponding rendering node can delete the eye-care shader.

[0372] S1604, The view root implements sending drawing logic 2 to the hardware-accelerated image rendering component.

[0373] Similarly, after each frame refresh, the view root implementation can send drawing logic 2, so that the GPU can render the new frame image.

[0374] S1605, the hardware-accelerated image rendering component obtains rendering instruction 2 based on drawing logic 2.

[0375] Since the eye protection shader has been removed, the rendering instructions 2 obtained from the hardware-accelerated image encapsulation no longer include eye protection instructions based on the eye protection shader.

[0376] S1606, The hardware-accelerated image rendering component sends rendering instructions to the GPU.

[0377] S1607, GPU executes rendering instruction 2 to draw each layer.

[0378] Since rendering instruction 2 does not include eye-care instructions, the GPU will not further fuse the eye-care shader onto the original content of the target layer, thus failing to achieve an eye-care effect.

[0379] Similarly, subsequent steps such as image compositing and display can be performed to finally obtain image 4 and send it to the display screen for display.

[0380] S1608, The display screen shows image 4.

[0381] If window B is still running in the foreground, then window B is also included in image 4, but the blue in window B is not enhanced.

[0382] Therefore, it should be noted that Scheme 1 and Scheme 2 can be combined. Below, using Scheme 1 (where the target window includes a video playback window) and Scheme 2 (where the target window includes a text and image reading window) as examples, several typical combination methods are introduced:

[0383] Combining Method 1 (see Figure 17(a)), when the electronic device is running a video playback window 1701 in the foreground, Method 1 can be used, allowing the GPU to perform an eye-protection algorithm on the target layers involved in the video playback window 1701 during image compositing, thus achieving an eye-protection effect when playing videos. Subsequently, after switching to Figure 17(b), i.e., when the electronic device is running a text and image reading window 1702 in the foreground, Method 2 can be used, allowing the GPU to perform an eye-protection algorithm on the target layers involved in the text and image reading window 1702 during image rendering, thus achieving an eye-protection effect when reading.

[0384] Of course, the same applies when switching from a text and image reading window running in the foreground to a video playback window running in the foreground.

[0385] Combining Method 2, as shown in Figure 18, if the electronic device runs a video playback window 1801 and a text and image reading window 1802 in a split-screen manner in the foreground, the electronic device can adopt Method 2, which allows the GPU to perform an eye protection algorithm on the target layer involved in the text and image reading window 1802 during the image rendering process, and can also adopt Method 1, which allows the GPU to perform an eye protection algorithm on the target layer involved in the video playback window 1801 during the image compositing process, thereby achieving an eye protection effect while playing videos and reading.

[0386] Of course, the electronic device can also run the video playback window and the text reading window in the foreground in the following ways: the text reading window runs in full screen, and the video playback window runs in a floating window; or, the video playback window runs in full screen, and the text reading window runs in a floating window, etc. This application does not make specific limitations on this.

[0387] By combining the above, electronic devices can adopt Scheme 2 for text and image reading windows, which can reduce the impact on GPU power consumption while achieving eye protection; and adopt Scheme 1 for video playback windows, which can achieve eye protection without GPU participation in image rendering.

[0388] In actual implementation, after detecting that the eye protection requirement is met, the electronic device can further decide whether to adopt Option 1 or Option 2, and achieve the eye protection effect based on the decision result. Specifically, the electronic device can make a decision based on the window whitelists of the target windows to which Option 1 and Option 2 apply respectively. If the current window (such as the video playback window 1701 mentioned above) is in the window whitelist of Option 1, then Option 1 is used to achieve the eye protection effect; if the current window (such as the video playback window 1702 mentioned above) is in the window whitelist of Option 2, then Option 2 is used to achieve the eye protection effect.

[0389] Furthermore, the window whitelist applicable to Scheme 2 can be a whitelist consisting of target windows that the view root implements to register for monitoring with the display engine service. The window whitelist applicable to Scheme 1 is a whitelist pre-configured in the display engine service. After detecting that the eye protection requirement is met, the display engine service can query whether the current window is a target window registered for monitoring. If it is a target window registered for monitoring, the display engine service can send the layer whitelist and algorithm parameters back to the view root implementation to achieve the eye protection effect through Scheme 2. If it is not a target window registered for monitoring, the display engine service can send the layer whitelist and algorithm parameters back to the image compositor to achieve the eye protection effect through Scheme 1.

[0390] Of course, if an electronic device has multiple windows running in the foreground, and at least two of these windows meet eye protection requirements, the electronic device can check the whitelist of each of the at least two windows.

[0391] If at least two windows are in the whitelist of Option 1, the electronic device can use Option 1 to achieve eye protection for at least two windows.

[0392] If at least two windows are in the whitelist of Option 2, the electronic device can use Option 2 to achieve eye protection for at least two windows.

[0393] If at least one part of two windows (such as the video playback window 1801 in Figure 18) is in the window whitelist of Scheme 1, and the other part (such as the text and image reading window 1802 in Figure 18) is in the window whitelist of Scheme 2, the electronic device can use Scheme 1 to achieve eye protection for one part of the windows and Scheme 2 to achieve eye protection for the other part of the windows.

[0394] This application also provides an electronic device, which may include a memory and one or more processors (such as a CPU, GPU, etc.). The memory and processor are coupled. The memory is used to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps performed by the device in the above method embodiments.

[0395] This application also provides a chip system including at least one processor and at least one interface circuit. The processor and the interface circuit are interconnected via lines. For example, the interface circuit can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit can be used to send signals to other devices (e.g., the processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application does not specifically limit this.

[0396] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the image processing method described above.

[0397] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the image processing method described above.

[0398] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the image processing methods in the above-described method embodiments.

[0399] In this embodiment, the electronic device, computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0400] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0401] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0402] The unit described as a separate component may or may not be physically separate. The component shown as a unit can be one physical unit or multiple physical units, that is, it can be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0403] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0404] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0405] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. An image processing method, characterized in that, The method includes: Run the first application, which includes a first window; Before the display duration of the first window reaches the first duration, generate the first layer content of the first target layer associated with the first window, and display the window content of the first window based on the first layer content; After the display duration of the first window reaches the first duration, the second layer content of the first target layer is generated, the value of the blue channel of the pixels in the second layer content is enhanced to obtain the third layer content, and the window content of the first window is displayed based on the third layer content.

2. The method according to claim 1, characterized in that, Enhancing the blue channel values ​​of pixels in the second layer content includes: Enhance the blue channel value of pixels in the second layer content during the image compositing or image rendering stage.

3. The method according to claim 2, characterized in that, Enhancing the blue channel value of pixels in the second layer content during the image compositing or image rendering stage includes: When the first window is a video playback window, the value of the blue channel of the pixels in the second layer content is enhanced during the image compositing stage; When the first window is a text and image reading window, the value of the blue channel of the pixels in the second layer content is enhanced during the image rendering stage.

4. The method according to claim 3, characterized in that, Before enhancing the blue channel values ​​of pixels in the second layer content during the image compositing stage, the method further includes: If the first window is a video window, set the GPU compositing flag of the first target layer to the first identifier; Based on the first identifier, it is determined that image synthesis will be performed using a GPU; During the image compositing stage, the blue channel values ​​of pixels in the second layer content are enhanced, including: The value of the blue channel of the pixels in the second layer content is enhanced during the image compositing stage performed by the GPU.

5. The method according to claim 3 or 4, characterized in that, After generating the second layer content of the first target layer, the method further includes: If the first window is neither the video playback window nor the text and image reading window, the blue channel value of the pixels in the second layer content is not enhanced, and the window content of the first window is displayed based on the second layer content.

6. The method according to any one of claims 2-5, characterized in that, The generation of the second layer content of the first target layer associated with the first window includes: The content of the second layer is obtained by rendering in the first target layer; During the image rendering stage, the blue channel values ​​of pixels in the second layer are enhanced to obtain the third layer content, including: During the image rendering stage, a first screenshot of the second layer content is obtained, and the first screenshot is fused with a first shader to obtain the third layer content. The first shader includes an algorithm to enhance the value of the blue channel. The step of displaying the window content of the first window based on the content of the third layer includes: The layer contents of multiple layers are combined to obtain the window content of the first window and displayed. The layer contents of the multiple layers include the content of the third layer.

7. The method according to any one of claims 2-5, characterized in that, The generation of the second layer content of the first target layer associated with the first window includes: The content of the second layer is obtained by rendering and / or refreshing in the first target layer; In the image compositing stage, the blue channel values ​​of pixels in the second layer are enhanced to obtain the third layer content, including: In the image compositing stage, a first screenshot of the second layer content is obtained, the first screenshot is corrected according to the resolution of the first display screen, and the corrected first screenshot is merged with the second shader to obtain the third layer content. The second shader includes an algorithm to enhance the value of the blue channel. The first display screen is the display screen that displays the first window. The step of displaying the window content of the first window based on the content of the third layer includes: The content of multiple layers is superimposed to obtain the window content of the first window and then displayed. The content of the multiple layers includes the content of the third layer.

8. The method according to any one of claims 1-7, characterized in that, Before enhancing the blue channel values ​​of pixels in the second layer content, the method further includes: Filter out invalid layers in the first target layer to obtain a filtered first target layer; wherein, the invalid layers include at least one of the following: a layer for drawing bullet screen content, a layer with a display frequency greater than a first frequency, and a layer with a buffer size greater than the resolution of the first display screen, wherein the first display screen is the display screen that displays the first window; Enhancing the blue channel values ​​of pixels in the second layer content includes: Enhance the blue channel value of pixels in the second layer content of the first target layer after filtering.

9. The method according to any one of claims 1-8, characterized in that, Enhancing the blue channel values ​​of pixels in the second layer content includes: Based on the resolution of the first display screen, the blur radius of the first target layer, and the buffer size of the first target layer, the value of the blue channel of the pixels in the content of the second layer is enhanced; The blur radius is used to indicate the sampling specification when enhancing the value of the blue channel, and the first display screen is the display screen that displays the first window.

10. The method according to claim 9, characterized in that, The blur radius is positively correlated with the degree of enhancement of the blue channel value; The first ratio is positively correlated with the enhancement level of the blue channel value, and the first ratio is the ratio of the buffer size of the first target layer to the resolution of the first display screen.

11. The method according to claim 9 or 10, characterized in that, The method is applied to an electronic device, and the method further includes: The blur radius is corrected based on the ambient light level of the environment in which the electronic device is located; The enhancement of the blue channel value in the content of the second layer based on the resolution of the first display screen, the blur radius of the first target layer, and the buffer size of the first target layer includes: Based on the resolution of the first display screen, the corrected blur radius, and the buffer size of the first target layer, the value of the blue channel of the pixels in the content of the second layer is enhanced.

12. The method according to claim 11, characterized in that, The step of correcting the blur radius of the first target layer based on the ambient light intensity of the environment in which the electronic device is located includes: The ambient light brightness is a first brightness, and the corrected blur radius is a first radius; The ambient light brightness is the second brightness, and the corrected blur radius is the second radius; Wherein, the first brightness is lower than the second brightness, and the first radius is greater than the second radius.

13. The method according to any one of claims 3-5, characterized in that, The method is applied to an electronic device, which includes a display engine service, an image synthesizer, and a view root implementation. When the first window is a video playback window, the blue channel value of pixels in the second layer content is enhanced during the image compositing stage, including: If the display engine service detects that the first window is a video playback window, the display engine service sends a first notification to the image synthesizer; In response to the first notification, the image synthesizer controls the enhancement of the blue channel value of pixels in the second layer content during the image compositing stage; When the first window is a text and image reading window, the blue channel value of the pixels in the second layer content is enhanced during the image rendering stage, including: If the display engine service detects that the first window is a text and image reading window, the display engine service sends a second notification to the view root implementation; In response to the second notification, the view root implementation controls the enhancement of the blue channel value of pixels in the second layer content during the image rendering phase.

14. The method according to claim 13, characterized in that, After running the first application, the method further includes: The view root implements the registration of the first window's listener with the display engine service, and the second type of window includes the window that the view root implements the registration of the listener; The display engine service sends a second notification to the view root implementation, including: The display engine service sends the second notification to the view root implementation that registered the listener for the first window.

15. An image processing method, characterized in that, The method includes: The first interface is displayed, which includes a video playback window and a text and image reading window; Generate the fourth layer content of the second target layer associated with the video playback window, enhance the blue channel value of the pixels in the fourth layer content during the image compositing stage to obtain the fifth layer content, and display the window content of the video playback window based on the fifth layer content; The sixth layer content associated with the third target layer of the image and text reading window is generated. During the image rendering stage, the blue channel value of the pixels in the sixth layer content is enhanced to obtain the seventh layer content. The window content of the image and text reading window is displayed based on the seventh layer content.

16. The method according to claim 15, characterized in that, Enhancing the blue channel values ​​of pixels in the fourth layer content during the image compositing stage includes: After the display duration of the video playback window reaches the first duration, the value of the blue channel of the pixels in the fourth layer content is enhanced during the image compositing stage; Enhancing the blue channel values ​​of pixels in the fifth layer content during the image rendering stage includes: After the display duration of the text and image reading window reaches the first duration, the value of the blue channel of the pixels in the fifth layer content is enhanced during the image rendering stage.

17. The method according to claim 15 or 16, characterized in that, The generation of the sixth layer content of the third target layer associated with the text and image reading window includes: The sixth layer content is obtained by rendering in the third target layer; During the image rendering stage, the blue channel values ​​of pixels in the sixth layer content are enhanced to obtain the seventh layer content, including: During the image rendering stage, a second screenshot of the content of the sixth layer is obtained, and the second screenshot is merged with the third shader to obtain the content of the seventh layer. The third shader includes an algorithm to enhance the value of the blue channel. The process of displaying the content of the image and text reading window based on the content of the seventh layer includes: The layer contents of multiple layers are combined to obtain the window content of the image and text reading window and displayed. The layer contents of the multiple layers include the content of the seventh layer.

18. The method according to any one of claims 15-17, characterized in that, The generation of the fourth layer content of the second target layer associated with the video playback window includes: The content of the fourth layer is obtained by rendering and / or refreshing in the second target layer; In the image compositing stage, the blue channel values ​​of pixels in the fourth layer content are enhanced to obtain the fifth layer content, including: In the image compositing stage, a third screenshot of the fourth layer content is obtained, the third screenshot is corrected according to the resolution of the second display screen, and the corrected third screenshot is merged with the fourth shader to obtain the fifth layer content. The fourth shader includes an algorithm to enhance the value of the blue channel. The process of displaying the video playback window content based on the content of the fifth layer includes: The content of the video playback window is obtained by overlaying the content of multiple layers, including the content of the fifth layer.

19. The method according to any one of claims 15-18, characterized in that, Before enhancing the blue channel values ​​of pixels in the fourth layer content during the image compositing stage, the method further includes: Filter out invalid layers in the second target layer to obtain a filtered second target layer; wherein, the invalid layers include at least one of the following: a layer for drawing bullet screen content, a layer with a display frequency greater than a first frequency, and a layer with a buffer size greater than the resolution of the second display screen, wherein the second display screen is a display screen that displays the video playback window; Enhancing the blue channel value of pixels in the fourth layer content during the image compositing stage includes: During the image compositing stage, the blue channel value of the pixels in the fourth layer content of the filtered second target layer is enhanced.

20. The method according to any one of claims 15-19, characterized in that, Enhancing the blue channel value of pixels in the fourth layer content during the image compositing stage includes: Based on the resolution of the second display screen, the blur radius of the second target layer, and the buffer size of the second target layer, the value of the blue channel of the pixels in the content of the fourth layer is enhanced; The blur radius is used to indicate the sampling specification when enhancing the value of the blue channel, and the second display screen is a display screen that shows the video playback window.

21. The method according to claim 20, characterized in that, The method is applied to an electronic device, and the method further includes: The blur radius is corrected based on the ambient light level of the environment in which the electronic device is located; The enhancement of the blue channel values ​​of pixels in the fourth layer content based on the resolution of the second display screen, the blur radius of the second target layer, and the buffer size of the second target layer includes: The blue channel values ​​of pixels in the fourth layer content are enhanced based on the resolution of the second display, the corrected blur radius, and the buffer size of the second target layer.

22. An electronic device, characterized in that, include: A display screen, one or more processors, and one or more memories; the one or more processors are coupled to the display screen and the one or more memories; the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1-21.

23. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-21.

24. A computer program product comprising computer instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-21.

Images