Noise monitoring method, electronic device and chip system

Abstract

This application provides a noise monitoring method, electronic device, and chip system, relating to the field of ambient light sensor technology, which can solve the problem of excessive power consumption in electronic devices. The detection method includes: an ambient light sensor in the electronic device collects ambient light at a collection cycle; before each ambient light collection, a memory write-back function is activated to obtain image noise during the ambient light collection period; after each ambient light collection ends, the memory write-back function is stopped to avoid the electronic device calculating image noise outside the ambient light collection period; power consumption is reduced by cyclically controlling the activation and deactivation of the memory write-back function. Since the noise interfering with ambient light may be related to the image displayed on the screen at the beginning of ambient light collection, the image can be forcibly refreshed after the memory write-back function is activated to obtain the currently displayed image on the screen, thereby obtaining the noise interfering with ambient light.

Description

[0001] This application is a divisional application of Chinese patent application filed on May 31, 2021, with application number 202110606261.8 and title "A noise monitoring method, electronic device and chip system". Technical Field

[0002] This application relates to the field of ambient light sensors, and more particularly to a control method for an electronic device, an electronic device, and a chip system. Background Technology

[0003] With the development of electronic devices, the screen-to-body ratio of these devices is increasing. To achieve the ultimate screen-to-body ratio, ambient light sensors can be placed beneath the OLED (Organic Light-Emitting Diode) screen. However, since the OLED screen itself emits light, the ambient light collected by the sensor placed beneath it includes light emitted by the OLED screen itself, leading to inaccuracies in the ambient light data collected by the sensor.

[0004] Currently, to accurately measure ambient light, one can obtain the ambient light collected by an ambient light sensor and the noise generated by the display screen of an electronic device. Then, the true ambient light is obtained based on the ambient light collected by the ambient light sensor and the noise generated by the display screen of the electronic device. In this method, the noise generated by the display screen of the electronic device is related to the image displayed on the screen; therefore, it is necessary to acquire the image displayed on the screen. However, acquiring the image and corresponding noise generated by the display screen consumes excessive power. Summary of the Invention

[0005] This application provides a control method for an electronic device, an electronic device, and a chip system, which solves the problem of excessive power consumption when the electronic device acquires noise.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] In a first aspect, embodiments of this application provide a noise monitoring method applied to an electronic device, the electronic device comprising: an HWC module, a display subsystem, and a noise algorithm library, the method comprising:

[0008] In response to receiving the first information, the HWC module sets the write-back flag to the first flag;

[0009] In response to receiving the first image, the HWC module queries the write-back flag to find that it is the first flag;

[0010] The HWC module sends the first image to the display subsystem based on the first marker;

[0011] The display subsystem stops storing a second image containing a first target image on the first image into the write-back memory of the electronic device, wherein the first target image is an image within a first region;

[0012] In response to the arrival of the first moment, the HWC module sets the write-back flag to the second flag;

[0013] The HWC module acquires the third image;

[0014] The HWC module queries the write-back flag and finds it to be the second flag;

[0015] The HWC module sends the third image and second information to the display subsystem based on the second tag. The second information is used to instruct the display subsystem to store the fourth image containing the second target image on the third image in the write-back memory of the electronic device.

[0016] In response to receiving the third image and the second information, the display subsystem stores a fourth image containing a second target image on the third image in the write-back memory of the electronic device, wherein the second target image is an image within the first region;

[0017] The HWC module obtains the second target image from the write-back memory;

[0018] The HWC module sends the second target image to the noise algorithm library;

[0019] The noise algorithm library calculates the noise of the first image based on the second target image.

[0020] In this embodiment, after the ambient light sensor finishes collecting ambient light each time, it can send first information to the AP processor via the SCP processor. On the AP processor side, the HWC module sets the write-back flag to the first flag, stopping the memory write-back function. If the electronic device refreshes the image, it will no longer acquire the target image of the refreshed image, nor will it calculate the image noise of the target image. Before the ambient light sensor collects ambient light again, for example, at the first moment, the HWC module can set the write-back flag to the second flag, and can also force a refresh of the image, for example, to the third image. When the write-back flag is set to the second flag, the memory write-back function is activated. If the image is refreshed, the target image of the refreshed image, i.e., the target image on the third image, can be obtained. The HWC will send the target image obtained from the third image to the noise algorithm library, which can calculate the image noise. In this way, it can be controlled that the HWC only acquires the target image of the currently refreshed image when there is a refreshed image during the ambient light sensor's acquisition of ambient light. Outside of the ambient light sensor's acquisition of ambient light, when there is a refreshed image, the HWC will no longer acquire the target image of the currently refreshed image. In practical applications, the write-back flag is used to set whether the HWC obtains the target image from the currently refreshed image. This application embodiment reduces power consumption by cyclically controlling the start and stop of the memory write-back function. In one possible implementation of the first aspect, the first information includes a first duration, which is the duration during which the display subsystem stops storing images to the write-back memory; the first moment is the moment after the first duration has elapsed since the write-back flag was set to the first flag.

[0021] Alternatively, the first information includes a first duration, a first value, and a second time, wherein the first duration is the duration during which the display subsystem stops storing images to the write-back memory, and the second time is the end time when the ambient light sensor of the electronic device collects the first value; the first time is the time after the second duration has elapsed since the write-back flag was set to the first flag, and the second duration is the first duration minus the delay duration, wherein the delay duration is the time when the HWC module receives the first information minus the second time duration.

[0022] In one possible implementation of the first aspect, the HWC module acquires the third image by:

[0023] The HWC module sends a first signal to the surface Flinger of the electronic device;

[0024] In response to receiving the first signal, the surface Flinger obtains the cached first display parameter and sends the first display parameter to the HWC module. The first display parameter is the latest cached display parameter among the display parameters cached by the surface Flinger.

[0025] The HWC module obtains the third image based on the first display parameters.

[0026] In one possible implementation of the first aspect, after the HWC module sets the write-back flag to the second flag and before the HWC module acquires the third image, the method further includes:

[0027] The HWC module obtains the time when the electronic device last refreshed the image;

[0028] If the time when the electronic device last refreshed the image meets the first preset condition, then the HWC module acquires the third image.

[0029] In one possible implementation of the first aspect, after the HWC module obtains the time of the last image refresh by the electronic device, it further includes:

[0030] If the last time the image was refreshed by the electronic device does not meet the first preset condition, the HWC module waits for the Surface Flinger module of the electronic device to send the second display parameters.

[0031] In one possible implementation of the first aspect, if the time of the last image refresh by the electronic device satisfies a first preset condition, then the HWC module acquires the first image, including:

[0032] If the time of the last image refresh by the electronic device meets the first preset condition, the HWC module waits for a second duration.

[0033] If the HWC module does not receive the third display parameter from Surface Flinger within the second time period, the HWC module acquires the first image.

[0034] In one possible implementation of the first aspect, the method further includes:

[0035] If the HWC module receives the fourth display parameter sent by Surface Flinger within the second time period, the HWC module obtains the fifth image based on the fourth display parameter.

[0036] The HWC module queries the write-back flag and finds it to be the second flag;

[0037] The HWC module sends the fifth image and the third information to the display subsystem based on the second marker;

[0038] In response to receiving the fifth image and the third information, the display subsystem stores a sixth image containing a third target image on the fifth image in the write-back memory of the electronic device, wherein the third target image is an image within the first region;

[0039] The HWC module retrieves the third target image from the write-back memory;

[0040] The HWC module sends the third target image to the noise algorithm library;

[0041] The noise algorithm library calculates the noise of the second image based on the third target image.

[0042] In one possible implementation of the first aspect, the first information includes a first value and a second time, wherein the second time is the end time when the ambient light sensor of the electronic device collects the first value;

[0043] The time when the electronic device last refreshed the image satisfies the first preset condition, including:

[0044] The last time the electronic device refreshed the image was later than the second time;

[0045] The electronic device's last image refresh time does not meet the first preset condition, including:

[0046] The time when the electronic device last refreshed the image is earlier than or equal to the second time.

[0047] In one possible implementation of the first aspect, the first information further includes a first value and a second time, wherein the second time is the end time when the ambient light sensor of the electronic device acquires the first value, and the time when the electronic device last refreshed the image satisfies a first preset condition including:

[0048] The first difference between the time when the electronic device last refreshed the image and the current time is less than the second difference between the second time and the current time;

[0049] The electronic device's last image refresh time does not meet the first preset condition, including:

[0050] The first difference between the time when the electronic device last refreshed the image and the current time is greater than or equal to the second difference between the second time and the current time.

[0051] In one possible implementation of the first aspect, the moment when the electronic device last refreshed the image satisfies a first preset condition includes:

[0052] The time when the electronic device last refreshed the image and the time when the HWC module last acquired the target image are both less than a first threshold; the target image is an image displayed on the screen located in the area above the ambient light sensor of the electronic device.

[0053] The electronic device's last image refresh time does not meet the first preset condition, including:

[0054] The time when the electronic device last refreshed the image and the time when the HWC module last acquired the target image are both greater than or equal to the first threshold.

[0055] In one possible implementation of the first aspect, the method further includes:

[0056] After the HWC module sets the write-back flag to the first flag, the HWC module monitors whether the data in the kernel node of the electronic device has changed, and the kernel node stores the brightness value.

[0057] In response to the detection of a change in data in the kernel node of the electronic device, the HWC module obtains a first brightness value from the kernel node;

[0058] After the HWC module obtains the first brightness value from the kernel node, in response to the detection of a change in the data in the kernel node of the electronic device, the HWC module obtains the second brightness value from the kernel node.

[0059] In response to the arrival of the first moment, the HWC module sends the second brightness value to the noise algorithm library.

[0060] In one possible implementation of the first aspect, the method further includes:

[0061] After the HWC module sets the write-back flag to the second flag, the HWC module monitors whether the data in the kernel node of the electronic device has changed, and the kernel node stores the brightness value;

[0062] In response to the detection of a change in data in the kernel node of the electronic device, the HWC module obtains a third brightness value from the kernel node;

[0063] The HWC module sends the third brightness value to the noise algorithm library;

[0064] After the HWC module sends the third brightness value to the noise algorithm library, in response to the detection of a change in the data in the kernel node of the electronic device, the HWC module obtains a fourth brightness value from the kernel node;

[0065] The HWC module sends the fourth brightness value to the noise algorithm library.

[0066] In one possible implementation of the first aspect, the noise algorithm library calculates the first image noise based on the second target image, including:

[0067] The noise algorithm library calculates the first image noise based on the second target image and the second brightness value.

[0068] In one possible implementation of the first aspect, the HWC module receiving the first image includes:

[0069] The HWC module receives the fifth display parameter sent by the Surface Flinger module of the electronic device;

[0070] The HWC module obtains the first image based on the fifth display parameter.

[0071] In one possible implementation of the first aspect, the first region is the area on the display screen of the electronic device located above the ambient light sensor of the electronic device.

[0072] In one possible implementation of the first aspect, before the HWC module acquires the moment when the electronic device last refreshed the image, it includes:

[0073] The HWC module receives the sixth display parameter sent by the Surface Flinger module of the electronic device;

[0074] The HWC module stores the moment when the HWC module receives the sixth display parameter;

[0075] The HWC module obtains the time of the last image refresh of the electronic device, including:

[0076] The HWC module acquires the stored time of receiving the sixth display parameter, which is the latest time of receiving the display parameter stored by the HWC module before acquiring the last time the image was refreshed on the electronic device.

[0077] In one possible implementation of the first aspect, the first display parameters include: the position, size, color, and storage address of the interface for synthesizing the third image on the display screen of the electronic device.

[0078] Secondly, embodiments of this application provide a noise monitoring method applied to an electronic device, the electronic device comprising: a first processor, the method comprising:

[0079] The first processor receives the first information;

[0080] After the first processor receives the first information, in response to receiving the first image, the first processor stops acquiring the first target image from the first image, where the first target image is an image within the first region;

[0081] After reaching the first moment, the first processor acquires the third image;

[0082] The first processor acquires a second target image from the third image, the second target image being an image within the first region.

[0083] In this embodiment, the ambient light sensor of the electronic device collects ambient light at a collection cycle. Before each ambient light collection, the memory write-back function is activated to obtain a target image of the refreshed image, thereby obtaining image noise during the ambient light collection period. After each ambient light collection, the memory write-back function is stopped to avoid the electronic device calculating image noise outside the ambient light collection period. Power consumption is reduced by cyclically controlling the activation and deactivation of the memory write-back function. Since the noise interfering with ambient light may be related to the image displayed on the screen at the beginning of ambient light collection, the image can be forcibly refreshed after the memory write-back function is activated, i.e., a third image can be obtained to acquire the image currently displayed on the screen, thereby obtaining the noise interfering with ambient light.

[0084] In one possible implementation of the second aspect, the method further includes:

[0085] In response to receiving the first information, the first processor sets the write-back flag to the first flag via the HWC module of the electronic device;

[0086] In response to receiving the first image, the first processor stopping the acquisition of the first target image from the first image includes:

[0087] In response to receiving the first image, the first processor queries the write-back flag as the first flag through the HWC module;

[0088] The first processor sends the first image to the display subsystem of the electronic device based on the first tag via the HWC module;

[0089] The first processor stops storing a second image containing a first target image on the first image in the write-back memory of the electronic device through the display subsystem; the first target image is an image within a first region.

[0090] The method further includes:

[0091] In response to the arrival of the first moment, the first processor sets the write-back flag to the second flag via the HWC module;

[0092] The first processor acquires a third image, and from the third image, the first processor acquires a second target image, the second target image being an image within the first region, including:

[0093] The first processor acquires the third image through the HWC module;

[0094] The first processor queries the write-back flag through the HWC module and finds it to be the second flag;

[0095] The first processor sends the third image and second information to the display subsystem based on the second tag through the HWC module. The second information is used to instruct the display subsystem to store the fourth image containing the second target image on the third image in the write-back memory of the electronic device.

[0096] In response to receiving the third image and the second information, the first processor stores a fourth image containing a second target image on the third image in the write-back memory of the electronic device through the display subsystem, wherein the second target image is an image within the first region;

[0097] The first processor obtains the second target image from the write-back memory through the HWC module;

[0098] The method further includes:

[0099] The first processor sends the second target image to the noise algorithm library through the HWC module;

[0100] The first processor calculates the first image noise based on the second target image using the noise algorithm library.

[0101] In one possible implementation of the second aspect, the first information includes a first duration, which is the duration during which the display subsystem stops storing images to the write-back memory; the first moment is the moment after the first duration has elapsed since the write-back flag was set to the first flag.

[0102] Alternatively, the first information includes a first duration, a first value, and a second time, wherein the first duration is the duration during which the display subsystem stops storing images to the write-back memory, and the second time is the end time when the ambient light sensor of the electronic device collects the first value; the first time is the time after the second duration has elapsed since the write-back flag was set to the first flag, and the second duration is the first duration minus the delay duration, wherein the delay duration is the time when the HWC module receives the first information minus the second time duration.

[0103] In one possible implementation of the second aspect, the first processor acquiring the first image via the HWC module includes:

[0104] The first processor sends a first signal to the surface Flinger of the electronic device through the HWC module;

[0105] In response to receiving the first signal, the surface Flinger obtains the cached first display parameter and sends the first display parameter to the HWC module. The first display parameter is the latest cached display parameter among the display parameters cached by the surface Flinger.

[0106] The HWC module obtains the third image based on the first display parameters.

[0107] In one possible implementation of the second aspect, after the first processor sets the write-back flag to the second flag via the HWC module, and before the first processor acquires the third image via the HWC module, the method further includes:

[0108] The first processor obtains the time of the last image refresh of the electronic device through the HWC module;

[0109] If the time when the electronic device last refreshed the image meets the first preset condition, then the first processor obtains the third image through the HWC module.

[0110] In one possible implementation of the second aspect, after the first processor obtains the time of the last image refresh of the electronic device through the HWC module, it further includes:

[0111] If the last time the image was refreshed by the electronic device does not meet the first preset condition, the first processor waits for the Surface Flinger module of the electronic device to send the second display parameters through the HWC module.

[0112] In one possible implementation of the second aspect, if the time of the last image refresh by the electronic device meets a first preset condition, then the first processor acquires the first image through the HWC module, including:

[0113] If the time when the electronic device last refreshed the image meets the first preset condition, the first processor waits for a second duration through the HWC module;

[0114] If the HWC module does not receive the third display parameter from Surface Flinger within the second time period, the first processor obtains the first image through the HWC module.

[0115] In one possible implementation of the second aspect, the method further includes:

[0116] If the HWC module receives the fourth display parameter sent by Surface Flinger within the second time period, the first processor obtains the fifth image based on the fourth display parameter through the HWC module.

[0117] The first processor queries the write-back flag through the HWC module and finds it to be the second flag;

[0118] The first processor sends the fifth image and the third information to the display subsystem based on the second tag via the HWC module;

[0119] In response to receiving the fifth image and the third information, the first processor stores a sixth image containing a third target image on the fifth image in the write-back memory of the electronic device through the display subsystem; the third target image is the image in the first region.

[0120] The first processor obtains the third target image from the write-back memory through the HWC module;

[0121] The first processor sends the third target image to the noise algorithm library of the electronic device through the HWC module;

[0122] The first processor calculates the second image noise based on the third target image using the noise algorithm library.

[0123] In one possible implementation of the second aspect, the first information includes a first value and a second time, wherein the second time is the end time when the ambient light sensor of the electronic device collects the first value;

[0124] The time when the electronic device last refreshed the image satisfies the first preset condition, including:

[0125] The last time the electronic device refreshed the image was later than the second time;

[0126] The electronic device's last image refresh time does not meet the first preset condition, including:

[0127] The time when the electronic device last refreshed the image was earlier than or equal to the second time;

[0128] Alternatively, the moment when the electronic device last refreshed the image satisfies the first preset condition includes:

[0129] The first difference between the time when the electronic device last refreshed the image and the current time is less than the second difference between the second time and the current time;

[0130] The electronic device's last image refresh time does not meet the first preset condition, including:

[0131] The first difference between the time when the electronic device last refreshed the image and the current time is greater than or equal to the second difference between the second time and the current time;

[0132] or,

[0133] The time when the electronic device last refreshed the image satisfies the first preset condition, including:

[0134] The time when the electronic device last refreshed the image and the time when the HWC module last acquired the target image are both less than a first threshold; the target image is an image displayed on the screen located in the area above the ambient light sensor of the electronic device.

[0135] The electronic device's last image refresh time does not meet the first preset condition, including:

[0136] The time when the electronic device last refreshed the image and the time when the HWC module last acquired the target image are both greater than or equal to the first threshold.

[0137] In one possible implementation of the second aspect, the method further includes:

[0138] After the first processor sets the write-back flag to the first flag through the HWC module; the first processor monitors whether the data in the kernel node of the electronic device has changed through the HWC module, and the kernel node stores the brightness value;

[0139] In response to the detection of a change in data in the kernel node of the electronic device, the first processor obtains a first brightness value from the kernel node through the HWC module;

[0140] After the first processor obtains a first brightness value from the kernel node through the HWC module, in response to detecting a change in the data in the kernel node of the electronic device, the first processor obtains a second brightness value from the kernel node through the HWC module.

[0141] In response to the arrival of the first moment, the first processor sends the second brightness value to the noise algorithm library through the HWC module.

[0142] In one possible implementation of the second aspect, the method further includes:

[0143] After the first processor sets the write-back flag to the second flag through the HWC module, the first processor monitors whether the data in the kernel node of the electronic device has changed through the HWC module. The kernel node stores the brightness value.

[0144] In response to the detection of a change in data in the kernel node of the electronic device, the first processor obtains a third brightness value from the kernel node through the HWC module;

[0145] The first processor sends the third brightness value to the noise algorithm library through the HWC module;

[0146] After the first processor sends the third brightness value to the noise algorithm library through the HWC module, in response to the detection of a change in the data in the kernel node of the electronic device, the first processor obtains a fourth brightness value from the kernel node through the HWC module.

[0147] The first processor sends the fourth brightness value to the noise algorithm library through the HWC module.

[0148] In one possible implementation of the second aspect, the first processor calculates the first image noise based on the second target image using the noise algorithm library, including:

[0149] The first processor calculates the first image noise based on the second target image and the second brightness value using the noise algorithm library.

[0150] In one possible implementation of the second aspect, the first processor receiving the first image via the HWC module includes:

[0151] The first processor receives the fifth display parameter sent by the Surface Flinger module of the electronic device through the HWC module;

[0152] The first processor obtains the first image based on the fifth display parameter through the HWC module.

[0153] In one possible implementation of the second aspect, the first region is the area on the display screen of the electronic device located above the ambient light sensor of the electronic device.

[0154] In one possible implementation of the second aspect, before the first processor obtains the time of the last image refresh of the electronic device through the HWC module, it includes:

[0155] The first processor receives the sixth display parameter sent by the Surface Flinger module of the electronic device through the HWC module;

[0156] The first processor stores the moment when the HWC module receives the sixth display parameter through the HWC module;

[0157] The first processor obtains the time of the last image refresh of the electronic device through the HWC module, including:

[0158] The first processor obtains the time when the sixth display parameter is received through the HWC module. The time when the sixth display parameter is received is the latest time when the HWC module receives the display parameter before the last time the image was refreshed on the electronic device.

[0159] In one possible implementation of the second aspect, the first display parameters include: the position, size, color, and storage address of the interface for synthesizing the third image on the display screen of the electronic device.

[0160] Thirdly, an electronic device is provided, including a processor for running a computer program stored in a memory to implement the method of any one of the first aspects or the method of any one of the second aspects of this application.

[0161] Fourthly, a chip system is provided, including a processor coupled to a memory, wherein the processor executes a computer program stored in the memory to implement the method of any of the second aspects of this application.

[0162] Fifthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by one or more processors, implements the method of any one of the first or second aspects of this application.

[0163] Sixthly, embodiments of this application provide a computer program product that, when run on a device, causes the device to execute the method of any one of the first aspects or any one of the second aspects of this application.

[0164] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0165] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0166] Figure 2 A positional relationship diagram of an ambient light sensor and a display screen in an electronic device provided in this application embodiment;

[0167] Figure 3 Another positional relationship diagram of the ambient light sensor and the display screen in the electronic device provided in the embodiments of this application;

[0168] Figure 4 Another positional relationship diagram of the ambient light sensor and the display screen in the electronic device provided in the embodiments of this application;

[0169] Figure 5 A positional relationship diagram of the target area on the display screen provided in this application embodiment;

[0170] Figure 6 A positional relationship diagram of an ambient light sensor and a target area on a display screen provided in an embodiment of this application;

[0171] Figure 7 A technical architecture diagram on which the ambient light detection method provided in the embodiments of this application depends;

[0172] Figure 8 A schematic diagram illustrating the ambient light acquisition cycle of the ambient light sensor provided in this application embodiment;

[0173] Figure 9 for Figure 8 A schematic diagram of the time nodes for image refresh and backlight adjustment within one acquisition cycle in the embodiment shown.

[0174] Figure 10 Based on Figure 7 The diagram shows a timing flowchart of an ambient light detection method implemented using the technical architecture shown.

[0175] Figure 11 for Figure 10 The illustrated embodiment shows the timing flowchart between the various modules in the AP processor provided in this application embodiment;

[0176] Figure 12 Another technical architecture diagram on which the ambient light detection method provided in the embodiments of this application depends;

[0177] Figure 13 Based on Figure 12 The illustrated technical architecture provides another timing flowchart for the ambient light detection method.

[0178] Figure 14 Based on Figure 9 The illustrated embodiment provides a schematic diagram of the calculation of integral noise for image noise and backlight noise at each time point;

[0179] Figure 15 A schematic diagram of each time node for image refresh and backlight adjustment along the time axis within an acquisition cycle provided in this application embodiment;

[0180] Figure 16 Based on Figure 15 The illustrated embodiment provides a schematic diagram of the calculation of integral noise for image noise and backlight noise at each time point;

[0181] Figure 17 A schematic diagram illustrating a start / stop scheme for CWB write-back function provided in an embodiment of this application;

[0182] Figure 18 This is a schematic diagram illustrating the process of the SCP processor transmitting first information to the AP processor according to an embodiment of this application.

[0183] Figure 19 A schematic diagram illustrating the start / stop scheme for the CWB write-back function of the forced image refresh provided in an embodiment of this application;

[0184] Figure 20 To adopt Figure 19 The implementation shown is a schematic diagram of events at various times for the start and stop scheme of CWB write-back function.

[0185] Figure 21 To adopt Figure 19 and Figure 20 The illustrated embodiment provides a schematic diagram of the integrated noise obtained by the start / stop scheme of the CWB write-back function.

[0186] Figure 22 A schematic diagram illustrating the refresh state and idle state of the display screen provided in an embodiment of this application;

[0187] Figure 23 A scheme for starting and stopping the CWB write-back function that forces image refresh when the display screen is idle for a long time, as provided in the embodiments of this application.

[0188] Figure 24 The adoption of the embodiments provided in this application Figure 23The scheme for starting and stopping the CWB write-back function of the forced image refresh is shown.

[0189] Figure 25 The adoption of the embodiments provided in this application Figure 17 The CWB write-back function start / stop scheme is shown below;

[0190] Figure 26 A schematic flowchart illustrating another CWB write-back function start / stop scheme provided in an embodiment of this application;

[0191] Figure 27 for Figure 26 A schematic diagram illustrating the method for determining whether to force an image refresh in the illustrated embodiment;

[0192] Figure 28 An embodiment of this application provides an example of using... Figure 26 The diagram shows the integrated noise obtained by the start-stop scheme.

[0193] Figure 29 Another approach provided for embodiments of this application Figure 26 The diagram shows the integrated noise obtained by the start-stop scheme.

[0194] Figure 30 Another approach provided for embodiments of this application Figure 26 The diagram shows the integral noise obtained by the start-stop scheme. Detailed Implementation

[0195] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details.

[0196] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0197] It should also be understood that in the embodiments of this application, "one or more" refers to one, two, or more; "and / or" describes the relationship between the associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0198] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," "fourth," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0199] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0200] The noise monitoring method provided in this application can be applied to electronic devices equipped with OLED screens. These electronic devices can be tablets, wearable devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other similar devices. This application does not limit the specific type of electronic device.

[0201] Figure 1 A schematic diagram of an electronic device is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a touch sensor 180K, an ambient light sensor 180L, etc.

[0202] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0203] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. For example, processor 110 is used to execute the noise monitoring method in the embodiments of this application.

[0204] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0205] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0206] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0207] USB interface 130 is an interface that conforms to the USB standard specification, specifically it can be a Mini USB interface, Micro USB interface, USB Type C interface, etc. USB interface 130 can be used to connect a charger to charge electronic device 100, and it can also be used for data transfer between electronic device 100 and peripheral devices.

[0208] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0209] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function. The data storage area may store data created during the use of electronic device 100.

[0210] In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0211] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0212] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance).

[0213] In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may also be located in the same device.

[0214] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0215] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0216] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1.

[0217] In some embodiments, at least some functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be disposed in the same device.

[0218] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0219] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0220] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. Wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. GNSS can include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0221] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0222] The audio module 170 is used to convert digital audio signals into analog audio signals for output, and also to convert analog audio inputs into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0223] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0224] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0225] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to monitoring voice information, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc. For example, microphone 170C can be used to collect voice information involved in the embodiments of this application.

[0226] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0227] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When a force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the touch operation intensity based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A.

[0228] In some embodiments, touch operations applied to the same touch location but with different touch intensity can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS message is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS message is executed.

[0229] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shake of the electronic device 100, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing game scenarios.

[0230] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover using the magnetic sensor 180D. Then, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0231] The 180E accelerometer can detect the magnitude of acceleration of electronic device 100 in various directions (typically three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to applications such as screen orientation switching and pedometers.

[0232] A distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, electronic device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0233] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 100 emits infrared light outward through the LED. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 may use the proximity sensor 180G to detect when a user holds the electronic device 100 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and locking of the screen.

[0234] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0235] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0236] The ambient light sensor 180L is used to sense the brightness of ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0237] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0238] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0239] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0240] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0241] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be an organic light-emitting diode (OLED). In some embodiments, electronic device 100 may include one or N displays screens 194, where N is a positive integer greater than 1.

[0242] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0243] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0244] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0245] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0246] This application does not specifically limit the structure of the execution entity for noise monitoring. Any entity can be configured to perform noise monitoring according to the method provided in this application by running a program that records a noise monitoring method according to this application. For example, the execution entity of the noise monitoring method provided in this application can be a functional module in an electronic device capable of calling and executing a program, or a communication device applied in an electronic device, such as a chip.

[0247] Figure 2 A frontal positional relationship diagram of the display screen and ambient light sensor in an electronic device provided in this application embodiment.

[0248] like Figure 2 As shown, the ambient light sensor's projection onto the electronic device's display screen is located in the upper half of the screen. When a user holds the electronic device, the ambient light sensor in the upper half detects the light intensity and color temperature of the environment in front of the device (the orientation of the display screen within the device). This light intensity and color temperature are used to adjust the brightness and color temperature of the display screen, thus achieving a better visual effect. For example, in a dark environment, the display screen will not be too bright to cause glare, and in a bright environment, the display screen will not be too dark to be easily visible.

[0249] like Figure 3The diagram shows the side positional relationship between the display screen and the ambient light sensor in an electronic device. The display screen of the electronic device, from top to bottom, includes: a glass cover (transparent), a display module, and a protective film. Here, "top" and "bottom" refer to the orientation when the display screen is placed upwards. Since the ambient light sensor needs to collect ambient light from above the display screen, a portion of the display module can be removed to house the ambient light sensor. This means the ambient light sensor is positioned below the glass cover and on the same layer as the display module. It should be noted that the detection direction of the ambient light sensor and the orientation of the display screen within the electronic device (…) Figure 3 The orientation of the central display screen in electronic devices is consistent with that of the upper part. Clearly, this placement of the ambient light sensor sacrifices a portion of the display area. When pursuing a high screen-to-body ratio, this ambient light sensor placement will no longer be suitable.

[0250] like Figure 4 The image shows another configuration of the ambient light sensor provided in this application embodiment. The ambient light sensor is moved from below the glass cover to below the display module. For example, the ambient light sensor is located below the active area (AA) of the OLED screen display module, where image content can be displayed. This configuration does not sacrifice the display area. However, OLED screens are self-emissive displays; when an image is displayed, the user can see the image from above. Similarly, the ambient light sensor located below the OLED screen can also collect the light corresponding to the image displayed on the OLED screen. Therefore, the ambient light collected by the ambient light sensor includes the light emitted by the display screen and the actual ambient light. To accurately obtain the actual ambient light, in addition to obtaining the ambient light collected by the ambient light sensor, it is also necessary to obtain the light emitted by the display screen.

[0251] pass Figure 4 It is understandable that, since the ambient light sensor is located below the AA area, its placement does not sacrifice the AA area in the display module. Therefore, the projection of the ambient light sensor onto the display screen can be located in any area of ​​the front of the display screen, and is not limited to the following setting: the projection of the ambient light sensor onto the display screen is located at the top front of the display screen.

[0252] Regardless of where the ambient light sensor is located below the AA zone, its projected area on the display screen is much smaller than the screen's actual area. Therefore, not all light emitted from the entire display screen interferes with the ambient light collected by the sensor. Instead, it is the light emitted from the display area above the sensor and the light emitted from the display area above and around the sensor that interferes with the ambient light collected by the sensor.

[0253] As an example, the photosensitive area of ​​an ambient light sensor has a photosensitive angle. The ambient light sensor can receive light within the photosensitive angle, but cannot receive light outside the photosensitive angle. Figure 5 Light emitted from point A, located above the ambient light sensor (within the sensing angle), and light emitted from point B, located a certain range above and around the ambient light sensor (also within the sensing angle), will both interfere with the ambient light collected by the sensor. Figure 5 Light emitted from point C, which is relatively far from the ambient light sensor (outside the sensing angle), will not interfere with the ambient light collected by the sensor. For ease of description, the display area that interferes with the ambient light collected by the sensor can be referred to as the target area (this target area can be referred to as the first area). The position of the target area on the display screen is determined by the specific position of the ambient light sensor below area AA. As an example, the target area can be a square area with a certain side length (e.g., 80 micrometers, 90 micrometers, 100 micrometers) centered on the center point of the ambient light sensor. Of course, the target area can also be other shaped areas that interfere with the light collected by the ambient light sensor, as determined by measurement.

[0254] As another example, Figure 6 This is a front view of the OLED screen of the electronic device provided in the embodiments of this application, as shown in the figure. Figure 6 As shown, the electronic device includes a housing, an OLED screen display interface, and an ambient light sensor located behind the ambient light sensor in the display area. The center point of the target area coincides with the center point of the ambient light sensor.

[0255] It should be noted that, as a device, the shape of an ambient light sensor may vary depending on the manufacturer. In this embodiment, the center point of the ambient light sensor is the center point of the photosensitive area where the sensor collects ambient light. Additionally, Figure 6The target area shown is larger than the projection area of ​​the ambient light sensor on the OLED screen. In practical applications, the target area may also be smaller than or equal to the projection area of ​​the ambient light sensor on the OLED screen. However, the target area is usually larger than the photosensitive area of ​​the ambient light sensor. As mentioned earlier, the actual ambient light is equal to the ambient light collected by the ambient light sensor minus the light emitted by the display screen. The light emitted by the display screen has been determined to be the light emitted by the target area. The light emitted by the target area is the light generated by the display content in the target area. The interference of the display content on the ambient light collected by the ambient light sensor comes from two parts: the RGB pixel information of the displayed image and the brightness of the displayed image. Through the above analysis, it can be understood that the interference on the ambient light collected by the ambient light sensor is respectively: the RGB pixel information of the image displayed in the target area and the brightness information of the target area. As an example, if the pixel value of a pixel is (r, g, b) and the brightness is L, then the normalized brightness of the pixel is: L × (r / 255) 2.2 , L×(g / 255) 2.2 , L×(b / 255) 2.2 .

[0256] For ease of description, the image corresponding to the target area can be denoted as the target image, and the interference of the RGB pixel information and brightness information of the target image on the ambient light collected by the ambient light sensor can be denoted as fusion noise. The ambient light collected by the ambient light sensor can be denoted as the initial ambient light, and the actual ambient light from outside can be denoted as the target ambient light.

[0257] From the above description, it can be concluded that the target ambient light is equal to the initial ambient light minus the fusion noise at each moment during the time period of acquiring the initial ambient light. In this embodiment, the process of calculating the fusion noise based on RGB pixel information and brightness information is referred to as the noise fusion process.

[0258] When the display screen is in display mode, the RGB pixel information of the image displayed in the target area may change, and the brightness information of the displayed image may also change. Whether the RGB pixel information or the brightness information of the image displayed in the target area changes, it may lead to changes in fusion noise. Therefore, it is necessary to calculate the subsequent fusion noise based on the changed information (RGB pixel information or brightness information). If the image in the target area remains unchanged for a long time, the fusion noise only needs to be calculated when the display screen brightness changes. Therefore, to reduce the frequency of calculating fusion noise, the target area can be an area where the image displayed on the display screen changes at a low frequency. For example, the status bar area at the top of the front of the electronic device. The projection of the ambient light sensor on the display screen is located slightly to the right of the status bar area. Of course, it can also be located slightly to the left of the status bar area, or in the middle of the status bar area; the specific location of the ambient light sensor is not limited in this embodiment.

[0259] The following will be through Figure 7 This application describes the technical architecture of the method for obtaining target ambient light from initial ambient light and content displayed on a screen, as provided in the embodiments of this application.

[0260] like Figure 7 As shown, the processor in the electronic device is a multi-core processor, which includes at least an AP (application processor) and an SCP (sensor coprocessor). The AP processor is the application processor in the electronic device; the operating system, user interface, and applications all run on the AP processor. The SCP processor is a coprocessor that assists the AP processor in tasks related to images, sensors (e.g., ambient light sensors), etc.

[0261] Figure 7 Only the AP processor and SCP processor are shown in the diagram. In practical applications, multi-core processors can also include other processors. For example, when the electronic device is a mobile phone, the multi-core processor can also include a baseband (BP) processor that runs the mobile phone's radio frequency communication control software and is responsible for sending and receiving data.

[0262] Figure 7 The AP processor shown only contains content relevant to the embodiments of this application. The implementation of the embodiments of this application depends on the following layers in the AP processor: Application, Java Framework, Native Framework, Hardware Abstraction Layer (HAL), Kernel, and Hardware.

[0263] Figure 7 The SCP processor in this application can be understood as a sensor hub, capable of controlling sensors and processing their data. Implementation of this application's embodiments relies on the following layers within the SCP processor: Hub APK, Hub FWK, Hub DRV, and Hubhardware.

[0264] The application layer of the AP processor contains various applications. Figure 7 The example shows application A and application B. Taking application A as an example, after the user launches application A, the screen will display the interface of application A. Specifically, application A sends the display parameters of the interface to be displayed (such as the memory address and color of the interface to be displayed) to the display engine service.

[0265] The display engine service in the AP processor sends the display parameters of the interface to be displayed to the SurfaceFlinger in the AP processor's native framework layer.

[0266] The SurfaceFlinger layer in the AP processor's native framework layer is responsible for controlling the blending of interfaces. As an example, it calculates the overlap area of ​​at least two overlapping interfaces. These interfaces could be the status bar, system tray, the application itself (the interface to be displayed by application A), wallpaper, background, etc. Therefore, SurfaceFlinger can obtain not only the display parameters of the interface to be displayed by application A, but also the display parameters of other interfaces.

[0267] The AP processor's hardware abstraction layer includes HWC (Hardware Composer HAL), which is the module responsible for interface composition and display in the system, providing hardware support for the SurfaceFlinger service. Step A1 involves SurfaceFlinger sending display parameters (e.g., memory address, color, etc.) of each interface to HWC for interface fusion via interfaces (e.g., setLayerBuffer, setLayerColor). In practical applications, display parameters may include the position, size, color, and storage address of the composite image interface on the electronic device's display screen.

[0268] Typically, in image composition (e.g., when an electronic device displays an image, it needs to composite the status bar, system bar, application itself, and wallpaper background), HWC obtains the composited image based on the display parameters of each interface through its underlying hardware (e.g., a hardware compositer). The HWC in the AP processor's hardware abstraction layer sends the hardware-composite image to the OLED driver, see step A2.

[0269] In practical applications, the HWC module can obtain the synthesized image based on the display parameters sent by surfaceFlinger in any way.

[0270] The OLED driver in the core layer of the AP processor sends the composite image to the display subsystem (DSS) in the hardware layer of the AP processor, as shown in step A3. The display subsystem (DSS) in the hardware layer of the AP processor can perform secondary processing on the composite image (e.g., HDR10 processing for image quality enhancement), and then send the processed image to the display. In practical applications, secondary processing may not be performed. Taking the absence of secondary processing as an example, the display subsystem in the hardware layer of the AP processor sends the composite image to the OLED screen for display.

[0271] Taking the launch of application A as an example, the composite image displayed on the OLED screen is the composite of the interface to be displayed by application A and the interface corresponding to the status bar.

[0272] Following the above method, an OLED screen can complete one image refresh and display.

[0273] In this embodiment of the application, the display subsystem (DSS) can control the display subsystem (DSS) to store the entire frame image (or the image larger than the target area in the entire frame image, or the image corresponding to the target area in the entire frame image) in the memory of the kernel layer of the AP processor before sending the secondary processed image (or the composite image) to the display. Since this process is a concurrent write-back of image frame data, this memory can be recorded as Concurrent Write Back (CWB) memory, see step A4.

[0274] This embodiment of the application takes the example of the display subsystem storing the entire frame image in the CWB memory of the AP processor. After the display subsystem successfully stores the entire frame image in the CWB memory, it can send a storage success signal to the HWC. The entire frame image corresponding to the image stored in the CWB memory by the display subsystem can be recorded as the image to be refreshed (the image to be refreshed can also be understood as the image after this refresh).

[0275] The AP processor can also be configured to allow HWC to access CWB memory. HWC can retrieve the target image from CWB memory after receiving a successful storage signal from the display subsystem, see step A5.

[0276] It should be noted that regardless of whether the CWB memory stores a complete frame of the image or a portion of it, HWC can obtain the target image from the CWB memory. The process of HWC obtaining the target image from the CWB memory can be referred to as HWC image matting from the CWB memory.

[0277] The range of the target image can be understood as the size of the range defined by the length and width of the target image. The range of the image to be refreshed is the range of the entire frame image, or the size of the range can be defined by the length and width.

[0278] As an example, the size of the image to be refreshed is X1 (pixels) × Y1 (pixels), the size of the target image is X2 (pixels) × Y2 (pixels), and the size of the image stored in CWB memory is X3 (pixels) × Y3 (pixels). X3 satisfies X1 ≥ X3 ≥ X2, and Y3 satisfies Y1 ≥ Y3 ≥ Y2.

[0279] Of course, when X3 = X1 and Y3 = Y1, the image stored in the CWB memory is a complete frame image. When X3 = X2 and Y3 = Y2, the image stored in the CWB memory is the target image.

[0280] Continuing with application A as an example, when application A needs to adjust brightness due to switching interfaces, application A sends the brightness to be adjusted to the display engine service.

[0281] The display engine service in the AP processor sends the brightness to be adjusted to the kernel node in the AP processor kernel layer, so that the relevant hardware can adjust the brightness of the OLED screen according to the brightness to be adjusted stored in the kernel node.

[0282] Using the method described above, an OLED screen can complete one brightness adjustment.

[0283] In this embodiment of the application, HWC can also be configured to obtain the brightness to be adjusted from the kernel node. The brightness to be adjusted can also be recorded as the brightness after this adjustment. See step A5' for details.

[0284] In its implementation, HWC can use the uevent mechanism to monitor whether the data stored in the kernel node has changed. Upon detecting a change in the kernel node's data, it retrieves the currently stored data, i.e., the brightness value to be adjusted (this brightness value is used to adjust the display screen's brightness, and therefore can also be referred to as the display screen's brightness value). After obtaining the target image or the brightness information to be adjusted, HWC can send it to the noise algorithm library in the hardware abstraction layer of the AP processor. See step A6. The noise algorithm library can calculate the fusion noise at the refresh time of the target image after each acquisition. It also calculates the fusion noise at the brightness adjustment time after each acquisition. The noise algorithm library stores the calculated fusion noise in its noise memory.

[0285] In practical applications, after the HWC acquires the target image, it can store the target image and send the storage address of the target image to a noise algorithm library. This noise algorithm library can cache the latest frame of the target image by recording the address. Similarly, after acquiring the brightness to be adjusted, the HWC can send the brightness to be adjusted to the noise algorithm library, which can cache the latest brightness value. For ease of description, subsequent embodiments of this application will be described as the HWC sending the target image to the noise algorithm library. In practical applications, this can mean that after the HWC acquires and stores the target image, it sends the storage address of the target image to the noise algorithm library.

[0286] As an example, after receiving the storage address of the first frame of the target image, the noise algorithm library caches the storage address of the first frame of the target image. Subsequently, each time a new storage address of the target image is received, that new storage address is used as the storage address of the latest cached target image. Similarly, after receiving the first brightness value, the noise algorithm library caches the first brightness value, and subsequently, each new brightness value is used as the latest cached brightness value. In this embodiment, the noise algorithm library caches the acquired target image and brightness value in a data storage repository. Both the target image and brightness value stored in the data storage repository can be considered as screen data; that is, the screen data stored in the data storage repository includes: target image and brightness value.

[0287] Furthermore, to describe the transmission relationship between the target image, the brightness to be adjusted, and other parameters, this embodiment uses the example of HWC sending the target image and the brightness to be adjusted to a noise algorithm library. In practical applications, the relationship between HWC and the noise algorithm library is that HWC calls the noise algorithm library. When HWC calls the noise algorithm library, HWC inputs the target image (the storage address of the target image) and the brightness to be adjusted, etc., as independent variables of the calculation model in the noise algorithm library. Other parameters will not be listed one by one.

[0288] Since brightness adjustment and image refresh are two completely independent processes, the image may be refreshed at a certain moment while the brightness remains unchanged. Therefore, when calculating the fusion noise at this moment, the target image corresponding to the refreshed image and the current brightness (the latest stored brightness value in the noise algorithm library before the time represented by the timestamp of the target image) are used. For ease of description, the fusion noise at the image refresh moment calculated due to image refresh can be denoted as the image noise at the image refresh moment. Similarly, if the image is not refreshed at a certain moment but the brightness is adjusted, the fusion noise at this moment is calculated using the adjusted brightness and the current target image (the latest stored target image in the noise algorithm library before the time represented by the timestamp of the brightness value). For ease of description, the fusion noise at the brightness adjustment moment calculated due to brightness adjustment can be denoted as the backlight noise at the brightness adjustment moment.

[0289] The target image and brightness sent by HWC to the noise algorithm library are both timestamped. Correspondingly, the image noise and backlight noise calculated by the noise algorithm library are also timestamped. The timestamp of the image noise is the same as the timestamp of the target image, and the timestamp of the backlight noise is the same as the timestamp of the brightness to be adjusted. Strictly speaking, the timestamp of the image noise should be the image refresh time. In practical applications, other time nodes close to the image refresh time can also be used, such as the start time (or end time, or any time in between) of HWC's image extraction from CWB memory to obtain the target image. Strictly speaking, the timestamp of the backlight noise should be the backlight adjustment time. In practical applications, other time nodes close to the backlight adjustment time can also be used, such as the start time (or end time, or any time in between) of HWC's acquisition of the brightness to be adjusted from the kernel node. The timestamps of the image noise and backlight noise facilitate subsequent denoising of the initial ambient light collected by the ambient light sensor over a time span to obtain the target ambient light. The noise algorithm library stores image noise and backlight noise in a noise memory. When storing image noise, the noise algorithm library also stores the timestamp of the image noise, and when storing backlight noise, the noise algorithm library also stores the timestamp of the backlight noise.

[0290] The ambient light sensor (ALS) in the SCP processor's co-hardware layer acquires initial ambient light at a certain acquisition cycle after startup (typically, the ambient light sensor starts up when the electronic device is powered on). The SCP processor's ambient light sensor transmits the initial ambient light information to the ambient light sensor driver (ALSDRV) in the SCP processor's co-drive layer (Hub DRV), see step E2.

[0291] The initial ambient light information transmitted from the SCP processor to the AP processor includes a first value, a first time, and a second time. The first value can be understood as the raw value of the initial ambient light, the first time is the start time of the integration of the first value by the ambient light sensor, and the second time is the end time of the integration of the first value by the ambient light sensor.

[0292] In the SCP processor's Hub DRV layer, the Ambient Light Sensor Driver (ALS DRV) preprocesses the initial ambient light information to obtain raw values ​​on the four RGBC channels. The SCP processor's Hub DRV layer then transmits these raw values ​​to the SCP processor's Ambient Light Sensor Application layer, as described in step E3.

[0293] The ambient light sensor of the SCP processor's co-application layer sends the raw values ​​on the four RGBC channels and other relevant data (e.g., the start and end times of each initial ambient light acquisition by the ambient light sensor) to the HWC of the AP processor via the first inter-core communication (communication between the ambient light sensor application of the SCP processor and the HWC of the AP processor), see step E4.

[0294] After the HWC in the AP processor obtains the initial ambient light data reported by the SCP processor, the HWC in the AP processor can send the initial ambient light data to the noise algorithm library. See step A6.

[0295] As mentioned earlier, the noise algorithm library can calculate the image noise at the image refresh time and the backlight noise at the brightness adjustment time, and store the calculated image noise and backlight noise in the noise memory of the noise algorithm library. In practical applications, the noise algorithm library can not only calculate the image noise at the image refresh time and the backlight noise at the brightness adjustment time, but also, after obtaining the start time and end time of the initial ambient light acquisition, obtain the integrated noise between the start time and the end time of the initial ambient light acquisition based on the image noise and backlight noise stored in the noise memory. The noise algorithm library can then subtract the integrated noise between the start time and the end time of the initial ambient light acquisition from the initial ambient light to obtain the target ambient light.

[0296] As described above, the noise algorithm library contains various computational models. For example, the first algorithm model is used to calculate the fusion noise based on the target image and brightness. The second algorithm model is used to obtain the integral noise from the start time to the end time of the initial ambient light acquisition based on the fusion noise at various times. The third algorithm model is used to obtain the target ambient light based on the initial ambient light and the integral noise. In practical applications, the noise algorithm library may also include other computational models. For example, in the process of obtaining the target ambient light based on the target image, brightness, and initial ambient light, if the raw values ​​of the four channels of the initial ambient light are filtered, there exists a model for filtering the raw values ​​of the four channels of the initial ambient light. This application embodiment will not exemplify other models one by one.

[0297] The input to the noise algorithm library includes: the target image and brightness acquired by HWC at various times, and the relevant data of the initial ambient light acquired by HWC from the SCP processor. The output of the noise algorithm library is: the raw value of the target ambient light, which can be denoted as the second value. In this embodiment, the process of HWC sending the target image, brightness, and initial ambient light to the noise algorithm library is denoted as step A6.

[0298] After obtaining the target ambient light, the noise calculation library needs to return the target data to HWC; this process is denoted as step A7. In practical applications, the output of the noise algorithm library is the raw value of the four channels of the target ambient light.

[0299] The HWC in the AP processor sends the raw values ​​of the four channels of the target ambient light returned by the noise algorithm library to the ambient light sensor application in the co-application layer of the SCP processor via the first inter-core communication, see step A8.

[0300] After obtaining the raw values ​​of the four target ambient light channels, the ambient light sensor application in the co-drive layer of the SCP processor stores these raw values ​​in the ambient light memory of the co-drive layer. See step E5.

[0301] The SCP processor's co-drive layer includes a computation module. This computation model obtains the raw values ​​of the four target ambient light channels from memory, see step E6. At the end of each integration, the ambient light sensor generates an integration interrupt signal. The ambient light sensor sends this interrupt signal to the ambient light sensor driver, which then invokes the computation module, triggering the computation model to obtain the raw values ​​of the four target ambient light channels from memory.

[0302] Since the ambient light sensor driver will only trigger the calculation module to obtain the raw value of the target ambient light after the current integration is completed, the raw value of the target ambient light obtained at this time is the raw value of the target ambient light from the previous integration cycle.

[0303] by Figure 8 Taking the illustrated embodiment as an example, after integration at time t1, the ambient light sensor obtains the initial ambient light from time t0 to time t1. The SCP processor sends the initial ambient light from time t0 to time t1 to the AP processor. The AP processor calculates and obtains the raw value of the target ambient light from time t0 to time t1. The AP processor then sends the raw value of the target ambient light from time t0 to time t1 to the SCP processor. The SCP processor stores the raw value of the target ambient light from time t0 to time t1 in its memory.

[0304] After integration at time t3, the ambient light sensor acquires the initial ambient light from time t2 to t3. The SCP processor sends this initial ambient light from time t2 to t3 to the AP processor. Each time the ambient light sensor completes integration, an integration interrupt signal is generated. The ambient light sensor sends this interrupt signal to the ambient light sensor driver, which then calls the calculation module, triggering the calculation model to retrieve the raw values ​​of the target ambient light from time t0 to t1 stored in memory. Since this occurs after time t3, the calculation module calculates the lux value of the target ambient light after time t3 based on the acquired raw values ​​of the target ambient light from time t0 to t1. In other words, the lux value of the target ambient light calculated by the SCP processor within period T2 is the lux value of the actual ambient light within period T1.

[0305] As mentioned earlier, the ambient light sensor in the SCP processor generates an integration interrupt signal after the integration ends (at time t3). This interrupt signal is sent to the ambient light sensor driver. After time t3, the initial ambient light for cycle T2 is sent to the AP processor. Only after the AP processor calculates and obtains the target ambient light will it send the target ambient light to the SCP processor. The SCP processor stores the target ambient light for cycle T2 in its memory. If the SCP processor uses the raw value of the target ambient light for cycle T2 to calculate the lux value, it needs to wait from the moment the ambient light sensor driver receives the integration interrupt signal until the AP processor transmits the target ambient light to the SCP processor's memory. Only then can the ambient light sensor driver in the SCP processor call the calculation module to retrieve the raw value of the target ambient light for cycle T2 from the memory. The waiting time includes at least the time required for the SCP processor to send the initial ambient light to the AP processor, the AP processor to calculate the target ambient light based on the initial ambient light and other relevant data, and the AP processor to send the target ambient light to the SCP processor's memory. This waiting time is relatively long and not fixed. Therefore, the ambient light sensor driver in the SCP processor can be configured to, upon receiving the integration interrupt signal from the second acquisition cycle, call the calculation module to retrieve the raw value of the target ambient light from memory for the previous cycle, and then calculate the lux value based on the raw value of the target ambient light for the previous cycle. The lux value of the target ambient light can be denoted as the third value, and the third value and the second value are the lux value and raw value of the same target ambient light.

[0306] by Figure 8 Taking the sampling period shown as an example, if the raw value of the initial ambient light sampled in sampling period T1 is the first value, then the raw value of the target ambient light corresponding to sampling period T1, obtained from the raw value of the initial ambient light sampled in sampling period T1, is the second value. The lux value of the target ambient light corresponding to sampling period T1, obtained from the raw value of the target ambient light corresponding to sampling period T1, is the third value. The raw value of the initial ambient light sampled in sampling period T2 can be recorded as the fourth value. The fourth value is the initial ambient light sampled in the sampling period following the sampling period corresponding to the first value (which could also be the sampling period corresponding to the second value, or the sampling period corresponding to the third value).

[0307] The computation module in the co-drive layer of the SCP processor obtains the lux value of the target ambient light based on the raw values ​​of the four channels of the target ambient light. The computation module in the SCP processor sends the calculated lux value of the target ambient light to the ambient light sensor application in the co-application layer through the interface of the co-frame layer, see steps E7 and E8.

[0308] The ambient light sensor application in the SCP processor's cooperating application layer transmits the lux value of the target ambient light to the light service in the native framework layer of the AP processor via inter-core communication (communication between the SCP processor and the AP processor's light service), see step E9.

[0309] The light service can send the lux value of the target ambient light to the display engine service. The display engine service can then send the lux value of the target ambient light to the upper layers so that applications in the application layer can determine whether to adjust the brightness. The display engine service can also send the lux value of the target ambient light to the kernel node so that the relevant hardware can adjust the brightness of the display screen based on the lux value of the target ambient light stored in the kernel node.

[0310] After describing the technical architecture upon which the method for obtaining target ambient light depends, the process of obtaining target ambient light from the perspective of the ambient light sensor's acquisition cycle will be described.

[0311] As illustrated by the above example, both the target image and the brightness to be adjusted are acquired by the HWC. Therefore, the processes of acquiring the target image and the brightness to be adjusted by the HWC have a sequential order. After acquiring the target image or the brightness to be adjusted, the HWC sends the target image or the brightness to be adjusted to the noise algorithm library. The process of the HWC sending the target image or the brightness to be adjusted to the noise algorithm library also has a sequential order. Correspondingly, the time when the noise algorithm library receives the target image and the brightness to be adjusted also has a sequential order. However, even though the time when the noise algorithm library receives the target image and the brightness to be adjusted also has a sequential order, since the HWC may acquire the target image and the brightness to be adjusted within the same time metric, the timestamps of the target image and the brightness to be adjusted may be the same. As an example, within the same millisecond (the 5th millisecond), the HWC first executes the acquisition of the brightness to be adjusted, and then executes the acquisition of the target image. Although there is a sequential order in the execution of the HWC, the timestamps of the target image and the brightness to be adjusted are both the 5th millisecond.

[0312] See Figure 8 The ambient light sensor collects ambient light over a specific time period T. Each collection period consists of three cycles: from t0 to t2 (collection period T1), from t2 to t4 (collection period T2), and from t4 to t6 (collection period T3). Within this collection period T1, the actual collection time is from t0 to t1; the sensor can be in a dormant state during the period from t1 to t2. This embodiment describes an example where the ambient light collection period is fixed (i.e., T1, T2, and T3 have the same value) and the integration time period is also fixed.

[0313] As an example, a data acquisition cycle of 350ms (t2-t0) can be used. Within one acquisition cycle, the ambient light sensor actually acquires data for 50ms (t1-t0), meaning the sensor will be in a sleep state for 300ms (t2-t1) within that cycle. The 350ms, 50ms, and 300ms values ​​in the above example are for illustrative purposes only and are not intended to be limiting.

[0314] For ease of description, the time period during which the ambient light sensor actually collects data (e.g., t0 to t1) can be denoted as the integration time period, and the time period during which the ambient light sensor does not start collecting data (e.g., t1 to t2) can be denoted as the non-integration time period.

[0315] The images displayed on the screen of an electronic device are refreshed at a certain frequency. Taking 60Hz as an example, this is equivalent to the screen refreshing 60 times per second, or once every 16.7ms. Therefore, when the screen displays an image, image refreshes occur within the sampling cycle of the ambient light sensor. When the displayed image refreshes, the AP processor executes... Figure 7 The technical architecture shown includes steps A1 to A6 (sending the target image). Starting from time t0, the HWC in the AP processor controls the CWB to continuously write back, meaning that as long as there is an image refresh, the above steps are repeatedly executed.

[0316] It should be noted that this embodiment uses a 60Hz refresh rate as an example. In practical applications, the refresh rate can also be 120Hz or other refresh rates. This embodiment takes the example that steps A1 to A6 (sending the target image) need to be repeated for each frame refresh. In practical applications, steps A1 to A6 (sending the target image) can also be repeated every one frame (or two frames, etc.).

[0317] Brightness adjustment does not follow a fixed periodicity; therefore, brightness adjustment may occur within the ambient light sensor's acquisition cycle. When brightness adjustment occurs, HWC also performs... Figure 7 Steps A5' to A6 (sending the brightness to be adjusted) in the technical architecture shown.

[0318] After each integration process by the ambient light sensor (i.e., after t1, t3, t5, etc.), the SCP processor reports the initial ambient light data collected during this integration process (e.g., the raw values ​​on the four channels of the initial ambient light and the start and end times of this integration process) to the HWC of the AP processor. The HWC of the AP processor sends the relevant data of the initial ambient light to the noise algorithm library, and the target ambient light is obtained by calculating through the noise algorithm library.

[0319] See Figure 9 Taking one acquisition cycle as an example, in t 01 Time (the same time as t0), t 03 Time, t 04 Time and t 11 All times are image refresh times, in t 02 Time and t 12 The time is the brightness adjustment time. Therefore, the AP processor can calculate and obtain t in real time. 01 Image noise at time t 02 Backlight noise at any time, t 03 Image noise at time t 04 Image noise at time t 11 Image noise at time t 12 Backlight noise at time t1. After the integration ends (time t1), the AP processor's noise memory stores: t 01 Image noise at time t 02 Backlight noise at any time, t 03 Image noise at time t 04 Image noise at any given moment.

[0320] After the integration ends (time t1), the ambient light sensor obtains the initial ambient light and the integration time period for this period. The SCP processor reports the initial ambient light data to the AP processor, and the noise calculation module in the AP processor obtains t from the noise memory based on the start and end times of the integration time period. 01 Image noise at time t 02 Backlight noise at any time, t 03 Image noise at time t 04 Image noise at any given time. The noise calculation library calculates the target ambient light based on the initial ambient light collected during the integration time period and the image noise and backlight noise affecting the current integration time period.

[0321] During the non-integral time period (t1 to t2), because HWC continuously controls CWB writeback, HWC controls t... 11 The refreshed image at each time step also underwent image matting to obtain the target image, and the noise algorithm library also calculated t. 11 Image noise at any given time. Non-integral time interval t. 12 The brightness changes over time, and the noise algorithm library also calculates t. 12 The backlight noise at each time step. However, when calculating the target ambient light, the required fusion noise is the fusion noise that interferes with the initial ambient light obtained during this integration time period. Therefore, t is not required. 11 Image noise at time t 12The backlight noise at any given moment can also be used to obtain the target ambient light during the current integration time period. In practical applications, the noise algorithm library obtains t... 11 Image noise at time t 12 After the backlight noise at a given moment, it is also necessary to... 11 Image noise at time t 12 The backlight noise at any given moment is stored in the noise memory.

[0322] The above examples are based on Figure 7 From the perspective of technical architecture, based on Figure 9 The process of acquiring target ambient light is described from the perspective of the acquisition cycle of the ambient light sensor. The following will combine... Figure 7 The technical architecture shown and Figure 9 Description of one acquisition cycle of the ambient light sensor shown. Figure 10 The illustrated embodiment provides a timing process diagram for acquiring target ambient light.

[0323] As described above, the processes of image refresh triggering AP processor to calculate image noise, brightness adjustment triggering AP processor to calculate backlight noise, and the SCP processor controlling the underlying hardware ambient light sensor to collect initial ambient light are all performed independently, without any temporal sequence. The AP processor's noise calculation library processes the target image, brightness, and initial ambient light obtained from these three independent processes to obtain the target ambient light.

[0324] Figure 10 Step numbers and in the illustrated embodiments Figure 7 In the illustrated technical architecture, steps with the same number represent the same steps being performed. To avoid repetition, in... Figure 7 The detailed descriptions in the illustrated embodiments are as follows: Figure 10 The embodiments shown will be described in a simplified manner.

[0325] Combination Figure 9 Starting from time t0, the image is refreshed. At the same time, the ambient light sensor enters an integration time period and begins to collect the initial ambient light.

[0326] Correspondingly, in Figure 10 In step E1, the ambient light sensor in the co-processor's hardware layer enters an integration time period, from t0(t 01 The initial ambient light is collected at a certain time.

[0327] Step A1, image t0(t 01The interface is refreshed constantly. SurfaceFlinger in the native framework layer of the AP processor sends the display parameters of the interface to HWC in the hardware abstraction layer of the AP processor. HWC can send the received display parameters of each interface sent by SurfaceFlinger to the underlying hardware of HWC. The underlying hardware of HWC obtains the composite image of each interface based on the display parameters of each interface. The underlying hardware of HWC returns the composite image to HWC.

[0328] In step A2, the HWC in the hardware abstraction layer of the AP processor sends the synthesized image to the OLED driver in the AP processor kernel layer. In step A3, the OLED driver in the AP processor kernel layer sends the synthesized image to the display subsystem of the AP processor's hardware layer.

[0329] In step A4, the display subsystem in the hardware layer of the AP processor stores the image before display in the CWB memory in the kernel layer of the AP processor.

[0330] In this embodiment of the application, after sending the synthesized image to the OLED driver, the HWC will wait for a storage success signal sent by the display subsystem.

[0331] After the display subsystem successfully stores the image before display in the CWB memory, it sends a storage success signal to the HWC. Upon receiving the storage success signal from the display subsystem, the HWC extracts the target image from the image before display stored in the CWB memory in the kernel layer.

[0332] In step A5, the HWC in the hardware abstraction layer of the AP processor obtains the target image by extracting the image stored in the CWB memory in the kernel layer before display.

[0333] In step A6, after the HWC in the hardware abstraction layer of the AP processor obtains the target image, it sends the target image to the noise algorithm library of this layer. After receiving the target image, the noise algorithm library calculates t based on the target image and the cached current brightness information. 01 Image noise at any given moment. During the execution of steps A1 to A6, the ambient light sensor in the co-hardware layer of the SCP processor is continuously undergoing an integration process within one acquisition cycle.

[0334] Combination Figure 9 , in t 02 At time t, the ambient light sensor is still within the integration time period and is collecting the initial ambient light. 02 At any given moment, the brightness of the display screen changes, triggering execution. Figure 10 Step B1 in the process.

[0335] exist Figure 10 In step B1 ( Figure 7 In step A5' of the architecture shown, the HWC of the AP processor's hardware abstraction layer obtains t from the kernel node in the AP processor's kernel layer. 02 Brightness information at any given time.

[0336] Step B2 (Step A6), the HWC of the AP processor's Hardware Abstraction Layer will t 02 The brightness information at time t is sent to the noise algorithm library, which then calculates the brightness based on the value of t. 02 The brightness information at time t is calculated from the cached target image currently being displayed. 02 Backlight noise at any given moment.

[0337] During the execution of steps B1 to B2, the ambient light sensor in the co-hardware layer of the SCP processor is constantly in the integration process within a data acquisition cycle.

[0338] After step B2, the noise memory of the noise algorithm library stores t. 01 Image noise at time t 02 Backlight noise at any given moment.

[0339] Combination Figure 9 , in t 03 At time t, the ambient light sensor is still within the integration time period and is collecting the initial ambient light. 03 At any given moment, the image is refreshed.

[0340] exist Figure 10 Since the image has been refreshed, steps C1 to C6 are executed. Steps C1 to C6 can be referred to in the descriptions in A1 to A6, and will not be repeated here.

[0341] During the execution of steps C1 to C6, the ambient light sensor in the co-hardware layer of the SCP processor is still in the integration process within a collection cycle.

[0342] After step C6, the noise memory of the noise algorithm library stores t. 01 Image noise at time t 02 Backlight noise at time and t 03 Image noise at any given moment.

[0343] See Figure 9 , in t 04 At time t, the ambient light sensor is still within the integration time period and is collecting the initial ambient light. 04 At any given moment, the image is refreshed.

[0344] exist Figure 10Since the image has been refreshed, steps D1 to D6 are executed. Steps D1 to D6 can be referred to in the descriptions in A1 to A6, and will not be repeated here.

[0345] During the execution of steps D1 to D6, the ambient light sensor in the co-hardware layer of the SCP processor is still in the integration process within a collection cycle.

[0346] After step D6, the noise memory of the noise algorithm library stores t. 01 Image noise at time t 02 Backlight noise at any time, t 03 Image noise at time t 04 Image noise at any given moment.

[0347] Combination Figure 9 At time t1, the ambient light sensor completes its integration. After the ambient light sensor completes its integration (at time t1), it obtains the initial ambient light. Figure 10 In the process, the SCP processor begins executing steps E2, E3, and E4, sending the relevant data of the initial ambient light (raw values ​​on the four RGBC channels, integration start time, and integration end time) to the HWC of the AP processor's hardware abstraction layer.

[0348] Combination Figure 9 During non-integration periods, the image may also be refreshed (e.g., t). 11 Image refresh time (at any moment), brightness may also change (e.g., t). 12 (The brightness changes over time). Therefore, during the non-integration period, Figure 10 Steps F1 to F6 still exist in the process. Figure 10 Steps F1 to F5 in the original text are omitted; please refer to steps A1 to A5 for details, so that t 11 The image noise at each time step is stored in the noise memory of the noise algorithm library. During the non-integration time period, steps G1 to G2 still exist. Figure 9 Step G1 is omitted; please refer to step B1 for details, so that t 12 The backlight noise at any given time is stored in the noise memory of the noise algorithm library.

[0349] In step A6', the HWC in the hardware abstraction layer of the AP processor sends the initial ambient light data to the noise algorithm library. The noise algorithm library calculates the target ambient light based on the initial ambient light data and the image noise and backlight noise that interfere with the initial ambient light.

[0350] pass Figure 10It can be understood that the integration start and end times of the ambient light sensor are controlled by the clock corresponding to the ambient light sensor; the AP processor's calculation of image noise is controlled by the image refresh clock; and the AP processor's calculation of backlight noise is controlled by the backlight adjustment time. Therefore, the execution of step A1 (or steps C1, D1, and F1) is triggered by image refresh. The execution of step B1 (or step G1) is triggered by brightness adjustment. The integration start and end times of the ambient light sensor are strictly in accordance with the pre-set acquisition cycle and the duration of each integration. Therefore, the execution of step E2 is triggered by the event of the ambient light sensor's integration ending.

[0351] From the perspective of the triggering event, these three processes are completely independent. However, the results obtained by these three processes (image noise, backlight noise, and initial ambient light) are correlated through the denoising process after the integration time period of the ambient light sensor ends. The initial ambient light fused in the denoising process is the initial ambient light acquired by the ambient light sensor in the current acquisition cycle, and the image noise and backlight noise removed in the denoising process are image noise and backlight noise that can interfere with the initial ambient light acquired in the current acquisition cycle.

[0352] This application embodiment analyzes the structure of the under-screen ambient light to determine that factors interfering with the ambient light collected by the ambient light sensor include the display content of the display area directly above the photosensitive area of ​​the ambient light sensor and the area directly above the photosensitive area surrounding the sensor. This display content is further divided into two parts: RGB pixel information and brightness information of the displayed image. Therefore, the noise calculation library in this application embodiment obtains fused noise by fusing the RGB pixel information and brightness information of the target image. Then, the integral noise of the initial ambient light's integration time period is obtained based on the fused noise. By removing the integral noise that interferes with the initial ambient light obtained from the ambient light sensor's integration time period, the target ambient light is obtained. Because the interference is removed, an accurate target ambient light can be obtained, and the system has strong versatility.

[0353] Furthermore, since the AP processor of the electronic device can obtain the target image and brightness information, it also obtains image noise and backlight noise. The SCP processor can obtain the initial ambient light. Therefore, the SCP processor can send the initial ambient light to the AP processor, which processes the initial ambient light and the fused noise to obtain the target ambient light. This avoids the problem of excessive power consumption caused by the AP processor frequently sending the target image (or image noise) and brightness information (or backlight noise) to the SCP processor, resulting in too frequent inter-core communication.

[0354] Furthermore, the DSS in the AP processor can store the image before display (the image to be displayed on the screen during this refresh) in the CWB memory. The HWC in the AP processor extracts the target image from the image before display stored in the CWB memory, thereby calculating the fusion noise. This method of obtaining fusion noise is accurate and consumes less power.

[0355] It should also be noted that the brightness of the display screen only needs to be adjusted according to the target ambient light when an image is displayed. When the display screen is not displaying any image, the brightness adjustment is unnecessary. Therefore, the AP processor also needs to monitor the screen's on / off events. When the screen is on, the ambient light detection method provided in this embodiment is executed. When the screen is off, the AP processor may not execute steps A4 to A6. Similarly, the SCP processor can also control the ambient light sensor to stop collecting initial ambient light when the screen is off, and the SCP processor may not execute steps E2 to E5.

[0356] To gain a clearer understanding of the execution within the AP processor, a timing diagram is described, illustrating the relationships between the various modules within the AP processor. Figure 10 In the embodiment shown, t is obtained 01 Image noise at time t 02 The backlight noise at any given time will be used as an example for description.

[0357] Figure 11 In the illustrated embodiment, when refreshing the image, each module in the AP processor performs the following steps:

[0358] Step 1100: After obtaining the display parameters of the interface to be displayed from the application in the application layer, the display engine service sends the display parameters of the interface to be displayed to SurfaceFlinger.

[0359] Step 1101: After SurfaceFlinger obtains the display parameters of the interface to be displayed for application A from the display engine service, it sends the display parameters (e.g., memory address, color, etc.) of each interface (the interface to be displayed for application A, the status bar interface, etc.) to HWC through the interface (e.g., setLayerBuffer, setLayerColor).

[0360] Step 1102: After receiving the display parameters of each interface, HWC obtains the composite image through the underlying hardware of HWC based on the display parameters of the interface to be displayed.

[0361] Step 1103: After HWC obtains the image synthesized by the underlying hardware, it sends the synthesized image to the OLED driver.

[0362] Step 1104: After receiving the composited image sent by HWC, the OLED driver sends the composited image to the display subsystem.

[0363] Step 1105: After receiving the synthesized image, the display subsystem performs secondary processing on the synthesized image to obtain the image before it is sent for display.

[0364] Step 1106: The display subsystem stores the image to be displayed in the CWB memory.

[0365] It should be noted that because OLED screens need to refresh images, the display subsystem also needs to send the image before display to the display screen for display.

[0366] In the embodiments of this application, the steps of the display subsystem sending the image before display to the display screen for display and the steps of the display subsystem storing the image before display in the CWB memory are two independent steps, and there is no strict order between them.

[0367] Step 1107: After the display subsystem successfully stores the image before display in the CWB memory, it can send a storage success signal to the HWC.

[0368] Step 1108: After receiving the signal that storage was successful, HWC extracts the target image from the image stored in CWB memory before display. The moment when HWC starts to obtain the target image is used as the timestamp of the target image.

[0369] Step 1109: After HWC obtains the target image and timestamp, it sends the target image and timestamp to the noise algorithm library.

[0370] Step 1110: The noise algorithm library calculates the image noise (t) at the refresh time corresponding to the target image. 01 The image noise at a given time (time stamp). The timestamp of this image noise is the timestamp of the target image from which the image noise is obtained. The noise algorithm library stores the image noise and its timestamp.

[0371] During brightness adjustment, each submodule in the AP processor performs the following steps:

[0372] Step 1111: After obtaining the brightness to be adjusted from application A in the application layer, the display engine service sends the brightness to be adjusted to the kernel node.

[0373] Step 1112: After HWC detects a change in the data on the kernel node, it retrieves the brightness to be adjusted from the kernel node. The moment HWC retrieves the brightness to be adjusted from the kernel node is the timestamp of the brightness to be adjusted.

[0374] In practical applications, HWC continuously monitors whether data changes have occurred on the kernel node.

[0375] Step 1113: HWC sends the timestamps of the adjusted brightness and the brightness to be adjusted to the noise algorithm library.

[0376] Step 1114: The noise algorithm library calculates the backlight noise (t) at the adjustment time of the brightness to be adjusted. 02 (Backlight noise at a given moment). The timestamp of this backlight noise is the timestamp of the brightness to be adjusted when the backlight noise is obtained. The noise algorithm library stores the backlight noise and its timestamp.

[0377] After an integration period ends, the SCP processor sends the initial ambient light collected during the integration period to the HWC in the AP processor.

[0378] Step 1115: The HWC of the AP processor receives the initial ambient light and the integration start time and integration end time of the initial ambient light sent by the SCP processor.

[0379] Step 1116: After receiving the initial ambient light and the integration start time and integration end time of the initial ambient light from the SCP processor, HWC sends the initial ambient light and the integration start time and integration end time of the initial ambient light to the noise algorithm library.

[0380] Step 1117: The noise algorithm library calculates the integral noise based on the image noise and its corresponding timestamp, the backlight noise and its corresponding timestamp, and the integration start and end times of the initial ambient light. The noise algorithm library also calculates the backlight noise based on the integral noise and the initial ambient light.

[0381] This application focuses on describing the timing logic diagram between various modules when the AP processor obtains the target ambient light.

[0382] The above embodiments all take "After the AP processor acquires the target image and brightness information, the AP processor calculates the fusion noise, and after the SCP processor obtains the initial ambient light, it sends the initial ambient light to the AP processor. The AP processor processes the fusion noise to obtain the integral noise of the integral time period of the initial ambient light, and then obtains the target ambient light based on the initial ambient light and the integral noise" as an example.

[0383] In practical applications, the AP processor can also obtain the target image and brightness information, and then send the target image and brightness information to the SCP processor. The SCP processor fuses the target image and brightness information to obtain the fusion noise and the integral noise of the initial ambient light over the integration time period, and then obtains the target ambient light based on the fusion noise and the initial ambient light.

[0384] In practical applications, after the AP processor acquires the target image and brightness information, it calculates the fusion noise and sends the calculated fusion noise to the SCP processor. The SCP processor obtains the integrated noise for the integration time period based on the received fusion noise, and then obtains the target ambient light based on the integrated noise for the integration time period and the initial ambient light collected by the ambient light sensor.

[0385] See Figure 12 This provides a technical architecture for "calculating fusion noise in the AP processor, calculating integral noise in the SCP processor, and obtaining target ambient light based on integral noise and target ambient light" in the embodiments of this application.

[0386] As mentioned earlier, the process of obtaining the target ambient light can be briefly described as follows:

[0387] Step 1: Calculate image noise based on the target image.

[0388] Step 2: Calculate the backlight noise based on the brightness.

[0389] Step 3: Calculate the target ambient light (raw value on four channels) based on the image noise, backlight noise, and initial ambient light.

[0390] Figure 7 In the illustrated technical architecture, step 3, calculating the target ambient light based on image noise, backlight noise, and initial ambient light, is implemented in the AP processor's noise algorithm library. The AP processor's noise algorithm library can calculate image noise and backlight noise. The initial ambient light is obtained by the ambient light sensor of the SCP processor. Therefore, the AP processor's noise algorithm library needs to obtain relevant data on the initial ambient light reported by the SCP processor (steps E3 to E4). Finally, the AP processor needs to return the calculated values ​​of the four channels of the target ambient light to the SCP processor to obtain the Lux value of the target ambient light (steps A8, E5, and E6).

[0391] Figure 12 In the illustrated technical architecture, step 3, calculating the target ambient light based on image noise, backlight noise, and initial ambient light, is implemented in the denoising module of the SCP processor. Image noise and backlight noise are obtained by the AP processor, while the initial ambient light is obtained by the ambient light sensor of the SCP processor. Therefore, the denoising module of the SCP processor needs to acquire the image noise and backlight noise sent by the AP processor (steps A8, E5, and E6), and also needs the initial ambient light sent by the ambient light sensor of the SCP processor (step E3).

[0392] Based on the above analysis, Figure 7 In the technical architecture shown, the noise algorithm library of the AP processor needs to implement the calculations of steps 1 to 3. Figure 12 In the technical architecture shown, steps 1 and 2 need to be implemented in the noise algorithm library of the AP processor, and step 3 needs to be implemented in the computing module of the SCP processor.

[0393] To understand more clearly Figure 12 The process of obtaining the target ambient light corresponding to the technical architecture shown is as follows: Figure 13 The timing diagram is shown. Combined with... Figure 9 Events at various times are recorded, starting from time t0, when the image is refreshed. Simultaneously, the ambient light sensor enters an integration time period and begins collecting initial ambient light data.

[0394] Correspondingly, in Figure 13 In step E1, the ambient light sensor in the co-processor's hardware layer enters an integration time period, from t0(t 01 The initial ambient light is collected at a certain time.

[0395] Steps A1 to A6, refer to Figure 7 The following describes steps A1 to A6 in the illustrated embodiment.

[0396] Step A7, the noise algorithm library in the hardware abstraction layer of the AP processor will t 01 Image noise at any given time is sent to the HWC in the same layer.

[0397] Step A8, calculate t in the AP processor. 01 After removing image noise at time t, 01 The ambient light sensor application in the co-application layer sends the image noise at any given moment to the SCP processor.

[0398] Step A9 ( Figure 12 In step E5 of the architecture shown, the ambient light sensor application of the SCP processor's co-application layer will t 01 The image noise at any given moment is sent to the noise memory of the SCP processor co-drive layer.

[0399] Steps B1 to B2, refer to Figure 7 The following is a description of steps B1 to B2 in the illustrated embodiment.

[0400] Step B3, the noise algorithm library in the hardware abstraction layer of the AP processor will t 02 Backlight noise at any given moment is sent to the HWC on the same layer.

[0401] Step B4, calculate t in the AP processor. 02 After the backlight noise at time t, 02 The ambient light sensor application in the co-application layer sends backlight noise to the SCP processor at any given time.

[0402] Step B5 ( Figure 11 In step E5 of the architecture shown, the ambient light sensor application of the SCP processor's co-application layer... 02 The backlight noise at any given moment is sent to the noise memory of the SCP processor co-drive layer.

[0403] Steps C1 to C9 and steps D1 to D9 are described in the same way as steps A1 to A9, and will not be repeated here.

[0404] After the ambient light sensor completes integration, the SCP processor is triggered to execute step E2, which refers to... Figure 7 Description of the illustrated embodiment.

[0405] In steps E3 to E6, the denoising module in the SCP processor co-driving layer retrieves the fused noise from the noise memory of this layer and obtains the raw values ​​of the four channels of the initial ambient light from the ambient light sensor of this layer. The target ambient light is calculated based on the raw values ​​of the four channels of the initial ambient light and the image noise and backlight noise that interfere with the initial ambient light. During non-integration periods, the image may also be refreshed (e.g., t...). 11 Image refresh time (at any moment), brightness may also change (e.g., t). 12 (The brightness changes over time). Therefore, during the non-integration period, Figure 13 Steps F1 to F9 still exist in the process. Figure 13 Steps F1 to F5 are omitted; please refer to [link / reference] for details. Figure 13 Steps A1 to A5 in the above steps make t 11 Image noise at any given moment is stored in the noise memory of the SCP processor. During the non-integration time period, steps G1 to G5 still exist. Figure 13 Step G1 is omitted; please refer to [link / reference] for details. Figure 13 Step B1 in the middle, so that t 12 The backlight noise at any given time is stored in the noise memory of the noise algorithm library.

[0406] The following will introduce Figure 7 The noise algorithm library in the illustrated embodiment calculates the target ambient light based on the target image, brightness, and initial ambient light.

[0407] Step 1: For each target image acquired by the noise calculation library, the image noise at the refresh time of the target image is calculated based on the target image and the brightness of the display screen at the refresh time of the target image; for each brightness level acquired by the noise calculation library, the backlight noise at the brightness adjustment time is calculated based on the brightness level and the target image at the brightness adjustment time.

[0408] Although image noise and backlight noise have different names, the calculation process for both is based on a frame of the target image and a brightness value.

[0409] The target image consists of multiple pixels. First, a weighted sum is calculated based on the RGB values ​​of each pixel and its weighting coefficient to obtain the weighted RGB values ​​of the target image. The weighting coefficient for each pixel is determined by the distance between its coordinates and the reference coordinates of the target image. The coordinates of the center point of the photosensitive area of ​​the ambient light sensor can be used as the reference coordinates of the target image.

[0410] Step two: The noise calculation library obtains the fusion noise based on the weighted RGB values ​​and brightness of the target image. The fusion noise can be obtained through a lookup table (which contains the fusion noise corresponding to the weighted RGB values ​​and brightness of the target image) or through a pre-set functional relationship (the independent variables are the weighted RGB values ​​and brightness of the target image, and the dependent variable is the fusion noise). The resulting fusion noise is the raw value of four channels.

[0411] Step 3: The noise calculation library calculates the integrated noise of the initial ambient light within the integration time period based on the fused noise at each moment.

[0412] It should be noted that image noise is not generated by the image refresh process itself. During the integration time period, the interference with the initial ambient light before image refresh is the image noise corresponding to the image before refresh, and the interference with the initial ambient light after image refresh is the image noise corresponding to the image after refresh.

[0413] Similarly, backlight noise is not generated by the brightness adjustment process itself. During the integration time period, the interference to the initial ambient light before brightness adjustment is the backlight noise corresponding to the brightness before adjustment, and the interference to the initial ambient light after brightness adjustment is the backlight noise corresponding to the brightness after adjustment.

[0414] As mentioned earlier, the noise memory stores image noise and backlight noise calculated by the noise algorithm library at various times. The noise stored in the noise memory is collectively referred to as fused noise or first noise.

[0415] Step A1: The first processor retrieves the first noise from the exit position of the noise memory through the noise algorithm library, and the first processor updates the exit position of the noise memory or the first noise at the exit position through the noise algorithm library.

[0416] Step B1: If the timestamp corresponding to the first noise extracted is at or before the first time, the first processor continues to execute step A1 through the noise algorithm library until the first noise extracted is after the first time.

[0417] Step B2: If the first noise extracted is after the first time interval, the first processor executes the following steps using the noise algorithm library:

[0418] Step C1: If the timestamp of the first noise extracted is after the first time and before the second time, calculate the integral noise between the first time and the time corresponding to the timestamp of the first noise extracted at the last time based on the first noise extracted previously, and continue execution from step A1.

[0419] Step C2: If the timestamp of the first noise extracted is after the first time and after the second time, then calculate the integrated noise between the first time and the second time based on the first noise extracted last time, and continue to execute step D1;

[0420] Step C3: If the timestamp of the first noise extracted now is not the first time after the first time but before the second time, then calculate the integrated noise between the time corresponding to the timestamp of the first noise extracted last time and the time corresponding to the timestamp of the first noise extracted now based on the first noise extracted last time; and continue execution from step A1.

[0421] Step C4: If the timestamp of the first noise currently extracted is not the first time after the first time, but is after the second time, calculate the integral noise between the time corresponding to the timestamp of the first noise extracted last time and the second time based on the first noise extracted last time, and continue to execute step D1;

[0422] Step D1: Obtain the second value based on the integrated noise between the first time and the second time and the first value.

[0423] When the noise memory is a FIFO (First Input First Output) memory, it is a first-in-first-out dual-port buffer. One port is the memory's input port, and the other is its output port. In this memory structure, the first data to enter is the first to be removed, and the order in which data is removed is the same as the order in which it is entered. The output address of the FIFO memory is the memory address corresponding to its output port.

[0424] The process of removing data from the FIFO memory is as follows: starting from the exit position (the first position), the fusion noise stored in the exit position is removed. Then, the data in the second position starting from the exit position is moved to the exit position, the data in the third position starting from the exit position is moved to the second position starting from the exit position, and so on.

[0425] Of course, in practical applications, after removing the fusion noise stored at the first position (A1) starting from the exit position (the first position, A1), the exit position of the memory can be updated to the second position (A2). After removing the fusion noise stored at the current exit position (A2) again, the exit position of the memory is updated to the third position (A3) and so on.

[0426] The process of obtaining the second value based on the above calculations can be found by referring to... Figure 14 The embodiments to Figure 16 The illustrated embodiment.

[0427] See Figure 14 , Figure 14 The noise calculation library in the AP processor provided in this application describes the process of calculating integral noise based on image noise and backlight noise. Each step in this process can be compared... Figure 9 and Figure 10 Description of each time point in the illustrated embodiment: at t 01 The image is refreshed continuously to obtain t 01 Image noise at time t; 02 Adjust the brightness in real time to obtain t 02 Backlight noise at time t; 03 The image is refreshed continuously to obtain t 03 Image noise at time t; 04 The image is refreshed continuously to obtain t 04 Image noise at any given moment.

[0428] From t 01 Time to t 02 At time t, the displayed image is t 01 The image is constantly refreshed, and the screen brightness is t. 01 Brightness at time t 01 The brightness at time t is the brightness value stored in the noise algorithm library. 01 The latest stored brightness value before the current moment, t 01 The image noise at time t is 01 The constantly refreshed image is displayed at a brightness of t. 01 The noise is present under the condition of brightness at a given moment. Therefore, the initial ambient light contains noise with a duration of "t". 02 -t 01 ", timestamp is t 01 Image noise.

[0429] From t 02 Time to t 03 At time t, the brightness of the display screen is 02 The brightness is adjusted over time, and the image displayed on the screen is t. 01The image is refreshed in real time, t 02 The backlight noise at time t is 02 The adjusted brightness is displayed on the screen. 01 The noise in the image after time adjustment. Therefore, the initial ambient light contains noise of duration "t". 03 -t 02 ", timestamp is t 02 Backlight noise at any given moment.

[0430] From t 03 Time to t 04 At time t, the displayed image is t 03 The image is constantly refreshed, and the screen brightness is t. 02 Brightness after time adjustment, t 03 The image noise at time t is 03 The constantly refreshed image is displayed at a brightness of t. 02 Noise under the condition of adjusted brightness at different times. Therefore, the initial ambient light contains noise with a duration of "t". 04 -t 03 ", timestamp is t 03 Image noise.

[0431] From t 04 From time t1 to time t2, the displayed image is t 04 The image is constantly refreshed, and the screen brightness is t. 02 Brightness after time adjustment, t 04 The image noise at time t is 04 The constantly refreshed image is displayed at a brightness of t. 02 Noise under the condition of brightness adjustment at different times. Therefore, the initial ambient light contains noise with a duration of "t1-t". 04 ", timestamp is t 04 Image noise.

[0432] Based on the above understanding, when the AP processor calculates the integral noise:

[0433] t 01 Image noise at time t 01 Time to t 02 The initial ambient light at any given moment causes interference;

[0434] t 02 Backlight noise at time t 02 Time to t 03 The initial ambient light at any given moment causes interference;

[0435] t 03 Image noise at time t 03 Time to t 04The initial ambient light at any given moment causes interference;

[0436] t 04 Image noise at time t 04 The initial ambient light from time t1 to time t1 causes interference.

[0437] Therefore, t can be calculated separately. 01 Time to t 02 Integral noise at time t 02 Time to t 03 Integral noise at time t 03 Time to t 04 Integral noise at time t 04 Integral noise from time t1 to time t2.

[0438] For t 01 Time to t 02 The integral noise at time t is: (t 02 -t 01 ) / (t1-t0)×N t01 .

[0439] For t 02 Time to t 03 The integral noise at time t is: (t 03 -t 02 ) / (t1-t0)×N t02 .

[0440] For t 03 Time to t 04 The integral noise at time t is: (t 04 -t 03 ) / (t1-t0)×N t03 .

[0441] For t 04 The integral noise from time t1 to time t2 is: (t1-t 04 ) / (t1-t0)×N t04 .

[0442] Where, N t01 Indicates t 01 Timing-based fusion noise, N t02 Indicates t 02 Timing-based fusion noise, N t03 Indicates t 03 Timing-based fusion noise, N t04 Indicates t 04 Moment-level fusion noise.

[0443] And each sub-time period (t) within the integration time period 01 To t 02 , t02 To t 03 , t 03 To t 04 , t 04 The integral noise up to t1) is the integral noise for the entire integration time period.

[0444] In the example above, the start time of the integration time period is the same as the image refresh time, which means that the image noise at the start time of the integration time period can be obtained.

[0445] In practical applications, the start time of the integration period may not be the image refresh time or the backlight adjustment time. In this case, it is necessary to obtain the fusion noise corresponding to the most recent change time (image refresh time or backlight adjustment time) before the start of the current integration period.

[0446] See Figure 15 As shown, the noise calculation library in the AP processor provided in this embodiment of the application obtains the integration time period (t). 01 A schematic diagram of the integral noise from time t1 to time t2. 01 The current time point is no longer the start time of the current integration period, but rather a single image refresh time within that period. The most recent change time (image refresh time or brightness adjustment time) before the start of the current integration period is t. -1 The time is the moment the image is refreshed.

[0447] See Figure 16 As shown, if the most recent change time before the start of this integration time period is the image refresh time, then the image noise corresponding to that image refresh time will affect the time from t0 to t... 01 The initial ambient light at that moment caused interference.

[0448] Of course, if the most recent change occurred during a brightness adjustment, then the backlight noise corresponding to that brightness adjustment will affect the time from t0 to t1. 01 The initial ambient light at that moment caused interference.

[0449] Figure 16 In the embodiment shown, the integral noise corresponding to each sub-time period in the integral time period is as follows:

[0450] For time t0 to t 01 The integral noise at time t is: (t 01 -t0) / (t1-t0)×N t-1 .

[0451] For t 01 Time to t 02 The integral noise at time t is: (t 02 -t 01) / (t1-t0)×N t01 .

[0452] For t 02 Time to t 03 The integral noise at time t is: (t 03 -t 02 ) / (t1-t0)×N t02 .

[0453] For t 03 Time to t 04 The integral noise at time t is: (t 04 -t 03 ) / (t1-t0)×N t03 .

[0454] For t 04 The integral noise from time t1 to time t2 is: (t1-t 04 ) / (t1-t0)×N t04 .

[0455] Where, N t-1 Indicates t -1 Timing-based fusion noise, N t01 Indicates t 01 Timing-based fusion noise, N t02 Indicates t 02 Timing-based fusion noise, N t03 Indicates t 03 Timing-based fusion noise, N t04 Indicates t 04 Moment-level fusion noise.

[0456] As can be understood from the above example, the obtained integral noise is also the raw value on the four channels.

[0457] In the examples above, the timestamps are all different. In practical applications, HWC may perform both the process of acquiring the target image and the process of acquiring the brightness to be adjusted within a single time unit (e.g., 1 ms). However, the timestamps of the acquired target image and the brightness to be adjusted are the same at that moment.

[0458] If there are target images and brightness values ​​with the same timestamp, and the noise algorithm library receives the target image first, then the noise algorithm library first calculates the image noise based on the target image and the latest brightness value before the target image. When calculating the backlight noise corresponding to the brightness value, the backlight noise is calculated based on the target image and brightness value with the same timestamp.

[0459] If there are target images and brightness values ​​with the same timestamp, and the noise algorithm library receives the brightness value first, then the noise algorithm library first calculates the backlight noise based on the brightness value and the latest target image before the brightness value. When calculating the image noise corresponding to the target image, the image noise is calculated based on the target image and brightness value with the same timestamp.

[0460] The noise algorithm library receives the target image first, calculates the image noise, and stores it in the noise memory. The fused noise stored in the noise memory follows a temporal order; that is, before storing, it checks if the currently stored fused noise is after the timestamp of the previously stored fused noise. If it is, it is stored; otherwise, it is discarded. Therefore, the backlight noise calculated later is discarded.

[0461] In practical applications, the timestamp of the target image can be the moment when HWC begins retrieving the target image from CWB memory. The timestamp of the brightness value can be the moment when HWC begins retrieving the brightness value from the kernel node. During the process of retrieving the target image, HWC may switch to retrieving the brightness value. Therefore, although HWC retrieves the target image first and then the brightness value, the timestamp of the brightness value is later than the timestamp of the target image. In practical applications, HWC may first retrieve the brightness value and send it to the noise algorithm library, which calculates and stores the backlight noise. Then, HWC may retrieve the target image and send it to the noise algorithm library, which calculates and stores the image noise. This results in the timestamp of the currently stored image noise being earlier than the timestamp of the previously stored backlight noise.

[0462] Step 4: The noise algorithm library removes the integral noise from the initial ambient light over the entire integration time period to obtain the target ambient light.

[0463] In this embodiment, the initial ambient light data sent by the SCP processor to the HWC of the AP processor is in the form of RGBC four-channel raw values. The initial ambient light data sent by the HWC to the noise algorithm library is also in the form of RGBC four-channel raw values. The raw values ​​of the four channels of the integrated noise are obtained in step three. Therefore, in this step, the four-channel raw values ​​of the initial ambient light and the four-channel raw values ​​of the integrated noise can be calculated to obtain the raw values ​​of the four channels of the target ambient light.

[0464] After the noise algorithm library calculates the raw values ​​of the four channels of the target ambient light, it can send the raw values ​​of the four channels of the target ambient light to the SCP processor. The SCP processor calculates the lux value of the target ambient light based on the raw values ​​of the four channels of the target ambient light.

[0465] As an example, the lux value can be obtained by multiplying the raw value of each channel by a coefficient for each channel (which can be provided by the ambient light sensor manufacturer) and then weighting the values.

[0466] As mentioned earlier, each time the electronic device refreshes the image (or every one or two intervals, etc.), the DSS in the AP processor stores the image before display (which can be understood as the image to be refreshed in this refresh process or the image after this refresh) in the CWB memory. The HWC in the AP processor extracts the target image from the image to be refreshed stored in the CWB memory, and then sends the target image to the noise algorithm library. For ease of description, the step of the DSS storing the image before display in the CWB memory and the HWC obtaining the target image from the CWB memory is referred to as the CWB write-back function.

[0467] With the AP processor's CWB write-back function enabled, the DSS stores the image before display in the CWB memory every time the electronic device refreshes the image (or every one or two intervals, etc.). After storing the image in the CWB memory, the DSS sends a successful storage message to the HWC module. Upon receiving the successful storage message, the HWC retrieves the target image from the CWB memory. Correspondingly, the HWC module can send the target image to the noise algorithm library.

[0468] That is, when the AP processor's CWB write-back function is enabled, the AP processor executes the following when the electronic device refreshes the image each time (or every one or two intervals, etc.). Figure 7 Steps A1 to A6 in the above.

[0469] When the CWB write-back function of the AP processor is stopped, the electronic device will follow the refresh display procedure every time (or every one or two intervals, etc.) the image is refreshed. Figure 7 Steps A1 to A3 in the illustrated technical architecture. However, the DSS no longer stores the image before display in the CWB memory, and correspondingly, the AP processor no longer executes the subsequent related steps.

[0470] That is, when the CWB write-back function of the AP processor is stopped, the AP processor executes the following when the electronic device refreshes the image each time (or every one or two intervals, etc.): Figure 7 Steps A1 to A3 in the code are not executed. Figure 7 Steps A4 to A6 in the technical architecture shown.

[0471] As mentioned earlier, the noise algorithm library calculates image noise based on the received target image. Therefore, the corresponding image noise is only obtained when the electronic device refreshes the image during the AP processor's CWB write-back function activation state, and not when the electronic device refreshes the image during the AP processor's CWB write-back function deactivation state.

[0472] by Figure 9 and Figure 14 For example, after the integration ends (at time t1), when the noise algorithm library calculates the target ambient light from time t0 to time t1, the fusion noise used is: t 01 Image noise at time t 02 Backlight noise at any time, t 03 Image noise at time t 04 Image noise at any given time. Unnecessary fusion noise includes at least: t 11 Image noise at time t 12 Backlight noise at any given moment. This refers to the image noise and backlight noise between the start and end times of the integration period, which interferes with the initial ambient light acquired during the integration process. However, image noise and backlight noise outside the integration period may not interfere with the initial ambient light acquired during the integration process.

[0473] Therefore, to reduce power consumption, during the integration period of the ambient light sensor, the AP processor can initiate the CWB write-back function via HWC, and the AP processor will execute steps A4 to A6. During the non-integration period of the ambient light sensor, the CWB write-back function can be stopped via HWC, and the AP processor will no longer execute steps A4 to A6.

[0474] See Figure 17 , Figure 17 This is a schematic diagram of a method for starting and stopping the CWB write-back function, provided in an embodiment of this application.

[0475] like Figure 17 As shown, during the integration periods of the ambient light sensor (from t0 to t1, from t2 to t3, from t4 to t5), the HWC controls the CWB write-back function to start, and during the non-integration periods (from t1 to t2, from t3 to t4, from t5 to t6), the HWC controls the CWB write-back function to stop. This method ensures that image noise during each integration process is captured while also reducing the power consumption of the AP processor.

[0476] This application focuses on describing the start / stop method for the CWB write-back function. In this method, the HWC in the AP processor can monitor whether the data in the kernel node changes during both the integration and non-integration periods. When a change occurs, the HWC can obtain the brightness to be adjusted, and correspondingly, the noise algorithm library can calculate the backlight noise for the entire acquisition cycle.

[0477] The subsequent embodiments of this application all take the example that the HWC in the AP processor can monitor the changes in data in the kernel node during both the integration time period and the non-integration time period. When the data stored in the kernel node changes, the HWC obtains the brightness to be adjusted from the kernel node and transmits it to the noise algorithm library to calculate the backlight noise.

[0478] In addition, since the integration process of the ambient light sensor is controlled by the SCP processor, the SCP processor needs to send the time-related parameters of the ambient light sensor during the initial acquisition of ambient light to the AP processor.

[0479] As mentioned earlier, after obtaining the initial ambient light at the end of each integration, the SCP processor can transmit the initial ambient light and the time related to the integration process of this initial ambient light (e.g., the end time and duration of this integration, or the start and end times of this integration) to the HWC in the AP processor. The SCP processor can also send the start time (or a period before) of the next integration attempt by the ambient light sensor to acquire the initial ambient light as the start time of the CWB write-back function to the HWC in the AP processor. It can also send the end time (or a period after) of the next integration attempt to acquire the initial ambient light as the stop time of the CWB write-back function to the HWC in the AP processor. In other words, the SCP processor sends the start and stop times of the CWB write-back function to the HWC in the AP processor.

[0480] In practical applications, when the ambient light sensor's acquisition period is fixed, the SCP processor can send the integration start time of the next initial ambient light acquisition as the CWB write-back function activation time to the AP processor. The AP processor calculates the CWB write-back function termination time based on the received activation time and acquisition period. Alternatively, the SCP processor can report the first integration start time, integration duration, sampling period, etc. The AP processor then determines the CWB write-back function activation and termination times based on this data.

[0481] It should be noted that this application embodiment does not limit the time-related parameters sent by the SCP processor to the AP processor during the initial ambient light acquisition process by the ambient light sensor. The AP processor only needs to be able to determine the next startup time of the CWB write-back function based on the received time-related parameters.

[0482] Furthermore, the start time of the CWB write-back function does not completely coincide with the start time of integration, nor does the stop time completely coincide with the end time of integration. The time period corresponding to the start state of the CWB write-back function needs to include the integration time period within the acquisition cycle. Taking one acquisition cycle as an example, the start time of the CWB write-back function is earlier than or equal to the start time of the integration time period of one acquisition cycle, and the stop time of the CWB write-back function is later than or equal to the start time of the integration time period of one acquisition cycle.

[0483] As another example, the SCP processor can also send the initial ambient light, the integration duration corresponding to the initial ambient light (or the start time of this integration), the end time of this integration, and the sleep duration of the CWB write-back function to the HWC in the AP processor after the current integration ends. For ease of description, the information sent at the end of this integration can be collectively referred to as the first information. The first information is not limited to the above information and may include more or less of the above information.

[0484] As mentioned earlier, the noise algorithm library calculates the target ambient light based on the initial ambient light, the integration duration corresponding to the initial ambient light (or the start time of this integration), the end time of this integration, and the corresponding fusion noise. The detailed process is described in the above embodiments and will not be repeated here.

[0485] The HWC in the AP processor needs to determine when to start the CWB write-back function based on the AP processor's sleep duration.

[0486] Of course, the aforementioned first information can also be broken down into multiple sub-information messages and sent separately to the HWC of the AP processor. This application embodiment does not limit this.

[0487] Taking the above information as an example, the SCP processor can transmit the moment when the SCP processor sends the first information.

[0488] After receiving the first information, the HWC in the AP processor first controls the CWB write-back function to stop. Then, based on the partial information in the received first information, the HWC in the AP processor determines the start time of the CWB write-back function or how long it needs to wait to reach the start time of the CWB write-back function.

[0489] Since the CWB write-back function can be started before the start of the integration in the next cycle, it does not have to be strictly controlled at a certain point in time. Therefore, the start time of the CWB write-back function can be obtained in any of the following ways or in other ways not shown in the embodiments of this application.

[0490] The HWC in the AP processor obtains the inter-core communication delay based on the moment the SCP processor sends the first message and the moment the AP processor receives the first message. Then, the HWC in the AP processor calculates the remaining waiting time (sleep time minus inter-core communication delay) or the CWB write-back function startup time (the moment the AP processor receives the first message via the HWC plus the remaining waiting time) based on the inter-core communication delay and the sleep duration. Once the CWB write-back function startup time arrives, the HWC in the AP processor initiates the CWB write-back function.

[0491] As an example, when the total duration of the non-integration period is 300ms, the sleep duration can be 240ms, 250ms, 260ms, 270ms, 280ms, etc. Therefore, even with inter-core communication delay (e.g., 1ms), the CWB write-back function can be guaranteed to start before the next integration begins.

[0492] Of course, in practical applications, the AP processor can also calculate the start time of the CWB write-back function (the end time of integration plus the sleep duration) or the remaining waiting time before the start time of the CWB write-back function (the end time of integration plus the sleep duration minus the time when the AP processor receives the first information through the HWC) based on the integration end time and sleep duration sent by the SCP processor via HWC. In this example, the start time of the CWB write-back function is recorded as the first time. The first time is also the time after the second duration has elapsed since the matting mark was set to the first character. The sleep duration in the first information can be recorded as the first duration. The second duration is: the sleep duration minus the delay duration. The delay duration is: the time when the HWC module receives the first information minus the duration of the integration end time, and the integration end time can be recorded as the second time. As mentioned above, the first information received by the AP processor can also include the integration start time (first time), the integration end time (second time), the initial ambient light (first value), and the sleep duration (first duration), etc.

[0493] Since the cutout marker represents the start and stop of the CWB write-back function, the duration during which the CWB write-back function is stopped can also be understood as the duration during which the cutout marker is set to the first character.

[0494] As mentioned above, the start time of the CWB write-back function in this embodiment is not strictly fixed at a certain time. Therefore, other calculation methods can also be used in this embodiment, as long as the start time of the CWB write-back function is before the integration start time. Therefore, the AP processor can also use the sleep duration in the received first information as the sleep duration of the CWB write-back function, ignoring the communication delay.

[0495] The above examples all use the AP processor and SCP processor time alignment as an example. If the AP processor and SCP processor time are not aligned, the time difference when the two times are not aligned needs to be considered based on the time or moment obtained above.

[0496] As mentioned earlier, the CWB write-back function stops at the moment when the AP processor receives the first information through the HWC or after that. Therefore, after the CWB write-back function is started, the HWC will stop the CWB write-back function in the current start state at the moment when it receives the first information sent by the SCP processor or after that.

[0497] Based on the above, it can be understood that the CWB write-back function is disabled after integration is complete. The activation time of the CWB write-back function can be determined based on the AP processor's sleep duration.

[0498] See Figure 18 On the AP processor side, each time the electronic device refreshes the image, the Surface Flinger sends the interface display parameters to the HWC (see details). Figure 7 In the illustrated embodiment, step A1) involves the HWC obtaining the composite image based on the display parameters. The HWC needs to query the matting markers. If the matting marker indicates that the function is enabled, the HWC initiates the CWB write-back function, and the AP processor can then execute... Figure 7 Steps A1 to A6 in the technical architecture shown. That is, after the CWB write-back function is started, the noise algorithm library can calculate the image noise and backlight noise during the CWB write-back function startup period.

[0499] In this example, the cutout marker can be denoted as the write-back marker.

[0500] The image matting markers in the above embodiments can also exist in the form of identifiers. After the HWC receives the first information sent by the SCP processor, the HWC sets the identifier to a first character (e.g., 0, False). After waiting for the sleep duration, the HWC sets the identifier to a second character (e.g., 1, True). If the identifier is the first character (e.g., 0, False), the HWC controls the CWB write-back function to stop. If the identifier is the second character (e.g., 1, True), the HWC controls the CWB write-back function to start.

[0501] In this example, the first character can be denoted as the first marker, and the second character can be denoted as the second marker.

[0502] HWC controls whether the CWB write-back function is started or stopped by querying the identifier.

[0503] As an example, HWC executes each time Figure 7Before step A2 in the illustrated technical architecture, it's possible to check whether the identifier is currently the first or second character. If it's the second character, it indicates that the CWB write-back function is enabled. In this case, when HWC executes step A2, it transmits the information that needs to be cut out. After receiving the composited image and the information that needs to be cut out from HWC, the display subsystem stores the image to be displayed in CWB memory. If it's the first character, it indicates that the CWB write-back function is disabled. In this case, when HWC executes step A2, it no longer transmits the information that needs to be cut out (or transmits the information that doesn't need to be cut out along with it). After receiving the composited image from HWC, if the display subsystem doesn't receive the information that needs to be cut out (or receives the information that doesn't need to be cut out), it no longer stores the image to be displayed in CWB memory. HWC will then be unable to obtain the target image.

[0504] As an example, when the image matting marker is designated as the second marker, if the electronic device refreshes the image (which can be referred to as the fifth image), the surface flinger transmits the interface display parameters (which can be referred to as the fourth display parameters) to the HWC. After receiving the fourth display parameters, the HWC can call the underlying hardware to synthesize the image. When the HWC transmits the synthesized image (which can be the fifth image, or an image that has been processed to obtain the fifth image) to the display subsystem, it can also transmit the information that needs to be matted (which can be referred to as the third information). Upon receiving the fifth image and the third information, the display subsystem can store the fifth image, or a portion of the fifth image (which can be referred to as the sixth image), and the target image on the fifth image (which can be referred to as the third target image) in the CWB memory. The HWC obtains the target image from the CWB memory and sends it to the noise algorithm library. The noise algorithm library can obtain image noise based on the target image (which can be referred to as the second image noise).

[0505] As another example, when the matting marker is set to the first marker, if the electronic device refreshes the image (which can be denoted as the first image), the surface flinger transmits the interface display parameters (which can be denoted as the fifth display parameter) to the HWC. After receiving the fifth display parameter, the HWC can call the underlying hardware to synthesize the image. When the HWC transmits the synthesized image (which can be the first image or an image that has been processed to obtain the first image) to the display subsystem, it no longer transmits the information that needs to be matted. When the display subsystem receives the first image, it no longer stores the first image, a portion of the first image (which can be denoted as the second image), and the target image on the first image (which can be denoted as the first target image) in the CWB memory. Accordingly, the HWC cannot obtain the target image from the CWB memory, nor does it send the target image to the noise algorithm library. The noise algorithm library also no longer obtains image noise based on the target image.

[0506] like Figure 18 As shown on one side of the SCP processor, after the SCP processor is started, the ambient light sensor driver is initialized, and then ambient light integration is started according to the preset acquisition cycle.

[0507] After the ambient light integration is completed, the screen on / off state can also be monitored. The screen on / off state is monitored by the HWC on the AP processor side. After the AP processor monitors the screen on / off event, it sends relevant information to the SCP processor to trigger the change of the screen on / off state on the SCP processor side.

[0508] In the screen-on state, the SCP processor needs to send the initial ambient light it collects to the HWC of the AP processor. The HWC in the AP processor then sends the initial ambient light to the noise algorithm library, which calculates the raw value of the target ambient light based on the received initial ambient light. The AP processor then sends the raw value of the target ambient light to the ambient light memory of the SCP processor. Alternatively, in practical applications, the AP processor can also calculate the lux value of the target ambient light based on its raw value. The AP processor then sends the lux value of the target ambient light to the SCP processor.

[0509] See Figure 18 When the screen is on, the SCP processor needs to send the end time of this scoring session and the sleep duration to the AP processor.

[0510] The AP processor receives the end time of this integration and the sleep duration reported by the SCP processor. HWC sets the cutout marker to the first character; when the cutout marker is set to the first character, HWC will stop the CWB write-back function.

[0511] In the AP processor's HWC (Hidden Wound Control Center), the matting thread (the thread responsible for acquiring the target image) receives the first information and first sets the matting marker to the first character. Then, it calculates the required sleep duration and calls the `sleep` function based on this duration. When the matting thread calls the `sleep` function, it passes the required sleep duration (e.g., 270ms), and the matting thread will sleep for 270ms. After 270ms, the matting thread ends its sleep. After the matting thread finishes its sleep, it sets the matting marker to the second character, and the CWB write-back function is activated.

[0512] The SCP processor also needs to calculate the lux value of the target ambient light based on its raw value. Additionally, ambient light integration will continue at the start of the next integration iteration.

[0513] When the screen is off, the initial ambient light collected by the ambient light sensor is the actual ambient light. Therefore, the SCP processor no longer needs to report the lux value of the initial ambient light collected over the integration period to the AP processor. Since the CWB write-back function does not need to be activated to obtain relevant noise in the screen-off state, the SCP processor no longer calculates the time for the next CWB write-back function activation.

[0514] Of course, in some scenarios, such as face unlock when the screen is off, the electronic device needs to know whether the current environment is dark, and in darker environments, it needs to provide supplemental lighting for the face. Therefore, in such scenarios, the electronic device needs to know the lux value of the current ambient light. Thus, even when the screen is off, the ambient light sensor still needs to collect ambient light data, and when the SCP processor receives a face unlock request from the AP processor, it reports the collected ambient light lux value to the AP processor so that the AP processor can determine whether supplemental lighting is needed based on the received ambient light lux value.

[0515] This application focuses on describing how the HWC in the AP processor controls the start and stop of the CWB write-back function. Other details not shown can be found in the description of any of the above embodiments.

[0516] As mentioned earlier, the start and stop times of the CWB write-back function in the AP processor are determined by the data reported by the SCP processor. Considering the potential data transmission delay in inter-core communication between the AP processor and the SCP processor, it can be configured such that after the AP processor determines that the display is on, the HWC in the AP processor keeps the CWB write-back function constantly open. After the HWC receives the first information reported by the SCP processor, it begins to control the CWB write-back function to cycle through starting and stopping according to the start / stop scheme described in any of the above embodiments.

[0517] by Figure 9 Taking the illustrated embodiment as an example, if using Figure 17 The following are the methods for enabling and disabling the CWB write-back function:

[0518] HWC can obtain t 01 The target image at time t will also be calculated by the noise algorithm library. 01 Image noise at any given time;

[0519] HWC can obtain t 02 The noise algorithm library will also calculate the brightness value to be adjusted at time t. 02 Backlight noise at any given moment;

[0520] HWC can obtain t 03 The target image at time t will also be calculated by the noise algorithm library. 03 Image noise at any given time;

[0521] HWC can obtain t 04 The target image at time t will also be calculated by the noise algorithm library. 04 Image noise at any given time;

[0522] HWC no longer receives t 11 The target image at time t will not be calculated by the noise algorithm library. 11 Image noise at any given time;

[0523] HWC can obtain t 12 The brightness value to be adjusted at time t is not calculated by the noise algorithm library. 12 Backlight noise at any given moment.

[0524] If the electronic device refreshes the image at a frequency of 60Hz, and the video is played on the electronic device, using the CWB write-back function always-on scheme, then within a non-integral time period (300ms for example) of one acquisition cycle (350ms), HWC may acquire the target image 300ms / (1000ms / 60) = 18 times, and the noise algorithm library may calculate and store the image noise 18 times.

[0525] use Figure 17 The CWB write-back function start / stop scheme shown reduces the number of HWC image acquisition processes by 18 and the number of noise algorithm library calculations by 18 within one acquisition cycle (350ms). Clearly, adopting this scheme... Figure 17 The CWB write-back function start / stop scheme shown can reduce power consumption.

[0526] However, in Figure 15 and Figure 16 In the illustrated embodiment, if t -1 The current time is within the non-integration period of the previous acquisition cycle. Because the CWB write-back function stops during the non-integration period (steps A4 to A6 are no longer executed), the display subsystem no longer stores the image to be refreshed in the CWB memory. Consequently, the HWC does not obtain the t value. -1 The target image at time t was not obtained from the noise algorithm library. -1 Target image at time t and t -1 Image noise at time t. Correspondingly, t is not present in the noise memory. -1 Image noise at time t0. Therefore, when the noise algorithm library calculates the integral noise for each sub-time period, it will lose the information from time t0 to t... 01 The initial ambient light at time t0 contains interfering fusion noise. During the loss of the pair from time t0 to t... 01 If the initial ambient light at the corresponding time point contains interfering fusion noise, the noise algorithm library will use the t stored in the noise memory. -1The fusion noise before time t0 is used as interference from time t0 to time t1. 01 The initial ambient light fusion noise at any given moment leads to inaccurate calculation of the target ambient light.

[0527] To address this issue, the CWB write-back function can be activated before the start of each integration time period. After the CWB write-back function is activated, the image is forced to refresh once to ensure that the display subsystem stores the forced-refresh image in the CWB memory. The HWC can then extract the target image corresponding to the forced-refresh image from the CWB memory. Correspondingly, the noise algorithm library will also calculate the image noise corresponding to the forced-refresh time. In this embodiment, the forced-refresh image is designated as the third image.

[0528] As mentioned earlier, before the forced image refresh, the CWB write-back function has already been activated, meaning the cutout marker has been recorded as the second marker. Therefore, when the HWC module sends the forced refresh image to the display subsystem, it also transmits the information requiring cutout (this information can be recorded as the second information). Correspondingly, the display subsystem can store the forced refresh image, a portion of the forced refresh image (recorded as the fourth image), or the target image (recorded as the second target image) in the CWB memory. As mentioned earlier, this target image can acquire corresponding image noise (recorded as the first image noise).

[0529] The HWC contains an interface for forcibly refreshing the image. When the HWC determines that a forced image refresh is necessary, it calls this interface, and the electronic device forces a single image refresh. When the HWC calls this interface, it sends a first signal to the Surface Flinger. Upon receiving this first signal, the Surface Flinger retrieves the latest cached display parameter from its cache; this first display parameter is designated as the first display parameter. The Surface Flinger then sends this display parameter to the HWC module. Based on this parameter, the HWC calls the underlying hardware to obtain the composite image (this image is the third image). If the matting marker is set to the second marker, the HWC sends the composite image to the display subsystem, carrying the information about the matting required.

[0530] In practical applications, the latest cached display parameter in the Surface Flinger cache may be the display parameter corresponding to the previous image refresh. If the electronic device refreshes the image before a forced image refresh, and that refreshed image is the first image, then the latest cached display parameter in the Surface Flinger cache may be the fifth display parameter used to generate the first image. Therefore, the third image may be the same as the first image. Thus, the image that the electronic device uses for forced image refresh can be the image currently displayed on the electronic device's screen (after the last refresh to the first image, the electronic device's screen continues to display the first image). The process of forced image refresh is the same as the normal image refresh process, both involving Surface Flinger, HWC, OLED driver, display subsystem, and finally display. The specific process can be referred to the description in the above embodiments, and will not be repeated here.

[0531] In this embodiment, the purpose of forcibly refreshing the image is to ensure that the image currently displayed on the screen is the image from the last refresh of the screen. In practical applications, before sending the image to be displayed, the display subsystem can cache a frame of image, which can be understood as either the image currently displayed on the screen or the image from the last refresh of the screen. The HWC retrieves the image from the cache and then transmits the image and the information to be cut out to the display subsystem. The display subsystem can then save the image (or the area image of the image, or the target image corresponding to the image) in the CWB memory, and the HWC performs the step of retrieving the target image from the CWB memory.

[0532] As mentioned earlier, if HWC needs to perform matting on a refreshed image to obtain the target image, HWC can carry the matting information when transmitting the composite image. If HWC does not need to perform matting on the currently refreshed image, HWC can omit the matting information (or carry information indicating that matting is not required). The display subsystem uses whether the received image carries the matting information as the basis for storing it in CWB memory. If the received image carries the matting information, then... Figure 7 Steps A4 to A6 in the illustrated technical architecture are executed without further processing if the received image does not carry information requiring image matting (or carries information that does not require image matting). Figure 7 Steps A4 to A6 in the technical architecture shown.

[0533] The forced image refresh is performed after the CWB write-back function is activated, therefore, the AP processor executes... Figure 7 In steps A2 to A3 of the technical architecture shown, the transmitted data carries information that needs to be cut out.

[0534] See Figure 19 This application provides a start / stop scheme for forcibly refreshing the image once after activating the CWB write-back function at a first preset time before the integration begins. In this application and subsequent embodiments, for ease of drawing, the start time of the CWB write-back function and the time of forcibly refreshing the image are set to the same time. In practical applications, the image is forcibly refreshed only after the CWB write-back function has been activated. For ease of drawing, the stop time of the CWB write-back function and the integration end time are set to the same time. In practical applications, the stop time of the CWB write-back function can be later than the integration end time.

[0535] like Figure 19 As shown, the first preset time (t2-t) before the start of integration in each acquisition cycle. 1n t4-t 3n t6-t 5n The corresponding time (t) 1n t 3n t 5n After activating the CWB write-back function, the image is forcibly refreshed once. This can also be understood as a second preset time (t) after the start of the non-integral period of each acquisition cycle. 1n -t1、t 3n -t3、t 5n -t5) corresponds to the time (t 1n t 3n t 5n After activating the CWB write-back function, the image is forcibly refreshed once. The sum of the first preset time and the second preset time is the duration of a non-integral time period.

[0536] Taking the first acquisition cycle as an example, during the non-integral time period t of the first acquisition cycle (T1) 1n At a certain time, the HWC in the AP processor initiates the CWB write-back function and forces a refresh of the image after initiation. The HWC can obtain t 1n The target image corresponding to the time when the image is forcibly refreshed at each moment can be calculated by the noise algorithm library. 1n Image noise at time t, noise algorithm library will t 1n Image noise at any given time is stored in a noise memory. Other acquisition cycles can refer to the example of this acquisition cycle, and will not be repeated here.

[0537] To verify Figure 19 The CWB write-back function start / stop scheme shown will not lose the fusion noise that interferes with the initial ambient light acquired during the integration time period. See [link / reference]. Figure 20 The illustrated embodiment, in Figure 20 In the illustrated embodiment, the non-integration time period t before the integration start time of the second acquisition cycle (T2) is... 1nAt a certain time, the HWC in the AP processor initiates the CWB write-back function and forces a refresh of the image after initiation. The HWC can obtain t 1n The target image corresponding to the time of the forced image refresh is refreshed at each time. The noise algorithm library can cache t. 1n The target image at time t is obtained, and t is calculated. 1n Image noise at time t, noise algorithm library will t 1n Image noise at any given time is stored in a noise memory.

[0538] In t 1n From time t1 to the start of the integration time period (t2), there is no brightness adjustment or image refresh.

[0539] If there is only one brightness adjustment in the second acquisition cycle (T2): t 21 Brightness adjustment at any given time. The noise algorithm library can then be based on t. 1n The target image corresponding to the image refreshed at each time and t 21 The adjusted brightness at time t is obtained 21 Backlight noise at time t. The noise algorithm library will use t 21 The backlight noise at any given moment is sent to the noise memory.

[0540] After the integration time period of the second acquisition cycle ends (time t3), the noise memory stores t. 1n Image noise at time t 21 Backlight noise at any given moment.

[0541] See Figure 21 The integral noise of the initial ambient light interfering with the second acquisition cycle is as follows:

[0542] The duration is "from time t2 to t 21 The t of "moment" 1n Image noise at any given time;

[0543] Duration is "t" 21 t from time t2 21 Backlight noise at any given moment.

[0544] pass Figure 20 As can be understood from the illustrated embodiment, if a start-stop scheme that forcibly refreshes the image before integration begins is adopted:

[0545] If there is no image refresh between the time of this forced image refresh and the start time of the next integration, the noise algorithm library can also obtain the first sub-time period (from time t2 to t) that affects this integration time period. 21 Fusion noise at time (t) 1n (Moment-time fusion noise).

[0546] Furthermore, there is a brightness adjustment (t) between the time of this forced image refresh and the time of the next image refresh. 21 When adjusting the brightness at a given time, the target image (t) corresponding to the brightness adjustment time can be obtained. 1n The target image at time (t) is used to obtain the correct brightness adjustment time. 21 The backlight noise corresponding to (time).

[0547] When an electronic device plays video through its display screen, the image displayed on the screen may refresh at a frequency of 60Hz, that is, once every 16.7ms. The ambient light sensor's acquisition period can be set to 350ms, the integration time period can be set to 50ms, and the non-integration time period can be set to 300ms. Even before the start of the integration time period (e.g., t2-t) 1n =20ms) to start the CWB write-back function, and force a refresh of the image once after the CWB write-back function is started. Then, in one acquisition cycle, it is equivalent to reducing the process of HWC acquiring the target image and the noise algorithm library calculating the image noise by (300-20) / 16.7=16.8 times.

[0548] In the above embodiments, t2-t 1n =20ms, in practical applications, t2-t 1n It can also be equal to other duration values. In this embodiment of the application, for t2-t... 1n The corresponding duration is set to ensure that the noise algorithm library can obtain the target image and the image noise once before the integration begins. Therefore, the above embodiment can reduce processor power consumption while obtaining accurate target ambient light.

[0549] In practical applications, the display screen of electronic devices may not be constantly refreshing when it is on, or it may remain idle for a long time.

[0550] A display screen operates in two states when it is on: idle and refresh. In practical applications, the time of the last image refresh can be obtained, and the difference between the current time and the last image refresh time can be used to determine whether the display is currently in refresh or idle state. A threshold can be preset: if the difference between the current time and the last image refresh time is less than the threshold, the display is currently in refresh state; if the difference is greater than or equal to the threshold, the display is currently in idle state.

[0551] This application's embodiments are not intended to strictly distinguish between refresh state and idle state. They are merely meant to illustrate that when the image displayed on the electronic device's screen remains unchanged for an extended period (idle state), the image displayed on the screen is always the image corresponding to the last refreshed image.

[0552] As another example, when a user views an interface on an electronic device and does not perform any operation for a long time, and there is no animation on the current interface, the display is in an idle state before the screen goes black. When the electronic device's display is playing video, the display may refresh the image at a frequency of 60Hz, and the display is in a refresh state. In the embodiments of this application, the display is in a refresh state, and the image displayed on the display may or may not change. The content displayed on the display does not change in the refresh state because: the AP processor executes... Figure 7 The image obtained in steps A1 to A3 of the technical architecture shown is before the refresh, and the AP processor executes the process. Figure 7 The refreshed images obtained from steps A1 to A3 in the illustrated technical architecture are exactly the same. The display screen remains idle, and the displayed image does not change because AP processing is no longer being executed. Figure 7 In the illustrated technical architecture, steps A1 to A3 show that the subsystem still executes the AP processor for the last time according to the preset refresh frequency. Figure 7 The images obtained in steps A1 to A3 of the technical architecture shown are sent to the display screen of the electronic device for display.

[0553] See Figure 22 This is a schematic diagram of a refresh state and an idle state provided in an embodiment of this application. Figure 22 The screen remained lit throughout.

[0554] During the TS0 cycle, the various modules in the AP processor work together to synthesize the image 1 that the display screen needs to refresh during the TS1 cycle.

[0555] During the TS1 cycle, the display subsystem sends image 1 to the display screen, which displays image 1 synthesized by the various modules in the AP processor during the TS0 cycle. At the same time, the various modules in the AP processor synthesize image 2, which is to be refreshed on the display screen during the TS2 cycle.

[0556] During the TS2 cycle, the display subsystem sends image 2 to the display screen, which displays image 2 synthesized by the various modules in the AP processor during the TS1 cycle. At the same time, the various modules in the AP processor synthesize image 3, which is to be refreshed on the display screen during the TS3 cycle.

[0557] During the TS3 cycle, the display subsystem sends image 3 to the display screen, which displays image 3 synthesized by the various modules in the AP processor during the TS2 cycle. At the same time, the various modules in the AP processor synthesize image 4, which is to be refreshed on the display screen during the TS4 cycle.

[0558] The display enters an idle state at the start of the TS4 cycle.

[0559] During the TS4 cycle, the display subsystem sends image 4 to the display screen, which displays image 4 synthesized by the various modules in the AP processor during the TS3 cycle. The AP processor then stops synthesizing images to be refreshed.

[0560] During the TS5 cycle, the display subsystem sends image 4 to the display screen, which continues to display image 4, and the AP processor no longer synthesizes images to be refreshed.

[0561] During the TS6 cycle, the display subsystem sends image 4 to the display screen, which continues to display image 5, and the AP processor no longer synthesizes images to be refreshed.

[0562] In the above process, TS0 to TS3 represent the refresh state of the display screen, and TS4 to TS6 represent the idle state. During and after TS4, the electronic device does not perform an image refresh operation, and the display screen enters the idle state. Even after entering the idle state, the display subsystem continues to send the last synthesized image (Image 4) from the AP processor to the display screen at a preset frequency (the display screen's refresh rate). The displayed image (Image 4) is the image from the last refresh before the display screen switched to the idle state. Although the display subsystem still sends the last synthesized image (Image 4) from the AP processor to the display screen at the preset frequency (the display screen's refresh rate), the AP processor no longer executes... Figure 7 Steps A1 to A2 in the technical architecture described above.

[0563] Of course, in practical applications, TS0 to TS4 cycles can be recorded as the refresh state of the display screen, and TS5 to TS6 cycles can be recorded as the idle state of the display screen.

[0564] When the CWB write-back function is enabled, if the display is refreshing, the HWC can extract the target image corresponding to the currently refreshed image, and similarly, it can also extract the corresponding image noise. If the display is idle, even if the CWB write-back function is enabled, the AP processor will not execute... Figure 7 The illustrated embodiment describes the process of image synthesis through the coordination of various modules in steps A1 to A3. Correspondingly, the AP processor no longer executes... Figure 7In steps A4 to A6 of the embodiment, the noise algorithm library will not receive the target image or obtain image noise during the period when the display screen is idle.

[0565] If the display is idle for an extended period, such as one minute, it doesn't need to refresh the image during that time. However, a start-stop scheme that forces an image refresh before each ambient light sensor integration could result in a forced refresh every 350ms during that minute. This equates to approximately 60,000ms / 350ms = 171.4 image refreshes per minute. Therefore, this undoubtedly increases power consumption when the display is idle for extended periods.

[0566] To better understand why a forced image refresh scheme during the integration phase might increase processor power consumption when the display is idle for extended periods, we will... Figure 23 The example shown illustrates this.

[0567] See Figure 23 During the integration time period t in the first acquisition cycle 01 The image is refreshed once every moment, and the corresponding noise algorithm library stores t. 01 Image of time and t 01 Image noise at any given moment.

[0568] See Figure 23 , in t 01 After time t, the display shows the integral time interval t during the (M+1)th acquisition cycle. (2M)1 The image is refreshed again at any time; this example ignores brightness adjustment.

[0569] See Figure 23 t before the integration time period of the second acquisition cycle 1n After the CWB write-back function is activated, the image is forcibly refreshed once (for ease of description, assuming the forced image refresh and the activation of the CWB write-back function are within the same time unit, for example, both within 1ms). The AP processor executes steps A4 to A6 once, and the noise algorithm library obtains t. 1n Target image at time t and t 1n Image noise at any given moment.

[0570] See Figure 24 The integral noise of the integration time period in the second acquisition cycle is: t, with a duration equal to the integration time. 1n Image noise at any given moment.

[0571] See Figure 23 t before the integration time period of the third acquisition cycle 3nAfter the CWB write-back function is activated, the image is forcibly refreshed once. The AP processor executes steps A4 to A6 once, and the noise algorithm library obtains t. 3n Target image at time t and t 3n Image noise at any given moment.

[0572] See Figure 24 The integral noise of the integration time period in the third acquisition cycle is: t, with a duration equal to the integration time. 3n Image noise at any given moment.

[0573] ...

[0574] See Figure 23 t before the integration time period of the (M+1)th acquisition cycle (2M-1)n After the CWB write-back function is activated, the image is forcibly refreshed once. The AP processor executes steps A4 to A6 once, and the noise algorithm library obtains t. (2M-1)n Target image at time t and t (2M-1)n Image noise at any given moment.

[0575] See Figure 23 During the integration time period t in the (M+1)th acquisition cycle (2M)1 At each image refresh, the AP processor executes steps A1 to A6 once, and the noise algorithm library obtains t. (2M)1 Target image at time t and t (2M)1 Image noise at any given moment.

[0576] See Figure 24 The integral noise of the integral time period in the (M+1)th acquisition cycle is: with a duration of t. 2M To t (2M)1 time t (2M-1)n Image noise at time t and duration t (2M)1 To t 2M+1 time t (2M)1 Image noise at any given moment.

[0577] If according to Figure 19 The start / stop scheme of the embodiment shown Figure 23 and Figure 24 In the illustrated embodiment, from time t0 to t 2M At any given time (M acquisition cycles), the image is forcibly refreshed M times.

[0578] If the CWB write-back function is enabled (t) 1n t 1n ...t (2M-1)n After that, do not force a refresh of the image. See also... Figure 25 The integral noise of the second acquisition cycle is: t, the duration of continuous integration. 01The image noise at time t, the integral noise in the third acquisition cycle is: the duration of continuous integration t 01 The image noise at time t, ..., the integral noise in the (M+1)th acquisition cycle is: [details about duration t]. 2M To t (2M)1 time t 01 Image noise at time t and duration t (2M)1 To t 2M+1 time t (2M)1 Image noise at any given moment.

[0579] As mentioned earlier, the process of forcibly refreshing the image does not change the image displayed on the screen, i.e., t 01 The image refreshed at any time is the same as the image at the forced refresh time; correspondingly, t 01 The target image at time t is the same as the target image at the forced refresh time. If brightness adjustment is ignored, then t 01 The image noise at the current time is the same as the image noise at the time of forced refresh. If brightness adjustment is present, the target image used at the brightness adjustment time remains unchanged. Therefore, in some scenarios, it is unnecessary to force a refresh of the image.

[0580] The above analysis shows that when the display screen is idle for a long time, even if the image is not forcibly refreshed, it may not lose image noise that may interfere with the integration time period.

[0581] Of course, in the above embodiments, if t 01 If the time interval falls within the non-integral time period of the first acquisition cycle, the noise algorithm library may not be able to obtain t. 01 The target image and image noise at time t. This will require... 1n The image is forced to refresh constantly.

[0582] Based on the various embodiments described above, the embodiments of this application provide Figure 26 The technical solution shown. Figure 26 The illustrated embodiment includes the following steps:

[0583] Step 2601: The HWC in the AP processor starts the CWB write-back function at the first preset time before integration begins, and checks the moment when the image is refreshed on the display screen.

[0584] In this embodiment, regardless of whether the display screen needs to be forcibly refreshed, the CWB write-back function needs to be started at the first preset time before the integration begins, and then other factors are considered to determine whether a forced image refresh is required.

[0585] For ease of description, refer to Figure 27As shown, the time corresponding to the first preset time before the start of integration within a collection period (T2) is selected as the reference time, and this reference time is t. 3n This application embodiment requires the time corresponding to the first preset time before the integration begins (t) 3n (At any given moment) Initiate the CWB write-back function and view the moment the image was last refreshed on the display.

[0586] For ease of description, the time when the image is refreshed on the display screen can be denoted as t. k .

[0587] As an example, if the last time the image was refreshed is not within the current non-integration time period, then there is no need to force a refresh. The electronic device waits for the upper-layer application to transmit the display parameters of the interface to the display engine service, and then passes through the display engine service, Surface Flinger, HWC, etc. For HWC, HWC waits for the display parameters sent by Surface Flinger (this parameter can be referred to as the second display parameter).

[0588] The HWC module receives the display parameters sent by the Surface Flinger module of the electronic device and stores the time when the display parameters are received, which is the sixth display parameter.

[0589] The HWC module obtains the time when the electronic device last refreshed its image, which can be the time when it last received display parameters from the Surface Flinger module. The sixth display parameter can be set to the last display parameter obtained by the HWC module before obtaining the time when the electronic device last refreshed its image. Accordingly, the time when the image was last refreshed is the time when the HWC module received the sixth display parameter.

[0590] Step 2602: If the moment when the image is refreshed on the display screen is within the current non-integral time period, then wait for the second time (which can be recorded as the second duration).

[0591] In this embodiment, if the time difference between the last image refresh and the current time is greater than a difference threshold, it can be understood that the display screen has entered an idle state; if the time difference between the last image refresh and the current time is less than or equal to the difference threshold, it can be understood that the display screen has not yet entered a refresh state. The difference threshold can be determined based on empirical values.

[0592] The key point of this application's embodiments is to obtain the time of the most recent image refresh (when HWC performed image matting, the start time of HWC's image matting from the most recent image refresh can be obtained), so as to determine whether a forced image refresh is needed based on the time of the most recent image refresh (or the start time of HWC's image matting from the most recent image refresh).

[0593] The focus of this application's embodiments is not on confirming the current state of the display screen. The current state of the display screen is used to facilitate understanding of the reason for the increased power consumption of the display screen in the idle state in the above embodiments.

[0594] Reference Figure 27 In the embodiment shown, the time t is the moment when the image is refreshed on the display screen. k During the current non-integral time period (t3 to t) 3n If the display screen is between t, then the display screen is at t. 3n The image displayed at time t k Images that are constantly updated. k The time is between time t3 and t 3n The time interval between these moments is the period during which the CWB write-back function stops. That is, HWC has not acquired t. k The target image at time t, and correspondingly, the noise algorithm library did not obtain t. k Image noise at any given moment. If the image is not forcibly refreshed at this time, the following may occur:

[0595] (1) At t 3n The first change in the content displayed on the screen after a certain time (image refresh or brightness adjustment) is t. b Brightness adjustment at any time.

[0596] t b Time t 3n The time interval between time t4 and time t5, and t b Assuming there is no image refresh or brightness adjustment between time t4 and time t5. b The backlight noise at time t4 interferes with the initial ambient light during the integration time period from time t4 to time t5. In calculating t... b When dealing with backlight noise at time t, the latest target image cached by HWC is not t. k The target image at time t, but t k If a target image is obtained before time t, then t is calculated. b The backlight noise at any given time is incorrect. This causes the target ambient light calculated by the noise algorithm library for the integration time period from time t4 to t5 to be inaccurate.

[0597] t b At time t4, t bThe backlight noise at time t4 interferes with the initial ambient light during the integration time period from time t4 to time t5. In calculating t... b When dealing with backlight noise at time t, the latest target image cached by HWC is not t. k The target image at time t is calculated. b Errors in backlight noise at certain times cause inaccurate calculations of the target ambient light during the integral time period from time t4 to time t5 by the noise algorithm library.

[0598] t b Given that the time interval is between time t4 and time t5, the time interval from time t4 to time t5 is... b The initial ambient light between time points is affected by t k Image noise interference at time t. However, HWC did not obtain t. k Image noise at time t will be used during the integration process. k The fusion noise prior to time t4 (which could be backlight noise or image noise) is used as the basis for the fusion noise from time t4 to time t5. b The initial ambient light between time points introduces interference and fusion noise, causing inaccurate calculations of the target ambient light during the integration time interval from time t4 to t5 by the noise algorithm library. Additionally, t b The backlight noise at time t4 interferes with the initial ambient light during the integration time period from time t4 to time t5. In calculating t... b When dealing with backlight noise at time t, the latest target image cached by HWC is not t. k The target image at time t4 leads to errors in the calculated backlight noise. The ambient light of the target during the integration time period from time t4 to time t5 calculated by the noise algorithm library is inaccurate.

[0599] (2) At t 3n The last change (image refresh or brightness adjustment) to the content displayed on the screen after time t is... b Image refresh time. t b Given that the time interval is between time t4 and time t5, the time interval from time t4 to time t5 is... b The initial ambient light between time points is affected by t k The image noise at time t is an interference. However, the noise algorithm library does not calculate t. k Image noise at time t4. The noise algorithm library calculates the target ambient light from time t4 to t5 using the noise memory stored in the t5 timeframe. k The fusion noise before time step t is used as the fusion noise of the first sub-time step of the interference integration time step, which causes the target ambient light calculated by the noise algorithm library for the integration time step from time step t4 to time step t5 to be inaccurate.

[0600] The above analysis lists some cases. However, it can be understood from the above analysis that if the last time the display refreshed the image is within the current non-integral time period, then a forced image refresh is required to obtain the target image corresponding to the currently displayed image and the image noise corresponding to that target image. Of course, the target image corresponding to the currently displayed image and the image noise corresponding to that target image can be understood as t 3n The target image and image noise at any given time.

[0601] In another embodiment of this application, if the moment when the image is refreshed on the display screen is not within the current non-integration time period, then the image will no longer be forced to be refreshed.

[0602] In this embodiment of the application, if the time t is the time when the image is refreshed on the display screen... k Not at time t3 and t 3n The time interval could be the integration period within the current collection cycle, or within the previous or earlier collection cycle.

[0603] If t k During the integration period of this acquisition cycle, since the CWB write-back function is activated during the integration period of this acquisition cycle, HWC can obtain t. k The target image at time t can also be obtained from the noise algorithm library. k The target image and image noise are minimized at any given moment, therefore, there is no need to force an image refresh.

[0604] If t k In the previous acquisition cycle or an earlier acquisition cycle, since it was already based on the data acquired in the previous acquisition cycle... Figure 26 If the illustrated embodiment has been executed, then there is no need to consider whether to force a refresh of the image. This application will be discussed later. Figures 28 to 30 Verify this situation (t) k In the previous acquisition cycle or an earlier acquisition cycle, is it unnecessary to consider forcibly refreshing the image? See details in [link to relevant documentation]. Figures 28 to 30 The description.

[0605] In this embodiment of the application, the time t at which the image is refreshed on the display screen is determined. k The method for determining whether it falls within the non-integral time period of the current cycle can be referred to Figure 27 As shown in the diagram.

[0606] Method 1: Determine T22(t) 3n -t k ) and T21(t 3n The size of -t3). If T22(t 3n -t k ) less than T21(t 3nIf -t3), it indicates that the moment the image was refreshed on the display screen occurred within the current non-integral time period. Otherwise, it indicates that the moment the image was refreshed on the display screen did not occur within the current non-integral time period. In this embodiment, T22 can be recorded as the first difference, and T21 can be recorded as the second difference.

[0607] Method 2: Determine t k And the size of t3. If t k Greater than t3 and less than t 3n This indicates that the last time the image was refreshed on the display screen was within the current non-integral time period. Otherwise, it indicates that the last time the image was refreshed on the display screen was not within the current non-integral time period. In this embodiment, if an image refresh occurs at time t3, the CWB write-back function will stop after time t3. Therefore, if an image refresh occurs at time t3, HWC can obtain the target image at time t3, and the noise algorithm library can also obtain the target image noise at time t3. Therefore, T22(t 3n -t k ) equals T21(t 3n In the case of -t3), the display switches to idle mode within the current integration time period. Similarly, t k When the time is equal to t3, the display screen switches to idle state within the current integration time period.

[0608] Method 3: Check if the times of the most recent image refresh and the most recent image matting are less than a certain threshold (because the image refresh time and the start time of HWC acquiring the target image may differ). If they are less than the threshold, it means that the most recently refreshed image has undergone image matting by HWC, and therefore the most recently refreshed image is not in the current non-integration time period. If they are greater than or equal to the threshold, it means that the most recently refreshed image has not undergone image matting by HWC, and therefore is in the current non-integration time period. This threshold is set according to the actual situation. In this embodiment, this threshold can be denoted as the first threshold.

[0609] In this embodiment, the moment when the HWC obtains the display parameters of the interface from the surface filter can be used as the moment of image refresh. Alternatively, the moment when the HWC obtains the composited image through the underlying hardware can be used as the moment of image refresh. The moment when the display subsystem sends the image to the display can also be used as the image refresh moment. Regardless of which moment is used, there may be a slight difference between the moment of image refresh and the moment when the HWC begins to extract the image from the refreshed image, for example, 0.5ms, 0.8ms, 1ms, etc. Of course, the moment of image refresh and the moment when the HWC begins to extract the image from the refreshed image may also be equal. The moment of the next refreshed image is typically one refresh cycle away from the moment of the current refresh. For example, at a refresh rate of 60Hz, this refresh cycle is 1000ms / 60 = 16.7ms. At a refresh rate of 120Hz, this refresh cycle is 1000ms / 120 = 8.3ms. Therefore, the threshold in this example can be a value relatively small compared to the refresh cycle, such as 2ms.

[0610] Step 2603: If the electronic device refreshes the image during the second preset time period, then the image refresh will no longer be forced.

[0611] When an electronic device refreshes its image, it means that the HWC can receive the display parameters sent by the Surface Flinger (this display parameter is denoted as the fourth display parameter). The currently refreshed image can be denoted as the fifth image.

[0612] Step 2603': If the electronic device does not refresh the image during the second preset time period, then the image is forcibly refreshed.

[0613] In this embodiment of the application, if the electronic device is constantly refreshing the image, the image refresh action has already occurred during the second preset time period (e.g., 17ms), and the HWC has already obtained the latest target image. Therefore, delaying the second preset time before deciding whether to perform a forced image refresh can avoid adding an extra forced image refresh action and further reduce power consumption.

[0614] If the HWC module does not receive the display parameters (which can be referred to as the third display parameters) sent by Surface Flinger during the second preset time period, it needs to force a refresh of the image.

[0615] The following three examples (all assuming HWC performs image cutout each time the display refreshes the image) will verify the effect of the display being idle for a long time (in the above examples, if t k(In the case of the previous acquisition cycle or an earlier acquisition cycle) whether it is possible to obtain all image noise and backlight noise of the initial ambient light acquired during the interference integration time period.

[0616] See Figure 28 Taking an example where the image on the display screen was refreshed within the integral time period of the previous acquisition cycle, and the image has not been refreshed again since, the previous acquisition cycle is the first acquisition cycle, and the current acquisition cycle is the second acquisition cycle. In this example, t 1n Time, t 2n Time, t 3n The CWB write-back function is activated at all times.

[0617] In the first acquisition cycle, at time t0, the ambient light sensor begins acquiring the initial ambient light; at time t0, the CWB write-back function has already been activated. At time t... k The display screen last refreshed the image at the specified time, and the noise algorithm library obtained t. k Image noise at any given moment.

[0618] In t 1n The moment when the image was last refreshed on the display screen does not fall within the current non-integral time period. (Display screen t) 1n The image displayed at time t k The image displayed at any given time, assuming backlight adjustment is ignored, then t 1n The image noise at time t is k Image noise at time t. The noise algorithm library has already obtained t. k Image noise at time t, therefore, t 1n The image will no longer be forced to refresh.

[0619] In the second acquisition cycle, there is no image refresh, and brightness adjustment is ignored. After the integration in the second acquisition cycle, the fusion noise during the interference integration time period is t. k Image noise at any given moment.

[0620] In t 3n The moment the display last refreshed the image does not occur within the current non-integral time period. (Display t) 3n The image displayed at time t k The image displayed at any given time, assuming backlight adjustment is ignored, then t 3n The image noise at time t is k Image noise at time t. HWC has already obtained t. k Image noise at time t, therefore, t 3n The image will no longer be forced to refresh.

[0621] In the third acquisition cycle, there is no image refresh, and brightness adjustment is ignored. The processing procedure from the second acquisition cycle continues.

[0622] This example illustrates that if the last time the image was refreshed occurred within the integration time period of the previous acquisition cycle, and the image was not refreshed again, it is not necessary to force a refresh of the image to obtain the image noise that interferes with each integration time period.

[0623] See Figure 29 The time when the image is refreshed on the display screen is within the non-integral time period t of the previous acquisition cycle. 1n Taking a time period before the current acquisition time, without refreshing the image again, as an example. Here, the previous acquisition period is the first acquisition period, and the current acquisition period is the second acquisition period. In this example, t... 1n Time, t 2n Time, t 3n The CWB write-back function is activated at all times.

[0624] In the first acquisition cycle, at time t0, the ambient light sensor begins acquiring the initial ambient light; at time t0, the CWB write-back function has already been activated. At time t... k The display screen last refreshed the image at that time. At this point, the CWB write-back function was stopped, so the HWC did not receive the t value. k Image noise at any given moment.

[0625] In t 1n At any given moment, the time when the image is refreshed on the display screen is the non-integral time period, and a forced image refresh is required (this moment becomes the time t of the last image refresh). k '), obtain t 1n Image noise at any given moment.

[0626] Of course, in practical applications, even if a decision is made to force a refresh of the image, a certain amount of time can be waited. If the display refreshes the image at the specified refresh rate during this period, a forced refresh is unnecessary. If no refresh is detected after a certain period, a forced refresh can be performed. This example uses a forced refresh of the image as an example.

[0627] In the second acquisition cycle, there is no image refresh and brightness adjustment is ignored. After the integration of the second acquisition cycle, the fusion noise that interferes with the integration time period of the second acquisition cycle is t. 1n Image noise at any given moment.

[0628] In t 3n The time when the display last refreshed the image is not within the current non-integral time period. (Display t) 3n The image displayed at time t 1n The image displayed at any given time, assuming backlight adjustment is ignored, then t 3n The image noise at time t is 1nImage noise at time t. The noise algorithm library has already obtained t. 1n Noise at time t, therefore, t 3n The image will no longer be forced to refresh.

[0629] In the third acquisition cycle, there is no image refresh, and brightness adjustment is ignored. The processing procedure from the second acquisition cycle continues.

[0630] This example illustrates that if the moment the image is refreshed on the display screen is before the CWB write-back function is activated during the non-integration period of the previous acquisition cycle, and the image is not refreshed again, it is possible to obtain the image noise that interferes with the initial ambient light of each integration period without forcibly refreshing the image.

[0631] See Figure 30 The time when the image is refreshed on the display screen is within the non-integral time period t of the previous acquisition cycle. 1n Taking the example of a period after time t1 and before time t2, where the image has not been refreshed again. Here, the previous acquisition period is the first acquisition period, and the current acquisition period is the second acquisition period. In this example, t... 1n Time, t 2n Time, t 3n The CWB write-back function is activated at all times.

[0632] The first acquisition cycle, in t 1n At that moment, the CWB write-back function is activated.

[0633] In t k At any given moment, when the display refreshes the image, the noise algorithm library can obtain t. k Image noise in the displayed image at any given time.

[0634] In the second acquisition cycle, there is no image refresh, and brightness adjustment is ignored. After the integration in the second acquisition cycle, the fusion noise during the interference integration time period is t. k Image noise at any given time.

[0635] In t 3n The moment the display last refreshed the image does not occur within the current non-integral time period. (Display t) 3n The image displayed at time t k The image displayed at any given time, assuming backlight adjustment is ignored, then t 3n The image noise at time t is k Image noise at time t. The noise algorithm library has already obtained t. k Noise at time t, therefore, t 3n The image will no longer be forced to refresh.

[0636] In the third acquisition cycle, there is no image refresh, and brightness adjustment is ignored. The processing procedure from the second acquisition cycle continues.

[0637] This example illustrates that if the image on the display screen is refreshed after the CWB write-back function is activated during the non-integration period of the previous acquisition cycle, and the image is not refreshed again, it is possible to obtain image noise that interferes with the initial ambient light of each integration period without forcibly refreshing the image.

[0638] pass Figures 28 to 30 The example can be understood as follows: if the moment when the image is refreshed on the display screen is not within the current non-integral time period, then there is no need to force a refresh of the image.

[0639] The embodiments of this application adopt Figure 28 The flowchart shown can reduce processor power consumption, avoid not being able to obtain image noise that interferes with the integration time period, avoid not being able to obtain the target image when backlight noise that interferes with the integration time period is not available, and avoid negative gain when the display screen is idle for a long time.

[0640] As mentioned earlier, the HWC in the AP processor can monitor changes in data in the kernel node during both the integration and non-integration periods. When the data stored in the kernel node changes, the HWC obtains the brightness to be adjusted from the kernel node and transmits it to the noise algorithm library to calculate the backlight noise.

[0641] In practical applications, it can also be done in the following way.

[0642] When the CWB write-back function is stopped, HWC can obtain the brightness to be adjusted when it detects changes in the data in the kernel node. However, HWC no longer transmits the brightness value to be adjusted to the noise algorithm library, and correspondingly, the noise algorithm library no longer calculates and obtains the corresponding backlight noise.

[0643] If, during the CWB write-back function's pause (e.g., 1ms before CWB write-back function starts) or during its startup, the HWC does not detect any changes in the data stored in the kernel node during the CWB write-back function's shutdown period, it indicates that the display screen's brightness value has not changed. The HWC executes the CWB write-back function startup / shutdown method provided in any of the above embodiments.

[0644] When the CWB write-back function is about to start or is started, if the HWC detects a change in the data stored in the kernel node during the CWB write-back function's shutdown period, it indicates that the display brightness value has changed. The HWC then needs to send the latest detected brightness value to the noise algorithm library. Then, the HWC executes the CWB write-back function start / stop method provided in any of the above embodiments. During the CWB write-back function's shutdown period, if there are multiple brightness adjustments, the noise algorithm library only needs to know the value after the last brightness adjustment. That is, the HWC sends the brightness to be adjusted corresponding to the most recent brightness change detected by the display to the noise algorithm library. This avoids the noise algorithm library frequently calculating the backlight noise corresponding to the brightness to be adjusted, thereby reducing power consumption.

[0645] As an example, after the HWC module sets the cutout marker to the first character, the HWC module monitors whether the data in the kernel node of the electronic device changes, and the kernel node stores the brightness value.

[0646] In response to a change in the data in the kernel node of the electronic device, the HWC module obtains a first brightness value from the kernel node;

[0647] After the HWC module obtains the first brightness value from the kernel node, in response to a change in the data in the kernel node of the electronic device, the HWC module obtains the second brightness value from the kernel node.

[0648] In response to the arrival of the first moment, the HWC module sends the second brightness value to the noise algorithm library.

[0649] When calculating the image noise corresponding to the forced refresh image, the noise algorithm library calculates the first image noise based on the target image corresponding to the forced refresh image and the second brightness value.

[0650] After the HWC module sets the cutout marker to the second character, the HWC module monitors whether the data in the kernel node of the electronic device changes, and the kernel node stores the brightness value;

[0651] In response to a change in the data in the kernel node of the electronic device, the HWC module obtains a third brightness value from the kernel node;

[0652] The HWC module sends the second brightness value to the noise algorithm library;

[0653] After the HWC module sends the third brightness value to the noise algorithm library, in response to a change in the data in the kernel node of the electronic device, the HWC module obtains a fourth brightness value from the kernel node.

[0654] The HWC module sends the fourth brightness value to the noise algorithm library.

[0655] In the above embodiments, HWC may or may not force an image refresh.

[0656] If HWC forces an image refresh, the image noise corresponding to the refreshed image is calculated using the brightness value from the latest input noise algorithm library and the target image corresponding to the refreshed image. The backlight noise at the brightness adjustment time will not interfere with the initial ambient light acquired in the next integration time interval. The fusion noise that interferes with the initial ambient light acquired in the next integration time interval may be the image noise corresponding to the refreshed image. In this case, the image noise value is correct and will not cause errors in the target ambient light in the next integration time interval.

[0657] If HWC no longer forces an image refresh, it means that the noise algorithm library stores the target image of the image currently displayed on the screen (the latest frame target image stored in the noise algorithm library).

[0658] If the brightness adjustment occurs earlier than the refresh time of the currently displayed image, the backlight noise corresponding to the brightness adjustment will not interfere with the initial ambient light collected in the next integration time interval. The fusion noise that interferes with the initial ambient light collected in the next integration time interval may be image noise corresponding to the currently displayed image. However, if this image noise value is correct, it will not cause an error in the target ambient light for the next integration time interval.

[0659] If the brightness adjustment time is later than the refresh time of the currently displayed image, the image noise corresponding to the refresh time of the currently displayed image will not interfere with the initial ambient light collected in the next integration time period. The fusion noise that interferes with the initial ambient light collected in the next integration time period may be the backlight noise at the latest brightness adjustment time. This backlight noise is generated by the latest target image already acquired by the display and the adjusted brightness value at the latest time. Therefore, this backlight noise is correct and will not cause the target ambient light in the next integration time period to be incorrect.

[0660] Therefore, regardless of when the CWB write-back function is started, and regardless of whether the image is forcibly refreshed, if the HWC detects a brightness change during the CWB write-back function's shutdown period, then when the CWB write-back function is about to be started, the HWC sends the latest detected brightness value to the noise algorithm library. The noise algorithm library calculates the backlight noise. The HWC continues to execute the start / stop method of the CWB write-back function provided in any of the above embodiments.

[0661] As another embodiment of this application, during the CWB write-back function startup, HWC can retrieve the target image from the CWB write-back memory once every frame.

[0662] As an example, when the display refreshes at a frequency of 90Hz, it is equivalent to refreshing the image once every 1000ms / 90 = 11.11ms. Specifically, HWC retrieves the target image from CWB memory every frame as follows:

[0663] When the electronic device refreshes the image for the i-th time (taking the image cutout of the i-th refresh as an example), after the HWC obtains the composite image, it checks the cutout mark. The HWC finds that the cutout mark is the second character, and the HWC determines that the image refreshed this time is a cutout frame. The HWC then continues to perform the subsequent steps according to the above embodiment.

[0664] When the image is refreshed for the (i+1)th time, HWC obtains the composite image and checks the matting mark. HWC checks if the matting mark is the second character (during the CWB write-back function is activated, the matting mark is the second character). HWC obtains the time difference between the time when HWC last determined the matting frame (the time when the image was determined to be the matting frame during the i-th refresh) and the current time. If the time difference is less than the matting frame difference threshold (which can be 11.11ms, or other time values, such as 11.5ms, 11.8ms, 12ms, etc.), it means that the image refreshed in the previous refresh of the image for the (i+1)-th refresh is already the matting frame, and the image refreshed in the (i+1)-th refresh will no longer be matted.

[0665] When the image is refreshed for the (i+2)th time, HWC obtains the composite image and checks the matting mark. HWC checks if the matting mark is the second character (during the CWB write-back function is activated, the matting mark is the second character). HWC obtains the time difference between the last time the matting frame was determined (the time when the image was determined to be a matting frame during the i-th refresh) and the current time. If the time difference is greater than or equal to the matting frame difference threshold (which can be 11.11ms, or other time values, such as 11.5ms, 11.8ms, 12ms, etc.), then it means that the image refreshed for the (i+2)th time is a matting frame.

[0666] In the example above, the time difference is the time difference between the moment the HWC last determined the matting frame and the current moment. In practice, it could also be the time difference between the moment the HWC last transmitted the composite image (carrying the information to be matted) to the OLED driver and the current moment, or it could be the time difference between the moment the HWC last started matting and the current moment. The methods for obtaining the time difference described above are merely illustrative; in actual applications, other methods can be used to determine the time difference. Different methods of obtaining the time difference will result in different threshold values ​​for the matting frame difference.

[0667] As an example, the theoretical interval between two image refreshes is 11.1ms, and the current time is the moment when HWC obtains the cutout marker as the second character. If the time difference is the time difference between the last time the cutout frame was determined and the current time, then theoretically this time difference is 11.1ms (the previous frame was a cutout frame) or 22.2ms (the previous frame was not a cutout frame). Therefore, the cutout frame difference threshold can be any value between 11.1 and 22.2. If the time difference is the moment when HWC last transmitted the composite image (carrying the information to be cut out) to the OLED driver, then theoretically this time difference is (11.1-t)ms (the previous frame was a cutout frame) or (22.2-t)ms (the previous frame was not a cutout frame), where t is the time difference between the moment HWC determined the cutout frame and the moment the composite image was transmitted to the OLED driver. Therefore, the cutout frame difference threshold can be any value between 11.1-t and 22.2-t.

[0668] If the display refreshes at 120Hz, the 11.1ms in the example above needs to be changed to 16.7ms based on the 120Hz frequency. Therefore, when performing frame interpolation, the frame difference threshold is also related to the display's current refresh rate.

[0669] Of course, the above examples are for illustrative purposes only and do not impose any limitations on this application.

[0670] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0671] This application embodiment can divide an electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. The following description uses the example of dividing each functional unit according to each function as an example:

[0672] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0673] This application also provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0674] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to the first device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0675] This application also provides a chip system, which includes a processor coupled to a memory. The processor executes a computer program stored in the memory to implement the steps of any method embodiment of this application. The chip system can be a single chip or a chip module composed of multiple chips.

[0676] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0677] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0678] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for monitoring noise, characterized in that, Applied to an electronic device, the electronic device including: an HWC module, a display subsystem, and a noise algorithm library, the method includes: In response to receiving the first information, the HWC module sets the write-back flag to the first flag. When the write-back flag is the first flag, the display subsystem stops storing images in the write-back memory of the electronic device. In response to receiving the first image, the HWC module sends the first image to the display subsystem; The display subsystem stops storing the second image to the write-back memory of the electronic device. The second image is the image corresponding to the area containing the first target image on the first image. The first target image is the image within the first area, which is the area on the display screen of the electronic device located above the ambient light sensor of the electronic device. In response to the arrival of the first moment, the HWC module sets the write-back flag to the second flag. When the write-back flag is the second flag, the display subsystem begins to store the image in the write-back memory of the electronic device. The HWC module acquires the third image; The HWC module sends the third image to the display subsystem; In response to receiving the third image, the display subsystem stores a fourth image in the write-back memory of the electronic device. The fourth image is an image corresponding to the region of the third image that contains the second target image, and the second target image is an image within the first region. The HWC module obtains the second target image from the write-back memory; The HWC module sends the second target image to the noise algorithm library; The noise algorithm library calculates the noise of the first image based on the second target image.

2. The method as described in claim 1, characterized in that, The first information includes a first duration, which is the duration during which the display subsystem stops storing images to the write-back memory; the first moment is the moment after the first duration has elapsed since the write-back flag was set to the first flag. Alternatively, the first information includes a first duration, a first value, and a second time, wherein the first duration is the duration during which the display subsystem stops storing images to the write-back memory, and the second time is the end time when the ambient light sensor of the electronic device collects the first value; the first time is the time after the second duration has elapsed since the write-back flag was set to the first flag, and the second duration is the first duration minus the delay duration, wherein the delay duration is the time when the HWC module receives the first information minus the second time duration.

3. The method as described in claim 1 or 2, characterized in that, The HWC module acquires the third image including: The HWC module sends a first signal to the surface Flinger of the electronic device; In response to receiving the first signal, the surface Flinger obtains the cached first display parameter and sends the first display parameter to the HWC module. The first display parameter is the latest cached display parameter among the display parameters cached by the surface Flinger. The HWC module obtains the third image based on the first display parameters.

4. The method as described in claim 1, characterized in that, After the HWC module sets the write-back flag to the second flag, and before the HWC module acquires the third image, the following steps are also included: The HWC module obtains the time when the electronic device last refreshed the image; If the time when the electronic device last refreshed the image meets the first preset condition, then the HWC module acquires the third image.

5. The method as described in claim 4, characterized in that, After the HWC module obtains the time of the last image refresh of the electronic device, it also includes: If the last time the image was refreshed by the electronic device does not meet the first preset condition, the HWC module waits for the Surface Flinger module of the electronic device to send the second display parameters.

6. The method as described in claim 4, characterized in that, If the time of the last image refresh by the electronic device meets a first preset condition, then the HWC module acquires the first image, including: If the time of the last image refresh by the electronic device meets the first preset condition, the HWC module waits for a second duration. If the HWC module does not receive the third display parameter from Surface Flinger within the second time period, the HWC module acquires the first image.

7. The method as described in claim 6, characterized in that, The method further includes: If the HWC module receives the fourth display parameter sent by Surface Flinger within the second time period, the HWC module obtains the fifth image based on the fourth display parameter. The HWC module queries the write-back flag and finds it to be the second flag; The HWC module sends the fifth image and the third information to the display subsystem based on the second marker; In response to receiving the fifth image and the third information, the display subsystem stores a sixth image containing a third target image on the fifth image in the write-back memory of the electronic device, wherein the third target image is an image within the first region; The HWC module retrieves the third target image from the write-back memory; The HWC module sends the third target image to the noise algorithm library; The noise algorithm library calculates the noise of the second image based on the third target image.

8. The method according to any one of claims 4 to 7, characterized in that, The first information includes a first value and a second time, wherein the second time is the end time when the ambient light sensor of the electronic device collects the first value; The time when the electronic device last refreshed the image satisfies the first preset condition, including: The last time the electronic device refreshed the image was later than the second time; The electronic device's last image refresh time does not meet the first preset condition, including: The time when the electronic device last refreshed the image is earlier than or equal to the second time.

9. The method according to any one of claims 4 to 7, characterized in that, The first information also includes a first value and a second time, wherein the second time is the end time when the ambient light sensor of the electronic device collects the first value, and the time when the electronic device last refreshed the image satisfies the first preset condition including: The first difference between the time when the electronic device last refreshed the image and the current time is less than the second difference between the second time and the current time; The electronic device's last image refresh time does not meet the first preset condition, including: The first difference between the time when the electronic device last refreshed the image and the current time is greater than or equal to the second difference between the second time and the current time.

10. The method according to any one of claims 4 to 7, characterized in that, The time when the electronic device last refreshed the image satisfies the first preset condition, including: The time when the electronic device last refreshed the image and the time when the HWC module last acquired the target image are both less than a first threshold; the target image is an image displayed on the screen located in the area above the ambient light sensor of the electronic device. The electronic device's last image refresh time does not meet the first preset condition, including: The time when the electronic device last refreshed the image and the time when the HWC module last acquired the target image are both greater than or equal to the first threshold.

11. The method as described in claim 1, characterized in that, The method further includes: After the HWC module sets the write-back flag to the first flag, the HWC module monitors whether the data in the kernel node of the electronic device has changed, and the kernel node stores the brightness value. In response to the detection of a change in data in the kernel node of the electronic device, the HWC module obtains a first brightness value from the kernel node; After the HWC module obtains the first brightness value from the kernel node, in response to the detection of a change in the data in the kernel node of the electronic device, the HWC module obtains the second brightness value from the kernel node. In response to the arrival of the first moment, the HWC module sends the second brightness value to the noise algorithm library.

12. The method as described in claim 11, characterized in that, The method further includes: After the HWC module sets the write-back flag to the second flag, the HWC module monitors whether the data in the kernel node of the electronic device has changed, and the kernel node stores the brightness value; In response to the detection of a change in data in the kernel node of the electronic device, the HWC module obtains a third brightness value from the kernel node; The HWC module sends the third brightness value to the noise algorithm library; After the HWC module sends the third brightness value to the noise algorithm library, in response to the detection of a change in the data in the kernel node of the electronic device, the HWC module obtains a fourth brightness value from the kernel node; The HWC module sends the fourth brightness value to the noise algorithm library.

13. The method as described in claim 11 or 12, characterized in that, The noise algorithm library calculates the noise in the first image based on the second target image, including: The noise algorithm library calculates the first image noise based on the second target image and the second brightness value.

14. The method as described in claim 1, characterized in that, The HWC module receives the first image including: The HWC module receives the fifth display parameter sent by the Surface Flinger module of the electronic device; The HWC module obtains the first image based on the fifth display parameter.

15. The method as described in claim 4, characterized in that, Before the HWC module obtains the time of the last image refresh of the electronic device, it includes: The HWC module receives the sixth display parameter sent by the Surface Flinger module of the electronic device; The HWC module stores the moment when the HWC module receives the sixth display parameter; The HWC module obtains the time of the last image refresh of the electronic device, including: The HWC module acquires the stored time of receiving the sixth display parameter, wherein the time of receiving the sixth display parameter is the latest time of receiving the display parameter stored by the HWC module before the time of the last image refresh of the electronic device.

16. The method as described in claim 3, characterized in that, The first display parameters include one or more of the following: the position, size, color, and storage address of the interface that synthesizes the third image on the display screen of the electronic device.

17. A method for monitoring noise, characterized in that, Applied to an electronic device, the electronic device including a first processor, the method includes: The first processor receives first information, which instructs the first processor to stop acquiring the target image from the refreshed image; After the first processor receives the first information, in response to receiving the first image, the first processor stops acquiring the first target image from the first image. The first target image is an image within a first region, and the first region is the area on the display screen of the electronic device located above the ambient light sensor of the electronic device. After reaching the first moment, the first processor acquires the third image; The first processor acquires a second target image from the third image, wherein the second target image is an image within the first region; The first processor calculates the first image noise based on the second target image.

18. The method as described in claim 17, characterized in that, The method further includes: In response to receiving the first information, the first processor sets the write-back flag to the first flag via the HWC module of the electronic device; In response to receiving the first image, the first processor stopping the acquisition of the first target image from the first image includes: In response to receiving the first image, the first processor queries the write-back flag as the first flag through the HWC module; The first processor sends the first image to the display subsystem of the electronic device based on the first tag via the HWC module; The first processor stops storing a second image containing a first target image on the first image in the write-back memory of the electronic device through the display subsystem; the first target image is an image within a first region. The method further includes: In response to the arrival of the first moment, the first processor sets the write-back flag to the second flag via the HWC module; The first processor acquires a third image, and from the third image, the first processor acquires a second target image, the second target image being an image within the first region, including: The first processor acquires the third image through the HWC module; The first processor queries the write-back flag through the HWC module and finds it to be the second flag; The first processor sends the third image and second information to the display subsystem based on the second tag through the HWC module. The second information is used to instruct the display subsystem to store the fourth image containing the second target image on the third image in the write-back memory of the electronic device. In response to receiving the third image and the second information, the first processor stores a fourth image containing a second target image on the third image in the write-back memory of the electronic device through the display subsystem, wherein the second target image is an image within the first region; The first processor obtains the second target image from the write-back memory through the HWC module; The method further includes: The first processor sends the second target image to the noise algorithm library through the HWC module; The first processor calculates the first image noise based on the second target image using the noise algorithm library.

19. The method as described in claim 18, characterized in that, The first information includes a first duration, which is the duration during which the display subsystem stops storing images to the write-back memory; the first moment is the moment after the first duration has elapsed since the write-back flag was set to the first flag. Alternatively, the first information includes a first duration, a first value, and a second time, wherein the first duration is the duration during which the display subsystem stops storing images to the write-back memory, and the second time is the end time when the ambient light sensor of the electronic device collects the first value; the first time is the time after the second duration has elapsed since the write-back flag was set to the first flag, and the second duration is the first duration minus the delay duration, wherein the delay duration is the time when the HWC module receives the first information minus the second time duration.

20. The method as described in claim 18, characterized in that, The first processor acquires the first image through the HWC module, including: The first processor sends a first signal to the surface Flinger of the electronic device through the HWC module; In response to receiving the first signal, the surface Flinger obtains the cached first display parameter and sends the first display parameter to the HWC module. The first display parameter is the latest cached display parameter among the display parameters cached by the surface Flinger. The HWC module obtains the third image based on the first display parameters.

21. The method as described in claim 18, characterized in that, After the first processor sets the write-back flag to the second flag through the HWC module, and before the first processor acquires the third image through the HWC module, the method further includes: The first processor obtains the time of the last image refresh of the electronic device through the HWC module; If the time when the electronic device last refreshed the image meets the first preset condition, then the first processor obtains the third image through the HWC module.

22. The method as described in claim 21, characterized in that, After the first processor obtains the time of the last image refresh of the electronic device through the HWC module, it further includes: If the last time the image was refreshed by the electronic device does not meet the first preset condition, the first processor waits for the Surface Flinger module of the electronic device to send the second display parameters through the HWC module.

23. The method as described in claim 21, characterized in that, If the time of the last image refresh by the electronic device meets a first preset condition, then the first processor acquires the first image through the HWC module, including: If the time when the electronic device last refreshed the image meets the first preset condition, the first processor waits for a second duration through the HWC module; If the HWC module does not receive the third display parameter from Surface Flinger within the second time period, the first processor obtains the first image through the HWC module.

24. The method as described in claim 23, characterized in that, The method further includes: If the HWC module receives the fourth display parameter sent by Surface Flinger within the second time period, the first processor obtains the fifth image based on the fourth display parameter through the HWC module. The first processor queries the write-back flag through the HWC module and finds it to be the second flag; The first processor sends the fifth image and the third information to the display subsystem based on the second tag via the HWC module; In response to receiving the fifth image and the third information, the first processor stores a sixth image containing a third target image on the fifth image in the write-back memory of the electronic device through the display subsystem; the third target image is the image in the first region. The first processor obtains the third target image from the write-back memory through the HWC module; The first processor sends the third target image to the noise algorithm library of the electronic device through the HWC module; The first processor calculates the second image noise based on the third target image using the noise algorithm library.

25. The method according to any one of claims 21 to 24, characterized in that, The first information includes a first value and a second time, wherein the second time is the end time when the ambient light sensor of the electronic device collects the first value; The time when the electronic device last refreshed the image satisfies the first preset condition, including: The last time the electronic device refreshed the image was later than the second time; The electronic device's last image refresh time does not meet the first preset condition, including: The time when the electronic device last refreshed the image was earlier than or equal to the second time; Alternatively, the moment when the electronic device last refreshed the image satisfies the first preset condition includes: The first difference between the time when the electronic device last refreshed the image and the current time is less than the second difference between the second time and the current time; The electronic device's last image refresh time does not meet the first preset condition, including: The first difference between the time when the electronic device last refreshed the image and the current time is greater than or equal to the second difference between the second time and the current time; or, The time when the electronic device last refreshed the image satisfies the first preset condition, including: The time when the electronic device last refreshed the image and the time when the HWC module last acquired the target image are both less than a first threshold; the target image is an image displayed on the screen located in the area above the ambient light sensor of the electronic device. The electronic device's last image refresh time does not meet the first preset condition, including: The time when the electronic device last refreshed the image and the time when the HWC module last acquired the target image are both greater than or equal to the first threshold.

26. The method according to any one of claims 18 to 24, characterized in that, The method further includes: After the first processor sets the write-back flag to the first flag through the HWC module; the first processor monitors whether the data in the kernel node of the electronic device has changed through the HWC module, and the kernel node stores the brightness value; In response to the detection of a change in data in the kernel node of the electronic device, the first processor obtains a first brightness value from the kernel node through the HWC module; After the first processor obtains a first brightness value from the kernel node through the HWC module, in response to detecting a change in the data in the kernel node of the electronic device, the first processor obtains a second brightness value from the kernel node through the HWC module. In response to the arrival of the first moment, the first processor sends the second brightness value to the noise algorithm library through the HWC module.

27. The method as described in claim 26, characterized in that, The method further includes: After the first processor sets the write-back flag to the second flag through the HWC module, the first processor monitors whether the data in the kernel node of the electronic device has changed through the HWC module. The kernel node stores the brightness value. In response to the detection of a change in data in the kernel node of the electronic device, the first processor obtains a third brightness value from the kernel node through the HWC module; The first processor sends the third brightness value to the noise algorithm library through the HWC module; After the first processor sends the third brightness value to the noise algorithm library through the HWC module, in response to the detection of a change in the data in the kernel node of the electronic device, the first processor obtains a fourth brightness value from the kernel node through the HWC module. The first processor sends the fourth brightness value to the noise algorithm library through the HWC module.

28. The method as described in claim 26, characterized in that, The first processor calculates the first image noise based on the second target image using the noise algorithm library, including: The first processor calculates the first image noise based on the second target image and the second brightness value using the noise algorithm library.

29. The method according to any one of claims 18 to 24, characterized in that, The first processor receives the first image through the HWC module, including: The first processor receives the fifth display parameter sent by the Surface Flinger module of the electronic device through the HWC module; The first processor obtains the first image based on the fifth display parameter through the HWC module.

30. The method according to any one of claims 17 to 24, characterized in that, The first region is the area on the display screen of the electronic device located above the ambient light sensor of the electronic device.

31. The method as described in claim 21, characterized in that, Before the first processor obtains the time of the last image refresh of the electronic device through the HWC module, it includes: The first processor receives the sixth display parameter sent by the Surface Flinger module of the electronic device through the HWC module; The first processor stores the moment when the HWC module receives the sixth display parameter through the HWC module; The first processor obtains the time of the last image refresh of the electronic device through the HWC module, including: The first processor obtains the time when the sixth display parameter is received through the HWC module. The time when the sixth display parameter is received is the latest time when the HWC module receives the display parameter before the last time the image was refreshed on the electronic device.

32. The method as described in claim 20, characterized in that, The first display parameters include: the position, size, color, and storage address of the interface that synthesizes the third image on the display screen of the electronic device.

33. An electronic device, characterized in that, The electronic device includes a first processor for running a computer program stored in a memory to enable the electronic device to perform the method as claimed in any one of claims 1 to 16 or the method as claimed in any one of claims 17 to 32.

34. A chip system, characterized in that, The method includes a first processor coupled to a memory, the first processor executing a computer program stored in the memory to implement the method as described in any one of claims 17 to 32.

35. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a processor, implements the method as claimed in any one of claims 1 to 16 or the method as claimed in any one of claims 17 to 32.

Images