Backlight adjustment method, device and storage medium
By determining and utilizing the feature information of the target focus area and eye area in the electronic device, and combining the neural network model for backlight adjustment, the problem of single and poor backlight adjustment function in the prior art is solved, and more intelligent and Personalized backlight adjustments.
Patent Information
- Application Number
- CN202011311137.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-20
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-11-20
AI Technical Summary
The backlight adjustment function of existing electronic devices is too single, the adjustment effect is poor, and lacks intelligence, so it is impossible to effectively combine user eye information for personalized adjustments.
By determining the user's feature information for the target area of interest and eye area of the display screen, the target backlight adjustment parameters are determined using feature extraction and neural network model to achieve intelligent backlight adjustment.
It improves the intelligence and adaptability of backlight adjustment, making the adjusted backlight more in line with user preferences, suitable for human eyes, and improves the display effect.
Smart Images

Figure CN114519891B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of backlight adjustment, and in particular to a backlight adjustment method, device and storage medium. Background Art
[0002] With the widespread popularization of electronic devices (such as mobile phones, tablet computers, etc.), electronic devices can support more and more applications and have more and more powerful functions. Electronic devices are developing in a diversified and personalized direction and have become an indispensable electronic product in users' lives.
[0003] In our daily life, backlight adjustment has become an essential skill for electronic devices. However, at present, backlight adjustment is often simply implemented based on ambient light. The function is too single and the adjustment effect is poor. Therefore, the problem of how to improve the intelligence of backlight adjustment needs to be solved urgently. Summary of the invention
[0004] The embodiments of the present application provide a backlight adjustment method, device and storage medium, which can improve the intelligence of backlight adjustment.
[0005] In a first aspect, an embodiment of the present application provides a backlight adjustment method, which is applied to an electronic device, and the method includes:
[0006] Determine a regional image of a user's target focus area on a display screen;
[0007] Extracting features from the regional image to obtain first feature information;
[0008] Acquire an eye region image of the user, and perform feature extraction on the eye region image to obtain second feature information;
[0009] Determine a target backlight adjustment parameter according to the first characteristic information and the second characteristic information;
[0010] Backlight adjustment is performed according to the target backlight adjustment parameter.
[0011] In a second aspect, an embodiment of the present application provides a backlight adjustment device, which is applied to an electronic device. The device includes: a first determination unit, a first extraction unit, a second extraction unit, a second determination unit and an adjustment unit, wherein:
[0012] The first determining unit is used to determine a region image of a target focus region of the display screen by the user;
[0013] The first extraction unit is used to extract features from the regional image to obtain first feature information;
[0014] The second extraction unit is used to obtain an eye region image of the user, and perform feature extraction on the eye region image to obtain second feature information;
[0015] The second determining unit is used to determine a target backlight adjustment parameter according to the first characteristic information and the second characteristic information;
[0016] The adjustment unit is used to adjust the backlight according to the target backlight adjustment parameter.
[0017] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program comprises instructions for executing some or all of the steps described in the method described in the first aspect of the embodiment of the present application.
[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program is executed by a processor to implement part or all of the steps described in the method described in the first aspect of the embodiment of the present application.
[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the method described in the first aspect of the embodiment of the present application. The computer program product may be a software installation package.
[0020] Implementing the embodiments of the present application has the following beneficial effects:
[0021] It can be seen that the backlight adjustment method, device and storage medium described in the embodiments of the present application are applied to electronic devices to determine the regional image of the user's target focus area on the display screen, perform feature extraction on the regional image to obtain a first feature information, obtain the user's eye area image, and perform feature extraction on the eye area image to obtain a second feature information, determine the target backlight adjustment parameters based on the first feature information and the second feature information, and adjust the backlight based on the target backlight adjustment parameters. When adjusting the backlight, the backlight can be adjusted in combination with the changes in the user's eye information, that is, the stimulation of the background to the eyeball, so that the adjusted backlight is more in line with the user's preferences and makes the backlight adjustment effect more suitable for the human eye, which helps to improve the intelligence of the backlight adjustment.
[0022] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0024] Figure 1 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;
[0025] Figure 2 It is a schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;
[0026] Figure 3 is a software architecture diagram of an eye tracking system provided in an embodiment of the present application;
[0027] Figure 4A is a flow chart of a backlight adjustment method provided in an embodiment of the present application;
[0028] Figure 4B It is a demonstration schematic diagram of the application scenario provided by the embodiment of the present application;
[0029] Figure 5 is an interactive diagram of a backlight adjustment method provided in an embodiment of the present application;
[0030] Figure 6 is another hardware structure diagram of an electronic device provided in an embodiment of the present application;
[0031] Fig. 7A is a structural schematic diagram of a backlight adjustment device provided in an embodiment of the present application;
[0032] Figure 7B It is a structural schematic diagram of another backlight adjustment device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0034] The following are detailed descriptions of each.
[0035] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.
[0036] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0037] Below, some terms in this application are explained to facilitate understanding by those skilled in the art.
[0038] Electronic devices may have a backlight adjustment function, such as smart phones, vehicle-mounted equipment (navigation devices, driving recorders, radar rangefinders, ETC payment devices, etc.), wearable devices, smart watches, walkie-talkies, smart glasses, wireless Bluetooth headsets, computing devices, routers or other processing devices connected to wireless modems, as well as various forms of user equipment (UE), mobile stations (MS), virtual reality / augmented reality devices, terminal devices, etc.
[0039] Electronic devices may also include smart home devices, which may be at least one of the following: smart speakers, smart cameras, smart rice cookers, smart wheelchairs, smart massage chairs, smart furniture, smart dishwashers, smart televisions, smart refrigerators, smart electric fans, smart heaters, smart clothes drying racks, smart lights, smart routers, smart switches, smart switch panels, smart humidifiers, smart air conditioners, smart doors, smart windows, smart stoves, smart disinfection cabinets, smart toilets, sweeping robots, etc., without limitation.
[0040] In the first part, the software and hardware operating environment of the technical solution disclosed in this application is introduced as follows.
[0041] As shown in the figure, Figure 11 shows a schematic diagram of the structure of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a compass 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195.
[0042] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0043] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor AP, a modem processor, a graphics processor GPU, an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor NPU, etc. Among them, different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 101 may also include one or more processors 110. Among them, the controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. In some other embodiments, a memory may also be provided in the processor 110 for storing instructions and data. Exemplarily, the memory in the processor 110 may be a high-speed cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. In this way, repeated access is avoided, the waiting time of the processor 110 is reduced, and the efficiency of the electronic device 101 in processing data or executing instructions is improved.
[0044] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface and / or a USB interface, etc. Among them, the USB interface 130 is an interface that complies with the USB standard specification, and can specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 101, and can also be used to transmit data between the electronic device 101 and a peripheral device. The USB interface 130 can also be used to connect headphones to play audio through the headphones.
[0045] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0046] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.
[0047] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193 and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.
[0048] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, the baseband processor, and the like.
[0049] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0050] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G / 6G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.
[0051] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), UWB modules, etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.
[0052] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
[0053] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini light-emitting diode (miniLED), MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or more display screens 194.
[0054] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.
[0055] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.
[0056] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include one or more cameras 193.
[0057] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
[0058] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
[0059] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0060] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.
[0061] The internal memory 121 can be used to store one or more computer programs, which include instructions. The processor 110 can execute the above instructions stored in the internal memory 121, so that the electronic device 101 performs the method of displaying page elements provided in some embodiments of the present application, as well as various applications and data processing. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as a gallery, contacts, etc.). The data storage area can store data (such as photos, contacts, etc.) created during the use of the electronic device 101. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, flash memory components, universal flash storage (UFS), etc. In some embodiments, the processor 110 can execute the method of displaying page elements provided in the embodiments of the present application, as well as other applications and data processing by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor 110. The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.
[0062] The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.
[0063] Among them, the pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.
[0064] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., X, Y, and Z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.
[0065] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.
[0066] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.
[0067] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.
[0068] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.
[0069] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.
[0070] For example, Figure 2 The software structure diagram of the electronic device 100 is shown. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, respectively, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages.
[0071] like Figure 2 As shown, the application layer may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.
[0072] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.
[0073] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.
[0074] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.
[0075] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
[0076] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.
[0077] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).
[0078] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.
[0079] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.
[0080] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.
[0081] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.
[0082] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
[0083] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.
[0084] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.
[0085] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
[0086] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.
[0087] A 2D graphics engine is a drawing engine for 2D drawings.
[0088] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.
[0089] In the second part, the backlight adjustment method and device disclosed in the embodiments of the present application are introduced as follows.
[0090] like Figure 3 As shown, Figure 3It is a software architecture diagram of a backlight adjustment method provided in an embodiment of the present application. The software architecture diagram includes four layers, wherein the first layer is an application layer, which may include applications such as e-books, browsers, launchers, systems, unlocking, mobile payments, and points of interest tracking. The second layer may include an eye tracking service (OEyeTrackerService), which specifically includes: eye tracking authorization (OEyeTrackerAuthentication), eye tracking strategy (OEyeTrackerStrategy), eye tracking algorithm (OEyeTrackerAlgo) and eye tracking parameters (OEyeTrackerParams) and other modules, wherein OEyeTrackerService is connected to the application of the first layer through the eye tracking SDK (OEyeTrackerSDK) interface; the second layer also includes a camera NDK interface (CameraNDKInterface), a camera service (CameraService), and CameraNDKInterface is connected to OEyeTrackerService, and CameraService is connected to CameraNDKInterface. The third layer is the hardware abstraction layer, which may include Google HAL Interface, Qualcomm HAL Interface, electronic image stabilization module, Cam X, Chi-cdk, etc. Qualcomm HAL Interface can connect to the electronic image stabilization module, Google HAL Interface is connected to the CameraService of the second layer, Qualcomm HAL Interface is connected to Google HAL Interface, Cam X is connected to Qualcomm HAL Interface and Chi-cdk respectively, and the fourth layer is the underlying driver, which includes RGB sensor, digital signal processor (DSP), infrared sensor (IR sensor), laser and light-emitting diode (LED), etc. IR sensor is connected to Cam X of the third layer. The connection between OEyeTrackerService and OEyeTrackerSDK, the connection between CameraService and CameraNDKInterface, and the connection between Google HAL Interface and CameraService are all through the Binder architecture.
[0091] Among them, OEyeTrackerSDK is responsible for providing common applications with APIs for obtaining gaze points and inputs in the form of jar / aar packages. OEyeTrackerService is responsible for managing gaze point algorithms, gaze point post-processing, input processing, authentication and parameter settings. EyeTrackerAlgo is the core algorithm for eye tracking, including the algorithm for determining the gaze point function in this application. OEyeTrackerStrategy is related to algorithm post-processing, such as filtering, gaze point jump, gaze point conversion monitoring, and gaze point input. OEyeTrackerAuthentication calls back each module and is responsible for authenticating whether the requester is allowed. OEyeTrackerParam is responsible for parsing configuration and hot updating configuration. The electronic image stabilization module is used to realize the electronic image stabilization function. The principle is to first fix the CCD on a bracket that can move up, down, left and right, and use the gyroscope to sense the direction and amplitude of the camera shake. The sensor then transmits this data to the processor for screening and amplification, and calculates the CCD movement that can offset the shake.
[0092] The eye gaze point is the gaze point position of the user's eye on the plane where the electronic device is located. The eye tracking SDK interface is a software development kit (SDK) interface provided by the electronic device for the eye tracking application, which is responsible for providing the eye tracking application with an application programming interface (API) interface for obtaining the gaze point and input. The eye tracking service can also call the camera application through the camera Native Development Kit (NDK) interface, and the camera application can call the camera to collect images of the user's eye area through the camera.
[0093] like Figure 4A As shown, Figure 4A is a flow chart of a backlight adjustment method provided in an embodiment of the present application, which is applied to Figure 1-Figure 3 In any electronic device shown in the figure, the method comprises:
[0094] 401. Determine a region image of a target focus region of a display screen by a user.
[0095] Among them, the electronic device may include a camera. When the user looks at the display screen, the camera can track the user's eyes, not only to obtain the user's eye area image, but also to obtain the user's target focus area for the display screen, so as to obtain the regional image of the target focus area by screenshot or screenshot. The camera can also be a single camera, dual cameras or multiple cameras. The single camera can be an infrared camera, a visible light camera (normal viewing angle camera or wide-angle camera), and the dual cameras can be a normal viewing angle camera + a wide-angle camera, or an infrared camera + a visible light camera.
[0096] In a specific implementation, when the camera can be a front-facing camera, such as Figure 4B As shown, when the target object takes a selfie or activates the eye tracking function, the camera can be used to obtain a first image of the user's selfie and the eye gaze point of the target object in the first image.
[0097] 402. Perform feature extraction on the regional image to obtain first feature information.
[0098] Among them, in the embodiment of the present application, the specific feature extraction method can be at least one of the following: color information extraction, feature point extraction, overall feature extraction, etc., which are not limited here. Color information extraction can be achieved through color space transformation, for example, RGB to YUV, etc. The specific algorithm for feature point extraction can be at least one of the following: Harris corner point extraction algorithm, scale-invariant feature extraction algorithm, etc., which are not limited here. The specific algorithm for overall feature extraction can be at least one of the following: average grayscale, mean square error, average gradient, clarity, etc., which are not limited here. The electronic device can perform feature extraction on the regional image based on the above-mentioned feature extraction method to obtain the first feature information. The first feature information can be at least one of the following: color information, feature point information, overall feature information, etc., which are not limited here.
[0099] 403. Obtain an eye region image of the user, and perform feature extraction on the eye region image to obtain second feature information.
[0100] In an embodiment of the present application, the electronic device can photograph the user through a camera to obtain a photographed image, perform area extraction on the photographed image to obtain an eye area image, and perform feature extraction on the eye area image to obtain second feature information. The second feature information can be at least one of the following: eyeball radius, pupil radius, eye opening degree, the proportional relationship between the pupil radius and the eyeball radius, etc., which is not limited here.
[0101] 404. Determine a target backlight adjustment parameter according to the first characteristic information and the second characteristic information.
[0102] Among them, in the embodiment of the present application, the first characteristic information reflects the backlight information to a certain extent, and the second characteristic information reflects the user's adaptation degree to the focus area to a certain extent. Furthermore, the backlight adjustment can be achieved through the correlation between the two. The backlight effect after adjustment is more suitable for the human eye and can achieve personalized ("customized") backlight adjustment effect.
[0103] Optionally, the above step 404, determining the target backlight adjustment parameter according to the first characteristic information and the second characteristic information, may include the following steps:
[0104] A41 inputs the first feature information and the second feature information into a target neural network model to obtain a target operation result;
[0105] A42. Determine the target backlight adjustment parameter corresponding to the target calculation result according to a preset mapping relationship between the calculation result and the backlight adjustment parameter.
[0106] Among them, in the embodiments of the present application, the target neural network model can be at least one of the following: a convolutional neural network model, a fully connected neural network model, a pulse neural network model, a recurrent neural network model, etc., which are not limited here.
[0107] In a specific implementation, the electronic device can pre-store a mapping relationship between preset calculation results and backlight adjustment parameters, and the electronic device can input the first feature information and the second feature information into the target neural network model to obtain the target calculation result. Then, the target backlight adjustment parameters corresponding to the target calculation result can be determined according to the mapping relationship between the preset calculation results and the backlight adjustment parameters. The target neural network model can be used to express the degree of adaptation of the user's eyes, and then, backlight adjustment can be achieved based on the degree of adaptation of the image of the user's focus area to the user's eyes.
[0108] Further, optionally, before the above step 401, determining the area image of the target focus area of the display screen by the user, the following steps may also be included:
[0109] A1. Obtain the background sample feature set of the user's attention area;
[0110] A2, a set of eyeball sample features based on the eye area image corresponding to the user's attention area;
[0111] A3. Input the background sample feature set and the eye sample feature set into a preset neural network model for training to obtain the target neural network model.
[0112] Among them, the preset neural network model can be at least one of the following: convolutional neural network model, fully connected neural network model, pulse neural network model, recurrent neural network model, etc., which are not limited here. Specifically, before step 401, the electronic device can obtain the background sample feature set of the user's focus area, and the eye sample feature set of the eye area image corresponding to the user's focus area, and then input the background sample feature set and the eye sample feature set into the preset neural network model for training to obtain a target neural network model. In this way, the target neural network model can express the degree of adaptation of the eyeball to the display screen, and then, the backlight adjustment can be achieved according to the degree of adaptation, so that the display effect is more suitable for the human eye.
[0113] Optionally, when the first feature information includes target color distribution state, average brightness value and target feature point distribution density, and the second feature information includes target eye feature parameters, the above step 404, determining the target backlight adjustment parameters according to the first feature information and the second feature information, may include the following steps:
[0114] B41. Determine a first eye feature parameter corresponding to the average brightness value according to a preset mapping relationship between brightness values and eye feature parameters;
[0115] B42. Determine a target first adjustment factor corresponding to the target color distribution state according to a preset mapping relationship between the color distribution state and the first adjustment factor;
[0116] B43, determining a target second adjustment factor corresponding to the target feature point distribution density according to a preset mapping relationship between the feature point distribution density and the second adjustment factor;
[0117] B44, adjusting the first eye feature parameter according to the target first adjustment factor and the target second adjustment factor to obtain a reference eye feature parameter;
[0118] B45, determining a target deviation between the reference eye feature parameter and the target eye feature parameter;
[0119] B46. Determine the target backlight adjustment parameter corresponding to the target deviation according to a preset mapping relationship between the deviation and the backlight adjustment parameter.
[0120] In a specific implementation, the electronic device may pre-store a mapping relationship between a preset brightness value and an eye feature parameter, a mapping relationship between a preset color distribution state and a first adjustment factor, a mapping relationship between a preset feature point distribution density and a second adjustment factor, and a mapping relationship between a preset deviation and a backlight adjustment parameter, wherein the value range of the first adjustment factor and the value range of the second adjustment factor may be set by the user or by the system default, for example, the value range of the first adjustment factor may be -0.12 to 0.12, and the value range of the second adjustment factor may be -0.08 to 0.08.
[0121] In a specific implementation, when the first feature information includes the target color distribution state, the average brightness value, and the target feature point distribution density, and the second feature information includes the target eye feature parameter, the electronic device can determine the first eye feature parameter corresponding to the average brightness value according to the mapping relationship between the preset brightness value and the eye feature parameter, and determine the target first adjustment factor corresponding to the target color distribution state according to the mapping relationship between the preset color distribution state and the first adjustment factor, and determine the target second adjustment factor corresponding to the target feature point distribution density according to the mapping relationship between the preset feature point distribution density and the second adjustment factor, and then adjust the first eye feature parameter according to the target first adjustment factor and the target second adjustment factor to obtain the reference eye feature parameter, which can be calculated according to the following formula:
[0122] Reference eye feature parameter = first eye feature parameter * (1 + first adjustment factor) * (1 + second adjustment factor)
[0123] Furthermore, the electronic device can determine a target deviation between a reference eye feature parameter and a target eye feature parameter, for example, target deviation = |reference eye feature parameter-target eye feature parameter|. Finally, the electronic device can determine the target backlight adjustment parameter corresponding to the target deviation according to a mapping relationship between a preset deviation and a backlight adjustment parameter. In this way, the color of the backlight image, the image detail features, and the background brightness can be used to comprehensively evaluate the impact of the user's focus area on the user's eyes. Furthermore, the backlight adjustment parameters can be accurately determined, which helps to achieve accurate backlight adjustment.
[0124] 405. Perform backlight adjustment according to the target backlight adjustment parameter.
[0125] In a specific implementation, the electronic device may adjust the backlight of the target area of interest according to the target backlight adjustment parameter, or may adjust the backlight of the entire display screen. Specifically, for example, when the value range of the target backlight adjustment parameter is between -1 and 1, such as, when the value range is -0.1 to 0.1, the backlight adjustment may be implemented by referring to the following formula:
[0126] Adjusted backlight parameter = current backlight parameter * (1 + target backlight adjustment parameter)
[0127] For another example, when the target backlight adjustment parameter is a specific backlight value, the backlight adjustment can be implemented by referring to the following formula:
[0128] Adjusted backlight parameter = current backlight parameter + target backlight adjustment parameter
[0129] Of course, other adjustment methods are also possible, which are not limited here and depend on the actual situation.
[0130] In an embodiment of the present application, gaze point information is obtained through eye tracking technology, and the user's focus area is determined according to the position of the gaze point. At the same time, combined with the user's eye information (for example, information such as pupil size), the appropriate backlight brightness for the user in certain areas can be determined. When determining the focus area, it is necessary to extract the features of the relevant area, such as the color or objects in the area, and then obtain a backlight adjustment model through sample training. The model can reflect the value of the backlight brightness that suits the user's preference under the premise that the color and objects in the content of the focus area are determined, and whether it suits the user's backlight preference can be determined based on changes in eye information. For example, when the pupil size (there is a certain relationship between the pupil size and the amount of light entering, that is, it reflects whether the brightness is suitable for the user's current needs) changes to a certain value, it can be considered that the current brightness is suitable for the user's preference.
[0131] Optionally, the above step 405, performing backlight adjustment according to the target backlight adjustment parameter, may include the following steps:
[0132] The backlight of the target area of interest is adjusted according to the target backlight adjustment parameter.
[0133] In a specific implementation, the electronic device can adjust the backlight of the target focus area according to the target backlight adjustment parameter. For example, the adjustment of the brightness value is not for the entire display area, but mainly for the area that the user currently focuses on, so as to save the power consumption of the device on the basis of meeting the user's needs (for example, the backlight brightness of the non-focus area can be reduced accordingly based on the brightness adjustment value of the focus area to achieve a smooth brightness effect from bright to dark). When adjusting the brightness of the user's focus area, the characteristics of the user's interest area are determined according to the characteristics of the user's interest area. For example, for color, the user may want the brightness of green or blue to be slightly higher, so the brightness of the relevant color area should be increased accordingly during the adjustment. In addition, the same brightness adjustment can be performed on the area related to the user's current focus area by content recognition (for example, if the user focuses on the text part, the backlight adjustment of the text part of the entire display area is performed to a higher standard).
[0134] In summary, the embodiments of the present application avoid the problem of excessive increase in power consumption caused by traditional adjustment strategies through differentiated backlight adjustment of the display area of the display screen; at the same time, the embodiments of the present application combine the changes in user eye information (pupil size and other information) when adjusting the backlight, so that the adjusted backlight is more in line with the user's preferences, achieving a better backlight adjustment effect.
[0135] Optionally, before the above step 401, determining the area image of the target focus area of the display screen by the user, may further include the following steps:
[0136] C1. determining a target gaze point on the display screen;
[0137] C2, determining the fixation duration for the target fixation point;
[0138] C3. When the gaze duration is greater than a preset duration, an area within a preset range including the target gaze point is used as the target attention area, and a regional image corresponding to the target attention area is acquired.
[0139] The preset duration can be set by the user or by the system default, and the preset range can be set by the user or by the system default. The electronic device can determine the target gaze point on the display screen, and can also determine the gaze duration for the target gaze point, and when the gaze duration is greater than the preset duration, the area within the preset range including the target gaze point is taken as the target focus area, and the regional image corresponding to the target focus area is obtained. For example, the area within the preset range centered on the target gaze point can be taken as the target focus area, and for another example, the display screen can be divided into multiple areas in advance, and the area where the target gaze point is located is taken as the target focus area.
[0140] Further, optionally, the above step C3, taking the area within the preset range including the target gaze point as the target focus area, may include the following steps:
[0141] C31. Acquire target size information of the display area of the display screen;
[0142] C32, determining a target distance between the user's eyes and the display screen;
[0143] C33, determining a reference focus range corresponding to the target distance according to a preset mapping relationship between the distance and the focus range;
[0144] C34. Determine a target optimization coefficient corresponding to the target size information according to a preset mapping relationship between the size information and the optimization coefficient;
[0145] C35, optimizing the reference attention range according to the target optimization coefficient to obtain a target attention range;
[0146] C36. Use the area of the target attention range centered on the target attention point as the target attention area.
[0147] The electronic device may pre-store a mapping relationship between a preset distance and a focus range and a mapping relationship between preset size information and an optimization coefficient.
[0148] In a specific implementation, the electronic device can obtain the target size information of the display area of the display screen, and determine the target distance between the user's eyes and the display screen, and then determine the reference focus range corresponding to the target distance according to the mapping relationship between the preset distance and the focus range, and determine the target optimization coefficient corresponding to the target size information according to the mapping relationship between the preset size information and the optimization coefficient, and optimize the reference focus range according to the target optimization coefficient to obtain the target focus range, wherein the value range of the optimization system can be set by the user or by the system default, for example, the optimization coefficient can be -0.1 to 0.1, and the optimization process can be: target focus range = reference focus range * (1 + target optimization coefficient), finally, the electronic device can use the area of the target focus range as the target focus area as the center, in this way, the size of the user's focus area can be adjusted in combination with the screen size of the display screen and the distance between the user and the display screen, so that the area of the user's focus can be accurately determined to a certain extent.
[0149] To illustrate, in a specific implementation, the electronic device can start the eye tracking module when the user adjusts the backlight adjustment mode to automatic, capture and analyze the user's eye signal, obtain the area of user attention, and at the same time, extract the image of the area of attention, and analyze the feature information and related color information of the object contained in the image of the area of attention, and transmit the obtained feature information and color information to the model training module (such as the target neural network model). In addition, the user's eye feature information, such as the relative size of the pupil, etc., is obtained at the same time, and the eye information is transmitted to the model training module at the same time. The model training module can obtain a backlight adjustment model that can reflect the user's preference through sample training, and then automatically and differentially adjust the backlight brightness value according to the new feature area and the characteristics of the user's eye information. The specific process is as follows:
[0150] 1. Change the backlight adjustment to automatic mode;
[0151] 2. Start the eye recognition module;
[0152] 3. Extract and judge the user's gaze point information (for example, the gaze time for a certain gaze area).
[0153] 4. If the gaze point information meets a certain requirement (gaze time is greater than t), the image information of the relevant gaze area will be extracted;
[0154] 5. Send the image information of the area of interest to the feature extraction algorithm for processing, and the algorithm outputs the processed feature information (color information and object feature information) to the model training algorithm;
[0155] 6. At the same time, extract the user's eye information and use it as input for the model training algorithm;
[0156] 7. Model training: A backlight adjustment model that can reflect user preferences is obtained through sample training;
[0157] 8. Determine the appropriate backlight value for the key area using the backlight adjustment model obtained based on the new eye tracking data and user eye information;
[0158] 9. Make differentiated backlight adjustments.
[0159] The use of eye information in the backlight adjustment model needs to focus on some information that can reflect the user's subjective backlight preference and objective indicators. For example, the size of the pupil can reflect the objective impact of the current backlight on the user. When the backlight intensity changes, the size of the pupil will automatically adjust to the new backlight intensity like an aperture; the proportion of the user's eye opening also reflects whether the current backlight is suitable for the user's preference. If the backlight is dim, the user will increase the proportion of eye opening. Similarly, if the backlight is too bright, the user will reduce the proportion of eye opening. In addition, the relationship between backlight intensity and eye information in the user's manual backlight adjustment mode can also be collected as a sample for model training.
[0160] For the training and correction of the backlight adjustment model, the following information can be used as input to the training algorithm: gaze point information (analyzing the characteristics of the gaze area based on the gaze point information), eyeball information (including the size of the eye opening and the pupil size), and the backlight brightness value suitable for the user (not necessarily exists. If the user is currently adjusting manually, the current brightness value is used as the appropriate brightness value, and the pupil size and the ratio of eye opening at this time are used as the standard value (i.e., the backlight standard suitable for the user)). It should be noted that the training process does not necessarily rely on the parameters of manual backlight adjustment. You can also choose an objective and scientific indicator as a standard for training (for example, what value of the reasonable pupil size and what degree of eye opening is the most suitable user experience for people); adjust the backlight according to the training results, and then correct the model according to the user's response after the adjustment.
[0161] In the specific implementation, since the distance between the user and the screen will affect the absolute value of the collected eye parameters (for example, the size of the eyes and the size of the pupil in the image), some ratio values can be used as the data of the eyeball information, such as a / b as the parameter of the eye opening size (a is the longitudinal opening distance of the eyelid, and b is the distance between the left and right corners of the eye area); similarly, for the pupil size parameter, d' / d (d' is the pupil diameter, and d is the diameter of the black eyeball area) can also be used as pupil data. The model can be simply defined as M, which can be expressed as: M<-(a / b, d' / d, image content corresponding to the gaze point).
[0162] It can be seen that the backlight adjustment method described in the embodiment of the present application is applied to an electronic device, determines a regional image of a user's target focus area on a display screen, performs feature extraction on the regional image to obtain a first feature information, obtains a user's eye regional image, and performs feature extraction on the eye regional image to obtain a second feature information, determines a target backlight adjustment parameter based on the first feature information and the second feature information, and performs backlight adjustment based on the target backlight adjustment parameter. When adjusting the backlight, the backlight can be adjusted in combination with changes in the user's eye information, that is, the stimulation of the background to the eyeball, so that the adjusted backlight is more in line with the user's preferences and makes the backlight adjustment effect more suitable for the human eye, which helps to improve the intelligence of the backlight adjustment.
[0163] With the above Figure 4A For the embodiments shown, please refer to Figure 5 , Figure 5 : is a flow chart of a backlight adjustment method provided in an embodiment of the present application. As shown in the figure, it is applied to Figure 1-Figure 3 For any electronic device shown in the figure, the backlight adjustment method includes:
[0164] 501. Determine a target gaze point on the display screen.
[0165] 502. Determine a fixation duration for the target fixation point.
[0166] 503. When the gaze duration is greater than a preset duration, a region within a preset range including the target gaze point is used as the target attention region, and a region image corresponding to the target attention region is acquired.
[0167] 504. Perform feature extraction on the regional image to obtain first feature information.
[0168] 505. Obtain an eye region image of the user, and perform feature extraction on the eye region image to obtain second feature information.
[0169] 506. Determine a target backlight adjustment parameter according to the first characteristic information and the second characteristic information.
[0170] 507. Perform backlight adjustment according to the target backlight adjustment parameter.
[0171] The specific description of the above steps 501 to 507 can be referred to as follows Figure 4A The corresponding steps of the described backlight adjustment method will not be repeated here.
[0172] It can be seen that the backlight adjustment method described in the embodiment of the present application can adjust the backlight by combining the changes in the user's eye information, that is, the stimulation of the background to the eyeball, so that the adjusted backlight is more in line with the user's preferences and the backlight adjustment effect is more suitable for the human eye, which helps to improve the intelligence of the backlight adjustment.
[0173] In accordance with the above embodiment, please refer to Figure 6 , Figure 6 : is a structural diagram of an electronic device provided in an embodiment of the present application. As shown in the figure, the electronic device includes a processor, a memory, a communication interface, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the processor. In the embodiment of the present application, the program includes instructions for executing the following steps:
[0174] Determine a regional image of a user's target focus area on a display screen;
[0175] Extracting features from the regional image to obtain first feature information;
[0176] Acquire an eye region image of the user, and perform feature extraction on the eye region image to obtain second feature information;
[0177] Determine a target backlight adjustment parameter according to the first characteristic information and the second characteristic information;
[0178] Backlight adjustment is performed according to the target backlight adjustment parameter.
[0179] It can be seen that the electronic device described in the embodiments of the present application determines the regional image of the user's target focus area on the display screen, performs feature extraction on the regional image to obtain a first feature information, obtains the user's eye regional image, and performs feature extraction on the eye regional image to obtain a second feature information, determines the target backlight adjustment parameters according to the first feature information and the second feature information, and adjusts the backlight according to the target backlight adjustment parameters. When adjusting the backlight, it can combine the changes in the user's eye information, that is, the stimulation of the background to the eyeball to adjust the backlight, so that the adjusted backlight is more in line with the user's preferences and makes the backlight adjustment effect more suitable for the human eye, which helps to improve the intelligence of the backlight adjustment.
[0180] Optionally, in determining the target backlight adjustment parameter according to the first feature information and the second feature information, the program includes instructions for executing the following steps:
[0181] Inputting the first feature information and the second feature information into a target neural network model to obtain a target operation result;
[0182] According to a preset mapping relationship between the calculation result and the backlight adjustment parameter, the target backlight adjustment parameter corresponding to the target calculation result is determined.
[0183] Optionally, before determining the area image of the target area of interest of the user on the display screen, the program further includes instructions for executing the following steps:
[0184] Obtain a background sample feature set of the user's attention area;
[0185] An eyeball sample feature set based on an eye area image corresponding to the user focus area;
[0186] The background sample feature set and the eye sample feature set are input into a preset neural network model for training to obtain the target neural network model.
[0187] Optionally, when the first feature information includes a target color distribution state, an average brightness value, and a target feature point distribution density, and the second feature information includes a target eye feature parameter, in determining the target backlight adjustment parameter according to the first feature information and the second feature information, the program includes instructions for executing the following steps:
[0188] Determine the first eye feature parameter corresponding to the average brightness value according to a preset mapping relationship between the brightness value and the eye feature parameter;
[0189] Determine a target first adjustment factor corresponding to the target color distribution state according to a preset mapping relationship between the color distribution state and the first adjustment factor;
[0190] Determine the target second adjustment factor corresponding to the target feature point distribution density according to the preset mapping relationship between the feature point distribution density and the second adjustment factor;
[0191] Adjusting the first eye feature parameter according to the target first adjustment factor and the target second adjustment factor to obtain a reference eye feature parameter;
[0192] Determining a target deviation between the reference eye feature parameter and the target eye feature parameter;
[0193] According to a preset mapping relationship between the deviation degree and the backlight adjustment parameter, the target backlight adjustment parameter corresponding to the target deviation degree is determined.
[0194] Optionally, in the aspect of performing backlight adjustment according to the target backlight adjustment parameter, the program includes instructions for executing the following steps:
[0195] The backlight of the target area of interest is adjusted according to the target backlight adjustment parameter.
[0196] Optionally, in the aspect of determining the area image of the target focus area of the display screen by the user, the program includes instructions for executing the following steps:
[0197] determining a target gaze point on the display screen;
[0198] Determining a fixation duration for the target fixation point;
[0199] When the gaze duration is greater than a preset duration, an area within a preset range including the target gaze point is used as the target attention area, and a regional image corresponding to the target attention area is acquired.
[0200] Optionally, in the aspect of taking the area within the preset range including the target gaze point as the target focus area, the program includes instructions for executing the following steps:
[0201] Acquiring target size information of a display area of the display screen;
[0202] determining a target distance between the user's eyes and the display screen;
[0203] According to a preset mapping relationship between the distance and the focus range, a reference focus range corresponding to the target distance is determined;
[0204] Determine the target optimization coefficient corresponding to the target size information according to the mapping relationship between the preset size information and the optimization coefficient;
[0205] Optimizing the reference attention range according to the target optimization coefficient to obtain a target attention range;
[0206] The area of the target attention range with the target attention point as the center is used as the target attention area.
[0207] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.
[0208] The embodiment of the present application can divide the electronic device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.
[0209] Fig. 7A 700 is a functional unit block diagram of a backlight adjustment device 700 involved in an embodiment of the present application. The backlight adjustment device 700 is applied to an electronic device, and the device 700 includes: a first determination unit 701, a first extraction unit 702, a second extraction unit 703, a second determination unit 704 and an adjustment unit 705, wherein:
[0210] The first determining unit 701 is used to determine a region image of a target focus region of a display screen by a user;
[0211] The first extraction unit 702 is used to extract features from the regional image to obtain first feature information;
[0212] The second extraction unit 703 is used to obtain an eye region image of the user, and perform feature extraction on the eye region image to obtain second feature information;
[0213] The second determining unit 704 is used to determine a target backlight adjustment parameter according to the first characteristic information and the second characteristic information;
[0214] The adjusting unit 705 is used to adjust the backlight according to the target backlight adjustment parameter.
[0215] It can be seen that the backlight adjustment device described in the embodiment of the present application is applied to an electronic device, determines a regional image of a user's target focus area on a display screen, performs feature extraction on the regional image to obtain a first feature information, obtains a user's eye regional image, and performs feature extraction on the eye regional image to obtain a second feature information, determines a target backlight adjustment parameter based on the first feature information and the second feature information, and performs backlight adjustment based on the target backlight adjustment parameter. When adjusting the backlight, it is possible to combine changes in the user's eye information, that is, the stimulation of the background to the eyeball to perform backlight adjustment, so that the adjusted backlight is more in line with the user's preferences and makes the backlight adjustment effect more suitable for the human eye, which helps to improve the intelligence of the backlight adjustment.
[0216] Optionally, in determining the target backlight adjustment parameter according to the first feature information and the second feature information, the program includes instructions for executing the following steps:
[0217] Inputting the first feature information and the second feature information into a target neural network model to obtain a target operation result;
[0218] According to a preset mapping relationship between the calculation result and the backlight adjustment parameter, the target backlight adjustment parameter corresponding to the target calculation result is determined.
[0219] Alternatively, if Figure 7B As shown, Figure 7B for Fig. 7A Another variant structure of the backlight adjustment device 700 shown in FIG. Fig. 7A In comparison, the method may further include: a model training unit 706, which is specifically as follows:
[0220] The model training unit 706 is used to perform the following functions:
[0221] Before determining the area image of the target focus area of the display screen by the user, the method further includes:
[0222] Obtain a background sample feature set of the user's attention area;
[0223] An eyeball sample feature set based on an eye area image corresponding to the user focus area;
[0224] The background sample feature set and the eye sample feature set are input into a preset neural network model for training to obtain the target neural network model.
[0225] Optionally, when the first feature information includes a target color distribution state, an average brightness value, and a target feature point distribution density, and the second feature information includes a target eye feature parameter, in determining the target backlight adjustment parameter according to the first feature information and the second feature information, the second determining unit 704 is specifically used to:
[0226] Determine the first eye feature parameter corresponding to the average brightness value according to a preset mapping relationship between the brightness value and the eye feature parameter;
[0227] Determine a target first adjustment factor corresponding to the target color distribution state according to a preset mapping relationship between the color distribution state and the first adjustment factor;
[0228] Determine the target second adjustment factor corresponding to the target feature point distribution density according to the preset mapping relationship between the feature point distribution density and the second adjustment factor;
[0229] Adjusting the first eye feature parameter according to the target first adjustment factor and the target second adjustment factor to obtain a reference eye feature parameter;
[0230] Determining a target deviation between the reference eye feature parameter and the target eye feature parameter;
[0231] According to a preset mapping relationship between the deviation degree and the backlight adjustment parameter, the target backlight adjustment parameter corresponding to the target deviation degree is determined.
[0232] Optionally, in the aspect of performing backlight adjustment according to the target backlight adjustment parameter, the adjustment unit 705 is specifically configured to:
[0233] The backlight of the target area of interest is adjusted according to the target backlight adjustment parameter.
[0234] Optionally, in determining the area image of the target attention area of the display screen by the user, the first determining unit 701 is specifically configured to:
[0235] determining a target gaze point on the display screen;
[0236] Determining a fixation duration for the target fixation point;
[0237] When the gaze duration is greater than a preset duration, an area within a preset range including the target gaze point is used as the target attention area, and a regional image corresponding to the target attention area is acquired.
[0238] Further, optionally, in the aspect of taking the area within the preset range including the target gaze point as the target focus area, the first determining unit 701 is specifically configured to:
[0239] Acquire target size information of a display area of the display screen;
[0240] determining a target distance between the user's eyes and the display screen;
[0241] According to a preset mapping relationship between the distance and the focus range, a reference focus range corresponding to the target distance is determined;
[0242] Determine the target optimization coefficient corresponding to the target size information according to the mapping relationship between the preset size information and the optimization coefficient;
[0243] Optimizing the reference attention range according to the target optimization coefficient to obtain a target attention range;
[0244] The area of the target attention range with the target attention point as the center is used as the target attention area.
[0245] It should be noted that the first determination unit 701, the first extraction unit 702, the second extraction unit 703, the second determination unit 704, the adjustment unit 705 and the model training unit 706 can all be implemented by a processor, and the first determination unit 701, the first extraction unit 702, the second extraction unit 703 can also be implemented by a camera.
[0246] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.
[0247] The embodiment of the present application also provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, the computer program is operable to cause a computer to execute some or all of the steps of any method described in the method embodiment. The computer program product may be a software installation package, and the computer includes an electronic device.
[0248] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0249] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0250] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.
[0251] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0252] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0253] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.
[0254] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.
[0255] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of the present application. At the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A backlight adjustment method, characterized in that: The method comprises: Determine a regional image of a user's target focus area on a display screen; Extracting features from the regional image to obtain first feature information; Acquire an eye region image of the user, and perform feature extraction on the eye region image to obtain second feature information; determining a target backlight adjustment parameter according to the first characteristic information and the second characteristic information; Performing backlight adjustment according to the target backlight adjustment parameter; Wherein, when the first feature information includes a target color distribution state, an average brightness value, and a target feature point distribution density, and the second feature information includes a target eye feature parameter, determining the target backlight adjustment parameter according to the first feature information and the second feature information includes: Determine the first eye feature parameter corresponding to the average brightness value according to a preset mapping relationship between the brightness value and the eye feature parameter; Determine a target first adjustment factor corresponding to the target color distribution state according to a preset mapping relationship between the color distribution state and the first adjustment factor; Determine the target second adjustment factor corresponding to the target feature point distribution density according to the preset mapping relationship between the feature point distribution density and the second adjustment factor; Adjusting the first eye feature parameter according to the target first adjustment factor and the target second adjustment factor to obtain a reference eye feature parameter; Determining a target deviation between the reference eye feature parameter and the target eye feature parameter; According to a preset mapping relationship between the deviation degree and the backlight adjustment parameter, the target backlight adjustment parameter corresponding to the target deviation degree is determined.
2. The method according to claim 1, characterized in that The determining the target backlight adjustment parameter according to the first feature information and the second feature information includes: Inputting the first feature information and the second feature information into a target neural network model to obtain a target operation result; According to a preset mapping relationship between the calculation result and the backlight adjustment parameter, the target backlight adjustment parameter corresponding to the target calculation result is determined.
3. The method according to claim 2, characterized in that Before determining the area image of the target focus area of the display screen by the user, the method further includes: Obtain a background sample feature set of the user's attention area; An eyeball sample feature set based on an eye area image corresponding to the user focus area; The background sample feature set and the eye sample feature set are input into a preset neural network model for training to obtain the target neural network model.
4. The method according to any one of claims 1 to 3, characterized in that: The step of adjusting the backlight according to the target backlight adjustment parameter includes: The backlight of the target area of interest is adjusted according to the target backlight adjustment parameter.
5. The method according to any one of claims 1 to 3, characterized in that: The determining of the regional image of the target focus area of the display screen by the user includes: determining a target gaze point on the display screen; Determining a fixation duration for the target fixation point; When the gaze duration is greater than a preset duration, an area within a preset range including the target gaze point is used as the target attention area, and a regional image corresponding to the target attention area is acquired.
6. The method according to claim 5, characterized in that The step of taking an area within a preset range including the target gaze point as the target focus area comprises: Acquiring target size information of a display area of the display screen; determining a target distance between the user's eyes and the display screen; According to a preset mapping relationship between the distance and the focus range, a reference focus range corresponding to the target distance is determined; Determine the target optimization coefficient corresponding to the target size information according to the mapping relationship between the preset size information and the optimization coefficient; Optimizing the reference attention range according to the target optimization coefficient to obtain a target attention range; The area of the target attention range centered on the target gaze point is used as the target attention area.
7. A backlight adjustment device, characterized in that: Applied to electronic equipment, the device comprises: a first determining unit, a first extracting unit, a second extracting unit, a second determining unit and an adjusting unit, wherein: The first determining unit is used to determine a region image of a target focus region of the display screen by the user; The first extraction unit is used to extract features from the regional image to obtain first feature information; The second extraction unit is used to obtain an eye region image of the user, and perform feature extraction on the eye region image to obtain second feature information; The second determining unit is used to determine a target backlight adjustment parameter according to the first characteristic information and the second characteristic information; The adjustment unit is used to adjust the backlight according to the target backlight adjustment parameter; Wherein, when the first feature information includes a target color distribution state, an average brightness value, and a target feature point distribution density, and the second feature information includes a target eye feature parameter, determining the target backlight adjustment parameter according to the first feature information and the second feature information includes: Determine the first eye feature parameter corresponding to the average brightness value according to a preset mapping relationship between the brightness value and the eye feature parameter; Determine a target first adjustment factor corresponding to the target color distribution state according to a preset mapping relationship between the color distribution state and the first adjustment factor; Determine the target second adjustment factor corresponding to the target feature point distribution density according to the preset mapping relationship between the feature point distribution density and the second adjustment factor; Adjusting the first eye feature parameter according to the target first adjustment factor and the target second adjustment factor to obtain a reference eye feature parameter; Determining a target deviation between the reference eye feature parameter and the target eye feature parameter; According to a preset mapping relationship between the deviation degree and the backlight adjustment parameter, the target backlight adjustment parameter corresponding to the target deviation degree is determined.
8. An electronic device, characterized in that: The method comprises a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for executing the steps in the method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 6.
