Information processing method and electronic equipment

Abstract

The embodiment of the invention provides an information processing method and electronic equipment, and relates to the technical field of intelligent interaction. And the energy consumption problem of the electronic equipment in an eye movement recognition scene is improved. According to the specific scheme, a first interface is displayed, wherein the first interface comprises first information; when the electronic equipment is located in the space with the ambient light brightness value larger than the first value, the target camera is controlled to collect a first image; the resolution of the first image is a first resolution; under the condition that the first image indicates that the fixation point of the user is located in the display area of the first information, displaying a second interface; or, when the electronic equipment is located in the space in which the ambient light brightness value is not greater than the second value, controlling the target camera to collect the second image; the second value is not greater than the first value, the resolution of the second image is a second resolution, and the second resolution is smaller than the first resolution; and displaying a second interface under the condition that the second image indicates that the fixation point of the user is located in the display area of the first information.

Description

Technical Field

[0001] The present application relates to the field of intelligent interaction technology, and in particular to an information processing method and electronic equipment. Background Art

[0002] Eye movement recognition technology tracks eye movement by detecting the position of the eye's gaze point or the movement of the eyeball relative to the head. At present, the application effect of eye movement recognition technology in the field of human-computer interaction is very good, which improves the human-computer interaction efficiency of electronic devices.

[0003] However, after enabling eye movement recognition technology, the energy consumption of electronic devices also increases significantly, which directly affects the battery life of electronic devices. Summary of the invention

[0004] In view of this, the present application provides an information processing method and an electronic device, which reduce the total energy consumption generated during eye movement recognition while ensuring the quality of eye movement recognition service.

[0005] In a first aspect, an embodiment of the present application provides an information processing method, which is applied to an electronic device, and includes: displaying a first interface, where the first interface includes first information.

[0006] When the electronic device is located in a space where the ambient light brightness value is greater than a first value, the target camera is controlled to capture a first image; the resolution of the first image is a first resolution, and the first image includes an eye image of the user; when the first image indicates that the user's gaze point is located in a display area of ​​the first information, a second interface is displayed, and the second interface is an application interface corresponding to the first information;

[0007] When the electronic device is located in a space where the ambient light brightness value is not greater than a second value, the target camera is controlled to capture a second image, the second value is not greater than the first value, the resolution of the second image is a second resolution, the second resolution is smaller than the first resolution, and the second image includes an eye image of the user; when the second image indicates that the user's gaze point is within the display area of ​​the first information, the second interface is displayed.

[0008] In the above embodiment, the electronic device uses eye movement recognition technology to recognize the user's interactive operation, and uses different image output resolutions in different ambient light scenes, which ensures the accuracy of interactive operation recognition and effectively reduces the energy consumption of the electronic device.

[0009] In some embodiments, before controlling the target camera to capture the first image, the method further includes: after the target camera is powered on, detecting a first ambient brightness value, the first ambient brightness value being greater than a first value, and writing first camera parameters including a first resolution into the target camera.

[0010] The first camera parameters are camera parameters that can control the target camera to output images according to the first resolution. In addition to the first resolution, the first camera parameters may also include other types of parameters.

[0011] In the above embodiment, the electronic device can distinguish whether the space where the electronic device is located is a space where the ambient light brightness value is greater than the first value by detecting the ambient light brightness value. After determining that the electronic device is located in a space where the ambient light brightness value is greater than the first value, the first camera parameter is configured for the target camera so that the target camera can output images according to the first resolution, ensuring that the collected image data can be used to calculate the eye movement data.

[0012] In some embodiments, before controlling the target camera to capture the second image, the method also includes: after the target camera is powered on, detecting a second ambient brightness value, the second ambient brightness value is not greater than the second value; and writing second camera parameters including a second resolution into the target camera.

[0013] In the above embodiment, the electronic device can distinguish whether the space where the electronic device is located is a space where the ambient light brightness value is not greater than the second value by detecting the ambient light brightness value. After determining that the electronic device is located in a space where the ambient light brightness value is not greater than the second value, the second camera parameter is configured for the target camera so that the target camera can output images according to the second resolution, thereby reducing the energy consumption of the electronic device.

[0014] In some embodiments, before controlling the target camera to capture the first image, the method also includes: after the target camera is powered on, writing second camera parameters carrying the second resolution into the target camera; controlling the target camera to capture the third image; and when the third image is an overexposed image, writing first camera parameters containing the first resolution into the target camera.

[0015] In the above embodiment, the electronic device can determine whether the electronic device is located in a space where the ambient light brightness value is greater than the first value through the collected image data (that is, the third image). For example, the image data is collected by default at the second resolution with lower energy consumption. If the collected image data is overexposed, the image is immediately collected at the first resolution to ensure the accuracy of eye movement recognition.

[0016] In some embodiments, before writing the first camera parameters including the first resolution into the target camera, the method further includes: determining that the number of overexposed pixels in the third image exceeds the first number, wherein the overexposed pixels are pixels having pixel values ​​greater than the first pixel threshold.

[0017] In some embodiments, before controlling the target camera to capture the second image, the method also includes: after the target camera is powered on, writing second camera parameters carrying the second resolution into the target camera; controlling the target camera to capture a fourth image; and determining that the overexposed pixels in the fourth image do not exceed a first number, wherein the overexposed pixels are pixels having pixel values ​​greater than a first pixel threshold.

[0018] In the above embodiment, the electronic device first outputs the image according to the default second resolution with lower energy consumption. If it is determined that the collected image data does not have the problem of overexposure, the second resolution is continued to be used for image collection, and the corresponding eye movement data is calculated, thereby reducing the energy consumption of the device while ensuring the accuracy of the calculated eye movement data.

[0019] In some embodiments, before controlling the target camera to capture the first image, the method also includes: after the target camera is powered on, detecting a second ambient brightness value, the second ambient brightness value is not greater than the second value; writing second camera parameters carrying the second resolution into the target camera; controlling the target camera to capture a fifth image, the resolution of the fifth image is the second resolution; when the fifth image is an overexposed image, writing first camera parameters containing the first resolution into the target camera.

[0020] In some embodiments, the target camera is a time-of-flight TOF camera, and controlling the target camera to capture a first image includes: when the transmitter of the target camera is turned on, controlling the target camera to capture a first image frame; when the transmitter of the target camera is turned off, controlling the target camera to capture a second image frame, and the first image includes a first image frame and a second image frame.

[0021] In some embodiments, the target camera is a time-of-flight TOF camera, and controlling the target camera to capture the second image includes: controlling the target camera to capture a third image frame when a transmitter of the target camera is turned on, and the second image includes the third image frame.

[0022] In the above embodiment, the same camera, that is, the TOF camera, can be used for image acquisition in different ambient light scenes. In bright light scenes, for example, in a space where the ambient light brightness value is greater than the first value, the image data output by the TOF camera is improved, and in the process of calculating the eye movement data, it is affected by the light. In dark light scenes, for example, in a space where the ambient light brightness value is not greater than the second value, the advantages of the TOF camera are brought into play, and images that are more suitable for calculating the eye movement data are acquired, while also reducing unnecessary energy consumption.

[0023] In some embodiments, the electronic device includes a secure memory, and after controlling the target camera to capture the first image, the method also includes: when the first image is captured, instructing the target camera to write the first image into the secure memory, and obtaining a corresponding first file descriptor; using the first file descriptor to search for the first image; after finding the first image, determining corresponding first eye movement data based on the first image, the first eye movement data including first position information and a corresponding first dwell time, the first position information indicating the position of the user's gaze point on the display screen; determining that the first position information is located within the display area of ​​the first information, and that the first dwell time reaches a preset duration.

[0024] In the above embodiments, the security of image data containing human eyes can be guaranteed, and the information security of users can be protected.

[0025] In some embodiments, the electronic device is located in a space where the ambient light brightness value is greater than a first value, including: the electronic device is located in a space with ambient light; the electronic device is located in a space where the ambient light brightness value is not greater than a second value, including: the electronic device is located in a space without ambient light.

[0026] In some embodiments, the first information includes a notification push message, a reminder capsule control, and a page turning control.

[0027] In a second aspect, an electronic device provided by an embodiment of the present application includes a display screen, one or more cameras, a memory and one or more processors; the one or more cameras include a TOF camera, and the display screen, the camera, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, the camera is used to collect image data, and the memory is used to store computer program code, the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the one or more processors are used to: display a first interface, the first interface includes first information; when the electronic device is located in a space where the ambient light brightness value is greater than the first value, control the target camera to collect a first image; the resolution of the first image is the first resolution, and the first image includes an eye image of the user; when the first image indicates that the user's gaze point is located in the display area of ​​the first information, display a second interface, and the second interface is an application interface corresponding to the first information; or, when the electronic device is located in a space where the ambient light brightness value is not greater than the second value, control the target camera to collect a second image; the second value is not greater than the first value, the resolution of the second image is the second resolution, the second resolution is less than the first resolution, and the second image includes an eye image of the user; when the second image indicates that the user's gaze point is located in the display area of ​​the first information, display the second interface.

[0028] In some embodiments, before controlling the target camera to capture the first image, the one or more processors are used to: after the target camera is powered on, detect a first ambient brightness value that is greater than a first value; and write first camera parameters including a first resolution into the target camera.

[0029] In some embodiments, before controlling the target camera to capture the second image, the one or more processors are used to: detect a second ambient brightness value after the target camera is powered on, and the second ambient brightness value is not greater than the second value; and write second camera parameters including a second resolution into the target camera.

[0030] In some embodiments, before controlling the target camera to capture the first image, the one or more processors are used to: write the second camera parameters carrying the second resolution into the target camera after the target camera is powered on; control the target camera to capture the third image; and when the third image is an overexposed image, write the first camera parameters containing the first resolution into the target camera.

[0031] In some embodiments, before writing the first camera parameters including the first resolution into the target camera, the one or more processors are used to: determine that the number of overexposed pixels in the third image exceeds the first number, wherein the overexposed pixels are pixels having pixel values ​​greater than the first pixel threshold.

[0032] In some embodiments, before controlling the target camera to capture the second image, the one or more processors are used to: write the second camera parameters carrying the second resolution into the target camera after the target camera is powered on; control the target camera to capture a fourth image; and determine that the overexposed pixels in the fourth image do not exceed the first number, wherein the overexposed pixels are pixels having pixel values ​​greater than the first pixel threshold.

[0033] In some embodiments, before controlling the target camera to capture the first image, the one or more processors are used to: detect a second ambient brightness value after the target camera is powered on, and the second ambient brightness value is not greater than the second value; write second camera parameters carrying the second resolution into the target camera; control the target camera to capture a fifth image, and the resolution of the fifth image is the second resolution; when the fifth image is an overexposed image, write first camera parameters containing the first resolution into the target camera.

[0034] In some embodiments, the target camera is a time-of-flight TOF camera, and the one or more processors are used to: control the target camera to capture a first image frame when the transmitter of the target camera is turned on; and control the target camera to capture a second image frame when the transmitter of the target camera is turned off, and the first image includes the first image frame and the second image frame.

[0035] In some embodiments, the target camera is a time-of-flight TOF camera, and the one or more processors are used to: control the target camera to capture a third image frame when the transmitter of the target camera is turned on, and the second image includes the third image frame.

[0036] In some embodiments, the electronic device includes a secure memory. After controlling the target camera to capture a first image, the one or more processors are used to: when the first image is captured, instruct the target camera to write the first image into the secure memory and obtain a corresponding first file descriptor; use the first file descriptor to search for the first image; after finding the first image, determine the corresponding first eye movement data based on the first image, the first eye movement data including first position information and a corresponding first dwell time, the first position information indicating the position of the user's gaze point on the display screen; determine that the first position information is located within the display area of ​​the first information, and that the first dwell time reaches a preset duration.

[0037] In some embodiments, the electronic device is located in a space where the ambient light brightness value is greater than a first value, including: the electronic device is located in a space with ambient light; the electronic device is located in a space where the ambient light brightness value is not greater than a second value, including: the electronic device is located in a space without ambient light.

[0038] In a third aspect, an embodiment of the present application provides a computer storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method in the above-mentioned first aspect and possible embodiments thereof.

[0039] In a fourth aspect, the present application provides a computer program product. When the computer program product is run on the above-mentioned electronic device, the electronic device executes the method in the above-mentioned first aspect and possible embodiments thereof.

[0040] It can be understood that the electronic devices, computer storage media and computer program products provided in the above-mentioned aspects are all applied to the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 An example diagram of a scenario of using an electronic device with one hand provided in an embodiment of the present application;

[0042] Figure 2A One of the exemplary diagrams of the working principle of the TOF camera provided in the embodiment of the present application;

[0043] Figure 2B The second example diagram of the working principle of the TOF camera provided in the embodiment of the present application;

[0044] Figure 3 An example diagram of the hardware architecture of an electronic device provided in an embodiment of the present application;

[0045] Figure 4 An example diagram of the software and hardware structure of an electronic device provided in an embodiment of the present application;

[0046] Figure 5 An example diagram of the interaction between various modules when the electronic device provided in an embodiment of the present application performs eye movement recognition;

[0047] Figure 6 is an example diagram of image data output by a TOF camera in some embodiments;

[0048] Fig. 7A One of the example scenes of eye movement recognition provided in the embodiment of the present application;

[0049] Figure 7B The second example diagram of the scene of eye movement recognition provided in the embodiment of the present application;

[0050] Figure 7C The third example diagram of the scene of eye movement recognition provided in the embodiment of the present application;

[0051] Figure 8 One of the flowcharts of the information processing method provided in the embodiment of the present application;

[0052] Fig. 9 Execute in different scenarios Figure 8 An example diagram of the information processing method shown;

[0053] Fig.10 One of the signaling interaction diagrams of the information processing method provided in the embodiment of the present application;

[0054] Fig.11 The second flowchart of the information processing method provided in the embodiment of the present application;

[0055] Fig.12 The second signaling interaction diagram of the information processing method provided in the embodiment of the present application;

[0056] Fig.13 The third signaling interaction diagram of the information processing method provided in the embodiment of the present application. DETAILED DESCRIPTION

[0057] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0058] In order to make the following embodiments clear and concise, a brief introduction to related concepts or technologies is first given:

[0059] Eye movement recognition refers to the tracking of eye movements by detecting the position of the eye's gaze point or the movement of the eyeball relative to the head. At present, the application effect of eye movement recognition in the field of human-computer interaction is very good. Eye movement recognition can effectively simplify the difficulty of operation in the process of human-computer interaction. For example, Figure 1 As shown, when the user gazes at the display area a of the electronic device, the electronic device can detect that the user's gaze point is located in the display area a through eye movement recognition. In this scenario, it can be determined that the user clicks the display area a.

[0060] In some embodiments, the electronic device needs to collect one or more frames of continuous image data containing the user's eyes through a time of flight (TOF) camera and / or an RGB camera. Based on the above image data, eye movement data can be calculated, and the eye movement data can include the user's eye gaze point, gaze duration, and eye movement trajectory. The electronic device identifies the user's eye movement interaction operation based on the above eye movement data.

[0061] The TOF camera may include a transmitter (TX) and a receiver (RX), wherein TX is used to transmit infrared light or laser pulses, and RX is used to receive reflected light and form an image. After TX emits infrared light or laser pulses and RX receives the corresponding reflected light, the TOF camera can use TOF imaging technology to form an image. When TX works normally, the image data collected by RX can be called a TX-ON frame.

[0062] TOF imaging technology refers to the imaging technology in which a TOF camera emits a group of infrared light (or laser pulses) that are invisible to the human eye, which are reflected after encountering an object and end when reflected to the camera. The time difference or phase difference from emission to reflection back to the camera is calculated, and the data is collected to form a set of distance and depth data, thereby obtaining a three-dimensional 3D model. In other words, TOF imaging technology adds depth information from the Z axis direction on the basis of traditional 2D XY axis imaging, and finally generates 3D image information.

[0063] like Figure 2AAs shown, when TX works normally, the image data collected by RX can be called TX-ON frame.

[0064] In addition, in some embodiments, the TOF camera can also turn off TX. After turning off TX, Figure 2B As shown, TX no longer emits light waves. When TX does not emit light waves, the RX of the TOF camera can receive ambient reflected light and perform imaging based on the received ambient reflected light. After turning off TX, the image data collected by RX can also be called a TX-OFF frame. The above TX-OFF frame is usually a "black image" without a clear face image.

[0065] An RGB camera, that is, a red (R), green (G), and blue (B) three-color camera, can obtain images consisting of multiple colors by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other.

[0066] In some embodiments, since image data containing eyes involves the user's privacy and security information, the image data collected by the TOF camera and RGB camera and used to calculate eye movement data needs to be isolated and protected using the secure memory in the rich execution environment (REE) + trusted execution environment (TEE) architecture.

[0067] The above-mentioned rich execution environment (REE), which can also be called a rich execution environment, a common execution environment, or an untrusted execution environment, refers to the system operating environment of an electronic device, and the above-mentioned system can refer to an operating system. REE has good openness and extensibility.

[0068] The above-mentioned trusted execution environment (TEE), also known as the security side or security zone, is an area that requires authorization to access. TEE refers to an independent security area established inside the processor through hardware resource isolation. TEE and REE coexist in the operating environment of electronic devices, but TEE and REE are completely isolated in hardware. TEE has its own operating space and is equipped with strict protection measures. Therefore, it has a higher security level than REE, and REE cannot access the relevant resources of TEE, so that TEE has the security ability to protect the internal code and information of TEE, and can resist attacks on the REE side and resist security threats.

[0069] REE+TEE architecture refers to an architecture that provides services for applications through the combination of TEE and REE. In other words, TEE and REE exist together in electronic devices. For example, TEE can implement an operating mechanism isolated from REE through hardware support. TEE has its own operating space, with a higher security level than REE, and can protect assets in TEE (such as data, software, etc.) from software attacks. Only authorized security software can be executed in TEE, and it also protects the confidentiality of the resources and data of the security software. Compared with REE, TEE can better protect the security of data and resources due to its protection mechanisms such as isolation and permission control.

[0070] A trusted application (TA) is an application running in TEE that can provide security services to client applications (CA) running in REE, such as key generation and key management, security authentication, intelligent perception and other security tasks.

[0071] CA usually refers to an application running in REE, which can call TA through the client application programming interface (API) and instruct TA to perform corresponding security operations.

[0072] Figure 3 A schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application.

[0073] like Figure 3 As shown, the electronic device (such as a mobile phone) may include: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a SIM card interface 295, etc.

[0074] Among them, the above-mentioned sensor module 280 may include sensors such as pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor and bone conduction sensor.

[0075] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0076] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0077] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0078] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0079] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0080] It is understandable that the interface connection relationship between the modules illustrated in this embodiment is only a schematic illustration and does not constitute a structural limitation of the electronic device. In other embodiments, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0081] The charging management module 240 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. While the charging management module 240 charges the battery 242, it can also power the electronic device through the power management module 241.

[0082] The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the external memory, the display screen 294, the camera 293, and the wireless communication module 260. In some embodiments, the power management module 241 and the charging management module 240 can also be set in the same device.

[0083] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor, etc. In some embodiments, the antenna 1 of the electronic device is coupled with the mobile communication module 250, and the antenna 2 is coupled with the wireless communication module 260, so that the electronic device can communicate with the network and other devices through wireless communication technology, such as communication with wearable devices.

[0084] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0085] The mobile communication module 250 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to electronic devices. The mobile communication module 250 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation.

[0086] The mobile communication module 250 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some functional modules of the mobile communication module 250 can be set in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 250 can be set in the same device as at least some modules of the processor 210.

[0087] The wireless communication module 260 can provide wireless communication solutions for electronic devices including WLAN (such as (wirelessfidelity, Wi-Fi) network), Bluetooth (bluetooth, BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), nearfield communication technology (nearfield communication, NFC), infrared technology (infrared, IR), etc.

[0088] Among them, GNSS may include Beidou navigation satellite system (BDS), global positioning system (GPS), global navigation satellite system (GLONASS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation systems (SBAS).

[0089] The wireless communication module 260 may be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 210. The wireless communication module 260 may also receive signals to be sent from the processor 210, modulate the frequencies of the signals, amplify the signals, and convert the signals into electromagnetic waves for radiation via the antenna 2.

[0090] The electronic device implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0091] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel.

[0092] The electronic device can realize the shooting function through the ISP, the camera 293, the video codec, the GPU, the display screen 294 and the application processor, etc. The ISP is used to process the data fed back by the camera 293.

[0093] The camera 293 is used to capture static images or videos. In some embodiments, the electronic device may include 1 or N cameras 293, where N is a positive integer greater than 1. The N cameras 293 may include a TOF camera and an RGB camera.

[0094] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.

[0095] The internal memory 221 may be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 221. For example, in an embodiment of the present application, the processor 210 may execute the instructions stored in the internal memory 221, and the internal memory 221 may include a program storage area and a data storage area.

[0096] The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0097] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0098] The software system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes a system with a layered architecture as an example to exemplify the software structure of the electronic device.

[0099] Figure 4 It is a block diagram of the software and hardware structure of the electronic device according to an embodiment of the present invention.

[0100] The layered architecture of electronic devices divides software and hardware into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the electronic device may include, for example, an application layer, an application framework layer, a hardware abstraction layer (HAL), a kernel layer, and a hardware layer. Of course, it may also include Figure 4 Layers not shown in the diagram, such as the runtime and system libraries.

[0101] Exemplarily, the application layer may include a series of applications.

[0102] like Figure 4 As shown, the application layer may include applications such as notifications, settings, and system user interfaces (UI). In some embodiments, the application layer may also include Figure 4 Applications not shown, such as capsule reminder applications, call, camera, map, navigation and other applications.

[0103] Among them, the settings application can turn on or off the smart perception function. After detecting that the user has turned on the smart perception function in the settings application, the electronic device can capture images and identify the user's eye movement data from the captured images, so as to achieve eye movement interaction based on the user's eye movement data. Optionally, in response to the user turning on the smart perception function in the settings application, the electronic device can display a guidance interface for eye movement calibration. The guidance interface may include guidance text information and guidance images or symbols for guiding the user to complete the eye movement calibration process, so as to guide the user to complete the specified eye movement according to the guidance text information and the guidance image or symbol guidance interface. After that, the electronic device can complete the eye movement calibration process according to the specified eye movement completed by the user.

[0104] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0105] like Figure 4 As shown, the application framework layer may include intelligent perception services and camera services, etc.

[0106] The intelligent perception service is used to process the eye movement recognition requests sent by the application and return the processing results to the application to implement the eye movement recognition function.

[0107] The camera service is used to manage the work of the camera. For example, it can control the camera to be turned on and off, and control the camera shooting parameters such as resolution and frame rate.

[0108] In other embodiments, the application framework layer may also include Figure 4 Modules not shown in the figure, such as window manager, content provider, view system, resource manager, notification manager, etc.

[0109] The hardware abstraction layer (HAL) can encapsulate the drivers in the kernel layer and provide a calling interface to the application framework layer, shielding the implementation details of the low-level hardware.

[0110] like Figure 4 As shown, the HAL of the electronic device includes at least a camera HAL and an intelligent perception control module.

[0111] The camera HAL is the core software framework of the camera, which can include sensor nodes and image front end (IFE) nodes. The sensor nodes and IFE nodes are components (nodes) in the transmission path (also called transmission pipeline) of image data and control instructions created by the camera HAL.

[0112] The intelligent perception control module is the core software application of intelligent perception. The intelligent perception control module includes the intelligent perception client application (CA), and the intelligent perception CA runs in the REE environment.

[0113] A smart perception trusted application (TA) is an application for smart perception that runs in a TEE environment.

[0114] In addition, the smart perception control module may also include: a camera control module and a smart perception HAL. Among them, the camera control module can be used to control the camera service to perform specified types of tasks, such as instructing the camera service to call the camera HAL to create a transmission channel, etc. The above-mentioned smart perception HAL can call smart perception-related algorithm modules, such as image processing algorithms, eye movement recognition algorithms, eye movement calibration algorithms, face detection algorithms, etc. Among them, the above-mentioned image processing algorithm can be used to convert the image data collected by the TOF camera into a TOF infrared (infrared radiation, IR) image, the above-mentioned eye movement recognition algorithm can calculate the corresponding eye movement data based on the image data, the above-mentioned eye movement calibration algorithm is used to start the eye movement calibration process, and the above-mentioned face detection algorithm can detect whether a face appears in the image data, etc.

[0115] like Figure 4 As shown in Figure 1, the kernel layer is the layer between hardware and software. The kernel layer contains at least the camera driver. In addition, the kernel layer can also include Figure 4 Drivers not shown, such as display driver, sensor driver, etc.

[0116] Among them, the camera driver is the driver layer of the Camera device, which is mainly responsible for the interaction with the camera hardware.

[0117] like Figure 4 As shown, the hardware layer includes a display, a TOF camera, an RGB camera, an IFE module, an IPE module, an IFE-Lite module, a secure buffer, etc.

[0118] Among them, the image front end (IFE) module can perform image processing such as automatic exposure algorithm (AE) and automatic white balance (AWB) on the image data.

[0119] The lightweight image pre-processing (IFE-Lite) module is a simplified version of the IFE module and can be used to forward image data without processing the image data during the forwarding process.

[0120] The image processing engine (IPE) module is used to perform hardware noise reduction, cropping, noise reduction, color processing, detail enhancement and other image processing tasks on the image. Among them, hardware noise reduction includes, for example, multi-frame noise reduction (MFNR) and multi-frame super resolution (MFSR).

[0121] Secure memory refers to a memory with security protection function, which can be used to store image data collected by TOF camera and RGB camera. In some embodiments, the image acquired by TOF camera is forwarded by IFE-Lite module and stored in secure memory, and the image acquired by RGB camera can be passed to IPE module after image pre-processing by IFE module, and stored in secure memory by IPE module. The above secure memory can shield the software module on REE side, so that the software module on REE side cannot directly access the data in secure memory, thereby improving the security of data.

[0122] Combine the following Figure 5 , the interaction between the software modules involved in the method provided in the embodiment of the present application is described.

[0123] like Figure 5 As shown, in an electronic device with REE+TEE architecture, applications (such as setting applications, notification applications, capsule prompt applications), system UI, smart perception services, smart perception control modules, camera services, camera HAL, camera driver, TOF camera, RGB camera, IFE-Lite, IFE module, and IPE module are located on the REE side, and smart perception TA and secure memory are located on the TEE side.

[0124] In the application layer, applications (such as settings applications, notification applications, and capsule prompt applications) can call the system UI according to business logic. The above-mentioned system UI can interact with the intelligent perception service in the application framework layer in response to the call of the application. For example, after the settings application detects the user's instruction to turn on the eye movement recognition function, it can call the system UI to instruct the system UI to interact with the intelligent perception service and turn on the eye movement recognition service. For another example, when the notification application receives a message to be pushed, it can also call the system UI to instruct the system UI to interact with the intelligent perception service and turn on the eye movement recognition service. For another example, when the message capsule provided by the capsule prompt application is displayed, or when the information displayed by the message capsule is updated, the system UI can also be called to instruct the system UI to interact with the intelligent perception service and turn on the eye movement recognition service.

[0125] like Figure 5 As shown, the intelligent perception service in the application framework layer can interact with the intelligent perception control module of the HAL layer in response to the call of the system UI.

[0126] The above-mentioned intelligent perception control module can control the power-on of the specified camera (for example, RGB camera and TOF camera) and enable the preset working mode (for example, eye tracking acquisition mode) through the camera service. As an implementation method, the intelligent perception control module can interact with the camera service by calling the interface (vendor native development kit, VNDK).

[0127] like Figure 5 As shown, the camera service can interact with the camera HAL in the HAL layer in response to the instructions of the intelligent perception control module. The camera HAL may include sensor nodes and IFE nodes. It is understandable that the sensor nodes and IFE nodes in the transmission paths corresponding to different cameras may be different. For the same camera, when different working modes are used (for example, different output resolutions are used), the sensor nodes and IFE nodes of the corresponding transmission paths are also different. The above transmission path can be used to write camera parameters to the corresponding camera.

[0128] In addition, the sensor node and the IFE node can interact with the camera driver in the kernel layer. For example, the sensor node can determine the camera parameters required when the corresponding camera (such as an RGB camera or a TOF camera) enables a preset working mode, and write the determined camera parameters to the corresponding camera through the camera driver, so that the camera collects image data according to the preset working mode (for example, eye tracking acquisition mode).

[0129] The IPE module can forward the image data (e.g., image data a) collected by the RGB camera to the secure memory for storage. The IFE-Lite module can forward the image data (e.g., image data b) collected by the TOF camera to the secure memory for storage.

[0130] The camera driver can also be used to obtain the file descriptor (FD) of the image data collected by the TOF camera from the IFE-Lite module, such as Figure 5 The above camera driver can also obtain the FD of the image data collected by the RGB camera from the IPE module, such as Figure 5 FD2 in . FD1 is used to indicate the storage location of the image data collected by the TOF camera in the secure memory. FD2 is used to indicate the storage location of the image data collected by the RGB camera in the secure memory.

[0131] Afterwards, the camera driver sends FD1 (or FD2) to the camera HAL, which then sends it to the camera service. The camera service can send FD1 (or FD2) to the smart perception control module. In response to FD1 (or FD2), the smart perception control module can send a query instruction carrying FD1 (or FD2) to the smart perception TA on the TEE side. In response to the query instruction, the smart perception TA can read the image data in the storage location indicated by FD1 (or FD2) from the secure memory, such as image data a (or image data b). After reading the image data (image data a or image data b), the smart perception TA can also encrypt the image data (image data a or image data b), for example, encrypting it based on the encryption module in the smart perception TA.

[0132] Afterwards, the smart perception TA sends the encrypted image data (image data a or image data b) to the smart perception control module on the REE side. Among them, the encryption key can also be transmitted between the smart perception TA and the smart perception control module. The smart perception control module can decrypt the encrypted image data with the key, and calculate the corresponding eye movement data based on the decrypted image data (image data a or image data b). The smart perception control module sends the eye movement data to the smart perception service, and the smart perception service sends the eye movement data to the system UI, which is passed to the application subscribed to the eye movement recognition service by the system UI. In this way, the above application determines the user's gaze point and gaze action based on the eye movement data, and performs operations on the interactive object corresponding to the gaze point according to the user's gaze action.

[0133] It should be noted that when the electronic device turns on dual streams for eye movement recognition, such as turning on the TOF camera and the RGB camera at the same time for eye movement recognition, the FDs sent back by the camera driver to the intelligent perception control module include FD1 and FD2. In addition, the image data read by the intelligent perception TA also includes image data a and image data b.

[0134] When the electronic device starts a single stream for eye movement recognition, for example, when the TOF camera is turned on for eye movement recognition, the FD sent back by the camera driver to the smart perception control module is FD1, and the image data read by the smart perception TA is image data b. For example, when the RGB camera is turned on for eye movement recognition, the FD sent back by the camera driver to the smart perception control module is FD2, and the image data read by the smart perception TA also includes image data a.

[0135] In addition, the solid arrows in the figure can be used to represent control flow, and the dashed arrows can be used to represent data flow.

[0136] Optionally, after the smart perception TA on the TEE side reads the image data from the secure memory according to the FD, it can also perform eye movement recognition on the read image data to obtain the user's eye movement data. Afterwards, the smart perception TA sends the eye movement data to the smart perception control module, and the smart perception control module sends the eye movement data to the smart perception service. The smart perception service sends the eye movement data to the system UI, which is passed to the application subscribed to the eye movement recognition service by the system UI. In this way, the application subscribed to the eye movement recognition service can determine the user's gaze point and gaze action based on the eye movement data, and perform operations on the interactive object corresponding to the gaze point according to the user's gaze action, which can ensure the security of eye movement recognition.

[0137] In some embodiments, Figure 6 As shown, the image data b output by the TOF camera consists of a TX-OFF frame and a TX-ON frame, wherein the TX-ON frame is used to calculate the eye movement data, and the TX-OFF frame is used to eliminate the influence of overexposed (overexposed) pixels in the TX-ON frame, thereby improving the success rate and accuracy of calculating the eye movement data. However, the resolution of a TX-OFF frame is 1phase (640*483), and the resolution of a TX-ON frame is 1phase (640*483). The image data b output by the TOF camera includes a TX-OFF frame and a TX-ON frame, and its output resolution is 2phase (640*966). Obviously, the TOF camera acquires images at the output resolution of 2phase for a long time.

[0138] In order to improve the above problems, an embodiment of the present application provides an information processing method, which uses eye movement recognition to process interactive objects (such as information) displayed by an electronic device. By dynamically adjusting the output resolution of the TOF camera, the power consumption of the electronic device is reduced while ensuring the accuracy of eye movement recognition.

[0139] The following, in conjunction with the accompanying drawings, introduces the information processing method provided in the embodiment of the present application from multiple dimensions such as application scenarios and implementation principles.

[0140] One of the exemplary application scenarios: When the eye movement recognition function is turned on in an electronic device, if the notification application has permission to enable the eye movement recognition function, when the notification application pushes an application message of any application, it can collect image data containing the user's eyes and calculate the corresponding eye movement data based on the image data. Based on the eye movement data, it can be determined whether the user has indicated to view the application message, that is, the user's contactless operation can be identified. After determining that the user has indicated to view the application message based on the user's eye movement data, the electronic device can jump to display the application interface corresponding to the application message.

[0141] like Fig. 7A As shown, when the SMS application of the electronic device receives a SMS message, such as a verification code SMS message, the notification application of the electronic device can display a push banner 701 of the SMS message. During the display of the push banner 701, the notification application can call the smart perception service through the system UI to calculate the eye movement data indicating the user's current gaze point, such as the SMS application subscribing to the eye movement recognition service through the system UI.

[0142] For example, when the user is gazing at the display area of ​​the push banner 701, the electronic device can calculate the corresponding eye movement data based on the image data containing the user's eyes collected by the TOF camera and the RGB camera, and the eye movement data can indicate that the user's gaze point is located in the display area of ​​the push banner 701. Then, in response to the eye movement data indicating that the user's gaze point is located in the display area of ​​the push banner 701, the electronic device can display the information interface 702 provided by the SMS application, and the information interface 702 includes the complete content of the SMS message displayed on the push banner 701, that is, the complete content of the verification code SMS message.

[0143] Exemplary application scenario 2: When the eye movement recognition function is turned on in an electronic device, if the capsule prompt application has permission to enable the eye movement recognition function, the capsule prompt application can also use the eye movement recognition function to identify the user's contactless operation.

[0144] Among them, the reminder capsule of the display capsule reminder application can be used to display application messages from other applications in the electronic device, and can also provide an activation entrance for other applications.

[0145] like Figure 7B As shown, when the capsule reminder application of the electronic device receives an application message from the taxi-hailing application, such as an order acceptance message indicating that the reserved vehicle has been accepted, the electronic device can display a brief order acceptance message on the reminder capsule 703.

[0146] While content is displayed on the reminder capsule 703, the capsule reminder application can call the smart perception service through the system UI to calculate eye movement data indicating the user's current gaze point, such as the capsule reminder application subscribing to the eye movement recognition service through the system UI.

[0147] For example, when the user is looking at the display area of ​​the reminder capsule 703, the electronic device can calculate the corresponding eye movement data based on the image data containing the user's eyes collected by the TOF camera and the RGB camera, and the eye movement data can indicate that the user's gaze point is located in the display area of ​​the reminder capsule 703. Then, in response to the eye movement data, the electronic device can display the reminder window 704 and hide the reminder capsule 703. The reminder window 704 contains the detailed content of the order acceptance message.

[0148] Exemplary application scenario three: When the eye movement recognition function is turned on in an electronic device, if the novel application has permission to enable the eye movement recognition function, the novel application can also use the eye movement recognition function to achieve contactless page turning.

[0149] like Figure 7C As shown, while the electronic device is displaying the application interface of the novel application, such as application interface 705, the novel application can call the intelligent perception service through the system UI to calculate eye movement data indicating the user's current gaze point, such as the novel application subscribing to the eye movement recognition service through the system UI.

[0150] For example, when the user gazes at the display area of ​​the page turning control 706 in the application interface 705, the electronic device can calculate the corresponding eye movement data based on the image data containing the user's eyes collected by the TOF camera and the RGB camera, and the eye movement data can indicate that the user's gaze point is located in the display area of ​​the page turning control 706. Then, in response to the eye movement data, the electronic device can display the application interface 707, and the application interface 707 and the application interface 705 are both interfaces for displaying novel content in the novel application, and the application interface 707 can be the next page of the application interface 705.

[0151] Other possible application scenarios: When the eye movement recognition function is turned on in an electronic device, if the short video application has the permission to enable the eye movement recognition function, the short video application can also use the eye movement recognition function to achieve contactless video switching. For example, if it is detected that the user's gaze point is located in the application interface provided by the short video application, in the display area for switching videos, and the stay time exceeds the preset duration, the video can be switched. If the news application has the permission to enable the eye movement recognition function, the news application can also use the eye movement recognition function to achieve contactless instructions to view different news content.

[0152] In an embodiment of the present application, an information processing method is provided. After the electronic device applies the information processing method, in any of the aforementioned scenarios, the power consumption of the TOF camera of the electronic device in a bright environment is higher than that of the TOF camera in a dark environment. In addition, the image output resolution of the TOF camera in a bright environment is higher than that of the TOF camera in a dark environment.

[0153] In some embodiments, Figure 8 As shown, the above information processing method may include the following steps:

[0154] S101, in response to satisfying a preset condition, obtaining an ambient light brightness value.

[0155] For example, the above preset condition may indicate a scenario in which the eye movement recognition process needs to be executed, and the manner of satisfying the preset condition may include any one of the following:

[0156] 1) After the electronic device turns on the eye movement recognition function, it will detect the user's gaze point position and gaze duration in real time. In this scenario, the eye movement recognition function is turned on and it can be determined that the preset conditions are met.

[0157] Correspondingly, after turning on the eye movement recognition function, the electronic device can periodically obtain the ambient light brightness value in the space where it is located.

[0158] 2) After the electronic device turns on the eye movement recognition function, when it detects that the user's face is facing the display screen of the electronic device, it can trigger the detection of the user's gaze point position and gaze duration. In this scenario, when it is detected that the user's face is facing the display screen of the electronic device, it can be determined that the preset condition is met.

[0159] Correspondingly, after the eye movement recognition function is turned on, the electronic device detects the brightness value of the ambient light in the space in response to detecting that the face of the person is facing the display screen of the electronic device.

[0160] 3) When the electronic device displays a specified type of display content, trigger detection of the user's gaze point position and gaze duration. The specified type of display content may be display content provided by an application with permission to use eye movement recognition. In addition, the specified type of display content is interactive, that is, the specified type of display content can respond to user operations and execute corresponding business logic. For example, the specified type of display content may be Fig. 7A Push banner 701, Figure 7B A reminder capsule 703 having display content and / or Figure 7C The page turning control 706, etc.

[0161] In this scenario, when the electronic device displays a specified type of display content, it can be determined that the preset condition is met.

[0162] Correspondingly, after the eye movement recognition function is turned on, the electronic device can detect the brightness value of the ambient light in the space in response to displaying a specified type of display content.

[0163] 4) After the eye movement recognition function is turned on in the electronic device, when the ambient light sensor of the electronic device detects that the change in the brightness of the ambient light is greater than a preset threshold, it is determined that the preset condition is met.

[0164] Correspondingly, when the ambient light brightness detected by the ambient light sensor varies greatly (greater than a preset threshold), the detected ambient light brightness value may be reported to various software and hardware modules that register to obtain the ambient light brightness value.

[0165] S102: When the ambient light brightness indicates that the environment is currently bright, control the target camera to acquire first image data with a first resolution.

[0166] The bright environment mentioned above refers to an environment brightness scene where overexposure is likely to occur when collecting image data. The dark environment, which is opposite to the bright environment, refers to an environment brightness scene where overexposure is not likely to occur when collecting image data.

[0167] In some embodiments, the electronic device may determine whether the environment in which it is located is a bright environment based on the ambient light brightness value.

[0168] Exemplarily, a brightness threshold (e.g., 50,000 lux) can be preconfigured in the electronic device. When the ambient light brightness value is greater than the above brightness threshold, it is judged that the current electronic device is in a bright environment. When the ambient light brightness value is not greater than the above brightness threshold, it is judged that the current electronic device is in a dark environment.

[0169] As another example, multiple brightness thresholds may be configured in the electronic device, such as brightness threshold 1 and brightness threshold 2, wherein brightness threshold 1 is greater than brightness threshold 2. When the ambient light brightness value is greater than brightness threshold 1, it is determined that the current electronic device is in a bright environment. When the ambient light brightness value is less than brightness threshold 2, it is determined that the current electronic device is in a dark environment. When the ambient light brightness value is between brightness threshold 1 and brightness threshold 2, if the electronic device was previously determined to be in a dark environment, then it is determined that the electronic device is still in a dark environment this time. If the electronic device was previously determined to be in a bright environment, then it is determined that the electronic device is still in a bright environment this time.

[0170] In some embodiments, the target camera may be a camera that needs to participate in eye movement recognition. The image data collected by the target camera may be used to calculate eye movement data. For example, the target camera may be a TOF camera in an electronic device. For another example, the target camera may also be an RGB camera that needs to participate in eye movement recognition. The electronic device may select a TOF camera or an RGB camera as the target camera according to pre-configured rules. Exemplarily, the pre-configured rules may be to decide which target camera to enable based on the ambient light brightness. The pre-configured rules may also be to decide which target camera to enable based on whether the user wears glasses and the ambient light brightness. In the embodiments of the present application, the pre-configured rules are not specifically limited.

[0171] When the target camera is a TOF camera, the first resolution may be 2phase. In this scenario, the TOF camera needs to collect a TX-OFF frame and a TX-ON frame respectively, and then output the first image data (also referred to as a TOF RAW image) composed of the TX-OFF frame and the TX-ON frame. After that, the process enters S104.

[0172] S103: When the ambient light brightness indicates that the environment is currently dark, control the target camera to acquire first image data using a second resolution.

[0173] When the target camera is a TOF camera, the second resolution may be 1phase. In this scenario, the TOF camera only needs to collect one TX-ON frame each time, so that the output first image data only includes the TX-ON frame. Then, the process enters S104.

[0174] The above embodiment takes the target camera as a TOF camera as an example to describe an implementation method in which the electronic device dynamically adjusts the image output resolution of the TOF camera according to the ambient light brightness value. When the target camera is an RGB camera, the image output resolution of the RGB camera can also be dynamically adjusted according to the ambient light brightness value.

[0175] S104: After obtaining the first image data, calculate corresponding eye movement data.

[0176] In some embodiments, the electronic device can calculate the corresponding eye movement data based on the first image data collected by the TOF camera and the RGB image collected by the RGB camera.

[0177] As an implementation method, the electronic device can process the first image data (TOF RAW image) to obtain a corresponding TOF IR image. The process of processing the first image data to obtain the TOF IR image can refer to related technologies and will not be described in detail here.

[0178] The electronic device can perform face detection on the TOF IR image (or the RGB image output by the RGB camera) and determine the face area rectangular frame and the face key point array.

[0179] According to the determined face area rectangular frame and face key point array, combined with the preconfigured eye movement recognition algorithm, the gaze point position coordinates are determined. The gaze point position coordinates can be determined with a specified position point in the display screen (which can be the upper left corner of the display screen) as the reference origin. The implementation principle of the above eye movement recognition algorithm can refer to related technologies and will not be described here. After calculating the eye movement data, it can be determined whether the user has made a contactless instruction, such as whether to instruct to check the message, whether to instruct to check the reminder capsule, and whether to instruct to turn the page.

[0180] In some embodiments, after the electronic device turns on the eye movement recognition function, it can collect image data and calculate eye movement data multiple times during operation to provide eye movement recognition services to users in a timely manner.

[0181] like Fig. 9 As shown, when the target camera is a TOF camera, after determining for the Mth time that the preset conditions are met, an ambient light brightness value of 1 is detected, and the ambient light brightness value 1 indicates that the current environment is bright. In this scenario, the TOF camera can be configured with camera parameters including the first resolution. After the TOF camera is configured with camera parameters including the first resolution, each time the TOF camera acquires image data, TX can be turned on to acquire a frame of Tx-ON frames, and TX can be turned off to acquire a frame of TX-OFF frames, and the first image data including TX-ON frames and TX-OFF frames can be actually output. The TX-ON frame and TX-OFF frame in the first image data can be two frames of data continuously acquired by the TOF camera, and the order during acquisition is not specifically limited. When the resolution of the TX-ON frame and the TX-OFF frame is 1phase, the resolution of the first image data is 2phase.

[0182] Understandably, in bright environments, the TX-ON frames collected by the TOF camera are likely to have overexposure problems. In this scenario, an accurate TOF IR map cannot be calculated based on the TX-ON frame alone, which in turn affects the accuracy of the subsequently calculated eye movement data. In this scenario, before calculating the TOF IR map, it is necessary to use the TX-OFF frame corresponding to the TX-ON frame (continuous with the acquisition time of the TX-ON frame) to eliminate the overexposure problem in the TX-ON frame.

[0183] After the preset condition is determined to be met for the Nth time, an ambient light brightness value of 2 is detected, indicating that the current environment is dark. In this scenario, the camera parameters including the second resolution can be configured in the TOF camera. After the TOF camera is configured with the camera parameters including the second resolution, each time the TOF camera collects image data, TX can be turned on to collect a TX-ON frame, and the first image data actually output is the TX-ON frame. The resolution of the first image data is 1phase.

[0184] Understandably, in dark environments, the probability of overexposure in the TX-ON frames collected by the TOF camera is low. In this scenario, accurate TOF IR images can be calculated based on the TX-ON frames alone, thereby ensuring the accuracy of the eye movement data calculated later. In dark environments, the process of collecting TX-OFF frames can be reduced, reducing the energy consumption of the TOF camera.

[0185] The above M and N are two different positive integers, such as Fig. 9 As shown in the figure, the image resolution of the TOF camera is different in a bright environment and a dark environment, and the power consumption generated when actually executing the first image data acquisition is also different. For example, in a bright environment, two frames of data are collected and combined into the first image data, and in a dark environment, only one frame of data is collected as the first image data. In this way, while ensuring the accuracy of the eye movement data, the energy consumption of the electronic device to realize eye movement recognition is reduced.

[0186] As a way to achieve this, Fig.10 As shown, in the process of implementing the above method, the signaling interaction steps between the various software and hardware modules in the electronic device are as follows:

[0187] S201, the application sends an eye movement recognition registration request to the intelligent perception service.

[0188] In some embodiments, when the preset conditions are met, the application can send an eye movement recognition registration request to the smart perception service through the system UI. The above eye movement recognition registration request can be used to register a callback corresponding to eye movement recognition on the smart perception service so that the application can obtain the eye movement recognition result. The above preset conditions can be referred to the introduction in the previous embodiment, and are briefly described here:

[0189] Taking displaying a specified type of display content for an electronic device in a manner that meets preset conditions as an example, when the specified type of display content includes a push banner from a notification application, after the notification application instructs the display of the push banner, the system UI can also be called to instruct the system UI to send an eye movement recognition registration request to the smart perception service to start the eye movement recognition process.

[0190] Taking the example of enabling the eye movement recognition function for an electronic device in a manner that meets preset conditions, the setting application can periodically call the system UI, instructing the system UI to send an eye movement recognition registration request to the smart perception service, and start the eye movement recognition process. In this way, the process enters S202.

[0191] S202: The smart perception service sends a first request to the smart perception control module in response to the eye movement recognition registration request.

[0192] In some embodiments, the smart perception service may respond to the eye movement recognition registration request and may instruct the subsequent process of executing the eye movement recognition. In addition, after the smart perception service obtains the eye movement recognition result (e.g., eye movement data), the eye movement recognition result may be fed back to the corresponding application through the system UI through the above registered callback.

[0193] The first request is a request instruction to start eye movement recognition. In some embodiments, the intelligent perception control module can respond to the first request, call the camera service, related algorithm modules, etc., collect image data, and calculate the corresponding eye movement data based on the image data to realize eye movement recognition. For implementation details, please refer to S203 to S225:

[0194] S203: The intelligent perception control module determines the target camera in response to the first request.

[0195] It is understandable that before the intelligent perception control module performs eye movement recognition, it is necessary to determine the camera used to collect image data, that is, the target camera. Usually, the target camera can be the front camera of the electronic device, and the lens of the front camera is located on the same side as the display screen of the electronic device, and can include a TOF camera and an RGB camera.

[0196] Exemplarily, the intelligent perception control module may determine the TOF camera in the front camera as the target camera. In another exemplary embodiment, the intelligent perception control module may also determine the RGB camera in the front camera as the target camera. In another exemplary embodiment, the intelligent perception control module may also determine the RGB camera and the TOF camera in the front camera as the target camera at the same time.

[0197] In some embodiments, the rule for selecting the target camera (the pre-configured rule mentioned in the above embodiment) can be to combine the ambient light brightness to determine the target camera. The specific implementation details can be referred to the relevant technology and will not be repeated here.

[0198] S204, the intelligent perception control module determines the target resolution 1.

[0199] When it is recognized that the current environment is bright, the target resolution 1 is the first resolution. When it is recognized that the current environment is dark, the target resolution 1 is the second resolution, and the first resolution is greater than the second resolution.

[0200] In some embodiments, the above S204 may refer to S102 and S103 of the aforementioned embodiments, and the specific details are not repeated here.

[0201] In addition, after determining the target resolution 1, the intelligent perception control module can trigger the creation of a transmission path corresponding to the target camera through the camera service. The transmission path is used to transmit control instructions 1 to the target camera. The above control instruction 1 can be a camera parameter 1 carrying the target resolution 1, which can instruct the target camera to use the target resolution 1 for image acquisition. In addition, the above transmission channel is also used to transmit the image data 1 acquired by the target camera. The resolution of the above image data 1 is the target resolution 1, and the implementation details are as follows S205~S207:

[0202] S205, the intelligent perception control module sends a second request to the camera service, carrying the target camera information and target resolution 1.

[0203] S206: The camera service sends a second request to the camera HAL.

[0204] S207, the camera HAL creates a channel corresponding to the target camera according to the information of the target camera and the target resolution 1.

[0205] The second request is an instruction for instructing the target camera to perform image acquisition according to the target resolution 1. Exemplarily, the second request may include information of the target camera (e.g., camera ID) and the size of the output resolution to be configured (e.g., target resolution 1). Exemplarily, the second request may also include information such as the output frame rate and data stream format to be configured for the target camera.

[0206] In some embodiments, the smart perception control module may send a second request to the camera service through a vendor native development kit (VNDK) interface, and the camera service passes the second request to the camera HAL.

[0207] In some embodiments, the camera HAL responds to the second request and creates a corresponding transmission path according to the target camera information in the second request and the target resolution 1. It can be understood that the above transmission path at least includes a path corresponding to the sensor node and a path corresponding to the IFE node.

[0208] Exemplarily, the above-mentioned creation of the corresponding transmission path may include: creating sensor nodes and IFE nodes, and determining other software and hardware modules in the path corresponding to the sensor nodes and the IFE nodes.

[0209] Exemplarily, when the target camera is a TOF camera, the path corresponding to the sensor node can be: a path consisting of sensor node-camera driver-TOF camera-IFE-Lite-secure memory. The path corresponding to the IFE node can be: a path consisting of IFE-Lite-camera driver-IFE node. The camera HAL can connect the output port of the sensor node and the input port of the IFE node at the HAL layer. Thus, the path corresponding to the sensor node and the path corresponding to the IFE node can form a closed-loop path. After the path is created, the hardware in the path is powered on (that is, the hardware circuit is powered on), waiting for the image acquisition instruction from the intelligent perception control module, triggering the target camera to acquire the image, and transmitting it back.

[0210] Exemplarily, when the target camera is an RGB camera, the path corresponding to the sensor node can be: a path consisting of sensor node-camera driver-RGB camera-IFE module-IPE module-secure memory. The path corresponding to the IFE node can be: a path consisting of IFE module-camera driver-IFE node. After the above path is created, the hardware in the path is powered on (i.e., the hardware circuit is powered on), waiting for the image acquisition instruction from the intelligent perception control module, triggering the target camera to acquire the image, and transmitting it back.

[0211] It is understandable that when the target camera information carried in the second request is different, the created paths are also different. In addition, for the same target camera, when the target resolution 1 carried in the second request is different, the created paths are also different, for example, the created sensor nodes and IFE nodes are different.

[0212] S208, the camera HAL sends a notification 1 to the camera service, indicating that the channel is successfully created.

[0213] S209, the camera service sends notification 1 to the intelligent perception control module.

[0214] In some embodiments, after successfully establishing the path corresponding to the target camera, the camera HAL sends a notification 1 (also referred to as a success notification) to the smart perception control module through the camera service, indicating that the transmission path for collecting image data required for eye movement recognition has been established. In response to the notification 1, the smart perception control module may execute S210.

[0215] In other embodiments, when the camera HAL fails to establish a path corresponding to the target camera, the camera HAL sends a failure notification to the smart perception control module through the camera service, indicating that the transmission path establishment failed. In this scenario, the smart perception control module can re-initiate the establishment of the transmission channel in response to the failure notification, for example, re-determine the target camera and instruct to create a path corresponding to the newly determined target camera.

[0216] S210, the intelligent perception control module sends an image acquisition instruction to the camera service.

[0217] The image acquisition instruction indicates that the target camera can acquire image data and transmit it back.

[0218] S211, the camera service calls the camera HAL to instruct the target camera to capture images.

[0219] S212, the camera HAL sends camera parameter 1, including target resolution 1, to the camera driver.

[0220] S213, the camera driver sends camera parameter 1 to the target camera.

[0221] S214, the target camera loads camera parameters 1.

[0222] It can be understood that when creating the transmission path of the target camera, the transmission path is set to be a path for configuring camera parameter 1 (including target resolution 1). In this transmission path, the sensor node can obtain camera parameter 1 including target resolution 1.

[0223] In some embodiments, in response to the camera service calling the camera HAL, the sensor node in the camera HAL may pass the camera parameter 1 to the camera driver. Then, the camera parameter 1 is converted into a control instruction recognizable by the kernel, such as control instruction 1, and then the sensor node writes the control instruction 1 to the target camera through the camera driver, instructing the target camera to configure the output resolution to the target resolution 1.

[0224] S215, the camera driver instructs the target camera to collect image data.

[0225] S216, the target camera collects image data.

[0226] In some embodiments, after writing camera parameter 1 (also referred to as control instruction 1) into the target camera, the camera driver may send a streaming instruction to the target camera, instructing the target camera to start image data acquisition and return the FD corresponding to the acquired image data.

[0227] S217, the target camera writes the collected image data 1 into the secure memory, and obtains a first FD indicating its storage location.

[0228] The image data 1 may be the first image data mentioned in S102 and S103. There is a corresponding relationship between the first FD and the storage address of the image data 1 in the secure memory. Other software modules can query the image data 1 based on the first FD through intelligent perception of TA. For details, refer to S223 in the subsequent embodiments.

[0229] S218, the target camera sends a notification 2 to the camera driver, carrying the first FD.

[0230] The above notification 2 may indicate that the target camera has captured image data, such as image data 1. And has stored the image data 1 in the secure memory. The notification 2 may carry the FD corresponding to the image data 1, such as the first FD. The first FD may be used to query the image data 1 from the secure memory.

[0231] S219: The camera driver sends a first FD to the camera HAL.

[0232] S220: The camera HAL sends a first FD to the camera service.

[0233] S221, the camera service sends a first FD to the intelligent perception control module.

[0234] In this way, the intelligent perception control module can use the first FD to obtain the corresponding image data 1 from the secure memory through the intelligent perception TA. In this way, the intelligent perception control module can calculate the corresponding eye movement data based on the image data 1.

[0235] S222, the smart sensing control module sends a first FD to the smart sensing TA.

[0236] S223, the intelligent sensing TA reads out the image data 1 from the secure memory according to the first FD.

[0237] Among them, the secure memory includes the correspondence between different FDs (including the first FD) and different storage addresses. The smart perception TA can read the image data 1 from the secure memory by sending a query instruction carrying the first FD to the secure memory.

[0238] S224, the smart perception TA sends image data 1 to the smart perception control module.

[0239] In some embodiments, after the smart perception TA sends the image data 1 to the smart perception control module, the image data 1 may also be encrypted. After the encrypted image data 1 is sent to the smart perception control module, the corresponding encryption key is sent to the smart perception control module, so that the smart perception control module can use the key to decrypt the image data 1.

[0240] S225, the intelligent perception control module calculates the corresponding eye movement data 1 based on the image data 1.

[0241] In some embodiments, the intelligent perception control module may call a related algorithm module to process the image data 1 to obtain corresponding eye movement data, such as eye movement data 1. The eye movement data 1 may be the eye movement data calculated based on the first image data mentioned in S104.

[0242] For example, when image data 1 is an image captured by a TOF camera, the intelligent perception control module first uses an image processing algorithm to process image data 1 to obtain a corresponding TOF IR image. Then, the face detection algorithm is used to extract a face area rectangular frame and a face key point array from the TOF IR image. In this way, the extracted face area rectangular frame and face key point array are input into the eye movement recognition module to calculate the corresponding eye movement data 1.

[0243] In other embodiments, the above S224 and S225 may also be replaced by: after the smart perception TA obtains the image data 1, the corresponding eye movement data 1 is calculated based on the image data 1. Afterwards, the smart perception TA may send the eye movement data 1 to the smart perception control module.

[0244] S226, the intelligent perception control module sends eye movement data 1 to the intelligent perception service.

[0245] S227, the intelligent perception service sends eye movement data 1 to the application.

[0246] In some embodiments, the application can determine whether the user makes a contactless interaction instruction based on the eye movement data 1. For example, if the eye movement data 1 indicates that the user's gaze point is located on the interactive display content and the stay time exceeds a preset time, it can be determined that the user makes an interaction instruction for the interactive display content without touching the display screen, which is referred to as a contactless instruction.

[0247] When determining that the user has made a contactless instruction, the application can also respond accordingly. Fig. 7A As shown, the SMS application determines that the user has made an instruction to click on the push banner 701 based on the eye movement data 1. For example, if the eye movement data 1 indicates that the user's gaze point is located at the display position of the push banner 701 and stays there for a preset time, the SMS application can call the system UI to display the information interface 702. For another example, Figure 7B As shown, the capsule prompt application determines that the user has made an instruction to view the reminder capsule 703 based on the eye movement data 1. For example, the eye movement data 1 indicates that the user's gaze point is located at the display position of the reminder capsule 703 and stays there for a preset time. The capsule prompt application can call the system UI to display the reminder window 704.

[0248] In other embodiments, each time the electronic device performs eye movement recognition, it first controls the use of a second resolution with lower energy consumption to collect image data. If the image data collected using the second resolution has an overexposure problem, it then controls the target camera to use the first resolution to collect images.

[0249] For example, Fig.11 As shown, the method provided in the embodiment of the present application may also include the following steps:

[0250] B101, in response to satisfying a preset condition, controlling the target camera to acquire second image data at a second resolution.

[0251] It is understandable that the target camera uses the second resolution to collect image data, which consumes less energy than using the first resolution to collect image data. The above-mentioned method of meeting the preset conditions can refer to the aforementioned embodiment and will not be repeated here. When the preset conditions are met, the electronic device needs to execute the eye movement recognition process, for example, it is necessary to control the target camera to collect image data, and calculate the corresponding eye movement data based on the collected image data.

[0252] In some embodiments, in response to satisfying a preset condition, the target camera is controlled by default to calculate the eye movement data according to the second image data of the second resolution. If the second image data does not have an overexposure problem, the target camera is controlled to collect images and calculate the eye movement data according to the second resolution. If the second image data has an overexposure problem, the process enters B102.

[0253] B102: When the number of over-exposure pixels in the second image data exceeds a first number, control the target camera to collect third image data according to the first resolution.

[0254] For example, if the pixel value of a pixel in the image data exceeds a preset pixel value (e.g., 3200), it can be determined that the pixel has an overexposure problem, which can be called an overexposed pixel. In the same image data, when the overexposed pixel exceeds a first number, it can be determined that the image data has an overexposed problem.

[0255] In some embodiments, when the number of over-exposure pixels in the second image data exceeds a first number, the target camera is controlled to collect image data according to a first resolution, such as third image data.

[0256] B103: Calculate corresponding eye movement data based on the third image data.

[0257] In some embodiments, before B102, the electronic device may also perform calculation of eye movement data based on the second image data to avoid frame skipping.

[0258] In other embodiments, if the second image data has an overexposure problem, the second image data may be discarded, and the eye movement data may be calculated based on the third image data after the third image data is acquired according to the first resolution.

[0259] In other embodiments, before B103, it is also possible to check whether the third image data still has the problem of overexposure. If the third image data does not have the problem of overexposure, the target camera can be controlled to change the output resolution to the second resolution to reduce the energy consumption of the device.

[0260] As a way to achieve this, Fig.12 As shown, in the process of implementing the above method, the signaling interaction steps between the various software and hardware modules in the electronic device are as follows:

[0261] S301, the application sends an eye movement recognition registration request to the intelligent perception service.

[0262] S302: The smart perception service sends a first request to the smart perception control module in response to the eye movement recognition registration request.

[0263] S303: The intelligent perception control module determines the target camera in response to the first request.

[0264] In some embodiments, the implementation details of the above S301 to S303 may refer to S201 to S303 and will not be described in detail here.

[0265] S304: The intelligent perception control module sends a third request to the camera service, carrying the target camera information and the second resolution.

[0266] S305: The camera service sends a third request to the camera HAL.

[0267] Among them, the third request is similar to the second request, and the third request can be used to instruct the target camera to capture images according to the second resolution. Exemplarily, the third request may include information of the target camera (such as camera ID) and the size of the output resolution to be configured (such as the second resolution). Exemplarily, the third request may also include information such as the output frame rate and data stream format to be configured by the target camera.

[0268] In some embodiments, the smart perception control module may send a third request to the camera service through the VNDK interface, and the camera service passes the second request to the camera HAL.

[0269] S306: The camera HAL creates a first channel corresponding to the target camera according to the information of the target camera and the second resolution.

[0270] In some embodiments, the implementation details of the above S306 can refer to S207, which will not be described here. The above first path can be used to transmit the control instruction 2, and the control instruction 2 can indicate the camera parameter 2 carrying the second resolution. In addition, the first path can be used to obtain the image data collected by the target camera according to the second resolution.

[0271] S307, the camera HAL sends a notification 1 to the camera service, indicating that the channel is successfully created.

[0272] S308, the camera service sends notification 1 to the intelligent perception control module.

[0273] S309, the intelligent perception control module sends an image acquisition instruction to the camera service, wherein the image acquisition instruction indicates that the target camera can acquire image data and transmit it back.

[0274] S310: The camera service calls the camera HAL to instruct the target camera to acquire an image.

[0275] S311, the camera HAL sends camera parameter 2, including the second resolution, to the camera driver.

[0276] S312, the camera driver sends camera parameter 2 to the target camera.

[0277] S313, the target camera loads camera parameters 2.

[0278] In some embodiments, the implementation details of the above S309 to S313 may refer to the above S210 to S214 and will not be described in detail here.

[0279] S314, the camera driver instructs the target camera to collect image data.

[0280] S315, the target camera collects image data.

[0281] S316, the target camera writes the collected image data 2 into the secure memory, and obtains a second FD indicating its storage location.

[0282] Among them, there is a corresponding relationship between the above-mentioned second FD and the storage address of the image data 2 in the secure memory. Other software modules can query the image data 2 based on the second FD through intelligent perception TA. For specific details, please refer to S322 in the subsequent embodiments.

[0283] The image data 2 may be image data acquired at a second resolution, that is, the second image data mentioned in the above embodiment.

[0284] S317, the target camera sends notification 3 to the camera driver, carrying the second FD.

[0285] The notification 3 is used to indicate that the image data 2 of the second resolution has been acquired and stored in the secure memory.

[0286] S318: The camera driver sends a second FD to the camera HAL.

[0287] S319, the camera HAL sends the second FD to the camera service.

[0288] S320, the camera service sends a second FD to the intelligent perception control module.

[0289] In this way, the intelligent perception control module can use the second FD to obtain the corresponding image data 2 from the secure memory through the intelligent perception TA. In this way, the intelligent perception control module can calculate the corresponding eye movement data based on the image data 2.

[0290] S321, the smart sensing control module sends a second FD to the smart sensing TA.

[0291] S322, the intelligent sensing TA reads out the image data 2 from the secure memory according to the second FD.

[0292] S323, the smart perception TA sends image data 2 to the smart perception control module.

[0293] S324, the intelligent perception control module detects that the number of overexposed pixels in image data 2 exceeds a first number.

[0294] The overexposed pixel refers to a pixel whose pixel value is greater than a preset pixel value (eg, 3200). The first number may be a preconfigured empirical value, for example, 1000.

[0295] When the number of overexposed pixels in the image data 2 exceeds the first number, it can be determined that the image data 2 has an overexposed problem. In response to determining that the image data 2 has an overexposed problem, the intelligent perception control module can instruct to change the output resolution of the target camera. This ensures the accuracy of the eye movement data calculated based on the subsequently collected image data. That is, the process can enter S327.

[0296] In some embodiments, after the intelligent perception control module obtains the image data 2, it can also calculate the corresponding eye movement data based on the image data 2, such as referred to as eye movement data 2. That is, after S323, S325 and S326 can also be executed. In addition, there is no necessary order between the above S325 and S324.

[0297] S325, when the eye movement data 2 is calculated based on the image data 2, the smart perception control module sends the eye movement data 2 to the smart perception service.

[0298] S326, the intelligent perception service sends eye movement data 2 to the application.

[0299] S327, the intelligent perception control module sends a fourth request to the camera service, carrying a first resolution for the target camera, where the first resolution is greater than the second resolution.

[0300] S328: The camera service sends a fourth request to the camera HAL.

[0301] Among them, the above-mentioned fourth request is similar to the second request and will not be elaborated here.

[0302] S329: The camera HAL creates a second path corresponding to the target camera according to the first resolution.

[0303] The second path can be used to transmit the control instruction 3, which can indicate the camera parameter 3 carrying the first resolution. In addition, the second path can be used to obtain the image data collected by the target camera according to the first resolution.

[0304] S330, the camera HAL sends a notification 1 to the camera service, indicating that the channel is successfully created.

[0305] S331, the camera service sends notification 1 to the intelligent perception control module.

[0306] S332, the intelligent perception control module sends an image acquisition instruction to the camera service.

[0307] The image acquisition instruction indicates that the target camera can acquire image data and transmit it back.

[0308] S333, the camera service calls the camera HAL to instruct the target camera to acquire images.

[0309] S334, the camera HAL sends camera parameter 3, including the first resolution, to the camera driver.

[0310] S335, the camera driver sends camera parameter 3 to the target camera.

[0311] S336, the target camera loads camera parameters 3.

[0312] S337, the camera driver instructs the target camera to collect image data.

[0313] S338, the target camera collects image data.

[0314] S339, the target camera writes the collected image data 3 into the secure memory, and obtains a third FD indicating its storage location.

[0315] The image data 3 may be image data collected at the first resolution, that is, the third image data mentioned in the above embodiment. There is a corresponding relationship between the third FD and the storage address of the image data 3 in the secure memory. Other software modules can query the image data 3 based on the third FD through intelligent perception of TA. For details, refer to S345 in the subsequent embodiments.

[0316] S340, the target camera sends a notification 4 to the camera driver, carrying the third FD.

[0317] The notification 4 is used to indicate that the image data 3 of the first resolution has been acquired and stored in the secure memory.

[0318] S341, the camera driver sends the third FD to the camera HAL.

[0319] S342, the camera HAL sends the third FD to the camera service.

[0320] S343, the camera service sends the third FD to the intelligent perception control module.

[0321] S344, the smart sensing control module sends a third FD to the smart sensing TA.

[0322] S345, the intelligent perception TA reads the image data 3 from the secure memory according to the third FD.

[0323] S346, the smart perception TA sends image data 3 to the smart perception control module.

[0324] S347, the intelligent perception control module calculates eye movement data 3 based on image data 3.

[0325] S348, the intelligent perception control module sends eye movement data 3 to the intelligent perception service.

[0326] S349, the intelligent perception service sends eye movement data 3 to the application.

[0327] In other embodiments, after S346, it is also possible to detect whether the image data 3 has an overexposure problem. If the image data 3 does not have an overexposure problem, for example, the number of overexposure pixels is less than the first number, the target camera may be reconfigured to acquire images using a second resolution.

[0328] In other embodiments, in a scenario where the electronic device needs to perform eye movement recognition, before the target camera has not yet produced an image, the output resolution of the target camera can be determined according to the ambient light brightness. After the target camera actually produces an image, it is determined whether to adjust the output resolution of the target camera based on whether the collected image data has an overexposure problem.

[0329] like Fig.13 As shown, the above method may further include the following steps:

[0330] S401, the application sends an eye movement recognition registration request to the intelligent perception service.

[0331] S402: The smart perception service sends a first request to the smart perception control module in response to the eye movement recognition registration request.

[0332] S403: The intelligent perception control module determines the target camera in response to the first request.

[0333] In some embodiments, the implementation details of the above S401 to S403 may refer to S201 to S303 and will not be described in detail here.

[0334] S404: When it is determined that the current environment is dark, a third request is sent, carrying information of the target camera and the second resolution.

[0335] In other embodiments, when it is determined that the current environment is bright, the third request is sent carrying the information of the target camera and the first resolution.

[0336] S405: The camera service sends a third request to the camera HAL.

[0337] S406: The camera HAL creates a first channel corresponding to the target camera according to the information of the target camera and the second resolution.

[0338] S407, the camera HAL sends a notification 1 to the camera service, indicating that the channel is successfully created.

[0339] S408, the camera service sends notification 1 to the intelligent perception control module.

[0340] S409, the intelligent perception control module sends an image acquisition instruction to the camera service.

[0341] The image acquisition instruction indicates that the target camera can acquire image data and transmit it back.

[0342] S410: The camera service calls the camera HAL to instruct the target camera to acquire an image.

[0343] S411, the camera HAL sends camera parameter 2, including the second resolution, to the camera driver.

[0344] S412: The camera driver sends camera parameter 2 to the target camera.

[0345] S413, the target camera loads camera parameters 2.

[0346] In some embodiments, the implementation details of the above S409 to S413 may refer to the above S210 to S214, which will not be described in detail here.

[0347] S414, the camera driver instructs the target camera to collect image data.

[0348] S415, the target camera collects image data.

[0349] S416, the target camera writes the collected image data 4 into the secure memory, and obtains a fourth FD indicating its storage location.

[0350] There is a corresponding relationship between the fourth FD and the storage address of the image data 4 in the secure memory, and other software modules can intelligently sense TA and query the image data 4 based on the fourth FD.

[0351] S417, the target camera sends a notification 5 to the camera driver, carrying the fourth FD.

[0352] The notification 4 is used to indicate that the image data 4 of the second resolution has been acquired and stored in the secure memory.

[0353] S418: The camera driver sends the fourth FD to the camera HAL.

[0354] S419, the camera HAL sends the fourth FD to the camera service.

[0355] S420, the camera service sends the fourth FD to the intelligent perception control module.

[0356] In this way, the intelligent perception control module can use the fourth FD to obtain the corresponding image data 4 from the secure memory through the intelligent perception TA. In this way, the intelligent perception control module can calculate the corresponding eye movement data based on the image data 4.

[0357] S421, the smart sensing control module sends a fourth FD to the smart sensing TA.

[0358] S422, the intelligent sensing TA reads the image data 4 from the secure memory according to the fourth FD.

[0359] S423, the smart perception TA sends image data 4 to the smart perception control module.

[0360] S424, the intelligent perception control module detects that the number of overexposed pixels in image data 4 exceeds a first number.

[0361] S425, when the eye movement data 4 is calculated based on the image data 4, the smart perception control module sends the eye movement data 4 to the smart perception service.

[0362] S426, the intelligent perception service sends eye movement data 4 to the application.

[0363] S427, the intelligent perception control module sends a fourth request to the camera service, carrying a first resolution for the target camera, where the first resolution is greater than the second resolution.

[0364] In some embodiments, there is no necessary order between the above S424 and S327.

[0365] S428: The camera service sends a fourth request to the camera HAL.

[0366] Among them, the above-mentioned fourth request is similar to the second request and will not be elaborated here.

[0367] S429: The camera HAL creates a second path corresponding to the target camera according to the first resolution.

[0368] S430, the camera HAL sends a notification 1 to the camera service, indicating that the channel is successfully created.

[0369] S431, the camera service sends notification 1 to the intelligent perception control module.

[0370] S432, the intelligent perception control module sends an image acquisition instruction to the camera service.

[0371] S433, the camera service calls the camera HAL to instruct the target camera to capture images.

[0372] S434, the camera HAL sends camera parameter 3, including the first resolution, to the camera driver.

[0373] S435, the camera driver sends camera parameter 3 to the target camera.

[0374] S436, the target camera loads camera parameters 3.

[0375] S437, the camera driver instructs the target camera to collect image data.

[0376] S438, the target camera collects image data.

[0377] S439, the target camera writes the collected image data 5 into the secure memory, and obtains the fifth FD indicating its storage location.

[0378] The image data 5 may be image data collected at the first resolution, that is, the third image data mentioned in the above embodiment. There is a corresponding relationship between the fifth FD and the storage address of the image data 5 in the secure memory, and other software modules can query the image data 5 based on the fifth FD through intelligent perception of TA.

[0379] S440, the target camera sends a notification 6 to the camera driver, carrying the fifth FD.

[0380] The notification 6 is used to indicate that the image data 5 of the first resolution has been acquired and stored in the secure memory.

[0381] S441, the camera driver sends the fifth FD to the camera HAL.

[0382] S442, the camera HAL sends the fifth FD to the camera service.

[0383] S443, the camera service sends the fifth FD to the intelligent perception control module.

[0384] S444, the smart sensing control module sends the fifth FD to the smart sensing TA.

[0385] S445, the intelligent perception TA reads out the image data 5 from the secure memory according to the fifth FD.

[0386] S446, the smart perception TA sends image data 5 to the smart perception control module.

[0387] S447, the intelligent perception control module calculates the eye movement data 5 based on the image data 5.

[0388] S448, the intelligent perception control module sends eye movement data 5 to the intelligent perception service.

[0389] S449, the intelligent perception service sends eye movement data 5 to the application.

[0390] In some embodiments, the implementation details of the above S401 to S449 can be referred to Fig.10 and Fig.12 The implementation details of each step in the method shown are not described here.

[0391] In other possible embodiments, after the intelligent perception control module obtains the image data 5, in addition to calculating the corresponding eye movement data based on the image data 5, it can also detect whether the image data 5 still has an overexposure problem. If the image data 5 does not have an overexposure problem, the target camera can be controlled again to adjust the output resolution to the second resolution.

[0392] In other embodiments, before S404, the electronic device recognizes that it is in a bright environment, and the above S404 can be replaced by: when it is determined that it is currently in a bright environment, sending a third request carrying the information of the target camera and the first resolution. In this way, the image data 4 mentioned in S416 to S425 can be image data collected by the target camera at the first resolution. The above S424 can be replaced by: the intelligent perception control module detects that the overexposed pixels in the image data 4 do not exceed the first number. S427 to S449 can be replaced by: the intelligent perception control module sends a fourth request to the camera service, carrying the second resolution for the target camera. Then, the camera parameter 2 carrying the second resolution is configured to the target camera, instructing the target camera to collect images at the second resolution, and based on the collected image data, calculate the corresponding eye movement data.

[0393] The embodiment of the present application also provides an electronic device, which may include: a camera, a display screen, a memory, and one or more processors. The memory is coupled to the processor. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can execute the various steps performed by the electronic device in the above embodiment. Of course, the electronic device includes but is not limited to the above memory and one or more processors.

[0394] In some embodiments, the electronic device may display a first interface. Exemplarily, the first interface may be any interface displayed by the electronic device, such as: Fig. 7A and Figure 7B The first interface can also be an application interface of a specific application, for example, Figure 7C An application interface 705 of a novel application is shown.

[0395] The first interface includes the first information. For example, the first information can be Fig. 7AThe push banner 701 on the displayed main interface. For another example, the first information may be Figure 7B The reminder capsule 703 on the displayed main interface. For another example, the first information may be Figure 7C Page turning controls 706 are shown.

[0396] In an exemplary scenario, when the electronic device is located in a space where the ambient light brightness value is greater than a first value, the electronic device can control the target camera to capture a first image.

[0397] The first value may be the 50000 lux mentioned in the above embodiment. Exemplarily, the electronic device being located in a space where the ambient light brightness value is greater than the first value may be the electronic device being located in a space with ambient light.

[0398] The target camera may be one of the front cameras of the electronic device. For example, the target camera may be a TOF camera. For example, the target camera may be an RGB camera.

[0399] The resolution of the first image is the first resolution (2phase), and the first image includes an eye image of the user. When the first image indicates that the user's gaze point is located in the display area of ​​the first information, the second interface is displayed, and the second interface is the application interface corresponding to the first information. For example, the second interface can be Fig. 7A The information interface 702 shown. For another example, the second interface can also be Figure 7B The main interface in which the reminder window 704 is displayed. For another example, the second interface may also be Figure 7C The application interface 707 in.

[0400] In another exemplary scenario, when the electronic device is located in a space where the ambient light brightness value is not greater than a second value, the target camera is controlled to capture a second image. The second value is not greater than the first value. In some embodiments, the second value may be equal to the first value, such as also equal to 50,000 lux. In other embodiments, the second value may be less than the first value, such as equal to 1,000 lux. Exemplarily, the electronic device is located in a space where the ambient light brightness value is not greater than the second value, including: the electronic device is located in a space without ambient light.

[0401] The resolution of the second image is a second resolution (1phase), and the second resolution is less than the first resolution. The second image includes an eye image of the user. When the second image indicates that the user's gaze point is located within the display area of ​​the first information, the second interface is displayed.

[0402] In scenario 1, before controlling the target camera to collect image data, the electronic device may first control the target camera to power on. After the target camera is powered on, a first ambient brightness value is detected, and the first ambient brightness value is greater than the first value. The electronic device writes a first camera parameter (camera parameter 3) including a first resolution into the target camera.

[0403] In scenario 2, before controlling the target camera to collect image data, the electronic device can first control the target camera to power on. After the target camera is powered on, a second ambient brightness value is detected, and the second ambient brightness value is not greater than the second value, and a second camera parameter (camera parameter 2) including a second resolution is written into the target camera.

[0404] The above scenarios 1 and 2 can be electronic equipment applications Fig.10 The following are possible scenarios that may occur after the method shown.

[0405] In scenario 3, before controlling the target camera to collect image data, the electronic device may first control the target camera to power on. After the target camera is powered on, the second camera parameters carrying the second resolution are written into the target camera. The target camera is controlled to collect a third image (e.g., image data 2). When the third image is an overexposed image, the first camera parameters containing the first resolution are written into the target camera.

[0406] The above scenario 3 can be an electronic device application Fig.12 After using the method, the collected image data shows overexposed scenes.

[0407] In some embodiments, the electronic device can determine whether the third image is overexposed by determining whether the number of overexposed pixels in the third image exceeds the first number, wherein the overexposed pixel is a pixel whose pixel value is greater than the first pixel threshold (preset pixel value).

[0408] In scenario 4, before controlling the target camera to collect image data, the electronic device may first control the target camera to power on. After the target camera is powered on, write the second camera parameter carrying the second resolution into the target camera; control the target camera to collect a fourth image; and determine that the overexposed pixel points in the fourth image do not exceed the first number, wherein the overexposed pixel points are pixel points whose pixel values ​​are greater than the first pixel threshold.

[0409] The above scenario 4 can be an electronic device application Fig.12 After the method shown, the collected image data does not show any overexposed scenes.

[0410] In scenario 5, before controlling the target camera to collect image data, the electronic device may first control the target camera to power on. After the target camera is powered on, a second ambient brightness value is detected, and the second ambient brightness value is not greater than the second value. The second camera parameters carrying the second resolution are written into the target camera. The target camera is controlled to collect a fifth image (e.g., the image data 4 mentioned in the above embodiment), and the resolution of the fifth image is the second resolution. In the case where the fifth image is an overexposed image, the first camera parameters containing the first resolution are written into the target camera.

[0411] The above scenario 5 can be an electronic device application Fig.13 The following are possible scenarios that may occur after the method shown.

[0412] In some embodiments, the target camera is a time-of-flight TOF camera, and controlling the target camera to capture the first image includes: when the transmitter of the target camera is turned on, controlling the target camera to capture the first image frame, that is, the TX-ON frame. When the transmitter of the target camera is turned off, controlling the target camera to capture the second image frame, that is, the TX-OFF frame. The above-mentioned first image (the image captured by the TOF camera using the first resolution) includes the first image frame and the second image frame.

[0413] In some embodiments, the target camera is a time-of-flight TOF camera, and controlling the target camera to capture the second image includes: when the transmitter of the target camera is turned on, controlling the target camera to capture a third image frame (TX-ON frame), and the second image (the image captured by the TOF camera using the second resolution) includes the third image frame.

[0414] In some embodiments, the electronic device includes a secure memory. After controlling the target camera to capture the first image, when the first image is captured, the electronic device can instruct the target camera to write the first image into the secure memory, obtain the corresponding first file descriptor (e.g., the third FD), and use the first file descriptor to search for the first image.

[0415] After the first image is found, the corresponding first eye movement data (e.g., eye movement data 3) is determined according to the first image, the first eye movement data including first position information and a corresponding first dwell time, the first position information indicating the position of the user's gaze point on the display screen. It is determined that the first position information is located within the display area of ​​the first information, and the first dwell time reaches a preset duration.

[0416] The embodiment of the present application also provides a chip system, which can be applied to the electronic device in the aforementioned embodiment. The chip system includes at least one processor and at least one interface circuit. The processor can be the processor in the above-mentioned electronic device. The processor and the interface circuit can be interconnected by lines. The processor can receive and execute computer instructions from the memory of the above-mentioned electronic device through the interface circuit. When the computer instructions are executed by the processor, the electronic device can execute the various steps executed by the electronic device in the above-mentioned embodiment. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiment of the present application.

[0417] In some embodiments, through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0418] Each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0419] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk.

[0420] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. An information processing method, characterized in that: Used in electronic equipment, including: Displaying a first interface, wherein the first interface includes first information; When the electronic device is located in a space where the ambient light brightness value is greater than a first value, controlling the target camera to capture a first image; the resolution of the first image is a first resolution, and the first image includes an eye image of the user; when the first image indicates that the user's gaze point is located in a display area of ​​the first information, displaying a second interface, the second interface being an application interface corresponding to the first information; Alternatively, when the electronic device is located in a space where the ambient light brightness value is not greater than a second value, the target camera is controlled to capture a second image; the second value is not greater than the first value, the resolution of the second image is a second resolution, the second resolution is smaller than the first resolution, and the second image includes an eye image of the user; when the second image indicates that the user's gaze point is within the display area of ​​the first information, the second interface is displayed.

2. The method according to claim 1, characterized in that Before controlling the target camera to capture the first image, the method further includes: After the target camera is powered on, a first environment brightness value is detected, and the first environment brightness value is greater than the first value; Writing first camera parameters including a first resolution into the target camera.

3. The method according to claim 1 or 2, characterized in that: Before controlling the target camera to capture the second image, the method further includes: After the target camera is powered on, a second environment brightness value is detected, and the second environment brightness value is not greater than the second value; Writing second camera parameters including a second resolution into the target camera.

4. The method according to claim 1, characterized in that: Before controlling the target camera to capture the first image, the method further includes: After the target camera is powered on, writing second camera parameters carrying the second resolution into the target camera; Controlling the target camera to capture a third image; When the third image is an overexposed image, first camera parameters including a first resolution are written into the target camera.

5. The method according to claim 4, characterized in that Before writing the first camera parameters including the first resolution into the target camera, the method further includes: It is determined that the number of overexposed pixels in the third image exceeds a first number, wherein the overexposed pixels are pixels having pixel values ​​greater than a first pixel threshold.

6. The method according to claim 1, characterized in that Before controlling the target camera to capture the second image, the method further includes: After the target camera is powered on, writing second camera parameters carrying the second resolution into the target camera; Controlling the target camera to capture a fourth image; It is determined that the number of overexposed pixels in the fourth image does not exceed a first number, wherein the overexposed pixels are pixels having pixel values ​​greater than a first pixel threshold.

7. The method according to claim 1, characterized in that Before controlling the target camera to capture the first image, the method further includes: After the target camera is powered on, a second environment brightness value is detected, and the second environment brightness value is not greater than the second value; Writing second camera parameters carrying the second resolution into the target camera; Controlling the target camera to capture a fifth image, wherein the resolution of the fifth image is the second resolution; When the fifth image is an overexposed image, first camera parameters including a first resolution are written into the target camera.

8. The method according to any one of claims 1 to 7, characterized in that: The target camera is a time-of-flight TOF camera, and controlling the target camera to collect the first image includes: When the transmitter of the target camera is turned on, controlling the target camera to capture a first image frame; When the transmitter of the target camera is turned off, the target camera is controlled to capture a second image frame, and the first image includes the first image frame and the second image frame.

9. The method according to any one of claims 1 to 8, characterized in that: The target camera is a time-of-flight TOF camera, and controlling the target camera to collect the second image includes: When the transmitter of the target camera is turned on, the target camera is controlled to capture a third image frame, and the second image includes the third image frame.

10. The method according to claim 8, characterized in that The electronic device includes a secure memory, and after the control target camera acquires the first image, the method further includes: When the first image is acquired, instruct the target camera to write the first image into a secure memory and obtain a corresponding first file descriptor; searching for the first image using the first file descriptor; After finding the first image, determining corresponding first eye movement data according to the first image, the first eye movement data including first position information and a corresponding first dwell time, the first position information indicating the position of the user's gaze point on the display screen; It is determined that the first location information is located within a display area of ​​the first information and that a first stay time reaches a preset duration.

11. The method according to claim 1, characterized in that: The electronic device is located in a space where the ambient light brightness value is greater than a first value, including: the electronic device is located in a space with ambient light; the electronic device is located in a space where the ambient light brightness value is not greater than a second value, including: the electronic device is located in a space without ambient light.

12. The method according to any one of claims 1 to 11, characterized in that: The first information includes notification push information, a reminder capsule control, and a page turning control.

13. An electronic device, characterized in that: The electronic device includes a display screen, one or more cameras, a memory, and one or more processors; the one or more cameras include a TOF camera, and the display screen, the camera, the memory, and the processor are coupled; The display screen is used to display the image generated by the processor, the camera is used to collect image data, and the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method described in any one of claims 1-12.

14. A readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 12.

Images