Face recognition method and electronic device

Abstract

Embodiments of the present application provide a face recognition method and an electronic device, and relate to the technical field of terminals. When a low-resolution TOF camera sensor is enabled, the accuracy of face recognition is ensured, and the success rate of face recognition is improved. The specific scheme is as follows: a first operation is received, the first operation triggers the electronic device to perform face recognition; in response to the first operation, an RGB camera sensor is controlled to collect a first image, and a TOF camera sensor is controlled to collect a second image, the first image and the second image being image data collected synchronously by the RGB camera sensor and the TOF camera sensor; first detection is performed based on the first image; the first detection is an image detection item for an image region in which eyes are displayed in the image data, and is used to detect whether a user has an intention to participate in face recognition; in a case where the first image passes the first detection, face recognition is performed based on the second image.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a face recognition method and electronic device. Background Technology

[0002] A Time-of-Flight (TOF) camera sensor consists of a transmitter (TX) and a receiver (RX). The TX emits infrared light or laser pulses, while the RX receives the reflected light and forms an image. Compared to RGB camera sensors, TOF camera sensors offer advantages in low-light image acquisition. Currently, TOF camera sensors are widely used in various electronic devices.

[0003] However, for low-resolution TOF camera sensors, when capturing images of faces with squinting eyes, it is difficult to extract eye feature information from the face image, resulting in the failure of the pre-detection of face recognition (such as eye gaze detection, eye opening and closing detection, etc.), and thus, the failure of face recognition. Summary of the Invention

[0004] This application provides a face recognition method and electronic device that improves the success rate of face recognition when a low-resolution TOF camera sensor is enabled.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, embodiments of this application provide a face recognition method applied to an electronic device. The electronic device is equipped with an RGB camera sensor and a TOF camera sensor. The method includes: receiving a first operation, which triggers the electronic device to perform face recognition. In response to the first operation, the RGB camera sensor and the TOF camera sensor can be controlled to simultaneously output images. For example, the RGB camera sensor can be controlled to acquire a first image. The first image acquired by the RGB camera sensor has higher resolution and carries richer feature information of facial key points (e.g., eyes). Alternatively, the TOF camera sensor can be controlled to acquire a second image. The image quality of the second image acquired by the TOF camera sensor is not affected by the ambient light level. The electronic device can perform a first detection based on the first image. The first detection is an image detection item targeting the image region displaying the eyes in the image data, used to detect whether the user intends to participate in face recognition. If the first image passes the first detection, face recognition is performed based on the second image.

[0007] Understandably, the aforementioned first detection, as a pre-detection step in face recognition, can ensure the reliability of the face recognition results.

[0008] In the above embodiments, the first image and the second image are acquired simultaneously, and the subject appearing in both images is the same. Using the first image, which carries richer feature information of facial key points, for the first detection improves the error tolerance of the first detection. Using the second image, whose image quality is less affected by ambient light, for face recognition ensures the accuracy of the face recognition results. Thus, the electronic device improves the success rate of face recognition while ensuring the reliability of the results.

[0009] In some embodiments, a communication line is configured between an RGB camera sensor and a TOF camera sensor. The step of the RGB camera sensor acquiring a first image and the TOF camera sensor acquiring a second image in response to the first operation includes: in response to the first operation, the RGB camera sensor acquiring the first image, and the RGB camera sensor sending first synchronization information to the TOF camera sensor via the communication line; the TOF camera sensor acquiring the second image in response to the first synchronization information.

[0010] In the above embodiments, the synchronous image output of the RGB camera sensor and the TOF camera sensor is realized at the hardware level, resulting in better synchronization and ensuring the reliability of the subsequent first detection and face recognition performed by fusing the first and second images.

[0011] In some embodiments, before performing the first detection based on the first image, the method further includes: performing the first detection based on the first infrared (IR) image of the second image; wherein performing the first detection based on the first image includes: performing the first detection based on the first image in response to the first IR image failing the first detection.

[0012] In some embodiments, if the face recognition fails, the method further includes: controlling an RGB camera sensor to acquire a fourth image and a TOF camera sensor to acquire a fifth image, wherein the fourth and fifth images are image data acquired synchronously by the RGB camera sensor and the TOF camera sensor; performing the first detection based on the second IR image of the fifth image; and performing face recognition based on the second IR image after the second IR image passes the first detection.

[0013] Understandably, if the eyes of the face appearing in the second image are small, the IR map of the second image is more likely to fail the first detection. Conversely, if the eyes of the face appearing in the second image are large, the IR map of the second image is more likely to pass the first detection.

[0014] In an exemplary scenario, after applying the above method, if a user with squinting eyes makes their first action and then obscures the RGB camera sensor, face recognition fails. If a user with squinting eyes makes their first action and the RGB camera sensor is not obscured, face recognition succeeds. In contrast, if a user with wide-open eyes makes their first action, face recognition succeeds regardless of whether the RGB camera sensor is obscured.

[0015] In some embodiments, the electronic device further includes a display screen; after receiving the first operation and before the RGB camera sensor acquires the first image and the TOF camera sensor acquires the second image, the method further includes: in response to the first operation, controlling the RGB camera sensor and the TOF camera sensor to power on; when the ambient light brightness in the space where the electronic device is located is not greater than a first threshold, adjusting the background brightness of the display screen from a first value to a second value, wherein the first value is the background brightness of the display screen before the RGB camera sensor is powered on.

[0016] In the above embodiments, when the ambient light brightness in the space where the electronic device is located is not greater than the first threshold, the image quality of the image data collected by the RGB camera sensor will be affected. By brightening the background light brightness of the display screen, the background light is used to provide local supplementary lighting for the user's face, thereby ensuring the image quality of the image area displaying key facial points in the first image and reducing the false judgment rate of the first detection.

[0017] In some embodiments, after performing a first detection based on a first image, the method further includes: if the first image fails the first detection, displaying a first reminder message, the first reminder message indicating that face recognition has failed.

[0018] In the above embodiments, if neither the first image nor the second image passes the first detection, the face recognition is determined to be a failure, indicating that there is a risk in the face recognition. In this scenario, there is no need to continue face recognition based on the second image, thus ensuring the reliability of the obtained face recognition result.

[0019] In some embodiments, after displaying the first reminder information, the method further includes: when the ambient light brightness in the space where the electronic device is located is not greater than a first threshold, increasing the background brightness of the display screen from a third value to a fourth value; wherein the third value is the background brightness of the display screen before displaying the first reminder information.

[0020] In some embodiments, by increasing the background light brightness, the influence of image data acquired by the RGB camera sensor in low ambient light brightness is reduced, thus avoiding the image data acquired by the RGB camera sensor failing the first detection due to ambient light brightness.

[0021] In some embodiments, the electronic device is pre-configured with a first mapping relationship, which indicates the positional relationship between pixels presenting the same content in the image data synchronously acquired by the RGB camera sensor and the TOF camera sensor; before performing face recognition based on the second image, the method further includes: adjusting the image coordinates of each pixel in the first image according to the first mapping relationship to obtain a third image; obtaining the first pixel of the image region displaying the eyes in the third image, and obtaining the second pixel of the image region displaying the eyes in the first IR image of the second image; determining a first quantity based on the image coordinates of the first pixel and the second pixel, wherein the first quantity is the number of pixels with different image coordinates corresponding to the first pixel and the second pixel; and determining that the first quantity is not greater than a preset quantity threshold.

[0022] In the above embodiments, when the number of the first IR images corresponding to the first image and the second image is not greater than a preset number threshold, it is determined that the second image and the first image can be aligned. The synchronization between the first image and the second image can be verified. This prevents malicious third parties from circumventing the first detection by injecting image data.

[0023] In some embodiments, face recognition based on a second image includes: extracting a first face feature from a first IR image of the second image; if the first face feature matches a pre-configured second face feature, the face recognition result is successful; if the first face feature does not match the second face feature, the face recognition result is unsuccessful.

[0024] In some embodiments, before performing face recognition based on the second image, the method further includes: performing anti-spoofing / liveness detection on a first IR image of the second image; and determining that the subject corresponding to the second image is a real person. This improves the reliability of the face recognition results.

[0025] In some embodiments, the first detection includes human eye gaze detection and / or human eye opening / closing detection.

[0026] In some embodiments, a first detection is performed based on a first infrared (IR) image of the second image, including: extracting eye feature information from the first IR image; and, if the extraction of eye feature information from the first IR image fails, determining that the first IR image has failed human eye gaze detection and / or human eye opening / closing detection.

[0027] In some embodiments, the first detection based on the first infrared (IR) image of the second image further includes: if first eye feature information is extracted from the first IR image, determining whether the user's gaze point is located on the display screen of the electronic device based on the first eye feature information; if the user's gaze point is determined to be outside the display screen, determining that the first IR image fails the human eye gaze detection; or, if first eye feature information is extracted from the first IR image, determining whether the user's eyes are open based on the first eye feature information; if the user's eyes are determined to be closed, determining that the first IR image fails the human eye open / closed detection.

[0028] In some embodiments, the electronic device is configured with secure memory. After the RGB camera sensor acquires a first image and the TOF camera sensor acquires a second image, the method further includes: controlling the RGB camera sensor to write the first image into the secure memory, and controlling the TOF camera sensor to write the second image into the secure memory; before performing a first detection based on the first image, the method further includes: reading the first image and the second image from the secure memory.

[0029] In the above embodiments, secure memory is used to improve the security of images acquired by the camera sensor.

[0030] Secondly, an electronic device is provided in this application embodiment. The electronic device includes a display screen, multiple camera sensors, a memory, and one or more processors. The multiple camera sensors include a TOF camera sensor and an RGB camera sensor. The display screen, the camera sensors, the memory, and the processor are coupled. The display screen is used to display images generated by the processor. The camera sensors are used to acquire image data. The memory is used to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the one or more processors are used to: receive a first operation, which triggers the electronic device to perform face recognition; respond to the first operation, control the RGB camera sensor to acquire a first image, and the TOF camera sensor to acquire a second image, wherein the first image and the second image are image data acquired synchronously by the RGB camera sensor and the TOF camera sensor; perform a first detection based on the first image; wherein the first detection is an image detection item targeting the image area displaying the eyes in the image data, used to detect whether the user has the intention to participate in face recognition; and perform face recognition based on the second image if the first image passes the first detection.

[0031] In some embodiments, a communication line is configured between the RGB camera sensor and the TOF camera sensor, and one or more processors are configured to: in response to a first operation, instruct the RGB camera sensor to acquire a first image, and instruct the RGB camera sensor to send first synchronization information to the TOF camera sensor via the communication line. Thus, the TOF camera sensor can acquire a second image in response to the first synchronization information.

[0032] In some embodiments, prior to performing a first detection based on the first image, one or more processors are configured to: perform a first detection based on a first infrared (IR) image of a second image; wherein performing a first detection based on the first image includes: performing a first detection based on the first image in response to the first IR image failing the first detection.

[0033] In some embodiments, the electronic device further includes a display screen; after receiving the first operation, the RGB camera sensor acquires a first image, and before the TOF camera sensor acquires a second image, one or more processors are configured to: in response to the first operation, control the RGB camera sensor and the TOF camera sensor to power on; when the ambient light brightness in the space where the electronic device is located is not greater than a first threshold, increase the background brightness of the display screen from a first value to a second value; wherein the first value is the background brightness of the display screen before the RGB camera sensor is powered on.

[0034] In some embodiments, after the first detection is performed based on the first image, one or more processors are configured to: display a first reminder message if the first image fails the first detection, the first reminder message indicating that the face recognition has failed.

[0035] In some embodiments, after displaying the first reminder information, one or more processors are configured to: increase the background brightness of the display screen from a third value to a fourth value when the ambient light brightness in the space where the electronic device is located is not greater than a first threshold; wherein the third value is the background brightness of the display screen before displaying the first reminder information.

[0036] In some embodiments, an electronic device is pre-configured with a first mapping relationship, which indicates the positional relationship between pixels presenting the same content in the image data synchronously acquired by the RGB camera sensor and the TOF camera sensor; before performing face recognition based on the second image, one or more processors are configured to: adjust the image coordinates of each pixel in the first image according to the first mapping relationship to obtain a third image; acquire a first pixel in the image region displaying the eyes in the third image, and acquire a second pixel in the image region displaying the eyes in the first IR image of the second image; determine a first quantity based on the image coordinates of the first pixel and the second pixel, wherein the first quantity is the number of pixels with different image coordinates corresponding to the first pixel and the second pixel; and determine that the first quantity is not greater than a preset quantity threshold.

[0037] In some embodiments, one or more processors are configured to: extract a first face feature from a first IR map of a second image; if the first face feature matches a pre-configured second face feature, the face recognition result is successful; if the first face feature does not match the second face feature, the face recognition result is unsuccessful.

[0038] In some embodiments, if the face recognition fails, one or more processors are configured to: control an RGB camera sensor to acquire a fourth image and a TOF camera sensor to acquire a fifth image, wherein the fourth and fifth images are image data acquired synchronously by the RGB camera sensor and the TOF camera sensor; perform a first detection based on the second IR image of the fifth image; and perform face recognition based on the second IR image after the second IR image passes the first detection.

[0039] In some embodiments, before performing face recognition based on the second image, one or more processors are configured to: perform anti-spoofing / liveness detection on a first IR image of the second image; and determine that the subject corresponding to the second image is a real person.

[0040] In some embodiments, the first detection includes human eye gaze detection and / or human eye opening / closing detection.

[0041] In some embodiments, one or more processors are configured to: extract eye feature information from a first IR image; and, if extracting eye feature information from the first IR image fails, determine that the first IR image has not passed the human eye gaze detection and / or human eye opening / closing detection.

[0042] In some embodiments, one or more processors are configured to: when first eye feature information is extracted from a first IR image, determine whether a user's gaze point is located on the display screen of an electronic device based on the first eye feature information; when it is determined that the user's gaze point is located outside the display screen, determine that the first IR image fails the human eye gaze detection; or, when first eye feature information is extracted from the first IR image, determine whether a user's eyes are open based on the first eye feature information; when it is determined that the user's eyes are closed, determine that the first IR image fails the human eye open / closed detection.

[0043] In some embodiments, the electronic device is configured with secure memory; after the RGB camera sensor acquires a first image and the TOF camera sensor acquires a second image, one or more processors are configured to: control the RGB camera sensor to write the first image into the secure memory, and control the TOF camera sensor to write the second image into the secure memory; before the first detection is performed based on the first image, one or more processors are configured to: read the first image and the second image from the secure memory.

[0044] Thirdly, embodiments of this application provide a computer storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the methods described in the first aspect and its possible embodiments.

[0045] Fourthly, this application provides a computer program product that, when run on the aforementioned electronic device, causes the electronic device to perform the methods described in the first aspect and its possible embodiments.

[0046] Understandably, the electronic devices, computer storage media, and computer program products provided in the above aspects are all applied to the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0047] Figure 1 This is an example diagram illustrating the working principle of the TOF camera sensor provided in the embodiments of this application;

[0048] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0049] Figure 3 Example diagrams of the soft and hard structures of an electronic device provided in this application embodiment;

[0050] Figure 4This is an example diagram illustrating the instruction transmission between various modules on the REE side after the application initiates face recognition, as provided in this embodiment of the application.

[0051] Figure 5 This application provides an example diagram illustrating data transmission between various modules on the REE side and TEE side after the camera sensor is powered on, as shown in the embodiments of this application.

[0052] Figure 6 An example diagram illustrating a scenario where a payment application initiates facial recognition, as provided in an embodiment of this application.

[0053] Figure 7 Example diagram of the change in background brightness of the display screen before and after initiating face recognition in a low-light scene provided in this application embodiment;

[0054] Figure 8 This is one of the flowcharts of the face recognition method provided in the embodiments of this application;

[0055] Figure 9 The second flowchart of the face recognition method provided in the embodiments of this application;

[0056] Figure 10 The third flowchart of the face recognition method provided in this application embodiment;

[0057] Figure 11 This is an example diagram illustrating the pixel mapping relationship between an RGB camera sensor and a TOF camera sensor provided in an embodiment of this application.

[0058] Figure 12 This is a signaling interaction diagram of the face recognition method provided in the embodiments of this application. Detailed Implementation

[0059] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0060] To make the following embodiments clear and concise, a brief introduction to the relevant concepts or technologies is given first:

[0061] A time-of-flight (TOF) camera sensor can include a transmitter (TX) and a receiver (RX). For example... Figure 1As shown, TX is used to emit infrared light or laser pulses, and RX is used to receive the reflected light and form an image. After the TX emits infrared light or laser pulses and the RX receives the corresponding reflected light, the TOF camera sensor can use TOF imaging technology to form an image.

[0062] The aforementioned Time-of-Flight (TOF) imaging technology refers to a technology where a TOF camera sensor emits a set of infrared light (or laser pulses) invisible to the human eye. Upon encountering an object, the light is reflected back to the camera sensor. The time difference or phase difference between the emission and reflection is calculated and collected to form a set of distance and depth data, thereby obtaining a three-dimensional 3D model. In other words, TOF imaging technology adds depth information from the Z-axis to traditional 2D XY-axis imaging, ultimately generating 3D image information.

[0063] An RGB camera sensor, which has three color channels (red (R), green (G), and blue (B), can produce images with multiple colors by varying these three color channels and superimposing them.

[0064] Facial recognition technology is a biometric identification technology based on facial feature information. For example, the implementation principle of facial recognition technology is roughly as follows: Facial feature information is extracted from image data containing a face. Then, the extracted facial feature information is compared with the facial feature information corresponding to pre-configured identity information. If the matching degree reaches a preset standard, the face in the image data is determined to belong to the person indicated by the aforementioned identity information, which can be called successful facial recognition. If the matching degree does not reach the preset standard, the face in the image data is determined not to belong to the person indicated by the aforementioned identity information, which can be called failed facial recognition. Specific details can be found in related technologies, which will not be elaborated here.

[0065] In some embodiments, image data containing human faces acquired by both TOF and RGB camera sensors can be used for face recognition processing. Of course, each type of image data has its advantages in face recognition processing. For example, image data acquired by a TOF camera sensor in low-light scenes can still meet the requirements for face recognition. Image data acquired by an RGB camera sensor has higher resolution and contains richer image information; therefore, performing various types of image recognition based on RGB camera sensor data yields more accurate results.

[0066] In some embodiments, since image data containing faces involves user privacy and security information, the image data collected by the TOF camera sensor and the RGB camera sensor for face recognition needs to be isolated and protected using secure memory in the rich execution environment (REE) + trusted execution environment (TEE) architecture.

[0067] The aforementioned rich execution environment (REE), also known as a rich execution environment, ordinary execution environment, or untrusted execution environment, refers to the system operating environment of electronic devices. The system in question can refer to an operating system. REEs are characterized by their openness and scalability.

[0068] The aforementioned Trusted Execution Environment (TEE), also known as a secure side or secure zone, is an area that requires authorization to access. A TEE refers to an independent secure area established within the processor through hardware resource isolation. The TEE and REE coexist in the operating environment of an electronic device, but the TEE and REE are completely isolated in hardware. The TEE has its own operating space with strict protection measures, therefore, it has a higher level of security than the REE. Furthermore, the REE cannot access the resources of the TEE, thus enabling the TEE to protect its internal code and information, and to resist attacks from the REE side, resisting security threats.

[0069] The REE+TEE architecture refers to an architecture where TEE and REE work together to provide services to applications. In other words, TEE and REE coexist within the electronic device. For example, TEE, through hardware support, can achieve an isolated operating mechanism from REE. TEE has its own runtime environment, offering a higher level of security than REE, and can protect assets within the TEE (such as data and software) from software attacks. Only authorized security software can execute within the TEE, which also protects the confidentiality of the security software's resources and data. Compared to REE, due to its isolation and access control mechanisms, TEE can better protect the security of data and resources.

[0070] A trusted application (TA) is an application running in a TEE that provides security services to client applications (CAs) running in an REE, such as key generation and management, security authentication, and facial recognition.

[0071] CA typically refers to an application running in REE, which can call TA through a client application programming interface (API) and instruct TA to perform corresponding security operations.

[0072] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0073] like Figure 2 As shown, an electronic device (such as a mobile phone) may include: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, buttons 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a SIM card interface 295, etc.

[0074] The aforementioned sensor module 280 may include sensors such as pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, and bone conduction sensors.

[0075] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0076] Processor 210 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0077] A controller can be the nerve center and command center of an electronic device. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0078] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0079] In some embodiments, the processor 210 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0080] It is understood that the interface connection relationships between the modules illustrated in this embodiment are merely illustrative and do not constitute a structural limitation on the electronic device. In other embodiments, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0081] The charging management module 240 receives charging input from a charger, which can be a wireless charger or a wired charger. While charging the battery 242, the charging management module 240 can also supply power to the electronic device via the power management module 241.

[0082] The power management module 241 connects the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, and supplies power to the processor 210, internal memory 221, external memory, display 294, camera 293, and wireless communication module 260, etc. In some embodiments, the power management module 241 and the charging management module 240 may also be housed in the same device.

[0083] The wireless communication function of an electronic device can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor, and baseband processor. In some embodiments, antenna 1 of the electronic device is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, enabling the electronic device to communicate with networks and other devices, such as wearable devices, through wireless communication technology.

[0084] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0085] The mobile communication module 250 can provide solutions for wireless communication applications in electronic devices, including 2G / 3G / 4G / 5G. The mobile communication module 250 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation.

[0086] The mobile communication module 250 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna 1. In some embodiments, at least some functional modules of the mobile communication module 250 can be housed in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 250 and at least some modules of the processor 210 can be housed in the same device.

[0087] The wireless communication module 260 can provide solutions for wireless communication applications in electronic devices, including WLAN (such as wireless fidelity, Wi-Fi) networks, Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, and other wireless communication technologies.

[0088] GNSS can include the BeiDou Navigation Satellite System (BDS), the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0089] The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signal, and sends the processed signal to processor 210. The wireless communication module 260 can also receive signals to be transmitted from processor 210, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0090] Electronic devices implement display functions through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0091] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel.

[0092] Electronic devices can perform shooting functions through an ISP, camera 293, video codec, GPU, display 294, and application processor. The ISP is used to process the data fed back by the camera 293.

[0093] Camera 293 is used to capture still images or videos. In some embodiments, the electronic device may include one or N cameras 293, where N is a positive integer greater than 1. The aforementioned N cameras 293 may include cameras with TOF camera sensors and cameras with RGB camera sensors.

[0094] The external storage interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 210 through the external storage interface 220 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0095] Internal memory 221 can be used to store computer executable program code, which includes instructions. Processor 210 executes various functional applications and data processing of the electronic device by running the instructions stored in internal memory 221. For example, in this embodiment, processor 210 can execute instructions stored in internal memory 221, which may include a program storage area and a data storage area.

[0096] The program storage area can store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.). The data storage area can store data created during the use of the electronic device (such as audio data, phonebook, etc.). Furthermore, the internal memory 221 can include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0097] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0098] The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses a layered architecture system as an example to illustrate the software structure of an electronic device.

[0099] Figure 3 This is a block diagram of the hardware and software structure of an electronic device according to an embodiment of the present invention.

[0100] The layered architecture of electronic devices divides software and hardware into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, an electronic device may include, for example, an application layer, an application framework layer, a hardware abstraction layer (HAL), a kernel layer, and a hardware layer. Of course, it may also include... Figure 3 Layers not shown in the diagram include, for example, runtime and system libraries.

[0101] For example, the application layer may include a series of applications.

[0102] like Figure 3 As shown, the application layer can include settings applications, payment applications, lock screen applications, face recognition software development kits (SDKs), etc.

[0103] The face recognition SDK can interact with the application framework layer to instruct the face recognition process, such as sending a face recognition request to the application framework layer.

[0104] The aforementioned settings application, payment application, and lock screen application can be examples of applications already installed in an electronic device. In some other embodiments, the application layer may also include... Figure 3 Applications not shown include, for example, social applications, calling applications, camera applications, maps, navigation applications, and face recognition software development kits (SDKs).

[0105] In some embodiments, the aforementioned settings application, payment application, lock screen application, and other applications may generate a business requirement for facial recognition during operation. For example, while the electronic device is running a settings application, it generates a business requirement to perform facial recognition after detecting a user's instruction to change the password. Similarly, while the electronic device is running a payment application, it generates a business requirement to perform facial recognition after detecting a user's instruction to make a payment. Furthermore, the lock screen application may also generate a business requirement to perform facial recognition after detecting a user's instruction to unlock.

[0106] After an application generates a business requirement for facial recognition, it can call the facial recognition SDK, which instructs the application framework layer to execute the facial recognition process. The operation that triggers the application to initiate facial recognition can be referred to as the first operation.

[0107] For example, the application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0108] like Figure 3 As shown, the application framework layer can include face recognition services and camera services, etc.

[0109] The face recognition service is used to process face recognition requests sent by the face recognition SDK and return the processing results to the application to realize the face recognition function.

[0110] Camera services are used to manage the operation of camera sensors. For example, they can control the camera sensor to turn on and off, and control parameters such as resolution and frame rate when the camera sensor captures images.

[0111] In other embodiments, the application framework layer described above may further include Figure 3 Modules not shown in the diagram include, for example, the window manager, content provider, view system, resource manager, notification manager, etc.

[0112] For example, a hardware abstraction layer (HAL) can encapsulate drivers in the kernel layer and provide an interface for calling the application framework layer, shielding the implementation details of the lower-level hardware.

[0113] like Figure 3 As shown, the HAL of an electronic device includes at least a camera HAL and a face recognition control module.

[0114] The camera HAL is the core software framework of the camera, which can include sensor nodes and image front end (IFE) nodes. Sensor nodes and IFE nodes are components (nodes) in the transmission path (also called transmission pipeline) for image data and control commands created by the camera HAL.

[0115] The face recognition control module is the core software application of face recognition. This module includes the face recognition client application (CA), which runs in the REE environment.

[0116] In addition, within the TEE environment, a trusted application (TA) for face recognition runs, corresponding to the face recognition CA. For example, the face recognition CA can communicate with the face recognition TA, instructing the face recognition TA to perform face recognition-related image processing tasks, such as image transformation, image detection, feature extraction, liveness detection, and feature comparison.

[0117] In addition, the face recognition control module may also include a camera control module and a face recognition HAL. The camera control module can be used to control the camera service to perform specified types of tasks, such as instructing the camera service to call the camera HAL to create a transmission channel.

[0118] The aforementioned face recognition HAL can call face recognition-related algorithm modules, such as TOF processing algorithms and face detection algorithms, to perform face recognition-related image processing tasks on the REE side. Specifically, the TOF processing algorithm can convert image data acquired by the TOF camera sensor into TOF infrared (IR) images, and the face detection algorithm can detect whether a face appears in the image data. It is understood that the electronic device can perform face recognition-related image processing tasks on both the TEE and REE sides. In some embodiments, face recognition-related image processing tasks can be pre-configured to be performed on the TEE side to ensure the security of face recognition. In other embodiments, face recognition-related image processing tasks can also be performed on the REE side; this application does not specifically limit this.

[0119] like Figure 3 As shown, the kernel layer is the layer between hardware and software. The kernel layer contains at least the camera driver. Additionally, the kernel layer may include... Figure 3 Drivers not shown, such as display drivers, sensor drivers, etc.

[0120] The camera driver is the driver layer for camera devices, and it is mainly responsible for the interaction with the camera sensor.

[0121] like Figure 3 As shown, the hardware layer includes a display, a TOF camera sensor, an RGB camera sensor, an IFE module, an IPE module, an IFE-Lite module, and a secure buffer.

[0122] The image front end (IFE) module can perform image processing on image data, such as automatic exposure (AE) and automatic white balance (AWB).

[0123] The Lightweight Image Preprocessing (IFE-Lite) module is a simplified version of the IFE module. It can be used to forward image data without processing the image data during the forwarding process.

[0124] The image processing engine (IPE) module is used to perform image processing tasks such as hardware noise reduction, cropping, color processing, and detail enhancement. Hardware noise reduction includes processes such as multi-frame noise reduction (MFNR) and multi-frame super resolution (MFSR).

[0125] Secure memory refers to memory with security protection features, which can be used to store image data acquired by TOF camera sensors and RGB camera sensors. In some embodiments, images acquired by the TOF camera sensor are forwarded by the IFE-Lite module and then stored in secure memory. Images acquired by the RGB camera sensor can be pre-processed by the IFE module and then passed to the IPE module, which stores them in secure memory. This secure memory can shield the software modules on the REE side, preventing them from directly accessing the data in secure memory and improving data security.

[0126] In some embodiments, the application on the REE side can control the RGB camera sensor or the TOF camera sensor to acquire image data for face recognition, according to business needs.

[0127] like Figure 4 As shown, when an application (such as a settings app, payment app, or lock screen app) generates a business need for facial recognition during runtime, it can request facial recognition services from the application framework layer. For example, the application can call the facial recognition SDK, and the facial recognition SDK, in response to the application's call, sends a facial recognition request to the facial recognition service in the application framework layer.

[0128] like Figure 4 As shown, in response to the face recognition request, the face recognition service can, through the HAL layer, instruct the TOF camera sensor or RGB camera sensor to power on and acquire image data for face recognition. For example, in response to the face recognition request, the face recognition service can call the face recognition control module of the HAL layer.

[0129] The face recognition control module responds to calls from the face recognition service by powering on a specified camera sensor (e.g., an RGB camera sensor or a TOF camera sensor) via the camera service. As one implementation, the face recognition control module can interact with the camera service by calling an interface (vendor native development kit, VNDK).

[0130] like Figure 4 As shown, the aforementioned camera service can interact with the camera HAL in the HAL layer in response to instructions from the face recognition control module. It instructs the camera HAL to create a transmission path corresponding to the specified camera sensor, which can then be used to manage and control the corresponding camera sensor.

[0131] For example, the aforementioned transmission path includes at least a sensor node, an IFE node, a specified camera sensor, and a corresponding camera driver. It is understood that the sensor node and IFE node in the transmission path may differ for different camera sensors. Creating the transmission path corresponding to the specified camera sensor may include: creating the sensor node and IFE node corresponding to the specified camera sensor, and waking up the specified camera sensor.

[0132] Additionally, sensor nodes and IFE nodes can interact with camera drivers in the kernel layer. For example, a sensor node can determine the camera parameters required when a specified camera sensor enables a preset operating mode, and write the determined camera parameters to the specified camera sensor through the camera driver, so that the specified camera sensor can acquire image data according to the preset operating mode (e.g., face capture mode).

[0133] In some embodiments, when the electronic device is in a bright light scene, the specified camera sensor can be an RGB camera sensor. When the electronic device is in a dark light scene, the specified camera sensor can be a TOF camera sensor. Furthermore, the electronic device can determine whether it is in a bright light scene or a dark light scene based on the detected ambient light intensity. For example, if the detected ambient light intensity value is greater than a preset brightness threshold, the environment is determined to be a bright light scene. Conversely, if the detected ambient light intensity value is not greater than the preset brightness threshold, the environment is determined to be a dark light scene. In this way, the acquired image data can be guaranteed to meet the image quality requirements for face recognition under different ambient light intensity scenarios.

[0134] Currently, before performing facial recognition based on image data, electronic devices require one or more verification-based image detections. These verification-based image detections are also known as user verification detections. These detections determine whether a user is willing to participate in facial recognition, preventing malicious third parties from unknowingly stealing the user's facial information for facial recognition. For example, a malicious third party could secretly photograph the user's face when they are not looking at the screen and then use that information for facial recognition. Another example is when a user is asleep with their eyes closed, a malicious third party could use the user's electronic device to secretly photograph their face and then use that information for facial recognition. To prevent such malicious facial recognition, electronic devices can be equipped with one or more verification-based image detection functions.

[0135] For example, the user verification detection described above can be based on image detection that includes an image region containing the human eye, such as human eye gaze detection, human eye opening and closing detection, etc.

[0136] However, when the specified camera sensor is a TOF camera sensor, if the user's eyes are small or the user is squinting, the image data captured by the TOF camera sensor will have low resolution and will be difficult to pass the aforementioned user verification detection, causing the face recognition triggered by the application to fail.

[0137] To address the aforementioned issues, this application provides a face recognition method applied to an electronic device. The electronic device can control a Time-of-Flight (TOF) camera sensor and an RGB camera sensor to simultaneously output images. This allows for user verification detection based on the image data synchronously acquired by the RGB camera sensor. If the image data acquired by the RGB camera sensor is used for user verification detection, face recognition is then performed based on the image data acquired by the TOF camera sensor. This approach improves the success rate of face recognition while ensuring its security.

[0138] The implementation details of the face recognition method provided in the embodiments of this application are described below with reference to the accompanying drawings and based on the above-described hardware and software structure.

[0139] In this embodiment of the application, when the specified camera sensor is TOF, the camera service can instruct the camera HAL to create transmission paths for the RGB camera sensor and the TOF camera sensor, respectively.

[0140] The transmission path of the aforementioned RGB camera sensor can consist of the RGB camera sensor, the corresponding sensor node, the corresponding IFE node, the corresponding camera driver, the IFE module, the IPE module, and secure memory. Details regarding the creation of this path can be found in subsequent embodiments and will not be elaborated here. Similarly, the transmission path of the aforementioned TOF camera sensor can consist of the TOF camera sensor, the corresponding sensor node, the corresponding IFE node, the corresponding camera driver, the IFE module, the IPE module, and secure memory. Details regarding the creation of this path can be found in subsequent embodiments and will not be elaborated here.

[0141] After creating the transmission paths for the RGB camera sensor and the TOF camera sensor, power on the hardware modules in the transmission paths. For example, ... Figure 4 As shown, the power-on of the RGB camera sensor is controlled by the camera driver corresponding to the RGB camera sensor, and the power-on of the TOF camera sensor is controlled by the camera driver corresponding to the TOF camera sensor.

[0142] After the RGB camera sensor and the TOF camera sensor are powered on, they begin to acquire image data.

[0143] For example, such as Figure 5 As shown, a communication line is configured between the RGB camera sensor and the TOF camera sensor. This communication line can be used to transmit synchronization pulse signals, such as XVS signals. These synchronization pulse signals are used to support the synchronous acquisition of image data by the RGB camera sensor and the TOF camera sensor. Additionally, this synchronization pulse signal can also be referred to as the first synchronization information.

[0144] In this setup, the RGB camera sensor acts as the transmitter of the synchronization pulse signal, and the TOF camera sensor acts as the receiver. After both the RGB and TOF camera sensors are powered on, the RGB camera sensor sends a synchronization pulse signal to the TOF camera sensor for each frame of image data it acquires (e.g., image data b), instructing the TOF camera sensor to also acquire a frame of image data (e.g., image data a) synchronously. This ensures a one-to-one correspondence between the image data acquired by the RGB and TOF camera sensors, achieving hardware-level synchronization. It should be noted that there is a very small delay between the acquisition of image data b by the RGB camera sensor and the acquisition of image data a by the TOF camera sensor; this delay is negligible and can be considered as synchronization.

[0145] For example, such as Figure 5As shown, the RGB camera sensor acquires image data b and can send a synchronization pulse signal to the TOF camera sensor. In response to this synchronization pulse, the TOF camera sensor can acquire one frame of image data, referred to as image data a. In subsequent embodiments, image data a and image data b may refer to two frames of image data synchronously acquired by the RGB camera sensor and the TOF camera sensor. For example, image data a may include a second image and a fifth image. Image data b may include a first image and a fourth image. The first and second images are synchronously acquired image data. The fourth and fifth images are synchronously acquired image data.

[0146] After the RGB camera sensor acquires image data b, it can be passed to the IFE module. The IFE module can perform image preprocessing on the image data b, such as converting it to YUV format. Then, the processed image data b is passed to the IPE module, which forwards the image data b to secure memory for storage.

[0147] After the TOF camera sensor acquires image data 'a', it can be passed to the IFE-Lite module. The IFE-Lite module can then forward the image data 'a' to secure memory for storage.

[0148] The camera driver can also be used to obtain the file descriptor (FD) of image data 'a' from the IFE-Lite module, such as... Figure 5 The FD1 in the above camera driver can also obtain the FD of image data b from the IPE module, such as FD1. Figure 5 In the context of FD2, FD1 represents the storage location of image data 'a' acquired by the TOF camera sensor in secure memory, and FD2 represents the storage location of image data 'b' acquired by the RGB camera sensor in secure memory.

[0149] The camera driver then sends FD1 and FD2 to the camera HAL, which in turn sends them to the camera service. The camera service can then send FD1 and FD2 to the face recognition control module. The face recognition control module can then send a data transmission carrying FD1 and FD2 to the face recognition TA on the TEE side.

[0150] In response to FD1 and FD2, the face recognition TA reads image data, such as image data a and image data b, from the secure memory via the trusted camera service on the TEE side, located at the storage locations indicated by FD1 and FD2. After reading the image data (image data a and image data b), the face recognition TA can perform face recognition-related image processing tasks based on the image data (image data a or image data b).

[0151] For example, a face recognition TA may include an acquisition module and a face processing module.

[0152] The acquisition module can interact with the trusted camera service. After receiving FD1 and FD2 from the REE side, the acquisition module sends FD1 and FD2 to the trusted camera service, instructing the trusted camera service to read the corresponding image data a and image data b from secure memory. Then, the acquisition module receives the image data a and image data b returned by the trusted camera service.

[0153] The aforementioned face processing module can perform face recognition-related image processing tasks based on image data a and image data b, such as user verification detection, image feature alignment, anti-spoofing / liveness recognition, face recognition, and one or more other image processing tasks.

[0154] For example, such as Figure 5 As shown, the TEE side is pre-configured with TOF processing algorithm, face recognition algorithm, face detection algorithm, anti-spoofing / liveness recognition algorithm, and image feature alignment algorithm.

[0155] The aforementioned TOF processing algorithm is used to convert the raw (RAW) image (e.g., image data a) acquired by the TOF camera sensor into an IR image.

[0156] The aforementioned face detection algorithm is used to extract facial feature information from the IR image corresponding to image data b or image data a, such as facial key point information (e.g., feature information corresponding to the eyes, nose, mouth, etc. in the image), face bounding box position information (a box indicating the position of the face in the image), face angle information, face occlusion information, etc.

[0157] The aforementioned facial recognition algorithm is used for identity verification based on facial feature information.

[0158] The aforementioned anti-counterfeiting / liveness recognition algorithm is used to identify whether the object participating in face recognition is a real person, and to prevent malicious third parties from using photos to participate in face recognition. The implementation principle of this anti-counterfeiting / liveness recognition algorithm can be found in relevant technologies, which will not be elaborated here.

[0159] The image feature alignment algorithm described above is used to verify whether image data a and image data b were acquired synchronously by the RGB camera sensor and the TOF camera sensor, thus avoiding the problem of tampering with the images acquired by the RGB camera sensor or the TOF camera sensor. Its implementation details will be introduced in subsequent embodiments and will not be repeated here.

[0160] In some embodiments, after receiving image data a and image data b, the face processing module can call one or more of the above-mentioned algorithm modules through the algorithm adaptation layer to perform one or more face recognition-related image processing tasks.

[0161] Taking face recognition as an example, the face processing module, through the algorithm adaptation layer, calls the Time-of-Flight (TOF) processing algorithm to convert image data 'a' into an Inverse Recognition (IR) image, and then calls the face detection algorithm to extract facial feature information from the IR image. Next, the face processing module reads multiple pre-stored identity information entries and their corresponding authentication face features from the secure storage chip on the TEE side. Thus, the face processing module can use the algorithm adaptation layer to call the face recognition module, combining the authentication face features corresponding to the identity information to perform face recognition. For example, it performs feature matching based on the facial feature information extracted from the IR image and the authentication face features. If the extracted facial feature information matches the authentication face features corresponding to any identity information, the face recognition is considered successful. If the extracted facial feature information does not match any of the authentication face features, the face recognition is considered to have failed. In this way, the face processing module obtains the face recognition result.

[0162] In some embodiments, an authentication TA also runs on the TEE side. After the face processing module obtains the face recognition result, the authentication TA can retrieve the face recognition result from the face recognition TA. Then, the authentication TA can pass the face recognition result to the face recognition service on the REE side. In this way, the face recognition service can pass the face recognition result to the application that requested the face recognition.

[0163] In the above embodiments, synchronous image output from the RGB camera sensor and the TOF camera sensor is achieved at the hardware level. By utilizing the synchronicity of the image data acquired by the RGB camera sensor and the image data acquired by the TOF camera sensor, the image data acquired by the RGB camera sensor improves the fault tolerance rate of the user verification and detection stage during face recognition based on the image data acquired by the TOF camera sensor, thus increasing the overall fault tolerance rate of the face recognition process and reducing the failure rate of the electronic device performing the face recognition process.

[0164] In other possible embodiments, timestamps can also be used to synchronize image data acquired by the RGB camera sensor and the TOF camera sensor on the timeline. For example, after receiving image data acquired by the TOF camera sensor, the IFE-lite module can assign a timestamp to the image data, and correspondingly, the FD (File Depository) of the image data can also carry this timestamp. Similarly, after receiving image data acquired by the RGB camera sensor, the IFE module can also assign a timestamp to the image data, and the FD of the image data can also carry this timestamp. Thus, when the face recognition TA receives FD1, it can also search for FD2 in the FDs from the RGB camera sensor based on the timestamp carried in FD1. The acquisition time indicated by the timestamp carried in FD2 is closest to the acquisition time indicated by the timestamp carried in FD1.

[0165] Take, for example, a scenario where a user transfers money using an electronic device. Figure 6 As shown, after the electronic device detects a user's instruction to transfer funds, for example, when the user opens a payment application, a transfer interface 601 is displayed. During the display of the transfer interface 601, the electronic device can receive the user's configured transfer information (e.g., recipient account, transfer amount, etc.). Subsequently, in response to the user's instruction to confirm the transfer, for example, clicking the "Next" control 602 in the transfer interface 601, a face recognition window 603 is displayed, prompting the user to participate in face recognition.

[0166] Additionally, in response to a user's instruction to confirm a transfer, the payment application in the electronic device can also request to initiate a facial recognition process via a facial recognition SDK. Subsequently, if it is determined that image data for facial recognition is acquired by a TOF camera sensor, such as... Figure 4 As shown, it is possible to control the TOF camera sensor and the RGB camera sensor to be powered on simultaneously.

[0167] In some possible embodiments, in low-light scenarios, such as when the electronic device detects that the ambient light level is below a preset brightness threshold (first threshold), after the TOF camera sensor and the RGB camera sensor are powered on, the background light level of the electronic device's display screen can be increased, thereby improving the image quality of the image data acquired by the RGB camera sensor. Thus, as... Figure 7 As shown, in a low-light scenario, before face recognition begins, for example, before the face recognition window 603 is displayed, the background brightness value of the electronic device can be value 1 (first value). After face recognition begins, for example, after the face recognition window 603 is displayed, the background brightness value of the electronic device can be value 2 (second value), where value 2 is greater than value 1. Thus, the backlight brightness value of the electronic device will increase before and after the face recognition window 603 is displayed.

[0168] In other embodiments, the background light brightness value may not be adjusted immediately after the TOF camera sensor and the RGB camera sensor are powered on. That is, there is no need to detect the ambient light brightness in the environment, nor to increase the background light brightness of the electronic device's display screen. This application does not specifically limit this.

[0169] Additionally, in some embodiments, such as Figure 8 and Figure 9 As shown, after the TOF camera sensor and the RGB camera sensor are powered on, they simultaneously acquire image data a and image data b. Image data a is a RAW image acquired by the TOF camera sensor, and the electronic device can convert image data a into an IR image; details of its implementation can be found in [reference needed]. Figure 5 For example, the TOF camera sensor stores the acquired image data 'a' in secure memory. After the face recognition TA reads the image data 'a' from the secure memory, it calls the TOF processing algorithm through the algorithm adaptation layer to convert the image data 'a' into an IR image.

[0170] After displaying the face recognition window 603, if the image data a acquired by the TOF camera sensor contains an image of a squinting face, for example, ... Figure 8 As shown, when a user squints while the TOF camera sensor is capturing an image, or when a user participating in face recognition has small eyes, the electronic device uses the IR map of image data a, resulting in a lower probability of user verification detection.

[0171] After displaying the face recognition window 603, if the image data a acquired by the TOF camera sensor contains an image of a face with wide-open eyes, that is, as shown... Figure 9 As shown, when a user opens their eyes wide while the TOF camera sensor captures an image, or when the user participating in face recognition has large eyes, the electronic device uses the IR map of image data a to increase the probability of user verification detection, such as eye gaze detection and eye opening / closing detection.

[0172] In scenarios where the IR map of image data 'a' cannot be verified by user authentication, such as when a user is squinting during face recognition, ... Figure 8 As shown, the steps for an electronic device to perform a face recognition method are as follows:

[0173] S1, based on the IR map of image data a, performs human eye gaze detection and human eye opening / closing detection.

[0174] In some embodiments, the above-described eye gaze detection and eye opening / closing detection are merely examples of user verification detection. In other embodiments, more or fewer types of user verification detection may be performed. For example, only eye gaze detection or eye opening / closing detection may be performed. Alternatively, other types of detection may be performed. Furthermore, the above-described user verification detection may also be referred to as the first detection.

[0175] In some embodiments, users can configure different face recognition scenarios and the types of user verification detection to be enabled. For example, when an electronic device responds to a user's action, it can be configured not to enable eye gaze detection during face unlock. Thus, in scenarios where face recognition is initiated by a lock screen application, the aforementioned user verification detection can include only eye opening / closing detection.

[0176] For example, the process of eye gaze detection is as follows: After the face processing module in the face recognition TA converts image data a (e.g., the second image) into an IR image (the first IR image), the face processing module can call a face detection algorithm through the algorithm adaptation layer to extract facial feature information from the IR image of image data a. For example, it can extract facial key point information (e.g., feature information corresponding to the eyes, nose, mouth, etc. in the image), face bounding box position information (a box indicating the position of the face in the image), face angle information, and face occlusion information.

[0177] Then, the face processing module calculates the corresponding eye-tracking data based on the facial feature information from the IR map (e.g., eye feature information and / or facial angle information from facial landmark information). The aforementioned eye feature information can also be referred to as eye feature information. This eye-tracking data indicates the location of the user's gaze point. When the eye-tracking data indicates that the user's gaze point is located on the electronic device's display screen, it is determined that eye gaze detection has been successful.

[0178] If the eye-tracking data can indicate that the user's gaze point is outside the display screen, or if the facial feature information from the first IR map is insufficient to calculate the eye-tracking data, or if no eye feature information is extracted from the first IR map, the electronic device can determine from the IR map of image data a that the eye gaze detection has not been passed.

[0179] For example, the process of detecting whether a person's eyes are open or closed is as follows: After the face processing module in the face recognition TA converts image data 'a' into an IR image, the face processing module can call a face detection algorithm through the algorithm adaptation layer to extract facial feature information from the IR image of image data 'a'. Understandably, before detecting whether a person's eyes are open or closed, facial features have already been extracted from the IR image of image data 'a', so the step of calling the face detection algorithm can be skipped.

[0180] Then, the face processing module determines the user's eye opening / closing state based on the eye feature information from the IR image (e.g., the first eye feature information extracted from the first IR image). If the eye feature information indicates that the user's eyes are open, the eye opening / closing detection is passed. If the eye feature information indicates that the user's eyes are closed, the eye opening / closing detection is failed. Additionally, if eye feature information is not successfully extracted from the IR image of image data a, the eye opening / closing detection is also considered a failure. If the extracted eye feature information cannot determine whether the eyes are open or closed, the eye opening / closing detection is also considered a failure.

[0181] Furthermore, there is no necessary order between the aforementioned eye gaze detection and eye opening / closing detection. Both eye gaze detection and eye opening / closing detection rely on eye feature information extracted from image data. In some embodiments, after converting image data a into an IR image, if the electronic device fails to extract eye feature information from the IR image, it can be determined that both eye gaze detection and eye opening / closing detection have failed, and the process proceeds to S2. In some embodiments, if the electronic device extracts eye feature information from the IR image and simultaneously determines that the user is in a closed-eye state, or determines that the user's gaze point is outside the display screen of the electronic device, the process can also proceed to S2.

[0182] S2, based on image data b, performs human eye gaze detection and human eye opening / closing detection.

[0183] In some embodiments, image data b and image data a are acquired synchronously. Image data b (e.g., the first image) comes from an RGB camera sensor, has higher resolution, and contains richer image information. In this embodiment, the electronic device can perform user verification detection (e.g., eye gaze detection and eye opening / closing detection) again based on image data b. The implementation details of S2 are the same as those of S1, except that the image data used in S1 is the IR map of image data a, while the image data used in S2 is image data b, which will not be elaborated here.

[0184] After image data b passes the eye gaze detection and eye opening / closing detection, the process can proceed to S3. If image data b fails the eye gaze detection and eye opening / closing detection, the face recognition initiated by the payment application is deemed to have failed.

[0185] After determining that facial recognition has failed, the electronic device can display a prompt message, such as a first reminder message, to remind the user to participate in facial recognition again. For example, this prompt message can be a text reminder, such as displaying the text "Recognition failed. Please move closer to the display screen and face the screen again to perform facial recognition."

[0186] In addition to displaying prompts, electronic devices can also detect the ambient light level. If the ambient light level is lower than a preset brightness threshold, the device's background brightness will be increased. For example, the background brightness of the display screen can be increased from a third value to a fourth value. The third value refers to the background brightness of the display screen before the first prompt is displayed.

[0187] After brightening the background light of the electronic device, the face recognition process is executed based on the newly acquired image data a and image data b. For example, the process re-enters S1.

[0188] S3, based on the IR map of image data a, performs face recognition processing.

[0189] In some embodiments, the face processing module reads multiple pre-stored identity information entries and the corresponding authentication face features (second face features) from a secure storage chip. The authentication face features can be information pre-stored by the electronic device in response to user actions. For example, the authentication face information could be the face feature information entered into the electronic device when the user configures the device's unlock password (or account payment password). The identity information can indicate the identity of the user corresponding to the authentication face feature; different users have different identity information, and each identity information also corresponds to the permissions granted to that user.

[0190] Then, the face processing module can call the face recognition algorithm through the algorithm adaptation layer. Based on the face feature information extracted from the IR image of image data a (face feature information a, or the first face feature) and the authenticated face feature, a face feature comparison is performed. If face feature information a matches the authenticated face feature a successfully, and the identity information a corresponding to the authenticated face feature a has payment permissions, the face recognition result is considered successful. If face feature information a does not match any of the authenticated face features, the face recognition result is considered failed. If face feature information a matches the authenticated face feature a successfully, but the identity information a corresponding to the authenticated face feature a does not have payment permissions, the face recognition result is also considered failed.

[0191] In other possible embodiments, the electronic device may skip S1 and execute S2 after synchronously acquiring image data a and image data b. This application does not specifically limit this.

[0192] In practical applications, there are also scenarios where the IR map of image data 'a' can be verified by the user. For example, after a face recognition failure based on the first and second images, the user widens their eyes and re-participates in face recognition. Another example is a user widening their eyes during face recognition. Yet another example is a user with larger eyes participating in face recognition.

[0193] In the above scenarios, such as Figure 9 As shown, the steps for an electronic device to perform a face recognition method are as follows:

[0194] A1, based on the IR map of image data a, performs human eye gaze detection and human eye opening / closing detection.

[0195] In some embodiments, the implementation details of A1 above can be found in S1, and will not be repeated here. After the IR map based on image data a passes human eye gaze detection and human eye opening / closing detection, the process can proceed to A2. The IR map of image data a mentioned in A1 above can also be referred to as the second IR map.

[0196] A2, based on the IR map of image data a, performs face recognition processing.

[0197] In some embodiments, the implementation details of A2 described above can be found in S3, and will not be repeated here. Thus, compared to Figure 8 The scenario shown skips the steps of human eye gaze detection and human eye opening / closing detection based on image data b.

[0198] In some embodiments, before performing face recognition based on the IR image of image data a, the electronic device can also determine the validity of image data a and image data b. For example, through image feature alignment processing, it can be determined whether image data b is an image acquired synchronously with image data a and has not been tampered with. Another example is through anti-spoofing / liveness detection to determine whether the face appearing in the IR image of image data a is from a real person. If it is determined that image data a is an image obtained from photographing a real person, image data a is determined to be valid image data. If it is determined that image data a is not an image obtained from photographing a real person, image data a is determined to be invalid image data.

[0199] In an exemplary embodiment, after the IR map of image data a or image data b has been verified by user verification, if both image data a and image data b are determined to be valid, face recognition processing can be performed based on the IR map of image data a.

[0200] like Figure 10 As shown, before S3, it may also include:

[0201] B1 performs image feature alignment processing based on the IR maps of image data b and image data a.

[0202] For example, the above image feature alignment process may include:

[0203] (1) Process image data b according to the pre-defined pixel mapping relationship to obtain image data c.

[0204] Understandably, RGB camera sensors and TOF camera sensors are configured in different positions on electronic devices. When both sensors simultaneously capture images of the same object, due to differences in shooting position and angle, the image coordinates of pixels presenting the same content may differ between the image data acquired by the RGB camera sensor and the TOF camera sensor. The aforementioned pixel mapping relationship (first mapping relationship) can indicate the positional relationship between pixels presenting the same content in two frames of images acquired synchronously by the RGB and TOF camera sensors.

[0205] for example, Figure 11 As shown, when calibrating the pixel mapping relationship, while the RGB camera sensor acquires image data d, the TOF camera sensor is simultaneously controlled to acquire image data e. Pixel a in image data d and pixel b in image data e are pixels presenting the same content. Pixel a has image coordinates (2, 2) in image data d, and pixel b has image coordinates (3, 3) in image data e. Thus, the calibrated pixel mapping relationship includes the mapping coordinates (2, 2) and (3, 3) of image coordinates (2, 2). Of course, the pixel mapping relationship also includes the mapping coordinates of other image coordinates in the image data acquired by the RGB camera sensor.

[0206] When processing image data b using pixel mapping relationships, each pixel in image data b can be moved from its current image coordinates to its corresponding mapped coordinates according to the pixel mapping relationship, thus obtaining image data c. For example, the pixel with image coordinates (2,2) in image data b can be moved to (3,3).

[0207] (2) Compare the image coordinates of the pixels in the IR map of image data c with those in the IR map of image data a, and determine the alignment result between the IR maps of image data c and image data a.

[0208] For example, the image coordinates of each pixel in image region a displaying the eye in image data c (the third image) are obtained; that is, the first pixel in the third image is obtained. The image coordinates of each pixel in image region b displaying the eye in the IR image of image data a are obtained; that is, the second pixel in the first IR image is obtained. The number of pixels with different image coordinates between image region a and image region b is counted (1, the first number). If the number 1 exceeds a preset number (a preset number threshold), the alignment result is determined to not meet the preset condition. If the number 1 does not exceed the preset number, the alignment result is determined to meet the preset condition.

[0209] Understandably, image data synchronously acquired by the RGB image sensor and the TOF camera sensor, after pixel mapping processing, results in images where pixels displaying the same content have the same image coordinates. Utilizing this characteristic, if there are many pixels with different image coordinates between image regions a and b displaying the eye, there may be a problem of image data a and image data b being out of sync, or image data b or image data a being tampered with. In this scenario, image data b can be determined to be invalid. Conversely, if there are few pixels with different image coordinates between image regions a and b displaying the eye, it indicates that image data a and image data b were acquired synchronously. In this scenario, image data b can be determined to be valid.

[0210] If the alignment result meets the preset conditions, the process can proceed to step B2; alternatively, step B2 can be skipped, and the process can proceed directly to step S3. If the alignment result does not meet the preset conditions, the face recognition initiated by the payment application is deemed to have failed. After determining that the face recognition has failed, the electronic device can display a prompt message to remind the user to participate in face recognition again. For example, this prompt message can be a text reminder, such as displaying the text "Recognition failed. Please move closer to the display screen and face recognition again."

[0211] In addition to displaying prompts, the electronic device can also detect the ambient light level. If the ambient light level is lower than a preset brightness threshold, the background brightness of the electronic device is increased. After increasing the background brightness, the face recognition process is executed based on the newly acquired image data a and image data b. For example, the process re-enters S1.

[0212] B2, based on the IR map of image data a, performs anti-counterfeiting / liveness recognition.

[0213] In some embodiments, the face processing module can call anti-spoofing / liveness recognition algorithms through the algorithm adaptation layer to perform anti-spoofing / liveness recognition on the IR map of image data a. For implementation details, please refer to relevant technologies, which will not be elaborated here.

[0214] Understandably, if the IR image of image data a passes the anti-counterfeiting / liveness detection, then image data a is considered valid. In this scenario, the process can proceed to S3. If the IR image of image data a fails the anti-counterfeiting / liveness detection, then image data a is considered invalid. In this scenario, the face recognition initiated by the payment application can be deemed a failure.

[0215] In another exemplary embodiment, after the IR map of image data a or image data b has been verified by user authentication, if image data a is determined to be valid, face recognition processing can be performed based on the IR map of image data a. For example, in Figure 8 In the scenario shown, after user verification detection is performed on image data b, anti-spoofing / liveness recognition is performed based on the IR map of image data a. If the IR map of image data a passes the anti-spoofing / liveness recognition, face recognition processing is performed based on the IR map of image data a.

[0216] For example, in Figure 9 In the scenario shown, after user verification detection is performed on image data a, anti-spoofing / liveness recognition is conducted based on the IR map of image data a. If the IR map of image data a passes the anti-spoofing / liveness recognition, face recognition processing is performed based on the IR map of image data a.

[0217] As one implementation method, such as Figure 12 As shown, after the application initiates face recognition, the signaling interaction steps between the various software and hardware modules in the electronic device are as follows:

[0218] S101, the application sends a face recognition registration request to the face recognition service.

[0219] In some embodiments, when a business requirement for facial recognition arises, the application can send a facial recognition registration request to the facial recognition service through the facial recognition SDK. This facial recognition registration request can be used to register a corresponding callback for facial recognition on the facial recognition service, so that the application can obtain the facial recognition result.

[0220] for example, Figure 6 As shown, after receiving a confirmation of the transfer, the payment application generates a facial recognition service request. In response to this request, it sends a facial recognition registration request to the facial recognition service.

[0221] S102, the face recognition service responds to the face recognition registration request by sending a first request to the face recognition control module.

[0222] The first request mentioned above is a request instruction to initiate facial recognition.

[0223] In some embodiments, the face recognition service may respond to a face recognition registration request by instructing the execution of subsequent face recognition procedures. Furthermore, after the face recognition service obtains the face recognition result, it can use the aforementioned registration callback to feed the face recognition result back to the corresponding application via the face recognition SDK.

[0224] S103, the face recognition control module responds to the first request and identifies the target camera sensor.

[0225] Understandably, before performing face recognition, the face recognition control module needs to determine the camera sensor used to acquire image data, i.e., the target camera sensor. Typically, this target camera sensor can be the camera sensor of the front-facing camera of an electronic device. The lens of the front-facing camera is located on the same side as the display screen of the electronic device. The front-facing camera can include a camera configured with a TOF camera sensor and a camera configured with an RGB camera sensor.

[0226] For example, the face recognition control module can identify the TOF camera sensor in the front-facing camera as the target camera sensor. As another example, the face recognition control module can also identify the RGB camera sensor in the front-facing camera as the target camera sensor.

[0227] In some embodiments, the rule for selecting the target camera sensor may be based on a combination of ambient light intensity. Specific implementation details can be found in related technologies and will not be elaborated upon here.

[0228] S104, if the target camera sensor is a TOF camera sensor, the face recognition control module sends a second request to the camera service, which carries the camera identifiers of the TOF camera sensor and the RGB camera sensor.

[0229] S105, the camera service sends a second request to the camera HAL.

[0230] S106, in response to the second request, the camera HAL creates a transmission path for the TOF camera sensor and the RGB camera sensor.

[0231] In some embodiments, the face recognition control module can send a second request to the camera service through the vendor native development kit (VNDK) interface, and the camera service will pass the second request to the camera HAL.

[0232] In some embodiments, the camera HAL responds to a second request and creates a transmission path corresponding to the TOF camera sensor and a transmission path corresponding to the RGB camera sensor based on the camera identifiers (identifiers of the TOF camera sensor and the RGB camera sensor) in the second request. It is understood that the aforementioned transmission paths include at least a path corresponding to the sensor node and a path corresponding to the IFE node.

[0233] For example, the creation of the corresponding transmission path may include: creating sensor nodes and IFE nodes, and determining other software and hardware modules in the path corresponding to the sensor nodes and IFE nodes.

[0234] Taking the creation of a transmission path for an RGB camera sensor as an example, the path corresponding to the sensor node can be: sensor node - camera driver - RGB camera sensor - IFE module - IPE module - secure memory. The path corresponding to the IFE node can be: IFE module - camera driver - IFE node. After the above paths are created, the hardware in the paths is powered on (i.e., the hardware circuit is powered on), waiting for the image acquisition command from the face recognition control module to trigger the RGB camera sensor to acquire images and transmit them back.

[0235] Taking the creation of a transmission path for a TOF camera sensor as an example, the path corresponding to a sensor node can be: sensor node - camera driver - TOF camera sensor - IFE - Lite - secure memory. The path corresponding to an IFE node can be: IFE - Lite - camera driver - IFE node. The camera HAL can connect the output port of the sensor node and the input port of the IFE node at the HAL layer. Thus, the path corresponding to the sensor node and the path corresponding to the IFE node can form a closed loop. After the path is created, the hardware in the path is powered on (i.e., the hardware circuit is powered on), waiting for the synchronization pulse signal from the RGB camera sensor to trigger the TOF camera sensor to acquire and transmit images back.

[0236] S107, the camera HAL sends notification 1 to the camera service, indicating that the path was successfully created.

[0237] S108, the camera service sends a notification 1 to the face recognition control module.

[0238] In some embodiments, after successfully establishing the corresponding transmission paths between the TOF camera sensor and the RGB camera sensor, the camera HAL sends notification 1 (also known as a success notification) to the face recognition control module via the camera service, indicating that the transmission path for acquiring the image data required for face recognition has been established. In response to notification 1, the face recognition control module executes step S109.

[0239] In other embodiments, if the camera HAL fails to establish a transmission path corresponding to the TOF camera sensor or the RGB camera sensor, it sends a failure notification to the face recognition control module via the camera service, indicating that the transmission path establishment has failed. In this scenario, the face recognition control module can respond to the failure notification and re-initiate the establishment of the transmission channel.

[0240] S109, the face recognition control module sends an image acquisition command to the camera service.

[0241] S110, the camera service calls the camera HAL to instruct the RGB camera sensor to acquire images.

[0242] S111, the camera HAL, through the camera driver, instructs the RGB camera sensor to acquire images.

[0243] The aforementioned camera driver includes the camera driver corresponding to an RGB camera sensor, and may also include the camera driver corresponding to a TOF camera sensor.

[0244] In some embodiments, after S109 and before S110, the camera service can also instruct the camera driver to write the camera parameters required for operation to the RGB camera sensor and the TOF camera sensor through the camera HAL. For implementation details, please refer to the relevant technology, which will not be elaborated here.

[0245] S112, Camera driver instructs the RGB camera sensor to acquire image data.

[0246] S113, the RGB camera sensor acquires image data b.

[0247] In some embodiments, after writing camera parameters to the RGB camera sensor, the camera driver can send a start-up command to the RGB camera sensor, instructing the RGB camera sensor to start image data acquisition and send back the FD corresponding to the acquired image data.

[0248] S114, the RGB camera sensor writes the acquired image data b to secure memory and obtains FD2 indicating its storage location.

[0249] There is a correspondence between FD2 and the storage address of image data b in secure memory. Other software modules can use face recognition TA to query image data b based on FD2.

[0250] S115, the RGB camera sensor sends a synchronization pulse signal to the TOF camera sensor.

[0251] In some embodiments, there is no necessary order between S115 and S114. After the RGB camera sensor acquires image data b, it can immediately send a synchronization pulse signal to the TOF camera sensor.

[0252] S116, the TOF camera sensor acquires image data a.

[0253] S117, the TOF camera sensor writes the acquired image data a into the secure memory and obtains FD1 indicating its storage location.

[0254] S118, the RGB camera sensor sends notification 2 to the camera driver, carrying FD2.

[0255] Notification 2 above can indicate that the RGB camera sensor has acquired image data, such as image data b, and that image data b has been stored in secure memory. Notification 2 can carry the FD corresponding to image data b, such as FD2.

[0256] Furthermore, there is no necessary order between S118 and S115. After S114, that is, after the RGB camera sensor obtains FD2, it can send notification 2 to the camera driver.

[0257] S119, the TOF camera sensor sends notification 3 to the camera driver, carrying FD1.

[0258] The aforementioned notification 3 can indicate that the TOF camera sensor has acquired image data, such as image data a, and that image data a has been stored in secure memory. Notification 3 can also carry the FD corresponding to image data a, such as FD1.

[0259] S120, the camera driver sends FD1 and FD2 to the camera HAL.

[0260] In some embodiments, the camera driver of the RGB camera sensor sends FD2 to the camera HAL, and the camera driver of the TOF camera sensor sends FD1 to the camera HAL.

[0261] S121, the camera HAL sends FD1 and FD2 to the camera service.

[0262] In some embodiments, the camera HAL can send pairs of FD1 and FD2 to the camera service to facilitate the subsequent retrieval of synchronously acquired image data a and image data b. For example, after receiving an FD2, the camera HAL waits for an FD1 from the TOF camera sensor. After receiving FD1, it sends both FD1 and FD2 to the camera service. Alternatively, after receiving FD1 and FD2, the camera HAL searches for the target FD corresponding to each FD2 within FD1. The timestamp of the target FD indicates a later acquisition time than the timestamp of FD2. Furthermore, compared to other FD1s, the time interval between the timestamp of the target FD and the timestamp of FD2 is the shortest.

[0263] S122, the camera service sends FD1 and FD2 to the face recognition control module.

[0264] S123, the face recognition control module sends FD1 and FD2 to the face recognition TA.

[0265] S124, the face recognition TA retrieves image data a and image data b from secure memory based on FD1 and FD2.

[0266] S125, Face recognition TA determines the face recognition result based on image data a and image data b.

[0267] In some embodiments, the implementation details of S125 described above can be found in [reference]. Figure 8 , Figure 9 and Figure 10 The embodiments shown are not described in detail here.

[0268] S126, Authenticate TA to obtain face recognition results.

[0269] In some embodiments, the authentication TA also runs on the TEE side, and the authentication TA can periodically query the face recognition results from the face recognition TA. In other embodiments, after obtaining the face recognition results, the face recognition TA actively sends the face recognition results to the authentication TA.

[0270] S127, the certified TA sends the face recognition result to the face recognition service.

[0271] S128, The face recognition service sends the face recognition result to the application.

[0272] In some embodiments, the facial recognition service can send the facial recognition results back to all applications that have registered to obtain them. This allows applications to execute corresponding responses based on the facial recognition results. For example, if the facial recognition result is successful, the payment application can execute the actual transfer process. If the facial recognition result is unsuccessful, the payment application can instruct the display to show a prompt message, prompting the user to move closer to the display and re-participate in facial recognition. Alternatively, the payment application can also instruct the display to increase the backlight brightness.

[0273] In other embodiments, the display screen can be controlled to brighten the background light after the RGB camera sensor is powered on. This application does not specifically limit this aspect.

[0274] This application also provides an electronic device that may include a camera sensor, a display screen, a memory, and one or more processors. The memory and processors are coupled. The memory stores computer program code, which includes computer instructions. When the processor executes the computer instructions, it causes the electronic device to perform the various steps performed by the electronic device in the above embodiments. Of course, the electronic device includes, but is not limited to, the memory and one or more processors described above.

[0275] This application also provides a chip system that can be applied to the electronic devices described in the foregoing embodiments. The chip system includes at least one processor and at least one interface circuit. The processor may be the processor in the aforementioned electronic device. The processor and the interface circuit are interconnected via wiring. The processor can receive and execute computer instructions from the memory of the aforementioned electronic device through the interface circuit. When the computer instructions are executed by the processor, the electronic device can perform the various steps performed by the electronic device in the foregoing embodiments. Of course, the chip system may also include other discrete devices, and this application does not specifically limit this.

[0276] In some embodiments, as described above, those skilled in the art will clearly understand that, for the sake of convenience and brevity, the division of the functional modules described above is merely an example. In practical applications, the functions described above can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0277] In the embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0278] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0279] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. A face recognition method, characterized by, The method is applied to an electronic device configured with an RGB camera sensor and a TOF camera sensor, and comprises the following steps: receiving a first operation triggering the electronic device to perform face recognition; in response to the first operation, controlling the RGB camera sensor to capture a first image and the TOF camera sensor to capture a second image, the first image and the second image being image data synchronously captured by the RGB camera sensor and the TOF camera sensor; performing first detection based on the first image; wherein the first detection is an image detection item for an image region displaying eyes in the image data, and is used to detect whether the user has the intention to participate in the face recognition; in the case that the first image passes the first detection, performing face recognition based on a first infrared (IR) image of the second image.

2. The method of claim 1, wherein, A communication line is configured between the RGB camera sensor and the TOF camera sensor, and the RGB camera sensor captures the first image and the TOF camera sensor captures the second image in response to the first operation, comprising the following steps: in response to the first operation, the RGB camera sensor captures the first image and sends first synchronization information to the TOF camera sensor through the communication line; the TOF camera sensor captures the second image in response to the first synchronization information.

3. The method according to claim 1 or 2, characterized in that, Before the first detection based on the first image, the method further comprises the following steps: performing the first detection based on a first infrared (IR) image of the second image; wherein the first detection based on the first image comprises the following steps: in response to the first IR image failing to pass the first detection, performing the first detection based on the first image.

4. The method according to claim 1 or 2, characterized in that, The electronic device further comprises a display screen; after receiving the first operation, before the RGB camera sensor captures the first image and the TOF camera sensor captures the second image, the method further comprises the following steps: in response to the first operation, controlling the RGB camera sensor and the TOF camera sensor to be powered on; when the ambient light brightness in the space where the electronic device is located is not greater than a first threshold value, adjusting the background light brightness of the display screen from a first value to a second value; wherein the first value is the background light brightness of the display screen before the RGB camera sensor is powered on.

5. The method according to claim 1 or 2, characterized in that, After the first detection based on the first image, the method further comprises the following steps: in the case that the first image fails to pass the first detection, displaying first reminder information indicating that the face recognition fails.

6. The method of claim 5, wherein, After displaying the first reminder information, the method further comprises the following steps: when the ambient light brightness in the space where the electronic device is located is not greater than a first threshold value, adjusting the background light brightness of the display screen of the electronic device from a third value to a fourth value; wherein the third value is the background light brightness of the display screen before the first reminder information is displayed.

7. The method according to claim 1 or 2, characterized in that, The electronic device is pre-configured with a first mapping relationship, which indicates a positional relationship between pixel points presenting the same content in image data synchronously collected by the RGB camera sensor and the TOF camera sensor; Before the face recognition based on the second image, the method further includes: adjusting image coordinates of each pixel point in the first image according to the first mapping relationship to obtain a third image; obtaining a first pixel point of an image region displaying eyes in the third image, and obtaining a second pixel point of an image region displaying eyes in a first IR image of the second image; determining a first quantity according to image coordinates of the first pixel point and image coordinates of the second pixel point, the first quantity being a quantity of pixel points with different corresponding image coordinates between the first pixel point and the second pixel point; determining that the first quantity is not greater than a preset quantity threshold.

8. The method of claim 1 or 2, wherein, The face recognition based on the second image includes: extracting a first face feature from the first IR image of the second image; in a case where the first face feature matches a second face feature pre-configured, a result of the face recognition is successful; in a case where the first face feature does not match the second face feature, the result of the face recognition is failed.

9. The method of claim 8, wherein, in a case where the result of the face recognition is failed, the method further includes: controlling the RGB camera sensor to collect a fourth image and the TOF camera sensor to collect a fifth image, wherein the fourth image and the fifth image are image data synchronously collected by the RGB camera sensor and the TOF camera sensor; based on a second IR image of the fifth image, performing the first detection; after the second IR image passes the first detection, performing face recognition based on the second IR image.

10. The method of claim 1 or 2, wherein, Before the face recognition based on the second image, the method further includes: performing anti-counterfeiting / liveness detection on a first IR image of the second image; determining that a photographed object corresponding to the second image is a real person.

11. The method of claim 1 or 2, wherein, The first detection includes eye gaze detection and / or eye open / close detection.

12. The method of claim 3, wherein, The first detection includes eye gaze detection and / or eye open / close detection; The first detection based on the first infrared IR image of the second image includes: extracting eye feature information from the first IR image; in a case where the eye feature information fails to be extracted from the first IR image, determining that the first IR image fails to pass the eye gaze detection and / or the eye open / close detection.

13. The method of claim 12, wherein, The first detection based on the first infrared IR image of the second image further includes: in a case where first eye feature information is extracted from the first IR image, judging whether a gaze point of a user is located on a display screen of the electronic device based on the first eye feature information; in a case where it is determined that the gaze point of the user is located outside the display screen, determining that the first IR image fails to pass the eye gaze detection. Or, in the case of extracting the first eye feature information from the first IR image, determining whether the user's eye is in an open state based on the first eye feature information; In the case of determining that the user's eye is in a closed eye state, determining that the first IR image fails the human eye open-close detection.

14. The method of claim 1 or 2, wherein, The electronic device is configured with a secure memory; After the RGB camera sensor collects a first image and the TOF camera sensor collects a second image, the method further comprises: controlling the RGB camera sensor to write the first image to the secure memory, and controlling the TOF camera sensor to write the second image to the secure memory; Before the first detection based on the first image, the method further comprises: reading the first image and the second image from the secure memory.

15. An electronic device, comprising: The electronic device includes a display screen, a plurality of camera sensors, a memory, and one or more processors; the plurality of camera sensors includes a TOF camera sensor and an RGB camera sensor, the display screen, the camera sensor, the memory, and the processor are coupled; The display screen is used to display the image generated by the processor, the camera sensor is used to collect image data, the memory is used to store computer program code, the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method as claimed in any one of claims 1-14.

16. A readable storage medium, characterized by, The computer instructions, when executed on an electronic device, cause the electronic device to perform the method as claimed in any one of claims 1-14.

Images