Face recognition method and electronic equipment

Abstract

The embodiment of the invention provides a face recognition method and electronic equipment, and relates to the technical field of terminals. When a low-resolution TOF camera sensor is started, the accuracy of face recognition is guaranteed, and the success rate of face recognition is improved. According to the specific scheme, a first operation is received, and the first operation triggers the electronic equipment to carry out face recognition; in response to the first operation, the RGB camera sensor is controlled to collect a first image, the TOF camera sensor is controlled to collect a second image, and the first image and the second image are image data synchronously collected by the RGB camera sensor and the TOF camera sensor; performing first detection based on the first image; wherein the first detection is an image detection item for an image area displaying eyes in the image data, and is used for detecting whether a user has an intention of participating in face recognition; and under the condition that the first image passes the first detection, performing face recognition based on the second image.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a face recognition method and electronic device. Background Art

[0002] The TOF camera sensor includes a transmitter (TX) and a receiver (RX). TX is used to transmit infrared light or laser pulses, and RX is used to receive reflected light and form an image. Compared with RGB camera sensors, TOF camera sensors have more advantages in image acquisition in dark light scenes. Currently, TOF camera sensors have been widely used in various electronic devices.

[0003] However, for TOF camera sensors with lower resolution, when a face image containing squinting eyes is captured, it is very difficult to extract the eye feature information from the face image, resulting in the failure of the pre-detection of face recognition (such as eye gaze detection, eye open and closed detection, etc.), and thus leading to the failure of face recognition. Summary of the invention

[0004] The embodiments of the present application provide a face recognition method and an electronic device, which improve the success rate of face recognition when a low-resolution TOF camera sensor is enabled.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a face recognition method, which is applied to an electronic device, wherein an RGB camera sensor and a TOF camera sensor are configured in the electronic device, and the method comprises: receiving a first operation, wherein the first operation triggers the electronic device to perform face recognition. In response to the first operation, the RGB camera sensor and the TOF camera sensor can be controlled to output images synchronously. For example, the RGB camera sensor is controlled to capture a first image. The first image captured by the RGB camera sensor has a higher resolution and carries more abundant feature information of key points of the face (such as eyes). For another example, the TOF camera sensor can also be controlled to capture a second image, and the image quality of the second image captured by the TOF camera sensor is not affected by the ambient light brightness of the environment. The electronic device can perform a first detection based on the first image. Among them, the first detection is an image detection item for an image area showing eyes in the image data, which is used to detect whether the user has the intention to participate in face recognition. When the first image passes the first detection, face recognition is performed based on the second image.

[0007] It can be understood that the above-mentioned first detection, as a pre-detection of face recognition, can ensure the reliability of the face recognition result.

[0008] In the above embodiment, the first image and the second image are images collected synchronously, and the photographed object appears in the first image and the second image is the same. Using the first image with richer feature information of facial key points for the first detection can improve the fault tolerance of the first detection. Using the second image whose image quality is less affected by ambient light for face recognition can ensure the accuracy of the face recognition result. In this way, the electronic device improves the success rate of face recognition while ensuring the reliability of the face recognition result.

[0009] In some embodiments, a communication line is configured between the RGB camera sensor and the TOF camera sensor, and in response to the first operation, the RGB camera sensor captures a first image, and the TOF camera sensor captures a second image, including: in response to the first operation, the RGB camera sensor captures the first image, and the RGB camera sensor sends first synchronization information to the TOF camera sensor through the communication line; the TOF camera sensor captures the second image in response to the first synchronization information.

[0010] In the above embodiment, the synchronous image output of the RGB camera sensor and the TOF camera sensor is realized from the hardware layer, and the synchronization effect is better, thereby ensuring the reliability of the first detection and face recognition performed by the subsequent fusion of the first image and the second image.

[0011] In some embodiments, before performing a first detection based on the first image, the method further includes: performing a first detection based on a first infrared IR map of the second image; wherein, performing the first detection based on the first image includes: in response to the first IR map failing the first detection, performing a first detection based on the first image.

[0012] In some embodiments, when the result of face recognition is failure, the method also includes: controlling the RGB camera sensor to capture a fourth image and the TOF camera sensor to capture a fifth image, wherein the fourth image and the fifth image are image data synchronously captured by the RGB camera sensor and the TOF camera sensor; performing the first detection based on the second IR map of the fifth image; and performing face recognition based on the second IR map after the second IR map passes the first detection.

[0013] It is understandable that if the eyes of the face appearing in the second image are smaller, the possibility that the IR map of the second image cannot pass the first detection is higher. If the eyes of the face appearing in the second image are larger, the possibility that the IR map of the second image passes the first detection is higher.

[0014] In an exemplary scenario, after applying the above method, if the user with squinting eyes performs the first operation and blocks the RGB camera sensor, face recognition fails. If the user with squinting eyes performs the first operation and does not block the RGB camera sensor, face recognition succeeds. In contrast, if the user with wide eyes performs the first operation, face recognition succeeds regardless of whether the RGB camera sensor is blocked.

[0015] In some embodiments, the electronic device also includes a display screen; after receiving the first operation, the RGB camera sensor captures the first image, and before the TOF camera sensor captures the second image, the method also includes: in response to the first operation, controlling the RGB camera sensor and the TOF camera sensor to power on; when the ambient light brightness in the space where the electronic device is located is not greater than a first threshold, adjusting the background light brightness of the display screen from a first value to a second value, wherein the first value is the background light brightness of the display screen before the RGB camera sensor is powered on.

[0016] In the above embodiment, when the ambient light brightness in the space where the electronic device is located is not greater than the first threshold, the image quality of the image data collected by the RGB camera sensor will be affected. By brightening the background light brightness of the display screen and using the background light to perform local fill light on the user's face, the image quality of the image area showing the key points of the face in the collected first image is guaranteed, and the misjudgment rate of the first detection is reduced.

[0017] In some embodiments, after performing a first detection based on the first image, the method further includes: if the first image fails the first detection, displaying a first reminder message indicating that face recognition has failed.

[0018] In the above embodiment, when both the first image and the second image fail the first detection, it is determined that the face recognition has failed, indicating that there is a risk in the face recognition. In this scenario, there is no need to continue face recognition based on the second image, so as to ensure the reliability of the face recognition result.

[0019] In some embodiments, after displaying the first reminder message, the method further includes: when the ambient light brightness in the space where the electronic device is located is not greater than a first threshold, adjusting the background light brightness of the display screen from a third value to a fourth value; wherein the third value is the background light brightness of the display screen before displaying the first reminder message.

[0020] In some embodiments, by increasing the background light brightness, the impact of low ambient light brightness on the image data collected by the RGB camera sensor is reduced, thereby preventing the image data collected by the RGB camera sensor from failing to pass the first detection due to the ambient light brightness.

[0021] In some embodiments, a first mapping relationship is pre-configured in the electronic device, and the first mapping relationship indicates the positional relationship between pixels presenting the same content in the image data synchronously collected by the RGB camera sensor and the TOF camera sensor; before performing face recognition based on the second image, the above method also includes: adjusting the image coordinates of each pixel in the first image according to the first mapping relationship to obtain a third image; obtaining a first pixel in the image area showing the eye in the third image, and obtaining a second pixel in the image area showing the eye in the first IR image of the second image; determining a first number based on the image coordinates of the first pixel and the image coordinates of the second pixel, the first number being the number of pixels between the first pixel and the second pixel whose corresponding image coordinates are different; and determining that the first number is not greater than a preset number threshold.

[0022] In the above embodiment, when the first number between the first IR images of the third image corresponding to the first image and the second image is not greater than the preset number threshold, it is determined that the second image can be aligned with the first image, and the first image and the second image can be aligned, and the synchronization between the first image and the second image can be verified. In this way, a malicious third party is prevented from circumventing the first detection by injecting image data.

[0023] In some embodiments, face recognition is performed based on the second image, including: a first facial feature extracted from a first IR map of the second image; when the first facial feature matches a pre-configured second facial feature, the result of face recognition is success; when the first facial feature does not match the second facial feature, the result of face recognition is failure.

[0024] In some embodiments, before performing face recognition based on the second image, the method further includes: performing anti-counterfeiting / liveness detection on the first IR image of the second image; and determining that the photographed object corresponding to the second image is a real person. In this way, the reliability of the face recognition result can be improved.

[0025] In some embodiments, the first detection includes human eye gaze detection and / or human eye opening and closing detection.

[0026] In some embodiments, a first detection is performed based on a first infrared IR map of the second image, including: extracting eye feature information from the first IR map; in case the extraction of eye feature information from the first IR map fails, determining that the first IR map fails to pass the human eye gaze detection and / or the human eye open and closed detection.

[0027] In some embodiments, performing a first detection based on the first infrared IR map of the second image also includes: in a case where first eye feature information is extracted from the first IR map, judging whether the user's gaze point is located on the display screen of the electronic device based on the first eye feature information; in a case where it is determined that the user's gaze point is outside the display screen, determining that the first IR map has not passed the human eye gaze detection; or, in a case where the first eye feature information is extracted from the first IR map, judging whether the user's eyes are in an open state based on the first eye feature information; in a case where it is determined that the user's eyes are in a closed state, determining that the first IR map has not passed the human eye open and closed detection.

[0028] In some embodiments, a secure memory is configured in the electronic device. After the RGB camera sensor captures the first image and the TOF camera sensor captures the second image, the method further includes: controlling the RGB camera sensor to write the first image into the secure memory, and controlling the TOF camera sensor to write the second image into the secure memory; before performing a first detection based on the first image, the method further includes: reading the first image and the second image from the secure memory.

[0029] In the above embodiment, the security of the images captured by the camera sensor is improved by using the secure memory.

[0030] In a second aspect, an electronic device provided by an embodiment of the present application includes a display screen, multiple camera sensors, a memory and one or more processors; the multiple camera sensors include a TOF camera sensor and an RGB camera sensor, and the display screen, the camera sensor, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, the camera sensor is used to collect image data, and the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, one or more processors are used to: receive a first operation, the first operation triggers the electronic device to perform face recognition; in response to the first operation, control the RGB camera sensor to collect a first image, and the TOF camera sensor to collect a second image, the first image and the second image are image data collected synchronously by the RGB camera sensor and the TOF camera sensor; perform a first detection based on the first image; wherein the first detection is an image detection item for an image area showing eyes in the image data, which is used to detect whether the user has the intention to participate in face recognition; if the first image passes the first detection, perform face recognition based on the second image.

[0031] In some embodiments, a communication line is configured between the RGB camera sensor and the TOF camera sensor, and one or more processors are used to: in response to the first operation, instruct the RGB camera sensor to capture the first image, and instruct the RGB camera sensor to send the first synchronization information to the TOF camera sensor through the communication line. In this way, the TOF camera sensor can capture the second image in response to the first synchronization information.

[0032] In some embodiments, before performing a first detection based on the first image, one or more processors are used to: perform a first detection based on a first infrared IR map of the second image; wherein, performing the first detection based on the first image includes: in response to the first IR map failing the first detection, performing a first detection based on the first image.

[0033] In some embodiments, the electronic device also includes a display screen; after receiving the first operation, the RGB camera sensor captures the first image, and before the TOF camera sensor captures the second image, one or more processors are used to: in response to the first operation, control the RGB camera sensor and the TOF camera sensor to power on; when the ambient light brightness in the space where the electronic device is located is not greater than a first threshold, adjust the background light brightness of the display screen from a first value to a second value; wherein the first value is the background light brightness of the display screen before the RGB camera sensor is powered on.

[0034] In some embodiments, after the first detection is performed based on the first image, one or more processors are used to: display a first reminder message if the first image fails the first detection, and the first reminder message indicates that the face recognition failed.

[0035] In some embodiments, after displaying the first reminder message, one or more processors are used to: when the ambient light brightness in the space where the electronic device is located is not greater than a first threshold, adjust the background light brightness of the display screen from a third value to a fourth value; wherein the third value is the background light brightness of the display screen before displaying the first reminder message.

[0036] In some embodiments, a first mapping relationship is preconfigured in the electronic device, and the first mapping relationship indicates the positional relationship between pixels presenting the same content in the image data synchronously collected by the RGB camera sensor and the TOF camera sensor; before performing face recognition based on the second image, one or more processors are used to: adjust the image coordinates of each pixel in the first image according to the first mapping relationship to obtain a third image; obtain a first pixel in the image area showing the eye in the third image, and obtain a second pixel in the image area showing the eye in the first IR image of the second image; determine a first number based on the image coordinates of the first pixel and the image coordinates of the second pixel, the first number being the number of pixels between the first pixel and the second pixel whose corresponding image coordinates are different; and determine that the first number is not greater than a preset number threshold.

[0037] In some embodiments, one or more processors are used to: extract a first facial feature from a first IR map of a second image; when the first facial feature matches a pre-configured second facial feature, the result of facial recognition is success; when the first facial feature does not match the second facial feature, the result of facial recognition is failure.

[0038] In some embodiments, when the result of face recognition is failure, one or more processors are used to: control the RGB camera sensor to capture a fourth image and the TOF camera sensor to capture a fifth image, wherein the fourth image and the fifth image are image data synchronously captured by the RGB camera sensor and the TOF camera sensor; perform a first detection based on the second IR map of the fifth image; and perform face recognition based on the second IR map after the second IR map passes the first detection.

[0039] In some embodiments, before performing face recognition based on the second image, one or more processors are used to: perform anti-counterfeiting / liveness detection on the first IR image of the second image; and determine that the subject corresponding to the second image is a real person.

[0040] In some embodiments, the first detection includes human eye gaze detection and / or human eye opening and closing detection.

[0041] In some embodiments, one or more processors are used to: extract eye feature information from the first IR image; if the extraction of eye feature information from the first IR image fails, determine that the first IR image has not passed the human eye gaze detection and / or human eye open and closed detection.

[0042] In some embodiments, one or more processors are used to: when first eye feature information is extracted from a first IR map, determine whether the user's gaze point is located on the display screen of the electronic device based on the first eye feature information; when it is determined that the user's gaze point is outside the display screen, determine that the first IR map does not pass the human eye gaze detection; or, when first eye feature information is extracted from the first IR map, determine whether the user's eyes are in an open state based on the first eye feature information; when it is determined that the user's eyes are in a closed state, determine that the first IR map does not pass the human eye open and closed detection.

[0043] In some embodiments, a secure memory is configured in the electronic device; after the RGB camera sensor captures the first image and the TOF camera sensor captures the second image, one or more processors are used to: control the RGB camera sensor to write the first image into the secure memory, and control the TOF camera sensor to write the second image into the secure memory; before performing the first detection based on the first image, one or more processors are used to: read the first image and the second image from the secure memory.

[0044] In a third aspect, an embodiment of the present application provides a computer storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method in the above-mentioned first aspect and possible embodiments thereof.

[0045] In a fourth aspect, the present application provides a computer program product. When the computer program product is run on the above-mentioned electronic device, the electronic device executes the method in the above-mentioned first aspect and possible embodiments thereof.

[0046] It can be understood that the electronic devices, computer storage media and computer program products provided in the above-mentioned aspects are all applied to the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 An example diagram of the working principle of the TOF camera sensor provided in the embodiment of the present application;

[0048] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0049] Figure 3 An example diagram of the soft and hard structures of an electronic device provided in an embodiment of the present application;

[0050] Figure 4An example diagram of the instruction transmission between various modules on the REE side after the application provided in the embodiment of the present application initiates face recognition;

[0051] Figure 5 This is an example diagram of data transmission between various modules on the REE side and the TEE side after the camera sensor provided in an embodiment of the present application starts streaming;

[0052] Figure 6 An example diagram of a scenario in which a payment application initiates face recognition provided in an embodiment of the present application;

[0053] Figure 7 An example diagram of the change in background brightness of a display screen before and after initiating face recognition in a dark light scene provided in an embodiment of the present application;

[0054] Figure 8 One of the step flow charts of the face recognition method provided in the embodiment of the present application;

[0055] Fig. 9 The second step flow chart of the face recognition method provided in the embodiment of the present application;

[0056] Fig.10 The third step flow chart of the face recognition method provided in the embodiment of the present application;

[0057] Fig.11 An example diagram of the pixel mapping relationship between the RGB camera sensor and the TOF camera sensor provided in the embodiment of the present application;

[0058] Fig.12 A signaling interaction diagram of the face recognition method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0060] In order to make the following embodiments clear and concise, a brief introduction to related concepts or technologies is first given:

[0061] A time of flight (TOF) camera sensor may include a transmitter (TX) and a receiver (RX). Figure 1As shown in the figure, TX is used to transmit infrared light or laser pulses, and RX is used to receive reflected light and form an image. After TX emits infrared light or laser pulses and RX receives the corresponding reflected light, the TOF camera sensor can use TOF imaging technology to form an image.

[0062] The above-mentioned TOF imaging technology refers to the imaging technology that the TOF camera sensor emits a group of infrared light (or laser pulses) that are invisible to the human eye, which are reflected after encountering an object and reflected back to the camera sensor, and the time difference or phase difference from emission to reflection back to the TOF camera sensor is calculated, and the data is collected to form a set of distance depth data, thereby obtaining a three-dimensional 3D model. In other words, TOF imaging technology is based on the traditional 2D XY axis imaging, adding depth information from the Z axis direction, and finally generating 3D image information.

[0063] An RGB camera sensor, that is, a camera sensor having three color channels: red (R), green (G), and blue (B), can obtain an image including multiple colors by changing the three color channels and superimposing them on each other.

[0064] Face recognition technology is a biometric technology that identifies an individual based on facial feature information. Exemplarily, the implementation principle of face recognition technology is as follows: facial feature information is extracted from image data containing a face, and then the extracted facial feature information is compared with the facial feature information corresponding to the pre-configured identity information. If the degree of match between the two reaches a preset standard, it is determined that the face in the image data belongs to the person indicated by the above identity information, which can also be called a successful face recognition. If the degree of match between the two does not reach the preset standard, it is determined that the face appearing in the image data does not belong to the person indicated by the above identity information, which can also be called a failed face recognition. For specific details, please refer to the relevant technology, which will not be repeated here.

[0065] In some embodiments, the image data containing faces collected by the TOF camera sensor and the RGB camera sensor can be used for face recognition processing. Of course, in the face recognition process, the image data collected by the TOF camera sensor and the RGB camera sensor each have their own advantages. For example, the image data collected by the TOF camera sensor in a dark scene can still meet the requirements of face recognition. The image data collected by the RGB camera sensor has a higher resolution and contains richer image information. Based on the image data collected by the RGB camera sensor, various types of image recognition are performed, and the recognition results are more accurate.

[0066] In some embodiments, since image data containing faces involves the user's privacy and security information, the image data collected by the TOF camera sensor and the RGB camera sensor for face recognition needs to be isolated and protected using the secure memory in the rich execution environment (REE) + trusted execution environment (TEE) architecture.

[0067] The above-mentioned rich execution environment (REE), which can also be called a rich execution environment, a common execution environment, or an untrusted execution environment, refers to the system operating environment of an electronic device, and the above-mentioned system can refer to an operating system. REE has good openness and extensibility.

[0068] The above-mentioned trusted execution environment (TEE), also known as the security side or security zone, is an area that requires authorization to access. TEE refers to an independent security area established inside the processor through hardware resource isolation. TEE and REE coexist in the operating environment of electronic devices, but TEE and REE are completely isolated in hardware. TEE has its own operating space and is equipped with strict protection measures. Therefore, it has a higher security level than REE, and REE cannot access the relevant resources of TEE, so that TEE has the security ability to protect the internal code and information of TEE, and can resist attacks on the REE side and resist security threats.

[0069] REE+TEE architecture refers to an architecture that provides services for applications through the combination of TEE and REE. In other words, TEE and REE exist together in electronic devices. For example, TEE can implement an operating mechanism isolated from REE through hardware support. TEE has its own operating space, with a higher security level than REE, and can protect assets in TEE (such as data, software, etc.) from software attacks. Only authorized security software can be executed in TEE, and it also protects the confidentiality of the resources and data of the security software. Compared with REE, TEE can better protect the security of data and resources due to its protection mechanisms such as isolation and permission control.

[0070] A trusted application (TA) is an application running in TEE that can provide security services to client applications (CA) running in REE, such as key generation and key management, security authentication, face recognition and other security tasks.

[0071] CA usually refers to an application running in REE, which can call TA through the client application programming interface (API) and instruct TA to perform corresponding security operations.

[0072] Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0073] like Figure 2 As shown, the electronic device (such as a mobile phone) may include: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a SIM card interface 295, etc.

[0074] Among them, the above-mentioned sensor module 280 may include sensors such as pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor and bone conduction sensor.

[0075] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0076] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0077] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0078] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0079] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0080] It is understandable that the interface connection relationship between the modules illustrated in this embodiment is only a schematic illustration and does not constitute a structural limitation of the electronic device. In other embodiments, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0081] The charging management module 240 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. While the charging management module 240 charges the battery 242, it can also power the electronic device through the power management module 241.

[0082] The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the external memory, the display screen 294, the camera 293, and the wireless communication module 260. In some embodiments, the power management module 241 and the charging management module 240 can also be set in the same device.

[0083] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor, etc. In some embodiments, the antenna 1 of the electronic device is coupled with the mobile communication module 250, and the antenna 2 is coupled with the wireless communication module 260, so that the electronic device can communicate with the network and other devices through wireless communication technology, such as communication with wearable devices.

[0084] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0085] The mobile communication module 250 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to electronic devices. The mobile communication module 250 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation.

[0086] The mobile communication module 250 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some functional modules of the mobile communication module 250 can be set in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 250 can be set in the same device as at least some modules of the processor 210.

[0087] The wireless communication module 260 can provide wireless communication solutions for electronic devices including WLAN (such as (wirelessfidelity, Wi-Fi) network), Bluetooth (bluetooth, BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), nearfield communication technology (nearfield communication, NFC), infrared technology (infrared, IR), etc.

[0088] Among them, GNSS may include Beidou navigation satellite system (BDS), global positioning system (GPS), global navigation satellite system (GLONASS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation systems (SBAS).

[0089] The wireless communication module 260 may be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 210. The wireless communication module 260 may also receive signals to be sent from the processor 210, modulate the frequencies of the signals, amplify the signals, and convert the signals into electromagnetic waves for radiation via the antenna 2.

[0090] The electronic device implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0091] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel.

[0092] The electronic device can realize the shooting function through the ISP, the camera 293, the video codec, the GPU, the display screen 294 and the application processor, etc. The ISP is used to process the data fed back by the camera 293.

[0093] The camera 293 is used to capture static images or videos. In some embodiments, the electronic device may include 1 or N cameras 293, where N is a positive integer greater than 1. The N cameras 293 may include a camera with a TOF camera sensor and a camera with an RGB camera sensor.

[0094] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.

[0095] The internal memory 221 may be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 221. For example, in an embodiment of the present application, the processor 210 may execute the instructions stored in the internal memory 221, and the internal memory 221 may include a program storage area and a data storage area.

[0096] The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0097] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0098] The software system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes a system with a layered architecture as an example to exemplify the software structure of the electronic device.

[0099] Figure 3 It is a block diagram of the software and hardware structure of the electronic device according to an embodiment of the present invention.

[0100] The layered architecture of electronic devices divides software and hardware into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the electronic device may include, for example, an application layer, an application framework layer, a hardware abstraction layer (HAL), a kernel layer, and a hardware layer. Of course, it may also include Figure 3 Layers not shown in the diagram, such as the runtime and system libraries.

[0101] Exemplarily, the application layer may include a series of applications.

[0102] like Figure 3 As shown, the application layer may include setting applications, payment applications, lock screen applications, face recognition development kits (software development kits, SDKs), etc.

[0103] Among them, the face recognition SDK can interact with the application framework layer to indicate the face recognition process, for example, sending a face recognition request to the application framework layer.

[0104] The above-mentioned setting application, payment application, and lock screen application may be examples of applications installed in the electronic device. In some other embodiments, the application layer may also include Figure 3 Applications not shown include, for example, social applications, calls, cameras, maps, navigation, face recognition development kit (SDK), and other applications.

[0105] In some embodiments, the above-mentioned setting application, payment application, lock screen application and other applications may generate business needs for face recognition during operation. For example, when the electronic device is running the setting application, after detecting the user's instruction to change the password, a business need to perform face recognition is generated. For another example, when the electronic device is running the payment application, after detecting the user's instruction to pay, a business need to perform face recognition is generated. For another example, after detecting the user's instruction to unlock, the lock screen application may also generate a business need to perform face recognition.

[0106] After the application generates a business demand for face recognition, the face recognition SDK can be called to instruct the application framework layer to execute the face recognition process through the face recognition SDK. Among them, the operation that triggers the application to initiate face recognition can be called the first operation.

[0107] Exemplarily, the application framework layer provides an application programming interface (API) and a programming framework for the application programs in the application layer. The application framework layer includes some predefined functions.

[0108] like Figure 3 As shown, the application framework layer may include face recognition services and camera services, etc.

[0109] The face recognition service is used to process the face recognition requests sent by the face recognition SDK and return the processing results to the application to implement the face recognition function.

[0110] The camera service is used to manage the work of the camera sensor. For example, it can control the camera sensor to be turned on and off, and control the resolution, frame rate and other parameters of the camera sensor.

[0111] In other embodiments, the application framework layer may also include Figure 3 Modules not shown in the figure, such as window manager, content provider, view system, resource manager, notification manager, etc.

[0112] Exemplarily, the hardware abstraction layer (HAL) can encapsulate the driver in the kernel layer and provide a calling interface to the application framework layer, shielding the implementation details of the low-level hardware.

[0113] like Figure 3 As shown, the HAL of the electronic device includes at least a camera HAL and a face recognition control module.

[0114] The camera HAL is the core software framework of the camera, which can include sensor nodes and image front end (IFE) nodes. The sensor nodes and IFE nodes are components (nodes) in the transmission path (also called transmission pipeline) of image data and control instructions created by the camera HAL.

[0115] The face recognition control module is the core software application of face recognition. The face recognition control module includes a face recognition client application (CA), and the face recognition CA runs in the REE environment.

[0116] In addition, in the TEE environment, a face recognition trusted application (TA) corresponding to the face recognition CA is running. Exemplarily, the face recognition CA can communicate with the face recognition TA and instruct the face recognition TA to perform image processing tasks related to face recognition, such as image conversion, image detection, feature extraction, liveness detection, feature comparison, etc.

[0117] In addition, the face recognition control module may also include: a camera control module and a face recognition HAL. The camera control module may be used to control the camera service to perform a specified type of task, such as instructing the camera service to call the camera HAL to create a transmission channel.

[0118] The above-mentioned face recognition HAL can call algorithm modules related to face recognition, such as TOF processing algorithm, face detection algorithm, etc., and perform image processing tasks related to face recognition on the REE side. Among them, the above-mentioned TOF processing algorithm can be used to convert the image data collected by the TOF camera sensor into a TOF infrared (infrared radiation, IR) image, and the above-mentioned face detection algorithm can detect whether a face appears in the image data, etc. It can be understood that the electronic device can perform image processing tasks related to face recognition on both the TEE side and the REE side. In some embodiments, it can be pre-configured to perform image processing tasks related to face recognition on the TEE side to ensure the security of face recognition. In other embodiments, image processing tasks related to face recognition can also be performed on the REE side, and the embodiments of the present application are not specifically limited to this.

[0119] like Figure 3 As shown in Figure 1, the kernel layer is the layer between hardware and software. The kernel layer contains at least the camera driver. In addition, the kernel layer can also include Figure 3 Drivers not shown, such as display driver, sensor driver, etc.

[0120] Among them, the camera driver is the driver layer of the Camera device, which is mainly responsible for the interaction with the camera sensor.

[0121] like Figure 3 As shown, the hardware layer includes display, TOF camera sensor, RGB camera sensor, IFE module, IPE module, IFE-Lite module, secure buffer, etc.

[0122] Among them, the image front end (IFE) module can perform image processing such as automatic exposure algorithm (AE) and automatic white balance (AWB) on the image data.

[0123] The lightweight image pre-processing (IFE-Lite) module is a simplified version of the IFE module and can be used to forward image data without processing the image data during the forwarding process.

[0124] The image processing engine (IPE) module is used to perform hardware noise reduction, cropping, noise reduction, color processing, detail enhancement and other image processing tasks on the image. Among them, hardware noise reduction includes, for example, multi-frame noise reduction (MFNR) and multi-frame super resolution (MFSR).

[0125] Secure memory refers to a memory with security protection function, which can be used to store image data collected by TOF camera sensors and RGB camera sensors. In some embodiments, the image acquired by the TOF camera sensor is forwarded by the IFE-Lite module and stored in the secure memory, and the image acquired by the RGB camera sensor can be passed to the IPE module after image pre-processing by the IFE module, and stored in the secure memory by the IPE module. The above-mentioned secure memory can shield the software module on the REE side, so that the software module on the REE side cannot directly access the data in the secure memory, thereby improving the security of the data.

[0126] In some embodiments, the application on the REE side can control the RGB camera sensor or the TOF camera sensor to collect image data for face recognition according to business needs.

[0127] like Figure 4 As shown, when an application (such as a setting application, a payment application, or a lock screen application) generates a business need for face recognition during operation, it can request the face recognition service from the application framework layer. For example, the application can call the face recognition SDK, and the face recognition SDK responds to the call of the application and sends a face recognition request to the face recognition service in the application framework layer.

[0128] like Figure 4 As shown, in response to the face recognition request, the face recognition service can instruct the TOF camera sensor or the RGB camera sensor to power on and collect image data for face recognition through the HAL layer. For example, in response to the face recognition request, the face recognition service can call the face recognition control module of the HAL layer.

[0129] The face recognition control module controls the power-on of a specified camera sensor (e.g., an RGB camera sensor or a TOF camera sensor) through the camera service in response to the call of the face recognition service. As an implementation method, the face recognition control module can interact with the camera service by calling an interface (vendor native development kit, VNDK).

[0130] like Figure 4 As shown, the camera service can interact with the camera HAL in the HAL layer in response to the instruction of the face recognition control module, and instruct the camera HAL to create a transmission path corresponding to the specified camera sensor, which can be used to control the corresponding camera sensor.

[0131] Exemplarily, the transmission path includes at least a sensor node, an IFE node, a designated camera sensor, and a corresponding camera driver. It is understandable that the sensor nodes and IFE nodes in the transmission path corresponding to different camera sensors may be different. The creation of the transmission path corresponding to the designated camera sensor may include: creating a sensor node and an IFE node corresponding to the designated camera sensor, and waking up the designated camera sensor.

[0132] In addition, the sensor node and the IFE node can interact with the camera driver in the kernel layer. For example, the sensor node can determine the camera parameters required when the specified camera sensor enables the preset working mode, and write the determined camera parameters to the specified camera sensor through the camera driver, so that the specified camera sensor collects image data according to the preset working mode (for example, face collection mode).

[0133] In some embodiments, the electronic device is in a bright light scene, and the designated camera sensor may be an RGB camera sensor. The electronic device is in a dark light scene, and the designated camera sensor may be a TOF camera sensor. In addition, the electronic device may determine whether it is currently in a bright light scene or a dark light scene based on the detected ambient light brightness. For example, if the ambient light brightness value is detected to be greater than a preset brightness threshold, the environment is determined to be a bright light scene. For another example, if the ambient light brightness value is detected to be not greater than a preset brightness threshold, the environment is determined to be a dark light scene. In this way, in different ambient light brightness scenes, it can be ensured that the collected image data meets the image quality requirements of face recognition.

[0134] At present, electronic devices actually need to perform one or more verification-type image detections before performing face recognition based on image data. Among them, the above-mentioned one or more verification-type image detections can also be called user verification detection. The one or more verification-type image detections can be used to determine whether the user has the intention to participate in face recognition, so as to avoid the user's facial information being stolen by a malicious third party for face recognition without knowing it. For example, when the user is not looking at the display screen, a malicious third party secretly takes a picture of the user's face for face recognition. For another example, when the user closes his eyes and sleeps, a malicious third party uses the user's electronic device to secretly take a picture of the user's face for face recognition. In order to avoid the above-mentioned malicious secretly taking pictures for face recognition, the above-mentioned one or more verification-type image detection functions can be set in electronic devices.

[0135] Exemplarily, the user verification detection may be based on image detection of an image region containing human eyes, such as human eye gaze detection, human eye open / closed detection, and the like.

[0136] However, when the designated camera sensor is a TOF camera sensor, if the user's eyes are small or the user squints, the image data collected by the TOF camera sensor will be difficult to pass the above-mentioned user verification detection due to its low resolution, resulting in the failure of the face recognition triggered by the application this time.

[0137] In order to improve the above problems, an embodiment of the present application provides a face recognition method, which is applied to an electronic device. Among them, the electronic device can control the TOF camera sensor and the RGB camera sensor to output images synchronously. In this way, user verification detection can be performed based on the image data synchronously collected by the RGB camera sensor. If the image data collected by the RGB camera sensor passes the user verification detection, face recognition is performed based on the image data collected by the TOF camera sensor. While ensuring the security of face recognition, the success rate of face recognition is improved.

[0138] In conjunction with the accompanying drawings, based on the above-mentioned software and hardware structure, the implementation details of the face recognition method provided in the embodiment of the present application are introduced.

[0139] In an embodiment of the present application, when the designated camera sensor is TOF, the camera service may instruct the camera HAL to separately create a transmission path for the RGB camera sensor and a transmission path for the TOF camera sensor.

[0140] Among them, the transmission path of the above-mentioned RGB camera sensor can be composed of an RGB camera sensor, a sensor node corresponding to the RGB camera sensor, a corresponding IFE node, a corresponding camera driver, an IFE module, an IPE module, and a secure memory. The details of creating this path can refer to the subsequent embodiments, which will not be repeated here. The transmission path of the above-mentioned TOF camera sensor can be composed of a TOF camera sensor, a sensor node corresponding to the TOF camera sensor, a corresponding IFE node, a corresponding camera driver, an IFE module, an IPE module, and a secure memory. The details of creating this path can refer to the subsequent embodiments, which will not be repeated here.

[0141] After creating the transmission path for the RGB camera sensor and the transmission path for the TOF camera sensor, control the hardware modules in the transmission path to power on. For example, Figure 4 As shown, the RGB camera sensor is powered on by controlling the camera driver corresponding to the RGB camera sensor, and the TOF camera sensor is powered on by controlling the camera driver corresponding to the TOF camera sensor.

[0142] After the RGB camera sensor and the TOF camera sensor are powered on, the RGB camera sensor and the TOF camera sensor start to collect image data.

[0143] For example, Figure 5 As shown, a communication line is configured between the RGB camera sensor and the TOF camera sensor. The communication line can be used to transmit a synchronization pulse signal, such as an XVS signal. The synchronization pulse signal is used to support the RGB camera sensor and the TOF camera sensor to synchronously collect image data. In addition, the synchronization pulse signal can also be called the first synchronization information.

[0144] Among them, the RGB camera sensor can be the sending end of the synchronization pulse signal, and the TOF camera sensor can be the receiving end of the synchronization pulse signal. After the RGB camera sensor and the TOF camera sensor are powered on, each time the RGB camera sensor collects a frame of image data (such as image data b), it will send a synchronization pulse signal to the TOF camera sensor, indicating that the TOF camera sensor also synchronously collects a frame of image data (such as image data a). In this way, the image data collected by the RGB camera sensor and the TOF camera sensor can maintain a one-to-one correspondence, and the RGB camera sensor and the TOF camera sensor realize acquisition synchronization at the hardware layer. It should be noted that there will be a small delay between the above-mentioned RGB camera sensor collecting image data b and the TOF camera sensor collecting image data a, which can be ignored and it is considered that their acquisition is synchronized.

[0145] For example, Figure 5As shown, the RGB camera sensor collects image data b and can send a synchronization pulse signal to the TOF camera sensor. In response to the synchronization pulse information, the TOF camera sensor can collect a frame of image data, such as image data a. The image data a and image data b mentioned in the subsequent embodiments may refer to two frames of image data synchronously collected by the RGB camera sensor and the TOF camera sensor. For example, image data a may include the second image and the fifth image. Image data b includes the first image and the fourth image. Among them, the first image and the second image are synchronously collected image data. The fourth image and the fifth image are synchronously collected image data.

[0146] After the RGB camera sensor acquires the image data b, the image data b can also be passed to the IFE module, which can perform image pre-processing on the image data b, such as converting the image data b into a YUV format. Then, the processed image data b is passed to the IPE module, which forwards the image data b to the secure memory for storage.

[0147] After the TOF camera sensor collects the image data a, the image data a can also be passed to the IFE-Lite module. The IFE-Lite module can forward the image data a to the secure memory for storage.

[0148] The camera driver can also be used to obtain the file descriptor (FD) of the image data a from the IFE-Lite module, such as Figure 5 The above camera driver can also obtain the FD of image data b from the IPE module, such as Figure 5 FD2 in . FD1 is used to indicate the storage location of image data a collected by the TOF camera sensor in the secure memory. FD2 is used to indicate the storage location of image data b collected by the RGB camera sensor in the secure memory.

[0149] Afterwards, the camera driver sends FD1 and FD2 to the camera HAL, which then sends them to the camera service. The camera service can send FD1 and FD2 to the face recognition control module. Afterwards, the face recognition control module can send FD1 and FD2 to the face recognition TA on the TEE side.

[0150] In response to FD1 and FD2, the face recognition TA reads the image data in the storage location indicated by FD1 and FD2 from the secure memory through the trusted camera service on the TEE side, such as image data a and image data b. After reading the image data (image data a and image data b), the face recognition TA can perform image processing tasks related to face recognition based on the image data (image data a or image data b).

[0151] Exemplarily, the face recognition TA may include an acquisition module and a face processing module.

[0152] The acquisition module can interact with the trusted camera service. After receiving FD1 and FD2 from the REE side, the acquisition module sends FD1 and FD2 to the trusted camera service, instructing the trusted camera service to read the corresponding image data a and image data b from the secure memory. Then, the acquisition module receives the image data a and image data b returned by the trusted camera service.

[0153] The above-mentioned face processing module can perform image processing tasks related to face recognition based on image data a and image data b, such as one or more image processing tasks such as user verification detection, image feature alignment, anti-counterfeiting / liveness recognition, and face recognition.

[0154] For example, Figure 5 As shown, the TEE side is pre-configured with TOF processing algorithm, face recognition algorithm, face detection algorithm, anti-counterfeiting / liveness recognition algorithm, and image feature alignment algorithm.

[0155] The TOF processing algorithm is used to convert the original (RAW) image (eg, image data a) captured by the TOF camera sensor into an IR image.

[0156] The above-mentioned face detection algorithm is used to extract facial feature information from the IR map corresponding to the image data b or the image data a, such as facial key point information (for example, feature information corresponding to the eyes, nose, mouth, etc. appearing in the image), face frame position information (a frame indicating the position where the face is displayed in the image), face angle information, face occlusion information, etc.

[0157] The above-mentioned face recognition algorithm is used to perform identity recognition based on face feature information.

[0158] The above-mentioned anti-counterfeiting / liveness recognition algorithm is used to identify whether the object participating in face recognition is a real person, so as to prevent a malicious third party from using photos to participate in face recognition. The implementation principle of the anti-counterfeiting / liveness recognition algorithm can be referred to related technologies and will not be elaborated here.

[0159] The above-mentioned image feature alignment algorithm is used to verify whether image data a and image data b are synchronously collected by the RGB camera sensor and the TOF camera sensor, so as to avoid the problem that the images collected by the RGB camera sensor or the TOF camera sensor are tampered with. The implementation details are introduced in the subsequent embodiments and will not be repeated here.

[0160] In some embodiments, after receiving the image data a and the image data b, the face processing module can call the above one or more algorithm modules through the algorithm adaptation layer to perform one or more image processing tasks related to face recognition.

[0161] Taking the face recognition task as an example, the face processing module calls the TOF processing algorithm through the algorithm adaptation layer to convert the image data a into an IR image, and then calls the face detection algorithm to extract the face feature information from the above IR image. Then, the face processing module reads the pre-stored multiple identity information and the authentication face features corresponding to each identity information from the secure storage chip on the TEE side. In this way, the face processing module can call the face recognition module through the algorithm adaptation layer, and perform face recognition in combination with the authentication face features corresponding to the identity information. For example, feature matching is performed based on the face feature information extracted from the IR image and the authentication face features. If the extracted face feature information successfully matches the authentication face features corresponding to any identity information, the face recognition is determined to be successful. If the extracted face feature information does not match all the authentication face features, the face recognition is determined to have failed. In this way, the face processing module can obtain the face recognition result.

[0162] In some embodiments, an authentication TA is also running on the TEE side. After the face processing module obtains the face recognition result, the authentication TA can obtain the above face recognition result from the face recognition TA. Afterwards, the authentication TA can pass the face recognition result to the face recognition service on the REE side. In this way, the face recognition service can pass the face recognition result to the application requesting face recognition.

[0163] In the above embodiment, the synchronous output of the RGB camera sensor and the TOF camera sensor is realized from the hardware layer. By utilizing the synchronization of the image data collected by the RGB camera sensor and the image data collected by the TOF camera sensor, in the process of face recognition based on the image data collected by the TOF camera sensor, the image data collected by the RGB camera sensor is used to improve the fault tolerance of the user verification detection link, and also improve the fault tolerance of the entire face recognition process, thereby reducing the failure rate of the face recognition process executed by the electronic device.

[0164] In other possible embodiments, the timestamp can also be used to synchronize the image data collected by the RGB camera sensor and the TOF camera sensor on the time axis. Exemplarily, after receiving the image data collected by the TOF camera sensor, the IFE-lite module can assign a timestamp to the image data, and correspondingly, the FD of the image data can also carry the timestamp. After the IFE module receives the image data collected by the RGB camera sensor, it can also assign a timestamp to the image data, and the FD of the image data can also carry the timestamp. In this way, when the face recognition TA receives FD1, it can also search for FD2 in the FD from the RGB camera sensor according to the timestamp carried in FD1. Among them, the acquisition time indicated by the timestamp carried by FD2 is closest to the acquisition time indicated by the timestamp carried by FD1.

[0165] Take the scenario where a user operates an electronic device to transfer money as an example. Figure 6 As shown, after the electronic device detects the user's instruction to transfer money, for example, the user clicks on the payment application, a transfer interface 601 is displayed. During the display of the transfer interface 601, the electronic device can receive the user's configuration transfer information (such as the receiving account, the transfer amount, etc.). Afterwards, in response to the user's instruction to determine the transfer operation, for example, clicking the "Next" control 602 in the transfer interface 601, a face recognition window 603 is displayed to prompt the user to participate in face recognition.

[0166] In addition, in response to the user's instruction to determine the transfer operation, the payment application in the electronic device can also request to start the face recognition process through the face recognition SDK. Figure 4 As shown, the TOF camera sensor and the RGB camera sensor can be powered on at the same time.

[0167] In some possible embodiments, in a dark light scene, for example, when the electronic device detects that the ambient light brightness value is lower than a preset brightness threshold (first threshold), after the TOF camera sensor and the RGB camera sensor are powered on, the background light brightness value of the display screen of the electronic device can be increased to improve the image quality of the image data collected by the RGB camera sensor. Figure 7 As shown, in a dark light scene, before the face recognition is started, for example, before the face recognition window 603 is displayed, the background light brightness value of the electronic device may be 1 (first value). After the face recognition is started, for example, after the face recognition window 603 is displayed, the background light brightness value of the electronic device may be 2 (second value), where value 2 is greater than value 1. In this way, the backlight brightness value of the electronic device will increase before and after the face recognition window 603 is displayed.

[0168] In other embodiments, after the TOF camera sensor and the RGB camera sensor are powered on, the background light brightness value may not be adjusted immediately. That is, there is no need to detect the ambient light brightness in the environment, and the background light brightness of the display screen of the electronic device will not be increased. The embodiments of the present application do not make specific limitations on this.

[0169] In addition, in some embodiments, Figure 8 and Fig. 9 As shown in the figure, after the TOF camera sensor and the RGB camera sensor are powered on, the TOF camera sensor and the RGB camera sensor synchronously collect image data a and image data b. Among them, image data a is the RAW image collected by the TOF camera sensor, and the electronic device can convert image data a into an IR image; its implementation details can be referred to Figure 5 For example, the TOF camera sensor stores the collected image data a in the secure memory. After the face recognition TA reads the image data a from the secure memory, it calls the TOF processing algorithm through the algorithm adaptation layer to convert the image data a into an IR image.

[0170] After the face recognition window 603 is displayed, if the image data a collected by the TOF camera sensor includes a face image with squinting eyes, for example, Figure 8 As shown, the user squints when the TOF camera sensor captures the image, or the user participating in face recognition has small eyes, and the electronic device uses the IR map of the image data a, and the probability of user verification detection is low.

[0171] After the face recognition window 603 is displayed, if the image data a collected by the TOF camera sensor includes a face image with eyes wide open, that is, Fig. 9 As shown, when the user opens his eyes wide when the TOF camera sensor captures images, or when the user participating in face recognition has large eyes, the electronic device uses the IR map of image data a to increase the probability of user verification detection, such as eye gaze detection and eye open and closed detection.

[0172] In the scenario where the IR image of image data a cannot be detected by user verification, for example, when the user narrows his eyes to participate in face recognition, such as Figure 8 As shown, the steps of the electronic device performing the face recognition method are as follows:

[0173] S1, based on the IR map of image data a, perform eye gaze detection and eye opening and closing detection.

[0174] In some embodiments, the above-mentioned human eye gaze detection and human eye opening and closing detection are only examples of user verification detection. In other embodiments, more or fewer types of user verification detection may be performed. For example, only human eye gaze detection or human eye opening and closing detection may be performed. For another example, other types of detection may be performed. In addition, the above-mentioned user verification detection may also be referred to as the first detection.

[0175] In some embodiments, the user can configure different face recognition scenarios and the types of user verification detection that need to be enabled. For example, the electronic device responds to the user's operation and is configured not to enable eye gaze detection when face unlocking. In this way, in the scenario where the lock screen application initiates face recognition, the above user verification detection may only include eye opening and closing detection.

[0176] Exemplarily, the process of performing eye gaze detection is as follows: after the face processing module in the face recognition TA converts the image data a (such as the second image) into an IR map (the first IR map), the face processing module can call the face detection algorithm through the algorithm adaptation layer to extract face feature information from the IR map of the image data a. For example, the face key point information (such as feature information corresponding to the eyes, nose, mouth, etc. appearing in the image), face frame position information (a frame indicating the display position of the face in the image), face angle information, face occlusion information, etc. are extracted.

[0177] Then, the face processing module calculates the corresponding eye movement data based on the face feature information from the IR image (such as the eye feature information and / or face angle information in the face key point information). The above eye feature information can also be called eye feature information. The above eye movement data can indicate the position of the user's gaze point. When the eye movement data can indicate that the user's gaze point is located on the display screen of the electronic device, it is determined that the eye gaze detection has been passed.

[0178] When the eye movement data can indicate that the user's gaze point is outside the display screen, or the facial feature information from the first IR map is insufficient to calculate the eye movement data, or the eye feature information is not extracted from the first IR map, the electronic device can determine that the eye gaze detection has not passed based on the IR map of image data a.

[0179] As another example, the process of performing eye opening and closing detection is as follows: after the face processing module in the face recognition TA converts the image data a into an IR map, the face processing module can call the face detection algorithm through the algorithm adaptation layer to extract the face feature information from the IR map of the image data a. It can be understood that before performing the eye opening and closing detection, the face features have been extracted from the IR map of the image data a, and the step of calling the face detection algorithm can be skipped.

[0180] Then, the face processing module determines the open or closed state of the user's eyes based on the eye feature information from the IR image (such as the first eye feature information extracted from the first IR image). If the eye feature information indicates that the user's eyes are in an open state, it is determined that the eye open or closed detection has been passed. If the eye feature information indicates that the user's eyes are in a closed state, it is determined that the eye open or closed detection has not been passed. In addition, if the eye feature information is not successfully extracted from the IR image of the image data a, it is also determined that the eye open or closed detection has not been passed. If the extracted eye feature information cannot determine whether the eyes are open or closed, it is also determined that the eye open or closed detection has not been passed.

[0181] In addition, there is no necessary order between the above-mentioned human eye gaze detection and human eye opening and closing detection. The above-mentioned human eye gaze detection and human eye opening and closing detection both rely on human eye feature information extracted from the image data. In some embodiments, after converting the image data a into an IR image, if the electronic device fails to successfully extract the human eye feature information from the IR image, it can be determined that both the human eye gaze detection and the human eye opening and closing detection have failed, and the process enters S2. In some embodiments, if the electronic device extracts the human eye feature information from the IR image and at the same time determines that the user is in a closed-eye state, or determines that the user's gaze point is outside the display screen of the electronic device, the process can also enter S2.

[0182] S2, based on the image data b, performs eye gaze detection and eye opening and closing detection.

[0183] In some embodiments, image data b and image data a are collected synchronously, and image data b (e.g., the first image) comes from an RGB camera sensor, has a higher resolution, and contains richer image information. In this embodiment, the electronic device can perform user verification detection (e.g., human eye gaze detection and human eye opening and closing detection) again based on image data b. The implementation details of S2 above are the same as those of S1, except that the image data used in S1 is the IR map of image data a, and the image data used in S2 is image data b, which will not be described here.

[0184] After the image data b passes the eye gaze detection and the eye open / closed detection, the process may enter S3. If the image data b fails the eye gaze detection and the eye open / closed detection, it is determined that the face recognition initiated by the payment application fails.

[0185] After determining that face recognition has failed, the electronic device may display a prompt message, such as a first reminder message, to remind the user to re-participate in face recognition. Exemplarily, the prompt message may be a text reminder, such as displaying a text reminder "Recognition failed, please move closer and face the display screen to re-perform face recognition."

[0186] In addition to displaying the prompt information, the electronic device can also detect the ambient light brightness value of the environment in which it is located, and if the ambient light brightness value is lower than the preset brightness threshold, brighten the background light brightness of the electronic device. For example, the background light brightness of the display screen is increased from the third value to the fourth value. The third value is the background light brightness of the display screen before the first reminder information is displayed.

[0187] After the background light brightness of the electronic device is brightened, the face recognition process is executed based on the re-collected image data a and image data b, for example, the process enters S1 again.

[0188] S3, performing face recognition processing based on the IR map of the image data a.

[0189] In some embodiments, the face processing module reads a plurality of pre-stored identity information and the authentication face features (second face features) corresponding to each identity information from the secure storage chip. The authentication face features may be information pre-stored by the electronic device in response to user operations. For example, the authentication face information may be the face feature information entered into the electronic device when the user configures the unlock password of the device (or the payment password of the account). The identity information may indicate the identity of the user corresponding to the authentication face feature. Different users have different identity information, and each identity information also corresponds to the permissions granted to the user.

[0190] Then, the face processing module can call the face recognition algorithm through the algorithm adaptation layer, and perform face feature comparison based on the face feature information (face feature information a, or called the first face feature) extracted from the IR map of the image data a and the authenticated face feature. If the face feature information a matches the authenticated face feature a successfully, and the identity information a corresponding to the authenticated face feature a has payment authority, the face recognition result can be determined to be a face recognition success. If the face feature information a does not match all authenticated face features, the face recognition result can be determined to be a face recognition failure. If the face feature information a matches the authenticated face feature a successfully, but the identity information a corresponding to the authenticated face feature a does not have payment authority, the face recognition result is also determined to be a face recognition failure.

[0191] In other possible embodiments, the electronic device may also skip S1 and execute S2 after synchronously acquiring image data a and image data b, which is not specifically limited in the embodiments of the present application.

[0192] In actual applications, there are also scenarios where the IR image of the image data a can be verified by the user, for example, after the face recognition based on the first image and the second image fails, the user opens his eyes and re-participates in the face recognition. Another example is a scenario where the user opens his eyes and participates in the face recognition. Another example is a scenario where a user with large eyes participates in the face recognition.

[0193] In the above scenario, if Fig. 9 As shown, the steps of the electronic device performing the face recognition method are as follows:

[0194] A1, based on the IR map of image data a, perform eye gaze detection and eye opening and closing detection.

[0195] In some embodiments, the implementation details of A1 above can refer to S1, which will not be described here. After the IR map based on image data a passes the human eye gaze detection and human eye opening and closing detection, the process can enter A2. Among them, the IR map of image data a mentioned in A1 above can also be called a second IR map.

[0196] A2, performs face recognition processing based on the IR map of image data a.

[0197] In some embodiments, the implementation details of A2 above can be referred to S3, which will not be described here. Figure 8 In the scenario shown, the steps of performing eye gaze detection and eye opening and closing detection based on image data b are skipped.

[0198] In some embodiments, before performing face recognition based on the IR map of image data a, the electronic device may also determine the validity of image data a and image data b. For example, through image feature alignment processing, it is determined whether image data b is an image collected synchronously with image data a and has not been tampered with. For another example, through anti-counterfeiting / liveness recognition, it is determined whether the face appearing in the IR map of image data a is from a real person. If it is determined that image data a is an image obtained by photographing a real person, it is determined that image data a is valid image data. If it is determined that image data a is not an image obtained by photographing a real person, it is determined that image data a is not valid image data.

[0199] In an exemplary embodiment, after the IR map of image data a or image data b is detected by user authentication, if it is determined that both image data a and image data b are valid, face recognition processing may be performed based on the IR map of image data a.

[0200] like Fig.10 As shown, before S3, it can also include:

[0201] B1, based on the IR map of image data b and image data a, image feature alignment processing is performed.

[0202] Exemplarily, the above-mentioned image feature alignment process may include:

[0203] (1) According to the pre-calibrated pixel mapping relationship, image data b is processed to obtain image data c.

[0204] It is understandable that the RGB camera sensor and the TOF camera sensor are configured at different positions on the electronic device. When the RGB camera sensor and the TOF camera sensor simultaneously shoot the same object, due to the differences in shooting positions and shooting angles, the image coordinates of the pixels presenting the same content between the image data collected by the RGB camera sensor and the image data collected by the TOF camera sensor may be different. The above-mentioned pixel mapping relationship (first mapping relationship) can indicate the positional relationship between the pixels presenting the same content between the two frames of images synchronously collected by the RGB camera sensor and the TOF camera sensor.

[0205] for example, Fig.11 As shown, when calibrating the pixel mapping relationship, when the RGB camera sensor collects image data d, the TOF camera sensor is synchronously controlled to collect image data e. The pixel a in the image data d and the pixel b in the image data e are pixels presenting the same content, wherein the image coordinates of the pixel a in the image data d are (2, 2), and the image coordinates of the pixel b in the image data e are (3, 3). In this way, the calibrated pixel mapping relationship includes the image coordinates (2, 2) and the mapping coordinates (3, 3) of the image coordinates (2, 2). Of course, the pixel mapping relationship also includes other image coordinates and the mapping coordinates of other image coordinates in the image data collected by the RGB camera sensor.

[0206] When processing image data b using the pixel mapping relationship, each pixel in image data b can be migrated from the current image coordinates to the mapping coordinates corresponding to the image coordinates according to the pixel mapping relationship to obtain image data c. For example, the pixel with image coordinates (2,2) in image data b can be migrated to (3,3).

[0207] (2) Compare the image coordinates of the pixel points of the IR map of the image data c and the image data a to determine the alignment result between the IR map of the image data c and the image data a.

[0208] Exemplarily, the image coordinates of each pixel point of the image area a showing the eye in the image data c (third image) are obtained, that is, the first pixel point in the third image is obtained. The image coordinates of each pixel point of the image area b showing the eye in the IR map of the image data a are obtained, that is, the second pixel point in the first IR map is obtained. The number 1 (first number) of pixel points with different image coordinates between the image area a and the image area b is counted. When the number 1 exceeds the preset number (preset number threshold), it is determined that the alignment result does not meet the preset condition. When the number 1 does not exceed the preset number, it is determined that the alignment result meets the preset condition.

[0209] It can be understood that the image data synchronously collected by the RGB album sensor and the TOF camera sensor, and the image data obtained after the pixel mapping relationship processing, present the same content of the pixel coordinates of the same. Using the above characteristics, when there are many pixels with different image coordinates between the image area a and the image area b showing the eyes, there may be a problem that the image data a and the image data b are not synchronized, or the image data b or the image data a is tampered with. In this scenario, it can be determined that the image data b is invalid image data. When there are fewer pixels with different image coordinates between the image area a and the image area b showing the eyes, it indicates that the image data a and the image data b are synchronously collected image data. In this scenario, it can be determined that the image data b is valid image data.

[0210] If the alignment result meets the preset conditions, the process can enter B2, or, without executing B2, the process directly enters S3. If the alignment result does not meet the preset conditions, it is determined that the face recognition initiated by the payment application has failed. After determining that the face recognition has failed, the electronic device can display a prompt message to remind the user to re-participate in the face recognition. Exemplarily, the prompt message can be a text reminder, such as displaying a text reminder "Recognition failed, please move closer and face the display screen to re-perform face recognition."

[0211] In addition to displaying the prompt information, the electronic device can also detect the ambient light brightness value of the environment in which it is located, and if the ambient light brightness value is lower than the preset brightness threshold, the background light brightness of the electronic device is brightened. After brightening the background light brightness of the electronic device, the face recognition process is executed based on the re-collected image data a and image data b, such as the process enters S1 again.

[0212] B2, anti-counterfeiting / liveness recognition is performed based on the IR map of image data a.

[0213] In some embodiments, the face processing module can call the anti-counterfeiting / liveness recognition algorithm through the algorithm adaptation layer to perform anti-counterfeiting / liveness recognition on the IR map of the image data a. The implementation details can be referred to related technologies and will not be elaborated here.

[0214] It can be understood that after the IR image of image data a passes the anti-counterfeiting / liveness recognition, it can be determined that image data a is valid image data. In this scenario, the process can enter S3. In the case that the IR image of image data a does not pass the anti-counterfeiting / liveness recognition, it can be determined that image data a is invalid image data. In this scenario, it can be determined that the face recognition initiated by the payment application this time has failed.

[0215] In another exemplary embodiment, after the IR map of image data a or image data b is detected by user verification, if image data a is determined to be valid, face recognition processing can be performed based on the IR map of image data a. Figure 8 In the scenario shown, after image data b passes user authentication, anti-counterfeiting / liveness recognition is performed based on the IR map of image data a. When the IR map of image data a passes anti-counterfeiting / liveness recognition, face recognition processing is performed based on the IR map of image data a.

[0216] For example, in Fig. 9 In the scenario shown, after image data a passes user authentication, anti-counterfeiting / liveness recognition is performed based on the IR map of image data a. When the IR map of image data a passes anti-counterfeiting / liveness recognition, face recognition processing is performed based on the IR map of image data a.

[0217] As a way to achieve this, Fig.12 As shown, after the application initiates face recognition, the signaling interaction steps between the various software and hardware modules in the electronic device are as follows:

[0218] S101, the application sends a face recognition registration request to the face recognition service.

[0219] In some embodiments, when a business need for face recognition arises, the application can send a face recognition registration request to the face recognition service through the face recognition SDK. The face recognition registration request can be used to register a callback corresponding to face recognition on the face recognition service so that the application can obtain the face recognition result.

[0220] for example, Figure 6 As shown, after receiving the operation of confirming the transfer, the payment application generates a business demand for face recognition. In response to the business demand for face recognition, a face recognition registration request is sent to the face recognition service.

[0221] S102: The face recognition service sends a first request to the face recognition control module in response to the face recognition registration request.

[0222] The first request is a request instruction to start face recognition.

[0223] In some embodiments, the face recognition service may respond to the face recognition registration request and may indicate the subsequent process of performing face recognition. In addition, after the face recognition service obtains the face recognition result, the face recognition result may be fed back to the corresponding application through the face recognition SDK through the above registered callback.

[0224] S103: The face recognition control module determines a target camera sensor in response to the first request.

[0225] It is understandable that before the face recognition control module performs face recognition, it is necessary to determine the camera sensor used to collect image data, that is, the target camera sensor. Usually, the target camera sensor can be the camera sensor of the front camera of the electronic device, the lens of the front camera is located on the same side as the display screen of the electronic device, and the front camera can include a camera equipped with a TOF camera sensor and a camera equipped with an RGB camera sensor.

[0226] Exemplarily, the face recognition control module may determine the TOF camera sensor in the front camera as the target camera sensor. Also exemplary, the face recognition control module may also determine the RGB camera sensor in the front camera as the target camera sensor.

[0227] In some embodiments, the rule for selecting the target camera sensor may be to combine the ambient light brightness to determine the target camera sensor. For specific implementation details, reference may be made to related technologies and will not be repeated here.

[0228] S104: When the target camera sensor is a TOF camera sensor, the face recognition control module sends a second request to the camera service, where the second request carries the camera identifiers of the TOF camera sensor and the RGB camera sensor.

[0229] S105: The camera service sends a second request to the camera HAL.

[0230] S106 : The camera HAL creates a transmission path between the TOF camera sensor and the RGB camera sensor in response to the second request.

[0231] In some embodiments, the face recognition control module may send a second request to the camera service through a vendor native development kit (VNDK) interface, and the camera service passes the second request to the camera HAL.

[0232] In some embodiments, the camera HAL responds to the second request and creates a transmission path corresponding to the TOF camera sensor and a transmission path corresponding to the RGB camera sensor according to the camera identifier (the identifier of the TOF camera sensor and the identifier of the RGB camera sensor) in the second request. It can be understood that the above transmission paths at least include a path corresponding to the sensor node and a path corresponding to the IFE node.

[0233] Exemplarily, the above-mentioned creation of the corresponding transmission path may include: creating sensor nodes and IFE nodes, and determining other software and hardware modules in the path corresponding to the sensor nodes and the IFE nodes.

[0234] Taking the creation of the transmission path of the RGB camera sensor as an example, the path corresponding to the sensor node can be: sensor node-camera driver-RGB camera sensor-IFE module-IPE module-secure memory. The path corresponding to the IFE node can be: IFE module-camera driver-IFE node. After the above path is created, the hardware in the path is powered on (that is, the hardware circuit is powered on), waiting for the image acquisition instruction from the face recognition control module, triggering the RGB camera sensor to acquire the image, and transmitting it back.

[0235] Taking the creation of a transmission path for a TOF camera sensor as an example, the path corresponding to the sensor node can be: a path consisting of sensor node-camera driver-TOF camera sensor-IFE-Lite-secure memory. The path corresponding to the IFE node can be: a path consisting of IFE-Lite-camera driver-IFE node. The camera HAL can connect the output port of the sensor node and the input port of the IFE node at the HAL layer. Thus, the path corresponding to the sensor node and the path corresponding to the IFE node can form a closed-loop path. After the path is created, the hardware in the path is powered on (i.e., the hardware circuit is powered on), waiting for the synchronization pulse signal from the RGB camera sensor, triggering the TOF camera sensor to acquire the image, and transmitting it back.

[0236] S107, the camera HAL sends a notification 1 to the camera service, indicating that the channel is successfully created.

[0237] S108, the camera service sends notification 1 to the face recognition control module.

[0238] In some embodiments, after successfully establishing the paths corresponding to the TOF camera sensor and the RGB camera sensor, the camera HAL sends a notification 1 (also referred to as a success notification) to the face recognition control module through the camera service, indicating that the transmission path for collecting image data required for face recognition has been established. In response to the notification 1, the face recognition control module may execute S109.

[0239] In some other embodiments, when the camera HAL fails to establish a path corresponding to the TOF camera sensor or the RGB camera sensor, the camera HAL sends a failure notification to the face recognition control module through the camera service, indicating that the transmission path establishment failed. In this scenario, the face recognition control module can re-initiate the establishment of the transmission channel in response to the failure notification.

[0240] S109, the face recognition control module sends an image acquisition instruction to the camera service.

[0241] S110: The camera service calls the camera HAL to instruct the RGB camera sensor to acquire an image.

[0242] S111, the camera HAL instructs the RGB camera sensor to acquire an image through the camera driver.

[0243] The camera driver includes a camera driver corresponding to an RGB camera sensor, and of course, may also include a camera driver corresponding to a TOF camera sensor.

[0244] In some embodiments, after S109 and before S110, the camera service may also instruct the camera driver through the camera HAL to write the camera parameters required for operation into the RGB camera sensor and the TOF camera sensor. For implementation details, please refer to the relevant technology and will not be described here.

[0245] S112: The camera driver instructs the RGB camera sensor to collect image data.

[0246] S113, the RGB camera sensor collects image data b.

[0247] In some embodiments, after writing the camera parameters into the RGB camera sensor, the camera driver may send a start flow instruction to the RGB camera sensor to instruct the RGB camera sensor to start image data acquisition and return the FD corresponding to the acquired image data.

[0248] S114, the RGB camera sensor writes the collected image data b into the secure memory and obtains FD2 indicating its storage location.

[0249] There is a corresponding relationship between the above FD2 and the storage address of the image data b in the secure memory, and other software modules can query the image data b based on FD2 through face recognition TA.

[0250] S115, the RGB camera sensor sends a synchronization pulse signal to the TOF camera sensor.

[0251] In some embodiments, there is no necessary order between the above S115 and S114. After the RGB camera sensor collects the image data b, it can immediately send a synchronization pulse signal to the TOF camera sensor.

[0252] S116, the TOF camera sensor collects image data a.

[0253] S117, the TOF camera sensor writes the collected image data a into the secure memory and obtains FD1 indicating its storage location.

[0254] S118, the RGB camera sensor sends notification 2 to the camera driver, carrying FD2.

[0255] The notification 2 may indicate that the RGB camera sensor has captured image data, such as image data b, and has stored the image data b in the secure memory. The notification 2 may carry the FD corresponding to the image data b, such as FD2.

[0256] In addition, there is no necessary order between the above S118 and the above S115. After S114, that is, after the RGB camera sensor obtains FD2, the notification 2 can be sent to the camera driver.

[0257] S119, the TOF camera sensor sends notification 3 to the camera driver, carrying FD1.

[0258] The notification 3 may indicate that the TOF camera sensor has captured image data, such as image data a, and has stored the image data a in the secure memory. The notification 3 may carry the FD corresponding to the image data a, such as FD1.

[0259] S120: The camera driver sends FD1 and FD2 to the camera HAL.

[0260] In some embodiments, the camera driver of the RGB camera sensor sends FD2 to the camera HAL, and the camera driver of the TOF camera sensor sends FD1 to the camera HAL.

[0261] S121, the camera HAL sends FD1 and FD2 to the camera service.

[0262] In some embodiments, the camera HAL may send a pair of FD1 and FD2 to the camera service to facilitate the subsequent search for synchronously acquired image data a and image data b. For example, after receiving an FD2, the camera HAL waits for FD1 from the TOF camera sensor. After receiving FD1, the FD1 and FD2 are sent to the camera service. For another example, after receiving FD1 and FD2, the camera HAL searches for the target FD corresponding to each FD2 in FD1. The acquisition time indicated by the timestamp of the target FD is later than the acquisition time indicated by the timestamp of FD2. In addition, compared with other FD1s, the time interval between the acquisition time indicated by the timestamp of the target FD and the acquisition time indicated by the timestamp of FD2 is the shortest.

[0263] S122, the camera service sends FD1 and FD2 to the face recognition control module.

[0264] S123, the face recognition control module sends FD1 and FD2 to the face recognition TA.

[0265] S124, face recognition TA obtains image data a and image data b from the secure memory according to FD1 and FD2.

[0266] S125, face recognition TA determines the face recognition result based on image data a and image data b.

[0267] In some embodiments, the implementation details of the above S125 can be referred to Figure 8 , Fig. 9 and Fig.10 The embodiments shown will not be described in detail here.

[0268] S126, authenticating TA to obtain face recognition results.

[0269] In some embodiments, the authentication TA also runs on the TEE side, and the authentication TA can periodically query the face recognition result from the face recognition TA. In other embodiments, after obtaining the face recognition result, the face recognition TA actively sends the face recognition result to the authentication TA.

[0270] S127, authenticate the TA and send the face recognition result to the face recognition service.

[0271] S128, the face recognition service sends the face recognition result to the application.

[0272] In some embodiments, the face recognition service can feedback the face recognition results to all applications registered to obtain face recognition results, so that the applications can execute corresponding responses according to the face recognition results. For example, if the face recognition result is a successful face recognition, the payment application can execute the actual transfer process. For example, if the face recognition result is a failed face recognition, the payment application can instruct the display screen to display a prompt message, prompting the user to move closer to the display screen and re-participate in face recognition. Of course, the payment application can also instruct the display screen to brighten the background brightness.

[0273] In other embodiments, after the RGB camera sensor is powered on, the display screen can be controlled to brighten the background light brightness, which is not specifically limited in the embodiments of the present application.

[0274] The embodiment of the present application also provides an electronic device, which may include: a camera sensor, a display screen, a memory, and one or more processors. The memory is coupled to the processor. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can execute the various steps performed by the electronic device in the above embodiment. Of course, the electronic device includes but is not limited to the above memory and one or more processors.

[0275] The embodiment of the present application also provides a chip system, which can be applied to the electronic device in the aforementioned embodiment. The chip system includes at least one processor and at least one interface circuit. The processor can be the processor in the above-mentioned electronic device. The processor and the interface circuit can be interconnected by lines. The processor can receive and execute computer instructions from the memory of the above-mentioned electronic device through the interface circuit. When the computer instructions are executed by the processor, the electronic device can execute the various steps executed by the electronic device in the above-mentioned embodiment. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiment of the present application.

[0276] In some embodiments, through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0277] Each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0278] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk.

[0279] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A face recognition method, characterized in that: Applied to an electronic device, wherein the electronic device is equipped with an RGB camera sensor and a TOF camera sensor, the method comprises: Receiving a first operation, wherein the first operation triggers the electronic device to perform face recognition; In response to the first operation, controlling the RGB camera sensor to capture a first image and the TOF camera sensor to capture a second image, wherein the first image and the second image are image data synchronously captured by the RGB camera sensor and the TOF camera sensor; Performing a first detection based on the first image; wherein the first detection is an image detection item for an image area showing eyes in the image data, and is used to detect whether the user has the intention to participate in the face recognition; When the first image passes the first detection, face recognition is performed based on the second image.

2. The method according to claim 1, characterized in that A communication line is configured between the RGB camera sensor and the TOF camera sensor, and in response to the first operation, the RGB camera sensor collects a first image, and the TOF camera sensor collects a second image, including: In response to the first operation, the RGB camera sensor captures a first image, and the RGB camera sensor sends first synchronization information to the TOF camera sensor through the communication line; The TOF camera sensor acquires the second image in response to the first synchronization information.

3. The method according to claim 1 or 2, characterized in that: Before performing the first detection based on the first image, the method further includes: Performing the first detection based on the first infrared IR image of the second image; The performing a first detection based on the first image includes: In response to the first IR map failing the first detection, the first detection is performed based on the first image.

4. The method according to any one of claims 1 to 3, characterized in that The electronic device further includes a display screen; after receiving the first operation, the RGB camera sensor acquires a first image, and before the TOF camera sensor acquires a second image, the method further includes: In response to the first operation, controlling the RGB camera sensor and the TOF camera sensor to power on; When the ambient light brightness in the space where the electronic device is located is not greater than a first threshold, adjusting the background light brightness of the display screen from a first value to a second value; The first value is the background light brightness of the display screen before the RGB camera sensor is powered on.

5. The method according to any one of claims 1 to 3, characterized in that: After performing the first detection based on the first image, the method further includes: If the first image fails the first detection, a first reminder message is displayed, where the first reminder message indicates that face recognition has failed.

6. The method according to claim 5, characterized in that After displaying the first reminder information, the method further includes: When the ambient light brightness in the space where the electronic device is located is not greater than the first threshold, adjusting the background light brightness of the display screen of the electronic device from the third value to a fourth value; The third value is the background light brightness of the display screen before the first reminder information is displayed.

7. The method according to any one of claims 1 to 6, characterized in that A first mapping relationship is pre-configured in the electronic device, where the first mapping relationship indicates a positional relationship between pixels presenting the same content in the image data synchronously collected by the RGB camera sensor and the TOF camera sensor; Before performing face recognition based on the second image, the method further includes: According to the first mapping relationship, the image coordinates of each pixel in the first image are adjusted to obtain a third image; Acquire a first pixel point of an image area showing an eye in the third image, and acquire a second pixel point of an image area showing an eye in the first IR image of the second image; Determine a first number according to the image coordinates of the first pixel point and the image coordinates of the second pixel point, where the first number is the number of pixels whose corresponding image coordinates are different between the first pixel point and the second pixel point; It is determined that the first quantity is not greater than a preset quantity threshold.

8. The method according to any one of claims 1 to 7, characterized in that The performing face recognition based on the second image includes: a first facial feature extracted from a first IR image of the second image; In the case where the first facial feature matches the pre-configured second facial feature, the result of the facial recognition is success; When the first facial feature does not match the second facial feature, the result of the facial recognition is failure.

9. The method according to claim 8, characterized in that When the face recognition result is a failure, the method further includes: Controlling the RGB camera sensor to collect a fourth image and the TOF camera sensor to collect a fifth image, wherein the fourth image and the fifth image are image data synchronously collected by the RGB camera sensor and the TOF camera sensor; The first detection is performed based on the second IR map of the fifth image; and after the second IR map passes the first detection, face recognition is performed based on the second IR map.

10. The method according to any one of claims 1 to 9, characterized in that Before performing face recognition based on the second image, the method further includes: performing anti-counterfeiting / liveness detection on the first IR image of the second image; It is determined that the photographed object corresponding to the second image is a real person.

11. The method according to any one of claims 1 to 10, characterized in that The first detection includes human eye gaze detection and / or human eye opening and closing detection.

12. The method according to claim 3, characterized in that The first detection includes human eye gaze detection and / or human eye opening and closing detection; The performing the first detection based on the first infrared IR image of the second image includes: extracting eye feature information from the first IR image; In the event that the extraction of eye feature information from the first IR image fails, it is determined that the first IR image fails the human eye gaze detection and / or the human eye open / closed detection.

13. The method according to claim 12, characterized in that The performing the first detection based on the first infrared IR image of the second image further includes: In a case where the first eye feature information is extracted from the first IR image, judging whether the gaze point of the user is located on the display screen of the electronic device based on the first eye feature information; In a case where it is determined that the gaze point of the user is outside the display screen, determining that the first IR image fails the human eye gaze detection; Alternatively, when first eye feature information is extracted from the first IR image, judging whether the user's eyes are in an open state based on the first eye feature information; When it is determined that the eyes of the user are in a closed state, it is determined that the first IR image does not pass the human eye opening and closing detection.

14. The method according to any one of claims 1 to 13, characterized in that The electronic device is provided with a secure memory; After the RGB camera sensor acquires the first image and the TOF camera sensor acquires the second image, the method further includes: Control the RGB camera sensor to write the first image into the secure memory, and control the TOF camera sensor to write the second image into the secure memory; Before performing the first detection based on the first image, the method further includes: The first image and the second image are read out from the secure memory.

15. An electronic device, characterized in that: The electronic device comprises a display screen, a plurality of camera sensors, a memory and one or more processors; the plurality of camera sensors comprises a TOF camera sensor and an RGB camera sensor, and the display screen, the camera sensor, the memory and the processor are coupled; The display screen is used to display the image generated by the processor, the camera sensor is used to collect image data, and the memory is used to store computer program code, wherein the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1 to 14.

16. A readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 14.

Images