A face recognition method and apparatus

Abstract

Embodiments of the present application provide a face recognition method and device, and relate to the field of terminals, and can improve the security of face recognition. The method is applied to an electronic device, the electronic device includes a TOF camera module, the TOF camera module includes a transmitter for transmitting light signals and an image sensor for receiving reflected light and imaging, and the method includes: receiving a first operation of a user, the first operation being used to trigger face recognition; controlling the transmitter to work at a first light intensity; determining whether the transmitter is in a normal working state; in the case that the transmitter is in the normal working state, controlling the transmitter to work at a second light intensity, the second light intensity being greater than the first light intensity; controlling the image sensor to collect image data; and performing face recognition based on the image data. The determination of whether the transmitter is in the normal working state can be determined by a camera HAL in the electronic device through a camera driving module from the working state parameter of the transmitter obtained from the transmitter.

Description

TECHNICAL FIELD

[0001] The present application relates to the terminal field, and in particular to a face recognition method and device. BACKGROUND

[0002] At present, face recognition is widely used in electronic devices for identity authentication. The current face recognition adopts a planar (2D) face feature detection technology, and the extracted face features are 2D features, which are vulnerable to false attacks (for example, impersonation by the owner's photo), and the security is not high. SUMMARY

[0003] The embodiments of the present application provide a face recognition method and device, which can improve the security of face recognition.

[0004] In a first aspect, the embodiments of the present application provide a face recognition method applied to an electronic device, the electronic device comprising a time-of-flight (TOF) camera module, the TOF camera module comprising a transmitter for transmitting light signals and an image sensor for receiving reflected light and imaging, the method comprising: receiving a first operation of a user, the first operation being used to trigger face recognition; controlling the transmitter to work at a first light intensity; determining whether the transmitter is in a normal working state; in the case that the transmitter is in the normal working state, controlling the transmitter to work at a second light intensity, the second light intensity being greater than the first light intensity; controlling the image sensor to collect image data; and performing face recognition based on the image data.

[0005] Based on the method provided by the embodiments of the present application, it is determined whether the transmitter works at the first light intensity in the normal working state, and if the transmitter is in the normal working state, the transmitter is controlled to work at the second light intensity. The second light intensity is greater than the first light intensity. Since the first light intensity is small, the first light intensity will not cause harm to the human eye, and the safety of the human eye can be ensured. If the transmitter can normally work to emit light signals of the first light intensity, it means that the transmitter is intact and undamaged, so that the electronic device can normally control the transmitter. When the transmitter works at the second light intensity, the safety of the human eye can also be ensured, and the problem of emitting light signals that harm the human eye due to damage will not occur. Moreover, when the transmitter works at the second light intensity, the image sensor can collect more accurate image data, so that face recognition can be more accurate.

[0006] In addition, the image data collected by the TOF camera module is 3D face data, and compared with 2D face data, face recognition based on 3D face data is more secure and accurate.

[0007] In a possible implementation, the light signal emitted by the emitter when operating at the first current value is of a first light intensity, and the light signal emitted by the emitter when operating at the second current value is of a second light intensity, the second current value being greater than the first current value. That is, the greater the current value at which the emitter operates, the greater the light intensity of the light signal emitted by the emitter.

[0008] In a possible implementation, determining whether the emitter is in the normal operating state comprises: determining a first parameter of the emitter, the first parameter being used to indicate the operating state of the emitter; determining that the emitter is in the normal operating state if the first parameter is used to indicate that the operating state of the emitter is the normal operating state; and determining that the emitter is in the abnormal operating state if the first parameter is used to indicate that the operating state of the emitter is the abnormal operating state. The first parameter of the emitter can be generated after the emitter operates at the first current value and emits the light signal of the first light intensity.

[0009] In a possible implementation, the method further comprises: in the case where the emitter is in the abnormal operating state, controlling the emitter to operate at a third light intensity, the third light intensity being 0; controlling the image sensor to collect image data; and performing face recognition based on the image data. Since the emitter is in the abnormal operating state, the emitter can not operate at this time, which can avoid the problem that a light signal that is harmful to the human eye is emitted due to damage to the emitter.

[0010] In a possible implementation, performing face recognition based on the image data comprises: obtaining a grayscale image and a depth image based on the image data; performing face comparison based on the grayscale image and performing anti-counterfeiting detection based on the depth image to obtain a face recognition result. If the face comparison result meets a first preset condition and the anti-counterfeiting detection result meets a second preset condition, the face recognition result is considered to be successful, and thus an operation such as unlocking can be performed.

[0011] In a possible implementation, the first operation comprises an operation for unlocking the electronic device, an operation for online payment, an operation for recording a face, or an operation for security registration or login of an application. The first operation may, for example, be an operation such as pressing a power key, clicking, swiping, or the like, and the application does not limit this.

[0012] In a possible implementation, the method further includes: determining whether to perform unlocking according to the face recognition result; if the face recognition result is successful, performing unlocking; if the face recognition result is failed, not performing unlocking or displaying unlocking failure; or determining whether to perform payment according to the face recognition result; if the face recognition result is successful, performing payment; if the face recognition result is failed, not performing payment or displaying payment failure; or determining whether to perform face entry according to the face recognition result; if the face recognition result is successful, performing face entry; if the face recognition result is failed, not performing face entry or displaying face entry failure; or determining whether to perform registration or login according to the face recognition result; if the face recognition result is successful, performing registration or login; if the face recognition result is failed, not performing registration or login or displaying registration or login failure. That is, the present application can be applied to face recognition when payment or transfer is performed (for example, a user performs a payment or transfer operation in a payment application / financial application / chat application / shopping application (for example, ), face security verification when the user registers or logs in an application (for example, the user performs a registration or login operation in ), and the like, without limitation.

[0013] In a possible implementation, the method further includes: prompting the user of unlocking failure in the case that the emitter is in an abnormal working state; or prompting the user of payment failure; or prompting the user of face entry failure; or prompting the user of registration or login failure. Since the emitter is in an abnormal working state, the emitter does not work at this time, and thus the face cannot be successfully recognized, but the problem that the emitter emits a light signal that harms the eyes due to damage can be avoided.

[0014] In a possible implementation, the electronic device includes a camera hardware abstraction layer (HAL) and a camera driver module, the camera HAL includes a sensor node, and the control of the emitter to work at the first light intensity includes: the sensor node determines that a working mode of the camera module is a first working mode; the first working mode is used to instruct the emitter to work at a first current value; the sensor node sends a configuration parameter of the first working mode to the camera driver module; the camera driver module writes the configuration parameter of the first working mode into a register of the TOF camera module; the camera driver module sends a message of configuration parameter write completion to the sensor node; in response to receiving the message of configuration parameter write completion, the sensor node sends a first start command to the camera driver module; the camera driver module sends a second start command to the TOF camera module; and the emitter works at the first current value, and the light signal emitted by the emitter when working at the first current value is the first light intensity. In this way, the emitter can be controlled to work at the first light intensity through the camera HAL (including the sensor node) and the camera driver module in the electronic device.

[0015] In a possible implementation, the electronic device further includes a first application, a face recognition software development kit (SDK), a face recognition service, a face recognition control module, and a camera service. After receiving the first operation of the user, the method further includes: the first application calling the face recognition SDK to perform face recognition; the first application corresponds to the first operation, and the first application includes a lock screen application, a shopping application, a chat application, or a financial application; the face recognition SDK sending a face recognition request to the face recognition service; the face recognition request carries an identifier of a face recognition type, a resolution size of an image, and a data stream format; the face recognition service sending the face recognition request to the face recognition control module; the face recognition control module matching a camera module according to the face recognition request; the face recognition control module sending a first request for opening the camera module to the camera service; the first request for opening the camera module carries a security identifier, an identifier (ID) of the camera module, a resolution of the image, and the data stream format; the security identifier is used to apply for a secure memory; and the camera service sends a second request for opening the camera module to the camera HAL, and the second request carries the security identifier, the ID of the camera module, the resolution of the image, and the data stream format. In this way, the sensor node in the camera HAL can obtain the security identifier, the ID of the camera module, the resolution of the image, and the data stream format, and can determine the working mode of the camera module according to the information subsequently.

[0016] In a possible implementation, the sensor node determines the working mode of the camera module as the first working mode, specifically including: the sensor node determining the working mode of the camera module as the first working mode according to the resolution of the image, the data stream format, and a preset rule. In this way, the sensor node can determine that the working mode of the camera module is the first working mode.

[0017] In a possible implementation, after the camera service sends the second request for opening the camera module to the camera HAL, the method further includes: the camera HAL creating a path for transmitting a data stream and a control stream according to the ID of the camera module, the resolution of the image, and the data stream format; the camera HAL returning a result of creating the path to the camera service; the result of creating the path is successful; the camera service returning a message that the camera module is opened to the face recognition control module; the face recognition control module sending a data request to the camera service, the data request being used to acquire the data stream; and the camera service calling the camera HAL to acquire the data stream.

[0018] In a possible implementation, after the transmitter works at the first current value, the method further includes: the image sensor acquiring a light signal within an exposure time corresponding to the first working mode; and the image sensor acquiring first image data based on the received light signal.

[0019] In a possible implementation, after the image sensor acquires the first image data, the method further includes: the image sensor sending a request for acquiring the first parameter to the emitter; the image sensor receiving the first parameter from the emitter; and the image sensor obtaining the first original RAW Data based on the first image data and the first parameter, wherein the first parameter is used to indicate the working state of the emitter at the first current value.

[0020] In a possible implementation, the electronic device further includes an image processing module, a first memory, and a face recognition TA, and determining whether the emitter is in the normal working state includes: after the image sensor obtains the first original RAW Data based on the first image data and the first parameter, the image sensor sends the first RAW Data to the image processing module; the image processing module sends the first RAW Data to the first memory for storage, and the first memory corresponds to the first FD; the image processing module sends the first FD to the camera driver module; the camera driver module sends the first FD to the camera HAL; in response to receiving the first FD, the camera HAL sends a request for reading the first parameter to the camera driver module, and the first parameter is used to indicate the working state of the emitter; the camera driver module reads the first parameter from the register of the emitter and sends the first parameter to the sensor node of the camera HAL; and the camera HAL determines whether the emitter is in the normal working state according to the first parameter.

[0021] In a possible implementation, the camera HAL determines whether the emitter is in the normal working state according to the first parameter, specifically including: if the first parameter is used to indicate that the working state of the emitter is the normal working state, the camera HAL determines that the emitter is in the normal working state; and if the first parameter is used to indicate that the working state of the emitter is the abnormal working state, the camera HAL determines that the emitter is in the abnormal working state. In this way, the face recognition can determine whether the emitter is in the normal working state according to the first parameter in the first RAW Data.

[0022] In a possible implementation, the method further includes: determining, by the camera HAL, that the working mode of the camera module is the second working mode if the working state of the emitter is the normal working state, the second working mode being used to indicate that the emitter works at the second current value; sending, by the camera HAL, the configuration parameters of the second working mode to the camera drive module; writing, by the camera drive module, the configuration parameters of the second working mode into a register of the TOF camera module; sending, by the camera drive module, a message indicating that the configuration parameter writing is completed to the sensor node; sending, by the sensor node, a third start command to the camera drive module in response to receiving the message indicating that the configuration parameter writing is completed; sending, by the camera drive module, a fourth start command to the TOF camera module; and working, by the emitter, at the second current value, the light signal emitted by the emitter when working at the second current value being the second light intensity. In this way, the camera HAL can determine that the working mode of the camera module is the second working mode according to the normal working state of the emitter. The camera HAL (including the sensor node) and the camera drive module in the electronic device can control the emitter to work at the second light intensity.

[0023] In a possible implementation, after the emitter works at the second current value, the method further includes: acquiring, by the image sensor, the light signal within the exposure time corresponding to the second working mode; and obtaining, by the image sensor, the second image data based on the received light signal.

[0024] In a possible implementation, after the image sensor obtains the second image data, the method further includes: sending, by the image sensor, a request for obtaining a second parameter to the emitter; receiving, by the image sensor, the second parameter from the emitter; and obtaining, by the image sensor, the second RAW Data based on the first image data and the second parameter, the second parameter being used to indicate the working state of the emitter at the second current value.

[0025] In a possible implementation, the electronic device further includes a second memory and a face recognition trusted application (TA), and performing face recognition based on the image data includes: sending, by the image sensor, the second RAW Data to the image processing module; storing, by the image processing module, the second RAW Data in the second memory; reading, by the face recognition TA, the second RAW Data from the second memory according to a second FD; obtaining, by the face recognition TA, a first grayscale image and a first depth image based on the image data in the second RAW Data; and obtaining, by the face recognition TA, a face recognition result by performing face comparison based on the first grayscale image and performing anti-fake detection based on the first depth image. In this way, the face recognition TA can determine the face recognition result based on the second RAW Data.

[0026] In a possible implementation, after the image processing module sends the second RAW Data to the second memory for storage, the method further includes: the image processing module sends the second FD to the camera driver module; the camera driver module sends the second FD to the camera HAL; the camera HAL sends the second FD to the camera service through a preset interface; the camera service sends the second FD to the face recognition control module; and the face recognition control module sends the second FD to the face recognition TA. In this way, the face recognition TA can obtain the second FD, so that the second RAW Data can be read according to the second FD.

[0027] In a possible implementation, the method further includes: the face recognition TA sends the face recognition result to the face recognition control module; the face recognition control module sends the face recognition result to the face recognition service; the face recognition service sends the face recognition result to the face recognition SDK; the face recognition SDK sends the face recognition result to the first application; and in response to receiving the face recognition result, the first application performs unlocking, and the face recognition result is success. In this way, when the face recognition result is success, the unlocking can be successfully performed.

[0028] In a possible implementation, in the case that the emitter is in an abnormal working state, the emitter is controlled to work at a third light intensity, and specifically includes: in response to receiving that the working state of the emitter is an abnormal working state, the camera HAL determines that the working mode of the camera module is a third working mode, the third working mode is used to indicate that the emitter works at a third current value, and the third current value is 0; the camera HAL sends configuration parameters of the third working mode to the camera driver module; the camera driver module writes the configuration parameters of the third working mode into a register of the TOF camera module; the camera driver module sends a message that the configuration parameter writing is completed to the sensor node; in response to receiving the message that the configuration parameter writing is completed, the sensor node sends a fifth start command to the camera driver module; the camera driver module sends a sixth start command to the TOF camera module; and the emitter does not work. Since the emitter is in an abnormal working state, the emitter does not work at this time, and the problem that the light signal that harms the human eye is emitted due to the damage of the emitter can be avoided.

[0029] In a possible implementation, after the emitter does not work, the method further includes: the image sensor receives a light signal within an exposure time corresponding to the third working mode; and the image sensor obtains third image data based on the received light signal.

[0030] In a possible implementation, the method further includes: the face recognition TA obtaining a second grayscale image and a second depth image based on the third image data; performing face comparison based on the second grayscale image and performing anti-counterfeiting detection based on the second depth image, and obtaining the face recognition result as a failure. Since the image data (the third image data) obtained by the image sensor is usually a "black image" without a clear face image when the emitter is not working (not powered on or not emitting light), the face recognition result is a failure.

[0031] In a possible implementation, the method further includes: the face recognition TA sending the face recognition result to the face recognition control module; the face recognition control module sending the face recognition result to the face recognition service; the face recognition service sending the face recognition result to the face recognition SDK; the face recognition SDK sending the face recognition result to the first application; and in response to receiving the face recognition result, the first application does not perform unlocking or displays an unlocking failure, and the face recognition result is a failure. Since the emitter is in an abnormal working state and the emitter is not working at this time, the face recognition result is a failure, so that the unlocking fails, but this can avoid the problem of emitting light signals that are harmful to the eyes of a person due to damage to the emitter.

[0032] In a second aspect, the present application provides a chip system, which includes one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected through a circuit. The chip system described above can be applied to an electronic device including a communication module and a memory. The interface circuit is configured to receive a signal from the memory of the electronic device and send the received signal to the processor, and the signal includes computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device can perform the method as described in the first aspect and any possible design manner thereof.

[0033] In a third aspect, the present application provides a computer readable storage medium, which includes computer instructions. When the computer instructions are run on an electronic device (such as a mobile phone), the electronic device executes the method as described in the first aspect and any possible design manner thereof.

[0034] In a fourth aspect, the present application provides a computer program product, which, when run on a computer, causes the computer to execute the method as described in the first aspect and any possible design manner thereof.

[0035] In a fifth aspect, an embodiment of the present application provides a face recognition apparatus, comprising a processor, the processor and a memory are coupled, and the memory stores program instructions, when the program instructions stored in the memory are executed by the processor, the apparatus implements the method in the first aspect and any possible design of the first aspect. The apparatus can be an electronic device or a server device; or can be a component of the electronic device or the server device, such as a chip.

[0036] In a sixth aspect, an embodiment of the present application provides a face recognition apparatus, the apparatus can be divided into different logical units or modules according to functions, each unit or module performs different functions, so that the apparatus performs the method in the first aspect and any possible design of the first aspect.

[0037] It can be understood that the beneficial effects of the chip system in the second aspect, the computer readable storage medium in the third aspect, the computer program product in the fourth aspect, and the apparatus in the fifth aspect and the sixth aspect can refer to the beneficial effects in the first aspect and any possible design of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 A structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0039] Figure 2 A principle schematic diagram of a TOF imaging technology provided by an embodiment of the present application;

[0040] Figure 3 A software module architecture schematic diagram provided by an embodiment of the present application;

[0041] Figure 4 An interaction schematic diagram between software modules provided by an embodiment of the present application;

[0042] Figure 5 Another interaction schematic diagram between software modules provided by an embodiment of the present application;

[0043] Figure 6 A signal interaction schematic diagram provided by an embodiment of the present application;

[0044] Figure 7 A display schematic diagram provided by an embodiment of the present application;

[0045] Figure 8 Another display schematic diagram provided by an embodiment of the present application;

[0046] Figure 9 Another display schematic diagram provided by an embodiment of the present application;

[0047] Figure 10 Another signal interaction schematic diagram provided for an embodiment of the present application;

[0048] Figure 11 A chip structure schematic diagram provided for an embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the description of the present application, unless otherwise specified, “at least one” refers to one or more, and “a plurality of” refers to two or more than two. In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, the same items or similar items with basically the same functions and roles are distinguished by using “first”, “second”, etc. The skilled in the art can understand that “first”, “second”, etc. do not limit the quantity and execution order, and “first”, “second”, etc. also do not necessarily mean different.

[0050] In order to make the description of each embodiment below clear and simple, first, a brief introduction of related concepts or technologies is given:

[0051] The rich execution environment (REE), which can also be referred to as a rich execution environment or a normal execution environment or an untrusted execution environment, refers to a system running environment of a mobile terminal, in which Android, IOS and Linux operating systems can be run. The REE has good openness and expansibility but low security.

[0052] The trusted execution environment (TEE), which can also be referred to as a secure side or a secure area, is an area that needs to be authorized to access. The TEE coexists with the REE in the running environment of an electronic device, and is supported by hardware to realize isolation from the REE, has security capabilities and can resist software attacks that the conventional REE side is vulnerable to. The TEE has its own running space and defines strict protection measures, so it has a higher security level than the REE and can protect assets (assets) in the TEE, such as data, software, etc., from software attacks and resist specific types of security threats.

[0053] The REE+TEE architecture refers to an architecture in which a TEE and an REE jointly provide services for an application. That is, the TEE and the REE coexist in an electronic device. By way of example, the TEE can implement an isolated running mechanism from the REE with the support of hardware. The TEE has its own running space and is of a higher security level than the REE, and can protect assets (such as data and software) in the TEE from software attacks. Only authorized security software can execute in the TEE, and the TEE also protects the confidentiality of resources and data of the security software. Compared with the REE, the TEE can better protect the security of data and resources due to its protection mechanisms such as isolation and permission control.

[0054] The TA (trusted application) is an application running in the TEE and can provide security services for a CA running outside the TEE, such as entering a password, generating a transaction signature, and face recognition.

[0055] The CA (client application) is an application running in the REE. The CA can call the TA through a client (Client) application programming interface (application programming interface, API) and instruct the TA to perform a corresponding security operation.

[0056] Software development kit (software development kit, SDK): in a broad sense, refers to a collection of related documents, examples, and tools that assist in the development of a certain type of software.

[0057] RAW data, that is, raw data, can be understood as "unprocessed and uncompressed data". In the embodiments of the present application, the RAW data can refer to the original data converted by the TOF camera from the light source signal captured to a digital signal. The RAW data also records some metadata (Metadata) generated by the camera shooting.

[0058] Metadata, also known as intermediate data or relay data, is data about data, mainly information describing data properties (property). In the embodiments of the present application, the Metadata can indicate the working mode of the camera, the light current value, the TOF camera device working state, the exposure value, and the like.

[0059] The TOF camera (TOF camera module) can include a transmitter (TX) and a receiver (RX), the TX is used to emit infrared light or laser pulses, and the RX is used to receive reflected light and form an image. Since the TX can autonomously emit light signals for imaging, the TOF image is not affected by most light in the environment, so that the application of the TOF image in the unlocking business can improve the security of face recognition.

[0060] Time of flight (TOF) imaging technology refers to emitting a group of infrared light (or laser pulses) that cannot be seen by human eyes, reflecting after encountering an object, ending at a camera, calculating the time difference or phase difference from emission to reflection to the camera, and collecting data to form a group of distance depth data, thereby obtaining a three-dimensional 3D model imaging technology. That is, the TOF imaging technology adds depth information from the Z-axis direction on the basis of the traditional 2D XY-axis imaging, and finally generates 3D image information.

[0061] When the TOF imaging technology is used, infrared light, laser and the like need to be projected to a face. In order to ensure the safety of human eyes, the optical power of the light emitted by TX needs to be detected, so as to ensure that the optical power of the light emitted by TX is within the safety range of human eyes, and the light signal emitted by TX does not harm human eyes.

[0062] The embodiment of the present application provides a face recognition method, which uses a TOF camera to collect images. The TOF camera can first work in an eye safety mode to determine whether TX is abnormal. If TX works normally, the TOF camera can be used to collect images normally, so that the optical power of the light emitted by TX is within the safety range of human eyes. If TX is abnormal, TX can be turned off to avoid the light signal emitted by TX from harming human eyes.

[0063] Figure 1 A structural schematic diagram of an electronic device 100 provided by the embodiment of the present application is provided.

[0064] As shown in Figure 1 The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, and the like.

[0065] Among them, the sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0066] It can be understood that the structure illustrated in the embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 can include more or fewer components than illustrated, or combine certain components, or split certain components, or different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0067] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0068] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching instructions and executing instructions.

[0069] The memory can also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that the processor 110 has just used or repeatedly uses. If the processor 110 needs to use the instructions or data again, it can be directly called from the memory. Avoiding repeated access, reducing the waiting time of the processor 110, thus improving the efficiency of the system.

[0070] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0071] It can be understood that the interface connection relationship between the modules shown in the embodiments is only illustrative and does not constitute a limitation on the structure of the electronic device 100. In other embodiments, the electronic device 100 can also use different interface connection manners or combinations of multiple interface connection manners.

[0072] The charging management module 140 is configured to receive charging input from a charger. The charging management module 140 can also supply power to the electronic device while charging the battery 142 via the power management module 141.

[0073] The power management module 141 is configured to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to supply power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160, etc. In other embodiments, the power management module 141 can also be disposed in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0074] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0075] Antenna 1 and antenna 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: antenna 1 can be multiplexed as a diversity antenna for wireless local area networks.

[0076] Mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc. applied on electronic device 100. Mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. Mobile communication module 150 can receive electromagnetic waves from antenna 1, and perform filtering, amplification, etc. on the received electromagnetic waves, and transmit them to a modem processor for demodulation. Mobile communication module 150 can also amplify signals modulated by the modem processor and convert them into electromagnetic waves radiated by antenna 1.

[0077] The modem processor can include a modulator and a demodulator. The modulator is used to modulate low-frequency baseband signals to be sent into medium-high frequency signals. The demodulator is used to demodulate received electromagnetic wave signals into low-frequency baseband signals. The demodulator then transmits the demodulated low-frequency baseband signals to the baseband processor for processing. The low-frequency baseband signals processed by the baseband processor are transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to loudspeakers 170A, microphones 170B, etc.), or displays images or videos through display screen 194.

[0078] Wireless communication module 160 can provide a solution for wireless communication including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. applied on electronic device 100. Wireless communication module 160 can be one or more devices integrated with at least one communication processing module. Wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and transmits the processed signals to processor 110. Wireless communication module 160 can also receive signals to be sent from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves radiated by antenna 2.

[0079] In some embodiments, the antenna 1 and the mobile communication module 150 of the electronic device 100 are coupled, and the antenna 2 and the wireless communication module 160 are coupled, so that the electronic device 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include a global positioning system (GPS), a global navigation satellite system (GLONASS), a beidu navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or a satellite based augmentation systems (SBAS).

[0080] The electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0081] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), a light-emitting diode (LED), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED).

[0082] The electronic device 100 can realize the shooting function through the ISP, camera 193, video codec, GPU, display screen 194 and application processor. The ISP is used to process the data fed back by the camera 193. The camera 193 is used to capture still images or videos. The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0083] The camera 193 may include 1 to N cameras. For example, the electronic device may include 2 front cameras and 4 rear cameras. Among them, the front camera may include a TOF camera. The TOF camera includes TX and RX, TX can be used to transmit light signals (infrared light or laser pulses), and RX can be used to receive imaging. TX can be, for example, an infrared light transmitter. RX can be, for example, a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor.

[0084] For example, Figure 2As shown in (a) of FIG. 1, a light signal (infrared light or laser pulse) can be continuously sent to a target (e.g., a user) to be measured by a light emitter (Tx) of a TOF camera, and a light signal returned by the target to be measured can be received at a sensor end (Rx) of the TOF camera, as shown in (b) of FIG. 1. Figure 2 As shown in (b) of FIG. 1, depth information of the target to be measured can be obtained according to a phase difference (delay) between the emitted and received light signals.

[0085] The Tx and the Rx can exchange information through a bus. For example, the Rx can send a configuration parameter to the Tx through the bus (e.g., a serial peripheral interface (SPI) bus), and the configuration parameter can be used to indicate an address of a register corresponding to the Tx and a value for the register. For example, the address of the register corresponding to the Tx can be 0x11, and a current value can be stored in a storage space corresponding to 0x11. The Tx can work at the corresponding current value based on the corresponding configuration parameter, so as to emit a light signal with a corresponding light intensity. The Rx can obtain corresponding image data based on reflected light of the light signal with the corresponding intensity emitted by the emitter. It should be noted that the Tx can emit light signals with different light intensities when working at different current values. For example, the Tx can emit a light signal with a first light intensity when working at a first current value. The Tx can emit a light signal with a second light intensity when working at a second current value. The second current value is greater than the first current value. The second light intensity is greater than the first light intensity. The image data obtained by the Rx based on the reflected light of the light signals with different intensities is also different. For example, when the Tx emits a light signal with the first light intensity by working at the first current value, the Rx obtains first image data within a corresponding exposure time; when the Tx emits a light signal with the second light intensity by working at the second current value, the Rx obtains second image data within a corresponding exposure time; and the second image data is different from the first image data.

[0086] When the Tx works at a corresponding current value, the working state of the Tx can be determined, and the working state can be normal or abnormal. The Rx can request the working state of the Tx through the bus, and the Tx can feed back the working state (e.g., normal or abnormal) of the Tx to the Rx through the bus, so that the Rx can obtain the working state of the Tx. The Rx can pack the working state of the Tx, the working state of the Rx, and the working mode of the two in a first data packet (e.g., Metadata). The Rx can also pack the Metadata and the image data obtained based on the reflected light in a second data packet (e.g., RAW Data).

[0087] The NPU is a neural-network (NN) computing processor. By drawing on the structure of a biological neural network, for example, by drawing on the transmission mode between human brain neurons, the NPU can quickly process input information and can also continuously self-learn. Through the NPU, the electronic device 100 can implement intelligent cognition and other applications, for example, image recognition, face recognition, voice recognition, text understanding, and the like.

[0088] The external memory interface 120 can be used to connect an external memory card, for example, a Micro SD card, to implement the expansion of the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are saved in the external memory card. The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 implements various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. For example, in the embodiments of the present application, the processor 110 can run the instructions stored in the internal memory 121, and the internal memory 121 can include a storage program area and a storage data area. The storage program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, and the like), and the like. The storage data area can store data created during the use of the electronic device 100 (such as audio data, a phonebook, and the like), and the like. In addition, the internal memory 121 can include a high-speed random access memory and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.

[0089] The electronic device 100 can implement an audio function through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, and the like. For example, music playing, recording, and the like.

[0090] The audio module 170 is used to convert digital audio information into an analog audio signal output and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode an audio signal. The speaker 170A, also called a “loudspeaker”, is used to convert an audio electrical signal into a sound signal. The receiver 170B, also called a “earpiece”, is used to convert an audio electrical signal into a sound signal. The microphone 170C, also called a “microphone”, “sounder”, is used to convert a sound signal into an electrical signal. The earphone interface 170D is used to connect a wired earphone.

[0091] The buttons 190 include a power button, a volume button, etc. The buttons 190 can be mechanical buttons. They can also be touch buttons. The electronic device 100 can receive button inputs and generate key signal inputs related to the user settings and function controls of the electronic device 100. The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts or for touch vibration feedback. The indicator 192 can be an indicator light that can be used to indicate the charging status, power changes, messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or removing it from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc.

[0092] The methods in the following embodiments can all be implemented in the electronic device 100 having the above hardware structure.

[0093] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0094] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through interfaces. In some embodiments, the Android system may include an application layer, an application framework layer, an Android runtime (Android runtime) and a system library, a hardware abstraction layer (HAL) and a kernel layer. It should be noted that the embodiments of the present application are illustrated by taking the Android system as an example. In other operating systems (such as Hongmeng system, IOS system, etc.), as long as the functions implemented by each functional module are similar to those of the embodiments of the present application, the solutions of the present application can also be implemented.

[0095] Among them, the application layer can include a series of application packages.

[0096] like Figure 3 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, lock screen application, setting application, etc. Of course, the application layer may also include other application packages, such as payment application, shopping application, banking application, etc., which are not limited in this application.

[0097] The application with the function of entering a face is used for face unlocking. The lock screen application has a function of responding to a user's unlocking operation (for example, pressing the power key) to perform unlocking. The lock screen application can perform face unlocking, fingerprint unlocking, password unlocking, and the like. The embodiments of the present application mainly take face unlocking as an example for description.

[0098] The application framework layer provides an application programming interface (API) and a programming framework for the application in the application layer. The application framework layer includes some pre-defined functions. For example, it can include an activity manager, a window manager, a content provider, a view system, a resource manager, a notification manager, a camera service, a face recognition service, and the like. The embodiments of the present application do not make any limitation thereto.

[0099] The system library can include a plurality of function modules. For example, a surface manager, media libraries, OpenGL ES, SGL, and the like.

[0100] The surface manager is used for managing a display subsystem and providing fusion of 2D and 3D layers for a plurality of applications.

[0101] The media library supports playback and recording of a plurality of commonly used audio, video formats, and the like, and static image files. The media library can support a plurality of audio and video encoding formats, for example, MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, and the like.

[0102] The OpenGL ES is used for realizing three-dimensional graphics drawing, image rendering, synthesis, and layer processing, and the like.

[0103] The SGL is a drawing engine for 2D drawing.

[0104] The Android runtime includes a core library and a virtual machine. The Android runtime is responsible for scheduling and management of the Android system. The core library includes two parts: one part is a function function required to be called by the java language, and the other part is the core library of the Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java files of the application layer and the application framework layer into binary files. The virtual machine is used for performing functions of management of an object life cycle, stack management, thread management, security and exception management, and garbage collection.

[0105] The HAL layer is a packaging of the Linux kernel driver, which provides an interface upward and shields the implementation details of the low-level hardware.

[0106] The HAL layer can include a Wi-Fi HAL, an audio HAL, a camera HAL (Camera HAL), a face recognition control module (Face CA), and the like.

[0107] The camera HAL is a core software framework of the Camera, and can include a sensor node and an image front end (IFE) node. The sensor node and the IFE node are components (nodes) in a transmission path (also referred to as a transmission pipeline) of image data and control instructions created by the camera HAL.

[0108] The face recognition control module is a core software framework / application of face recognition.

[0109] Face Trusted Application (Face TA): an application for face recognition running in a TEE environment. In the embodiments of the present application, the Face TA is referred to as a face recognition TA.

[0110] The kernel layer is a layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0111] The camera driver is a driver layer of the Camera device, and is mainly responsible for interaction with the hardware.

[0112] The hardware layer includes a display, a TOF camera, an IFE module, and a secure buffer, and the like.

[0113] The secure buffer refers to a memory with a security protection function, and can be used to store raw data collected by the TOF camera.

[0114] The TOF camera, which can also be referred to as a TOF sensor, can include a transmitter (TX) and a receiver (RX). The TX is used to emit infrared light or laser pulses, and the RX is used to receive reflected light and form an image.

[0115] IFE module (IFE-Lite): can be referred to as an image front end module, and can be used to forward image data without processing the image data in the forwarding process.

[0116] The software modules involved in the face recognition method provided by the embodiments of the present application and the interaction between the modules are described below. Figure 4As shown, the lock screen application in the application layer can interact with the face recognition SDK, the face recognition SDK can interact with the face recognition service in the framework layer by calling a preset application programming interface (API) interface, the face recognition service can interact with the face recognition control module in the HAL layer, the face recognition control module can interact with the camera HAL in the HAL layer through the camera service in the framework layer, or the face recognition control module can directly interact with the camera HAL in the HAL layer. The camera HAL can include a sensor node and an IFE node. The sensor node can interact with the camera driver module in the kernel layer, and the camera driver module can be used to drive the TOF camera in the hardware layer to collect image data in a default working mode (for example, a human eye safety mode, which can be specifically referred to as the description in S112 below). The IFE module can store the image data collected by the TOF camera into the secure memory. The storage location of the image data collected by the TOF camera in the secure memory can be represented using a file descriptor (FD). The IFE module can send the FD (for example, FD1) of the image data to the camera driver module, and the camera driver module can pass the FD1 to the IFE node of the camera HAL. After the IFE node receives the FD1, the IFE node can pass the FD1 to the sensor node of the camera HAL. After the sensor node receives the FD1, a process of reading the working state of the TOF camera is triggered. That is, the sensor node can read the working state register (that is, a register used to store the working state of the TOF camera) of the TOF camera through the camera driver module, determine the human eye safety detection result according to the value of the working state register, and switch the working mode of the TOF camera according to the human eye safety detection result. After the working mode of the TOF camera is switched, the camera HAL can continue to interact with the camera driver module, so that the camera driver module can drive the TOF camera to collect image data in the switched working mode (for example, a face ID mode, which can be specifically referred to as the description in S112 below). The image data can be stored in the secure memory. The FD (for example, FD2) corresponding to the image data can be passed to the face recognition TA through the IFE module, the camera driver module, the IFE node, the camera service and the face control module. The face recognition TA can read the image data from the secure memory according to the FD2 and process the image data, and feed back the processing result (face recognition success or face recognition failure) to the face recognition control module. The face recognition control module can feed back the processing result to the lock screen application through the face recognition service and the face recognition SDK, so that the lock screen application determines whether to unlock (if the face recognition is successful, unlock; if the face recognition fails, do not unlock, that is, unlock failure). Figure 4 The solid arrow in the figure can be used to represent the control flow, and the dashed arrow can be used to represent the data flow.

[0117] Specifically, as shown in Figure 5 the sensor node in the camera HAL can be used to select the working mode of the TOF camera, including the human eye safety mode (the first working mode), the face ID mode (the second working mode), and the TX off mode (the third working mode), etc. The specific working modes can be referred to the description of S112 below. The default initial working mode of the TOF camera can be the human eye safety mode. When the TOF camera works in the human eye safety mode, the human eye safety current value (the first current value) calibrated by the production line can be read from the memory, the human eye safety mode configuration is updated according to the current value, and the configuration parameters of the human eye safety mode can be sent to the camera driving module. The camera driving module can be used to drive the TOF camera in the hardware layer to collect image data in the default working mode (for example, the human eye safety mode, which can be referred to the description of S112 below). The storage location of the image data collected by the TOF camera in the secure memory can be represented by FD. The IFE module can send the FD (for example, FD1) of the image data to the camera driving module, the camera driving module can pass the FD1 to the IFE node of the camera HAL, the IFE node can pass the FD1 to the sensor node of the camera HAL after receiving the FD1, and the sensor node can read the working state register (that is, the register used to store the working state of the TOF camera) of the TOF camera through the camera driving module, and determine the human eye safety detection result according to the value of the working state register. After obtaining the human eye safety detection result, the mode switching process can be performed. Specifically, if the human eye safety detection result is successful, the TOF camera can be switched to the face ID mode. If the human eye safety detection result fails, the TOF camera can be switched to the TX off mode. In this way, the human eye safety can be ensured.

[0118] For ease of understanding, the method provided by the embodiments of the present application is specifically introduced below with reference to the accompanying drawings.

[0119] As shown in Figure 6 the present application provides a method for human eye safety detection and face recognition based on TOF image, and the flow is as follows:

[0120] S101, the lock screen application calls the face recognition SDK to perform face recognition.

[0121] When the user's unlocking operation (the first operation) is detected, the lock screen application calls the face recognition SDK to perform face recognition. The user's unlocking operation includes picking up the phone, pressing the power button, operating on the screen (clicking, sliding, etc.), or pulling out the charging line.

[0122] Meanwhile, the lock screen application can register a callback to the face recognition SDK, and the registration of the callback is to return the face recognition result to the lock screen application when the face recognition SDK obtains the face recognition result.

[0123] S102, the face recognition SDK sends a face recognition request to the face recognition service.

[0124] The face recognition request carries an identification of a face recognition type, a resolution size of an image, and a data stream format. The face recognition type includes a 2D face recognition type (e.g., which can correspond to identification 0) and a 3D face recognition type (e.g., which can correspond to identification 1).

[0125] For example, the face recognition type carried in the face recognition request can be 1 (i.e., the 3D face recognition type), the resolution size of the image can be 1280x2898 pixels, and the data stream format can be raw image format (RAW) 16.

[0126] Meanwhile, the face recognition SDK can register a callback to the face recognition service, and the registration of the callback is to return the face recognition result to the face recognition SDK when the face recognition service obtains the face comparison result.

[0127] S103, the face recognition service sends a face recognition request to the face recognition control module.

[0128] The face recognition request can refer to the description of S102, which will not be repeated here.

[0129] That is, the face recognition SDK can notify the face recognition control module to perform face recognition through the face recognition service. The face recognition service can send the face recognition request received from the face recognition SDK to the face recognition control module.

[0130] Meanwhile, the face recognition service can register a callback to the face recognition control module, and the registration of the callback is to return the face comparison result to the face recognition service when the face recognition control module obtains the face comparison result.

[0131] S104, in response to receiving the face recognition request, the face recognition control module matches the camera according to the face recognition request.

[0132] Specifically, the face recognition control module can obtain the identification of the face recognition type, the resolution size of the image, and the data stream format from the face recognition request, and determine the matched camera by querying the camera service from the camera capability.

[0133] It should be understood that, in the booting process of the electronic device, the camera service can send a camera capability query request to the camera HAL, the camera capability query request being used to request to query the camera capability supported by the electronic device. After receiving the camera capability query request, the camera HAL can send the camera capability supported by the electronic device to the camera service, and the camera service can store the received camera capability supported by the electronic device. The camera capability supported by the electronic device includes the camera identity (ID) of each camera, the maximum resolution size supported, the format of the data stream, and whether the camera supports collecting depth information, and the like.

[0134] For example, it is assumed that three cameras are installed on the mobile phone, and the capability information of the three cameras can be as shown in Table 1:

[0135] Table 1

[0136] Camera ID Mounting position Maximum resolution supported Data stream format Depth information 1 Rear-facing 4096x3072 pixel YUY No 2 Front-facing 3264x2448 pixel YUY No 3 Front-facing 1280x2898 pixel RAW16 Yes

[0137] The camera with the camera ID of 3 is a TOF camera and supports collecting depth information. The cameras with the camera IDs of 1 and 2 are ordinary cameras and do not support collecting depth information. Of course, more front or rear cameras can be installed on the mobile phone, for example, the mobile phone can be installed with 2 front cameras and 4 rear cameras.

[0138] The face recognition control module can send a camera capability query request to the camera service, the camera service can send the camera capability supported by the electronic device to the face recognition control module, and the face recognition control module can determine the matched camera according to the camera capability supported by the electronic device, for example, the matched camera can be determined as the camera with the ID of 3 (i.e., the TOF camera).

[0139] It should be noted that Table 1 is only an example, and the data stream format corresponding to each camera can include multiple types. For example, the camera with the camera ID of 1 can correspond to not only the YUY data stream format but also the RAW16 data stream format, which is not limited in the present application.

[0140] S105, the face recognition control module sends a request to open a camera (Camera) to the camera service.

[0141] Exemplarily, the face recognition control module can send a request for opening the Camera to the camera service through a vendor native development kit (VNDK) interface. The request for opening the Camera carries information such as a security identifier, a camera ID, a resolution size, and a data stream format. The security identifier is used to indicate that the data is stored in a secure buffer. That is, the security identifier can be used to apply for a piece of secure memory, which is used to store the data collected by the camera subsequently. For example, the security identifier can be 1 or 0, 1 indicating that the data is stored in the secure buffer and 0 indicating that the data is stored in a non-secure buffer.

[0142] Exemplarily, the security identifier carried in the request for opening the Camera can be 1 (that is, the data is stored in the secure buffer), the resolution size of the image can be 1280x2898 pixels, the data stream format can be RAW16, and the camera ID can be 3.

[0143] Meanwhile, the face recognition control module can register a callback with the camera service, and the callback is used to notify the face recognition control module that the camera service has completed the opening of the Camera after the camera service completes the opening of the Camera.

[0144] S106, in response to receiving the request for opening the Camera, the camera service sends a request for opening the Camera to the camera HAL, and the request for opening the Camera carries information such as the security identifier, the camera ID, the resolution size, and the data stream format.

[0145] During the process of invoking the camera HAL by the camera service, the camera service can send the camera HAL information such as the security identifier, the camera ID, the resolution of the image, and the data stream format. The camera HAL can cache the information such as the security identifier, the camera ID, the resolution of the image, and the data stream format for a preset time.

[0146] Meanwhile, the camera service can register a callback with the camera HAL, and the callback is used to notify the camera service of the result of creating a path by the camera HAL.

[0147] S107, the camera HAL creates a corresponding path according to the camera ID, the resolution of the image, and the data stream format.

[0148] The camera HAL can select available nodes according to the camera ID, resolution, and data stream format, and create a corresponding path according to the available nodes. For example, if the resolution is 1280x2898 pixels, the data stream format is RAW16, and the camera ID is 3, it can be determined that the sensor node and the IFE node are selected. This is because the sensor node and the IFE node can support the transmission of data with a resolution of 1280x2898 pixels, a data stream format of RAW16, and a camera ID of 3 collected by the camera.

[0149] The path corresponding to the sensor node can be a path composed of a sensor node, a camera driver, a TOF camera, an IFE module, and a secure memory. The path corresponding to the IFE node can be a path composed of an IFE module (carrying an FD), a camera driver, and an IFE node. The camera HAL can connect the output port of the sensor node and the input port of the IFE node at the HAL layer. Thus, the path corresponding to the sensor node and the path corresponding to the IFE node can form a closed loop path. After the path is created, the hardware in the path is powered on (i.e., the hardware circuit is powered on) and waits for a data request.

[0150] S108, the camera HAL returns the result of creating the path to the camera service.

[0151] The result of creating the path can be success or failure. If the result of creating the path is failure, the camera HAL notifies the camera service that the path creation fails. If the result of creating the path is success, the camera HAL notifies the camera service that the path creation is successful, and the subsequent steps S109 and the like can be continued.

[0152] S109, in response to receiving the notification that the path creation is successful, the camera service returns a message that the camera opening is completed to the face recognition control module.

[0153] It can be understood that the camera opening completion refers to that the preparation work (e.g., camera parameter configuration, power-on, and the like) before photographing or video shooting by the camera has been completed.

[0154] S110, in response to receiving the message that the camera opening is completed, the face recognition control module sends a data request to the camera service.

[0155] The data request is used to request to obtain the data stream of the camera.

[0156] S111, in response to receiving the data request sent by the face recognition control module, the camera service calls the camera HAL to obtain the data stream.

[0157] S112, the camera HAL selects a camera working mode through the sensor node.

[0158] Specifically, the sensor node can select the camera working mode corresponding to the sensor node through the camera resolution and the data stream format cached in S106. For example, the sensor node can select the camera working mode corresponding to the sensor node by looking up a table (for example, Table 2).

[0159] Table 2

[0160]

[0161] Among them, the EyeSafe Mode refers to a mode in which the Tx of the TOF camera works at a small current (a current smaller than a preset threshold, a first current value). The EyeSafe Mode is used to check whether the TOF camera is damaged. The Face ID Mode refers to a mode in which the Tx of the TOF camera works at a normal current (a second current value within a preset threshold range). The Face ID Mode is used for scenarios such as secure face unlocking and secure payment. The second current value is greater than the first current value. The Tx OFF Mode refers to a mode in which the Tx of the TOF camera is not powered on (so as not to emit light). The Tx OFF Mode is used in the case where the Tx device of the TOF camera is damaged or cannot work normally. This is because the use of the Tx device in the damaged state will have an adverse effect on the human eye. Therefore, when it is detected that the Tx device of the TOF camera is damaged, the Tx OFF Mode is adopted to make the Tx device not powered on, so as to avoid causing harm to the human eye.

[0162] Of course, the working mode of the camera can also include more, which is not limited by the present application.

[0163] According to Table 2, when the maximum value of the resolution of the image is 1280x2898 pixels and the data stream format is Raw, the working mode of the camera can include the EyeSafe Mode, the Face ID Mode, and the Tx OFF Mode. The sensor node can set the initial working mode of the camera as the EyeSafe Mode by default. When the working mode of the camera is the EyeSafe Mode, the sensor node can read the EyeSafe current value (i.e., the current value that is harmless to the human eye) calibrated by the production line from the memory (for example, oeminfo), and update the EyeSafe Mode setting of the TOF camera according to the EyeSafe current value. For example, the address of the current register of the TOF camera can be obtained by looking up the table, and the EyeSafe current value is written into the current register of the TOF camera. It should be understood that the sensor node can store the addresses of various registers of the TOF camera, and the addresses of various registers of the TOF camera can be as shown in Table 3.

[0164] Table 3

[0165] Register identification Storage data type Address 1 Current 0x1 2 Resolution 0x2 3 Data stream format 0x3 4 TOF camera device working state 0x4 5 TOF camera working mode 0x5

[0166] For example, by referring to Table 3, it can be determined that the address of the register corresponding to the current value is 0x1, and thus the eye-safe current value can be written into the storage space corresponding to 0x1.

[0167] In S113, the sensor node sends the configuration parameter of the eye-safe mode to a camera driver module in the Kernel layer.

[0168] For example, the configuration parameter of the eye-safe mode can be: the current value is 700 mA, the IR grayscale image exposure time is 10 μs, and the depth image exposure time is 10 μs.

[0169] In S114a, the camera driver module writes (updates) the configuration parameter of the eye-safe mode into the register of the TOF camera.

[0170] That is, the camera driver module can send the configuration parameter of the eye-safe mode to the TOF camera.

[0171] For example, the camera driver module can write the configuration parameter of the eye-safe mode into the register of RX of the TOF camera through an inter-integrated circuit (I2C). The address corresponding to the register of RX can be 0x01. The register corresponding to RX can include multiple registers, which are not limited by the present application. That is, the configuration parameter of the eye-safe mode can be sent to RX of the TOF camera through I2C. The configuration parameter of the eye-safe mode includes the configuration parameter for RX and TX. For example, the configuration parameter for TX can be a first current value. The configuration parameter for RX can be an exposure time. RX can write the configuration parameter corresponding to TX into the register corresponding to TX through a SPI bus. The address of the register corresponding to TX can be 0x11. The register corresponding to TX can include multiple registers, which are not limited by the present application.

[0172] In S114b, the camera driver module sends a stream on command / instruction (second start command) to the TOF camera.

[0173] The stream on command is used to drive the TOF camera to collect data.

[0174] It should be noted that before S114b and after S114a, the camera driver module can also send a message indicating that the writing of the configuration parameter is completed to the sensor node; in response to receiving the message indicating that the writing of the configuration parameter is completed, the sensor node sends a start command (first start command) to the camera driver module.

[0175] S115, in response to receiving the stream on command, the TOF camera collects RAW Data1 based on the eye safety mode.

[0176] Specifically, in response to receiving the stream on command, the RX can send a request for the light-emitting signal to the TX, and the TX works at a corresponding current value (first current value) to send a light signal of a first light intensity; the RX receives the light signal at a corresponding exposure time (for example, 10us), and the light signal received by the RX includes reflected light of the light signal of the first light intensity. Based on the received light signal, the RX obtains first image data.

[0177] That is, RAW Data 1 (raw data 1) refers to the image data (first image data) obtained by the Rx of the TOF camera receiving reflected light and imaging when the Tx of the TOF camera works at the eye safety current value calibrated on the production line to emit a light signal to the face. Wherein, the light signal emitted by the Tx working at the eye safety current value calibrated on the production line is of a first light intensity.

[0178] Wherein, the RAW Data contains Metadata. For example, the Metadata stores information such as the working mode of the current TOF camera (for example, eye safety mode), the light-emitting current value (for example, eye safety current value calibrated on the production line), the working state of the TOF camera device (for example, normal or abnormal), and the image exposure value (for example, 10us).

[0179] S116, the TOF camera sends the RAW Data 1 collected based on the eye safety mode to the IFE module.

[0180] For example, the TOF camera can transmit the RAW Data 1 collected by the TOF camera to the IFE module through the mobile industry processor interface (MIPI). The IFE module can also be referred to as an image front-end processing module (IFE-Lite), and the IFE module can not process the RAW Data 1.

[0181] S117, the IFE module sends the RAW Data 1 to the secure buffer for storage.

[0182] The storage location of the RAW Data 1 collected by the TOF camera based on the eye safety mode in the secure buffer can be represented by FD1.

[0183] For example, when FD1 is 69, it can represent that the storage location is XX secure buffer; when FD1 is 96, it can represent that the storage location is YY non-secure buffer (normal memory).

[0184] S118, the IFE module sends the FD1 to the camera driver module.

[0185] S119, the camera driver module sends the FD1 to the IFE node.

[0186] S120, the IFE node sends the FD1 to the sensor node of the camera HAL.

[0187] S121, in response to receiving the FD1, the sensor node sends a request for reading the working status register to the camera driver module, and the request carries the address of the working status register.

[0188] The request for reading the register is used to request the camera driver module to read the value of the register storing the working status of the TOF camera.

[0189] S122, the camera driver module reads the value of the working status register according to the address of the working status register.

[0190] The working status register can be set on the TOF camera, so the camera driver module can read the value of the corresponding register from the TOF camera according to the address of the working status register.

[0191] It should be noted that the working status of the TOF camera is stored in the working status register. The working status of the TOF camera can be represented by 0 or 1, 0 can represent that the working status is abnormal, and 1 can represent that the working status is normal.

[0192] S123, the camera driver module sends the value of the working status register to the sensor node of the camera HAL.

[0193] S124, the sensor node of the camera HAL calculates the eye safety detection result according to the value of the working status register.

[0194] If the device working status is normal (for example, the value of the working status register is 0), the eye safety detection result is safe / normal (or the eye safety detection is successful). If the device working status is abnormal (for example, the value of the working status register is 1), the eye safety detection result is unsafe / abnormal (or the eye safety detection fails).

[0195] S125, the sensor node of the camera HAL determines the working mode of the TOF camera based on the eye safety detection result.

[0196] If the eye safety detection result is safe (normal), it is determined that the working mode of the TOF camera is the face ID mode; if the eye safety detection result is unsafe (abnormal), it is determined that the working mode of the TOF camera is the TX off mode.

[0197] It should be noted that the sensor node can store configuration parameters corresponding to the face ID mode and the TXOFF mode.

[0198] For example, the configuration parameters corresponding to the face ID mode can be: the current value (second current value) is 2800mA, the IR grayscale image exposure time is 500μs, the depth is yes, and the image exposure time is 800μs. The configuration parameters corresponding to the TXOFF mode can be: the current value (third current value) is 0mA, the IR grayscale image exposure time is 10μs, the depth is no, and the image exposure time is 10μs.

[0199] It should be noted that the execution order of S101-S125 is not limited in the embodiments of the present application. In some embodiments, after S101-S107 are executed, S112-S124 can be directly executed, S124 is executed again after S108-S111, and S125 can be executed again after S111. Of course, S101-S125 can also have other combination orders to ensure that the sensor node of the camera HAL can obtain the eye safety detection result, so as to determine the working mode of the TOF camera, which will not be described here.

[0200] The following will be described taking the working mode of the TOF camera determined by the sensor node as the face ID mode as an example. After S125, S126-S142 can also be included:

[0201] S126, the sensor node sends the configuration parameters of the face ID mode to the camera driving module.

[0202] S127, the camera driving module writes the configuration parameters of the face ID mode into the register of the TOF camera to drive the TOF camera to collect data based on the face ID mode.

[0203] That is, the camera driving module can send the configuration parameters of the face ID mode to the TOF camera.

[0204] For example, the camera driving module can write the configuration parameters of the face ID mode into the register of the TOF camera through I2C. That is, the camera driving module can send the configuration parameters of the face ID mode to the TOF camera through I2C.

[0205] S128, the TOF camera collects RAW Data 2 based on the face ID mode.

[0206] The RAW Data 2 can be image data obtained by the TOF camera when the TOF camera transmits light signals to the face at a second current value (e.g., 2800 mA) and receives reflected light and images the reflected light. The light signals transmitted by the TOF camera at the second current value have a second light intensity. The second light intensity is greater than the first light intensity.

[0207] The RAW Data 2 includes Metadata. For example, the Metadata stores information such as a working mode (e.g., face ID mode) of the TOF camera, a lighting current value (e.g., 2800 mA), a working state (e.g., normal) of the TOF camera device, and an image exposure time (e.g., 800 μs).

[0208] S129. The TOF camera sends the RAW Data 2 to the IFE module.

[0209] For example, the TOF camera can transmit the RAW Data 2 collected by the TOF camera based on the face ID mode to the IFE module through MIPI.

[0210] S130. The IFE module stores the RAW Data 2 in the secure memory.

[0211] The storage location of the RAW Data 2 collected by the TOF camera based on the face ID mode in the secure memory can be denoted as FD2.

[0212] The FD2 in this step can be the same as or different from the FD1 in S117. When the FD2 in this step is the same as the FD1 in S117, the RAW Data 2 collected by the TOF camera based on the face ID mode is stored in the same secure memory as the RAW Data 1 collected by the TOF camera based on the eye safety mode in S117. The RAW Data 1 collected by the TOF camera based on the eye safety mode in S117 can be deleted, so that the RAW Data 2 collected by the TOF camera based on the face ID mode can be stored in the secure memory again. When the FD2 is different from the FD1, the RAW Data 2 collected by the TOF camera based on the face ID mode and the RAW Data 1 collected by the TOF camera based on the eye safety mode in S117 can be stored in different secure memories.

[0213] S131. The IFE module sends the FD2 to the camera driving module.

[0214] S132. The camera driving module sends the FD2 to the IFE node.

[0215] S133, the IFE node sends the FD2 to the camera service through the interface of the camera HAL.

[0216] S134, the camera service sends the FD2 to the face recognition control module.

[0217] S135, the face recognition control module sends the FD2 to the face recognition TA.

[0218] S136, the face recognition TA reads RAW Data 2 from the secure memory according to the FD2.

[0219] S137, the face recognition TA obtains a face recognition result according to the RAW Data 2.

[0220] Specifically, the face recognition TA can obtain the working mode of the TOF camera from the Metadata in the RAW Data 2, which can be a face ID mode for example. Then, the face recognition TA can process the second image data in the RAW Data 2 to obtain a first grayscale image and a first depth image through a TOF algorithm, and perform face recognition based on the first grayscale image, and anti-counterfeit detection based on the first depth image, so as to obtain the face recognition result.

[0221] It should be noted that the face recognition TA also stores the face information of the user previously entered, and the TOF algorithm can convert the face information entered by the user into a grayscale image and a depth image. If the grayscale image corresponding to the face information entered by the user (RAW Data 2 collected by the TOF camera based on the face ID mode) and the face information entered previously (RAW Data collected by the electronic device when the user performs the face entry operation) matches, it can be considered that it is the same user (i.e., the user performing the face entry operation and the user performing the unlocking operation is the same user), and if the currently collected face information includes depth information, it can be considered that the current user is real and reliable (not a photo, video, etc. fake), at this time, it can be considered that the face of the current user is safe, that is, the face recognition result is successful. If the grayscale image corresponding to the face information entered by the user (RAW Data 2 collected by the TOF camera based on the face ID mode) and the face information entered previously (RAW Data collected by the electronic device when the user performs the face entry operation) does not match, or if the currently collected face information does not include depth information, it is considered that the face of the current user is not safe, that is, the face recognition result is failed.

[0222] S138, the face recognition TA sends the face recognition result to the face recognition control module.

[0223] S139, the face recognition control module sends the face recognition result to the face recognition service.

[0224] The face recognition control module may transmit the face recognition result (success or failure) to the face recognition service based on the callback previously registered by the face recognition service (in S103 ).

[0225] S140. The face recognition service transmits the face recognition result to the face recognition SDK.

[0226] The face recognition service passes the face recognition result (success or failure) to the face recognition SDK based on the callback previously registered by the face recognition SDK (in S102).

[0227] S141. The face recognition SDK transmits the face recognition result to the lock screen application.

[0228] The face recognition SDK passes the face recognition result (success or failure) to the lock screen application based on the callback registered by the lock screen application previously (in S101).

[0229] S142. The lock screen application decides whether to unlock the screen based on the face recognition result.

[0230] If the facial recognition result is successful, the lock screen application can be successfully unlocked, and the electronic device can display the desktop or application interface (system application or third-party application). If the facial recognition result is unsuccessful, the lock screen application will not be unlocked, that is, the face unlocking has failed. After the face unlocking fails, the lock screen application can disable the face recognition function for a period of time (for example, 5 minutes) when the face recognition failed.

[0231] For example, if the user sets up face unlock, such as Figure 7 As shown in (a) in FIG, when the user picks up the phone for face recognition, in response to the user picking up the phone, as shown in FIG. Figure 7 As shown in (b) in FIG. 7 , the mobile phone may display a lock screen interface 701 . During the face recognition process, the mobile phone may display an unlock icon 702 and a prompt text “Recognizing face” 703 on the lock screen interface 701 .

[0232] If face recognition is successful, Figure 8 As shown in (a) of FIG, an interface 704 may be displayed, which may include an unlock icon 705 (open, which can vividly prompt the user that the face unlock is successful) and a prompt text "Swipe up to enter" 706. In response to the user's swipe up operation, the mobile phone may display the desktop or the interface of an application (system application or third-party application). Or, as Figure 8 As shown in (b), if the face recognition is successful, the mobile phone can be unlocked directly without the need for additional user operation, that is, the desktop 707 can be displayed immediately (or the application interface can be directly displayed).

[0233] If face recognition fails,Figure 9 As shown in (a) in the figure, an interface 708 may be displayed, which may include an unlock icon 709 (in a closed state, which can vividly prompt the user that the face unlocking is not successful) and a prompt text "Unrecognized, double-click the screen to try again" 710. In response to the user's double-click operation, the mobile phone can perform face recognition again (i.e., collect the user's face information again for comparison and anti-counterfeiting judgment). Or, in response to the user's upward sliding operation in the interface 708, such as Figure 9 As shown in (b), the mobile phone can display an interface 711. After entering the interface 711, the mobile phone can perform face recognition again. The interface 711 may include a face recognition icon 712 and a prompt text "Face recognition in progress" 713. If the recognition is still unsuccessful, Figure 9 As shown in (c), the mobile phone can display an interface 714, which can include a prompt text 715 "Unsuccessful recognition, click here to try again". The user can click the corresponding position to re-trigger face recognition, or can also enter a password through the soft keyboard 716 to unlock, avoiding the problem of poor user experience caused by unsuccessful recognition.

[0234] It should be noted that the above embodiment uses the method flow of face unlocking by lock screen application as an example to illustrate the working mode selection of TOF camera. The working mode selection of TOF camera can also be applied to payment or transfer (for example, the user is in payment application / financial application / chat application / shopping application (for example, ) for payment or transfer operations), face recognition for user security registration or login to applications (for example, users in This application does not limit the scenarios such as registering or logging in (for example, in the lock screen application). That is, the lock screen application can be replaced with a shopping application, chat application, payment application, banking application, or financial application, etc. This application does not limit it.

[0235] The following is an example of the working mode of the TOF camera determined by the sensor node as TX off mode. Figure 10 As shown, after S125, S150-S167 may also be included:

[0236] S150: The sensor node of the camera HAL determines that the operating mode of the TOF camera is the TX off mode based on the eye safety detection result.

[0237] S151. The sensor node sends the configuration parameters of the TX off mode to the camera driver module.

[0238] S152. The camera driver module writes (updates) the configuration parameters of the TX off mode into the register of the TOF camera to drive the TOF camera to collect data.

[0239] That is, the camera driving module can send the configuration parameters of the TX-off mode to the TOF camera.

[0240] For example, the camera driving module can write the configuration parameters of the TX-off mode into the TOF camera register through I2C. That is, the camera driving module sends the configuration parameters of the TX-off mode to the TOF camera through I2C.

[0241] S153, the TOF camera collects RAW Data 3 based on the TX-off mode.

[0242] The RAW Data 3 can be the image data (third image data) obtained by the TOF camera receiving reflected light (no emitted light or environmental emitted light) and imaging when the TOF camera Tx is not powered and emits no light. It is usually a "black picture" without a clear face image.

[0243] The Metadata is contained in the RAW Data 3. For example, the Metadata stores information such as the current working mode of the TOF camera (for example, TX-off mode), the size of the lighting current value (third current value) (for example, 0 mA), the working state of the TOF camera device (for example, abnormal), and the image exposure time (for example, 10 μs).

[0244] S154, the TOF camera transmits the RAW Data 3 to the IFE module.

[0245] For example, the TOF camera can transmit the RAW Data 3 collected by the TOF camera to the IFE module through MIPI.

[0246] S155, the IFE module sends the RAW Data 3 to the secure memory for storage.

[0247] The storage location of the RAW Data 3 collected by the TOF camera based on the TX-off mode in the secure memory can be represented by FD3.

[0248] FD3 in this step can be the same as or different from FD1 in S117. When FD3 in this step is the same as FD1 in S117, RAW Data 3 collected by the TOF camera based on the TX-off mode is stored in the same piece of secure memory as RAW Data 1 collected by the TOF camera based on the eye-safe mode in S117. RAW Data 1 collected by the TOF camera based on the eye-safe mode in S117 can be deleted, so that RAW Data 3 collected by the TOF camera based on the TX-off mode can be re-stored in the secure memory. When FD3 is different from FD1, RAW Data 3 collected by the TOF camera based on the TX-off mode and RAW Data 1 collected by the TOF camera based on the eye-safe mode in S117 can be stored in different pieces of secure memory.

[0249] S156, the IFE module sends FD3 to the camera driving module.

[0250] S157, the camera driving module sends FD3 to the IFE node.

[0251] S158, the IFE node sends FD3 to the camera service through the interface of the camera HAL.

[0252] S159, the camera service sends FD3 to the face recognition control module.

[0253] S160, the face recognition control module sends FD3 to the face recognition TA.

[0254] S161, the face recognition TA reads RAW Data 3 from the secure memory according to FD3.

[0255] S162, the face recognition TA obtains a face recognition result according to RAW Data 3.

[0256] Specifically, the face recognition TA can acquire that the current mode is the TX-off mode from the Metadata data in RAW Data 3 collected by the TOF camera based on the TX-off mode, then obtain a second grayscale image and a second depth image based on the third image data through the TOF algorithm, perform face recognition based on the second grayscale image through the face ID algorithm, and perform anti-counterfeiting detection based on the second depth image, to obtain the face recognition result.

[0257] It should be noted that when the TOF camera works in the TX-off mode, the face recognition result is failure. This is because the TOF camera cannot emit light in the TX-off mode, so the TOF camera cannot collect a clear face image. Even if the user currently unlocking is an authenticated user (i.e., the machine owner), the face recognition result is still failure.

[0258] S163、The face recognition TA passes the face recognition result (as failure) to the face recognition control module.

[0259] That is, the face recognition TA can inform the face recognition control module that the face recognition result is failure.

[0260] S164、The face recognition control module passes the face recognition result (as failure) to the face recognition service.

[0261] The face recognition control module passes the face recognition result (as failure) to the face recognition service based on the callback registered by the face recognition service before. That is, the face recognition control module can inform the face recognition service that the face recognition result is failure.

[0262] S165、The face recognition service passes the face recognition result (as failure) to the face recognition SDK.

[0263] The face recognition service passes the face recognition result (as failure) to the face recognition SDK based on the callback registered by the face recognition SDK before. That is, the face recognition service can inform the face recognition SDK that the face recognition result is failure.

[0264] S166、The face recognition SDK passes the face recognition result (as failure) to the lock screen application.

[0265] The face recognition SDK passes the face recognition result (as failure) to the lock screen application based on the callback registered by the lock screen application before. That is, the face recognition SDK can inform the lock screen application that the face recognition result is failure.

[0266] S167、The lock screen application decides not to unlock according to the face recognition result (as failure).

[0267] Since the face recognition result is failure, the lock screen application does not unlock.

[0268] For example, if the face recognition fails, the face recognition result is failure. Figure 9As shown in (a) in the figure, an interface 708 may be displayed. The interface 708 may include an unlock icon 709 (in a closed state, which can vividly prompt the user that the face unlocking is not successful) and a prompt text "Unrecognition is not successful, double-click the screen to try again" 710. In response to the user's double-click operation, the mobile phone can perform face recognition again (i.e., collect the user's face information again for comparison and anti-counterfeiting judgment). Alternatively, in response to the user's upward sliding operation in the interface 708, the mobile phone can display an interface 711. After entering the interface 711, the mobile phone can perform face recognition again. The interface 711 may include a face recognition icon 712 and a prompt text "Face recognition in progress" 713. If the recognition is still not successful, the mobile phone can display an interface 714. The interface 714 may include a prompt text "Unrecognition is not successful, click here to try again" 715. The user can click the corresponding position to re-trigger face recognition, or can also enter a password through the soft keyboard 716 to unlock, so as to avoid the problem of poor user experience caused by unsuccessful recognition.

[0269] Some embodiments of the present application provide an electronic device, which may include: a touch screen, a memory, and one or more processors. The touch screen, the memory, and the processor are coupled. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can perform the various functions or steps performed by the electronic device in the above method embodiment. The structure of the electronic device can refer to Figure 1 The structure of the electronic device 100 is shown.

[0270] The present application also provides a chip system (eg, a system on a chip (SoC)). Figure 11 As shown, the chip system includes at least one processor 1101 and at least one interface circuit 1102. The processor 1101 and the interface circuit 1102 can be interconnected via lines. For example, the interface circuit 1102 can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit 1102 can be used to send signals to other devices (such as the processor 1101 or the touch screen of an electronic device). Exemplarily, the interface circuit 1102 can read instructions stored in the memory and send the instructions to the processor 1101. When the instructions are executed by the processor 1101, the electronic device can execute the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiments of the present application.

[0271] The embodiments of the present application further provide a TOF camera, which can be used to implement the human eye safety mode, the face ID mode and the TX closing mode in the above embodiments. An electronic device provided with the TOF camera can perform each function or step performed by the electronic device in the above method embodiments.

[0272] The embodiments of the present application further provide a computer readable storage medium, which comprises computer instructions. When the computer instructions are run on the above electronic device, the electronic device performs each function or step performed by the electronic device in the above method embodiments.

[0273] The embodiments of the present application further provide a computer program product, which, when run on an electronic device, causes the electronic device to perform each function or step performed by the electronic device in the above method embodiments.

[0274] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0275] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, another division mode can be adopted, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0276] The units described as separate components can or can not be physically separate, and the components shown as units can be one physical unit or multiple physical units, that is, can be located in one place or distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0277] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0278] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The software product is stored in a storage medium, including a plurality of instructions to make a device (which can be a single-chip microcomputer, a chip, etc.) or a processor execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0279] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any change or replacement within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A face recognition method applied to an electronic device, the electronic device comprising a time-of-flight (TOF) camera module, the TOF camera module comprising a transmitter for transmitting a light signal and an image sensor for receiving a reflected light and imaging, the electronic device further comprising a camera HAL, a camera driver module, an image processing module and a first memory, characterized in that, The method comprises: receiving a first operation of a user, the first operation being used to trigger face recognition; controlling the emitter to work at a first light intensity; the image sensor acquires first image data based on a light signal of the first light intensity, and obtains first RAW Data based on the first image data and a first parameter, the first parameter being used to indicate a working state of the emitter; the image sensor sends the first RAW Data to the image processing module; the image processing module stores the first RAW Data in the first memory, the first memory corresponding to a first FD; the image processing module sends the first FD to the camera HAL; in response to receiving the first FD, the camera HAL sends a request to read the first parameter to the camera driver module; the camera driver module reads the first parameter from a register of the emitter and sends the first parameter to the camera HAL; the camera HAL determines whether the emitter is in a normal working state according to the first parameter; in a case where the emitter is in the normal working state, controlling the emitter to work at a second light intensity, the second light intensity being greater than the first light intensity; controlling the image sensor to collect image data; performing the face recognition based on the image data.

2. The method of claim 1, wherein: the light signal emitted by the emitter when working at a first current value is the first light intensity, and the light signal emitted by the emitter when working at a second current value is the second light intensity, the second current value being greater than the first current value.

3. The method according to claim 1 or 2, characterized in that, the determination of whether the emitter is in the normal working state comprises: if the first parameter is used to indicate that the working state of the emitter is a normal working state, it is determined that the emitter is in the normal working state; and if the first parameter is used to indicate that the working state of the emitter is an abnormal working state, it is determined that the emitter is in the abnormal working state.

4. The method according to claim 1 or 2, characterized in that, The method further comprises: in a case where the emitter is in the abnormal working state, controlling the emitter to work at a third light intensity, the third light intensity being 0; controlling the image sensor to collect image data; performing the face recognition based on the image data.

5. The method according to claim 1 or 2, characterized in that, The performance of the face recognition based on the image data comprises: acquiring a grayscale image and a depth image based on the image data; performing face comparison based on the grayscale image and performing anti-fake detection based on the depth image to obtain a face recognition result.

6. The method of claim 1 or 2, wherein: the first operation comprises an operation for unlocking the electronic device, an operation for online payment, an operation for entering a face, or an operation for security registration or login of an application.

7. The method of claim 5, wherein, The method further comprises: determining whether to perform unlocking according to the face recognition result; if the face recognition result is successful, performing unlocking; if the face recognition result is unsuccessful, not performing unlocking or displaying that unlocking fails; or determining whether to perform payment according to the face recognition result; If the face recognition result is successful, payment is performed; If the face recognition result is unsuccessful, payment is not performed or a payment failure is displayed; or According to the face recognition result, it is determined whether to perform face registration or login; If the face recognition result is successful, face registration or login is performed; If the face recognition result is unsuccessful, face registration or login is not performed or a face registration or login failure is displayed; or According to the face recognition result, it is determined whether to perform registration or login; If the face recognition result is successful, registration or login is performed; If the face recognition result is unsuccessful, registration or login is not performed or a registration or login failure is displayed.

8. The method of claim 4, wherein, The method further comprises: In the case that the transmitter is in an abnormal working state, prompting the user of a failure to unlock, or prompting the user of a payment failure, or prompting the user of a face registration failure, or prompting the user of a registration or login failure.

9. The method of claim 1 or 2, wherein, The camera HAL comprises a sensor node, and the control of the transmitter to work at a first light intensity comprises: The sensor node determines that the working mode of the camera module is a first working mode; the first working mode is used to indicate that the transmitter works at a first current value; The sensor node sends configuration parameters of the first working mode to the camera drive module; The camera drive module writes the configuration parameters of the first working mode into a register of the TOF camera module; The camera drive module sends a message of completion of writing of configuration parameters to the sensor node; In response to receiving the message of completion of writing of configuration parameters, the sensor node sends a first start command to the camera drive module; The camera drive module sends a second start command to the TOF camera module; The transmitter works at the first current value, and the light signal emitted by the transmitter when working at the first current value is the first light intensity.

10. The method of claim 9, wherein, The electronic device further comprises a first application, a face recognition SDK, a face recognition service, a face recognition control module, and a camera service, and after receiving the first operation of the user, the method further comprises: The first application invokes the face recognition SDK for face recognition; the first application corresponds to the first operation, and the first application comprises a lock screen application, a shopping application, a chat application, or a financial application; The face recognition SDK sends a face recognition request to the face recognition service; the face recognition request carries an identifier of a face recognition type, a resolution size of an image, and a data stream format; The face recognition service sends the face recognition request to the face recognition control module; The face recognition control module matches a camera module according to the face recognition request; The face recognition control module sends a first request of opening the camera module to the camera service; the first request of opening the camera module carries a security identifier, an identifier ID of the camera module, a resolution of an image, and a data stream format; the security identifier is used to apply for a secure memory; The camera service sends a second request of opening the camera module to the camera HAL, and the second request carries the security identifier, an identifier ID of the camera module, a resolution of the image, and a data stream format.

11. The method of claim 10, wherein, The sensor node determines the working mode of the camera module as the first working mode, specifically including: The sensor node determines the working mode of the camera module as the first working mode according to the resolution of the image, the data stream format, and a preset rule.

12. The method according to claim 10 or 11, characterized in that, After the camera service sends the second request of opening the camera module to the camera HAL, the method further includes: The camera HAL creates a channel for transmitting a data stream and a control stream according to the ID of the camera module, the resolution of the image, and the data stream format; The camera HAL returns a result of creating the channel to the camera service, and the result of creating the channel is success; The camera service returns a message of completing opening the camera module to the face recognition control module; The face recognition control module sends a data request to the camera service, and the data request is used to acquire the data stream; The camera service calls the camera HAL to acquire the data stream.

13. The method of claim 10 or 11, wherein, After the transmitter works at the first current value, the method further includes: The image sensor acquires a light signal within an exposure time corresponding to the first working mode; The image sensor acquires first image data based on the received light signal.

14. The method of claim 13, wherein, The first RAW Data is obtained based on the first image data and the first parameter, including: The image sensor sends a request of acquiring the first parameter to the transmitter; The image sensor receives the first parameter from the transmitter; The image sensor obtains the first RAW Data based on the first image data and the first parameter.

15. The method of claim 14, wherein, The image processing module sends the first FD to the camera HAL, including: The image processing module sends the first FD to the camera driver module; The camera driver module sends the first FD to the camera HAL; The camera driver module sends the first parameter to the sensor node of the camera HAL, including: The camera driver module sends the first parameter to the sensor node of the camera HAL; The camera HAL determines whether the transmitter is in a normal working state according to the first parameter, including: The sensor node of the camera HAL determines whether the transmitter is in a normal working state according to the first parameter.

16. The method of claim 15, wherein, The camera HAL determines whether the transmitter is in a normal working state according to the first parameter, specifically including: If the first parameter is used to indicate that the working state of the transmitter is a normal working state, the camera HAL determines that the transmitter is in a normal working state; If the first parameter is used to indicate that the working state of the transmitter is an abnormal working state, the camera HAL determines that the transmitter is in an abnormal working state.

17. The method according to claim 15 or 16, characterized in that, The control of the transmitter working at a second light intensity includes: If the working state of the transmitter is a normal working state, the camera HAL determines that the working mode of the camera module is a second working mode, and the second working mode is used to indicate that the transmitter works at a second current value; The camera HAL sends configuration parameters of the second working mode to the camera driving module; The camera driving module writes the configuration parameters of the second working mode into a register of the TOF camera module; The camera driving module sends a message of configuration parameter writing completion to the sensor node; In response to receiving the message of configuration parameter writing completion, the sensor node sends a third start command to the camera driving module; The camera driving module sends a fourth start command to the TOF camera module; The transmitter works at the second current value, and the light signal emitted by the transmitter working at the second current value is the second light intensity.

18. The method of claim 17, wherein, After the transmitter works at the second current value, the method further includes: The image sensor acquires light signals within an exposure time corresponding to the second working mode; Based on the received light signals, the image sensor acquires second image data.

19. The method of claim 18, wherein, After the image sensor acquires the second image data, the method further includes: The image sensor sends a request for acquiring a second parameter to the transmitter; The image sensor receives the second parameter from the transmitter; The image sensor obtains second RAW data based on the second image data and the second parameter, and the second parameter is used to indicate the working state of the transmitter at the second current value.

20. The method of claim 19, wherein, The electronic device further includes a second memory and a face recognition TA, and the face recognition based on the image data includes: The image sensor sends the second RAW data to the image processing module; The image processing module stores the second RAW data in the second memory; the second memory corresponds to a second FD; The face recognition TA reads the second RAW data from the second memory according to the second FD; The face recognition TA acquires a first grayscale image and a first depth image according to image data in the second RAW data; The face recognition TA performs face comparison according to the first grayscale image and performs anti-fake detection according to the first depth image to obtain a face recognition result.

21. The method of claim 20, wherein, After the image processing module stores the second RAW data in the second memory, the method further includes: The image processing module sends the second FD to the camera driving module; The camera driving module sends the second FD to the camera HAL; The camera HAL sends the second FD to a camera service through a preset interface; The camera service sends the second FD to the face recognition control module; The face recognition control module sends the second FD to the face recognition TA.

22. The method of claim 20 or 21, wherein, The method further includes: The face recognition TA sends the face recognition result to the face recognition control module; The face recognition control module sends the face recognition result to the face recognition service; The face recognition service sends the face recognition result to the face recognition SDK; The face recognition SDK sends the face recognition result to the first application; In response to receiving the face recognition result, the first application performs unlocking, and the face recognition result is successful.

23. The method of claim 15 or 16, wherein, In the case that the emitter is in an abnormal working state, the emitter is controlled to work at a third light intensity, specifically including: In response to receiving the working state of the emitter as an abnormal working state, the camera HAL determines that the working mode of the camera module is a third working mode, and the third working mode is used to indicate that the emitter works at a third current value, and the third current value is 0; The camera HAL sends the configuration parameters of the third working mode to the camera driver module; The camera driver module writes the configuration parameters of the third working mode into the register of the TOF camera module; The camera driver module sends a configuration parameter write completion message to the sensor node; In response to receiving the configuration parameter write completion message, the sensor node sends a fifth start command to the camera driver module; The camera driver module sends a sixth start command to the TOF camera module; The emitter does not work.

24. The method of claim 23, wherein, After the emitter does not work, the method further includes: The image sensor receives a light signal within an exposure time corresponding to the third working mode; Based on the received light signal, the image sensor obtains third image data.

25. The method of claim 24, wherein, The method further includes: The face recognition TA obtains a second gray image and a second depth image based on the third image data; Based on the second gray image, face comparison is performed, and based on the second depth image, anti-fake detection is performed, obtaining a face recognition result as failure.

26. The method of claim 25, wherein, The method further includes: The face recognition TA sends the face recognition result to the face recognition control module; The face recognition control module sends the face recognition result to the face recognition service; The face recognition service sends the face recognition result to the face recognition SDK; The face recognition SDK sends the face recognition result to the first application; In response to receiving the face recognition result, the first application does not perform unlocking or displays unlocking failure, and the face recognition result is failure.

27. An electronic device, comprising: The electronic device includes a wireless communication module, a memory and one or more processors; the wireless communication module, the memory and the processor are coupled; Wherein, the memory is used to store computer program code, the computer program code includes computer instructions; when the computer instructions are executed by the processor, the electronic device executes the method as claimed in any one of claims 1-26.

28. A computer-readable storage medium, characterized in that, Including computer instructions; When the computer instructions run on an electronic device, the electronic device executes the method as claimed in any one of claims 1-26.

29. A chip system, characterized by The chip system comprises one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected through lines; The chip system is applied to an electronic device comprising a communication module and a memory; the interface circuits are used for receiving signals from the memory and sending the signals to the processors, the signals comprising computer instructions stored in the memory; when the processors execute the computer instructions, the electronic device executes the method as claimed in any one of claims 1-26.

Images