A face recognition method and device

Abstract

An embodiment of the present application provides a face recognition method and apparatus, which relate to the field of terminals and can improve the security of face recognition. The method is applied to an electronic device, and the electronic device includes a TOF camera module. The TOF camera module includes a transmitter for emitting an optical signal and an image sensor for receiving the reflected light and imaging. The method includes: receiving a first operation of a user, where the first operation is used to trigger face recognition; controlling the transmitter to work at a first light intensity; determining whether the transmitter is in a normal working state; when the transmitter is in a normal working state, controlling the transmitter to work at a second light intensity, where the second light intensity is greater than the first light intensity; controlling the image sensor to collect image data; and performing face recognition based on the image data. Among them, determining whether the transmitter is in a normal working state can be performed by the camera HAL in the electronic device.

Description

Technical Field

[0001] This application relates to the field of terminals, and in particular, to a face recognition method and apparatus. Background Art

[0002] Currently, face recognition is widely used in scenarios where electronic devices perform identity authentication. The current face recognition uses the technology of planar (2D) face feature detection, and the extracted face features are 2D features, which are vulnerable to false attacks (for example, impersonation by using the photo of the device owner), and the security is not high. Summary of the Invention

[0003] Embodiments of this application provide a face recognition method and apparatus, which can improve the security of face recognition.

[0004] In a first aspect, an embodiment of this application provides a face recognition method, which is applied to an electronic device. The electronic device includes a time-of-flight (TOF) camera module, and the TOF camera module includes a transmitter for emitting an optical signal and an image sensor for receiving the reflected light and imaging. The method includes: receiving a first operation of a user, where the first operation is used to trigger face recognition; controlling the transmitter to work at a first light intensity; determining whether the transmitter is in a normal working state; when the transmitter is in a normal working state, controlling the transmitter to work at a second light intensity, where the second light intensity is greater than the first light intensity; controlling the image sensor to collect image data; and performing face recognition based on the image data.

[0005] Based on the method provided by the embodiment of this application, it is determined whether the transmitter is in a normal working state when working at the first light intensity. If the transmitter is in a normal working state, the transmitter is then controlled to work at the second light intensity. The second light intensity is greater than the first light intensity. Since the first light intensity is small, the first light intensity will not cause harm to the human eye, which can ensure the safety of the human eye. If the transmitter can work normally and emit the optical signal with the first light intensity, it means that the transmitter is intact and not damaged. In this way, the electronic device can normally control the transmitter, and when the transmitter works at the larger second light intensity, it can also ensure the safety of the human eye and there will be no problem of emitting optical signals that harm the human eye due to damage. Moreover, when the transmitter works at the second light intensity, it can ensure that the image sensor collects more accurate image data, so that more accurate face recognition can be performed.

[0006] In addition, the image data collected by the TOF camera module is 3D face data. Compared with 2D face data, face recognition based on 3D face data is more secure and accurate.

[0007] In a possible implementation, the optical signal emitted by the transmitter when it operates at the first current value is the first light intensity, and the optical signal emitted by the transmitter when it operates at the second current value is the second light intensity, where the second current value is greater than the first current value. That is to say, the greater the current value at which the transmitter operates, the greater the light intensity of the optical signal emitted by the transmitter.

[0008] In a possible implementation, determining whether the transmitter is in a normal operating state includes: determining a first parameter of the transmitter, where the first parameter is used to indicate the operating state of the transmitter; if the first parameter is used to indicate that the operating state of the transmitter is the normal operating state, determining that the transmitter is in the normal operating state; if the first parameter is used to indicate that the operating state of the transmitter is an abnormal operating state, determining that the transmitter is in the abnormal operating state. The first parameter of the transmitter may be generated after the transmitter operates at the first current value and sends an optical signal with the first light intensity.

[0009] In a possible implementation, the method further includes: when the transmitter is in the abnormal operating state, controlling the transmitter to operate at a third light intensity, where the third light intensity is 0; controlling the image sensor to collect image data; performing face recognition based on the image data. Since the transmitter is in the abnormal operating state, the transmitter may not work at this time, which can avoid the problem of emitting optical signals that harm the human eye due to transmitter damage.

[0010] In a possible implementation, performing face recognition based on the image data includes: obtaining a grayscale image and a depth image based on the image data; performing face comparison based on the grayscale image and anti-counterfeiting detection based on the depth image to obtain the face recognition result. If the face comparison result meets the first preset condition and the anti-counterfeiting detection result meets the second preset condition, it can be considered that the face recognition result is successful, and thus operations such as unlocking can be performed.

[0011] In a possible implementation, the first operation includes an operation for unlocking the electronic device, an operation for online payment, an operation for entering a face, or an operation for securely registering or logging in to an application. The first operation may be, for example, pressing the power button, clicking, swiping, etc., and the present application does not make a limitation.

[0012] In a possible implementation, the method further includes: determining whether to perform unlocking according to the face recognition result; if the face recognition result is successful, performing unlocking; if the face recognition result is failed, not performing unlocking or displaying unlocking failure; or determining whether to perform payment according to the face recognition result; if the face recognition result is successful, performing payment; if the face recognition result is failed, not performing payment or displaying payment failure; or determining whether to perform face enrollment according to the face recognition result; if the face recognition result is successful, performing face enrollment; if the face recognition result is failed, not performing face enrollment or displaying face enrollment failure; or determining whether to perform registration or login according to the face recognition result; if the face recognition result is successful, performing registration or login; if the face recognition result is failed, not performing registration or login or displaying registration or login failure. That is, the present application can be applied to face recognition during payment or transfer (for example, when the user performs a payment or transfer operation in a payment application / financial management application / chat application / shopping application (such as ). The present application is not limited to scenarios such as face recognition during payment or transfer operations of the user in a payment application / financial management application / chat application / shopping application (such as ), or face security verification for secure registration or login of the user to an application program (such as when the user performs a registration or login operation in

[0013] In a possible implementation, the method further includes: when the transmitter is in an abnormal working state, prompting the user of unlocking failure; or prompting the user of payment failure; or prompting the user of face enrollment failure; or prompting the user of registration or login failure. Since the transmitter is in an abnormal working state and does not work at this time, it is impossible to successfully recognize the face, but the problem of emitting optical signals that can harm the human eye due to transmitter damage can be avoided.

[0014] In a possible implementation, the electronic device includes a camera hardware abstraction layer HAL and a camera driver module. The camera HAL includes a sensor node. Controlling the transmitter to work at a first light intensity includes: the sensor node determining that the working mode of the camera module is the first working mode; the first working mode being used to indicate that the transmitter works at a first current value; the sensor node sending the configuration parameters of the first working mode to the camera driver module; the camera driver module writing the configuration parameters of the first working mode into the register of the TOF camera module; the camera driver module sending a message indicating that the configuration parameter writing is completed to the sensor node; in response to receiving the message indicating that the configuration parameter writing is completed, the sensor node sending a first start command to the camera driver module; the camera driver module sending a second start command to the TOF camera module; the transmitter working at the first current value, and the optical signal emitted when the transmitter works at the first current value being the first light intensity. In this way, the camera HAL (including the sensor node) and the camera driver module in the electronic device can control the transmitter to work at the first light intensity.

[0015] In a possible implementation, the electronic device further includes a first application, a face recognition software development kit (SDK), a face recognition service, a face recognition control module, and a camera service. After receiving a first operation from the user, the method further includes: the first application invoking the face recognition SDK to perform face recognition; the first application corresponding to the first operation, and the first application includes a lock screen application, a shopping application, a chat application, or a financial management application; the face recognition SDK sending a face recognition request to the face recognition service; the face recognition request carrying an identifier of the face recognition type, the resolution size of the image, and the data stream format; the face recognition service sending the face recognition request to the face recognition control module; the face recognition control module matching the camera module according to the face recognition request; the face recognition control module sending a first request to open the camera module to the camera service; the first request to open the camera module carrying a security identifier, an identifier ID of the camera module, the resolution of the image, and the data stream format; the security identifier being used to apply for secure memory; the camera service sending a second request to open the camera module to the camera HAL, and the second request carrying the security identifier, the identifier ID of the camera module, the resolution of the image, and the data stream format. In this way, the sensor node in the camera HAL can obtain information such as the security identifier, the identifier ID of the camera module, the resolution of the image, and the data stream format, and subsequently can determine the working mode of the camera module based on this information.

[0016] In a possible implementation, the sensor node determining that the working mode of the camera module is the first working mode specifically includes: the sensor node determining that the working mode of the camera module is the first working mode according to the resolution of the image, the data stream format, and a preset rule. In this way, the sensor node can determine that the working mode of the camera module is the first working mode.

[0017] In a possible implementation, after the camera service sends a second request to open the camera module to the camera HAL, the method further includes: the camera HAL creating a path for transmitting data streams and control streams according to the ID of the camera module, the resolution of the image, and the data stream format; the camera HAL returning the result of creating the path to the camera service; the result of creating the path being successful; the camera service returning a message indicating that the camera module has been successfully opened to the face recognition control module; the face recognition control module sending a data request to the camera service, and the data request being used to obtain a data stream; the camera service invoking the camera HAL to obtain the data stream.

[0018] In a possible implementation, after the transmitter operates at a first current value, the method further includes: the image sensor obtaining an optical signal within the exposure time corresponding to the first working mode; based on the received optical signal, the image sensor obtaining first image data.

[0019] In a possible implementation, after the image sensor acquires the first image data, the method further includes: the image sensor sends a request for acquiring the first parameter to the transmitter; the image sensor receives the first parameter from the transmitter; the image sensor obtains the first original RAW Data based on the first image data and the first parameter; the first parameter is used to indicate the working state of the transmitter at the first current value.

[0020] In a possible implementation, the electronic device further includes an image processing module. Determining whether the transmitter is in a normal working state includes: after the image sensor obtains the first original RAW Data based on the first image data and the first parameter, the image sensor sends the first RAW Data to the image processing module; the first RAW Data includes the first Metadata and the first image data, and the first Metadata includes the first parameter; the image processing module deletes the first image data from the first RAW Data and sends the first Metadata to the camera driver module; the camera driver module sends the first Metadata to the camera HAL; the camera HAL obtains the first parameter from the first Metadata; the camera HAL determines whether the transmitter is in a normal working state according to the first Metadata. In this way, the camera HAL can determine the working state of the transmitter.

[0021] In a possible implementation, the camera HAL determines whether the transmitter is in a normal working state according to the first Metadata, specifically including: the camera HAL obtains the first parameter from the first Metadata; if the first parameter is used to indicate that the working state of the transmitter is a normal working state, the camera HAL determines that the transmitter is in a normal working state; if the first parameter is used to indicate that the working state of the transmitter is an abnormal working state, the camera HAL determines that the transmitter is in an abnormal working state. In this way, the camera HAL can determine the working state of the transmitter according to the first parameter.

[0022] In a possible implementation, controlling the transmitter to work at a second light intensity includes: in response to receiving that the working state of the transmitter is the normal working state, the camera HAL determines that the working mode of the camera module is the second working mode, and the second working mode is used to indicate that the transmitter works at a second current value; the camera HAL sends the configuration parameters of the second working mode to the camera driver module; the camera driver module writes the configuration parameters of the second working mode into the registers of the TOF camera module; the camera driver module sends a message indicating that the configuration parameters have been written successfully to the sensor node; in response to receiving the message indicating that the configuration parameters have been written successfully, the sensor node sends a third start command to the camera driver module; the camera driver module sends a fourth start command to the TOF camera module; the transmitter works at the second current value, and the optical signal emitted when the transmitter works at the second current value is the second light intensity. In this way, the camera HAL can determine that the working mode of the camera module is the second working mode according to the normal working state of the transmitter. Through the camera HAL (including the sensor node) and the camera driver module in the electronic device, the transmitter can be controlled to work at the second light intensity.

[0023] In a possible implementation, after the transmitter works at the second current value, controlling the image sensor to collect image data specifically includes: the image sensor obtains the optical signal within the exposure time corresponding to the second working mode; based on the received optical signal, the image sensor obtains the second image data.

[0024] In a possible implementation, after the image sensor obtains the second image data, the method further includes: the image sensor sends a request to obtain the second parameter to the transmitter; the image sensor receives the second parameter from the transmitter; the image sensor obtains the second RAW Data based on the first image data and the second parameter; the second parameter is used to indicate the working state of the transmitter at the second current value.

[0025] In a possible implementation, the electronic device further includes a first memory and a face recognition trusted application TA. Performing face recognition based on the image data includes: the image sensor sends the second RAW Data to the image processing module; the image processing module sends the second RAW Data to be stored in the first memory; the first memory corresponds to the first FD; the face recognition TA reads the second RAW Data from the first memory according to the first FD; the face recognition TA obtains the first grayscale image and the first depth image according to the image data in the second RAW Data; the face recognition TA performs face comparison according to the first grayscale image and anti-counterfeiting detection according to the first depth image to obtain the face recognition result. In this way, the face recognition TA can determine the face recognition result according to the second RAW Data.

[0026] In a possible implementation, after the image processing module sends the second RAW Data to be stored in the first memory, the method further includes: the image processing module sends the first FD to the camera driver module; the camera driver module sends the first FD to the camera HAL; the camera HAL sends the first FD to the camera service through a preset interface; the camera service sends the first FD to the face recognition control module; the face recognition control module sends the first FD to the face recognition TA. In this way, the face recognition TA can obtain the first FD, and thus can read the second RAW Data according to the first FD.

[0027] In a possible implementation, the method further includes: the face recognition TA sends the face recognition result to the face recognition control module; the face recognition control module sends the face recognition result to the face recognition service; the face recognition service sends the face recognition result to the face recognition SDK; the face recognition SDK sends the face recognition result to the first application; in response to receiving the face recognition result, the first application performs unlocking, and the face recognition result is successful. In this way, when the face recognition result is successful, unlocking can be successful.

[0028] In a possible implementation, in the case where the transmitter is in an abnormal working state, controlling the transmitter to work at a third light intensity specifically includes: in response to receiving that the working state of the transmitter is an abnormal working state, the camera HAL determines that the working mode of the camera module is the third working mode, and the third working mode is used to instruct the transmitter to work at a third current value, and the third current value is 0; the camera HAL sends the configuration parameters of the third working mode to the camera driver module; the camera driver module writes the configuration parameters of the third working mode into the register of the TOF camera module; the camera driver module sends a message indicating that the configuration parameter writing is completed to the sensor node; in response to receiving the message indicating that the configuration parameter writing is completed, the sensor node sends a fifth start command to the camera driver module; the camera driver module sends a sixth start command to the TOF camera module; the transmitter does not work. Since the transmitter is in an abnormal working state and does not work at this time, the problem of emitting light signals that can harm the human eye due to transmitter damage can be avoided.

[0029] In a possible implementation, after the transmitter does not work, the method further includes: the image sensor receives the optical signal during the exposure time corresponding to the third working mode; based on the received optical signal, the image sensor acquires the third image data.

[0030] In a possible implementation, the method further includes: the face recognition TA obtains a second grayscale image and a second depth map based on the third image data; performs face comparison based on the second grayscale image and anti-counterfeiting detection based on the second depth map, and obtains that the face recognition result is a failure. Since when the transmitter does not work (is not powered on and does not emit light), the image data (the third image data) obtained by the image sensor is usually a "black image" without a clear face image, the face recognition result is a failure.

[0031] In a possible implementation, the method further includes: the face recognition TA sends the face recognition result to the face recognition control module; the face recognition control module sends the face recognition result to the face recognition service; the face recognition service sends the face recognition result to the face recognition SDK; the face recognition SDK sends the face recognition result to the first application; in response to receiving the face recognition result, the first application does not perform unlocking or displays an unlocking failure, and the face recognition result is a failure. Since the transmitter is in an abnormal working state and does not work at this time, the face recognition result is a failure, resulting in an unlocking failure, but this can avoid the problem of emitting light signals that can harm the human eyes due to transmitter damage.

[0032] In a second aspect, the present application provides a chip system, which includes one or more interface circuits and one or more processors. The interface circuits and the processors are interconnected by lines. The above chip system can be applied to an electronic device including a communication module and a memory. The interface circuit is used to receive a signal from the memory of the electronic device and send the received signal to the processor, and the signal includes computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device can execute the method as described in the first aspect and any of its possible design manners.

[0033] In a third aspect, the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions run on an electronic device (such as a mobile phone), the electronic device is caused to execute the method as described in the first aspect and any of its possible design manners.

[0034] In a fourth aspect, the present application provides a computer program product, which when running on a computer, causes the computer to execute the method as described in the first aspect and any of its possible design manners.

[0035] Fifth aspect, an embodiment of the present application provides a face recognition device, including a processor, the processor is coupled to a memory, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the device implements the method described in the first aspect and any possible design manner thereof. The device may be an electronic device or a server device; or may be a component of an electronic device or a server device, such as a chip.

[0036] Sixth aspect, an embodiment of the present application provides a face recognition device, which can be divided into different logical units or modules according to functions, and each unit or module performs different functions to enable the device to execute the method described in the first aspect and any possible design manner thereof.

[0037] It can be understood that the beneficial effects that can be achieved by the chip system described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the devices described in the fifth aspect and the sixth aspect can refer to the beneficial effects in the first aspect and any possible design manner thereof, which will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0039] Figure 2 It is a schematic principle diagram of a TOF imaging technology provided by an embodiment of the present application;

[0040] Figure 3 It is a schematic diagram of a software module architecture provided by an embodiment of the present application;

[0041] Figure 4 It is a schematic diagram of the interaction between software modules provided by an embodiment of the present application;

[0042] Figure 5 It is another schematic diagram of the interaction between software modules provided by an embodiment of the present application;

[0043] Figure 6 It is a schematic diagram of signal interaction provided by an embodiment of the present application;

[0044] Figure 7 It is a schematic diagram of display provided by an embodiment of the present application;

[0045] Figure 8 It is another schematic diagram of display provided by an embodiment of the present application;

[0046] Figure 9 It is another schematic diagram of display provided by an embodiment of the present application;

[0047] Figure 10 Another signal interaction schematic diagram provided by an embodiment of the present application;

[0048] Figure 11 Another signal interaction schematic diagram provided by an embodiment of the present application;

[0049] Figure 12 Another signal interaction schematic diagram provided by an embodiment of the present application;

[0050] Figure 13 A schematic diagram of a chip structure provided by an embodiment of the present application. Detailed implementation manners

[0051] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application. Among them, in the description of the present application, unless otherwise specified, "at least one" means one or more, and "a plurality" means two or more than two. In addition, in order to clearly describe the technical solutions in the embodiments of the present application, in the embodiments of the present application, terms such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the terms such as "first" and "second" do not limit the quantity and execution order, and the terms such as "first" and "second" do not necessarily limit to be different.

[0052] For the sake of clear and concise description of the following embodiments, a brief introduction to relevant concepts or technologies is given first:

[0053] The rich execution environment (REE), which can also be referred to as the rich execution environment or the general execution environment or the untrusted execution environment, refers to the system running environment of the mobile terminal, in which operating systems such as Android, IOS, and Linux can run. The REE has good openness and scalability but low security.

[0054] The trusted execution environment (TEE), which can also be referred to as the secure side or the secure area, is an area that requires authorization to access. The TEE coexists with the REE in the running environment of the electronic device. Through the support of hardware, it realizes isolation from the REE, has security capabilities and can resist software attacks that the conventional REE side is vulnerable to. The TEE has its own running space and defines strict protection measures. Therefore, it has a higher security level than the REE and can protect the assets (such as data, software, etc.) in the TEE from software attacks and resist specific types of security threats.

[0055] The REE+TEE architecture refers to an architecture that provides services for applications through the combination of TEE and REE. That is to say, TEE and REE coexist in an electronic device. Exemplarily, with the support of hardware, TEE can implement an operating mechanism isolated from REE. TEE has its own operating space, with a higher security level than REE, and can protect the assets (such as data, software, etc.) in TEE from software attacks. Only authorized security software can execute in TEE, and at the same time, it also protects the confidentiality of the resources and data of the security software. Compared with REE, due to its protection mechanisms such as isolation and permission control, TEE can better protect the security of data and resources.

[0056] TA, that is, a trusted application, is an application running in TEE and can provide security services for CAs running outside TEE, such as entering passwords, generating transaction signatures, face recognition, etc.

[0057] CA, that is, a client application. CA generally refers to an application running in REE. CA can call TA through the client application programming interface (API) and instruct TA to perform corresponding security operations.

[0058] SDK: Generally speaking, it refers to a collection of relevant documents, examples, and tools for assisting in the development of a certain type of software.

[0059] RAW Data, that is, raw data, can be understood as "data that has not been processed and compressed". In the embodiments of this application, RAW Data can refer to the raw data obtained by the TOF camera converting the captured light source signal into a digital signal. Some metadata generated by the camera shooting is also recorded in RAW Data.

[0060] Metadata, also known as intermediary data or relay data, is data used to describe data (data about data), mainly information describing the properties of data. In the embodiments of this application, Metadata can indicate information such as the working mode of the camera, the value of the lighting current, the working state of the TOF camera device, and the exposure value.

[0061] The TOF camera (TOF camera module) can include a transmitter (TX) and a receiver (RX). The TX is used to emit infrared light or laser pulses, and the RX is used to receive the reflected light and form an image. Since the TX can independently emit light signals for imaging, the TOF image is not affected by most of the light in the environment. Thus, applying the TOF image in the unlocking service can improve the security of face recognition.

[0062] Time of flight (TOF) imaging technology refers to the emission of a group of infrared lights (or laser pulses) invisible to the human eye. After hitting an object, they are reflected and then received by a camera. The time difference or phase difference from emission to reflection back to the camera is calculated, and the data is collected to form a set of distance-depth data, thereby obtaining a three-dimensional 3D model. That is to say, TOF imaging technology adds depth information from the Z-axis direction to the basis of traditional 2D XY-axis imaging, and finally generates 3D image information.

[0063] When using TOF imaging technology, it is necessary to project infrared light, lasers, etc. onto the human face. For the safety of the human eye, it is necessary to detect the optical power of the light emitted by TX to ensure that it is within the safe range of the human eye and avoid the light signal emitted by TX from harming the human eye.

[0064] An embodiment of this application provides a face recognition method. The TOF camera is used to collect images. The TOF camera can first work in the human eye safety mode to determine whether TX is abnormal. If TX is working properly, the TOF camera can be used to collect images normally, which can ensure that the optical power of the light emitted by TX is within the safe range of the human eye. If TX is abnormal, TX can be turned off to avoid the light signal emitted by TX from harming the human eye.

[0065] Figure 1 It is a schematic structural diagram of an electronic device 100 provided by an embodiment of this application.

[0066] As Figure 1 shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0067] Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0068] It can be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In some other embodiments, the electronic device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0069] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0070] The controller may be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0071] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0072] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0073] It can be understood that the interface connection relationships between the modules illustrated in this embodiment are only illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0074] The charging management module 140 is configured to receive a charging input from a charger. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0075] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives inputs from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160, etc. In some other embodiments, the power management module 141 may also be disposed in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may also be disposed in the same device.

[0076] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0077] Antenna 1 and Antenna 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, Antenna 1 can be multiplexed as a diversity antenna for a wireless local area network.

[0078] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by Antenna 1, and perform processing such as filtering and amplification on the received electromagnetic waves, and then transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves through Antenna 1 and radiate it out.

[0079] The modulation and demodulation processor can include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to Speaker 170A, Receiver 170B, etc.), or displays an image or video through the display screen 194.

[0080] The wireless communication module 160 can provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves through Antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be transmitted from the processor 110, perform frequency modulation and amplification on it, and convert it into electromagnetic waves through Antenna 2 and radiate it out.

[0081] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, such that electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0082] Electronic device 100 implements a display function through a GPU, display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, and is connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0083] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), a light-emitting diode (LED), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc.

[0084] The electronic device 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc. The ISP is used to process the data fed back by the camera 193. The camera 193 is used to capture static images or videos. The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0085] The camera 193 can include 1 to N. For example, the electronic device can include 2 front cameras and 4 rear cameras. Among them, the front cameras can include a TOF camera. The TOF camera includes a TX and an RX. The TX can be used to emit optical signals (infrared light or laser pulses), and the RX can be used to receive imaging. The TX can be, for example, an infrared light emitter. The RX can be, for example, a complementary metal oxide semiconductor (CMOS) or a charge-coupled device (CCD) image sensor.

[0086] Exemplarily, such as Figure 2As shown in (a) therein, an optical signal (infrared light or laser pulse) can be continuously sent to a target to be measured (e.g., a user) by an optical transmitter (Tx) of a TOF camera, and the optical signal returned by the target to be measured is received at the sensor end (Rx) of the TOF camera. As Figure 2 shown in (b) therein, depth information of the target to be measured can be obtained according to the phase difference (delay) between the transmitted and received optical signals.

[0087] Among them, Tx and Rx can perform information interaction through a bus. For example, Rx can send configuration parameters to Tx through a bus (e.g., a Serial Peripheral Interface (SPI) bus), and the configuration parameters are used to indicate the address of the register corresponding to Tx and the value for the register. For example, the address of the register corresponding to Tx can be 0x11, and a current value can be stored in the storage space corresponding to 0x11. Tx can work at the corresponding current value based on the corresponding configuration parameter so as to emit an optical signal with a corresponding light intensity. Rx can obtain corresponding image data based on the reflected light of the optical signal with the corresponding intensity emitted by the transmitter. It should be noted that when Tx works at different current values, optical signals with different light intensities can be emitted. For example, when Tx works at a first current value, an optical signal with a first light intensity can be emitted. When Tx works at a second current value, an optical signal with a second light intensity can be emitted. The second current value is greater than the first current value. The second light intensity is greater than the first light intensity. The image data obtained by Rx based on the reflected light of optical signals with different intensities is also different. For example, when Tx works at the first current value and emits an optical signal with the first light intensity, Rx obtains first image data within a corresponding exposure time; when Tx works at the second current value and emits an optical signal with the second light intensity, Rx obtains second image data within a corresponding exposure time; the second image data is different from the first image data.

[0088] When Tx works at the corresponding current value, its working state can be judged, and the working state can be normal or abnormal. Rx can request the working state of Tx through the bus, and Tx can feedback its own working state (e.g., normal or abnormal) to Rx through the bus, so that Rx can obtain the working state of Tx. Rx can package the working state of Tx, its own working state, and their working modes in a first data packet (e.g., Metadata). Rx can also package Metadata and the image data obtained based on the reflected light in a second data packet (e.g., RAW Data).

[0089] The NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission pattern between human brain neurons, it can quickly process input information and can also continuously learn on its own. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, voice recognition, text understanding, etc.

[0090] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card. The internal memory 121 can be used to store computer-executable program code, and the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. For example, in the embodiment of the present application, the processor 110 can execute the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory and can also include non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0091] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor, etc. Such as music playback, recording, etc.

[0092] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert analog audio input into a digital audio signal. The audio module 170 can also be used for encoding and decoding audio signals. The speaker 170A, also known as the "speaker", is used to convert an audio electrical signal into a sound signal. The receiver 170B, also known as the "earpiece", is used to convert an audio electrical signal into a sound signal. The microphone 170C, also known as the "microphone", "transmitter", is used to convert a sound signal into an electrical signal. The headphone jack 170D is used to connect a wired headphone.

[0093] The button 190 includes a power-on button, volume buttons, etc. The button 190 can be a mechanical button or a touch button. The electronic device 100 can receive button inputs and generate key signal inputs related to the user settings and function controls of the electronic device 100. The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and also for touch vibration feedback. The indicator 192 can be an indicator light, which can be used to indicate the charging state, power change, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect the SIM card. The SIM card can be in contact with and separated from the electronic device 100 by being inserted into or removed from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc.

[0094] The methods in the following embodiments can all be implemented in the electronic device 100 with the above hardware structure.

[0095] The software system of the above-mentioned electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservices architecture, or cloud architecture. In the embodiments of the present invention, the Android system with a layered architecture is taken as an example to exemplarily illustrate the software structure of the electronic device 100.

[0096] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through interfaces. In some embodiments, the Android system can include an application layer, an application framework layer, Android runtime, system libraries, a hardware abstraction layer (HAL), and a kernel layer. It should be noted that the embodiments of the present application take the Android system as an example for illustration. In other operating systems (such as HarmonyOS, IOS system, etc.), as long as the functions implemented by each functional module are similar to those of the embodiments of the present application, the solutions of the present application can also be implemented.

[0097] Among them, the application layer can include a series of application packages.

[0098] As Figure 3 shown, the application packages can include applications such as a camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, lock screen application, settings application, etc. Of course, the application layer can also include other application packages, such as payment applications, shopping applications, bank applications, etc., which are not limited in this application.

[0099] Among them, the application is set to have the function of face entry, and the entered face is used for face unlocking. The lock screen application has the function of unlocking in response to the user's unlocking operation (for example, pressing the power button). The lock screen application can perform unlocking processes such as face unlocking, fingerprint unlocking, password unlocking, etc. In this embodiment of the application, face unlocking is mainly used as an example for illustration.

[0100] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions. For example, it may include an activity manager, a window manager, a content provider, a view system, a resource manager, a notification manager, a camera service, and a face recognition service, etc. This embodiment of the application places no restrictions on this.

[0101] The system library can include multiple functional modules. For example: surface manager, Media Libraries, OpenGL ES, SGL, etc.

[0102] The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0103] The media library supports the playback and recording of a variety of common audio and video formats, as well as static image files, etc. The media library can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0104] OpenGL ES is used to implement three-dimensional graphics drawing, image rendering, synthesis, and layer processing, etc.

[0105] SGL is a drawing engine for 2D drawing.

[0106] Android Runtime includes a core library and a virtual machine. Android runtime is responsible for the scheduling and management of the Android system. The core library consists of two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core library of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0107] The HAL layer is an encapsulation of the Linux kernel driver, providing an interface upward and shielding the implementation details of the low-level hardware.

[0108] The HAL layer may include Wi-Fi HAL, audio HAL, Camera HAL, and Face CA (Face Recognition Control Module), etc.

[0109] Among them, the Camera HAL is the core software framework of the camera. The Camera HAL may include a sensor node and an image front end (IFE). The sensor node and the IFE node are components (nodes) in the transmission path (also called the transmission pipeline) of the image data and control instructions created by the Camera HAL.

[0110] The face recognition control module is the core software framework / application for face recognition.

[0111] Face Trusted Application (Face TA): An application for face recognition running in the TEE environment. In the embodiments of the present application, Face TA is referred to as the face recognition TA.

[0112] The kernel layer is the layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0113] Among them, the camera driver is the driver layer of the Camera device and is mainly responsible for interacting with the hardware.

[0114] The hardware layer includes a display, a TOF camera, an IFE module, and a Secure Buffer, etc.

[0115] Among them, the secure memory refers to the memory with security protection functions and can be used to store the raw data collected by the TOF camera.

[0116] The TOF camera, also called the TOF sensor, may include a transmitter (TX) and a receiver (RX). The TX is used to emit infrared light or laser pulses, and the RX is used to receive the reflected light and form an image.

[0117] IFE module (IFE-Lite): It can be called the image front processing module and can be used to forward image data without processing the image data during the forwarding process.

[0118] Next, the software modules involved in the face recognition method provided by the embodiments of the present application and the interactions between the modules will be described. As Figure 4As shown in the figure, the lock screen application in the application layer can interact with the face recognition SDK. The face recognition SDK can interact with the face recognition service in the framework layer by calling a preset application programming interface (API). The face recognition service can interact with the face recognition control module in the HAL layer. The face recognition control module can interact with the camera HAL in the HAL layer through the camera service in the framework layer, or the face recognition control module can directly interact with the camera HAL in the HAL layer. The camera HAL can include a sensor node and an IFE node. The sensor node can interact with the camera driver module in the kernel layer. The camera driver module can be used to drive the TOF camera in the hardware layer to collect image data in a default working mode (for example, the eye-safe mode, specifically, see the description in S112 below). The image data collected by the TOF camera can include Metadata (the first Metadata). The IFE module can delete other data in the image data collected by the TOF camera, retain the Metadata, and can send the Metadata to the camera driver module. The camera driver module can pass the Metadata to the IFE node of the camera HAL. The IFE node can pass the Metadata to the sensor node. The sensor node can parse the Metadata, calculate the eye safety detection result, and switch the working mode of the TOF camera according to the eye safety monitoring result. The camera HAL can continue to interact with the camera driver module, so that the camera driver module drives the TOF camera to collect image data in the switched working mode (for example, the face ID mode, specifically, see the description in S112 below). The IFE module can store the image data collected by the TOF camera based on the face ID mode in the secure memory. The storage location of the image data collected by the TOF camera in the secure memory can be represented by a file descriptor (FD). The IFE module can pass the FD (the first FD) corresponding to the image data to the face recognition TA through the IFE module, the camera driver module, the IFE node, the camera service, and the face control module. The face recognition TA can read the image data from the secure memory (the first memory) according to the FD and process it, and feedback the processing result (face recognition success or face recognition failure) to the face recognition control module. The face recognition control module can feedback the processing result to the lock screen application through the face recognition service and the face recognition SDK, so that the lock screen application can determine whether to unlock (if face recognition is successful, unlock; if face recognition fails, do not unlock, that is, unlock failure). Among them, Figure 4 The solid arrows in the figure can be used to represent the control flow, and the dashed arrows can be used to represent the data flow.

[0119] Specifically, asFigure 5 As shown, the sensor node in the camera HAL can be used to select the working modes of the TOF camera, including the eye-safe mode (the first working mode), the face ID mode (the second working mode), the TX off mode (the third working mode), etc. The details of various working modes can be referred to the description in S112 below. The default initial working mode of the TOF camera can be the eye-safe mode. When the TOF camera works in the eye-safe mode, the eye-safe current value (the first current value) calibrated by the production line can be read from the memory, the configuration of the eye-safe mode can be updated according to this current value, and the configuration parameters of the eye-safe mode can be sent to the camera driver module. So that the TOF camera can collect image data based on the eye-safe mode, the IFE module can send the Metadata in the image data to the camera driver module, the camera driver module can send the Metadata to the IFE node, the IFE node can send the Metadata to the sensor node, the sensor node can parse the Metadata, calculate the eye-safe detection result, and then select the working mode according to the eye-safe monitoring result. Specifically, if the eye-safe detection result is successful, the TOF camera can switch to the face ID mode. If the eye-safe detection result fails, the TOF camera can switch to the TX off mode. In this way, eye safety can be ensured.

[0120] For ease of understanding, the method provided in the embodiments of the present application will be specifically introduced below with reference to the accompanying drawings.

[0121] As Figure 6 shown, the embodiments of the present application provide a method for eye-safe detection and face recognition based on TOF images, and the process is as follows:

[0122] S101. The lock screen application invokes the face recognition SDK for face recognition.

[0123] When the unlock operation (the first operation) of the user is detected, the lock screen application invokes the face recognition SDK for face recognition. Among them, the unlock operation of the user includes operations such as the user picking up the mobile phone, pressing the power button, operating on the screen (clicking, swiping, etc.), or unplugging the charging cable.

[0124] At the same time, the lock screen application can register a callback with the face recognition SDK. The function of registering this callback is that when the face recognition SDK obtains the face recognition result, it can return the face recognition result to the lock screen application.

[0125] S102. The face recognition SDK sends a face recognition request to the face recognition service.

[0126] Among them, the request for face recognition carries the identifier of the face recognition type, the resolution size of the image, and the data stream format. Among them, the face recognition types include 2D face recognition type (for example, it can correspond to the identifier 0) and 3D face recognition type (for example, it can correspond to the identifier 1).

[0127] Exemplarily, the face recognition type carried in the request for face recognition can be 1 (i.e., 3D face recognition type), the resolution size of the image can be 1280x2898 pixels (pixel), and the data stream format can be the raw image format (RAW) 16.

[0128] At the same time, the face recognition SDK can register a callback with the face recognition service. The role of registering this callback is that when the face recognition service obtains the face comparison result, it can return the face recognition result to the face recognition SDK.

[0129] S103. The face recognition service sends a request for face recognition to the face recognition control module.

[0130] The request for face recognition can refer to the description in S102 and will not be elaborated here.

[0131] That is to say, the face recognition SDK can notify the face recognition control module to perform face recognition through the face recognition service. The face recognition service can send the request for face recognition received from the face recognition SDK to the face recognition control module.

[0132] At the same time, the face recognition service can register a callback with the face recognition control module. The role of registering this callback is that when the face recognition control module obtains the face comparison result, it can return the face comparison result to the face recognition service.

[0133] S104. In response to receiving the request for face recognition, the face recognition control module matches the camera according to the request for face recognition.

[0134] Specifically, the face recognition control module can obtain the identifier of the face recognition type, the resolution size of the image, and the data stream format from the request for face recognition, and determine the matching camera by querying the camera capabilities from the camera service.

[0135] It should be understood that during the startup process of the electronic device, the camera service can send a camera capability query request to the camera HAL. The camera capability query request is used to request a query of the camera capabilities supported by the electronic device. After receiving the camera capability query request, the camera HAL can send the capabilities of the cameras supported by the electronic device to the camera service, and the camera service can store the received capabilities of the cameras supported by the electronic device. Among them, the camera capabilities supported by the electronic device include the camera identity (ID) of each camera, the maximum resolution size supported, the format of the data stream, and whether the camera supports collecting depth information, etc.

[0136] Exemplarily, assume that three cameras are installed on the mobile phone, and the capability information of these three cameras can be as shown in Table 1:

[0137] Table 1

[0138] Camera ID Installation Location Maximum Supported Resolution Data Stream Format Depth Information 1 Rear 4096x3072 pixel YUY No 2 Front 3264x2448 pixel YUY No 3 Front 1280x2898 pixel RAW16 Yes

[0139] Among them, the camera with camera ID 3 is a TOF camera and supports collecting depth information. The cameras with camera IDs 1 and 2 are ordinary cameras and do not support collecting depth information. Of course, more front or rear cameras can be installed on the mobile phone. For example, the mobile phone can install 2 front cameras and 4 rear cameras.

[0140] The face recognition control module can send a camera capability query request to the camera service. The camera service can send the capabilities of the cameras supported by the electronic device to the face recognition control module. The face recognition control module can determine the matching camera according to the capabilities of the cameras supported by the electronic device. For example, it can be determined that the matching camera is the camera with ID 3 (i.e., the TOF camera).

[0141] It should be noted that Table 1 is only an example, and the data stream format corresponding to each camera can include multiple types. For example, the camera with camera identity 1 can not only correspond to the YUY data stream format, but also correspond to the RAW16 data stream format. This application does not make any limitations.

[0142] S105. The face recognition control module sends a request to open the Camera to the camera service.

[0143] Exemplarily, the face recognition control module may send a request to open the Camera to the camera service through the vendor native development kit (VNDK) interface. Among them, the request to open the Camera carries information such as a security identifier, a camera ID, the size of the resolution, and the data stream format. Among them, the security identifier is used to indicate that the data is stored in a secure buffer. That is to say, the security identifier can be used to apply for a piece of secure memory, which will be used to store the data collected by the camera later. For example, the security identifier can be 1 or 0. 1 means that the data is stored in the secure buffer, and 0 means that the data is stored in the non-secure buffer.

[0144] Exemplarily, the security identifier carried in the request to open the Camera can be 1 (that is, the data is stored in the secure buffer), the size of the image resolution can be 1280x2898 pixel, the data stream format can be RAW16, and the camera ID can be 3.

[0145] At the same time, the face recognition control module can register a callback with the camera service. Registering this callback is used to notify the face recognition control module that the Camera has been opened after the camera service completes opening the camera.

[0146] S106. In response to receiving the request to open the Camera, the camera service sends a request to open the Camera to the camera HAL. The request to open the Camera carries information such as a security identifier, a camera ID, the size of the resolution, and the data stream format.

[0147] During the process of the camera service calling the camera HAL, the camera service can send information such as a security identifier, a camera ID, the resolution of the image, and the data stream format to the camera HAL. The camera HAL can cache the information such as the security identifier, the camera ID, the resolution of the image, and the data stream format for a preset time.

[0148] At the same time, the camera service can register a callback with the camera HAL. This callback is used for the camera HAL to notify the camera service of the result of creating the path.

[0149] S107. The camera HAL creates a corresponding path according to the camera ID, the resolution of the image, and the data stream format.

[0150] The camera HAL can select available nodes based on the camera ID, resolution, and data stream format, and then create corresponding paths according to the available nodes. Exemplarily, if the resolution is 1280x2898 pixels, the data stream format is RAW16, and the camera ID is 3, then it can be determined to select the sensor node and the IFE node. This is because the sensor node and the IFE node can support the transmission of data with a resolution of 1280x2898 pixels and a data stream format of RAW 16 collected by the camera with a camera ID of 3.

[0151] Among them, the path corresponding to the sensor node can be: the path composed of the sensor node - camera driver - TOF camera - IFE module - secure memory. The path corresponding to the IFE node can be: the path composed of the IFE module (carrying FD) - camera driver - IFE node. The camera HAL can connect the output port of the sensor node and the input port of the IFE node at the HAL layer. Thus, the path corresponding to the sensor node and the path corresponding to the IFE node can form a closed-loop path. After the path creation is completed, the hardware in the path is powered on (i.e., the hardware circuit is energized) and waits for a data request.

[0152] S108. The camera HAL returns the result of creating the path to the camera service.

[0153] Among them, the result of creating the path can be success or failure. If the result of creating the path is failure, the camera HAL notifies the camera service that the path creation fails. If the result of creating the path is success, the camera HAL notifies the camera service that the path creation is successful, and S109 and its subsequent steps can be continued.

[0154] S109. In response to receiving the notification that the path creation is successful, the camera service returns a message indicating that the camera is opened successfully to the face recognition control module.

[0155] It can be understood that the successful opening of the camera means that the preparatory work before the camera takes pictures or videos (such as camera parameter configuration, power-on, etc.) has been completed.

[0156] S110. In response to receiving the message indicating that the camera is opened successfully, the face recognition control module sends a data request to the camera service.

[0157] Among them, the data request is used to request to obtain the data stream of the camera.

[0158] S111. In response to receiving the data request sent by the face recognition control module, the camera service calls the camera HAL to obtain the data stream.

[0159] S112. The camera HAL selects the camera working mode through the sensor node.

[0160] Specifically, the sensor node can select the camera working mode corresponding to the sensor node according to the camera resolution and data stream format cached in S106. Exemplarily, the sensor node can select the camera working mode corresponding to the sensor node by looking up a table (e.g., Table 2).

[0161] Table 2

[0162]

[0163] Among them, the EyeSafe Mode means that the TX of the TOF camera works in a small current (a current less than a preset threshold, the first current value). The EyeSafe Mode is used to check whether the TOF camera is damaged. The Face ID Mode means that the TX of the TOF camera works in a normal current (the second current value, within the preset threshold range). The Face ID Mode is used for scenarios such as secure face unlocking and secure payment. Among them, the second current value is greater than the first current value. The TX OFF Mode means that the TX of the TOF camera is not powered on (and thus does not emit light). The TX OFF Mode is used when it is detected that the TX device of the TOF camera is damaged or cannot work properly. This is because if the TX device is damaged and still powered on, it may have an adverse impact on the human eye. Therefore, when it is detected that the TX device of the TOF camera is damaged, the TX OFF Mode is adopted to make the TX device not powered on, thereby avoiding harm to the human eye.

[0164] Of course, the working modes of the camera can also include more, which are not limited in this application.

[0165] According to Table 2, when the maximum value of the image resolution is 1280x2898 pixel and the data stream format is Raw, the camera working modes can include the EyeSafe Mode, the Face ID Mode, the TX OFF Mode, etc. The sensor node can default the initial working mode of the camera to the EyeSafe Mode. When the camera working mode is the EyeSafe Mode, the sensor node can read the factory-calibrated eye-safe current value (i.e., the current value that does not harm the human eye) from the memory (e.g., oeminfo), and update the eye-safe mode setting of the TOF camera according to the eye-safe current value. For example, the address of the current register of the TOF camera can be obtained by looking up a table, and the eye-safe current value is written into the current register of the TOF camera. It should be understood that the sensor node can store the addresses of each register of the TOF camera, and the addresses of each register of the TOF camera can be as shown in Table 3.

[0166] Table 3

[0167] Register Identification Stored Data Type Address 1 Current 0x1 2 Resolution 0x2 3 Data Stream Format 0x3 4 TOF Camera Device Working Status 0x4 5 Working Mode of TOF Camera 0x5

[0168] Exemplarily, by querying Table 3, it can be determined that the address of the register corresponding to the current value is 0x1, so that the eye-safe current value can be written into the storage space corresponding to 0x1.

[0169] S113. The sensor node sends the configuration parameters of the eye-safe mode to the camera driver module in the Kernel layer.

[0170] Exemplarily, the configuration parameters of the eye-safe mode can be: the current value is 700 mA, the exposure time of the IR grayscale image is 10 μs, and the exposure time of the depth map is 10 μs.

[0171] S114a. The camera driver module writes (updates) the configuration parameters of the eye-safe mode into the registers of the TOF camera.

[0172] That is, the camera driver module can send the configuration parameters of the eye-safe mode to the TOF camera.

[0173] Exemplarily, the camera driver module can write the configuration parameters of the eye-safe mode into the registers of the RX of the TOF camera through the inter-integrated circuit (I2C). The address corresponding to the registers of the RX can be 0x01. There can be multiple registers corresponding to the RX, which are not limited in this application. That is, the configuration parameters of the eye-safe mode can be sent to the RX of the TOF camera through I2C. Among them, the configuration parameters of the eye-safe mode include the configuration parameters for the RX and the TX. For example, the configuration parameter for the TX can be the first current value. The configuration parameter for the RX can be the exposure time. The RX can write the configuration parameter corresponding to the TX into the register corresponding to the TX through the SPI bus. The address of the register corresponding to the TX can be 0x11. There can be multiple registers corresponding to the TX, which are not limited in this application.

[0174] S114b. The camera driver module sends a start (stream on) command / instruction (the second start command) to the TOF camera.

[0175] The stream on command is used to drive the TOF camera to perform data acquisition.

[0176] It should be noted that before S114b and after S114a, the camera driver module can also send a message indicating that the configuration parameter writing is completed to the sensor node; in response to receiving the message indicating that the configuration parameter writing is completed, the sensor node sends a start command (the first start command) to the camera driver module.

[0177] S115. In response to receiving the stream on command, the TOF camera acquires RAW Data1 based on the eye-safe mode.

[0178] Specifically, in response to receiving the stream on command, the RX can send a request for a light emission signal to the TX, and the TX works at a corresponding current value (the first current value) to send a light signal with the first light intensity; the RX receives the light signal during a corresponding exposure time (e.g., 10 μs), and the light signal received by the RX includes the reflected light of the light signal with the first light intensity. Based on the received light signal, the RX obtains the first image data.

[0179] That is to say, RAW Data 1 (raw data 1) refers to the image data (the first image data) obtained when the Rx of the TOF camera receives the reflected light and forms an image while the Tx of the TOF camera emits a light signal at the eye-safe current value calibrated on the production line to the human face. Among them, when the Tx works at the eye-safe current value calibrated on the production line, the light signal emitted is the first light intensity.

[0180] Among them, Metadata is included in the RAW Data. Exemplarily, Metadata stores information such as the working mode of the current TOF camera (e.g., the eye-safe mode), the value of the lighting current (e.g., the eye-safe current value calibrated on the production line), the working state of the TOF camera device (e.g., normal or abnormal), and the exposure value of the image (e.g., 10 μs).

[0181] S116. The TOF camera sends the RAW Data 1 acquired based on the eye-safe mode to the IFE module.

[0182] Exemplarily, the TOF camera can transmit the RAW Data 1 acquired by the TOF camera to the IFE module through the mobile industry processor interface (MIPI). The IFE module can also be referred to as the image pre-processing module (IFE-Lite), and the IFE module may not process the RAW Data 1.

[0183] S117. The IFE module deletes the data in the RAW Data 1 except for the Metadata.

[0184] In the embodiments of the present application, the Metadata in RAW Data 1 can be represented as Metadata 1. RAW Data 1 may include Metadata 1 and image data. The data other than Metadata is the image data. Since the image data involves user privacy, it is necessary to ensure the security of the image data. However, since the IFE module is in the REE environment and cannot ensure the security of the image data, the image data can be deleted in time to prevent the image data from being misappropriated.

[0185] S118. The IFE module sends Metadata 1 to the camera driver module.

[0186] S119. The camera driver module sends Metadata 1 to the IFE node of the camera HAL.

[0187] S120. The IFE node of the camera HAL sends Metadata 1 to the sensor node of the camera HAL.

[0188] S121. The sensor node of the camera HAL obtains the eye safety detection result according to Metadata 1.

[0189] The sensor node can determine whether the TOF camera is in the eye safety working mode according to the Metadata. If so (i.e., working in the eye safety working mode), it determines whether the device working state is normal. If the device working state is normal, the eye safety detection result is safe / normal (or the eye safety detection is successful). If the device working state is abnormal, the eye safety detection result is unsafe / abnormal (or the eye safety detection fails).

[0190] S122. The sensor node of the camera HAL determines the working mode of the TOF camera based on the eye safety detection result.

[0191] If the eye safety detection result is safe (normal), it is determined that the working mode of the TOF camera is the face ID mode; if the eye safety detection result is unsafe (abnormal), it is determined that the working mode of the TOF camera is the TX off mode.

[0192] It should be noted that the sensor node may store the configuration parameters corresponding to the face ID mode and the TX off mode.

[0193] Exemplarily, the configuration parameters corresponding to the face ID mode can be: the current value (the second current value) is 2800 mA, the exposure time of the IR grayscale image is 500 μs, depth is yes, and the exposure time of the image is 800 μs. The configuration parameters corresponding to the TXOFF mode can be: the current value (the third current value) is 0 mA, the exposure time of the IR grayscale image is 10 μs, depth is no, and the exposure time of the image is 10 μs.

[0194] The following takes the working mode of the TOF camera determined by the sensor node as the face ID mode as an example for description, including S123 - S139:

[0195] S123. The sensor node sends the configuration parameters of the face ID mode to the camera driver module.

[0196] S124. The camera driver module writes the configuration parameters of the face ID mode into the register of the TOF camera to drive the TOF camera to perform data acquisition based on the face ID mode.

[0197] That is, the camera driver module can send the configuration parameters of the face ID mode to the TOF camera.

[0198] Exemplarily, the camera driver module can write the configuration parameters of the face ID mode into the TOF camera register through I2C. That is, the camera driver module can send the configuration parameters of the face ID mode to the TOF camera through I2C.

[0199] S125. The TOF camera acquires RAW Data 2 based on the face ID mode.

[0200] Among them, RAW Data 2 can be the image data (the second image data) obtained when the Tx of the TOF camera works at the second current value (for example, 2800 mA) to emit light signals to the human face and the Rx of the TOF camera receives the reflected light and forms an image. The light signal emitted when the Tx of the TOF camera works at the second current value is the second light intensity. The second light intensity is greater than the first light intensity.

[0201] Among them, Metadata is included in RAW Data 2. Exemplarily, Metadata stores information such as the current working mode of the TOF camera (for example, the face ID mode), the magnitude of the lighting current value (for example, 2800 mA), the working state of the TOF camera device (for example, normal), and the exposure time of the image (for example, 800 μs).

[0202] S126. The TOF camera sends RAW Data 2 to the IFE module.

[0203] Exemplarily, the TOF camera can transmit RAW Data 2 collected in the face ID mode by the TOF camera to the IFE module through MIPI.

[0204] S127. The IFE module sends RAW Data 2 to be stored in the secure memory (Secure Buffer).

[0205] The storage location of RAW Data 2 collected by the TOF camera in the face ID mode in the secure memory (the first memory) can be represented by FD1 (the first FD). Exemplarily, when FD1 is 69, it can represent the storage location as XX secure memory; when FD1 is 96, it can represent the storage location as YY non-secure memory (ordinary memory).

[0206] S128. The IFE module sends FD1 to the camera driver module.

[0207] S129. The camera driver module sends FD1 to the IFE node.

[0208] S130. The IFE node sends FD1 to the camera service through the interface of the camera HAL.

[0209] S131. The camera service sends FD1 to the face recognition control module.

[0210] S132. The face recognition control module sends FD1 to the face recognition TA.

[0211] S133. The face recognition TA reads RAW Data 2 from the secure memory according to FD1.

[0212] S134. The face recognition TA obtains the face recognition result according to RAW Data 2.

[0213] Specifically, the face recognition TA can obtain the working mode of the TOF camera from the Metadata in RAW Data 2, for example, it can be the face ID mode. Then, the face recognition TA can process the second image data in RAW Data 2 through the TOF algorithm to obtain the first grayscale image and the first depth map, and then perform face recognition based on the first grayscale image through the face ID algorithm and anti-counterfeiting detection based on the first depth map, so as to obtain the face recognition result.

[0214] It should be noted that the face recognition TA also stores the face information previously entered by the user. The TOF algorithm can convert the face information entered by the user into a grayscale image and a depth image. If the grayscale image corresponding to the currently collected face information (RAW Data 2 collected by the TOF camera based on the face ID mode) matches the grayscale image of the previously entered face information (i.e., the RAW Data collected by the electronic device when the user performs the face entry operation), it can be considered the same user (i.e., the user performing the face entry operation and the unlocking operation is the same user). Moreover, if the currently collected face information includes depth information, it can be considered that the current user is real and trustworthy (not a disguise such as a photo or video). At this time, it can be considered that the face of the current user is secure, that is, the face recognition result is successful. If the grayscale image corresponding to the currently collected face information (RAW Data 2 collected by the TOF camera based on the face ID mode) does not match the grayscale image of the previously entered face information (i.e., the RAW Data collected by the electronic device when the user performs the face entry operation), or if the currently collected face information does not include depth information, it is considered that the face of the current user is not secure, that is, the face recognition result is failed.

[0215] S135. The face recognition TA sends the face recognition result to the face recognition control module.

[0216] S136. The face recognition control module sends the face recognition result to the face recognition service.

[0217] The face recognition control module can, based on the callback registered by the face recognition service previously (in S103), pass the face recognition result (success or failure) to the face recognition service.

[0218] S137. The face recognition service passes the face recognition result to the face recognition SDK.

[0219] The face recognition service, based on the callback registered by the face recognition SDK previously (in S102), passes the face recognition result (success or failure) to the face recognition SDK.

[0220] S138. The face recognition SDK passes the face recognition result to the lock screen application.

[0221] The face recognition SDK, based on the callback registered by the lock screen application previously (in S101), passes the face recognition result (success or failure) to the lock screen application.

[0222] S139. The lock screen application decides whether to unlock according to the face recognition result.

[0223] If the face recognition result is successful, the lock screen application can be successfully unlocked, so that the electronic device can display the desktop or the interface of an application (system application or third-party application). If the face recognition result is a failure, the lock screen application is not unlocked, that is, the face unlock fails. After the face unlock fails, the lock screen application can disable the face recognition function for a period of time (for example, 5 minutes) after the face recognition fails.

[0224] Exemplarily, if the user sets face unlock, as shown in (a) of Figure 7 , when the user picks up the mobile phone for face recognition, in response to the operation of the user picking up the mobile phone, as shown in (b) of Figure 7 , the mobile phone can display the lock screen interface 701. During the face recognition process, the mobile phone can display the unlock icon 702 and the prompt text "Recognizing face" 703 on the lock screen interface 701.

[0225] If the face recognition is successful, as shown in (a) of Figure 8 , the interface 704 can be displayed. The interface 704 can include an unlock icon 705 (in an open state, which can vividly prompt the user that the face unlock is successful) and the prompt text "Swipe up to enter" 706. In response to the user's swipe-up operation, the mobile phone can display the desktop or the interface of an application (system application or third-party application). Or, as shown in (b) of Figure 8 , if the face recognition is successful, without the user's additional operation, the mobile phone can be directly unlocked, that is, the desktop 707 (or directly display the interface of the application) can be immediately displayed.

[0226] If the face recognition fails, as shown in (a) of Figure 9 , the interface 708 can be displayed. The interface 708 can include an unlock icon 709 (in a closed state, which can vividly prompt the user that the face is not unlocked successfully) and the prompt text "Recognition failed, double-tap the screen to retry" 710. In response to the user's double-tap operation, the mobile phone can perform face recognition again (that is, collect the user's face information again for comparison and anti-counterfeiting judgment). Or, in response to the user's swipe-up operation in the interface 708, as shown in (b) of Figure 9 , the mobile phone can display the interface 711. After entering the interface 711, the mobile phone can perform face recognition again. The interface 711 can include a face recognition icon 712 and the prompt text "Performing face recognition" 713. If the recognition is still not successful, as shown in (c) of Figure 9 , the mobile phone can display the interface 714. The interface 714 can include the prompt text "Recognition failed, click here to retry" 715. The user can click on the corresponding position to re-trigger face recognition, or can also enter the password through the soft keyboard 716 to unlock, avoiding the problem of low user experience caused by continuous unsuccessful recognition.

[0227] It should be noted that the above embodiments illustrate the working mode selection of the TOF camera by taking the method flow of face unlocking in the lock screen application as an example. The working mode selection of the TOF camera can also be applied to face recognition during payment or transfer (for example, when the user performs payment or transfer operations in a payment application / financial management application / chat application / shopping application (such as ), and in scenarios such as face security verification when the user securely registers or logs in to an application program (for example, when the user performs registration or login operations). The present application is not limited. That is, the lock screen application can be replaced with a shopping application, a chat application, a payment application, a banking application, a financial management application, etc., and the present application is not limited.

[0228] The following takes the working mode of the TOF camera determined by the sensor node as the TX off mode as an example for illustration. As Figure 10 shown, it includes S140 - S157:

[0229] S140. The sensor node of the camera HAL determines that the working mode of the TOF camera is the TX off mode based on the human eye safety detection result.

[0230] S141. The sensor node sends the configuration parameters of the TX off mode to the camera driver module.

[0231] S142. The camera driver module writes (updates) the configuration parameters of the TX off mode into the register of the TOF camera to drive the TOF camera to collect data.

[0232] That is, the camera driver module can send the configuration parameters of the TX off mode to the TOF camera.

[0233] Exemplarily, the camera driver module can write the configuration parameters of the TX off mode into the TOF camera register through I2C. That is, send the configuration parameters of the TX off mode to the TOF camera through I2C.

[0234] S143. The TOF camera collects RAW Data 3 based on the TX off mode.

[0235] Among them, RAW Data 3 can be the image data (the third image data) obtained by the Rx of the TOF camera receiving the reflected light (no transmitted light or ambient transmitted light) and imaging when the Tx of the TOF camera is not powered on and does not emit light, usually a "black image" without a clear face image.

[0236] Among them, RAW Data 3 contains Metadata. Exemplarily, the Metadata stores information such as the working mode of the current TOF camera (e.g., Tx off mode), the value of the lighting current (the third current value) (e.g., 0 mA), the working state of the TOF camera device (e.g., abnormal), and the frame exposure time (e.g., 10 μs).

[0237] S144. The TOF camera transmits RAW Data 3 to the IFE module.

[0238] Exemplarily, the TOF camera can transmit RAW Data 3 collected by the TOF camera to the IFE module through MIPI.

[0239] S145. The IFE module sends RAW Data 3 to be stored in the secure memory.

[0240] The storage location of RAW Data 3 collected by the TOF camera based on the Tx off mode in the secure memory (the second memory) can be represented by FD2 (the second FD).

[0241] S146. The IFE module sends FD2 to the camera driver module.

[0242] S147. The camera driver module sends FD2 to the IFE node.

[0243] S148. The IFE node sends FD2 to the camera service through the interface of the camera HAL.

[0244] S149. The camera service sends FD2 to the face recognition control module.

[0245] S150. The face recognition control module sends FD2 to the face recognition TA.

[0246] S151. The face recognition TA reads RAW Data 3 from the secure memory according to FD2.

[0247] S152. The face recognition TA obtains the face recognition result according to RAW Data 3.

[0248] Specifically, the face recognition TA can obtain that the current is the Tx off mode from the Metadata data in RAW Data 3 collected by the TOF camera based on the Tx off mode, then obtain the second grayscale image and the second depth image through the TOF algorithm based on the third image data, perform face recognition based on the second grayscale image through the face ID algorithm, and perform anti-counterfeiting detection based on the second depth image to obtain the face recognition result.

[0249] It should be noted that when the TOF camera operates in the TX off mode, the face recognition result is a failure. This is because the TOF camera cannot emit light in the TX off mode, so the TOF camera cannot capture a clear face image. Even if the currently unlocked user is an authenticated user (i.e., the device owner), the face recognition result is still a failure.

[0250] S153. The face recognition TA transmits the face recognition result (which is a failure) to the face recognition control module.

[0251] That is, the face recognition TA can notify the face recognition control module that the face recognition result is a failure.

[0252] S154. The face recognition control module transmits the face recognition result (which is a failure) to the face recognition service.

[0253] Based on the previously registered callback of the face recognition service, the face recognition control module transmits the face recognition result (which is a failure) to the face recognition service. That is, the face recognition control module notifies the face recognition service that the face recognition result is a failure.

[0254] S155. The face recognition service transmits the face recognition result (which is a failure) to the face recognition SDK.

[0255] Based on the previously registered callback of the face recognition SDK, the face recognition service transmits the face recognition result (which is a failure) to the face recognition SDK. That is, the face recognition service can notify the face recognition SDK that the face recognition result is a failure.

[0256] S156. The face recognition SDK transmits the face recognition result (which is a failure) to the lock screen application.

[0257] Based on the previously registered callback of the lock screen application, the face recognition SDK transmits the face recognition result (which is a failure) to the lock screen application. That is, the face recognition SDK can notify the lock screen application that the face recognition result is a failure.

[0258] S157. The lock screen application decides not to unlock according to the face recognition result (which is a failure).

[0259] Since the face recognition result is a failure, the lock screen application does not unlock.

[0260] Exemplarily, if the face recognition fails, such as Figure 9As shown in (a) in [description], the interface 708 can be displayed. The interface 708 may include an unlock icon 709 (in a closed state, which can vividly prompt the user that the face unlocking is not successful) and a prompt text "Recognition failed. Double-tap the screen to retry" 710. In response to the user's double-tap operation, the mobile phone can perform face recognition again (that is, collect the user's face information again for comparison and anti-counterfeiting judgment). Or, in response to the user's upward swipe operation in the interface 708, the mobile phone can display the interface 711. After entering the interface 711, the mobile phone can perform face recognition again. The interface 711 may include a face recognition icon 712 and a prompt text "Performing face recognition" 713. If the recognition is still not successful, the mobile phone can display the interface 714. The interface 714 may include a prompt text "Recognition failed. Tap here to retry" 715. The user can click on the corresponding position to re-trigger face recognition, or can also enter the password through the soft keyboard 716 to unlock, avoiding the problem of low user experience caused by continuous unsuccessful recognition.

[0261] In still some other embodiments, as Figure 11 shown, the method for human eye safety detection based on TOF images (i.e., the process of determining the human eye safety detection result) may include S201 - S224:

[0262] S201. The lock screen application calls the face recognition SDK to perform face recognition.

[0263] S202. The face recognition SDK sends a request for face recognition to the face recognition service.

[0264] S203. The face recognition service sends a request for face recognition to the face recognition control module.

[0265] S204. In response to receiving the request for face recognition, the face recognition control module matches the camera according to the request for face recognition.

[0266] S205. The face recognition control module sends a request to the camera service to open the camera (Camera).

[0267] S206. In response to receiving the request to open the Camera, the camera service sends a request to open the Camera to the camera HAL. The request to open the Camera carries information such as a security identifier, a camera ID, the size of the resolution, and the data stream format.

[0268] S207. The camera HAL creates a corresponding path according to the camera ID, the resolution of the image, and the data stream format.

[0269] S201 - S207 can refer to S101 - S107 in the above embodiments and will not be elaborated here.

[0270] S208a. The sensor node of the camera HAL selects a working mode.

[0271] For the specific process, please refer to S112 and it will not be elaborated here.

[0272] S208b. The sensor node of the camera HAL sends the configuration parameters of the eye-safe mode to the camera driver module.

[0273] After the camera HAL successfully creates a path, it default selects the working mode of the TOF camera as the eye-safe mode and sends the configuration parameters of the eye-safe mode to the camera driver module.

[0274] S209. The camera driver module writes (updates) the configuration parameters of the eye-safe mode into the registers of the TOF camera.

[0275] S210. The camera driver module sends a stream on command / instruction to the TOF camera.

[0276] S211. The TOF camera acquires RAW Data 1 based on the eye-safe mode.

[0277] S212. The camera driver module sends a stop (stream off) command / instruction to the TOF camera.

[0278] After the camera driver module drives the TOF camera to acquire a frame of image data (acquire Raw Data once), it can instruct the TOF camera to stop acquiring data to save power consumption.

[0279] S213. The TOF camera sends the RAW Data 1 acquired based on the eye-safe mode to the IFE module.

[0280] S214. The IFE module deletes the data in RAW Data 1 except for Metadata.

[0281] In the embodiments of the present application, the Metadata in RAW Data 1 can be represented as Metadata 1.

[0282] S215. The IFE module sends Metadata 1 to the camera driver module.

[0283] S216. The camera driver module sends Metadata 1 to the IFE node of the camera HAL.

[0284] S217. The IFE node of the camera HAL sends Metadata 1 to the sensor node of the camera HAL.

[0285] S218. The sensor node of the camera HAL obtains the eye safety detection result based on Metadata 1.

[0286] The sensor node can determine whether the TOF camera is in the eye safety working mode according to the Metadata. If so (i.e., working in the eye safety working mode), it determines whether the device working state is normal. If the device working state is normal, the eye safety detection result is safe / normal (or the eye safety detection is successful). If the device working state is abnormal, the eye safety detection result is unsafe / abnormal (or the eye safety detection fails).

[0287] S219. The camera HAL returns the result of creating the path to the camera service.

[0288] Among them, the result of creating the path can be successful or failed. If the result of creating the path is failed, the camera HAL notifies the camera service that the path creation fails. If the result of creating the path is successful, the camera HAL notifies the camera service that the path creation is successful, and S109 and its subsequent steps can be continued.

[0289] S220. In response to receiving the notification of successful path creation, the camera service returns a message indicating that the camera is turned on to the face recognition control module.

[0290] It can be understood that the completion of turning on the camera means that the preparatory work before the camera takes pictures or videos (such as camera parameter configuration, power-on, etc.) has been completed.

[0291] S221. In response to receiving the message indicating that the camera is turned on, the face recognition control module sends a data request to the camera service.

[0292] Among them, the data request is used to request to obtain the data stream of the camera.

[0293] S222. In response to receiving the data request sent by the face recognition control module, the camera service calls the camera HAL to obtain the data stream.

[0294] S223. The sensor node of the camera HAL determines the working mode of the TOF camera based on the eye safety detection result.

[0295] If the eye safety detection result is safe (normal), it determines that the working mode of the TOF camera is the face ID mode; if the eye safety detection result is unsafe (abnormal), it determines that the working mode of the TOF camera is the TX off mode.

[0296] It should be noted that the configuration parameters corresponding to the face ID mode and the TX off mode can be stored in the sensor node.

[0297] Exemplarily, the configuration parameters corresponding to the face ID mode can be: the current value is 2800 mA, the exposure time of the IR grayscale image is 500 μs, depth is yes, and the exposure time of the image is 800 μs. The configuration parameters corresponding to the TXOFF mode can be: the current value is 0 mA, the exposure time of the IR grayscale image is 0 μs, depth is no, and the exposure time of the image is 0 μs.

[0298] When the working mode of the TOF camera determined by the sensor node is the face ID mode, as Figure 11 shown, after S223, the electronic device can further execute S224 - S240. S224 - S240 can refer to the relevant descriptions of S123 - S139 and will not be elaborated here. When the working mode of the TOF camera determined by the sensor node is the TX off mode, as Figure 12 shown, after S222, the electronic device can further execute S241 - S258. S241 - S258 can refer to the relevant descriptions of S140 - S157 and will not be elaborated here.

[0299] Some embodiments of the present application provide an electronic device, which may include: a touch screen, a memory, and one or more processors. The touch screen, the memory, and the processor are coupled. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can perform each function or step executed by the electronic device in the above - mentioned method embodiments. The structure of the electronic device can refer to Figure 1 the structure of the electronic device 100 shown.

[0300] Embodiments of the present application further provide a chip system (for example, a system on a chip (SoC)), as Figure 13 shown, the chip system includes at least one processor 1301 and at least one interface circuit 1302. The processor 1301 and the interface circuit 1302 can be interconnected by a line. For example, the interface circuit 1302 can be used to receive signals from other devices (such as the memory of an electronic device). Also for example, the interface circuit 1302 can be used to send signals to other devices (such as the processor 1301 or the touch screen of an electronic device). Exemplarily, the interface circuit 1302 can read the instructions stored in the memory and send the instructions to the processor 1301. When the instructions are executed by the processor 1301, the electronic device can perform each step in the above - mentioned embodiments. Of course, the chip system can also include other discrete devices, and the embodiments of the present application do not make specific limitations on this.

[0301] The embodiments of the present application further provide a TOF camera, which can be used to implement the eye-safe mode, face ID mode, TX off mode, etc. in the above embodiments. An electronic device equipped with this TOF camera can execute each function or step that the electronic device executes in the above method embodiments.

[0302] The embodiments of the present application further provide a computer-readable storage medium, which includes computer instructions. When the computer instructions run on the above electronic device, the electronic device is enabled to execute each function or step that the electronic device executes in the above method embodiments.

[0303] The embodiments of the present application further provide a computer program product. When the computer program product runs on an electronic device, the electronic device is enabled to execute each function or step that the electronic device executes in the above method embodiments.

[0304] Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0305] In the several embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the module or unit is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0306] The unit described as a separated component may or may not be physically separated. The component displayed as a unit may be a physical unit or multiple physical units, that is, it may be located in one place, or it may be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0307] In addition, each functional unit in the various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0308] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0309] The above content is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claimed rights.

Claims

1. A face recognition method, applied to an electronic device, the electronic device including a time-of-flight (TOF) camera module, the TOF camera module including a transmitter for emitting an optical signal and an image sensor for receiving reflected light and imaging, characterized in that The method includes: Receiving a first operation of a user, the first operation being used to trigger face recognition; Controlling the transmitter to work at a first light intensity; Determining whether the transmitter is in a normal working state; When the transmitter is in a normal working state, controlling the transmitter to work at a second light intensity, the second light intensity being greater than the first light intensity; Controlling the image sensor to collect image data; Performing the face recognition based on the image data; Wherein, the electronic device further includes an image processing module, a camera driver module, and a camera HAL. Determining whether the transmitter is in a normal working state includes: The image sensor obtains first image data; The image sensor receives a first parameter from the transmitter, the first parameter being used to indicate the working state of the transmitter; The image sensor obtains first RAW Data based on the first image data and the first parameter; The image sensor sends the first RAW Data to the image processing module; the first RAW Data includes first Metadata and the first image data, and the first Metadata includes the first parameter; The image processing module deletes the first image data from the first RAW Data and sends the first Metadata to the camera driver module; The camera driver module sends the first Metadata to the camera HAL; The camera HAL obtains the first parameter from the first Metadata; The camera HAL determines whether the transmitter is in a normal working state according to the first Metadata.

2. The method according to claim 1, wherein The camera HAL determines whether the transmitter is in a normal working state according to the first Metadata, specifically including: The camera HAL obtains the first parameter from the first Metadata; If the first parameter is used to indicate that the working state of the transmitter is a normal working state, the camera HAL determines that the transmitter is in a normal working state; If the first parameter is used to indicate that the working state of the transmitter is an abnormal working state, the camera HAL determines that the transmitter is in an abnormal working state.

3. The method according to claim 2, characterized in that, The camera HAL includes a sensor node. Controlling the transmitter to work at a second light intensity includes: If it is determined that the working state of the transmitter is a normal working state, the camera HAL determines that the working mode of the camera module is a second working mode, and the second working mode is used to indicate that the transmitter works at a second current value; The camera HAL sends the configuration parameters of the second working mode to the camera driver module; The camera driver module writes the configuration parameters of the second working mode into the register of the TOF camera module; The camera driver module sends a message indicating that the configuration parameter writing is completed to the sensor node; In response to receiving the message indicating that the configuration parameter writing is completed, the sensor node sends a third start command to the camera driver module; The camera driver module sends a fourth start command to the TOF camera module; The emitter operates at the second current value, and the optical signal emitted when the emitter operates at the second current value is the second light intensity.

4. The method according to claim 3, characterized in that After the emitter operates at the second current value, controlling the image sensor to collect image data specifically includes: The image sensor obtains an optical signal within the exposure time corresponding to the second operating mode; Based on the received optical signal, the image sensor obtains second image data.

5. The method according to claim 4, wherein After the image sensor obtains the second image data, the method further includes: The image sensor sends a request for obtaining second parameters to the emitter; The image sensor receives the second parameters from the emitter; The image sensor obtains second RAW Data based on the first image data and the second parameters; the second parameters are used to indicate the operating state of the emitter at the second current value.

6. The method according to claim 5, characterized in that, The electronic device further includes a first memory and a face recognition trusted application TA. Performing the face recognition based on the image data includes: The image sensor sends the second RAW Data to the image processing module; The image processing module sends the second RAW Data to be stored in the first memory; the first memory corresponds to the first FD; The face recognition TA reads the second RAW Data from the first memory according to the first FD; The face recognition TA obtains a first grayscale image and a first depth image according to the image data in the second RAW Data; The face recognition TA performs face comparison according to the first grayscale image and performs anti-counterfeiting detection according to the first depth image to obtain a face recognition result.

7. The method according to claim 6, wherein The electronic device further includes a camera service and a face recognition control module. After the image processing module sends the second RAW Data to be stored in the first memory, the method further includes: The image processing module sends the first FD to the camera driver module; The camera driver module sends the first FD to the camera HAL; The camera HAL sends the first FD to the camera service through a preset interface; The camera service sends the first FD to the face recognition control module; The face recognition control module sends the first FD to the face recognition TA.

8. The method according to claim 6 or 7, characterized in that, The electronic device further includes a face recognition service, a face recognition SDK, and a first application. The method further includes: The face recognition TA sends the face recognition result to the face recognition control module; The face recognition control module sends the face recognition result to the face recognition service; The face recognition service sends the face recognition result to the face recognition SDK; The face recognition SDK sends the face recognition result to the first application; In response to receiving the face recognition result, the first application performs unlocking, and the face recognition result is successful.

9. The method according to claim 2, wherein The camera HAL includes a sensor node. The method further includes: In response to receiving that the operating state of the transmitter is an abnormal operating state, the camera HAL determines that the operating mode of the camera module is the third operating mode, and the third operating mode is used to indicate that the transmitter operates at a third current value, and the third current value is 0; The camera HAL sends the configuration parameters of the third operating mode to the camera driver module; The camera driver module writes the configuration parameters of the third operating mode into the registers of the TOF camera module; The camera driver module sends a message indicating that the configuration parameter writing is completed to the sensor node; In response to receiving the message indicating that the configuration parameter writing is completed, the sensor node sends a fifth start command to the camera driver module; The camera driver module sends a sixth start command to the TOF camera module; The transmitter does not work.

10. The method according to claim 9, wherein After the transmitter does not work, the method further includes: The image sensor receives an optical signal during the exposure time corresponding to the third operating mode; Based on the received optical signal, the image sensor obtains third image data.

11. The method according to claim 10, characterized in that, The electronic device further includes a face recognition TA, and the method further includes: The face recognition TA obtains a second grayscale image and a second depth image based on the third image data; Based on the second grayscale image for face comparison and based on the second depth image for anti-counterfeiting detection, the face recognition result is obtained as a failure.

12. The method according to claim 11, wherein The electronic device further includes a face recognition service, a face recognition control module, a face recognition SDK, and a first application, and the method further includes: The face recognition TA sends the face recognition result to the face recognition control module; The face recognition control module sends the face recognition result to the face recognition service; The face recognition service sends the face recognition result to the face recognition SDK; The face recognition SDK sends the face recognition result to the first application; In response to receiving the face recognition result, the first application does not perform unlocking or displays a unlocking failure, and the face recognition result is a failure.

13. An electronic device, characterized in that, The electronic device includes: a wireless communication module, a memory, and one or more processors; the wireless communication module, the memory are coupled to the processor; Wherein, the memory is used to store computer program code, and the computer program code includes computer instructions; when the computer instructions are executed by the processor, the electronic device is caused to execute the method according to any one of claims 1-12.

14. A computer-readable storage medium, characterized in that, Includes computer instructions; When the computer instructions run on the electronic device, the electronic device is caused to execute the method according to any one of claims 1-12.

15. A chip system, characterized in that, The chip system includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected by lines; The chip system is applied to an electronic device including a communication module and a memory; the interface circuit is configured to receive a signal from the memory and send the signal to the processor, and the signal includes computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1-12.

Images