Face recognition method, electronic equipment, chip system and storage medium

Abstract

The embodiment of the invention provides a face recognition method, electronic equipment, a chip system and a storage medium, and relates to the technical field of terminals, the method is realized based on an REE + multi-TEE environment, face template data is stored in a security chip, and the security chip supports access of a first security virtual machine at a TEE side and does not support access of a main virtual machine at the REE side. In a face recognition / authentication scene, the electronic equipment creates a secure memory, a first secure virtual machine is accessible, a main virtual machine is not allowed to access, and under the condition that first face data acquired by camera hardware is written into the secure memory, the first secure virtual machine acquires the first face data from the secure memory and reads face template data from a secure chip; and comparing the first face data with the face template data to obtain a face recognition result. As the Android side cannot access the security memory and the security chip, the face data can be prevented from being acquired and damaged, and the security of the face data is improved.

Description

Technical Field

[0001] This application relates to the field of terminal technologies, and in particular, to a face recognition method, an electronic device, a chip system, and a storage medium. Background Art

[0003] Currently, face recognition is widely used in scenarios where electronic devices perform identity authentication, such as payment scenarios or unlocking scenarios. In face recognition / authentication scenarios, electronic devices collect face image data, and the face image data involves users' privacy and security information. Although electronic devices already provide face recognition functions, there is still a risk that users' face image data may be stolen or damaged in current face recognition technologies. Summary of the Invention

[0004] Embodiments of this application provide a face recognition method, an electronic device, a chip system, and a storage medium, which can improve the reliability and security of face recognition.

[0005] In a first aspect, embodiments of this application provide a face recognition method, which is applied to an electronic device. The electronic device includes a main virtual machine, a first secure virtual machine, and a second secure virtual machine; the method includes:

[0006] The main virtual machine receives a first operation of the user (the first operation is an operation that triggers the face recognition process); in response to the first operation, the main virtual machine creates a first secure memory and sets the access permission of the first secure memory; the main virtual machine calls the camera HAL to trigger the startup of the camera hardware; wherein, the camera hardware of the electronic device has the write permission to the first secure memory, and the first secure virtual machine and the second secure virtual machine have the read permission to the first secure memory;

[0007] When the first face data collected by the camera hardware is written into the first secure memory, the first secure virtual machine obtains the first face data from the first secure memory, and obtains face template data from the secure chip of the electronic device; the first secure virtual machine compares the first face data with the face template data to obtain a face recognition result;

[0008] Wherein, the first secure virtual machine has the access permission to the secure chip, and the main virtual machine does not have the access permission to the secure chip. Among them, the access permission to the secure chip includes writing data to the secure chip and reading data from the secure chip.

[0009] Through the face recognition method provided by the embodiments of the present application, the face template data is stored in the secure chip. The first secure virtual machine has access rights to the secure chip, while the main virtual machine on the Android side does not have access rights to the secure chip. In the face recognition / authentication scenario, the electronic device creates a secure memory and sets the access rights of the secure memory. The secure memory only allows the secure virtual machine to access, and does not allow the main virtual machine to access. When the first face data collected by the camera hardware is written into the first secure memory, the first secure virtual machine obtains the first face data from the first secure memory and reads the face template data from the secure chip, and compares the first face data with the face template data to obtain the face recognition result. Since the Android side cannot access the secure memory or the secure chip, the face data can be prevented from being obtained and damaged, and the reliability and security of the face data are improved.

[0010] Among them, the secure memory refers to the memory with security protection functions. The secure memory is the memory allocated for the Trusted Execution Environment (TEE) side of the secure operating environment. For example, the electronic device can allocate a part from the Double Data Rate Synchronous Dynamic Random Access Memory (DDR) as the secure memory for caching data. In the embodiments of the present application, the secure memory can be used to store the raw images collected by the camera. The electronic device can extract the face feature values based on the raw images, and further generate the face template data according to the face feature values. In the face recognition / authentication scenario, the electronic device can create a secure memory and set the access rights of the secure memory, allowing the TEE side to access the secure memory and not allowing the Android side to access the secure memory, so as to improve the security of the stored data to a certain extent.

[0011] Among them, the secure chip is an independent secure storage chip from the secure memory, and uses a specific encryption method for data storage. For example, the secure chip can be a SEC flash storage chip. The SEC flash storage chip defines a special interface and has specific read / write timings, which can prevent bus attacks and data theft. In the solution of the present application, the face data template is stored in the secure chip. The secure chip supports the access of the TEE side of the secure operating environment and does not support the access of the Android side, so as to improve the security of the stored data to a certain extent.

[0012] Among them, the main virtual machine runs in the general running environment REE (or the non-secure running environment REE). The first secure virtual machine runs in the first secure running environment TEE, or a self-developed TEE system. The second secure virtual machine runs in the second secure running environment TEE, or a native TEE system. The face recognition method provided by the present application is implemented based on a multi-TEE environment, and this face recognition method adopts the REE + multi-TEE architecture. Among them, the secure virtual machine can also be called a trusted virtual machine.

[0013] The improvements of this application compared with the prior art include:

[0014] (1) Based on the existing multi-TEE architecture, the face recognition TA is migrated from the native TEE system to the self-developed TEE system, so as to make full use of the self-developed TEE system to enhance various security features.

[0015] (2) The face template data is saved in the secure chip, which improves the reliability and security of the face data and prevents it from being obtained and damaged by hackers.

[0016] In the related art, when the user uses the secure storage function to save data, the data is encrypted inside the secure running environment TEE side and then saved to the relevant storage area on the REE side. For example, the file system SFS uses the file system on the non-secure running environment REE side for storage services. The data encrypted on the secure running environment TEE side will be stored in the storage medium on the REE side using the file system on the REE side. Since the face template data is stored in the Android-side file system SFS, the Android side can access the file system SFS. Although the face template data has been encrypted, this data storage method still has the risk of being damaged.

[0017] Compared with the related art solution of storing face template data through the file system SFS and having low security due to the Android side's accessibility, this application uses an independent secure storage hardware to store face template data, and automatically encrypts the face template data when writing it into the secure chip. This secure chip can only be accessed by the TEE side and cannot be accessed by the Android side. Therefore, the security of storing data is greatly improved.

[0018] The solution provided by this application is applied to face recognition / verification scenarios, such as payment verification scenarios or unlocking verification scenarios, which can improve the security of face verification.

[0019] Among them, the face recognition result is face recognition / verification successful or face recognition / verification failed.

[0020] For example, in the unlocking verification scenario, it is determined whether to perform unlocking according to the face recognition result; if the face recognition result is successful, unlocking is performed; if the face recognition result is failed, unlocking is not performed or unlocking failure is displayed.

[0021] For example, in the payment verification scenario, it is determined whether to perform payment according to the face recognition result; if the face recognition result is successful, payment is performed; if the face recognition result is failed, payment is not performed or payment failure is displayed.

[0022] For example, in the scenario of unlocking authentication / payment authentication, it is determined whether to perform face enrollment according to the face recognition result; if the face recognition result is successful, face enrollment is performed; if the face recognition result is failed, face enrollment is not performed or a face enrollment failure is displayed.

[0023] For example, in the scenario of unlocking authentication / payment authentication, it is determined whether to perform registration or login according to the face recognition result; if the face recognition result is successful, registration or login is performed; if the face recognition result is failed, registration or login is not performed or a registration or login failure is displayed.

[0024] That is, the solution of the present application can be applied to face recognition during payment or transfer (for example, when a user performs a payment or transfer operation in a payment application / financial management application / chat application / shopping application), and the present application is not limited to scenarios such as face security verification for secure registration or login of the user to the application program.

[0025] In a possible implementation manner, before the main virtual machine receives the first operation of the user, the method further includes: the main virtual machine receives a second operation of the user (the second operation is an operation to trigger the face enrollment process); the main virtual machine creates a second secure memory and sets the access permission of the second secure memory in response to the second operation; the camera hardware has the write permission for the second secure memory, and the first secure virtual machine and the second secure virtual machine have the read permission for the second secure memory; the main virtual machine calls the camera HAL to trigger the camera hardware to start; when the second face data collected by the camera hardware is written into the second secure memory, the first secure virtual machine obtains the second face data from the second secure memory and generates the face template data according to the feature value of the second face data; the first secure virtual machine encrypts the face template data and stores it in the secure chip.

[0026] In a possible implementation manner, after the first secure virtual machine encrypts the face template data and stores it in the secure chip, the method further includes: the main virtual machine calls the camera HAL to trigger the camera hardware to close and trigger the release of the second secure memory. After the face enrollment process ends, the camera hardware is closed and the secure memory is released to save resources.

[0027] In a possible implementation manner, after the first secure virtual machine obtains the face recognition result, the method further includes: the first secure virtual machine sends the face recognition result to the main virtual machine; the main virtual machine receives the face recognition result and performs a face recognition service according to the face recognition result, and the face recognition service includes face payment or face unlocking.

[0028] In a possible implementation, after the main virtual machine receives the face recognition result, the method further includes: the main virtual machine calls the camera HAL, triggers the camera hardware to turn off, and triggers the release of the first secure memory. After the face recognition process ends, turning off the camera hardware and releasing the secure memory saves resources.

[0029] In a possible implementation, the main virtual machine creates the first secure memory, including: the main virtual machine calls the camera HAL to create the first secure memory based on the operating memory of the electronic device. Among them, the operating memory can be provided by DDR.

[0030] In a possible implementation, the first secure virtual machine obtains the face template data from the secure chip of the electronic device, including: the first secure virtual machine calls the secure chip interface to obtain the encrypted face template data from the secure chip; the first secure virtual machine decrypts the face template data.

[0031] In a possible implementation, the first secure virtual machine includes a face recognition trusted application TA; the main virtual machine includes a first client application CA, a face recognition service module, a face recognition control module, and a secure runtime environment TEE driver.

[0032] In this case, the method further includes: in response to the user's first operation on the first client application CA, the first client application CA requests a face recognition service from the face recognition service module; the face recognition service module instructs the face recognition control module to start the face recognition process; the face recognition control module calls the secure runtime environment TEE driver to start the face recognition trusted application TA on the side of the first secure virtual machine.

[0033] Through the solution of this application, the trusted application TA can provide security services such as key generation and key management, security authentication, and face recognition for the client application CA running in the REE, ensuring the security of user data.

[0034] In a possible implementation, the first client application CA is a payment application or an unlocking application, and the unlocking application can be an application program for application lock or screen unlocking.

[0035] In a possible implementation, after the face recognition trusted application TA is started on the side of the first secure virtual machine, the face recognition TA obtains the first face data from the first secure memory, obtains the face template data from the secure chip and decrypts the face template data, and compares the first face data with the decrypted face template data to obtain the face recognition result.

[0036] In a possible implementation, the primary virtual machine further includes a second client application CA, and the second client application CA is used to set face template data. In this case, the method further includes: in response to a second operation of the user on the second client application CA, the second client application CA requests a face entry service from the face recognition service module; the face recognition service module instructs the face recognition control module to start a face entry process; the face recognition control module calls the TEE driver to start a face recognition TA on the side of the first secure virtual machine.

[0037] In a possible implementation, after starting the face recognition TA on the side of the first secure virtual machine, the face recognition TA obtains second face data collected by the camera hardware from the second secure memory, and generates face template data based on the feature values of the second face data.

[0038] In a possible implementation, the first secure virtual machine further includes a secure runtime environment TEE system service, and a secure chip interface is correspondingly set for the TEE system service.

[0039] Wherein, the face recognition TA calls the secure runtime environment TEE system service and the secure chip interface to encrypt and store the face template data in the secure chip; and the face recognition TA calls the secure runtime environment TEE system service and the secure chip interface to obtain the face template data from the secure chip.

[0040] In a possible implementation, the primary virtual machine further includes a first proxy service module, and the second secure virtual machine includes a second proxy service module.

[0041] In this case, the primary virtual machine sets the access permission of the first secure memory, including: the face recognition control module calls the first proxy service and the second proxy service to set the access permission of the first secure memory; the face recognition control module calls the secure runtime environment TEE driver to send information about the access permission of the first secure memory to the secure runtime environment TEE system service.

[0042] In a possible implementation, the first secure virtual machine obtains the first face data from the first secure memory, including: the face recognition TA calls the secure runtime environment TEE system service to obtain the first face data from the first secure memory.

[0043] In a possible implementation, the method further includes: when the camera hardware collects the first face data, the camera hardware calls the camera driver on the side of the primary virtual machine and the first proxy service to write the first face data into the first secure memory.

[0044] Second aspect, the present application provides a chip system, which includes one or more interface circuits and one or more processors. The interface circuits and the processors are interconnected by lines. The above chip system can be applied to an electronic device including a communication module and a memory. The interface circuit is configured to receive a signal from the memory of the electronic device and send the received signal to the processor, and the signal includes computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device can execute the method described in the first aspect and any possible design thereof.

[0045] Third aspect, the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions run on an electronic device (such as a mobile phone), the electronic device is caused to execute the method described in the first aspect and any possible design thereof.

[0046] Fourth aspect, the present application provides a computer program product, which when running on a computer causes the computer to execute the method described in the first aspect and any possible design thereof.

[0047] Fifth aspect, an embodiment of the present application provides a face recognition device, including a processor, the processor is coupled to a memory, and the memory stores program instructions. When the program instructions stored in the memory are executed by the processor, the device implements the method described in the first aspect and any possible design thereof. The device may be an electronic device or a server device; or may be a component of an electronic device or a server device, such as a chip.

[0048] Sixth aspect, an embodiment of the present application provides a face recognition device, and the device can be divided into different logical units or modules according to functions, and each unit or module performs different functions to cause the device to execute the method described in the first aspect and any possible design thereof.

[0049] It can be understood that the beneficial effects that can be achieved by the chip system described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the devices described in the fifth and sixth aspects can refer to the beneficial effects in the first aspect and any possible design thereof, and will not be elaborated here. Description of the Drawings

[0050] Figure 1 It is a schematic diagram of a face recognition scenario provided by an embodiment of the present application;

[0051] Figure 2 It is a schematic diagram of another face recognition scenario provided by an embodiment of the present application;

[0052] Figure 3A Schematic diagram of the architecture of REE and a single TEE provided by an embodiment of the present application;

[0053] Figure 3B Schematic diagram of the architecture of REE and a single TEE provided by an embodiment of the present application;

[0054] Figure 4 Schematic diagram of the architecture of REE and multiple TEEs provided by an embodiment of the present application;

[0055] Figure 5 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application;

[0056] Figure 6 Schematic diagram of a software module architecture provided by an embodiment of the present application;

[0057] Figure 7 Schematic diagram of the system framework of the face recognition method provided by an embodiment of the present application;

[0058] Figure 8 Schematic diagram of signal interaction of the face recognition method provided by an embodiment of the present application;

[0059] Figure 9 Schematic diagram of signal interaction of the face recognition method provided by an embodiment of the present application;

[0060] Figure 10 Schematic diagram of signal interaction of the face recognition method provided by an embodiment of the present application;

[0061] Figure 11 Schematic diagram of the structure of a chip provided by an embodiment of the present application. Detailed implementation manners

[0062] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application. Among them, in the description of the present application, unless otherwise specified, "at least one" means one or more, and "multiple" means two or more than two. In addition, in order to clearly describe the technical solutions in the embodiments of the present application, in the embodiments of the present application, terms such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the terms "first" and "second" do not limit the quantity and execution order, and the terms "first" and "second" do not necessarily limit to be different.

[0063] Face recognition technology is a biometric identification technology that performs identity recognition based on facial feature information. Exemplarily, the implementation principle of face recognition technology is roughly as follows: Facial feature information is extracted from image data containing a face, and then, the extracted facial feature information is compared with the facial feature information corresponding to pre-configured identity information. If the matching degree between the two reaches the preset standard, it is determined that the face in the image data belongs to the person indicated by the above identity information, which can also be called successful face recognition. If the matching degree between the two does not reach the preset standard, it is determined that the face appearing in the image data does not belong to the person indicated by the above identity information, which can also be called failed face recognition. For specific details, reference can be made to related technologies and will not be elaborated here.

[0064] The solution provided by this application is applied to face recognition / authentication scenarios, such as payment authentication scenarios or unlock authentication scenarios, aiming to enhance the security of face authentication.

[0065] Exemplarily, take the scenario where a user operates an electronic device to transfer money as an example. As Figure 1 shown, after the electronic device detects the user's operation indicating a transfer, the electronic device displays a transfer interface 11. During the period when the electronic device displays the transfer interface 11, the electronic device can receive transfer information input by the user (such as, recipient account number, transfer amount, etc.). After that, in response to the user's operation indicating the determination of the transfer, such as the user clicking the "Next" control 12 in the transfer interface 11, the electronic device activates the camera, starts collecting face image data, and displays a face recognition window 13, prompting the user to participate in face recognition.

[0066] The above scenario is a payment authentication scenario. If face recognition / authentication is successful, the payment is successful; if face recognition / authentication fails, the payment fails.

[0067] Another exemplarily, take the scenario where a user operates an electronic device to unlock the screen as an example. As Figure 2 shown, the electronic device is in a locked screen state. Assuming that the user has set the face unlock function, when the user picks up the electronic device for face recognition, in response to the user's operation of picking up the phone, the electronic device can display a locked screen interface 21. During the face recognition process of the electronic device, the electronic device collects face image data through the camera, and the electronic device can display an unlock icon 22 and a prompt text "Face recognition in progress" 23 on the locked screen interface 21.

[0068] The above scenario is an unlock authentication scenario. If face recognition / authentication is successful, the screen unlocks successfully; if face recognition / authentication fails, the screen unlock fails.

[0069] In the above face recognition / verification scenarios, the electronic device collects face image data, which involves the privacy and security information of users. Although the electronic device already provides face recognition function, there is still a risk that the user's face image data may be stolen or damaged by the current face recognition technology.

[0070] Currently, the electronic device adopts Figure 3A the architecture shown in the figure to isolate and protect the face image data, so as to avoid the face image data being stolen or damaged. As Figure 3A shown in the figure, this architecture consists of a Rich Execution Environment (REE) and a Trusted Execution Environment (TEE).

[0071] For the sake of clear and concise description of the following embodiments, a brief introduction to relevant concepts or technologies is given first:

[0072] The Rich Execution Environment (REE), also known as the rich execution environment or the ordinary execution environment or the insecure execution environment, refers to the system execution environment of the mobile terminal, in which operating systems such as Android, IOS, and Linux can run. The REE has good openness and scalability but low security.

[0073] The Trusted Execution Environment (TEE), also known as the secure side or the secure area, is an area that requires authorization to access. The TEE has its own running space and defines strict protection measures. Only authorized security software can execute in the TEE.

[0074] In comparison, the REE is an open environment vulnerable to attacks, such as the theft of sensitive data and the embezzlement of mobile payments, etc.; while the TEE is a secure area on the central processing unit, which can ensure that sensitive data is processed in an isolated and trusted environment, thus being protected from software attacks in the REE. In addition, compared with other trusted execution environments, the TEE can protect the integrity and confidentiality of trusted applications (TAs) end-to-end, and can provide stronger processing capabilities and larger memory spaces.

[0075] The REE+TEE architecture refers to an architecture that provides services for applications through the combination of the TEE and the REE. That is to say, the TEE and the REE coexist in the electronic device. With the support of hardware, the TEE can implement an operating mechanism isolated from the REE. Due to the protection mechanisms such as isolation and permission control of the TEE, it can better protect the security of data and resources.

[0076] A trusted application (TA) is an application that runs in the TEE and can provide security services for client applications (CA) running outside the TEE, such as entering passwords, generating transaction signatures, face recognition, etc.

[0077] A client application (CA) is an application that runs in the REE. The CA can call the TA through the client application programming interface (API) and instruct the TA to perform corresponding security operations.

[0078] TEE secure storage is used to save user sensitive data such as keys and other information. When the user uses the secure storage function to save data, the data will be encrypted within the TEE and then saved to the relevant storage area on the REE side. Currently, the commonly used TEE secure storage is the SFS secure storage. This SFS secure storage utilizes the REE-side file system for storage services. The data encrypted by the TEE will be stored on the REE-side storage medium using the non-secure side file system. Therefore, its capacity is large and it can store more files. However, since it utilizes the non-secure side file system, the stored files are visible on the REE side and the security is relatively low.

[0079] Refer again to Figure 3A As shown, the system architecture includes a main virtual machine VM and a secure virtual machine TVM. Among them, the main virtual machine VM is applied to the general running environment REE and runs the Android system. The client application CA runs on the main virtual machine VM side. For example, the client application CA can be a payment APP, a lock screen APP, or an app lock APP, etc., which require face recognition to complete payment or unlocking. The secure virtual machine TVM is applied to the secure running environment TEE, and the trusted application TA runs on the secure virtual machine TVM side. As Figure 3A shown, the client application CA and the trusted application TA run at the EL0 level, the system kernel and the TEE kernel run at the EL1 level, the virtual machine manager (Hypevisor) runs at the EL2 level, and the secure monitor runs at the EL3 level. The trusted application TA can provide security services such as key generation and key management, security authentication, and face recognition for the client application CA running in the REE. This architecture design realizes an operating mechanism that isolates the TEE from the REE and ensures the security of user data.

[0080] Figure 3B is based on Figure 3A The system architecture schematic diagram for the face recognition / authentication scenario implemented.

[0081] On the Android side, the client application CA (or the face recognition CA) runs. The client application CA calls the camera HAL to turn on the camera, sets the security mode, generates secure memory, and sets the usage permissions for the camera hardware and the TEE system. Exemplarily, the client application CA can be a payment application or an unlocking application.

[0082] On the TEE side, the face recognition trusted application TA (or the face recognition TA) runs. The face recognition TA can complete the following two tasks:

[0083] (1) Face data entry: In the face data entry mode, the face recognition TA calls the TEE HAL to access the secure memory, obtains the image captured by the camera, extracts the face data feature values based on the image captured by the camera, then forms the face template data based on the face data feature values, and then saves the face template data to the file system SFS (file system).

[0084] (2) Face recognition / authentication: In the face authentication mode, the face recognition TA calls the TEE HAL to access the secure memory, obtains the image captured by the camera, extracts the face feature values based on the image captured by the camera, and then compares the extracted face feature values with the face template data saved in the file system SFS.

[0085] It can be seen that in the face recognition / authentication scenario, running the face recognition TA in the TEE TVM, the face authentication TA can provide security services such as face recognition for the client application CA running in the REE. However, since the face template data is stored in the Android-side file system SFS, the Android side can access the file system SFS. Although the face template data has been encrypted, this data storage method still has the risk of being damaged.

[0086] To solve the above problems, this application provides a face recognition method based on a multi-TEE environment. Refer to Figure 4 As shown, this face recognition method adopts the REE + multi-TEE architecture. On the TEE side, it includes two TEE systems, namely the self-developed TEE system (referred to as the first TEE) and the native TEE system (referred to as the second TEE). The first TEE adopts the first secure virtual machine (referred to as TVM1), and the second TEE adopts the first secure virtual machine (referred to as TVM2). Among them, the secure virtual machine can also be called the trusted virtual machine.

[0087] In the multi-TEE environment, a hardware-level encrypted storage solution is provided based on the self-developed TEE system. Refer to Figure 4As shown in the figure, the present application uses an independent secure storage hardware (referred to as a secure chip secflash) to store face template data. Among them, based on the self-developed TEE system, the face template data is written into the secure chip and automatically encrypted during writing. This secure chip can only be accessed by the TEE side and cannot be accessed by the Android side.

[0088] The improvements of the present application compared with the prior art include:

[0089] (1) Based on the existing multi-TEE architecture, the face recognition TA is migrated from the native TEE system to the self-developed TEE system, so that various security features can be fully utilized by the self-developed TEE system.

[0090] (2) Saving the face template data in the secure chip improves the reliability and security of the face data and prevents it from being obtained and damaged by hackers.

[0091] Compared with the related technology that stores face template data through the file system SFS and has low security due to the Android side's accessibility, the present application uses independent secure storage hardware to store face template data, and automatically encrypts the face template data when writing it into the secure chip. This secure chip can only be accessed by the TEE side and cannot be accessed by the Android side. Therefore, the security of the stored data is greatly improved.

[0092] The electronic device described in the embodiment of the present application can be a mobile phone, a personal digital assistant (PDA), a tablet computer or other intelligent devices. A non-secure operating environment and a secure operating environment can be deployed on the electronic device. Among them, the non-secure operating environment is the REE on the electronic device, running operating systems such as Android, iOS, Windows Phone, etc.; the secure operating environment is the TEE, running a secure operating system. Among them, the software and hardware resources accessed by the TEE are isolated from the REE. The software and hardware resources on the electronic device can be respectively identified as two execution environment states. The software and hardware resources identified as the secure execution state can only be accessed by the TEE execution environment, and the software and hardware resources identified as the non-secure execution state can be accessed by both execution environments. The TEE constructs a secure operating environment isolated from the REE and can provide a secure execution environment for authorized trusted software.

[0093] Figure 5 It is a schematic structural diagram of an electronic device 100 provided for the embodiment of the present application.

[0094] As Figure 5As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0095] Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0096] It can be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0097] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0098] The controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0099] A memory can also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0100] In some embodiments, the processor 110 may include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0101] It can be understood that the interface connection relationships between the modules illustrated in this embodiment are only illustrative descriptions and do not constitute a structural limitation on the electronic device 100. In some other embodiments, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0102] The charging management module 140 is configured to receive a charging input from a charger. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0103] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives the inputs from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, the wireless communication module 160, etc. In some other embodiments, the power management module 141 may also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 may also be disposed in the same device.

[0104] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0105] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as the diversity antenna of the wireless local area network.

[0106] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform filtering, amplification and other processing on the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves through the antenna 1 and radiate it out.

[0107] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194.

[0108] The wireless communication module 160 may provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 for radiation.

[0109] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, such that electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0110] Electronic device 100 implements a display function through a GPU, display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, and is connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0111] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), a light-emitting diode (LED), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc.

[0112] The electronic device 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc. The ISP is used to process the data fed back by the camera 193. The camera 193 is used to capture static images or videos. The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0113] The camera 193 can include 1 to N. For example, the electronic device can include 2 front cameras and 4 rear cameras. Among them, the front cameras can include a TOF camera. The TOF camera includes a complementary metal oxide semiconductor (CMOS) or a charge-coupled device (CCD) image sensor.

[0114] The NPU is a neural-network (NN) computing processor. By drawing on the biological neural network structure, such as drawing on the transmission mode between human brain neurons, it can quickly process the input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, voice recognition, text understanding, etc.

[0115] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to implement the storage capacity expansion of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card. The internal memory 121 can be used to store computer-executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. For example, in the embodiment of the present application, the processor 110 can execute the instructions stored in the internal memory 121. The internal memory 121 can include a storage program area and a storage data area. Among them, the storage program area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.). The storage data area can store data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0116] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor, etc. For example, music playback, recording, etc.

[0117] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used for encoding and decoding audio signals. The speaker 170A, also known as the "loudspeaker", is used to convert an audio electrical signal into a sound signal. The receiver 170B, also known as the "earpiece", is used to convert an audio electrical signal into a sound signal. The microphone 170C, also known as the "microphone", "transmitter", is used to convert a sound signal into an electrical signal. The headphone jack 170D is used to connect a wired headphone.

[0118] The button 190 includes a power-on button, volume buttons, etc. The button 190 can be a mechanical button or a touch button. The electronic device 100 can receive button inputs and generate key signal inputs related to the user settings and function control of the electronic device 100. The motor 191 can generate a vibration prompt. The motor 191 can be used for incoming call vibration prompts and also for touch vibration feedback. The indicator 192 can be an indicator light and can be used to indicate the charging status, power change, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect the SIM card. The SIM card can be in contact with and separated from the electronic device 100 by inserting or removing it from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc.

[0119] The methods in the following embodiments can all be implemented in the electronic device 100 with the above hardware structure.

[0120] The software system of the above electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservices architecture, or cloud architecture. In this application embodiment, the Android system with a layered architecture is taken as an example to exemplarily illustrate the software structure of the electronic device 100.

[0121] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through interfaces. In some embodiments, as Figure 6 shown, the Android system can include an application layer, an application framework layer, Android runtime, system libraries, a hardware abstraction layer (HAL), and a kernel layer. It should be noted that this application embodiment takes the Android system as an example. In other operating systems (such as HarmonyOS, IOS system, etc.), as long as the functions implemented by each functional module are similar to those of this application embodiment, the solution of this application can also be implemented.

[0122] Among them, the application layer can include a series of application packages. As Figure 6 shown, the application packages can include camera applications, settings applications, and application programs applied to the face recognition scenario such as payment applications and lock screen applications. Of course, the application programs for the face recognition scenario can also include other application packages, such as shopping applications, bank applications, chat applications, or financial management applications, etc., which are not limited in this application.

[0123] Among them, the settings application has the function of inputting a face image, and the input face image is used in scenarios such as face payment or face unlocking. The lock screen application has the function of unlocking in response to a user's unlocking operation (for example, pressing the power button). The lock screen application can perform unlocking processes such as face unlocking, fingerprint unlocking, and password unlocking. In this embodiment of the application, face payment is taken as an example for illustration.

[0124] In this embodiment of the application, the settings application, the payment application, the lock screen application, etc. are client applications in the face recognition scenario, that is, face recognition CAs. Correspondingly, the applications in the TEE environment include face recognition TAs.

[0125] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions. For example, it may include an activity manager, a window manager, a content provider, a view system, a resource manager, a notification manager, a camera service (Camera Service), and a face recognition service, etc. This embodiment of the application does not impose any restrictions on this.

[0126] The system library can include multiple functional modules. For example: a surface manager, media libraries, OpenGL ES, SGL, etc.

[0127] The surface manager is used to manage the display subsystem and provides the fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0128] The media library supports the playback and recording of multiple common audio and video formats, as well as static image files, etc. The media library can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0129] OpenGL ES is used to implement three-dimensional graphics drawing, image rendering, synthesis, and layer processing, etc.

[0130] SGL is a drawing engine for 2D drawing.

[0131] The Android Runtime includes core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. The core libraries consist of two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core libraries of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files in the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as the management of object life cycles, stack management, thread management, security and exception management, and garbage collection.

[0132] The HAL layer is a wrapper for the Linux kernel driver, providing an interface upward and shielding the implementation details of the underlying hardware.

[0133] The HAL layer can include Wi-Fi HAL, audio HAL, Camera HAL, and a face recognition control module, etc.

[0134] Among them, the Camera HAL is the core software framework of the camera. The Camera HAL can include a sensor node and an image front end (IFE) node. The sensor node and the IFE node are components (nodes) in the transmission path (also called a transmission pipeline) of the image data and control instructions created by the Camera HAL.

[0135] Among them, the face recognition control module is the core software framework / application for face recognition. The face recognition control module can interact with the face recognition service in the application framework layer, can also interact with the Camera HAL in the HAL layer, and can also interact with the TEE driver in the kernel layer.

[0136] The kernel layer is the layer between the hardware and the software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0137] Among them, the camera driver is the driver layer of the Camera device, mainly responsible for the interaction with the hardware.

[0138] The hardware layer includes a display, a camera, a Secure Buffer, and a secure chip, etc.

[0139] Among them, the Secure Buffer refers to the memory with security protection functions. The Secure Buffer is the memory allocated for the TEE side. For example, an electronic device can allocate a part from the double data rate (DDR) synchronous dynamic random access memory as the Secure Buffer to cache data.

[0140] Among them, the security chip is a secure storage chip independent of the secure memory, and uses a specific encryption method for data storage. The security chip can be a SEC flash storage chip. The SEC flash storage chip defines a special interface and has specific read / write timings, which can prevent bus attacks and data theft. The security chip can also use other possible chips, such as a replay protected memory block (RPMB). In practical applications, RPMB is usually used to store some data that requires protection against illegal tampering, such as public keys, serial numbers, etc. related to fingerprint payment on mobile phones.

[0141] This application takes the SEC flash storage chip as an example for illustrative purposes.

[0142] In the embodiments of this application, the secure memory can be used to store the raw images (raw data) captured by the camera. The electronic device can extract face feature values based on the raw images, and further generate face template data according to the face feature values. In a face recognition / authentication scenario, the electronic device can create a secure memory and set the access permissions of the secure memory, allowing the TEE side to access the secure memory and not allowing the Android side to access the secure memory, thus improving the security of the stored data to a certain extent.

[0143] In the embodiments of this application, the security chip can be used to store face template data. The face template data is encrypted by a specific encryption method, and the security chip can only be accessed by the TEE side and not by the Android side, so the security of the stored data is greatly improved.

[0144] The following describes the software modules involved in the face recognition method provided by the embodiments of this application and the interactions between the modules.

[0145] Figure 7 The schematic diagram of the system architecture adopted by the overall solution of this application is shown. As Figure 7 shown, the REE side (main virtual machine) includes a settings application, a payment application / unlock application, a face recognition software development kit (SDK), a face recognition service, a face recognition control module, a camera service, a camera HAL, a camera driver, a TEE driver, a proxy service, etc.

[0146] Among them, the face recognition control module can interact with the face recognition TA of the first TEE side (secure virtual machine TVM1) through the TEE driver.

[0147] The camera HAL may include a sensor node and an IFE node. The sensor node may interact with a camera driver in the kernel layer, and the camera driver may be used to drive the camera in the hardware layer to collect image data.

[0148] The first TEE side (secure virtual machine TVM1) includes a face recognition TA and TEE system services. The TEE system services include a secure chip interface, and the first TEE side (secure virtual machine TVM1) can access the secure chip through this interface.

[0149] The second TEE side (secure virtual machine TVM2) includes a proxy service.

[0150] It should be noted that Figure 7 the system framework is for illustrative purposes. In actual implementation, the REE side (main virtual machine), the first TEE side (secure virtual machine TVM1), and the second TEE side (secure virtual machine TVM2) may include more or fewer modules, which are not limited in the embodiments of the present application. Among them, the main improvement point of the solution of the present application lies in the first TEE side (secure virtual machine TVM1).

[0151] As Figure 7 shown, the system architecture also includes a virtual machine manager. The virtual machine manager manages the life cycles of the main virtual machine on the REE side, the secure virtual machine TVM1 on the first TEE side, and the secure virtual machine TVM2 on the second TEE side, etc.

[0152] The system architecture also includes a resource manager, which is used to manage system resources and the access permissions of the secure virtual machine TVM1, the secure virtual machine TVM2, and the camera hardware to secure memory, etc.

[0153] The system architecture also includes hardware such as a camera, an image pre-processing IFE module, secure memory, and a secure chip.

[0154] Among them, the image pre-processing IFE module can store the image data collected by the camera into the secure memory. The storage location of the image data collected by the camera in the secure memory can be represented by a file descriptor (FD).

[0155] As Figure 7 shown, the payment application and the settings application in the application layer can respectively interact with the face recognition SDK. The face recognition SDK interacts with the face recognition service in the framework layer by calling a preset application programming interface (API) interface. The face recognition service can interact with the face recognition control module in the HAL layer. The face recognition control module can interact with the camera HAL in the HAL layer through the camera service in the framework layer, or the face recognition control module can directly interact with the camera HAL in the HAL layer.

[0156] Next, the overall process of the face recognition method provided by the embodiments of the present application will be described in stages. Among them, the overall process of the face recognition method includes a face template data entry process (abbreviated as face entry process) and a face authentication process.

[0157] The first stage: Face entry process

[0158] In the face entry process, the settings application in the application layer interacts with the face recognition service through the face recognition SDK.

[0159] The user starts the face entry process through the settings application, and the face recognition control module sends an instruction to the first TEE to load the face recognition trusted application TA (abbreviated as face recognition TA) on the first TEE side.

[0160] The face recognition control module calls the camera middleware service (such as the camera service, camera HAL, camera driver) to start the camera, and triggers the creation of a secure memory through the camera HAL to cache the image data collected by the camera. And set the access permission of the secure memory through the proxy services on the Android side and the second TEE side, so that both TVM1 of the first TEE and TVM2 of the second TEE can access this secure memory. Once the camera is started, only TVM1 of the first TEE and TVM2 of the second TEE can access the secure memory, and the Android side is not allowed to access this secure memory.

[0161] After the face recognition control module receives the notification that the camera hardware image is ready, the face recognition control module notifies the face recognition TA to collect face data by obtaining it from the secure memory. The face recognition TA extracts the face feature value based on the face data and generates the face template data according to the face feature value. Then, the face recognition TA saves the face template data to the secure chip through the secure chip interface provided by the first TEE.

[0162] The face recognition control module turns off the camera and takes back the access permission of the secure memory. The face entry process ends.

[0163] The second stage: Face recognition process

[0164] In the face recognition process, the payment application / unlock application in the application layer interacts with the face recognition service through the face recognition SDK.

[0165] In the face recognition / authentication scenarios such as when the user triggers the mobile phone unlocking / application lock service or an Android third-party application (payment application), the face authentication process is started. The face recognition control module sends an instruction to the first TEE to start the face recognition TA in the first TEE.

[0166] The face recognition control module calls the camera middle layer services (such as the camera service, camera HAL, camera driver) to start the camera, and triggers the creation of a secure memory through the camera HAL to cache the image data collected by the camera. And the access permissions of the secure memory are set through the proxy services on the Android side and the second TEE side, so that both the TVM1 of the first TEE and the TVM2 of the second TEE can access this secure memory. Once the camera is started, only the TVM1 of the first TEE and the TVM2 of the second TEE can access the secure memory, and the Android side is not allowed to access this secure memory.

[0167] After the face recognition control module receives the notification that the camera hardware image is ready, the face recognition control module notifies the face recognition TA to obtain the face data from the secure memory, and the face recognition TA extracts the face feature values based on the face data. The face data reads the face template data from the secure chip through the secure chip interface provided by the first TEE, and then compares the face template data with the real-time collected face data. If the comparison result meets the preset value, then the face recognition / authentication is successful. If the comparison result does not meet the preset value, then the face recognition / authentication fails.

[0168] The face recognition control module turns off the camera and withdraws the access permissions of the secure memory. The face recognition / authentication process ends.

[0169] For ease of understanding, the method provided in the embodiments of the present application will be specifically introduced below in conjunction with the accompanying drawings.

[0170] Face entry process

[0171] Based on the Figure 7 framework diagram of Figure 8 the present application, the face entry process in the face recognition method provided by the embodiments of the present application is shown as follows:

[0172] S101. In response to a user operation, set the application to trigger the start of the face entry process.

[0173] Among them, the setting application can call the face recognition SDK to perform face entry.

[0174] S102. The setting application sends a message to the face recognition service to indicate the start of the face entry process. Then, the face recognition service sends a message to the face recognition control module to indicate the start of the face entry process.

[0175] S103. The face recognition control module sends a message to the secure virtual machine TVM1 to indicate loading the face recognition TA on the secure virtual machine TVM1 side.

[0176] S104. The face recognition TA is loaded on the secure virtual machine TVM1 side.

[0177] Accordingly, the secure virtual machine TVM1 side can return a message indicating that the face recognition TA has been successfully loaded to the face recognition control module.

[0178] In S105 and S106, the face recognition control module calls the camera HAL to create a secure memory.

[0179] For example, the electronic device calls the camera HAL to allocate a portion of the memory from the DDR as the secure memory.

[0180] Accordingly, the memory device feeds back a message indicating that the secure memory has been successfully created to the camera HAL.

[0181] In S107, the camera HAL calls the proxy service of the main virtual machine VM and the proxy service of the secure virtual machine TVM2 to set the access permissions for the secure memory. Among them, the secure virtual machines TVM1, TVM2, and the camera hardware have the permission to access the secure memory, while the main virtual machine VM does not have the permission to access the secure memory.

[0182] In S108, the secure memory feeds back a message indicating that the permissions for the secure memory have been successfully set.

[0183] The message indicating that the permissions for the secure memory have been successfully set includes: the secure virtual machines TVM1, TVM2, and the camera hardware have the permission to access the secure memory.

[0184] Among them, the access permissions include write permissions and read permissions.

[0185] The camera hardware has write permissions. For example, the camera hardware (camera) can write face data into the secure memory.

[0186] The secure virtual machines TVM1 and TVM2 have read permissions. For example, the secure virtual machine TVM1 can read face data from the secure memory.

[0187] Among them, the message is passed to the camera hardware, the proxy service of TVM2, the proxy service of the main virtual machine VM, and the TEE system service of the secure virtual machine TVM1.

[0188] The above steps are the preparation stage of the face entry process, and the following steps are the specific execution stage of the face entry process.

[0189] In S109, the face recognition control module sends a message to the secure virtual machine TVM1, instructing the face recognition TA to execute the face entry process.

[0190] Among them, the face recognition control module can interact with the secure virtual machine TVM1 on the first TEE side through the TEE driver on the main virtual machine VM side. Among them, the face recognition control module can call the TEE driver to start the face recognition TA.

[0191] S110 and S111, the face recognition control module calls the camera HAL to start the camera hardware (camera).

[0192] Among them, the face recognition control module calls the camera HAL, and the camera HAL then calls the camera driver module, and the camera driver module triggers the start of the camera hardware (camera).

[0193] After the camera hardware (camera) starts, it begins to collect images.

[0194] S112, the camera hardware (camera) feeds back a message that the camera hardware is ready to the camera HAL, and then the camera HAL passes the message that the camera hardware is ready to the face recognition control module.

[0195] S113, the camera hardware (camera) collects face data and writes the collected face data into the secure memory.

[0196] As described above, the camera has the permission to write the collected images into the secure memory. Therefore, after the camera collects the face image, the camera writes the face image into the secure memory.

[0197] Among them, the face data is a face image. For example, the face image can be a 3D face image.

[0198] Among them, the camera is a TOF sensor or other 3D sensors.

[0199] Exemplarily, the present application can adopt a camera based on time of flight (TOF) imaging technology, and 3D face image features can be collected through the TOF camera.

[0200] Among them, the TOF imaging technology refers to a group of infrared lights (or laser pulses) that are invisible to the human eye being emitted outward, reflected after encountering an object, and then reflected back to the camera. The time difference or phase difference from the emission to the reflection back to the camera is calculated, and the data is collected to form a set of distance-depth data, thereby obtaining a three-dimensional 3D model imaging technology. That is to say, the TOF imaging technology adds depth information from the Z-axis direction on the basis of the traditional 2D XY-axis imaging, and finally generates 3D image information. Since the TOF image is not affected by most of the light in the environment, applying the TOF image in payment / unlock services can improve the reliability of face recognition.

[0201] S114, the secure memory feeds back a message that the face data has been written into the secure memory to the camera hardware (camera), and then the camera hardware (camera) feeds back a message that the face data has been written into the secure memory to the face recognition TA.

[0202] S115. The face recognition TA requests face data from the secure memory.

[0203] Among them, the face recognition TA calls the TEE system service of the secure virtual machine TVM1 to request face data from the secure memory.

[0204] S116. The secure memory returns face data to the face recognition TA.

[0205] Among them, the secure memory calls the TEE system service of the secure virtual machine TVM1 to return face data to the face recognition TA.

[0206] S117. The face recognition TA extracts face feature values based on the face data collected by the camera and generates face template data according to the face feature values.

[0207] S118 and S119. The face recognition TA calls the secure chip interface of the secure virtual machine TVM1 to encrypt the face template data and store it in the secure chip.

[0208] In the embodiment of the present application, the face recognition TA encrypts the face template data and then stores the encrypted face template data in the secure chip. The data encryption algorithm can be a preset encryption algorithm, such as a symmetric encryption algorithm, and the symmetric encryption algorithm can be the advanced encryption standard (AES) 256 algorithm.

[0209] S120. The secure chip feeds back a message indicating that face entry has been completed to the face recognition TA, and then the face recognition TA feeds back a message indicating that face entry has been completed to the face recognition control module.

[0210] S121 and S122. The face recognition control module calls the camera HAL to turn off the camera hardware (the camera).

[0211] Correspondingly, the camera hardware (the camera) can feed back a message indicating that the camera has been turned off to the camera HAL, and then the camera HAL feeds back a message indicating that the camera hardware (the camera) has been turned off to the face recognition control module.

[0212] S123. The face recognition control module calls the camera HAL to cancel the access permission to the secure memory. Among them, the camera HAL calls the proxy service of the main virtual machine VM and the proxy service of the secure virtual machine TVM2 to cancel the access permission of the secure virtual machines TVM1 and TVM2 to the secure memory.

[0213] S124 and S125. The face recognition control module calls the camera HAL to release the secure memory.

[0214] Moreover, the face recognition control module can call the TEE driver to turn off the face recognition TA.

[0215] The face enrollment process ends.

[0216] Face recognition / verification process

[0217] Figure 9 The face recognition / verification process in the face recognition method provided by the embodiments of the present application is shown. In combination with Figure 8 , as Figure 9 shown, after S119 (the security chip has stored the face template data), the present application further includes the following process:

[0218] S201. In response to a user operation, set an application to trigger and start the face enrollment process.

[0219] Among them, the set application can call the face recognition SDK to perform face recognition.

[0220] S202. The set application sends a message to the face recognition service to indicate starting the face recognition process. Then, the face recognition service sends a message to the face recognition control module to indicate starting the face recognition process.

[0221] S203. The face recognition control module sends a message to the secure virtual machine TVM1 to indicate starting the face recognition TA on the secure virtual machine TVM1 side.

[0222] S204. The face recognition TA is started on the secure virtual machine TVM1 side.

[0223] Correspondingly, the secure virtual machine TVM1 side can return a message to the face recognition control module indicating that the face recognition TA has been started.

[0224] S205 and S206. The face recognition control module calls the camera HAL to create a secure memory.

[0225] For example, the electronic device calls the camera HAL to allocate a part of the memory from the DDR as the secure memory.

[0226] Correspondingly, the memory device feeds back a message to the camera HAL indicating that the secure memory has been created.

[0227] S207. The camera HAL calls the proxy service of the main virtual machine VM and the proxy service of the secure virtual machine TVM2 to set the access permission of the secure memory, where the secure virtual machines TVM1, TVM2 and the camera hardware have the permission to access the secure memory, and the main virtual machine VM has no permission to access the secure memory.

[0228] S208. The secure memory feeds back a message indicating that the access permission of the secure memory has been set.

[0229] The message indicating that the permissions for the secure memory have been set includes: the secure virtual machines TVM1 and TVM2, and the camera hardware have permissions to access the secure memory. Among them, the access permissions include write permissions and read permissions. The camera hardware has write permissions. The camera hardware (camera) can write face data into the secure memory. The secure virtual machines TVM1 and TVM2 have read permissions. The secure virtual machine TVM1 can read face data from the secure memory.

[0230] Among them, this message is transmitted to the camera hardware, the proxy service of TVM2, the proxy service of the main virtual machine VM, and the TEE system service of the secure virtual machine TVM1.

[0231] The above steps are the preparation stage of the face recognition / authentication process, and the following steps are the specific execution stage of the face recognition / authentication process.

[0232] S209. The face recognition control module sends a message to the secure virtual machine TVM1, instructing the face recognition TA to execute the face enrollment process.

[0233] S210 and S211. The face recognition control module calls the camera HAL to start the camera hardware (camera).

[0234] Among them, the face recognition control module calls the camera HAL, and the camera HAL then calls the camera driver module, and the camera driver module triggers the start of the camera hardware (camera).

[0235] After the camera hardware (camera) starts, it begins to collect images.

[0236] S212. The camera hardware (camera) feeds back the message that the camera hardware is ready to the camera HAL, and then the camera HAL passes the message that the camera hardware is ready to the face recognition control module.

[0237] S213. The camera hardware (camera) collects face data and writes the collected face data into the secure memory.

[0238] S214. The secure memory feeds back the message that the face data has been written into the secure memory to the camera hardware (camera), and then the camera hardware (camera) feeds back the message that the face data has been written into the secure memory to the face recognition TA.

[0239] S215. The face recognition TA requests face data from the secure memory.

[0240] S216. The secure memory returns the face data to the face recognition TA.

[0241] The implementation process of the above steps S201 - S216 is similar to the above steps S101 - S116, and will not be elaborated here.

[0242] S217. The face recognition TA extracts face feature values based on the face data collected by the camera.

[0243] S218. The face recognition TA calls the security chip interface of the security virtual machine TVM1 to send a request message to the security chip, and this request message is used to read the face template data.

[0244] S219. The security chip calls the security chip interface of the security virtual machine TVM1 to return the face template data to the face recognition TA.

[0245] S220. The face recognition TA decrypts the face template data.

[0246] In the embodiment of the present application, the face template data obtained by the face recognition TA is encrypted face template data. The face recognition TA can use a preset key to decrypt the face template data to obtain the decrypted face template data.

[0247] S221. The face recognition TA compares the face feature values collected by the camera with the decrypted face template data.

[0248] S222. The face recognition TA determines whether the comparison result meets a preset threshold.

[0249] S223. In the case where the comparison result meets the preset threshold, the face recognition TA obtains the face recognition result: recognition successful. Recognition successful means that the identity authentication is successful.

[0250] In the case where the comparison result does not meet the preset threshold, the face recognition TA obtains the face recognition result: recognition failed. Recognition failed means that the identity authentication fails.

[0251] S224. The face recognition TA feeds back the face authentication result, and the camera HAL, the face recognition service, and the payment application / unlock application all receive the face authentication result.

[0252] S225. After receiving the face authentication result, the payment application / unlock application completes the payment or unlocking according to the face authentication result.

[0253] S226 and S227. After receiving the face authentication result, the face recognition control module calls the camera HAL to turn off the camera hardware (the camera).

[0254] Correspondingly, the camera hardware (the camera) can feed back a message that the camera has been turned off to the camera HAL, and then the camera HAL feeds back a message that the camera has been turned off to the face recognition control module.

[0255] S228. The face recognition control module calls the camera HAL to cancel the access permission to the secure memory. Among them, the camera HAL calls the proxy service of the main virtual machine VM and the proxy service of the secure virtual machine TVM2 to cancel the access permissions of the secure virtual machines TVM1 and TVM2 to the secure memory.

[0256] S229 and S230. The face recognition control module calls the camera HAL to trigger the release of the secure memory.

[0257] Moreover, the face recognition control module can call the TEE driver to close the face recognition TA.

[0258] The face recognition / authentication process ends.

[0259] The following combines Figure 10 Taking the face recognition / authentication scenario as the unlocking authentication scenario and the camera as a TOF camera as an example, an exemplary description is given of the face recognition / authentication solution provided by this application. The following steps are executed after Figure 8 S119 (storing the face template data in the secure chip).

[0260] S301. When detecting the user's unlocking operation, the lock screen application calls the face recognition SDK to perform face recognition.

[0261] Among them, the user's unlocking operation includes the user picking up the mobile phone, or pressing the power button, or operating on the screen (clicking, swiping, etc.), or unplugging the charging cable, etc.

[0262] At the same time, the lock screen application can register a callback with the face recognition SDK. The function of registering this callback is that when the face recognition SDK obtains the face recognition result, it can return the face recognition result to the lock screen application.

[0263] S302. The face recognition SDK sends a face recognition request to the face recognition service. Among them, the face recognition request carries the identifier of the face recognition type, the resolution size of the image, and the data stream format. Among them, the face recognition types include 2D face recognition type (for example, it can correspond to the identifier 0) and 3D face recognition type (for example, it can correspond to the identifier 1).

[0264] Exemplarily, the face recognition type carried in the face recognition request can be 1 (i.e., 3D face recognition type), the resolution size of the image can be 1280x2898 pixels (pixel), and the data stream format can be the raw image format (rawimage format, RAW) 16.

[0265] Meanwhile, the face recognition SDK can register a callback with the face recognition service. The purpose of registering this callback is that when the face recognition service obtains the face comparison result, it can return the face recognition result to the face recognition SDK.

[0266] S303. The face recognition service sends a face recognition request to the face recognition control module. That is to say, the face recognition SDK can notify the face recognition control module to perform face recognition through the face recognition service. The face recognition service can send the face recognition request received from the face recognition SDK to the face recognition control module.

[0267] Meanwhile, the face recognition service can register a callback with the face recognition control module. The purpose of registering this callback is that when the face recognition control module obtains the face comparison result, it can return the face comparison result to the face recognition service.

[0268] S304. In response to receiving the face recognition request, the face recognition control module matches a camera according to the face recognition request.

[0269] Specifically, the face recognition control module can obtain the identifier of the face recognition type, the resolution size of the image, and the data stream format from the face recognition request, and determine the matching camera by querying the camera capabilities from the camera service.

[0270] It should be understood that during the boot process of the electronic device, the camera service can send a camera capability query request to the camera HAL. The camera capability query request is used to request a query of the camera capabilities supported by the electronic device. After receiving the camera capability query request, the camera HAL can send the capabilities of the cameras supported by the electronic device to the camera service, and the camera service can store the received capabilities of the cameras supported by the electronic device. Among them, the capabilities of the cameras supported by the electronic device include the camera identity (ID) of each camera, the maximum resolution size supported, the format of the data stream, and whether the camera supports collecting depth information, etc.

[0271] Exemplarily, assume that there are three cameras installed on the mobile phone, and the capability information of these three cameras can be shown in Table 1:

[0272] Table 1

[0273] Camera ID Installation Location Maximum Supported Resolution Data Stream Format Depth Information 1 Rear 4096x3072 pixel YUY No 2 Front 3264x2448 pixel YUY No 3 Front 1280x2898 pixel RAW16 Yes

[0274] Among them, the camera with camera ID 3 can be a TOF camera and supports collecting depth information. The cameras with camera IDs 1 and 2 can be ordinary cameras and do not support collecting depth information. Of course, more front or rear cameras can be installed on the mobile phone. For example, the mobile phone can install 2 front cameras and 4 rear cameras.

[0275] The face recognition control module can send a camera capability query request to the camera service. The camera service can send the capabilities of the cameras supported by the electronic device to the face recognition control module. The face recognition control module can determine a matching camera based on the capabilities of the cameras supported by the electronic device. For example, it can determine that the matching camera is the camera with ID 3 (i.e., the TOF camera).

[0276] It should be noted that Table 1 is only an example, and the data stream format corresponding to each camera can include multiple types. For example, the camera with camera identifier 1 can correspond not only to the YUY data stream format but also to the RAW16 data stream format, which is not limited in this application.

[0277] S305. The face recognition control module sends a request to the camera service to open the camera (Camera).

[0278] Exemplarily, the face recognition control module can send a request for opening the Camera to the camera service through the vendor native development kit (VNDK) interface. Among them, the request for opening the Camera carries information such as a security identifier, a camera ID, the size of the resolution, and the data stream format.

[0279] Among them, the security identifier is used to indicate storing data in a secure Buffer. That is to say, the security identifier can be used to apply for a piece of secure memory for storing the data collected by the camera later. For example, the security identifier can be 1 or 0. 1 indicates storing data in the secure Buffer, and 0 indicates storing data in a non-secure Buffer.

[0280] Exemplarily, the security identifier carried in the request for opening the Camera can be 1 (i.e., storing data in the secure Buffer), the size of the image resolution can be 1280x2898 pixel, the data stream format can be RAW16, and the camera ID can be 3.

[0281] At the same time, the face recognition control module can register a callback with the Camera service. Registering this callback is used to notify the face recognition control module that the Camera has been opened after the camera service completes opening the camera.

[0282] S306. In response to receiving the request to open the Camera, the camera service sends a request to open the Camera to the camera HAL. The request to open the Camera carries information such as a security identifier, a camera ID, the size of the resolution, and the data stream format.

[0283] During the process of the camera service calling the camera HAL, the camera service can send information such as security identification, camera ID, image resolution, and data stream format to the camera HAL. The camera HAL can cache information such as security identification, camera ID, image resolution, and data stream format for a preset time.

[0284] Meanwhile, the camera service can register a callback with the camera HAL, which is used for the camera HAL to notify the camera service of the result of creating a path.

[0285] S307. The camera HAL creates a corresponding path according to the camera ID, image resolution, and data stream format.

[0286] The camera HAL can select available nodes according to the camera ID, resolution, and data stream format, and then create a corresponding path based on the available nodes. Exemplarily, if the resolution is 1280x2898 pixel, the data stream format is RAW16, and the camera ID is 3, it can be determined to select the sensor node and the IFE node. This is because the sensor node and the IFE node can support the transmission of data with a resolution of 1280x2898 pixel and a data stream format of RAW 16 collected by the camera with camera ID 3.

[0287] Among them, the path corresponding to the sensor node can be: the path composed of the sensor node - camera driver - TOF camera - IFE module - secure memory. The path corresponding to the IFE node can be: the path composed of the IFE module (carrying FD) - camera driver - IFE node. The camera HAL can connect the output port of the sensor node and the input port of the IFE node at the HAL layer. Thus, the path corresponding to the sensor node and the path corresponding to the IFE node can form a closed-loop path. After the path is created, the hardware in the path is powered on (i.e., the hardware circuit is energized) and waits for a data request.

[0288] S308. The camera HAL returns the result of creating a path to the camera service.

[0289] Among them, the result of creating a path can be success or failure. If the result of creating a path is failure, the camera HAL notifies the camera service that the path creation fails. If the result of creating a path is success, the camera HAL notifies the camera service that the path creation is successful, and S109 and its subsequent steps can be continued.

[0290] S309. In response to receiving the notification that the path creation is successful, the camera service returns a message indicating that the camera is successfully opened to the face recognition control module.

[0291] It can be understood that the successful opening of the camera means that the preparatory work before the camera takes pictures or videos (such as camera parameter configuration, power-on, etc.) has been completed.

[0292] S310. In response to receiving the message indicating that the camera is successfully turned on, the face recognition control module sends a data request to the camera service.

[0293] Among them, the data request is used to request to obtain the data stream of the camera.

[0294] S311. In response to receiving the data request sent by the face recognition control module, the camera service calls the camera HAL to obtain the data stream.

[0295] S312. The sensor node sends the configuration parameters of the camera to the camera driver (CameraDriver) module in the Kernel layer.

[0296] It should be understood that the sensor node can store the addresses of the respective registers of the TOF camera, and the addresses of the respective registers of the TOF camera can be as shown in Table 3.

[0297] Table 3

[0298] Register Identification Stored Data Type Address 1 Current 0x1 2 Resolution 0x2 3 Data Stream Format 0x3 4 TOF Camera Device Working Status 0x4 5 Working Mode of TOF Camera 0x5

[0299] S313. The camera driver module writes (updates) the configuration parameters of the camera into the registers of the TOF camera.

[0300] That is to say, the camera driver module can send the configuration parameters to the TOF camera.

[0301] S314. The camera driver module sends a start (stream on) command / instruction to the TOF camera.

[0302] The start command is used to drive the TOF camera to perform data acquisition.

[0303] It should be noted that before S314 and after S313, the camera driver module can also send a message indicating that the configuration parameter writing is completed to the sensor node; in response to receiving the message indicating that the configuration parameter writing is completed, the sensor node sends a start command to the camera driver module.

[0304] S315. In response to receiving the start command, the TOF camera acquires RAW Data.

[0305] Among them, the RAW Data contains Metadata. Exemplarily, the Metadata stores information such as the working mode of the current TOF camera, the working state of the TOF camera device (e.g., normal or abnormal), and the frame exposure value (e.g., 10 μs).

[0306] S316. The TOF camera sends the acquired RAW Data to the IFE module.

[0307] Exemplarily, the TOF camera can transmit the RAW Data collected by the TOF camera to the IFE module through the mobile industry processor interface (MIPI). The IFE module can also be referred to as an image pre-processing module (IFE-Lite), and the IFE module may not process the RAW Data.

[0308] S317. The IFE module sends the RAW Data to be stored in the Secure Buffer.

[0309] The storage location of the RAW Data collected by the TOF camera in the secure memory can be represented by FD1.

[0310] Exemplarily, when FD1 is 69, it can represent the storage location as XX secure memory; when FD1 is 96, it can represent the storage location as YY non-secure memory (ordinary memory).

[0311] S318. The IFE module sends FD1 to the camera driver module. The camera driver module sends FD1 to the IFE node. The IFE node sends FD1 to the camera service through the interface of the camera HAL. The camera service sends FD1 to the face recognition control module.

[0312] S319. The face recognition control module sends FD1 to the face recognition TA (FACE TA).

[0313] Among them, the face recognition TA contains a TOF algorithm and a face ID algorithm. Among them, the TOF algorithm is used to convert the RAW Data into a grayscale image and a depth image, and calculate whether the face is safe according to the grayscale image and the depth image (that is, whether the current user is the machine owner). The face ID algorithm is used for grayscale image matching and depth image anti-counterfeiting judgment.

[0314] S320. The face recognition TA reads the RAW Data from the secure memory according to FD1.

[0315] The face recognition TA can request the RAW Data from the secure memory according to FD1, and the secure memory sends the RAW Data to the face recognition TA.

[0316] S321. The face recognition TA obtains the face template data from the secure chip through the secure chip interface.

[0317] S322. The face recognition TA compares the RAW Data with the face template data to obtain the face recognition result.

[0318] The face recognition TA can process the RAW Data through the TOF algorithm to obtain the first grayscale image and the first depth image, and then perform face recognition based on the first grayscale image through the face ID algorithm, and perform anti-counterfeiting detection based on the first depth image, so as to obtain the face recognition result.

[0319] It should be noted that after the face recognition TA obtains the face template data, it can convert the face template data into a grayscale image and a depth image through the TOF algorithm.

[0320] If the grayscale image corresponding to the currently collected face information (the RAW Data currently collected by the TOF camera) matches the face template data (that is, the RAW Data collected by the electronic device when the user performs the face entry operation), it can be considered the same user (that is, the user performing the face entry operation and the unlocking operation is the same user). And if the currently collected face information includes depth information, it can be considered that the current user is real and trustworthy (not disguised by photos, videos, etc.). At this time, it can be considered that the face of the current user is secure, that is, the face recognition result is successful.

[0321] If the grayscale image corresponding to the currently collected face information (the RAW Data currently collected by the TOF camera) does not match the face template data (that is, the RAW Data collected by the electronic device when the user performs the face entry operation), or if the currently collected face information does not include depth information, it is considered that the face of the current user is not secure, that is, the face recognition result is recognition failure.

[0322] S323. The face recognition TA sends the face recognition result to the face recognition control module. The face recognition control module sends the face recognition result to the face recognition service. The face recognition service passes the face recognition result to the face recognition SDK. The face recognition SDK passes the face recognition result to the lock screen application.

[0323] The face recognition control module can, based on the callback registered by the face recognition service before (in S303), pass the face recognition result (success or failure) to the face recognition service.

[0324] The face recognition service, based on the callback registered by the face recognition SDK before (in S302), passes the face recognition result (success or failure) to the face recognition SDK.

[0325] The face recognition SDK, based on the callback registered by the lock screen application before (in S301), passes the face recognition result (success or failure) to the lock screen application.

[0326] S324. The lock screen application unlocks or does not unlock according to the face recognition result.

[0327] If the face recognition result is successful, the lock screen application can be successfully unlocked, so that the electronic device can display the desktop or the interface of an application (system application or third-party application).

[0328] If the face recognition result is failed, the lock screen application will not be unlocked, that is, the face unlock fails. After the face unlock fails, the lock screen application can disable the face recognition function for a period of time (for example, 5 minutes) after the face recognition fails.

[0329] It should be noted that the above embodiments are described by taking the method flow of face unlock by the lock screen application as an example. The above solution can also be applied to face recognition during payment or transfer (for example, when the user performs payment or transfer operations in a payment application / wealth management application / chat application / shopping application). The present application does not make any limitations in scenarios such as face security verification for user's secure registration or login to an application program. That is, the lock screen application can be replaced with a shopping application, a chat application, a payment application, a banking application, a wealth management application, etc., and the present application does not make any limitations.

[0330] Some embodiments of the present application provide an electronic device, which may include: a touch screen, a memory, and one or more processors. The touch screen, the memory, and the processor are coupled. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can perform each function or step executed by the electronic device in the above method embodiments. The structure of the electronic device can refer to Figure 4 the structure of the electronic device 100 shown.

[0331] Embodiments of the present application also provide a chip system, such as a system on a chip (SoC), as Figure 11 shown, the chip system includes at least one processor 1101 and at least one interface circuit 1102. The processor 1101 and the interface circuit 1102 can be interconnected through a line. For example, the interface circuit 1102 can be used to receive signals from other devices (such as the memory of an electronic device). For another example, the interface circuit 1102 can be used to send signals to other devices (such as the processor 1101 or the touch screen of an electronic device). Exemplarily, the interface circuit 1102 can read the instructions stored in the memory and send the instructions to the processor 1101. When the instructions are executed by the processor 1101, the electronic device can perform each step in the above embodiments. Of course, the chip system can also include other discrete devices, and the embodiments of the present application do not make specific limitations on this.

[0332] Embodiments of the present application also provide a TOF camera, which can be used to implement the above embodiments. An electronic device equipped with the TOF camera can perform each function or step executed by the electronic device in the above method embodiments.

[0333] The embodiments of the present application also provide a computer-readable storage medium, which includes computer instructions. When the computer instructions run on the above-mentioned electronic device, the electronic device is enabled to execute each function or step executed by the electronic device in the above method embodiments.

[0334] The embodiments of the present application also provide a computer program product. When the computer program product runs on an electronic device, the electronic device is enabled to execute each function or step executed by the electronic device in the above method embodiments.

[0335] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above function modules is used as an example. In actual applications, the above functions can be allocated to different function modules according to needs, that is, the internal structure of the device is divided into different function modules to complete all or part of the functions described above.

[0336] In the several embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0337] The unit described as a separated component may or may not be physically separated. The component displayed as a unit may be a physical unit or multiple physical units, that is, it may be located in one place, or may be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0338] In addition, each functional unit in the various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0339] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0340] The above content is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claimed rights.

Claims

1. A face recognition method, applied to an electronic device, the electronic device including a main virtual machine, a first secure virtual machine, and a second secure virtual machine; characterized in that, The method includes: The main virtual machine receives a first operation from the user, and the first operation is an operation to trigger a face recognition process; In response to the first operation, the main virtual machine creates a first secure memory and sets the access permission of the first secure memory; the camera hardware of the electronic device has the write permission to the first secure memory, and the first secure virtual machine and the second secure virtual machine have the read permission to the first secure memory; The main virtual machine calls the camera HAL to trigger the camera hardware to start; When the first face data collected by the camera hardware is written into the first secure memory, the first secure virtual machine obtains the first face data from the first secure memory and obtains face template data from the secure chip of the electronic device; The first secure virtual machine compares the first face data with the face template data to obtain a face recognition result; Among them, the first secure virtual machine has the read permission to the secure chip, and the main virtual machine does not have the read permission to the secure chip.

2. The method according to claim 1, characterized in that, The main virtual machine runs in a general running environment REE, the first secure virtual machine runs in a first secure running environment TEE, and the second secure virtual machine runs in a second secure running environment TEE.

3. The method according to claim 1, wherein Before the main virtual machine receives the first operation from the user, the method further includes: The main virtual machine receives a second operation from the user, and the second operation is an operation to trigger a face enrollment process; In response to the second operation, the main virtual machine creates a second secure memory and sets the access permission of the second secure memory; the camera hardware has the write permission to the second secure memory, and the first secure virtual machine and the second secure virtual machine have the read permission to the second secure memory; The main virtual machine calls the camera HAL to trigger the camera hardware to start; When the second face data collected by the camera hardware is written into the second secure memory, the first secure virtual machine obtains the second face data from the second secure memory and generates the face template data according to the feature value of the second face data; The first secure virtual machine encrypts the face template data and stores it in the secure chip.

4. The method according to claim 3, characterized in that, After the first secure virtual machine encrypts the face template data and stores it in the secure chip, the method further includes: The main virtual machine calls the camera HAL to trigger the camera hardware to close and trigger the release of the second secure memory.

5. The method according to any one of claims 1 to 4, characterized in that, After the first secure virtual machine obtains the face recognition result, the method further includes: The first secure virtual machine sends the face recognition result to the main virtual machine; The main virtual machine receives the face recognition result and executes a face recognition service according to the face recognition result, and the face recognition service includes face payment or face unlocking.

6. The method according to claim 5, characterized in that, After the main virtual machine receives the face recognition result, the method further includes: The main virtual machine calls the camera HAL to trigger the camera hardware to close and trigger the release of the first secure memory.

7. The method according to any one of claims 1 to 6, characterized in that, The main virtual machine creates a first secure memory, including: The main virtual machine calls the camera HAL and creates the first secure memory based on the running memory of the electronic device.

8. The method according to any one of claims 1 to 7, characterized in that, The first secure virtual machine obtains face template data from the secure chip of the electronic device, including: The first secure virtual machine calls the secure chip interface and obtains the encrypted face template data from the secure chip; The first secure virtual machine decrypts the face template data.

9. The method according to any one of claims 2 to 8, characterized in that, The first secure virtual machine includes a face recognition trusted application TA; the main virtual machine includes a first client application CA, a face recognition service module, a face recognition control module, and a secure runtime environment TEE driver; The method further includes: In response to a first operation of the user on the first client application CA, the first client application CA requests a face recognition service from the face recognition service module; The face recognition service module instructs the face recognition control module to start the face recognition process; The face recognition control module calls the secure runtime environment TEE driver to start the face recognition trusted application TA on the side of the first secure virtual machine.

10. The method according to claim 9, wherein The first client application CA is a payment application or an unlocking application.

11. The method according to claim 9 or 10, characterized in that, After starting the face recognition trusted application TA on the side of the first secure virtual machine, the face recognition TA obtains the first face data from the first secure memory, obtains the face template data from the secure chip and decrypts the face template data, and compares the first face data with the decrypted face template data to obtain the face recognition result.

12. The method according to any one of claims 9 to 11, characterized in that, The main virtual machine further includes a second client application CA for setting face template data; the method further includes: In response to a second operation of the user on the second client application CA, the second client application CA requests a face enrollment service from the face recognition service module; The face recognition service module instructs the face recognition control module to start the face enrollment process; The face recognition control module calls the TEE driver to start the face recognition TA on the side of the first secure virtual machine.

13. The method according to claim 12, wherein After starting the face recognition TA on the side of the first secure virtual machine, the face recognition TA obtains the second face data collected by the camera hardware from the second secure memory and generates the face template data according to the feature values of the second face data.

14. The method according to any one of claims 9 to 13, characterized in that, The first secure virtual machine further includes a secure runtime environment TEE system service, and the TEE system service is correspondingly provided with a secure chip interface; The face recognition TA calls the secure runtime environment TEE system service and the secure chip interface, encrypts the face template data and stores it in the secure chip; and the face recognition TA calls the secure runtime environment TEE system service and the secure chip interface to obtain the face template data from the secure chip.

15. The method according to claim 14, wherein The main virtual machine further includes a first proxy service module, and the second secure virtual machine includes a second proxy service module; The main virtual machine sets the access permission of the first secure memory, including: The face recognition control module invokes the first proxy service and the second proxy service to set the access permission of the first secure memory; The face recognition control module invokes the Trusted Execution Environment (TEE) driver of the secure runtime environment and sends the information of the access permission of the first secure memory to the TEE system service of the secure runtime environment.

16. The method according to claim 15, wherein The first secure virtual machine obtains the first face data from the first secure memory, including: The face recognition Trusted Application (TA) invokes the TEE system service of the secure runtime environment to obtain the first face data from the first secure memory.

17. The method according to claim 15 or 16, characterized in that, The method further includes: When the camera hardware captures the first face data, the camera hardware invokes the camera driver on the main virtual machine side and the first proxy service to write the first face data into the first secure memory.

18. An electronic device, characterized in that, The electronic device includes: one or more processors, and a memory; The memory is coupled to the one or more processors. The memory is used to store computer program code, and the computer program code includes computer instructions. The one or more processors invoke the computer instructions to cause the electronic device to execute the method according to any one of claims 1 to 17.

19. A chip system, characterized in that, The chip system is applied to an electronic device. The chip system includes one or more processors, and the one or more processors are used to invoke computer instructions to cause the electronic device to execute the method according to any one of claims 1 to 17.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions. When the instructions run on an electronic device, the instructions cause the electronic device to execute the method according to any one of claims 1 to 17.

Images