Face recognition method, electronic device, chip system, and storage medium

Abstract

Embodiments of the present application provide a face recognition method, an electronic device, a chip system and a storage medium, and relate to the technical field of terminals. The method is implemented based on a REE+multi-TEE environment, face template data is stored in a secure chip, and the secure chip supports access of a first secure virtual machine on the TEE side and does not support access of a main virtual machine on the REE side. In a face recognition / authentication scenario, the electronic device creates a secure memory that can be accessed by the first secure virtual machine and is not allowed to be accessed by the main virtual machine. In the case that first face data collected by camera hardware is written into the secure memory, the first secure virtual machine obtains the first face data from the secure memory and reads face template data from the secure chip, and compares the first face data with the face template data to obtain a face recognition result. Since the Android side cannot access the secure memory and the secure chip, the face data can be prevented from being acquired and damaged, and the security of the face data is improved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a face recognition method, electronic device, chip system, and storage medium. Background Technology

[0002] Currently, facial recognition is widely used for identity authentication on electronic devices, such as in payment or unlocking scenarios. In facial recognition / authentication scenarios, electronic devices collect facial image data, which involves users' privacy and security information. Although electronic devices already offer facial recognition functionality, current facial recognition technology still carries the risk of users' facial image data being stolen or compromised. Summary of the Invention

[0003] This application provides a face recognition method, electronic device, chip system, and storage medium, which can improve the reliability and security of face recognition.

[0004] In a first aspect, embodiments of this application provide a face recognition method applied to an electronic device, the electronic device including a main virtual machine, a first secure virtual machine, and a second secure virtual machine; the method includes:

[0005] The main virtual machine receives a first operation from the user (the first operation is an operation that triggers the face recognition process); in response to the first operation, the main virtual machine creates a first secure memory and sets the access permissions for the first secure memory; the main virtual machine calls the camera HAL to trigger the camera hardware to start; wherein, the camera hardware of the electronic device has write permissions to the first secure memory, and the first secure virtual machine and the second secure virtual machine have read permissions to the first secure memory.

[0006] When the first face data captured by the camera hardware is written to the first secure memory, the first secure virtual machine retrieves the first face data from the first secure memory and retrieves face template data from the security chip of the electronic device; the first secure virtual machine compares the first face data with the face template data to obtain the face recognition result;

[0007] The first secure virtual machine has access to the security chip, while the main virtual machine does not. Access to the security chip includes writing data to and reading data from it.

[0008] The face recognition method provided in this application stores face template data in a secure chip. A first secure virtual machine has access to the secure chip, while the main virtual machine on the Android side does not. In a face recognition / authentication scenario, the electronic device creates secure memory and sets access permissions for it. The secure memory is accessible only to the secure virtual machine, not the main virtual machine. When the first face data captured by the camera hardware is written to the first secure memory, the first secure virtual machine retrieves the first face data from the first secure memory and reads the face template data from the secure chip. The first face data is then compared with the face template data to obtain the face recognition result. Because the Android side cannot access the secure memory or the secure chip, the acquisition and corruption of face data can be prevented, improving the reliability and security of the face data.

[0009] Secure memory refers to memory with security protection features. Secure memory is memory allocated to the TEE (Technical Equipment Environment) side for a secure operating environment. For example, an electronic device can allocate a portion of double-data-rate (DDR) synchronous dynamic random access memory as secure memory for caching data. In this embodiment, secure memory can be used to store raw images captured by a camera. The electronic device can extract facial feature values ​​based on these raw images and further generate facial template data based on these features. In face recognition / authentication scenarios, the electronic device can create secure memory and set access permissions, allowing the TEE side to access secure memory but disallowing the Android side, thus improving the security of stored data to a certain extent.

[0010] The security chip is a secure storage chip independent of the secure memory, employing specific encryption methods for data storage. For example, the security chip could be an SEC flash memory chip. SEC flash memory chips define special interfaces and have specific read / write timings, preventing bus attacks and data theft. This application's solution stores face data templates through a security chip. The security chip supports access from the secure operating environment (TEE) side but not from the Android side, thus improving the security of the stored data to a certain extent.

[0011] The main virtual machine runs in a general-purpose runtime environment (REE) (or a non-secure runtime environment REE). The first secure virtual machine runs in a first secure runtime environment TEE, or a self-developed TEE system. The second secure virtual machine runs in a second secure runtime environment TEE, or a native TEE system. The face recognition method provided in this application is implemented based on a multi-TEE environment, and this face recognition method adopts an REE + multi-TEE architecture. The secure virtual machine can also be called a trusted virtual machine.

[0012] The improvements of this application compared to the prior art include:

[0013] (1) Based on the existing multi-TEE architecture, the face recognition TA is migrated from the native TEE system to the self-developed TEE system, so as to make full use of the self-developed TEE system to improve various security features.

[0014] (2) Saving face template data to a secure chip improves the reliability and security of face data, preventing it from being obtained and destroyed by hackers.

[0015] In related technologies, when users save data using the secure storage function, the data is encrypted within the secure operating environment (TEE) and then saved to the relevant storage area on the REE side. For example, the file system SFS utilizes the file system on the non-secure operating environment (REE) side for storage services. Data encrypted on the secure operating environment (TEE) side is stored on the REE side's storage medium using the REE side's file system. Because the face template data is stored in the Android-side file system SFS, the Android side can access this file system SFS. Although the face template data is encrypted, this data storage method still carries the risk of being compromised.

[0016] Compared with related technologies that store face template data through the SFS file system, which is accessible on the Android side and thus has low security, this application uses independent secure storage hardware to store face template data and automatically encrypts the face template data when writing it to the secure chip. This secure chip can only be accessed by the TEE side and is not accessible by the Android side, thus greatly improving the security of the stored data.

[0017] The solution provided in this application can be applied to facial recognition / authentication scenarios, such as payment authentication or unlocking authentication scenarios, and can improve the security of facial authentication.

[0018] The face recognition result is either "face recognition / authentication successful" or "face recognition / authentication failed".

[0019] For example, in the unlocking authentication scenario, the decision to unlock is made based on the face recognition result; if the face recognition result is successful, unlocking is performed; if the face recognition result is unsuccessful, unlocking is not performed or an unlocking failure is displayed.

[0020] For example, in a payment authentication scenario, the decision to proceed with payment is based on the facial recognition result; if the facial recognition result is successful, payment is executed; if the facial recognition result is unsuccessful, payment is not executed or a payment failure message is displayed.

[0021] For example, in unlocking / payment authentication scenarios, the system determines whether to perform face registration based on the face recognition result; if the face recognition result is successful, face registration is performed; if the face recognition result is unsuccessful, face registration is not performed or a face registration failure is displayed.

[0022] For example, in unlocking / payment authentication scenarios, the system determines whether to perform registration or login based on the facial recognition result; if the facial recognition result is successful, registration or login is performed; if the facial recognition result is unsuccessful, registration or login is not performed or a registration or login failure is displayed.

[0023] This application can be applied to facial recognition during payments or transfers (e.g., when users make payments or transfers in payment apps / financial apps / chat apps / shopping apps), or to facial security verification for users during secure registration or login to applications. This application does not limit the scope of the application.

[0024] In one possible implementation, before the main virtual machine receives the user's first operation, the method further includes: the main virtual machine receiving the user's second operation (the second operation being an operation that triggers the face registration process); the main virtual machine responding to the second operation by creating a second secure memory and setting access permissions for the second secure memory; the camera hardware having write permissions to the second secure memory, and the first and second secure virtual machines having read permissions to the second secure memory; the main virtual machine calling the camera HAL to trigger the camera hardware to start; when the second face data collected by the camera hardware is written to the second secure memory, the first secure virtual machine obtaining the second face data from the second secure memory and generating the face template data based on the feature values ​​of the second face data; the first secure virtual machine encrypting the face template data and storing it in the secure chip.

[0025] In one possible implementation, after the first secure virtual machine encrypts the face template data and stores it in the secure chip, the method further includes: the main virtual machine calling the camera HAL to trigger the camera hardware to shut down and triggering the release of the second secure memory. After the face registration process is completed, the camera hardware is shut down and the secure memory is released to save resources.

[0026] In one possible implementation, after the first secure virtual machine obtains the face recognition result, the method further includes: the first secure virtual machine sending the face recognition result to the main virtual machine; the main virtual machine receiving the face recognition result and performing face recognition services based on the face recognition result, the face recognition services including face payment or face unlock.

[0027] In one possible implementation, after the main virtual machine receives the face recognition result, the method further includes: the main virtual machine calling the camera HAL to trigger the camera hardware to shut down and triggering the release of the first safe memory. After the face recognition process is completed, the camera hardware is shut down and the safe memory is released to save resources.

[0028] In one possible implementation, the main virtual machine creates the first secure memory, which involves the main virtual machine invoking the camera HAL to create the first secure memory based on the electronic device's runtime memory. This runtime memory can be provided by DDR.

[0029] In one possible implementation, the first secure virtual machine obtains face template data from the security chip of the electronic device, including: the first secure virtual machine calling the security chip interface to obtain encrypted face template data from the security chip; and the first secure virtual machine decrypting the face template data.

[0030] In one possible implementation, the first secure virtual machine includes a face recognition trusted application (TA); the main virtual machine includes a first client application (CA), a face recognition service module, a face recognition control module, and a secure operating environment (TEE) driver.

[0031] In this case, the method further includes: in response to the first operation performed by the user on the first client application CA, the first client application CA requests a face recognition service from the face recognition service module; the face recognition service module instructs the face recognition control module to start the face recognition process; the face recognition control module calls the secure operating environment TEE driver to start the face recognition trusted application TA on the first secure virtual machine side.

[0032] Through the proposed solution, Trusted Application (TA) can provide security services such as key generation and management, security authentication, and facial recognition for client application CAs running on REE, thereby ensuring the security of user data.

[0033] In one possible implementation, the first client application CA is a payment application or an unlocking application, and the unlocking application can be an application used for application lock or screen unlock.

[0034] In one possible implementation, after the face recognition trusted application TA is started on the first secure virtual machine side, the face recognition TA obtains the first face data from the first secure memory, obtains the face template data from the secure chip and decrypts the face template data, and compares the first face data with the decrypted face template data to obtain the face recognition result.

[0035] In one possible implementation, the main virtual machine further includes a second client application CA, which is used to set face template data. In this case, the method further includes: in response to a second operation by the user on the second client application CA, the second client application CA requests a face enrollment service from the face recognition service module; the face recognition service module instructs the face recognition control module to start the face enrollment process; and the face recognition control module calls the TEE driver to start the face recognition TA on the first secure virtual machine side.

[0036] In one possible implementation, after the face recognition TA is started on the first secure virtual machine side, the face recognition TA obtains the second face data captured by the camera hardware from the second secure memory, and generates face template data based on the feature values ​​of the second face data.

[0037] In one possible implementation, the first secure virtual machine also includes a secure runtime environment (TEE) system service, which is equipped with a secure chip interface.

[0038] Specifically, the face recognition TA calls the secure operating environment TEE system service and the secure chip interface to encrypt the face template data and store it in the secure chip; and the face recognition TA calls the secure operating environment TEE system service and the secure chip interface to obtain the face template data from the secure chip.

[0039] In one possible implementation, the main virtual machine further includes a first proxy service module, and the second security virtual machine includes a second proxy service module.

[0040] In this case, the main virtual machine sets the access permissions for the first secure memory, including: the face recognition control module calls the first agent service and the second agent service to set the access permissions for the first secure memory; the face recognition control module calls the secure runtime environment TEE driver to send the access permission information for the first secure memory to the secure runtime environment TEE system service.

[0041] In one possible implementation, the first secure virtual machine obtains the first face data from the first secure memory, including: the face recognition TA calls the secure runtime environment (TEE) system service to obtain the first face data from the first secure memory.

[0042] In one possible implementation, the method further includes: when the camera hardware acquires the first face data, the camera hardware calls the camera driver on the main virtual machine side and the first proxy service to write the first face data into the first secure memory.

[0043] Secondly, this application provides a chip system including one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via lines. The aforementioned chip system can be applied to electronic devices including communication modules and memory. The interface circuits are used to receive signals from the memory of the electronic device and send the received signals to the processor, the signals including computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device can perform the methods described in the first aspect and any of its possible design embodiments.

[0044] Thirdly, this application provides a computer-readable storage medium including computer instructions. When the computer instructions are executed on an electronic device (such as a mobile phone), they cause the electronic device to perform the methods described in the first aspect and any of its possible design embodiments.

[0045] Fourthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the method described in the first aspect and any possible design thereof.

[0046] Fifthly, embodiments of this application provide a face recognition device, including a processor and a memory coupled together. The memory stores program instructions, which, when executed by the processor, cause the device to implement the method described in the first aspect and any possible design of the above. The device may be an electronic device or a server device; or it may be a component of an electronic device or a server device, such as a chip.

[0047] Sixthly, embodiments of this application provide a face recognition device, which can be divided into different logical units or modules according to function, each unit or module performing different functions, so that the device performs the method described in the first aspect and any possible design method described above.

[0048] It is understood that the beneficial effects achieved by the chip system described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the apparatus described in the fifth and sixth aspects can be referred to the beneficial effects in the first aspect and any of its possible design embodiments, which will not be repeated here. Attached Figure Description

[0049] Figure 1 A schematic diagram of a face recognition scenario provided in an embodiment of this application;

[0050] Figure 2 A schematic diagram illustrating another face recognition scenario provided in an embodiment of this application;

[0051] Figure 3A A schematic diagram of the architecture of the REE and a single TEE provided for embodiments of this application;

[0052] Figure 3B A schematic diagram of the architecture of the REE and a single TEE provided for embodiments of this application;

[0053] Figure 4 A schematic diagram of the architecture of the REE and multiple TEEs provided in the embodiments of this application;

[0054] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0055] Figure 6 A schematic diagram of a software module architecture provided in an embodiment of this application;

[0056] Figure 7 This is a schematic diagram of the system framework of the face recognition method provided in the embodiments of this application;

[0057] Figure 8 This is a schematic diagram of the signal interaction of the face recognition method provided in the embodiments of this application;

[0058] Figure 9 This is a schematic diagram of the signal interaction of the face recognition method provided in the embodiments of this application;

[0059] Figure 10 This is a schematic diagram of the signal interaction of the face recognition method provided in the embodiments of this application;

[0060] Figure 11 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0061] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. In the description of this application, unless otherwise stated, "at least one" refers to one or more, and "more than one" refers to two or more. Furthermore, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0062] Facial recognition technology is a biometric identification technology based on facial feature information. For example, the implementation principle of facial recognition technology is roughly as follows: Facial feature information is extracted from image data containing a face. Then, the extracted facial feature information is compared with the facial feature information corresponding to pre-configured identity information. If the matching degree reaches a preset standard, the face in the image data is determined to belong to the person indicated by the aforementioned identity information, which can be called successful facial recognition. If the matching degree does not reach the preset standard, the face in the image data is determined not to belong to the person indicated by the aforementioned identity information, which can be called failed facial recognition. Specific details can be found in related technologies and will not be elaborated here.

[0063] The solution provided in this application is applied to facial recognition / authentication scenarios, such as payment authentication or unlocking authentication scenarios, and aims to improve the security of facial authentication.

[0064] For example, consider a scenario where a user uses an electronic device to transfer funds. Figure 1 As shown, after the electronic device detects the user's instruction to transfer funds, it displays a transfer interface 11. During the display of the transfer interface 11, the electronic device can receive transfer information entered by the user (e.g., recipient account, transfer amount, etc.). Subsequently, in response to the user's instruction to confirm the transfer, such as clicking the "Next" control 12 in the transfer interface 11, the electronic device activates its camera, begins collecting facial image data, and displays a facial recognition window 13, prompting the user to participate in facial recognition.

[0065] The above scenario is a payment authentication scenario. If facial recognition / authentication is successful, the payment will be successful; if facial recognition / authentication fails, the payment will fail.

[0066] To illustrate further, consider the scenario of a user unlocking an electronic device's screen. For example... Figure 2 As shown, the electronic device is in a locked state. Assuming the user has enabled face unlock, when the user picks up the electronic device for face recognition, the device can display the lock screen interface 21 in response to the user picking up the phone. During face recognition, the electronic device captures facial image data through its camera, and the device can display an unlock icon 22 and the prompt text "Recognizing face" 23 on the lock screen interface 21.

[0067] The above scenario describes the unlocking and authentication process. If facial recognition / authentication is successful, the screen will unlock successfully; if facial recognition / authentication fails, the screen will unlock unsuccessfully.

[0068] In the aforementioned facial recognition / authentication scenarios, electronic devices collect facial image data. This facial image data involves users' privacy and security information. Although electronic devices have provided facial recognition functionality, current facial recognition technology still carries the risk of users' facial image data being stolen or destroyed.

[0069] Currently, electronic devices adopt Figure 3A The architecture shown isolates and protects facial image data to prevent it from being stolen or corrupted. For example... Figure 3A As shown, the architecture consists of a general runtime environment (REE) and a secure runtime environment (TEE).

[0070] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the relevant concepts or technologies is given first:

[0071] A rich execution environment (REE), also known as a general execution environment, a standard execution environment, or an insecure execution environment, refers to the system runtime environment for mobile devices, in which operating systems such as Android, iOS, and Linux can run. REEs offer good openness and scalability but have relatively low security.

[0072] A trusted execution environment (TEE), also known as a secure side or secure zone, is an area that requires authorization to access. A TEE has its own runtime space with strict protection measures defined, and only authorized secure software can execute within the TEE.

[0073] In comparison, an REE (Reliable Execution Environment) is an open environment vulnerable to attacks such as the theft of sensitive data and mobile payment fraud; while a TEE (Trusted Application Environment) is a secure zone on the central processing unit that ensures sensitive data is processed in an isolated and trusted environment, thus protecting it from software attacks originating from the REE. Furthermore, compared to other secure operating environments, a TEE can protect the integrity and confidentiality of trusted applications (TAs) end-to-end, providing stronger processing power and larger memory space.

[0074] The REE+TEE architecture refers to an architecture where TEE and REE work together to provide services to applications. In other words, TEE and REE coexist within the electronic device. TEE, through hardware support, can achieve an isolated operating mechanism from REE. Due to TEE isolation and access control mechanisms, data and resource security are better protected.

[0075] A trusted application (TA) is an application that runs within a TEE (Transaction Execution Environment) and provides security services to client applications (CAs) running outside the TEE, such as password input, transaction signature generation, and facial recognition.

[0076] A client application (CA) is an application that runs within the REE. The CA can invoke the TA (Task Agent) and instruct it to perform corresponding security operations through the client's application programming interface (API).

[0077] TEE secure storage is used to store sensitive user data such as keys. When users save data using the secure storage function, the data is encrypted within the TEE and then saved to the relevant storage area on the REE side. Currently, the commonly used TEE secure storage is SFS secure storage. This SFS secure storage utilizes the REE-side file system for storage services. The data encrypted by the TEE is stored on the REE-side storage medium using a non-secure file system. Therefore, it has a large capacity and can store many files. However, because it uses a non-secure file system, the stored files are visible to the REE side, resulting in lower security.

[0078] Refer again Figure 3A As shown, the system architecture includes a main virtual machine (VM) and a secure virtual machine (TVM). The VM is used in the general runtime environment (REE) and runs the Android system. Client applications (CAs) run on the VM side; for example, client applications (CAs) can be payment apps, lock screen apps, or app lock apps that require facial recognition for payment or unlocking. The TVM is used in the secure runtime environment (TEE), and trusted applications (TAs) run on the TVM side. Figure 3A As shown, the client application CA and trusted application TA run at EL0 level, the system kernel and TEE kernel run at EL1 level, the hypervisor runs at EL2 level, and the secure monitor runs at EL3 level. The trusted application TA provides security services such as key generation and management, security authentication, and facial recognition to the client application CA running on the REE. This architecture design achieves an isolation mechanism between the TEE and the REE, ensuring the security of user data.

[0079] Figure 3B Based on Figure 3A A schematic diagram of the system architecture implemented for face recognition / authentication scenarios.

[0080] On the Android side, a client application CA (or face recognition CA) runs. The client application CA calls the camera HAL to open the camera, set the security mode, generate secure memory, and set usage permissions for the camera hardware and TEE system. For example, the client application CA can be a payment application or an unlocking application.

[0081] On the TEE side, a Trusted Application (TA) for Face Recognition (or Face Recognition TA) runs. The Face Recognition TA can perform the following two tasks:

[0082] (1) Face data entry: In the face data entry mode, the face recognition TA calls the TEE HAL to access the secure memory, obtains the image captured by the camera, extracts the face data feature value based on the image captured by the camera, and then forms the face template data based on the face data feature value. Then the face template data is saved to the file system SFS (file system).

[0083] (2) Face recognition / authentication: In face authentication mode, the face recognition TA calls the TEE HAL to access secure memory, obtains the image captured by the camera, extracts the face feature value based on the image captured by the camera, and then compares the extracted face feature value with the face template data stored in the file system SFS.

[0084] As can be seen, in face recognition / authentication scenarios, running face recognition TAs and face authentication TAs within the TEE TVM can provide security services such as face recognition for client applications (CAs) running on the REE. However, because face template data is stored in the Android-side file system SFS, the Android side can access this SFS file system. Although the face template data is encrypted, this data storage method still carries the risk of being compromised.

[0085] To address the aforementioned issues, this application provides a face recognition method based on a multi-TEE environment, referencing... Figure 4 As shown, this face recognition method adopts an REE+ multi-TEE architecture, which includes two TEE systems on the TEE side: a self-developed TEE system (referred to as the first TEE) and a native TEE system (referred to as the second TEE). The first TEE uses a first secure virtual machine (referred to as TVM1), and the second TEE uses a first secure virtual machine (referred to as TVM2). The secure virtual machine can also be referred to as a trusted virtual machine.

[0086] In a multi-TEE environment, a hardware-level encrypted storage solution is provided based on a self-developed TEE system. (Reference) Figure 4As shown, this application uses independent secure storage hardware (called the secure chip secflash) to store face template data. The face template data is written to the secure chip based on the self-developed TEE system and is automatically encrypted during writing. The secure chip can only be accessed by the TEE side and cannot be accessed by the Android side.

[0087] The improvements of this application compared to the prior art include:

[0088] (1) Based on the existing multi-TEE architecture, the face recognition TA is migrated from the native TEE system to the self-developed TEE system, so as to make full use of the self-developed TEE system to improve various security features.

[0089] (2) Saving face template data to a secure chip improves the reliability and security of face data, preventing it from being obtained and destroyed by hackers.

[0090] Compared with related technologies that store face template data through the SFS file system, which is accessible on the Android side and thus has low security, this application uses independent secure storage hardware to store face template data and automatically encrypts the face template data when writing it to the secure chip. This secure chip can only be accessed by the TEE side and is not accessible by the Android side, thus greatly improving the security of the stored data.

[0091] The electronic device described in this application embodiment can be a mobile phone, a personal digital assistant (PDA), a tablet computer, or other smart devices. An insecure operating environment and a secure operating environment can be deployed on the electronic device. The insecure operating environment is the REE (Remote Execution Environment) on the electronic device, running operating systems such as Android, iOS, and Windows Phone; the secure operating environment is the TEE (Trusted Execution Environment), running a secure operating system. The software and hardware resources accessed by the TEE are isolated from the REE. The software and hardware resources on the electronic device can be identified into two execution environment states. Software and hardware resources identified as being in a secure execution state can only be accessed by the TEE execution environment, while software and hardware resources identified as being in a non-secure execution state can be accessed by both execution environments. The TEE constructs a secure operating environment isolated from the REE, providing a secure execution environment for authorized trusted software.

[0092] Figure 5 This is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of this application.

[0093] like Figure 5As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0094] The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0095] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0096] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0097] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0098] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0099] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0100] It is understood that the interface connection relationships between the modules illustrated in this embodiment are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0101] The charging management module 140 receives charging input from the charger. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0102] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0103] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0104] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network.

[0105] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1.

[0106] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194.

[0107] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0108] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0109] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0110] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), a light-emitting diode (LED), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0111] Electronic device 100 can perform shooting functions through an ISP, camera 193, video codec, GPU, display screen 194, and application processor. The ISP processes data fed back from the camera 193. The camera 193 captures still images or video. The digital signal processor processes digital signals, including digital image signals and other digital signals. The video codec compresses or decompresses digital video. Electronic device 100 can support one or more video codecs. Thus, electronic device 100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0112] Camera 193 may include 1 to N cameras. For example, an electronic device may include 2 front-facing cameras and 4 rear-facing cameras. Among them, the front-facing cameras may include TOF cameras. TOF cameras include complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensors.

[0113] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0114] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to achieve data storage functionality. For example, music, video, and other files can be saved on the external memory card. The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. For example, in this embodiment, the processor 110 can execute instructions stored in the internal memory 121, which may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0115] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0116] Audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Audio module 170 can also be used for audio signal encoding and decoding. Speaker 170A, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. Receiver 170B, also called a "handset," is used to convert audio electrical signals into sound signals. Microphone 170C, also called a "microphone" or "microphone unit," is used to convert sound signals into electrical signals. Headphone jack 170D is used to connect wired headphones.

[0117] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control. Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with electronic device 100. Electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc.

[0118] The methods described in the following embodiments can all be implemented in the electronic device 100 having the above-described hardware structure.

[0119] The software system of the aforementioned electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of the electronic device 100.

[0120] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through interfaces. In some embodiments, such as... Figure 6 As shown, the Android system may include an application layer, an application framework layer, an Android runtime and system libraries, a hardware abstraction layer (HAL), and a kernel layer. It should be noted that this application uses the Android system as an example; however, the solution of this application can also be implemented in other operating systems (such as HarmonyOS, iOS, etc.) as long as the functions implemented by each functional module are similar to those in the embodiments of this application.

[0121] The application layer can include a series of application packages. For example... Figure 6 As shown, the application package may include camera apps, settings apps, payment apps, lock screen apps, and other applications used in facial recognition scenarios. Of course, applications for facial recognition scenarios may also include other application packages, such as shopping apps, banking apps, chat apps, or financial management apps; this application does not limit the scope.

[0122] The settings application includes a function to register a facial image, which is used for scenarios such as facial payment or facial unlocking. The lock screen application has the function of unlocking in response to the user's unlocking operation (e.g., pressing the power button). The lock screen application can perform unlocking processes such as facial unlocking, fingerprint unlocking, and password unlocking. This application embodiment uses facial payment as an example for illustration.

[0123] In this embodiment, the settings application, payment application, lock screen application, etc., are client applications for the face recognition scenario, i.e., face recognition CA. Correspondingly, the applications in the TEE environment include face recognition TA.

[0124] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. These may include, for example, an activity manager, a window manager, a content provider, a view system, a resource manager, a notification manager, a camera service, and a face recognition service, etc., though this application embodiment does not impose any limitations on these.

[0125] A system library can include multiple functional modules. Examples include: surface manager, media libraries, OpenGL ES, SGL, etc.

[0126] The Surface Manager is used to manage the display subsystem and provides the fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0127] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0128] OpenGL ES is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0129] SGL is a 2D graphics engine.

[0130] The Android Runtime comprises the core libraries and the virtual machine. The Android Runtime is responsible for scheduling and managing the Android system. The core libraries consist of two parts: one part contains the functionalities that Java calls, and the other part contains the core Android libraries. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0131] The HAL layer is a wrapper around Linux kernel drivers, providing interfaces to the upper layers and shielding them from the implementation details of the lower-level hardware.

[0132] The HAL layer can include Wi-Fi HAL, audio HAL, camera HAL, and face recognition control module, etc.

[0133] The Camera HAL is the core software framework of the Camera, which can include sensor nodes and image front end (IFE) nodes. Sensor nodes and IFE nodes are components (nodes) in the image data and control command transmission path (also known as the transmission pipeline) created by the Camera HAL.

[0134] The face recognition control module is the core software framework / application for face recognition. It can interact with the face recognition service at the application framework layer, the camera HAL at the HAL layer, and the TEE driver at the kernel layer.

[0135] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0136] Among them, the camera driver is the driver layer of the camera device, which is mainly responsible for interacting with the hardware.

[0137] The hardware layer includes displays, cameras, secure memory (Secure Buffer), and security chips, among others.

[0138] Secure memory refers to memory with security protection features. Secure memory is memory allocated to the TEE side. For example, an electronic device can allocate a portion of double data rate synchronous dynamic random access memory (DDR) as secure memory for caching data.

[0139] The security chip is a secure storage chip independent of the secure memory, employing specific encryption methods for data storage. The security chip can be an SEC flash memory chip. SEC flash memory chips define special interfaces and have specific read / write timings, preventing bus attacks and data theft. Other possible chips can also be used, such as replay-protected memory blocks (RPMBs). In practical applications, RPMBs are typically used to store data requiring protection against unauthorized tampering, such as public keys and serial numbers related to fingerprint payments on mobile phones.

[0140] This application uses the SEC flash memory chip as an example for illustrative purposes.

[0141] In this embodiment, secure memory can be used to store raw images captured by the camera. The electronic device can extract facial feature values ​​based on these raw images and further generate facial template data based on these feature values. In face recognition / authentication scenarios, the electronic device can create secure memory and set access permissions for it, allowing the TEE side to access the secure memory while disallowing the Android side, thus improving the security of stored data to a certain extent.

[0142] In this embodiment, the security chip can be used to store face template data. The face template data is encrypted using a specific encryption method, and the security chip can only be accessed by the TEE side, while the Android side cannot access it, thus greatly improving the security of the stored data.

[0143] The following describes the software modules and interactions between modules involved in the face recognition method provided in the embodiments of this application.

[0144] Figure 7 A schematic diagram of the system architecture adopted by the overall solution of this application is shown. For example... Figure 7 As shown, the REE side (main virtual machine) includes settings application, payment application / unlock application, face recognition SDK (software development kit), face recognition service, face recognition control module, camera service, camera HAL, camera driver, TEE driver, proxy service, etc.

[0145] The face recognition control module can interact with the face recognition TA on the first TEE side (secure virtual machine TVM1) through the TEE driver.

[0146] The camera HAL can include sensor nodes and IFE nodes. Sensor nodes can interact with the camera driver in the kernel layer, and the camera driver can be used to drive the camera in the hardware layer to acquire image data.

[0147] The first TEE side (Secure Virtual Machine TVM1) includes a face recognition TA and TEE system services. The TEE system services include a security chip interface, through which the first TEE side (Secure Virtual Machine TVM1) can access the security chip.

[0148] The second TEE side (Secure Virtual Machine TVM2) includes proxy services.

[0149] It should be noted that, Figure 7 The system framework described herein is illustrative. In actual implementation, the REE side (main virtual machine), the first TEE side (security virtual machine TVM1), and the second TEE side (security virtual machine TVM2) may include more or fewer modules, and this application embodiment does not limit this. The main improvement of this application solution lies in the first TEE side (security virtual machine TVM1).

[0150] like Figure 7 As shown, the system architecture also includes a virtual machine manager. The virtual machine manager manages the lifecycle of the main virtual machine on the REE side, the security virtual machine TVM1 on the first TEE side, and the security virtual machine TVM2 on the second TEE side.

[0151] The system architecture also includes a resource manager, which manages system resources, as well as access permissions for secure virtual machines TVM1, TVM2, and camera hardware to secure memory.

[0152] The system architecture also includes hardware such as cameras, image preprocessing (IFE) modules, secure memory, and security chips.

[0153] The Image Preprocessing (IFE) module stores the image data captured by the camera in secure memory. The storage location of the image data in secure memory can be represented by a file descriptor (FD).

[0154] like Figure 7 As shown, the payment application and settings application in the application layer can interact with the face recognition SDK. The face recognition SDK interacts with the face recognition service in the framework layer by calling the preset application programming interface (API). The face recognition service can interact with the face recognition control module in the HAL layer. The face recognition control module can interact with the camera HAL in the HAL layer through the camera service in the framework layer, or the face recognition control module can interact directly with the camera HAL in the HAL layer.

[0155] The overall process of the face recognition method provided in this application embodiment will be described in stages below. The overall process of the face recognition method includes a face template data input process (hereinafter referred to as the face input process) and a face authentication process.

[0156] Phase 1: Face Enrollment Process

[0157] In the face registration process, the settings application in the application layer interacts with the face recognition service through the face recognition SDK.

[0158] The user initiates the face registration process through the settings application. The face recognition control module sends a command to the first TEE to load the face recognition trusted application TA (referred to as face recognition TA) on the first TEE side.

[0159] The face recognition control module calls camera middleware services (such as camera service, camera HAL, and camera driver) to start the camera. The camera HAL triggers the creation of a secure memory block to cache image data captured by the camera. Access permissions for this secure memory are set via proxy services on the Android side and the second TEE side, allowing both TVM1 and TVM2 of the first TEE to access it. Once the camera is started, only TVM1 and TVM2 of the first and second TEEs can access the secure memory; the Android side is not allowed to access it.

[0160] After receiving notification that the camera hardware image is ready, the face recognition control module instructs the face recognition TA to acquire face data from secure memory. The face recognition TA extracts facial feature values ​​based on the face data and generates face template data accordingly. Then, the face recognition TA saves the face template data to the secure chip via the secure chip interface provided by the first TEE.

[0161] The face recognition control module shuts down the camera and revokes access to the secure memory. The face registration process is now complete.

[0162] Phase Two: Facial Recognition Process

[0163] In the facial recognition process, the payment / unlocking application in the application layer interacts with the facial recognition service through the facial recognition SDK.

[0164] When a user triggers face recognition / authentication services such as phone unlocking / app lock or third-party Android applications (payment applications), the face authentication process is initiated. The face recognition control module sends a command to the first TEE, initiating the face recognition TA within the first TEE.

[0165] The face recognition control module calls camera middleware services (such as camera service, camera HAL, and camera driver) to start the camera. The camera HAL triggers the creation of a secure memory block to cache image data captured by the camera. Access permissions for this secure memory are set via proxy services on the Android side and the second TEE side, allowing both TVM1 and TVM2 of the first TEE to access it. Once the camera is started, only TVM1 and TVM2 of the first and second TEEs can access the secure memory; the Android side is not allowed to access it.

[0166] After receiving notification that the camera hardware image is ready, the face recognition control module instructs the face recognition TA to retrieve face data from secure memory. The face recognition TA then extracts facial feature values ​​based on the face data. The face data is then read from the secure chip via the secure chip interface provided by the first TEE, and the face template data is compared with the real-time acquired face data. If the comparison result meets the preset values, face recognition / authentication succeeds. If the comparison result does not meet the preset values, face recognition / authentication fails.

[0167] The face recognition control module shuts down the camera and revokes access to the secure memory. The face recognition / authentication process ends.

[0168] For ease of understanding, the methods provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0169] Face registration process

[0170] exist Figure 7 Based on the framework diagram, Figure 8 The following is an illustration of the face registration process in the face recognition method provided in this application embodiment:

[0171] S101. In response to user operation, the settings application triggers the start of the face registration process.

[0172] The settings application can call the face recognition SDK to register faces.

[0173] S102. The application sends a message to the face recognition service, instructing it to start the face enrollment process. Then, the face recognition service sends a message to the face recognition control module, instructing it to start the face enrollment process.

[0174] S103. The face recognition control module sends a message to the secure virtual machine TVM1, instructing that the face recognition TA be loaded on the secure virtual machine TVM1 side.

[0175] S104, Secure Virtual Machine TVM1 loads face recognition TA.

[0176] Correspondingly, the secure virtual machine TVM1 can return a message to the face recognition control module that the face recognition TA has been loaded.

[0177] S105 and S106, the face recognition control module calls the camera HAL to create secure memory.

[0178] For example, electronic devices may call the camera's HAL to allocate a portion of memory from DDR as safe memory.

[0179] Accordingly, the memory device sends a message to the camera HAL that the safe memory has been created.

[0180] S107. The camera HAL calls the agent services of the main virtual machine VM and the security virtual machine TVM2 to set access permissions for the secure memory. The security virtual machines TVM1, TVM2 and the camera hardware have permission to access the secure memory, while the main virtual machine VM does not have permission to access the secure memory.

[0181] S108, Safe Memory reports a message that the permissions for safe memory have been set.

[0182] The message that secure memory permissions have been set includes: secure virtual machines TVM1, TVM2, and camera hardware have permission to access secure memory.

[0183] Access permissions include write permissions and read permissions.

[0184] Camera hardware has write permissions. For example, camera hardware (camera) can write facial data to secure memory.

[0185] Secure virtual machines TVM1 and TVM2 have read permissions. For example, secure virtual machine TVM1 can read face data from secure memory.

[0186] The message is transmitted to the camera hardware, the agent service of TVM2, the agent service of the main virtual machine VM, and the TEE system service of the secure virtual machine TVM1.

[0187] The above steps are the preparation stage of the face registration process. The following steps are the specific execution stage of the face registration process.

[0188] S109. The face recognition control module sends a message to the secure virtual machine TVM1, instructing the face recognition TA to execute the face enrollment process.

[0189] The face recognition control module can interact with the secure virtual machine TVM1 on the first TEE side through the TEE driver on the main virtual machine (VM) side. Specifically, the face recognition control module can call the TEE driver to start the face recognition TA (Task Activation).

[0190] S110 and S111, the face recognition control module calls the camera HAL to start the camera hardware (camera).

[0191] The face recognition control module calls the camera HAL, which in turn calls the camera driver module, which then triggers the camera hardware (camera) to start.

[0192] After the camera hardware (camera) is powered on, it begins to capture images.

[0193] S112, The camera hardware (camera) sends a message to the camera HAL indicating that the camera hardware is ready, and then the camera HAL sends the message to the face recognition control module indicating that the camera hardware is ready.

[0194] S113. The camera hardware (camera) collects facial data and writes the collected facial data into secure memory.

[0195] As mentioned above, the camera has the authority to write the captured images into secure memory. Therefore, after the camera captures a face image, it writes the face image into secure memory.

[0196] Among them, facial data refers to facial images. For example, a facial image can be a 3D facial image.

[0197] The camera is a TOF sensor or other 3D sensor.

[0198] For example, this application may employ a camera based on time-of-flight (TOF) imaging technology, through which 3D facial image features can be acquired.

[0199] Time-of-Flight (TOF) imaging technology involves emitting a beam of infrared light (or laser pulses) invisible to the human eye. This light reflects off an object and reaches a camera. The time difference or phase difference between emission and reflection is calculated and collected to form a set of distance and depth data, thus creating a 3D model. In other words, TOF imaging adds depth information from the Z-axis to traditional 2D XY-axis imaging, ultimately generating a 3D image. Because TOF images are unaffected by most ambient light, applying them to payment / unlocking processes can improve the reliability of facial recognition.

[0200] S114, the secure memory sends a message to the camera hardware (camera) that the face data has been written to the secure memory, and then the camera hardware (camera) sends a message to the face recognition TA that the face data has been written to the secure memory.

[0201] S115, Face recognition TA requests face data from secure memory.

[0202] In this process, the face recognition TA calls the TEE system service of the secure virtual machine TVM1 to request face data from the secure memory.

[0203] S116. Secure memory returns face data to the face recognition TA.

[0204] Specifically, the secure memory calls the TEE system service of the secure virtual machine TVM1 to return face data to the face recognition TA.

[0205] S117. Face recognition TA extracts facial feature values ​​based on facial data collected by the camera and generates face template data based on the facial feature values.

[0206] S118 and S119, the face recognition TA calls the secure chip interface of the secure virtual machine TVM1 to encrypt the face template data and store it in the secure chip.

[0207] In this embodiment, the face recognition TA encrypts the face template data and then stores the encrypted face template data in a secure chip. The data encryption algorithm can be a preset encryption algorithm, such as a symmetric encryption algorithm, which can be the Advanced Encryption Standard (AES) 256 algorithm.

[0208] S120, the security chip sends a message to the face recognition TA that the face registration has been completed, and then the face recognition TA sends a message to the face recognition control module that the face registration has been completed.

[0209] S121 and S122, the face recognition control module calls the camera HAL to shut down the camera hardware (camera).

[0210] Accordingly, the camera hardware (camera) can send a message to the camera HAL that the camera is turned off, and then the camera HAL sends a message to the face recognition control module that the camera hardware (camera) is turned off.

[0211] S123. The face recognition control module calls the camera HAL to revoke access permissions to secure memory. Specifically, the camera HAL calls the agent service of the main virtual machine VM and the agent service of the secure virtual machine TVM2 to revoke access permissions of secure virtual machines TVM1 and TVM2 to secure memory.

[0212] S124 and S125, the face recognition control module calls the camera HAL and releases safe memory.

[0213] Furthermore, the face recognition control module can call the TEE driver to disable face recognition TA.

[0214] The face registration process is complete.

[0215] Facial recognition / authentication process

[0216] Figure 9 This illustration shows the face recognition / authentication process in the face recognition method provided in this application embodiment, combined with... Figure 8 ,like Figure 9 As shown, after S119 (the security chip has stored the face template data), this application also includes the following process:

[0217] S201. In response to user operation, the settings application triggers the start of the face registration process.

[0218] The settings application can call the face recognition SDK to perform face recognition.

[0219] S202. The application sends a message to the face recognition service, instructing it to start the face recognition process. Then, the face recognition service sends a message to the face recognition control module, instructing it to start the face recognition process.

[0220] S203. The face recognition control module sends a message to the secure virtual machine TVM1, instructing the face recognition TA to be started on the secure virtual machine TVM1 side.

[0221] S204, Start Face Recognition TA on the secure virtual machine TVM1 side.

[0222] Correspondingly, the secure virtual machine TVM1 can return a message to the face recognition control module that face recognition TA has been started.

[0223] S205 and S206, the face recognition control module calls the camera HAL to create secure memory.

[0224] For example, electronic devices may call the camera's HAL to allocate a portion of memory from DDR as safe memory.

[0225] Accordingly, the memory device sends a message to the camera HAL that the safe memory has been created.

[0226] S207. The camera HAL calls the agent services of the main virtual machine VM and the security virtual machine TVM2 to set access permissions for the secure memory. The security virtual machines TVM1, TVM2 and the camera hardware have permission to access the secure memory, while the main virtual machine VM does not have permission to access the secure memory.

[0227] S208, Safe Memory reports a message that the permissions for safe memory have been set.

[0228] The message indicating that secure memory permissions have been configured includes: Secure virtual machines TVM1 and TVM2, and the camera hardware have access to secure memory. These access permissions include write and read permissions. The camera hardware has write permissions. The camera hardware (camera) can write face data to secure memory. Secure virtual machines TVM1 and TVM2 have read permissions. Secure virtual machine TVM1 can read face data from secure memory.

[0229] The message is transmitted to the camera hardware, the agent service of TVM2, the agent service of the main virtual machine VM, and the TEE system service of the secure virtual machine TVM1.

[0230] The above steps are the preparation stage of the face recognition / authentication process. The following steps are the specific execution stage of the face recognition / authentication process.

[0231] S209. The face recognition control module sends a message to the secure virtual machine TVM1, instructing the face recognition TA to execute the face enrollment process.

[0232] S210 and S211, the face recognition control module calls the camera HAL to start the camera hardware (camera).

[0233] The face recognition control module calls the camera HAL, which in turn calls the camera driver module, which then triggers the camera hardware (camera) to start.

[0234] After the camera hardware (camera) is powered on, it begins to capture images.

[0235] S212, The camera hardware (camera) sends a message to the camera HAL indicating that the camera hardware is ready, and then the camera HAL sends the message to the face recognition control module indicating that the camera hardware is ready.

[0236] S213. The camera hardware (camera) collects facial data and writes the collected facial data into secure memory.

[0237] S214, the secure memory sends a message to the camera hardware (camera) that the face data has been written to the secure memory, and then the camera hardware (camera) sends a message to the face recognition TA that the face data has been written to the secure memory.

[0238] S215, Face recognition TA requests face data from secure memory.

[0239] S216, Secure memory returns face data to face recognition TA.

[0240] The implementation process of steps S201-S216 is similar to that of steps S101-S116 above, and will not be repeated here.

[0241] S217. Face recognition TA extracts facial feature values ​​based on facial data collected by a camera.

[0242] S218. The face recognition TA calls the security chip interface of the security virtual machine TVM1 to send a request message to the security chip. This request message is used to read face template data.

[0243] S219. The security chip calls the security chip interface of the security virtual machine TVM1 to return face template data to the face recognition TA.

[0244] S220, Face recognition TA decrypts face template data.

[0245] In this embodiment, the face template data obtained by the face recognition TA is encrypted face template data. The face recognition TA can use a preset key to decrypt the face template data to obtain the decrypted face template data.

[0246] S221. The face recognition system compares the facial feature values ​​captured by the camera with the decrypted face template data.

[0247] S222. The face recognition TA determines whether the comparison result meets the preset threshold.

[0248] S223. If the comparison result meets the preset threshold, the face recognition system obtains the face recognition result: recognition successful. Successful recognition means successful identity authentication.

[0249] If the comparison result does not meet the preset threshold, the face recognition system will return the following result: recognition failed. Recognition failure indicates that identity authentication has failed.

[0250] S224. The face recognition TA reports the face authentication result. The camera HAL, face recognition service, and payment / unlocking application all receive the face authentication result.

[0251] S225. After receiving the facial recognition result, the payment / unlocking application completes the payment or unlocking based on the facial recognition result.

[0252] S226 and S227: After receiving the face authentication result, the face recognition control module calls the camera HAL to shut down the camera hardware (camera).

[0253] Accordingly, the camera hardware (camera) can send a message to the camera HAL that the camera is off, and then the camera HAL sends a message to the face recognition control module that the camera is off.

[0254] S228. The face recognition control module calls the camera HAL to revoke access permissions to secure memory. Specifically, the camera HAL calls the agent service of the main virtual machine VM and the agent service of the secure virtual machine TVM2 to revoke access permissions of secure virtual machines TVM1 and TVM2 to secure memory.

[0255] S229 and S230: The face recognition control module calls the camera HAL to trigger a safe memory release.

[0256] Furthermore, the face recognition control module can call the TEE driver to disable face recognition TA.

[0257] The facial recognition / authentication process has ended.

[0258] The following is combined with Figure 10 Taking a face recognition / authentication scenario as the unlocking and authentication scenario and a TOF camera as an example, the face recognition / authentication scheme provided in this application is illustrated below. The following steps are... Figure 8 It is executed after S119 (which stores face template data in the security chip).

[0259] S301. When a user's unlock operation is detected, the lock screen application calls the face recognition SDK to perform face recognition.

[0260] The user's unlocking operations include picking up the phone, pressing the power button, operating on the screen (tapping, swiping, etc.), or unplugging the charging cable.

[0261] Meanwhile, the lock screen application can register a callback with the face recognition SDK. The purpose of registering this callback is to return the face recognition result to the lock screen application after the face recognition SDK obtains the face recognition result.

[0262] S302. The face recognition SDK sends a face recognition request to the face recognition service. The face recognition request carries an identifier for the face recognition type, the image resolution, and the data stream format. The face recognition type includes 2D face recognition (e.g., corresponding to identifier 0) and 3D face recognition (e.g., corresponding to identifier 1).

[0263] For example, the face recognition type carried in the face recognition request can be 1 (i.e., 3D face recognition type), the image resolution can be 1280x2898 pixels, and the data stream format can be raw image format (RAW)16.

[0264] Meanwhile, the face recognition SDK can register a callback with the face recognition service. The purpose of registering this callback is that when the face recognition service obtains the face comparison result, it can return the face recognition result to the face recognition SDK.

[0265] S303. The face recognition service sends a face recognition request to the face recognition control module. In other words, the face recognition SDK can notify the face recognition control module to perform face recognition through the face recognition service. The face recognition service can send the face recognition request received from the face recognition SDK to the face recognition control module.

[0266] Meanwhile, the face recognition service can register a callback with the face recognition control module. The purpose of registering this callback is that after the face recognition control module obtains the face comparison result, it can return the face comparison result to the face recognition service.

[0267] S304. In response to receiving a face recognition request, the face recognition control module matches the camera according to the face recognition request.

[0268] Specifically, the face recognition control module can obtain the face recognition type identifier, image resolution, and data stream format from the face recognition request, and determine the matching camera by querying the camera capabilities from the camera service.

[0269] It should be understood that during the power-on process of an electronic device, the camera service can send a camera capability query request to the camera HAL (Host Allocation Center). This request is used to query the camera capabilities supported by the electronic device. Upon receiving the request, the camera HAL can send the supported camera capabilities to the camera service, which can then store this information. These supported capabilities include the camera identifier (ID), maximum supported resolution, data stream format, and whether the camera supports depth information acquisition.

[0270] For example, suppose a mobile phone has three cameras installed, and the capability information of these three cameras can be shown in Table 1:

[0271] Table 1

[0272] Camera ID Installation location Maximum supported resolution Data stream format Depth Information 1 Rear 4096x3072 pixels YUY No 2 Front 3264x2448 pixels YUY No 3 Front 1280x2898 pixels RAW16 Yes

[0273] Among them, the camera with camera ID 3 can be a TOF camera, which supports the acquisition of depth information. Cameras with camera IDs 1 and 2 can be ordinary cameras, which do not support the acquisition of depth information. Of course, more front or rear cameras can be installed on a phone; for example, a phone can have two front cameras and four rear cameras.

[0274] The face recognition control module can send a camera capability query request to the camera service. The camera service can send the camera capabilities supported by the electronic device to the face recognition control module. The face recognition control module can determine the matching camera based on the camera capabilities supported by the electronic device. For example, it can determine that the matching camera is the camera with ID 3 (i.e., a TOF camera).

[0275] It should be noted that Table 1 is only an example, and the data stream format corresponding to each camera can include multiple formats. For example, the camera identified as 1 can correspond to not only the YUY data stream format, but also the RAW16 data stream format; this application does not impose any limitations.

[0276] S305, The face recognition control module sends a request to the camera service to open the camera.

[0277] For example, the face recognition control module can send a request to the camera service to open the camera via the vendor native development kit (VNDK) interface. This request includes information such as a security identifier, camera ID, resolution, and data stream format.

[0278] The security flag indicates that data should be stored in a secure buffer. In other words, the security flag can be used to request a secure memory space for storing data captured by the camera. For example, the security flag can be 1 or 0; 1 indicates that data should be stored in a secure buffer, and 0 indicates that data should be stored in a non-secure buffer.

[0279] For example, the security identifier carried in the request to open the Camera can be 1 (i.e., the data is stored in a secure buffer), the image resolution can be 1280x2898 pixels, the data stream format can be RAW16, and the camera ID can be 3.

[0280] Meanwhile, the face recognition control module can register a callback with the Camera service. This callback is used to notify the face recognition control module that the camera has been opened once the camera service has finished opening the camera.

[0281] S306. In response to receiving a request to open the Camera, the camera service sends a request to the camera HAL to open the Camera. The request to open the Camera carries information such as the security identifier, camera ID, resolution, and data stream format.

[0282] During the process of the camera service calling the camera HAL, the camera service can send information such as security identifier, camera ID, image resolution, and data stream format to the camera HAL. The camera HAL can cache this information, including the security identifier, camera ID, image resolution, and data stream format, for a preset time.

[0283] Additionally, the camera service can register a callback with the camera HAL, which is used by the camera HAL to notify the camera service of the result of creating the path.

[0284] S307, the camera HAL creates the corresponding path based on the camera ID, image resolution, and data stream format.

[0285] The camera HAL can select available nodes based on the camera ID, resolution, and data stream format, and then create corresponding pathways based on the available nodes. For example, if the resolution is 1280x2898 pixels, the data stream format is RAW16, and the camera ID is 3, then the sensor node and IFE node can be selected. This is because the sensor node and IFE node can support the transmission of data acquired by the camera with camera ID 3 at a resolution of 1280x2898 pixels and a data stream format of RAW 16.

[0286] The path corresponding to the sensor node can be: sensor node - camera driver - TOF camera - IFE module - secure memory. The path corresponding to the IFE node can be: IFE module (carrying FD) - camera driver - IFE node. The camera HAL can connect the output port of the sensor node and the input port of the IFE node at the HAL layer. Thus, the path corresponding to the sensor node and the path corresponding to the IFE node can form a closed loop. After the path is created, the hardware in the path is powered on (i.e., the hardware circuit is energized) and waits for data requests.

[0287] S308, The camera HAL returns the result of the path creation to the camera service.

[0288] The path creation process can result in either success or failure. If the path creation fails, the camera HAL notifies the camera service that the path creation failed. If the path creation is successful, the camera HAL notifies the camera service that the path creation was successful, and S109 and subsequent steps can continue.

[0289] S309. In response to receiving a notification that the path creation was successful, the camera service returns a message to the face recognition control module that the camera has been opened.

[0290] It is understandable that "camera turned on" means that the camera has completed the preparation work before taking a picture or video (such as camera parameter configuration, powering on, etc.).

[0291] S310. In response to receiving the message that the camera has been turned on, the face recognition control module sends a data request to the camera service.

[0292] The data request is used to request the camera's data stream.

[0293] S311. In response to receiving a data request from the face recognition control module, the camera service calls the camera HAL to obtain the data stream.

[0294] S312, the sensor node sends the camera's configuration parameters to the camera driver module in the kernel layer.

[0295] It should be understood that the sensor node can store the addresses of the various registers of the TOF camera, as shown in Table 3.

[0296] Table 3

[0297] Register identifier Storage data types address 1 Current 0x1 2 resolution 0x2 3 Data stream format 0x3 4 TOF camera device operating status 0x4 5 TOF camera working mode 0x5

[0298] S313, The camera driver module writes (updates) the camera's configuration parameters into the TOF camera's register.

[0299] In other words, the camera driver module can send configuration parameters to the TOF camera.

[0300] S314, The camera driver module sends a start (stream on) command / instruction to the TOF camera.

[0301] The start command is used to drive the TOF camera to acquire data.

[0302] It should be noted that before S314 and after S313, the camera driver module can also send a message indicating that the configuration parameters have been written. In response to receiving the message indicating that the configuration parameters have been written, the sensor node sends a start command to the camera driver module.

[0303] S315. In response to receiving the start command, the TOF camera acquires RAW data.

[0304] The RAW Data contains Metadata. For example, the Metadata stores information such as the current operating mode of the TOF camera, the operating status of the TOF camera device (e.g., normal or abnormal), and the image exposure value (e.g., 10 μs).

[0305] The S316 TOF camera sends the acquired RAW data to the IFE module.

[0306] For example, a TOF camera can transmit the RAW data it captures to an IFE module via a Mobile Industry Processor Interface (MIPI). The IFE module, also known as an Image Preprocessing Module (IFE-Lite), does not perform any processing on the RAW data.

[0307] The S317 IFE module sends RAW data to a secure buffer for storage.

[0308] The storage location of RAW data captured by a TOF camera in secure memory can be represented by FD1.

[0309] For example, when FD1 is 69, it can represent the storage location as XX secure memory; when FD1 is 96, it can represent the storage location as YY non-secure memory (normal memory).

[0310] The S318 and IFE modules send FD1 to the camera driver module. The camera driver module then sends FD1 to the IFE node. The IFE node sends FD1 to the camera service via the camera HAL interface. The camera service then sends FD1 to the face recognition control module.

[0311] S319, The face recognition control module sends FD1 to the face recognition TA (FACE TA).

[0312] The face recognition technology (TA) includes a Time-of-Flight (TOF) algorithm and a Face ID algorithm. The TOF algorithm converts RAW data into grayscale and depth maps, and then calculates the security of the face (i.e., whether the current user is the owner of the device) based on these maps. The Face ID algorithm is used for grayscale image matching and depth map verification to prevent spoofing.

[0313] S320, the face recognition TA reads RAW data from secure memory according to FD1.

[0314] The face recognition TA can request RAW data from the secure memory based on FD1, and the secure memory can send RAW data to the face recognition TA.

[0315] S321. The face recognition TA obtains face template data from the security chip through the security chip interface.

[0316] S322. The face recognition TA compares the RAW data with the face template data to obtain the face recognition result.

[0317] Face recognition technology can process RAW data using the Time-of-Flight (TOF) algorithm to obtain a first grayscale image and a first depth image. Then, it uses a face ID algorithm to perform face recognition based on the first grayscale image, and uses the first depth image and anti-spoofing detection to obtain the face recognition result.

[0318] It should be noted that after obtaining the face template data, the face recognition system can use the Time-of-Flight (TOF) algorithm to convert the face template data into grayscale and depth maps.

[0319] If the currently collected facial information (the RAW data currently collected by the TOF camera) matches the grayscale image corresponding to the facial template data (i.e., the RAW data collected by the electronic device when the user performs the facial registration operation), it can be considered that they are the same user (i.e., the user performing the facial registration operation and the user performing the unlocking operation are the same user). Furthermore, if the currently collected facial information includes depth information, it can be considered that the current user is real and trustworthy (not a photo, video, or other disguise). In this case, the current user's face can be considered secure, and the facial recognition result is successful.

[0320] If the currently collected face information (the RAW data currently collected by the TOF camera) does not match the grayscale image corresponding to the face template data (i.e., the RAW data collected by the electronic device when the user performs face registration), or if the currently collected face information does not include depth information, the current user's face is considered unsafe, that is, the face recognition result is recognition failure.

[0321] S323, The Face Recognition TA sends the face recognition result to the face recognition control module. The face recognition control module sends the face recognition result to the face recognition service. The face recognition service passes the face recognition result to the face recognition SDK. The face recognition SDK passes the face recognition result to the lock screen application.

[0322] The face recognition control module can pass the face recognition result (success or failure) to the face recognition service based on the callback registered in the previous face recognition service (in S303).

[0323] The face recognition service, based on the callback registered in the face recognition SDK in S302, passes the face recognition result (success or failure) to the face recognition SDK.

[0324] The face recognition SDK, based on the callback registered by the lock screen application in S301, passes the face recognition result (success or failure) to the lock screen application.

[0325] S324: Lock screen applications may or may not unlock based on facial recognition results.

[0326] If the facial recognition is successful, the lock screen application can be unlocked, allowing the electronic device to display the desktop or application (system application or third-party application) interface.

[0327] If facial recognition fails, the lock screen app will not unlock the device; that is, facial unlocking has failed. After facial unlocking fails, the lock screen app can disable the facial recognition function for a period of time (e.g., 5 minutes).

[0328] It should be noted that the above embodiments are illustrated using a lock screen application for face unlocking as an example. The above solution can also be applied to face recognition during payments or transfers (e.g., when users perform payments or transfers in payment applications / financial management applications / chat applications / shopping applications), and in scenarios such as face security verification for secure registration or login applications. This application is not limited in these scenarios. In other words, the lock screen application can be replaced with a shopping application, chat application, payment application, banking application, or financial management application, etc. This application is not limited in these scenarios.

[0329] Some embodiments of this application provide an electronic device that may include a touchscreen, a memory, and one or more processors. The touchscreen, memory, and processors are coupled. The memory stores computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps performed by the electronic device in the above method embodiments. The structure of the electronic device can be referred to... Figure 4 The structure of the electronic device 100 shown.

[0330] This application also provides a chip system, such as a system-on-a-chip (SoC), as shown in the embodiments. Figure 11 As shown, the chip system includes at least one processor 1101 and at least one interface circuit 1102. The processor 1101 and the interface circuit 1102 are interconnected via lines. For example, the interface circuit 1102 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 1102 can be used to send signals to other devices (e.g., the processor 1101 or the touchscreen of an electronic device). Exemplarily, the interface circuit 1102 can read instructions stored in the memory and send those instructions to the processor 1101. When the instructions are executed by the processor 1101, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.

[0331] This application also provides a TOF camera, which can be used to implement the above embodiments. An electronic device equipped with the TOF camera can perform various functions or steps performed by the electronic device in the above method embodiments.

[0332] This application also provides a computer-readable storage medium including computer instructions that, when executed on the electronic device, cause the electronic device to perform various functions or steps performed by the electronic device in the above method embodiments.

[0333] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to perform various functions or steps performed by the electronic device in the above method embodiments.

[0334] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0335] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0336] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0337] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0338] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0339] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A face recognition method applied to an electronic device, the electronic device comprising a main virtual machine, a first secure virtual machine, and a second secure virtual machine; characterized in that, The first secure virtual machine includes a trusted face recognition application (TA) and a secure operating environment (TEE) system service. The secure operating environment (TEE) system service is equipped with a corresponding secure chip interface. The method includes: The face recognition trusted application (TA) calls the secure operating environment (TEE) system service and the security chip interface to encrypt the face template data and store it in the security chip of the electronic device; the first secure virtual machine has read permission to the security chip, while the main virtual machine does not have read permission to the security chip. The main virtual machine receives a first operation from the user, which is an operation that triggers the face recognition process; In response to the first operation, the main virtual machine creates a first secure memory and sets access permissions for the first secure memory; the camera hardware of the electronic device has write permissions to the first secure memory, and the first secure virtual machine and the second secure virtual machine have read permissions to the first secure memory; The main virtual machine calls the camera HAL to trigger the camera hardware startup; When the first face data acquired by the camera hardware is written to the first secure memory, the first secure virtual machine retrieves the first face data from the first secure memory and calls the secure operating environment TEE system service and the secure chip interface to retrieve face template data from the secure chip; the first secure virtual machine compares the first face data with the face template data to obtain the face recognition result.

2. The method according to claim 1, characterized in that, The main virtual machine runs in a general runtime environment (REE), the first secure virtual machine runs in a first secure runtime environment (TEE), and the second secure virtual machine runs in a second secure runtime environment (TEE).

3. The method according to claim 1, characterized in that, Before the main virtual machine receives the user's first operation, the method further includes: The main virtual machine receives a second operation from the user, which is an operation to trigger the face registration process; In response to the second operation, the main virtual machine creates a second secure memory and sets access permissions for the second secure memory; the camera hardware has write permissions to the second secure memory, and the first secure virtual machine and the second secure virtual machine have read permissions to the second secure memory; The main virtual machine invokes the camera HAL to trigger the camera hardware startup; When the second face data acquired by the camera hardware is written to the second secure memory, the first secure virtual machine obtains the second face data from the second secure memory and generates the face template data based on the feature values ​​of the second face data; The first secure virtual machine encrypts the face template data and stores it in the secure chip.

4. The method according to claim 3, characterized in that, After the first secure virtual machine encrypts the face template data and stores it in the secure chip, the method further includes: The main virtual machine calls the camera HAL, triggering the camera hardware to shut down and triggering the second secure memory release.

5. The method according to claim 1, characterized in that, After the face recognition result is obtained in the first secure virtual machine, the method further includes: The first secure virtual machine sends the face recognition result to the main virtual machine; The main virtual machine receives the face recognition result and performs face recognition services based on the face recognition result, including face payment or face unlock.

6. The method according to claim 5, characterized in that, After the main virtual machine receives the face recognition result, the method further includes: The main virtual machine calls the camera HAL, triggering the camera hardware to shut down and the first secure memory to be released.

7. The method according to any one of claims 1 to 6, characterized in that, The main virtual machine creates the first secure memory, including: The main virtual machine invokes the camera HAL to create the first secure memory based on the running memory of the electronic device.

8. The method according to any one of claims 1 to 6, characterized in that, The step of obtaining face template data from the security chip includes: The encrypted face template data is obtained from the security chip; The face template data is decrypted.

9. The method according to any one of claims 1 to 6, characterized in that, The main virtual machine includes a first client application CA, a face recognition service module, a face recognition control module, and a secure operating environment TEE driver; The method further includes: In response to the user's first operation on the first client application CA, the first client application CA requests face recognition service from the face recognition service module; The face recognition service module instructs the face recognition control module to start the face recognition process; The face recognition control module calls the secure operating environment TEE driver to start the face recognition trusted application TA on the first secure virtual machine side.

10. The method according to claim 9, characterized in that, The first client application CA is a payment application or an unlocking application.

11. The method according to claim 9, characterized in that, After the face recognition trusted application TA is started on the first secure virtual machine side, the face recognition trusted application TA obtains the first face data from the first secure memory, obtains the face template data from the secure chip and decrypts the face template data, and compares the first face data with the decrypted face template data to obtain the face recognition result.

12. The method according to claim 9, characterized in that, The main virtual machine also includes a second client application CA, which is used to set face template data; the method further includes: In response to the user's second operation on the second client application CA, the second client application CA requests face enrollment service from the face recognition service module; The face recognition service module instructs the face recognition control module to start the face enrollment process; The face recognition control module calls the TEE driver to start the face recognition trusted application TA on the first secure virtual machine side.

13. The method according to claim 12, characterized in that, After the face recognition trusted application TA is started on the first secure virtual machine side, the face recognition trusted application TA obtains the second face data collected by the camera hardware from the second secure memory, and generates the face template data based on the feature values ​​of the second face data.

14. The method according to claim 9, characterized in that, The main virtual machine further includes a first agent service module, and the second security virtual machine includes a second agent service module; The main virtual machine sets access permissions for the first secure memory, including: The face recognition control module calls the first agent service and the second agent service to set access permissions for the first secure memory; The face recognition control module calls the secure operating environment TEE driver to send the access permission information of the first secure memory to the secure operating environment TEE system service.

15. The method according to claim 14, characterized in that, The first secure virtual machine retrieves the first face data from the first secure memory, including: The face recognition trusted application (TA) calls the secure operating environment (TEE) system service to obtain the first face data from the first secure memory.

16. The method according to claim 14, characterized in that, The method further includes: When the camera hardware acquires the first face data, the camera hardware calls the camera driver on the main virtual machine side and the first proxy service to write the first face data into the first secure memory.

17. An electronic device, characterized in that, The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 16.

18. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 16.

Images