Image processing method and electronic equipment

Abstract

The embodiment of the invention discloses an image processing method and electronic equipment, and the method comprises the steps: responding to a received user operation in REE, and obtaining TOF data through a TOF camera; processing the TOF data in the TEE to obtain a TOF image, and further, operating the TOF image through an NPU (Network Processing Unit) arranged on the TEE side and a hardware NPU running target algorithm to obtain an operation result; and executing an instruction corresponding to the user operation in the REE according to the operation result. According to the method, the hardware acceleration is realized and the operation efficiency is improved through the NPU operation in the TEE hardware.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to an image processing method and electronic equipment. Background Art

[0002] At present, the fields of terminal unlocking, payment and identity authentication all involve the processing of sensitive images such as faces and eyes. Usually, the central processing unit (CPU) is used to run artificial intelligence (AI) algorithms on the trusted execution environment (TEE) side of the terminal to process the images, and then the processing results are returned to the rich execution environment (REE) side of the terminal device.

[0003] However, due to the limitations on the number of CPU cores used on the TEE side and the processing performance of the CPU running AI algorithms, the efficiency of image processing is low. Summary of the invention

[0004] The embodiments of the present invention provide an image processing method and electronic device to solve the problem of low efficiency of image processing.

[0005] In a first aspect, an embodiment of the present invention provides an image processing method, which is applied to an electronic device, wherein the electronic device includes a trusted execution environment TEE and a universal execution environment REE, and the method includes:

[0006] In response to the received user operation, acquiring TOF data through the TOF camera in the REE;

[0007] The TOF data is processed in TEE to obtain a TOF image;

[0008] In TEE, the TOF image is operated by the target algorithm running on the neural network processor NPU to obtain the operation result;

[0009] In REE, the instructions corresponding to the user operations are executed according to the calculation results.

[0010] The above method realizes hardware acceleration and improves computing efficiency by computing on the TEE hardware NPU.

[0011] In combination with the first aspect, in a possible implementation, the method may further include: storing the TOF data and an identifier of the TOF data in a secure buffer memory area at the REE; and acquiring the TOF data from the secure buffer memory area based on the identifier at the TEE.

[0012] The above method provides a mechanism for data communication between REE and TEE, which is more secure.

[0013] In combination with the first aspect, in a possible implementation, the electronic device includes a TOF camera service in REE, a TOF camera hardware abstraction layer HAL, a client application CA, a HAL corresponding to CA, a service corresponding to CA, and a trusted application TA in TEE; the method may also include: the TOF camera sends an identifier to the TOF camera HAL; the TOF camera HAL sends an identifier to the TOF camera service; the TOF camera service sends an identifier to the HAL corresponding to CA; the HAL corresponding to CA sends an identifier to CA; the CA sends an identifier to TA; wherein TA is used to obtain TOF data from a secure buffer memory area based on the identifier.

[0014] Optionally, the TA is also used to process the TOF data to obtain a TOF image; the method may also include:

[0015] TA sends the TOF image to the neural network processor NPU.

[0016] Optionally, TA sends the TOF image to the neural network processor NPU, specifically including: TA stores the TOF image in a secure shared cache area; TA sends a first notification to the NPU, the first notification being used to indicate that the TOF image has been stored; and the NPU obtains the TOF image from the secure shared cache area in response to the first notification.

[0017] In combination with the first aspect, in a possible implementation, the method may also include: performing human eye safety detection based on TOF data in TEE to obtain a detection result, wherein the detection result is used to indicate whether the light sent by the TOF camera is safe for the human eye; when the detection result indicates that it is unsafe, stopping obtaining TOF data through the TOF camera.

[0018] The above method can timely conduct eye safety detection to ensure the safety of users.

[0019] Optionally, the electronic device also includes a TOF camera service in REE, a TOF camera hardware abstraction layer HAL, a TOF camera driver, a client application CA, a HAL corresponding to CA, a service corresponding to CA, and a trusted application TA in TEE; TA is used to perform human eye safety detection based on TOF data to obtain a detection result; the method may also include: TA sends the detection result to CA; CA sends the detection result to the HAL corresponding to CA; the HAL corresponding to CA sends the detection result to the TOF camera service; the TOF camera service sends the detection result to the TOF camera HAL; when the detection result of the TOF camera HAL is unsafe, the TOF camera driver sends a first instruction, and the first instruction is used to instruct the TOF camera driver to stop obtaining TOF data through the TOF camera.

[0020] In combination with the first aspect, in a possible implementation, the method may further include: determining, in the TEE, an exposure parameter value of the TOF camera based on the TOF data; and adjusting, in the REE, an exposure parameter of the TOF camera based on the exposure parameter value.

[0021] The above method can realize dynamic adjustment of the exposure parameters of the TOF camera to obtain higher quality TOF data.

[0022] Optionally, the electronic device also includes a TOF camera service in REE, a TOF camera hardware abstraction layer HAL, a TOF camera driver, a client application CA, a HAL corresponding to CA, a service corresponding to CA, and a trusted application TA in TEE; TA is used to determine the exposure parameter value of the TOF camera based on TOF data; the method may also include: TA sends the exposure parameter value to CA; CA sends the exposure parameter value to the HAL corresponding to CA; the HAL corresponding to CA sends the exposure parameter value to the TOF camera service; the TOF camera service sends the exposure parameter value to the TOF camera HAL; the TOF camera HAL sends the exposure parameter value to the TOF camera driver; the TOF camera driver sets the exposure parameter of the TOF camera to the exposure parameter value.

[0023] In combination with the first aspect, in a possible implementation, the electronic device also includes a TOF camera service, a TOF camera hardware abstraction layer HAL, a TOF camera driver, a client application CA, a HAL corresponding to CA, a service corresponding to CA, and a trusted application TA in TEE; the method may also include: the NPU sends the operation result to TA; TA sends the operation result to CA; CA sends the operation result to the HAL corresponding to CA; the HAL corresponding to CA sends the operation result to the service corresponding to CA, and the service corresponding to CA sends the operation result to the application; executing instructions corresponding to user operations based on the operation results in REE, including the application executing instructions corresponding to user operations based on the operation results.

[0024] Optionally, an implementation of NPU sending the operation result to TA may include: NPU storing the operation result in a secure shared cache area; NPU sending a second notification to TA, the second notification is used to indicate that the operation is completed; CA responds to the second notification and reads the operation result from the secure shared cache area.

[0025] In combination with the first aspect, in a possible implementation, in the TEE, the TOF image is operated by the target algorithm running on the neural network processor NPU. Before the operation result is obtained, the method may also include: in response to a call instruction of the REE, loading the target algorithm in the NPU, and the call instruction includes an identifier of the target algorithm.

[0026] In combination with the first aspect, in a possible implementation, the method may further include: configuring the mode of the NPU to a security mode.

[0027] In combination with the first aspect, in a possible implementation, the user operation is an unlocking operation detected by the electronic device when the electronic device is in a locked screen state; the TOF image includes a grayscale image and a depth image, and the target algorithm is a face recognition algorithm and a liveness verification algorithm;

[0028] In the REE, an instruction corresponding to the user operation is executed according to the calculation result, including: in the REE, when the calculation result indicates that the face recognition and the liveness verification algorithm are passed, the screen is unlocked.

[0029] In combination with the first aspect, in a possible implementation, the user operation is a payment operation detected by the electronic device in a payment interface; the TOF image includes a grayscale image and a depth image, and the target algorithm is a face recognition algorithm and a liveness verification algorithm;

[0030] In the REE, instructions corresponding to the user operation are executed according to the calculation results, including: in the REE, when the calculation results indicate that the face recognition and the liveness verification algorithm have been passed, payment is made.

[0031] In combination with the first aspect, in a possible implementation, the user operation is a user operation for turning on gesture control, the TOF image includes a grayscale image and a depth image, and the target algorithm is a gesture recognition algorithm and a liveness verification algorithm;

[0032] Executing an instruction corresponding to the user operation according to the calculation result in the REE includes: in the REE, when the calculation result indicates a first posture and passes a liveness verification algorithm, executing an instruction corresponding to the first posture.

[0033] In combination with the first aspect, in a possible implementation, the user operation is a user operation for turning on gesture control, the TOF image includes a grayscale image and a depth image, and the target algorithm is a gesture recognition algorithm and a liveness verification algorithm;

[0034] Executing an instruction corresponding to the user operation according to the operation result in the REE includes: in the REE, when the operation result indicates a first gesture and passes a liveness verification algorithm, executing an instruction corresponding to the first gesture.

[0035] In a second aspect, an embodiment of the present invention provides an electronic device, comprising multiple processors and one or more memories; wherein the multiple processors include a neural network processor NPU, the one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program code, the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the processor executes the first aspect or any possible implementation of the first aspect, the electronic device is the execution method of the present invention.

[0036] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, including computer instructions, which, when executed on an electronic device, enables the electronic device to execute the method of execution in the first aspect or any possible implementation of the first aspect.

[0037] In a fourth aspect, an embodiment of the present invention provides a computer program product, including computer instructions, which, when executed on an electronic device, causes the electronic device to execute the method of execution in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0040] Figure 2 It is a software and hardware architecture diagram of an electronic device provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application;

[0042] Figure 4 A flowchart of another image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0044] In the embodiments of the present application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0045] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0046] The terms involved in the embodiments of the present application are explained as follows.

[0047] (1) Time of flight (TOF) technology is a technology that calculates the distance traveled by measuring the time it takes light to travel a certain distance in a medium.

[0048] (2) TOF imaging refers to an imaging technology in which a group of infrared lights (laser pulses) invisible to the human eye are emitted outward, reflected after hitting an object, and end up being reflected to the camera. The time difference or phase difference from emission to reflection back to the camera is calculated, and the data is collected to form a set of distance and depth data, thereby obtaining a three-dimensional 3D model.

[0049] (3) A sensor that can achieve TOF imaging is a TOF sensor, also known as a TOF camera or a TOF camera. While acquiring depth image data, a TOF camera can also acquire infrared image data of the irradiated object.

[0050] In the present application, TOF data (TOF image data) can be raw data obtained by a TOF camera, or data after preliminary processing or preprocessing of the raw data. After processing, the TOF data can be converted into a TOF image, and the TOF image can include a grayscale image and / or a depth image. Among them, the grayscale image in the embodiment of the present application is an infrared image obtained by a TOF camera, which can also be called an IR image or an IR grayscale image.

[0051] Since infrared light and visible light are in different bands, TOF images are not affected by visible light. TOF images can be applied to face recognition, face verification, face unlocking, posture recognition, expression recognition, gaze recognition, gesture recognition and other fields to improve recognition accuracy. They can also be used in the field of liveness verification and have anti-counterfeiting capabilities.

[0052] (4) Rich execution environment (REE), generally referred to as the non-secure side or non-secure area, is a common environment for all mobile devices and runs a common OS (Operating System), such as Android, IOS, etc.

[0053] (5) Trusted execution environment (TEE), generally referred to as the security side or security zone, is an area that requires authorization to access; it is usually used for digital copyright management and privacy information protection.

[0054] (6) Vendor Native Development Kit (VNDK) is a set of static libraries specifically used by vendors to implement their hardware abstraction layer.

[0055] (7) Client Application (CA) is an application running in the REE environment, referred to as CA for short.

[0056] (8) Trusted Application (TA) is an application running in the TEE environment, referred to as TA for short.

[0057] (9) Fast Remote Procedure Call (RPC) is a type of remote procedure call based on the Client-Server working mode. The Fast RPC framework allows users to transparently perform remote method calls between applications and NPU processors. It is mainly used for loading, initializing and unloading the secure HTP (Hexagon Tensor Processor) (NPU in this application) and the algorithms running on it.

[0058] (10) The secure shared cache is a secure storage space created between the application running on the processor on the security side and the NPU for data exchange between the two parties.

[0059] (11) Shared Channel: A communication mechanism between the TA and the NPU on the TEE side, used to notify the other party of the status or to notify the other party to communicate data.

[0060] The embodiment of the present application provides an image processing method, which is applied to an electronic device, in response to a received user operation in the REE, TOF data is obtained through a time-of-flight TOF camera; the TOF data is processed in the TEE to obtain a TOF image, and further, a neural network processing unit (NPU) is set on the TEE side, and the TOF image is operated by running a target algorithm through the hardware NPU to obtain an operation result; and the instruction corresponding to the user operation is executed in the REE according to the operation result. The method realizes hardware acceleration and improves operation efficiency by operating the TEE hardware NPU.

[0061] The electronic device in this application may be a device with a TOF camera, including but not limited to smartphones, tablet computers, smart screens, wearable devices, vehicle-mounted devices, augmented reality (AR), virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), artificial intelligence (AI) terminals and other terminal devices. The embodiments of this application do not impose any restrictions on the specific types of electronic devices.

[0062] For example, Figure 1 A schematic structural diagram of an electronic device 100 provided in an embodiment of the present application is shown.

[0063] The electronic device 100 may include a processor 110, a neural network processor 111, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and the like.

[0064] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0065] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, etc. Different processing units may be independent devices or integrated into one or more processors.

[0066] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0067] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0068] The processor 110 can be coupled to a memory (such as the internal memory 121 and / or an external memory), call the computer instructions stored in the memory, and execute the image processing method on the REE side. For details, see the following Figure 3 The image processing method shown will not be described in detail here.

[0069] The electronic device 100 may also include one or more neural-network processing units (NPUs) 111. The NPU 111 may also be included in the processor 110. Like the above-mentioned processor 110, the NPU 100 is coupled to one or more memories to call the method stored in the memory to implement the image processing method executed by the NPU on the TEE side.

[0070] Exemplarily, the processor 110 in the embodiment of the present application may include one or more first processors located on the REE side, and one or more second processors located on the TEE side. The processor 110 includes a first processor and a second processor, and the first processor and the second processor may be the same processor divided by software. The first processor is used to call the method on the REE side and execute the image processing method on the REE side in the present application; the second processor and NPU 111 are used to call the method on the TEE side and execute the image processing method on the TEE side in the present application, for example, NPU 111 executes the method executed by the NPU on the TEE side, and the second processor executes the method in the TEE except the method executed by the NPU. For details, please refer to the following Figure 3 The image processing method shown will not be described in detail here.

[0071] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0072] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0073] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0074] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0075] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0076] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0077] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0078] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0079] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0080] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), the fifth generation of wireless communication system (5G), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0081] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0082] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0083] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0084] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0085] The camera 193 is used to capture still images or videos. The camera 193 may include a TOF camera. The TOF camera can not only collect depth data of the photographed object, but also obtain an infrared image of the photographed object. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0086] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0087] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0088] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0089] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0090] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0091] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0092] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0093] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0094] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0095] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0096] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0097] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0098] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0099] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0100] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0101] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser.

[0102] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0103] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0104] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0105] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0106] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0107] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0108] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.

[0109] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting the SIM card interface 195 or pulling the SIM card interface 195 out. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc.

[0110] The operating system of the electronic device 100 is divided into a general execution environment and a trusted execution environment, and the software and hardware of the general execution environment can adopt a layered architecture. Figure 2 As shown, taking the layered architecture Android system as an example, the hardware and software structure of the general execution environment of the electronic device 100 includes the application (APP) layer, the framework (FWK) layer, the hardware abstraction layer (HAL), the kernel layer, and the hardware layer. The trusted execution environment may include trusted applications, a neural network processor NPU, a secure buffer memory area for REE and TEE to exchange data, and a secure shared cache area for trusted applications and NPU to exchange data. They are explained separately as follows.

[0111] It should be understood that when the electronic device initializes the NPU, the mode of the NPU can be configured to a secure mode to divide the NPU into the TEE. For example, information indicating secure use can be sent to the NPU during initialization to configure it to a secure mode.

[0112] REE:

[0113] The application layer may include at least one application, which may be a system application or a third-party application, such as smart sensing, TOF camera, etc., or notification, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc. ( Figure 2 not shown).

[0114] Among them, the methods or services supported by the smart perception service may include but are not limited to gesture control, face unlocking, face payment, etc.

[0115] The framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. For example, it may include activity manager, window manager, content provider, view system, resource manager, notification manager, camera service (CameraService) and intelligent perception service.

[0116] The system architecture on the REE side can also include system libraries and Android Runtime (Android Runtime) ( Figure 2 The system library may include multiple functional modules, such as surface manager, media libraries, OpenGL ES, SGL, etc. The Android runtime includes a core library and a virtual machine.

[0117] The HAL layer is an encapsulation of the Linux kernel driver, providing an interface to the upper layer and shielding the implementation details of the low-level hardware.

[0118] The HAL layer can include the hardware abstraction module of the TOF camera, namely the TOF camera HAL (Camera HAL), and the intelligent perception Daemon module. Figure 2 The HAL layer may also include other hardware abstract modules, such as Wi-Fi HAL, audio HAL, ambient light sensor HAL, etc.

[0119] Among them, the intelligent perception Daemon module includes intelligent perception CA, intelligent perception HAL and TOF camera control unit.

[0120] The TOF camera control module can be used to implement data interaction between the smart perception HAL and the TOF camera service. For example, the smart perception HAL can send the AE value received from the smart perception TA on the TEE side to the TOF camera service through the TOF camera control module, and the TOF camera service can also send the TOE data identifier, such as the file descriptor (FileDescriptor, FD), to the smart perception HAL through the TOF camera control module. Data can be transmitted between the TOF camera control module and the TOF camera service through the VNDK interface.

[0121] The smart perception HAL is used to send the identifier of the TOF data obtained from the TOF camera service to the smart perception CA. The smart perception HAL is also used to initialize the smart perception HAL, initialize the smart perception CA, the smart perception TA and the NPU based on the request of the smart perception service.

[0122] The smart perception HAL can call the algorithm in the NPU through Fast RPC, for example, sending an initialization instruction to the NPU, the initialization instruction carries the identifier of the target algorithm, and the NPU can load the target algorithm after receiving the initialization instruction to initialize the NPU. Optionally, it can also include information for indicating safe use to divide the NPU into TEE.

[0123] The smart perception HAL can send an initialization instruction to the smart perception CA, and the initialization instruction can include an identifier of an image processing algorithm, so that the smart perception CA and TA can determine from the algorithm library that the image processing algorithm corresponding to the identifier is the image processing algorithm to be used.

[0124] Smart perception CA can be used to exchange information with smart perception TA on the TEE side. Smart perception CA can send the received TOF data identifier to smart perception TA, and can also receive the human eye safety detection results, calculation results and AE values ​​sent by smart perception TA.

[0125] The kernel layer can include the TOF camera driver and Fast RPC. The TOF camera driver is used to provide functional support for the TOF camera. Fast RPC is used to implement the intelligent perception HAL to call the NPU on the TEE side.

[0126] The hardware layer may include a TOF camera and an IFE module, wherein the TOF camera is used to collect TOF data and transmit the collected TOF data to a secure buffer memory area through the IFE module and a secure ISP channel.

[0127] The software and hardware architecture of REE is not limited to the modules shown above, and may also include other modules. The embodiment of the present application takes the intelligent sensing application as an example to illustrate, and it should be understood that Figure 2 The intelligent perception application in may also be other applications, such as the first application. Accordingly, Figure 2 The smart perception service, smart perception CA, smart perception TA, etc. corresponding to the smart perception application in the application are replaced by the service, CA, TA, etc. corresponding to the first application. If the first application is a setting application, the setting application has the function of recording a face, and the recorded face is used for face unlocking. The recorded face image can be stored in the NPU or secure buffer memory area on the TEE side.

[0128] TEE:

[0129] The TEE side may include but is not limited to at least one trusted application TA, such as a smart perception TA, which may include a smart perception module for implementing TOF data processing, algorithm invocation, task scheduling, HTP communication with the NPU, and data transmission with the smart perception CA.

[0130] The smart perception TA can also perform eye safety detection based on TOF data through an eye safety detection algorithm, and determine the AE value of the next frame through an AE prediction algorithm based on TOF data. It can also receive the calculation results sent by the smart perception TA, and can also send the eye safety detection results, calculation results and AE values ​​to the smart perception CA.

[0131] The intelligent sensing module is used for TOF-based identification, such as FD reading the TOF data from the security buffer memory area.

[0132] The intelligent perception module may include an algorithm adaptation layer and multiple algorithms. The algorithm adaptation layer is used to determine, based on the received initialization instruction, from multiple algorithms that the algorithm corresponding to the identifier included in the instruction is the required algorithm. And the TOF data is input into the determined algorithm to process the TOF image. Exemplarily, the determined algorithm includes an image processing algorithm, an eye safety detection algorithm, and an AE prediction algorithm.

[0133] The intelligent perception module can perform eye safety detection based on TOF data through an eye safety detection algorithm to obtain eye safety detection results, determine the AE value of the next frame of TOF data through an AE prediction algorithm based on the TOF data, and process the TOF data through an image processing algorithm to obtain a grayscale image and a depth image, and write the depth image and the grayscale image into a secure shared cache.

[0134] The intelligent perception module is also used to send a first notification to the NPU through a shared channel after writing the depth image and grayscale image into the secure shared cache to notify the NPU that the depth image and grayscale image have been written for reading.

[0135] The NPU may include a neural network control module and a variety of neural network models (also known as a variety of image recognition algorithms). The neural network control module provides interfaces (APIs) for each neural network model, which can be called by CA and TA, such as by smart perception TA or smart perception CA, to implement calculations through the image recognition algorithm in the NPU.

[0136] The neural network control module is used to determine the required target algorithm from multiple image recognition algorithms based on the identifier in the received initialization instruction, and is also used to receive an operation request from the smart perception TA to perform operations on the grayscale image and / or depth image through the target algorithm to obtain the operation result, and after writing the operation result to the secure shared cache, send a second notification to the smart perception TA (or other TA) through the shared channel to notify the completion of the operation or indicate that the operation result has been written.

[0137] Exemplarily, the target algorithm includes a face recognition algorithm and a liveness verification algorithm. The neural network control module is used to input the depth image and grayscale image read from the secure shared cache into the face recognition algorithm and the liveness verification algorithm for calculation, identify whether the face in the image is that of the owner of the device and whether it is alive, and write the calculation results into the secure shared cache.

[0138] It should be understood that the specific implementation of each of the above modules can also be found in the following Figure 3 The method shown will not be repeated here.

[0139] Combined with the above Figure 1 The electronic equipment and Figure 2 The software and hardware architecture of the electronic device is shown, and the image processing method involved in the embodiment of the present application is further introduced below.

[0140] See also Figure 3 , Figure 3 is a flow chart of recording eye movement calibration data provided by an embodiment of the present application. The method may include the above Figure 1 The electronic device shown is implemented, and the method may include but is not limited to some or all of the following steps:

[0141] S01, the application sends a service call request to the TOF camera service in response to a received user operation.

[0142] Among them, the service call request is used to request the call of the service. Smart perception can include multiple services, such as face unlocking, face payment, information authentication, gesture control, posture control, etc.

[0143] In the application scenario of face unlocking, the application program may be a setting application, and the user operation may be an unlocking operation of the electronic device. For example, when the electronic device is in the locked screen state, the motion sensor detects that the electronic device is picked up, or when a touch operation is detected in the locked screen state, etc. After detecting the unlocking operation, the electronic device triggers the process of unlocking the screen and executes the above S01. At this time, the instruction corresponding to the user operation may be an unlocking instruction, and the service call request is used to request to call the face unlocking service.

[0144] In the application scenario of face payment, the user operation can be a payment operation through a payment application, for example, the electronic device inputs a user operation for indicating payment in the payment interface of the payment application (such as Alipay or WeChat, etc.), and after detecting the payment operation, the electronic device triggers the secure payment process and executes the above S01. The instruction corresponding to the user operation can be a payment instruction, and the service call request is used to request the call of the face payment service.

[0145] In the application scenario of information authentication, the user operation may be a user operation that instructs identity authentication, and after detecting the user operation, the electronic device triggers the identity authentication process. The instruction corresponding to the user operation may be a verification instruction, and the service call request is used to request the service of information authentication.

[0146] In the application scenario of gesture control, the user operation may be a user operation for turning on gesture control. After detecting the user operation, the electronic device triggers the gesture control process and executes the above S01. The instruction corresponding to the user operation may be an instruction corresponding to the detected gesture, and the service call request is used to request the call of the gesture control service. For example, when the electronic device displays the main interface, the instruction corresponding to the gesture of sliding the hand to the right is to display the next page, and the instruction corresponding to the gesture of sliding the hand to the left is to display the previous page.

[0147] In the application scenario of gesture control, the user operation may be a user operation for turning on gesture control. After detecting the user operation, the electronic device triggers the gesture control process and executes the above S01. The instruction corresponding to the user operation may be an instruction corresponding to the detected gesture, and the service call request is used to request to call the gesture control service.

[0148] Not limited to the scenarios listed above, the image processing method provided in the embodiments of the present application can also be used in other scenarios, which are not limited here.

[0149] S02, the intelligent perception service sends a business request to the intelligent perception HAL.

[0150] S03, the intelligent perception HAL initializes the intelligent perception CA.

[0151] The smart perception HAL can send a first initialization instruction to the smart perception CA, where the first initialization instruction is used to instruct the smart perception CA to complete the initialization configuration to provide conditions for implementing the method to be implemented by the following smart perception CA.

[0152] Optionally, the smart perception HAL initializes itself, wherein the smart perception HAL initializes the camera control module to complete its own initialization configuration, so as to provide conditions for implementing the method to be implemented by the smart perception HAL described below.

[0153] S04, smart sensing CA initializes smart sensing TA.

[0154] The smart perception CA can send a second initialization instruction to the smart perception TA. In response to the second initialization instruction, the smart perception TA can initialize the port configuration and loaded algorithms, such as the image processing algorithm, the human eye safety detection algorithm, and the AE prediction algorithm identification, to provide conditions for implementing the method to be implemented by the following smart perception TA.

[0155] S05, the intelligent perception HAL initializes the NPU through Fast RPC.

[0156] Specifically, the smart perception HAL sends a third initialization instruction to the NPU via Fast RPC, and the NPU can initialize the configuration of the port in response to the third initialization instruction, select the algorithm corresponding to the identifier from the algorithm library, that is, the target algorithm, and load the target algorithm. The third initialization instruction can also include information for indicating safe use to divide the NPU into TEE.

[0157] It should be noted that the execution order of the above S03-S05 and the following S06-S15 is not specific.

[0158] S06, the intelligent perception HAL sends a collection request to the TOF camera service through the VNDK interface.

[0159] S07, the smart sensing HAL sends a processing request to the smart sensing TA through the smart sensing CA, wherein the processing request is used to request the smart sensing TA to process the TOF data collected by the TOF camera, such as executing the following S17-S19 and S26.

[0160] S08: The TOF camera service sends first information to the TOF camera HAL. The first information is used to instruct the TOF camera to obtain TOF data.

[0161] S09, the TOF camera HAL sends second information to the TOF driver. The second information is used to instruct the TOF camera to obtain TOF data.

[0162] S10: The TOF driver drives the TOF camera to collect TOF data in response to the second information.

[0163] Specifically, the TOF driver responds to the second information, starts the TOF camera (if the TOF is not currently started), and collects TOF data through the first exposure parameter value.

[0164] During the initial acquisition, the first exposure parameter value is a default exposure parameter. In subsequent acquisitions, the first exposure parameter is a received exposure parameter value (also called an AE value) for indicating the next frame of TOF data.

[0165] S11, the TOF camera sends the collected TOF data to the IFE module.

[0166] The IFE module can also be called the IFE life module. The IFE module is used to transmit TOF data and can also perform preliminary processing on TOF, such as format conversion. In a specific implementation, the ISP module can convert TOF data in TOF MIPI rawdata format into TOF unpacked Raw Data (data under an existing data protocol) and send it to the secure buffer memory area for storage.

[0167] S12, the IFE module stores the TOF data and the identifier of the TOF data (such as FD) in a secure buffer memory area.

[0168] The IFE module can send the TOF data collected by the TOF camera to a secure buffer memory area (Secure buffer) through a secure ISP. The IFE module can also generate an identifier corresponding to the TOF data, such as a file descriptor (FileDescriptor, FD).

[0169] It should be understood that the storage location and reading method of the TOF data in the security buffer memory area can be represented by FD. The following is an example of the identification of the TOF data as FD. It should be understood that the identification of the TOF data is not limited to FD, and the storage location and reading method of the TOF data can also be indicated by other methods.

[0170] S13, the IFE module sends the FD of TOF data to the TOF camera HAL.

[0171] It should be understood that the IFE module can transmit the FD to the smart perception HAL through the TOF driver, kernel, TOF camera HAL, and TOF camera service in sequence. Figure 3 Some steps are omitted, such as the steps of the IFE module sending FD to the TOF driver and the TOF driver sending FD to the TOF camera HAL.

[0172] S14, the TOF camera HAL sends the FD of TOF data to the TOF camera service.

[0173] S15, the TOF camera service sends the FD of TOF data to the intelligent perception HAL.

[0174] Specifically, the TOF camera service can send the FD of TOF data to the smart perception HAL through the VNDK interface.

[0175] S16, the smart sensing HAL sends the FD of TOF data to the smart sensing CA.

[0176] S17, the smart sensing CA sends the FD of TOF data to the smart sensing TA.

[0177] S18, the intelligent sensing TA reads the TOF data corresponding to the FD from the security buffer memory area based on the FD.

[0178] After S07 and S17, the smart sensing TA can execute S18.

[0179] S19, intelligent perception TA performs eye safety detection based on TOF data.

[0180] Since the TOF camera will actively illuminate when imaging, it is necessary to ensure the safety of human eyes at this time, so it is necessary to detect whether the human eyes are safe. In some embodiments, the smart sensing TA can identify whether the TOF camera is working in the human eye safety mode based on the TOF data. If so, the human eye safety detection result is safe, otherwise, the human eye safety detection result is unsafe.

[0181] S20, when the human eye safety detection result is safe, the intelligent perception TA calculates the AE value of the next frame TOF data.

[0182] The AE value may be an exposure parameter value, an exposure time, etc.

[0183] S21, the smart perception TA sends the human eye safety detection results and the calculated AE value to the smart perception CA.

[0184] S22, the smart perception CA sends the human eye safety detection result and AE value to the smart perception HAL.

[0185] S23, the intelligent perception HAL sends the eye safety detection results and AE value to the TOF camera service through the VNDK interface.

[0186] Optionally, when the eye safety detection result is unsafe, the smart perception HAL can also send a calculation result indicating a calculation failure to the application through the smart perception service. When the application receives the calculation result indicating a calculation failure, it can execute a second operation / instruction.

[0187] For example, in an application scenario of screen unlocking, the second operation may be outputting a prompt message indicating that the unlocking has failed.

[0188] S24, the TOF camera service sends the eye safety detection result and AE value to the TOF camera HAL.

[0189] S25, when the eye safety detection result of the TOF camera HAL is safe, the TOF camera HAL sends the AE value to the TOF driver.

[0190] Optionally, when the result of the eye safety detection is unsafe, the TOF camera HAL sends an indication message to the TOF driver to instruct to turn off the TOF. After receiving the indication message, the TOF driver sends a closing command to the TOF camera to turn off the TOF camera.

[0191] S26, the TOF driver adjusts the exposure parameter of the TOF camera to the AE value.

[0192] The TOF driver can adjust the exposure parameter of the TOF camera to the AE value, so that when the TOF camera acquires the next frame of TOF data, it is exposed according to the AE value.

[0193] It should be understood that the above-mentioned human eye safety detection results and AE values ​​can be sent separately.

[0194] S27, when the result of the human eye safety detection is safe, the intelligent perception TA processes the TOF data to obtain an infrared image and a depth image.

[0195] Smart Perception TA can run image processing algorithms to convert TOF data into grayscale images and depth images.

[0196] S28, the intelligent perception TA stores the grayscale image and the depth image in a secure shared cache.

[0197] S29, after storing the grayscale image and the depth image in the secure shared cache, the smart perception TA may send a first notification to the NPU. The first notification is used to indicate that the grayscale image and the depth image have been stored, and the notification may also include the physical address or identifier of the grayscale image and the depth image in the secure shared cache.

[0198] S30: After receiving the first notification, the NPU reads the grayscale image and the depth image from the secure shared cache.

[0199] S31, the NPU operates the grayscale image and the depth image through a target algorithm to obtain an operation result.

[0200] In different scenarios, the target algorithm used may be different, and the meaning of the obtained calculation results may also be different.

[0201] For example, in the application scenarios of face unlocking, payment and verification, the target algorithm is the face recognition algorithm and the liveness verification algorithm. At this time, the NPU identifies whether the face is the owner of the machine based on the grayscale image (or grayscale image and depth image). If so, it further detects whether the face is alive based on the grayscale image and depth image. If so, it determines that the operation result can be an indication information indicating that the verification is passed (or the verification is successful). When the face is not the owner of the machine or the face is not alive, it determines that the operation result can be an indication information indicating that the verification is not passed (or the verification is unsuccessful).

[0202] As another example, in an application scenario of posture / gesture control, the target algorithm is a posture recognition algorithm or a gesture recognition algorithm. At this time, the NPU recognizes the posture or gesture based on the grayscale image (or grayscale image and depth image), and the calculation result is the recognized posture or gesture or indication information used to indicate the recognized posture or gesture.

[0203] S32, the NPU stores the calculation result in the secure shared cache.

[0204] S33, the NPU sends a second notification to the smart sensing TA. The second notification is used to indicate that the operation result has been stored or the operation has been completed, and the notification may also include a physical address or identifier of the operation result in the secure shared cache.

[0205] S34, after receiving the second notification, the intelligent sensing TA reads the calculation result from the secure shared cache.

[0206] S35, the smart sensing TA sends the calculation result to the smart sensing CA.

[0207] S36, the smart perception CA sends the calculation result to the smart perception HAL.

[0208] S37, the intelligent perception HAL sends the calculation result to the intelligent perception service.

[0209] S38, the intelligent perception service / HAL sends the calculation result to the application.

[0210] S39, the application executes the instruction corresponding to the user operation based on the calculation result.

[0211] Among them, in different reference scenarios, the instructions (also called operations) executed by the application based on the calculation results are different.

[0212] For example, in the application scenario of face unlocking, when the calculation result indicates that the verification is passed (or the verification is successful), the application unlocks the screen. Conversely, when the calculation result indicates that the verification is not passed (or the verification is unsuccessful), the smart perception reference outputs an indication message indicating the verification failure and does not unlock the screen.

[0213] As another example, in the application scenario of face payment, when the calculation result indicates that the verification is passed (or the verification is successful), the application performs the payment operation. On the contrary, when the calculation result indicates that the verification is not passed (or the verification is unsuccessful), the smart perception reference outputs the indication information for indicating the verification failure and does not perform the payment.

[0214] As another example, in the application scenario of information authentication, when the calculation result indicates that the authentication is passed (or the authentication is successful), the application outputs a prompt message indicating that the authentication is successful or enters the main interface of the website for information authentication, etc. Conversely, when the calculation result indicates that the authentication is not passed (or the authentication is unsuccessful), the smart perception reference outputs an indication message indicating that the authentication fails.

[0215] As another example, in the application scenario of posture / gesture control, the application can store operations corresponding to multiple postures or gestures, and then when the calculation result is a recognized posture or gesture, execute the instruction corresponding to the recognized posture or gesture. For example, the instruction corresponding to the gesture of sliding the hand to the right is to display the next page, and the instruction corresponding to the gesture of sliding the hand to the left is to display the previous page.

[0216] It should be noted that the above description uses intelligent perception as an example. In other implementations, Figure 3 The smart perception service, smart perception CA, smart perception TA, etc. corresponding to the smart perception application in the application are replaced by the service, CA, TA, etc. corresponding to the first application. If the first application is a setting application, the setting application has the function of recording a face, and the recorded face is used for face unlocking. The recorded face image can be stored in the NPU or secure buffer memory area on the TEE side.

[0217] Figure 4 Another image processing method provided by the embodiment of the present application can be Figure 1 The electronic device shown is implemented, and the electronic device includes TEE and REE, and specifically may include TOF camera service, TOF camera HAL, client application CA, HAL corresponding to the CA, service corresponding to CA, and trusted application TA and NPU in TEE. The method may include but is not limited to some or all of the following steps:

[0218] S41: In response to the received user operation, in the REE, TOF data is acquired through the TOF camera.

[0219] The definition of the user operation and the specific implementation of S41 can be found in the above steps S01-S17, which will not be repeated here.

[0220] Optionally, TOF data can be transmitted between REE and TEE through a secure buffer memory area. Specifically, the electronic device can store TOF data and an identifier of the TOF data in the secure buffer memory area in REE, and obtain TOF data from the secure buffer memory area based on the identifier of the TOF data in TEE. The identifier of the TOF data can be FD, which is used to indicate the storage location and reading method of TOF.

[0221] In some embodiments, the method may further include: the TOF camera sends the identification of the TOF data to the TOF camera HAL; the TOF camera HAL sends the identification of the TOF data to the TOF camera service; the TOF camera service sends the identification of the TOF data to the HAL corresponding to the CA; the HAL corresponding to the CA sends the identification of the TOF data to the CA; the CA sends the identification of the TOF data to the TA. The TA is used to obtain the TOF data from the secure buffer memory area based on the identification of the TOF data.

[0222] Above Figure 3 Steps S01-S17 in the figure exemplarily describe an implementation of S41 using the intelligent perception CA and its corresponding HAL and services. For details, please refer to the above steps S01-S17, which will not be repeated here.

[0223] S42: Process the TOF data in TEE to obtain a TOF image.

[0224] The TOF image may include a grayscale image and / or a depth image. In the TEE, the TOF data is processed by TA to obtain a TOF image. The specific implementation can refer to the above step S27, which will not be repeated here.

[0225] In one implementation, the TA is further used to process the TOF data to obtain a TOF image; the method further includes:

[0226] TA sends the TOF image to the neural network processor NPU.

[0227] Among them, an implementation of TA sending the TOF image to the neural network processor NPU may include: TA stores the TOF image in a secure shared cache area; TA sends a first notification to the NPU, the first notification being used to indicate that the TOF image has been stored; and the NPU obtains the TOF image from the secure shared cache area in response to the first notification.

[0228] S43: In the TEE, the TOF image is operated by the target algorithm running on the neural network processor NPU to obtain the operation result. The specific implementation of S43 can be referred to the above S31, which will not be repeated here.

[0229] S44: Execute the instruction corresponding to the user operation in REE according to the calculation result.

[0230] Among them, the specific implementation of S44 can refer to the above S39, which will not be repeated here.

[0231] Specifically, the specific implementation of the transmission method of the operation result between TEE and REE can be: NPU sends the operation result to TA; TA sends the operation result to CA; CA sends the operation result to the HAL corresponding to CA; the HAL corresponding to CA sends the operation result to the service corresponding to CA, and the service corresponding to CA sends the operation result to the application; further, the application executes the instruction corresponding to the user operation based on the operation result. The specific implementation can be seen in the above S32-S38, which will not be repeated here.

[0232] Optionally, one implementation of NPU sending the operation result to TA may be: NPU stores the operation result in a secure shared cache area; NPU sends a second notification to TA, the second notification is used to indicate that the operation is completed; CA responds to the second notification and reads the operation result from the secure shared cache area. For specific implementation, please refer to S32-S34 above, which will not be repeated here.

[0233] In some embodiments, after the TEE obtains the TOF data, the method may further include: the electronic device performs a human eye safety test based on the TOF data in the TEE to obtain a test result, the test result is used to indicate whether the light sent by the TOF camera is safe for the human eye; when the test result indicates that it is unsafe, stop obtaining TOF data through the TOF camera. For specific implementation, please refer to the above Figure 3 Steps S19, S21-S24 are not described here in detail.

[0234] In some embodiments, after the TEE obtains the TOF data, the method may further include: the electronic device determines the exposure parameter value of the TOF camera based on the TOF data in the TEE; and adjusts the exposure parameter of the TOF camera based on the exposure parameter value in the REE. For specific implementation, please refer to the above Figure 3 Steps S20-S26 are not described here any more.

[0235] It should be understood that each step in the above method embodiment provided by the present application can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0236] The present application also provides an electronic device, which may include: a memory and a processor. The memory may be used to store a computer program; the processor may be used to call the computer program in the memory so that the electronic device performs each function or step performed by the electronic device in the above method embodiment.

[0237] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which, when executed on the electronic device, enables the electronic device to execute the functions or steps executed by the electronic device in the method embodiment.

[0238] The present application also provides a computer-readable storage medium, which stores a computer program (also referred to as code or instruction). When the computer program runs on the electronic device, the electronic device executes each function or step executed by the electronic device in the method embodiment.

[0239] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0240] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0241] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0242] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0243] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0244] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0245] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0246] The above is only a specific implementation of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the embodiments of the present application, which should be included in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device includes a trusted execution environment TEE and a universal execution environment REE, the method includes: In response to the received user operation, acquiring TOF data through a time-of-flight TOF camera in the REE; Processing the TOF data in the TEE to obtain a TOF image; In the TEE, the TOF image is operated by a target algorithm running on a neural network processor NPU to obtain an operation result; The instruction corresponding to the user operation is executed in the REE according to the calculation result.

2. The method according to claim 1, characterized in that The method further comprises: storing the TOF data and the identification of the TOF data in a secure buffer memory area at the REE; The TEE obtains the TOF data from the secure buffer memory area based on the identifier.

3. The method according to claim 2, characterized in that The electronic device includes a TOF camera service in the REE, a TOF camera hardware abstraction layer HAL, a client application CA, a HAL corresponding to the CA, a service corresponding to the CA, and a trusted application TA in the TEE; The method further comprises: The TOF camera sends the identifier to the TOF camera HAL; The TOF camera HAL sends the identifier to the TOF camera service; The TOF camera service sends the identifier to the HAL corresponding to the CA; The HAL corresponding to the CA sends the identifier to the CA; The CA sends the identifier to the TA; The TA is used to obtain the TOF data from the security buffer memory area based on the identifier.

4. The method according to claim 3, characterized in that The TA is also used to process the TOF data to obtain the TOF image; the method also includes: The TA sends the TOF image to the neural network processor NPU.

5. The method according to claim 4, characterized in that The TA sends the TOF image to the neural network processor NPU, specifically including: The TA stores the TOF image in a secure shared cache area; The TA sends a first notification to the NPU, where the first notification is used to indicate that the TOF image has been stored; In response to the first notification, the NPU obtains the TOF image from the secure shared cache area.

6. The method according to claim 1 or 2, characterized in that: The method further comprises: Performing a human eye safety test on the TEE based on the TOF data to obtain a test result, where the test result is used to indicate whether the light sent by the TOF camera is safe for human eyes; When the detection result indicates unsafe, stop acquiring TOF data through the TOF camera.

7. The method according to claim 6, characterized in that The electronic device also includes a TOF camera service in the REE, a TOF camera hardware abstraction layer HAL, a TOF camera driver, a client application CA, a HAL corresponding to the CA, a service corresponding to the CA, and a trusted application TA in the TEE; The TA is used to perform eye safety detection based on the TOF data to obtain a detection result; The method further comprises: The TA sends the detection result to the CA; The CA sends the detection result to the HAL corresponding to the CA; The HAL corresponding to the CA sends the detection result to the TOF camera service; The TOF camera service sends the detection result to the TOF camera HAL; When the detection result is unsafe, the TOF camera HAL sends a first instruction to the TOF camera driver, where the first instruction is used to instruct the TOF camera driver to stop acquiring TOF data through the TOF camera.

8. The method according to any one of claims 1, 2 and 6, characterized in that: The method further comprises: Determining, in the TEE, an exposure parameter value of the TOF camera based on the TOF data; The REE adjusts the exposure parameter of the TOF camera based on the exposure parameter value.

9. The method according to claim 8, characterized in that The electronic device also includes a TOF camera service in the REE, a TOF camera hardware abstraction layer HAL, a TOF camera driver, a client application CA, a HAL corresponding to the CA, a service corresponding to the CA, and a trusted application TA in the TEE; The TA is used to determine the exposure parameter value of the TOF camera based on the TOF data; The method further comprises: The TA sends the exposure parameter value to the CA; The CA sends the exposure parameter value to the HAL corresponding to the CA; The HAL corresponding to the CA sends the exposure parameter value to the TOF camera service; The TOF camera service sends the exposure parameter value to the TOF camera HAL; The TOF camera HAL sends the exposure parameter value to the TOF camera driver; The TOF camera driver sets the exposure parameter of the TOF camera to the exposure parameter value.

10. The method according to claim 1, characterized in that The electronic device also includes a TOF camera service in the REE, a TOF camera hardware abstraction layer HAL, a TOF camera driver, a client application CA, a HAL corresponding to the CA, a service corresponding to the CA, and a trusted application TA in the TEE; The method further comprises: The NPU sends the calculation result to the TA; The TA sends the calculation result to the CA; The CA sends the calculation result to the HAL corresponding to the CA; The HAL corresponding to the CA sends the calculation result to the service corresponding to the CA; The service corresponding to the CA sends the calculation result to the application program; The executing, in the REE, the instruction corresponding to the user operation based on the calculation result includes the application executing, based on the calculation result, the instruction corresponding to the user operation.

11. The method according to claim 10, characterized in that The NPU sends the operation result to the TA, specifically including: The NPU stores the operation result in a secure shared cache area; The NPU sends a second notification to the TA, where the second notification is used to indicate that the operation is completed; In response to the second notification, the CA reads the operation result from the secure shared cache area.

12. The method according to any one of claims 1 to 11, characterized in that: The TOF image is operated by a target algorithm running on a neural network processor NPU in the TEE, and before the operation result is obtained, the method further includes: In response to a call instruction of the REE, the target algorithm is loaded into the NPU, wherein the call instruction includes an identifier of the target algorithm.

13. The method according to any one of claims 1 to 12, characterized in that: The method further comprises: The mode of the NPU is configured as a safe mode.

14. The method according to any one of claims 1 to 13, characterized in that: The user operation is an unlocking operation detected by the electronic device when the electronic device is in a locked screen state; the TOF image includes a grayscale image and a depth image, and the target algorithm is a face recognition algorithm and a liveness verification algorithm; The executing the instruction corresponding to the user operation according to the calculation result in the REE includes: in the REE, when the calculation result indicates that the face recognition and the liveness verification algorithm are passed, unlocking the screen.

15. The method according to any one of claims 1 to 13, characterized in that: The user operation is a payment operation detected by the electronic device in a payment interface; the TOF image includes a grayscale image and a depth image, and the target algorithm is a face recognition algorithm and a liveness verification algorithm; The executing the instruction corresponding to the user operation according to the calculation result in the REE includes: in the REE, when the calculation result indicates that the face recognition and the liveness verification algorithm are passed, payment is made.

16. The method according to any one of claims 1 to 13, characterized in that: The user operation is a user operation for turning on gesture control, the TOF image includes a grayscale image and a depth image, and the target algorithm is a gesture recognition algorithm and a liveness verification algorithm; The executing the instruction corresponding to the user operation according to the calculation result in the REE includes: in the REE, when the calculation result indicates a first posture and passes a liveness verification algorithm, executing the instruction corresponding to the first posture.

17. The method according to any one of claims 1 to 13, characterized in that: The user operation is a user operation for turning on gesture control, the TOF image includes a grayscale image and a depth image, and the target algorithm is a gesture recognition algorithm and a liveness verification algorithm; The executing the instruction corresponding to the user operation according to the calculation result in the REE includes: in the REE, when the calculation result indicates a first gesture and passes a liveness verification algorithm, executing the instruction corresponding to the first gesture.

18. An electronic device, characterized in that: It includes multiple processors and one or more memories; wherein, the multiple processors include a neural network processor NPU, the one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program codes, and the computer program codes include computer instructions. When the one or more processors execute the computer instructions, the processors execute the method described in any one of claims 1-17.

19. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 17.

Images