Face recognition method and related equipment thereof

Abstract

The invention provides a face recognition method and related equipment thereof, and relates to the technical field of terminals, and the method comprises the steps: indicating a TOF camera to operate in response to a face recognition request; based on a first frame of original data collected by the TOF camera, a target distance is determined by using a distance measurement algorithm, and the distance measurement algorithm is operated on the REE side of the electronic device; determining a target exposure value based on the target distance; indicating the TOF camera to collect second frame original data by using the target exposure value; wherein the first frame of original data and the second frame of original data are used for face recognition in sequence. According to the method and the device, the distance measurement algorithm is added on the REE side, and the exposure value of the TOF camera when the original data is acquired is adaptively adjusted by using the measured distance, so that the quality of a picture can be improved, and the accuracy and the speed of unlocking are improved.

Description

Face recognition method and related equipment Technical Field

[0001] The present application relates to the field of terminal processing, and specifically to a face recognition method and related equipment. Background Art

[0002] Currently, most smart devices use time of flight (TOF) for face unlocking. TOF face unlocking uses a TOF camera to generate images. Specifically, a beam of invisible light is emitted, reflected by an object, and then reflected back to the camera. The time difference or phase difference between emission and reflection is then calculated. This data is collected and used to generate a depth map for liveness detection, anti-counterfeiting, and comparison. This method effectively prevents liveness attacks such as photos, videos, head models, and masks, improving the security of face unlocking.

[0003] However, during the face recognition process, the quality of images obtained by the TOF camera in different scenarios may vary. If poor quality images are used for unlocking, the unlocking will fail, which will result in a very slow unlocking speed and affect the user experience.

[0004] Summary of the Invention

[0005] This application provides a face recognition method and related equipment, which improves the quality of the output image and the accuracy and speed of unlocking by adding a ranging algorithm on the REE side and using the measured distance to adaptively adjust the exposure value when the TOF camera collects raw data.

[0006] In a first aspect, a face recognition method is provided, which is applied to an electronic device including a time-of-flight (TOF) camera. The method includes: instructing the TOF camera to operate in response to a face recognition request; determining a target distance using a ranging algorithm based on a first frame of raw data collected by the TOF camera, wherein the ranging algorithm operates on a REE side of the electronic device; determining a target exposure value based on the target distance; and instructing the TOF camera to collect a second frame of raw data using the target exposure value; wherein the first frame of raw data and the second frame of raw data are used sequentially for face recognition.

[0007] The TOF camera is one of the front cameras of electronic devices.

[0008] It should be understood that the target distance is used to indicate the distance from the measured object to the TOF camera. In the embodiment of the present application, the measured object is used to indicate the user's face.

[0009] In one possible implementation, before responding to a face recognition request, the TOF camera is in an off state or in a low power consumption mode.

[0010] In a possible implementation, the ranging algorithm runs on the HAL layer of the REE side of the electronic device.

[0011] In an embodiment of the present application, based on the imaging principle of the TOF camera, the function of the TOF camera to measure distance is utilized, or the raw data collected by the TOF camera includes a distance value, to increase the ranging algorithm to determine the target distance from the user's face to the TOF camera, and determine the optimal exposure value adapted to the target distance through the target distance; thereby, the exposure value can be updated, and the quality of the second frame image collected by the TOF camera can be improved, so that the second frame RAW Data can be successfully matched with the face.

[0012] Since the ranging algorithm does not need to obtain private information such as faces during the distance calculation process, the amount of data processed is small, the algorithm itself is also small, the calculation is simple, and the calculation speed and update speed are relatively fast; therefore, compared with the existing technology, the method provided in this application can shorten the time for correcting the exposure value, so that the exposure value can be updated and configured before the TOF camera obtains the second frame of RAW Data, thereby improving the quality of the second frame of RAW Data collected by the TOF camera, using the second frame of RAW Data to successfully identify faces, and improving the accuracy and speed of face recognition.

[0013] In combination with the first aspect, in certain implementations of the first aspect, the first frame of raw data includes a distance indicating depth information of a spatial point, with one pixel corresponding to one distance; based on the first frame of raw data captured by the TOF camera, a ranging algorithm is used to determine the target distance, including: performing a weighted summation of the distances corresponding to pixels at multiple preset positions in the first frame of raw data captured by the TOF camera to determine the target distance.

[0014] In this implementation, some distance values ​​in the original data collected by the TOF camera can be directly reused to measure the target distance, which has no impact on the source of the quantity and the amount of calculation is very small.

[0015] In combination with the first aspect, in certain implementations of the first aspect, determining the target exposure value based on the target distance includes: based on the target distance, querying a preset relationship table and determining the target exposure value; the preset relationship table includes multiple pairs of distances and exposure values ​​with corresponding relationships.

[0016] In this implementation, since the distance between the object being measured and the TOF camera has a certain correspondence with the image quality captured by the TOF camera at the same exposure value, that is, the closer the distance, the better the image quality, and conversely, the farther the distance, the worse the image quality. In this regard, in order to ensure that the quality of the first frame image captured at each distance can meet the requirements of face recognition, this application selects adaptive configuration of different exposure values ​​for shooting at different distances, thereby ensuring that the quality of the first frame image at each distance can meet the requirements of face recognition and achieve the purpose of fast unlocking. Based on this, a table representing the correspondence between distance and exposure value can be generated, and a direct query can be performed when determining the target exposure value, which is fast and accurate.

[0017] In combination with the first aspect, in some implementations of the first aspect, the pixels at the multiple preset positions include five pixels located at four corners and a center position.

[0018] In this implementation, in order to balance the computational complexity and the accuracy of the distance, the five positions are more representative.

[0019] In combination with the first aspect, in certain implementations of the first aspect, the first frame of raw data is collected by the TOF camera based on a preset initial exposure value.

[0020] The initial exposure value is different from the target exposure value, and the initial exposure value is fixed.

[0021] In combination with the first aspect, in some implementations of the first aspect, the method further includes: receiving a first operation; and issuing the face recognition request in response to the first operation.

[0022] In combination with the first aspect, in some implementations of the first aspect, when the electronic device is in a screen-off state, the first operation is a movement operation on the electronic device, or a pressing operation on the power button; when the electronic device is in a screen-on state, the first operation is a double-click operation on the screen.

[0023] In combination with the first aspect, in some implementations of the first aspect, before receiving the first operation, the method also includes: displaying a first interface; wherein, the first interface includes a first icon, the first icon is used to indicate a privacy moment mode or a target application, and the first operation is a click operation on the first icon.

[0024] In combination with the first aspect, in some implementations of the first aspect, the first operation is any one of a voice operation, an air gesture operation, and a gaze operation.

[0025] In this implementation, face recognition can also be performed using a distance measurement algorithm through non-contact operations.

[0026] In a second aspect, another face recognition method is provided, which is applied to a TOF camera in an electronic device. The method includes: in response to a face recognition request, the TOF camera operates; based on a ranging algorithm integrated in the TOF camera, a target distance is determined; based on the target distance, a target exposure value is determined; and using the target exposure value, a first frame of raw data is collected and output, and the first frame of raw data is used for face recognition.

[0027] In the embodiment of the present application, since the ranging algorithm is integrated into the TOF camera to calculate the distance, and the calculation process does not require obtaining privacy information such as the face, the amount of data processed is small, the calculation is simple, and the calculation speed and update speed are relatively fast; therefore, the method provided in the present application takes a very short time to correct the exposure value, so that the exposure value can be updated and configured before the TOF camera obtains the first frame of RAW Data, thereby improving the quality of the first frame of RAW Data collected by the TOF camera, successfully identifying the face using the first frame of RAW Data, and improving the accuracy and speed of face recognition.

[0028] In combination with the second aspect, in certain implementations of the first aspect, determining the target distance based on the ranging algorithm integrated in the TOF camera includes: collecting initial raw data based on a preset initial exposure value; determining the target distance based on the initial raw data using the ranging algorithm integrated in the TOF camera; and discarding the initial raw data.

[0029] It should be understood that the initial raw data is only used to measure the target distance and is not output. Therefore, it is equivalent to outputting the raw data collected for the second time in combination with the target exposure value as the first frame raw data.

[0030] In a third aspect, an electronic device is provided. The electronic device includes a TOF camera, one or more processors, and a memory. The memory is coupled to the one or more processors, and the memory is used to store computer program code. The computer program code includes computer instructions. The one or more processors call the computer instructions to cause the electronic device to execute:

[0031] In response to a face recognition request, instruct the TOF camera to run; based on a first frame of raw data collected by the TOF camera, determine a target distance using a ranging algorithm, wherein the ranging algorithm runs on the REE side of the electronic device; determine a target exposure value based on the target distance; instruct the TOF camera to collect a second frame of raw data using the target exposure value; wherein the first frame of raw data and the second frame of raw data are used in sequence for face recognition.

[0032] In a fourth aspect, an electronic device is provided, comprising a TOF camera, one or more processors, and a memory; the memory is coupled to the one or more processors, the memory being configured to store computer program code, the computer program code comprising computer instructions, and the one or more processors invoking the computer instructions to cause the electronic device to execute:

[0033] In response to a face recognition request, the TOF camera operates; determines a target distance based on a ranging algorithm integrated in the TOF camera; determines a target exposure value based on the target distance; and uses the target exposure value to collect and output a first frame of raw data, where the first frame of raw data is used for face recognition.

[0034] In a fifth aspect, an electronic device is provided, comprising a module / unit for executing the face recognition method in the first aspect or any one of the implementations of the first aspect.

[0035] In a sixth aspect, an electronic device is provided, comprising a module / unit for executing the face recognition method in the second aspect or any one of the implementations of the second aspect.

[0036] In the seventh aspect, an electronic device is provided, comprising a TOF camera, one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code comprising computer instructions, and the one or more processors calling the computer instructions to enable the electronic device to execute the face recognition method in any aspect or any implementation of any aspect.

[0037] In an eighth aspect, a chip system is provided, which is applied to an electronic device, and the chip system includes one or more processors, and the processors are used to call computer instructions to enable the electronic device to execute any aspect or any one of the face recognition methods in any aspect.

[0038] In the ninth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code is executed by an electronic device, the electronic device executes the face recognition method in any aspect or any implementation of any aspect.

[0039] In the tenth aspect, a computer program product is provided, comprising: a computer program code, which, when executed by an electronic device, enables the electronic device to execute the face recognition method in any aspect or any implementation of any aspect.

[0040] The face recognition method provided in this application is based on the imaging principle of the TOF camera. It calculates the distance of the object being measured by adding a ranging algorithm, and adaptively adjusts the exposure value of the TOF camera through the correspondence between the distance and the exposure value, thereby improving the quality of the images subsequently collected by the TOF camera.

[0041] Here, since the ranging algorithm does not need to obtain private information such as faces in the process of calculating distance, the amount of data processed is small, the algorithm itself is also small, the calculation is simple, and the speed is relatively fast; therefore, compared with the existing technology, the method provided in this application can shorten the time for correcting the exposure value, and then quickly adjust the exposure value to an appropriate level, quickly improve the quality of the output image, and achieve the purpose of improving the accuracy and speed of face recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] FIG1 is a schematic diagram of a face unlocking scenario provided by an embodiment of the present application;

[0043] FIG2 is a hardware system of an electronic device provided in an embodiment of the present application;

[0044] FIG3 is a schematic structural diagram of a TOF camera provided in an embodiment of the present application;

[0045] FIG4 is a schematic structural diagram of a transmitter and a receiver in a TOF camera provided in an embodiment of the present application;

[0046] FIG5 is a simplified schematic diagram, circuit diagram, and timing diagram of an iTOF camera provided in an embodiment of the present application;

[0047] FIG6 is a software system of an electronic device provided by the related art;

[0048] FIG7 is a timing diagram involved in FIG6;

[0049] FIG8 is a schematic diagram showing the corresponding relationship between distance and imaging effect provided by an embodiment of the present application;

[0050] FIG9 is a schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application;

[0051] FIG10 is a timing diagram of FIG9 provided in an embodiment of the present application;

[0052] FIG11 is a schematic diagram of the software architecture of another electronic device provided in an embodiment of the present application;

[0053] FIG12 is a timing diagram of FIG11 provided in an embodiment of the present application;

[0054] FIG13 is a flow chart of a face recognition method provided in an embodiment of the present application;

[0055] FIG14 is a flow chart of a face recognition method provided in an embodiment of the present application;

[0056] FIG15 is a schematic diagram of a face unlocking interface provided by an embodiment of the present application;

[0057] FIG16 is a schematic diagram of another face unlocking interface provided by an embodiment of the present application;

[0058] FIG17 is a schematic diagram of another face unlocking interface provided by an embodiment of the present application;

[0059] FIG18 is a schematic structural diagram of an electronic device suitable for the present application. DETAILED DESCRIPTION

[0060] In the embodiments of this application, the terms "first," "second," and the like are used for descriptive purposes only and should not be understood to indicate or imply relative importance or to implicitly indicate the number of the technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0061] In order to facilitate the understanding of the embodiments of the present application, the relevant concepts involved in the embodiments of the present application are first briefly described.

[0062] 1. A rich execution environment (REE), also known as a normal or untrusted execution environment, is a mobile operating environment that can run operating systems such as Android, iOS, and Linux. REEs offer openness and scalability but are less secure.

[0063] A trusted execution environment (TEE), also known as a secure side or secure zone, is an area requiring authorization for access. The TEE coexists with the REE in an electronic device. Through hardware-based isolation from the REE, the TEE provides security and resistance to software attacks that are common to the REE. The TEE has its own operating space and defines strict protection measures, resulting in a higher level of security than the REE. It can protect TEE assets, such as data and software, from software attacks and certain types of security threats.

[0064] The REE+TEE architecture combines a TEE and REE to provide services for applications. In other words, the TEE and REE coexist within electronic devices. For example, the TEE, supported by hardware, operates in isolation from the REE. The TEE has its own operating space, offering a higher level of security than the REE, protecting assets within the TEE from software attacks. Only authorized security software can execute within the TEE, which also protects the confidentiality of the software's resources and data. Compared to the REE, the TEE offers better protection for data and resource security due to its isolation and permission control mechanisms.

[0065] TA, or trusted application, is an application running in TEE that can provide security services for applications running outside TEE, such as password input, fingerprint recognition, face recognition, etc.

[0066] CA, or client application, usually refers to an application running in REE. CA can call TA through the client application programming interface (API) and instruct TA to perform corresponding security operations.

[0067] 2. Based on the modulation method, TOF can generally be divided into two types: pulse modulation and continuous wave modulation. Pulse modulation can directly measure distance based on the time difference between pulse transmission and reception. Pulse modulation is also called dTOF (direct TOF). Continuous wave modulation measures distance based on the phase difference between the sine waves at the receiving and transmitting ends. Therefore, continuous wave modulation is also called iTOF (indirect TOF).

[0068] 3. RAW Data, or raw data, can be understood as "unprocessed and uncompressed data." In this embodiment, RAW Data refers to the raw data that the TOF sensor (TOF camera) converts into digital signals from the captured light source signal. RAW Data also contains metadata generated during the capture process.

[0069] 4. Metadata, also known as intermediary data or relay data, is data about data, primarily describing the properties of the data. In this embodiment, metadata can indicate information such as the camera's operating mode, photocurrent value, the device operating status of the TOF camera, and exposure value.

[0070] 5. Automatic exposure (AE) refers to the automatic setting of exposure values ​​by electronic devices based on available lighting conditions. Electronic devices can automatically set the shutter speed and aperture value based on the exposure value of the currently captured image to achieve automatic exposure settings.

[0071] The exposure value is a combination of exposure time and aperture value to represent the light transmission capacity of the camera lens. The exposure value can be defined as:

[0072] Where N is the aperture value; t is the exposure time in seconds.

[0073] FIG1 is a schematic diagram showing a scenario of face unlocking using a TOF camera.

[0074] As shown in FIG1 , taking the electronic device 100 as a mobile phone as an example, a user holds the mobile phone and turns the screen of the mobile phone toward the face to unlock the phone using the imaging technology of the TOF camera.

[0075] It should be understood that in addition to being a mobile phone, the electronic device 100 can also be: a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, etc. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device 100.

[0076] FIG2 shows a hardware system of an electronic device 100 suitable for the present application.

[0077] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0078] It should be noted that the structure shown in FIG2 does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in FIG2, or the electronic device 100 may include a combination of some of the components shown in FIG2, or the electronic device 100 may include sub-components of some of the components shown in FIG2. The components shown in FIG2 may be implemented in hardware, software, or a combination of software and hardware.

[0079] Exemplarily, the processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). Different processing units may be independent devices or integrated devices. The controller may generate an operation control signal based on the instruction opcode and the timing signal to complete the control of instruction fetching and execution.

[0080] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0081] In some embodiments, the processor 110 may include one or more interfaces. For example, the processor 110 may include at least one of the following interfaces: an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and a USB interface.

[0082] Exemplarily, in an embodiment of the present application, the processor 110 can be used to execute the face recognition method provided by the embodiment of the present application; for example, in response to a face recognition request, instruct the TOF camera to run; based on the first frame of raw data collected by the TOF camera, the target distance is determined using a ranging algorithm, and the ranging algorithm runs on the REE side of the electronic device; based on the target distance, the target exposure value is determined; the TOF camera is instructed to collect a second frame of raw data using the target exposure value; wherein the first frame of raw data and the second frame of raw data are used in sequence for face recognition.

[0083] 2 is merely a schematic illustration and does not limit the connection relationship between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of the multiple connection modes in the above embodiments.

[0084] The wireless communication function of the electronic device 100 can be implemented through components such as the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.

[0085] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0086] Electronic device 100 can implement display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0087] Display screen 194 may be used to display images or videos.

[0088] Optionally, the display screen 194 can be used to display images or videos. The display screen 194 includes a display panel. The display panel can use a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini light-emitting diode (Mini LED), a micro light-emitting diode (Micro LED), a micro OLED, or a quantum dot light-emitting diode (QLED). In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0089] Exemplarily, the electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0090] Exemplarily, the ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the camera to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can perform algorithmic optimization on image noise, brightness, and color. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0091] Exemplarily, the camera 193 (also referred to as a lens) is used to capture still images or videos. It can be triggered to turn on through application instructions to implement the photo function, such as capturing images of any scene. The camera may include components such as an imaging lens, a filter, and an image sensor. The light emitted or reflected by the object enters the imaging lens, passes through the filter, and is finally converged on the image sensor. The imaging lens is mainly used to converge the light emitted or reflected by all objects in the photographic field of view (also referred to as the scene to be photographed, the target scene, or the scene image that the user expects to capture) to form an image; the filter is mainly used to filter out excess light waves in the light (for example, light waves other than visible light, such as infrared); the image sensor can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The image sensor is mainly used to perform photoelectric conversion on the received light signal, convert it into an electrical signal, and then transmit the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV.

[0092] In the embodiment of the present application, the camera 193 includes a TOF camera, and the TOF camera is a front camera. It should be understood that the front camera may include multiple cameras, one of which is a TOF camera.

[0093] FIG3 shows a schematic structural diagram of a TOF camera provided in an embodiment of the present application.

[0094] For example, as shown in Figure 3, a TOF camera mainly includes a transmitter and a receiver. The transmitter can be used to transmit a light signal (infrared light or laser pulse) to an object, and the receiver is used to receive the light signal returned from the object. The distance of the object is obtained by detecting the flight (round-trip) time or phase difference of the light pulse. In the embodiment of the present application, the object to be measured is a user's face.

[0095] FIG4 is a schematic structural diagram of a transmitting end and a receiving end included in the TOF camera shown in FIG3 .

[0096] As shown in (a) of Figure 4, the transmitting end includes a light source and an optical shaping element. The light source is used to emit laser light in a specific frequency band. The light source is a vertical cavity surface emitting laser (VCSEL), which is usually a two-dimensional light source composed of many sub-light sources arranged in a two-dimensional pattern. Compared with traditional light sources, it has the advantages of small size, small divergence angle, and concentrated energy. In addition, the light source composed of VCSELs can also be called a dot matrix projector. The optical shaping element is used to shape the laser emitted by the light source, diffusing the point light source into a uniform surface light source; the optical shaping element can be a diffraction optical element (DOE) or a diffuser. The DOE is used to shape the light by diffraction.

[0097] For example, the microstructure of the optical shaping element may be as shown in FIG4( b ).

[0098] As shown in (c) in Figure 4, the receiving end may include a lens, a filter, and an image sensor. The lens is made of transparent glass and is used for protection; the filter is used to filter out unwanted stray light to reduce interference. For example, if the emitting light source is invisible infrared light, the wavelength range is around 940nm. The filter can be a near-infrared narrow-band filter to filter out unwanted stray light other than infrared light. The internal structure of the image sensor in the TOF camera is similar to that of an ordinary image sensor. It can include a photosensitive array and an A / D converter, which is used to receive the light signal returned through the filter through the photosensitive array, and then convert the light signal into a digital image signal through the A / D converter, that is, to generate RAW data.

[0099] The RAW data includes six microframes of data for each pixel. From the RAW data, the IR value of each pixel is extracted and processed to obtain an infrared grayscale image (IR); the distance value of each pixel is extracted and processed to obtain a depth map. It should be understood that a depth map is a three-dimensional image, where the horizontal and vertical coordinates correspond to the pixel position, and the grayscale value of that position corresponds to the physical distance of the spatial point indicated by the pixel from the camera. Therefore, each pixel in the depth map can represent the three-dimensional coordinates of a point in space.

[0100] Figure 5 is a schematic diagram, circuit diagram and timing diagram corresponding to an iTOF camera provided in an embodiment of the present application.

[0101] As shown in (a) of Figure 5, the modulation block generates a pulse signal of a specific frequency, typically square wave pulse modulation, which is relatively easy to implement using digital circuits. The pulse signal is sent to the transmitter, which generates infrared light of the corresponding frequency for external emission. At the same time, the modulation block also sends the pulse signal of this specific frequency to the receiver for subsequent reference. The emitted infrared light then shines on the object being measured, and some of the infrared light is reflected back, passing through the lens and irradiating the photosensitive array in the receiver. This creates a phase difference between the original transmitted signal and the received signal, and depth information can be calculated from this phase difference.

[0102] As shown in Figure 5(b), the photosensitive array at the receiving end consists of an array of photosensitive units (such as photodiodes) that convert the returned infrared light into current. These units are connected to multiple high-frequency conversion switches, such as G0 and G1 in the figure, which direct the current into different charge storage capacitors, such as S0 and S1.

[0103] The duration of a single light pulse is very short, so the above process is repeated thousands of times until the exposure time is reached. For example, if the modulation frequency is 20 MHz, the period is 50 ms, and the exposure time is 100 μs, the process is repeated 2000 times. The values ​​in the photosensitive array are then read out, and the physical distance of each spatial point can be calculated based on these values.

[0104] For example, as shown in (c) of FIG5 , the speed of light is c, tp is the duration of the light pulse, S0 is used to represent the charge collected by the earlier shutter, and S1 represents the charge collected by the delayed shutter. Then, the distance d can be calculated by the following formula:

[0105] According to the above formula, each pixel in the TOF camera can be calculated to obtain a corresponding distance d. In addition, it can be seen that the minimum measured distance d is equal to 0.

[0106] Exemplarily, the digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0107] For example, a video codec is used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. This allows the electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0108] Exemplarily, the gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, the y-axis, and the z-axis) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse motion to achieve anti-shake. The gyroscope sensor 180B can also be used in scenarios such as navigation and somatosensory games.

[0109] For example, the accelerometer 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (typically the x-axis, y-axis, and z-axis). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. The accelerometer 180E can also be used to identify the posture of the electronic device 100, which can be used as an input parameter for applications such as landscape / portrait switching and pedometers.

[0110] Exemplarily, the distance sensor 180F is used to measure distance. The electronic device 100 can measure distance by infrared or laser. In some embodiments, for example, in a shooting scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0111] For example, ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light brightness. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touches.

[0112] For example, the fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0113] For example, the touch sensor 180K is also referred to as a touch-sensitive device. The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, which is also referred to as a touch screen. The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor 180K can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100 and at a different location from the display screen 194.

[0114] In addition, an operating system is run on the above hardware. The operating system of the operating system layer can be any one or more computer operating systems that implement business processing through processes. The operating system of the electronic device 100 can include but is not limited to The embodiment of the present application is not limited to an operating system such as Harmony. Applications can be installed and run on the operating system.

[0115] The operating system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The following takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100.

[0116] 6 is a software structure diagram of an electronic device 100 provided by the related art. The framework includes an Android system layer (REE) and a TEE, and the REE and the TEE can interact with each other.

[0117] The layered architecture divides software into several layers, each with distinct roles and divisions of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into five layers: applications, application framework, hardware abstraction layer (HAL), and kernel. Below the kernel layer is a hardware layer that can operate on both REE and TEE.

[0118] As shown in Figure 6, on the REE side, the application layer may include a series of application packages. For example, the application may include: lock screen, camera, gallery, music, video (other than the lock screen, not shown in Figure 6), etc.; this embodiment of the application does not impose any restrictions on this.

[0119] Among them, the lock screen application has the function of unlocking in response to the user's unlocking operation (for example, pressing the power button). The lock screen application can perform unlocking processing such as face unlocking, fingerprint unlocking, and password unlocking. In the embodiment of this application, face unlocking is mainly used as an example for explanation.

[0120] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example, as shown in Figure 6, the application framework layer may include a camera service and a face recognition service, etc., but this embodiment of the application does not impose any restrictions on this.

[0121] The HAL layer encapsulates the Linux kernel driver, providing an interface to the upper layer and shielding the implementation details of the lower-level hardware. The HAL layer includes the Camera HAL and the Face Recognition Control Module, which includes the Face HAL and Face CA. The Camera HAL is the core software framework for the camera, while the Face Recognition Control Module is the core software framework / application for face recognition.

[0122] As shown in FIG6 , on the TEE side, an application for face recognition is also running: Face Trusted Application (Face TA), or Face TA.

[0123] The kernel layer is the layer between hardware and software. It includes at least the camera driver and display driver. The camera driver is the driver layer for the camera and is primarily responsible for interacting with the hardware.

[0124] The hardware layer includes the display, TOF camera, and secure memory (Secure Buffer). For an introduction to the display and TOF camera, refer to the descriptions of Figures 2 through 4 above and will not be repeated here. Secure memory is a security-protected memory used to store the RAW data captured by the TOF camera.

[0125] Taking the detection of a user unlocking operation as an example, the interaction process between the modules in the electronic device 100 is introduced in combination with the software system shown in Figure 6. As shown in Figure 6, the process may include the following steps 1 to 9.

[0126] Step 1: In response to the user's unlock operation, the lock screen application sends an unlock request to the face recognition service.

[0127] Step 2: The face recognition service sends a face recognition request to the face recognition control module.

[0128] Step 3: The face recognition control module sends an instruction to the camera service to run the TOF camera.

[0129] Step 4: The camera service sends a command to the camera HAL to run the TOF camera.

[0130] Step 5: The camera HAL sends a command to the camera driver to drive the TOF camera.

[0131] Step 6: The TOF camera runs.

[0132] Step 7: The TOF camera collects RAW data and sends it to secure memory for storage.

[0133] Optionally, the TOF camera can continuously collect RAW data at preset intervals.

[0134] Step 8: The face recognition TA running on the TEE side performs image processing and face comparison based on the RAW data stored in the secure memory.

[0135] Step 9: Face TA returns the comparison result to the face recognition control module. If the comparison is successful, the face recognition control module will feedback the face recognition success upward, and the upper-layer application will be successfully unlocked. If the comparison fails, face recognition can be performed again based on the next frame of RAW data collected by the TOF camera.

[0136] FIG7 is a timing diagram corresponding to the face recognition process shown in FIG6 above.

[0137] For example, in the related technology, in response to the user's unlocking operation, after triggering face recognition, the TOF camera can continuously output images at a certain speed (such as an interval of 66ms), and send the collected multiple RAW data in sequence to buffer1, buffer2 to bufferN in the secure memory for storage.

[0138] On the TEE side, the facial recognition system (TA) can obtain the first frame of RAW data in the order stored in secure memory. It then processes the first frame of RAW data to obtain the first set of infrared grayscale and depth images. It then performs face comparison with the pre-stored user image based on this set of infrared grayscale and depth images. If the first frame of face comparison is unsuccessful, it performs image processing and face comparison based on the second frame of RAW data stored in secure memory, and so on.

[0139] In the existing technology, when the TOF camera collects the first frame of RAW data after operation, it is collected according to the pre-configured, fixed initial exposure value; if the face comparison using the first frame of RAW data is unsuccessful, in order to improve the image quality of the next frame used for unlocking, the electronic device will perform AE adjustment to update the exposure value of the TOF camera; ideally, the TOF camera should obtain the second frame of RAW data based on the updated exposure value, and then perform the next round of image processing and face comparison. However, as shown in Figure 7, in fact, since the image processing process for the first frame of RAW Data may take 91ms and the face comparison process may take another 72ms, when the exposure value is updated, the TOF camera may collect and store multiple frames of RAW Data according to the original initial exposure value and the interval of 66ms; in other words, the time for image processing and face comparison based on the first frame of RAW Data in the face TA is relatively long, resulting in a delay in the exposure value update time, so that the TOF camera will collect multiple frames of RAW Data with poor quality according to the initial exposure value, and these multiple frames of RAW Data with poor quality will continuously lead to unsuccessful recognition, which in turn leads to slow unlocking speed and poor user experience.

[0140] Therefore, how to improve the accuracy and speed of recognition has become an urgent problem to be solved.

[0141] In view of this, an embodiment of the present application provides a face recognition method, which is based on the imaging principle of the TOF camera, calculates the distance of the object being measured by adding a ranging algorithm, and adaptively adjusts the exposure value of the TOF camera through the correspondence between the distance and the exposure value, thereby improving the quality of the images subsequently collected by the TOF camera.

[0142] Here, since the ranging algorithm does not need to obtain private information such as faces in the process of calculating distance, the amount of data processed is small, the algorithm itself is also small, the calculation is simple, and the speed is relatively fast; therefore, compared with the existing technology, the method provided in this application can shorten the time for correcting the exposure value, and then quickly adjust the exposure value to an appropriate level, quickly improve the quality of the output image, and achieve the purpose of improving the accuracy and speed of face recognition.

[0143] For ease of understanding, the face recognition method provided by the embodiments of the present application is described in detail below with reference to the accompanying drawings. The methods in the following embodiments can all be implemented in the electronic device 100 having the above hardware structure.

[0144] Figure 8 shows a schematic diagram of the correspondence between the distance of the measured object and the imaging effect of the TOF camera provided by an embodiment of the present application. As shown in (a) of Figure 8, based on the analysis of the unlocking scene shown in Figure 1, it can be seen that under the same exposure value, when the distance between the user's face and the TOF camera in the front camera is different, the quality (such as brightness) of the first frame image (such as the infrared grayscale image extracted from the RAW Data) obtained when the TOF camera is unlocked will be different. During face recognition, the quality of the infrared grayscale image will directly affect the accuracy and stability of the feature extraction and comparison process.

[0145] For example, as shown in (b) in FIG8 , under the same exposure value, at a closer distance D1, such as a distance of 20 cm, the image quality obtained by the TOF camera is better, the brightness is brighter, and it can well reflect the user's facial details; at an intermediate distance D2, such as a distance of 40 cm, the image quality obtained by the TOF camera is average, the brightness is relatively dark, and some facial details are unclear; at a long distance D3, such as a distance of 60 cm, the image quality obtained by the TOF camera is poor, the overall brightness is darker, and the user's face is basically blurred.

[0146] Based on the above example analysis, it can be seen that under the same exposure value, the distance between the measured object (such as the user's face) and the TOF camera has a certain corresponding relationship with the image quality captured by the TOF camera, that is, the closer the distance, the better the image quality, and conversely, the farther the distance, the worse the image quality. In this regard, in order to ensure that the quality of the first frame image collected at each distance can meet the requirements of face recognition, this application selects different exposure values ​​for adaptive configuration for shooting at different distances, thereby ensuring that the quality of the first frame image at each distance can meet the requirements of face recognition and achieve the purpose of fast unlocking.

[0147] For example, the following Table 1 is used to illustrate the corresponding relationship between the distance to the measured object and the exposure value.

[0148] Table 1

[0149] It should be understood that the above is only an example. The correspondence between the distance to the measured object and the exposure value shown in Table 1 can be generated based on experimental data, historical unlocking data, simulation data, etc., and the embodiments of the present application do not impose any limitation on this.

[0150] It should also be understood that the above Table 1 can be stored in an electronic device or in a server. When the electronic device supports triggering unlocking, it is called from the server in response to the unlocking request; in addition, when the unlocking is successful based on the newly measured distance and exposure value, the distance and exposure value can be maintained in Table 1 for subsequent reference.

[0151] Example 1

[0152] FIG9 is a software structure block diagram of an electronic device 100 provided in an embodiment of the present application.

[0153] As shown in Figure 9, the HAL layer also includes an algorithm adaptation layer and a distance measurement algorithm. The algorithm adaptation layer is responsible for the interaction between the distance measurement algorithm and the face recognition control module. When the distance measurement algorithm is running, it can query Table 1 above based on the measured distance to the object and determine the corresponding exposure value.

[0154] The other structures in FIG9 are similar to those in FIG6 , and reference may be made to the introduction of FIG6 , which will not be repeated here.

[0155] Taking the detection of a user unlocking operation as an example, the interaction process between the modules in the electronic device 100 is introduced in combination with the software system shown in Figure 9. As shown in Figure 9, the process may include the following steps S201 to S211.

[0156] S201. In response to a first operation, a face recognition service receives a face recognition request.

[0157] The face recognition request is used to request face recognition.

[0158] Exemplarily, as shown in FIG15 , when the screen is off, the first operation may be a movement operation of the user on the electronic device, such as lifting the electronic device from a horizontal state to an inclined state; in response to the movement operation, the lock screen application may send a face recognition request to the face recognition service.

[0159] Alternatively, in the screen-off state, the first operation may be a user pressing the power button of the electronic device. In response to the pressing operation, the lock screen application may send a face recognition request to the face recognition service.

[0160] Exemplarily, as shown in FIG16 , when the screen is on, the first operation may be a double-click operation on the screen. In response to the double-click operation, the lock screen application may send a face recognition request to the face recognition service.

[0161] Exemplarily, the electronic device displays a first interface, which may be the interface corresponding to the control center. The first interface may include an icon that triggers the electronic device to enter a special mode (the privacy moment mode as shown in Figure 17); in response to a click operation on the icon, the control center may send a face recognition request to the face recognition service.

[0162] For example, the electronic device may display a desktop including an icon indicating an APP; in response to a click operation on the icon, the APP may send a face recognition request to the face recognition service.

[0163] Optionally, the first operation may also include a non-contact operation between the user and the electronic device, such as voice operation, air gesture operation, gaze operation, etc. The first operation may be set and adjusted as needed, and the embodiment of the present application does not impose any restrictions on this.

[0164] S202: The face recognition service sends a face recognition request to the face recognition control module.

[0165] S203: The face recognition control module sends a capture instruction to the camera service to capture an image using the TOF camera.

[0166] S204: The camera service sends a capture instruction to the camera HAL.

[0167] S205: The camera HAL sends an acquisition instruction to the camera driver.

[0168] S206 : In response to the acquisition instruction, the TOF camera runs and acquires RAW data.

[0169] Among them, the structure and imaging principle of the TOF camera can refer to the above introduction to Figures 3 to 5, and will not be repeated here.

[0170] Optionally, when the TOF camera is in a power-off state, upon receiving an acquisition instruction, the TOF camera can be powered on and start acquiring RAW data; or, the TOF camera can also be an AO (always on) camera, which always runs in low power mode. Upon receiving an acquisition instruction, it switches to normal mode and directly starts acquiring RAW data.

[0171] For example, the TOF camera can continuously collect RAW data at a preset interval, and the preset interval can be pre-set as needed, such as 50ms, 66ms, etc.

[0172] S207. The TOF camera sends the RAW data to the secure memory for storage.

[0173] S208 , the TOF camera sends the RAW data to the distance measurement algorithm, and uses the distance measurement algorithm to determine the target distance of the measured object.

[0174] Optionally, the distance measurement algorithm can extract the distance values ​​included in the pixels at multiple preset positions from the RAW data, and then use the weighted sum of the values ​​as the target distance corresponding to the measured object. In this embodiment of the application, the target distance is used to indicate the distance between the user's face and the TOF camera.

[0175] It should be understood that in order to balance the amount of calculation and the accuracy of the distance, this application selects the distance values ​​included in the pixels at multiple representative preset positions and uses the weighted sum of the distance values ​​as the target distance corresponding to the measured object. Of course, the target distance can also be determined in other ways, or the distance corresponding to the pixel at the center position can be directly used as the target distance. This embodiment of the application does not impose any restrictions on this.

[0176] For example, the ranging algorithm can extract the distance values ​​corresponding to the five pixels at the four corners and the center position from the received RAW Data, and then use the weighted summation result as the target distance corresponding to the object being measured; wherein the weight can be set as needed, for example, a larger weight can be assigned to the pixel at the center position. The specific setting and change can be made as needed, and the embodiment of the present application does not impose any restrictions on this.

[0177] The principle of TOF camera collecting RAW data can be referred to the above introduction to Figure 5, which will not be repeated here.

[0178] It should be understood that S207 and S208 can be executed simultaneously, or S207 can be executed first and then S208; or S208 can be executed first and then S207. The embodiments of the present application do not impose any restrictions on this.

[0179] Optionally, as another implementation, the ranging algorithm may also extract RAW Data from the secure memory to calculate the target distance.

[0180] S209 , the face recognition control module receives the target distance determined by the distance measurement algorithm, and determines the target exposure value corresponding to the target distance by looking up a table.

[0181] Among them, the determined target exposure value can be sent to the TOF camera for use through the face recognition control module, camera service, camera HAL, and camera driver.

[0182] It should be understood that the table stores multiple distances and multiple exposure values, as well as the corresponding relationship between distances and exposure values. Based on the determined target distance, the face recognition control module can quickly determine the optimal exposure value for capturing images using TOF at that distance by looking up the table.

[0183] The table may be in the form shown in Table 1 above, which will not be described in detail here.

[0184] Optionally, if the table does not record the exposure value corresponding to the current distance, the optimal exposure value corresponding to the current distance can be calculated by interpolation based on the exposure values ​​corresponding to the two distance values ​​closest to the current distance, such as an adjacent distance value that is smaller than the current distance and an adjacent distance value that is larger than the current distance. Of course, other algorithms can also be used for calculation, and this embodiment of the application does not impose any restrictions on this.

[0185] After the exposure value is corrected, the camera service sends the updated exposure value to the TOF camera to control the TOF camera to collect RAW data according to the new exposure value.

[0186] S210: The face TA running on the TEE side performs image processing and face comparison based on the RAW data stored in the secure memory.

[0187] An IR map can be obtained based on the IR value extracted from the RAW data, and a depth map can be obtained based on the distance value extracted from the RAW data; then, face comparison can be performed based on the IR map and the depth map.

[0188] Optionally, the image processing may include: RAW to IR image conversion, RAW to depth map conversion, and AE processing.

[0189] Among them, RAW to IR image is used to extract IR-related data from RAW data to obtain an IR map, and RAW to depth map is used to extract distance data from RAW data to obtain a depth map; AE processing is used to determine the exposure value based on the IR image and depth map.

[0190] It should be noted that the ranging algorithm determines the target distance based on the first frame of RAW Data. The purpose is to make the exposure value of the second frame acquired by the TOF camera roughly accurate and improve the quality of the second frame. The AE processing is performed on each frame of RAW Data collected by the TOF camera. The purpose is to provide the TOF camera with a more accurate exposure value.

[0191] Optionally, face comparison may include face detection, IR liveness detection, eye liveness detection, depth liveness detection, feature extraction / comparison, etc.

[0192] It should be understood that image processing and face comparison may also include other steps, and the detection order of multiple steps may be set and adjusted as needed, which is not limited in the embodiments of the present application.

[0193] S211. The face TA returns the comparison result to the face recognition control module. If the comparison is successful, the face recognition control module will feedback the success of face recognition upward. If the comparison fails, the comparison can be performed again based on the next frame of RAW data collected by the TOF camera.

[0194] Exemplarily, after S211, if the comparison is successful, the face recognition control module will feedback that the face recognition is successful. As shown in FIG15 , the electronic device can switch from the screen-off state to the desktop-displaying state; or, as shown in FIG16 , the electronic device can switch from the screen-on state to the desktop-displaying state.

[0195] For example, if the comparison is unsuccessful, the electronic device can switch from the screen-off state to the screen-on state, and display a prompt on the lock screen interface, such as "Face recognition failed", to prompt the user that the comparison was not successful and that face recognition needs to continue to unlock, or the user can choose to use other methods to unlock.

[0196] FIG10 is a timing diagram corresponding to the face recognition process shown in FIG9 above.

[0197] For example, in an embodiment of the present application, in response to the user's first operation, after triggering face recognition, the TOF camera can output images at a certain speed (such as an interval of 66ms) and send the collected multiple RAW data in sequence to buffer0, buffer1, buffer2 to bufferN of the secure memory for storage.

[0198] The TOF camera also sends the collected first frame RAW Data to the ranging algorithm. The ranging algorithm extracts the distance values ​​corresponding to the pixels at preset positions (such as the four corners and the center position) from the first frame RAW Data and performs a weighted sum calculation. Then, the ranging algorithm uses the calculation result as the target distance value, and, based on the target distance, finds the exposure value corresponding to the target distance from the table used to reflect the correspondence between distance and exposure value as the optimal exposure value applicable when using TOF to collect images at this distance. Then, it is sent to the TOF camera through the face recognition control module, camera service, etc. The TOF camera collects subsequent RAW Data based on the updated exposure value.

[0199] Here, since the ranging algorithm has a small amount of calculation and the exposure value is corrected quickly, the exposure value can usually be updated before the TOF camera obtains the second frame. Therefore, the TOF camera can obtain the RAW data of the second frame and subsequent frames based on the corrected exposure value and store it in the secure memory.

[0200] On the TEE side, the face agent can retrieve the first frame of RAW data, starting with the first frame, according to the order stored in secure memory buffer 0. Image processing and facial comparison are performed on this first frame. If facial comparison fails on the first frame, this indicates that the image quality captured by the TOF camera using the previously configured initial exposure value is insufficient and cannot meet facial recognition requirements. The face agent can then retrieve the second frame of RAW data from secure memory buffer 1 and perform image processing and facial comparison again. Because the second frame of RAW data is captured based on the corrected exposure value, which is the optimal exposure value at the current distance, facial comparison based on this second frame of RAW data is successful, leading to successful unlocking.

[0201] In addition, when face matching is successful based on the first frame of RAW Data, it means that the image quality captured by the TOF camera using the originally configured initial exposure value can meet the requirements of face recognition. Although the exposure value is updated based on the target distance, a second frame of face recognition is no longer required.

[0202] In the face recognition method provided in the present application, based on the imaging principle of the TOF camera, the function of the TOF camera to measure distance is utilized, or the data collected by the TOF camera includes a distance value, to increase the distance measurement algorithm to determine the target distance from the user's face to the TOF camera, and by looking up the correspondence table between the preset distance and the exposure value, to determine the optimal exposure value adapted to the target distance; thereby, the exposure value can be updated, and the quality of the second frame image collected by the TOF camera can be improved, so that the second frame RAW Data can be successfully matched with the face.

[0203] In this embodiment, since the ranging algorithm does not need to obtain private information such as faces during the distance calculation process, the amount of data processed is small, the algorithm itself is also small, the calculation is simple, and the calculation speed and update speed are relatively fast; therefore, compared with the existing technology, the method provided in this application can shorten the time for correcting the exposure value, so that the exposure value can be updated and configured before the TOF camera obtains the second frame of RAW Data, thereby improving the quality of the second frame of RAW Data collected by the TOF camera, successfully identifying the face using the second frame of RAW Data, and improving the accuracy and speed of face recognition.

[0204] Example 2

[0205] FIG11 is a software structure block diagram of another electronic device 100 provided in an embodiment of the present application.

[0206] As shown in Figure 11, a distance measurement algorithm for calculating the distance of the measured object can be integrated into the TOF camera at the hardware layer. In this way, after collecting the first frame of RAW Data, the TOF camera can extract the distance corresponding to the pixel at the preset position from the first frame of RAW Data, and use the distance measurement algorithm within the TOF camera to determine the target distance; then, the TOF camera sends the target distance to the face recognition control module.

[0207] The other structures in FIG11 are similar to those in FIG6 , and reference may be made to the introduction of FIG6 , which will not be repeated here.

[0208] Taking the detection of a user unlocking operation as an example, the interaction process between the modules in the electronic device is introduced in combination with the software system shown in Figure 11. As shown in Figure 11, the process may include the following steps S301 to S311.

[0209] S301: In response to a first operation, a face recognition service receives a face recognition request.

[0210] For the introduction of S301 , please refer to the above description of S201 , which will not be repeated here.

[0211] S302: The face recognition service sends a face recognition request to the face recognition control module.

[0212] S303: The face recognition control module sends a capture instruction to the camera service to capture an image using the TOF camera.

[0213] S304: The camera service sends a capture instruction to the camera HAL.

[0214] S305: The camera HAL sends an acquisition instruction to the camera driver.

[0215] S306 : In response to the acquisition instruction, the TOF camera runs and acquires RAW data.

[0216] At the same time, the ranging algorithm in the TOF camera determines the target distance of the measured object based on the RAW data, and based on the target distance, determines the corresponding target exposure value by looking up the table.

[0217] S307 , the TOF camera sends the RAW data collected using the target exposure value to the secure memory for storage.

[0218] It should be understood that in the TOF camera, since the ranging algorithm is integrated, it is equivalent to updating the exposure value internally before the TOF camera outputs the image; among them, the RAW Data collected based on the initial exposure value can be directly discarded after being used to calculate the target distance, and does not need to be output to the secure memory for storage, while the RAW Data collected using the target exposure value can be stored as the first frame data collected.

[0219] S308. The face TA running on the TEE side performs image processing and face comparison based on the RAW data stored in the secure memory.

[0220] S309 , the face TA returns the result of successful comparison to the face recognition control module; the face recognition control module feeds back the success of face recognition.

[0221] It should be understood that since the RAW data is collected based on the optimal exposure value found based on the distance, the face comparison can be successful.

[0222] FIG12 is a timing diagram corresponding to the face recognition process shown in FIG11 above.

[0223] For example, in an embodiment of the present application, in response to the user's first operation, after triggering face recognition, the TOF camera can output images at a certain speed (such as an interval of 66ms) and send the collected multiple RAW data in sequence to buffer0, buffer1, buffer2 to bufferN of the secure memory for storage.

[0224] Before the TOF camera outputs an image, the ranging algorithm integrated in the TOF camera can extract the distance values ​​corresponding to the pixels at preset positions (such as the four corners and the center position) from the RAW data collected according to the initial exposure value, and obtain the target distance after weighted summation. Moreover, based on the target distance, the exposure value corresponding to the target distance is found from the table used to reflect the correspondence between distance and exposure value as the optimal exposure value applicable when using the TOF camera to collect images at this distance; then, based on the determined optimal exposure value, the first frame of RAW data is collected and output.

[0225] Here, since the ranging algorithm is integrated into the TOF camera, the ranging algorithm has a small amount of calculation and the exposure value is corrected very quickly. Therefore, the exposure value can usually be updated before the TOF camera obtains the first frame of RAW data. As a result, the TOF camera can obtain the first frame of RAW data based on the corrected exposure value and store it in the secure memory.

[0226] On the TEE side, the face TA can obtain the RAW data starting from the first frame according to the storage order in secure memory buffer0, and perform image processing and facial comparison on the first frame of RAW data. In this embodiment, because the first frame of RAW data is obtained based on the corrected exposure value, the image quality can meet the requirements of facial recognition. Therefore, when performing facial comparison based on the first frame of RAW data, the comparison can be successful, and then the unlocking is successful.

[0227] In the face recognition method provided in the present application, based on the imaging principle of the TOF camera, the function of the TOF camera to measure distance is utilized or the data collected by the TOF camera includes a distance value, to increase the distance measurement algorithm to determine the target distance from the user's face to the TOF camera, and by looking up the correspondence table between the preset distance and the exposure value element, to determine the optimal exposure value adapted to the target distance; thereby, the exposure value can be updated immediately on the TOF camera, thereby improving the quality of the first frame of RAW Data collected by the TOF camera, so that the first frame of RAW Data can be successfully matched with the face.

[0228] Here, the RAW data collected using the initial exposure value can be discarded and not output after completing the distance measurement, while the RAW data collected using the updated exposure value can be output as the first frame of RAW data.

[0229] In this embodiment, since the ranging algorithm is integrated into the TOF camera, the ranging algorithm does not need to obtain private information such as faces during the distance calculation process. The amount of data processed is small, the algorithm itself is also small, the calculation is simple, and the calculation speed and update speed are relatively fast. Therefore, compared with the existing technology, the method provided in this application can shorten the time for correcting the exposure value, so that the exposure value can be updated and configured before the TOF camera obtains the first frame of RAW Data, thereby improving the quality of the first frame of RAW Data collected by the TOF camera, successfully identifying the face using the first frame of RAW Data, and improving the accuracy and speed of face recognition.

[0230] It should also be understood that since the ranging algorithm of Example 2 is integrated into the TOF camera, compared to Example 1, the exposure value is updated faster and RAW data with quality that meets the requirements for face recognition is obtained earlier. Therefore, the speed of face recognition is relatively faster.

[0231] FIG13 shows a flow chart of a face recognition method provided in an embodiment of the present application.

[0232] As shown in FIG. 13 , the method may apply the HAL layer in the electronic device 100 . The method 400 may include the following S410 to S450 , which will be described in detail below.

[0233] S410: In response to the face recognition request, instruct the TOF camera to operate.

[0234] S420: Determine the target distance based on the first frame of raw data collected by the TOF camera.

[0235] It should be understood that the RAW Data includes a distance value indicating depth information of a spatial point. Therefore, the distance corresponding to a pixel point at a preset position can be extracted from the RAW Data to calculate the target distance.

[0236] Optionally, when calculating the target distance, the calculation may be performed according to the relevant steps of the distance measurement algorithm introduced in S208 above, which will not be described in detail here.

[0237] The first frame of RAW data is collected according to the initial exposure value, which is a preset fixed value. The target distance is used to indicate the distance from the object to be measured to the TOF camera.

[0238] S430: Determine a target exposure value corresponding to the target distance based on the target distance.

[0239] Optionally, the target exposure value corresponding to the target distance is searched and determined from a preset relationship table, wherein the preset relationship table includes a plurality of pairs of distance values ​​and exposure values ​​having corresponding relationships.

[0240] Furthermore, the present application can also use historical data to calculate the correspondence between distance and exposure value to obtain a relational expression or a set of relations. Thus, when the target distance is determined, the corresponding relational expression or set of relations can be substituted into the relational expression or set of relations to calculate the corresponding exposure value. Of course, other methods can also be used to determine the target exposure value corresponding to the target distance. The concepts behind these methods all originate from the correspondence between distance and exposure value, and the present application embodiments do not impose any limitations on this.

[0241] S440: Instruct the TOF camera to collect the second frame of raw data using the target exposure value.

[0242] S450: If face recognition fails on the first frame of original data, perform face recognition on the second frame of original data.

[0243] Optionally, image processing and face comparison may be performed on both the first frame of RAW Data and the second frame of RAW Data.

[0244] For the introduction of image processing and face comparison, please refer to the description in S210 above, which will not be repeated here.

[0245] In the embodiment of the present application, since it is not necessary to obtain private information such as the face during the distance calculation process, the amount of data processed is small, the calculation is simple, and the calculation speed and update speed are relatively fast; therefore, the method provided in the present application takes a very short time to correct the exposure value, so that the exposure value can be updated and configured before the TOF camera obtains the second frame of RAW Data, thereby improving the quality of the second frame of RAW Data collected by the TOF camera, successfully identifying the face using the second frame of RAW Data, and improving the accuracy and speed of face recognition.

[0246] FIG14 shows a flow chart of another face recognition method provided in an embodiment of the present application.

[0247] As shown in FIG. 14 , the method may be applied to the TOF camera of the electronic device 100 . The method 500 may include the following S510 to S540 , which will be described in detail below.

[0248] S510: In response to a face recognition request, the TOF camera operates.

[0249] S520, the TOF camera collects raw data and determines the target distance.

[0250] The target distance is used to indicate the distance from the measured object to the TOF camera.

[0251] S530: The TOF camera determines a target exposure value corresponding to the target distance from a preset relationship table based on the target distance.

[0252] S540 and the TOF camera collect raw data again in combination with the target exposure value and output it as the first frame of raw data.

[0253] The previously collected raw data can be discarded after the target distance is determined, and the raw data collected again can be output as the first frame of raw data for face recognition.

[0254] Optionally, image processing and face comparison can be performed on the first frame of raw data.

[0255] For the introduction of image processing and face comparison, please refer to the description in S210 above, which will not be repeated here.

[0256] In the embodiment of the present application, since the distance calculation is integrated into the TOF camera and the calculation process does not require obtaining privacy information such as the face, the amount of data processed is small, the calculation is simple, and the calculation speed and update speed are relatively fast; therefore, the method provided in the present application takes a very short time to correct the exposure value, so that the exposure value can be updated and configured before the TOF camera obtains the first frame of RAW Data, thereby improving the quality of the first frame of RAW Data collected by the TOF camera, using the first frame of RAW Data to successfully identify the face, and improving the accuracy and speed of face recognition.

[0257] Figure 15 is a schematic diagram of the interface of a face unlocking scenario provided in an embodiment of the present application.

[0258] As shown in (a) of FIG15 , the electronic device is in a screen-off state. In this state, a user's movement operation on the electronic device may be received, such as lifting the electronic device from a horizontal state to a tilted state; in response to the movement operation, the lock screen application in the electronic device may send a face recognition request to the face recognition service; or, in this state, a user's pressing operation on the power button of the electronic device may be received; in response to the pressing operation, the lock screen application in the electronic device may send a face recognition request to the face recognition service; in response to the face recognition request, the face recognition method provided in the embodiment of the present application is executed; when the recognition is successful, the display may be switched to the desktop as shown in (b) of FIG15 . By combining the face recognition method provided in the embodiment of the present application, the accuracy and speed of unlocking are relatively high.

[0259] Figure 16 is a schematic diagram of the interface of another face unlocking scenario provided in an embodiment of the present application.

[0260] As shown in (a) of FIG16 , the electronic device is in a screen-on state. In this state, a double-click operation by the user on the display screen of the electronic device can be received; in response to the double-click operation, the lock screen application in the electronic device can send a face recognition request to the face recognition service; or, in this state, any one of a user's voice operation, air gesture operation, gaze operation, etc. can be received. In response to any operation, the lock screen application in the electronic device can send a face recognition request to the face recognition service; in response to the face recognition request, the face recognition method provided in the embodiment of the present application is executed; when the recognition is successful, the display can be switched to the desktop as shown in (b) of FIG16 . By combining the face recognition method provided in the embodiment of the present application, the accuracy and speed of unlocking are relatively high.

[0261] The face recognition method provided in the embodiments of the present application can be used in the face unlocking scenarios shown in Figures 15 to 16 above, and can also be used in application scenarios that trigger special modes or special APPs.

[0262] FIG17 is a schematic diagram of a scenario for entering a privacy moment mode provided in an embodiment of the present application.

[0263] In response to a user clicking a button indicating entry into Private Moments mode, the electronic device may invoke the facial recognition method provided in an embodiment of the present application to perform facial recognition. Simultaneously, the screen may display an interface as shown in FIG17(a), displaying a prompt prompting the user to enter Private Moments mode after verifying facial information. Upon successful facial recognition, Private Moments mode may be entered, and the electronic device may switch to displaying the desktop as shown in FIG17(b).

[0264] It should be noted that here, the "private moment mode" is the name of an application management mode, which may also be called super privacy mode, high security mode, etc., and this application does not limit this. After enabling the private moment module, the electronic device can retain only the functions of a few native system apps with very high security, disable other functions, and close other apps with lower security, thereby achieving state isolation in high security mode.

[0265] In addition to the above application scenarios, the face recognition method provided in the embodiments of the present application may also include but is not limited to the following scenarios:

[0266] For example, face-related application scenarios include face payment scenarios, background blur, portrait lighting effects, animated expressions, three-dimensional beauty, liveness detection, and vision correction.

[0267] It should be understood that the above is an example of an application scenario and does not limit the application scenario of this application.

[0268] Figure 18 shows a schematic diagram of the structure of another electronic device provided by the present application. The dotted lines in Figure 18 indicate that the unit or module is optional; the electronic device 600 can be used to implement the face recognition method described in the above method embodiment.

[0269] The electronic device 600 includes one or more processors 601, which can support the face recognition method in the method embodiment implemented by the electronic device 600. The processor 601 can be a general-purpose processor or a special-purpose processor. For example, the processor 601 can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0270] Optionally, the processor 601 may be used to control the electronic device 600, execute software programs, and process data of the software programs. The electronic device 600 may further include a communication unit 605 for implementing signal input (reception) and output (transmission).

[0271] For example, the electronic device 600 may be a chip, the communication unit 605 may be an input and / or output circuit of the chip, or the communication unit 605 may be a communication interface of the chip, and the chip may be a component of a terminal device or other electronic device.

[0272] For another example, the electronic device 600 may be a terminal device, and the communication unit 605 may be a transceiver of the terminal device, or the communication unit 605 may be a transceiver circuit of the terminal device.

[0273] The electronic device 600 may include one or more memories 602 on which a program 604 is stored. The program 604 can be executed by the processor 601 to generate instructions 603, so that the processor 601 executes the face recognition method described in the above method embodiment according to the instructions 603.

[0274] Optionally, data may also be stored in the memory 602 .

[0275] Optionally, the processor 601 may also read data stored in the memory 602 . The data may be stored at the same storage address as the program 604 , or may be stored at a different storage address from the program 604 .

[0276] Optionally, the processor 601 and the memory 602 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of the terminal device.

[0277] Exemplarily, the memory 602 may be used to store a program 604 related to the face recognition method provided in an embodiment of the present application, and the processor 601 may call a program 804 related to the face recognition method stored in the memory 602 to execute the face recognition method of the embodiment of the present application. For example, in response to a face recognition request, the TOF camera is instructed to operate; based on the first frame of raw data collected by the TOF camera, a distance to the target is determined using a ranging algorithm, where the ranging algorithm operates on the REE side of the electronic device; based on the target distance, a target exposure value is determined; the TOF camera is instructed to collect a second frame of raw data using the target exposure value; wherein the first frame of raw data and the second frame of raw data are sequentially used for face recognition.

[0278] Optionally, the present application also provides a computer program product, which, when executed by the processor 601, implements the face recognition method in any method embodiment of the present application.

[0279] For example, the computer program product may be stored in the memory 602 , such as a program 604 , which is converted into an executable target file that can be executed by the processor 601 after undergoing preprocessing, compilation, assembly, and linking.

[0280] Optionally, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the face recognition method of any method embodiment of the present application. The computer program may be a high-level language program or an executable target program.

[0281] For example, the computer-readable storage medium is memory 602. Memory 602 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0282] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0283] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0284] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the embodiments of the electronic device described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0285] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0286] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0287] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0288] In addition, the term "and / or" in this document simply describes an association between related objects, indicating that three possible relationships exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0289] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0290] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A face recognition method, characterized in that, Applied to an electronic device including a TOF camera, the method includes: In response to a face recognition request, instruct the TOF camera to operate; Based on the first frame of raw data collected by the TOF camera, use a ranging algorithm to determine the target distance, and the ranging algorithm runs on the REE side of the electronic device; Based on the target distance, determine the target exposure value; Instruct the TOF camera to collect the second frame of raw data using the target exposure value; Wherein, the first frame of raw data and the second frame of raw data are used for face recognition in sequence.

2. The method according to claim 1, wherein The first frame of raw data includes distances indicating the depth information of spatial points, and one pixel corresponds to one distance; Based on the first frame of raw data collected by the TOF camera, using a ranging algorithm to determine the target distance includes: After performing weighted summation on the distances corresponding to the pixels at multiple preset positions in the first frame of raw data collected by the TOF camera, determine the target distance.

3. The method according to claim 2, wherein Based on the target distance, determining the target exposure value includes: Based on the target distance, query a preset relationship table and determine the target exposure value; the preset relationship table includes multiple pairs of distances and exposure values with corresponding relationships.

4. The method according to claim 2 or 3, characterized in that, The pixels at the multiple preset positions include five pixels located at the four corners and the center position.

5. The method according to any one of claims 1 to 3, characterized in that, The first frame of raw data is collected by the TOF camera based on a preset initial exposure value.

6. The method according to any one of claims 1 to 5, characterized in that The method further includes: Receive a first operation; In response to the first operation, issue the face recognition request.

7. The method according to claim 6, wherein When the electronic device is in the screen-off state, the first operation is a movement operation on the electronic device or a pressing operation on the power button; When the electronic device is in the screen-on state, the first operation is a double-click operation on the screen.

8. The method according to claim 6, characterized in that Before receiving the first operation, the method further includes: displaying a first interface; Wherein, the first interface includes a first icon, and the first icon is used to indicate the privacy moment mode or the target application, and the first operation is a click operation on the first icon.

9. The method according to claim 6, wherein The first operation is any one of a voice operation, an air gesture operation, and a gaze operation.

10. A face recognition method, characterized in that, Applied to a TOF camera in an electronic device, the method includes: In response to a face recognition request, the TOF camera operates; Based on a ranging algorithm integrated in the TOF camera, determine the target distance; Based on the target distance, determine the target exposure value; Collect and output the first frame of raw data using the target exposure value, and the first frame of raw data is used for face recognition.

11. The method according to claim 10, characterized in that, Based on a ranging algorithm integrated in the TOF camera, determining the target distance includes: Collect initial raw data based on a preset initial exposure value; Based on the initial raw data, use the ranging algorithm integrated in the TOF camera to determine the target distance; Discard the initial raw data.

12. An electronic device, characterized in that, Includes: A TOF camera, one or more processors, and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute the method according to any one of claims 1 to 9, or, according to claim 10 or 11.

13. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system includes one or more processors, and the processors are used to call computer instructions to cause the electronic device to execute the method according to any one of claims 1 to 9, or, according to claim 10 or 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program runs on an electronic device, it causes the electronic device to execute the method according to any one of claims 1 to 9, or, according to claim 10 or 11.

Images