Image processing method and electronic equipment

Abstract

The invention provides an image processing method and electronic equipment, and the method carries out the processing of TOF data collected by a TOF camera through a low-power-consumption ISP, a low-power-consumption DSP and a low-power-consumption NPU in a low-power-consumption module when a smart perception service is completed, can carry out the processing of TOF RAW data without relying on a CPU, obtains a service result, and returns the service result to an application layer for response. The power consumption of the electronic equipment during TOF data processing can be effectively reduced, and the endurance of the electronic equipment is prolonged.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and in particular to an image processing method and electronic equipment. Background Art

[0002] Currently, applications such as terminal unlocking, payment, and identity authentication all involve processing sensitive images such as faces and eyes. Due to requirements for accuracy and ambient brightness, electronic devices rely on Time of Flight (TOF) cameras to perform some of these tasks. However, current electronic devices require extremely complex steps to process TOF RAW data, resulting in high power consumption when using TOF cameras, shortening battery life and impacting user experience.

[0003] Therefore, how to control the power consumption of electronic devices when processing TOF RAW data is an urgent problem to be solved. Summary of the Invention

[0004] An embodiment of the present invention provides an image processing method and an electronic device. When completing intelligent perception services, the method processes the TOF data collected by the TOF camera through the low-power ISP, low-power DSP, and low-power NPU in the low-power module. The TOF RAW data can be processed without relying on the CPU, and the service results are obtained and returned to the application layer for response. This can effectively reduce the power consumption of the electronic device when processing TOF data and extend the battery life of the electronic device.

[0005] In a first aspect, the present application provides an image processing method, which is applied to an electronic device, wherein the electronic device includes a time-of-flight TOF camera, a first image signal processor ISP, a first digital signal processor DSP, a first neural network processor NPU and an integrated circuit bus I 2 C, the method includes: collecting first data through the TOF camera, the first data including image data and image metadata, the image data is transmitted to the first DSP via the first ISP, and the image metadata is transmitted to the first DSP via the I 2 C is transmitted to the first DSP; the image metadata is processed by the first DSP to obtain first information, and the first information indicates whether the TOF camera is safe for human eyes; when the first information indicates that the TOF camera is safe for human eyes, the image data is processed by the first DSP to obtain a depth image and a grayscale image.

[0006] In this method, the first data may be TOF RAW data. The first ISP and the first DSP may be low-power ISPs and low-power DSPs. Unlike conventional ISPs and DSPs used in electronic devices to process most image processing, the first ISP and the first DSP may be low-power ISPs and low-power DSPs with lower performance but capable of performing some image processing operations. Specifically, the low-power modules of the first ISP and the first DSP can replace conventional ISPs, conventional DSPs, and CPUs to process TOF data, generating the depth image and grayscale image used to generate service results in smart sensing services. The grayscale image generated from TOF RAW data can also be called an infrared grayscale image. Because the reflectivity of the illuminated material under infrared light differs from that under visible light, infrared grayscale images exhibit image features different from those of typical visible light grayscale images. For example, a face may exhibit features such as bright / dark pupils, a reflective nose tip, and dark cheeks. When using the first ISP and the first DSP to process TOF data, the electronic device can obtain the depth image and grayscale image without waking up the CPU or the entire Android system, effectively reducing the power consumption of the electronic device.

[0007] In combination with the first aspect, in one possible implementation, the collecting of first data through the TOF camera includes: collecting first data through the TOF camera in response to a service request; after obtaining the depth image and the grayscale image, the method further includes: performing calculations on the depth image and the grayscale image through the first NPU to obtain a calculation result of the service corresponding to the service request.

[0008] In this embodiment, the services corresponding to the service request may include but are not limited to: 1. gesture recognition service; 2. device status recognition service and user status recognition service.

[0009] In some embodiments of the present application, some smart sensing services can be completed by ordinary cameras (such as RGB cameras), but some smart sensing services have high requirements for accuracy and ambient brightness, so these smart sensing services need to be completed based on TOF cameras.

[0010] In combination with the first aspect, in a possible implementation, the first electronic device further includes a central processing unit (CPU), and after responding to the service request, the CPU of the electronic device is in a dormant state.

[0011] Since the transmission link of TOF data is simplified when the first ISP and the first DSP are used to process TOF data, the first ISP and the first DSP do not rely on the CPU when processing TOF RAW data. The electronic device can obtain the depth image and the grayscale image without waking up the CPU and the entire Android system, which can effectively reduce the power consumption of the electronic device.

[0012] In addition, since the first data collected by the TOF camera is directly transmitted through two data paths (i.e., through I 2 C and the second ISP) are transmitted to the DSP without entering the buffer storage area. Therefore, in an embodiment, the DDR of the electronic device will not participate in the processing of the TOF RAW data (that is, after responding to the service request, the DDR of the electronic device can also be in a dormant state), and the power consumption of the electronic device is further reduced.

[0013] In combination with the first aspect, in a possible implementation manner, the method further includes: processing the image data by the first DSP to obtain second information, and the second information is transmitted to the I 2 C is transmitted to the TOF camera, and the second information represents the exposure parameters used by the TOF camera when collecting the next frame of data.

[0014] In this embodiment, the electronic device can dynamically adjust the exposure parameters of the TOF camera to obtain higher quality TOF data.

[0015] In combination with the first aspect, in a possible implementation, the method further includes: when the first information indicates that the TOF camera is unsafe for human eyes, stopping data collection through the TOF camera; when the first information indicates that the TOF camera is safe for human eyes, controlling the TOF camera to continue collecting data.

[0016] In this embodiment, the electronic device can perform eye safety detection in a timely manner to ensure the safety of the user.

[0017] In combination with the first aspect, in one possible implementation, the image data is transparently transmitted to the first DSP via the first ISP, and the image metadata is transparently transmitted to the first DSP via the I 2 C is transparently transmitted to the first DSP.

[0018] Transparent transmission, also known as transparent transmission (SerialNet), means that during the transmission process, it is transparent to the outside world. Regardless of the content or data protocol format being transmitted, no processing is done on the data to be transmitted. The content to be transmitted is simply treated as a set of binary data and transmitted perfectly to the destination node. It is equivalent to a data line or serial port line, while ensuring the quality of transmission without processing the transmitted business. In this embodiment, the image data is transparently transmitted to the first DSP via the first ISP, and the image metadata is transmitted via the I 2 C is transparently transmitted to the first DSP, which can effectively ensure the transmission quality of TOF RAW data and is conducive to improving the accuracy of the results obtained after the DSP processes the image data and image metadata.

[0019] In combination with the first aspect, in a possible embodiment, the electronic device further includes an ambient light sensor, which is used to collect light intensity data. In response to a service request, collecting the first data through the time-of-flight TOF camera includes: analyzing and processing the service request based on the DSP, and collecting the first data through the TOF camera when it is determined that the service corresponding to the service request is an eye tracking service; or analyzing and processing the service request based on the DSP, and collecting the first data through the TOF camera when it is determined that the service corresponding to the service request is not an eye tracking service and the light intensity data indicates that the ambient brightness is less than a first threshold.

[0020] In some embodiments, the front camera of the electronic device may simultaneously include at least one TOF camera and at least one RGB-AO camera. Among the intelligent perception services involved in the electronic device, most of the services can be completed by the first ISP, the first DSP, and the first NPU using TOF RAW data collected by the TOF camera, and can also be completed by the first ISP, the first DSP, and the first NPU using AO RAW data collected by the TOF camera.

[0021] In this embodiment, when the service request instruction indicates that the current service is an eye tracking-related service, regardless of the ambient light data indicating the light intensity in the current environment, the first DSP will determine that the TOF camera is a camera adapted to the current service, and process the TOF RAW data (i.e., the first data) collected by the TOF camera in combination with the first ISP, the first DSP, and the first NPU to obtain the service results of the eye tracking-related service. It can be understood that the eye tracking service needs to specifically determine the specific gaze area of the user's eyes. Compared with other smart perception services, the eye tracking service has higher requirements for the accuracy of the service results. The TOF camera has better light adaptability and can be combined with depth data for judgment, which is more accurate. The RGB camera is a 2D camera and cannot obtain depth information. Therefore, using the TOF camera to collect the user's eye movement data can ensure the accuracy of the service results of the eye tracking-related service.

[0022] When the service request instruction indicates that the current service is not an eye tracking-related service and is an intelligent perception service, the first DSP can further determine the light intensity in the current environment based on the ambient light data. When the light intensity in the current environment is too small (for example, less than the first threshold), the first DSP will determine that the TOF camera is a camera adapted to the current service, and process the TOFRAW data (i.e., the first data) collected by the TOF camera in combination with the first ISP, the first DSP, and the first NPU to obtain the service results of the intelligent perception service. Since the RGB-AO camera has a poor shooting effect in a dark environment, it will affect the accuracy of the service results, and the TOF camera has a better shooting effect in a dark environment. Therefore, when completing the intelligent perception service in a low-light environment, the TOF camera is used as a camera adapted to the current service to ensure the quality of the captured image and thus ensure the accuracy of the service results.

[0023] In combination with the first aspect, in one possible implementation, the electronic device further includes an RGB-AO camera, and the method further includes:

[0024] When it is determined that the service corresponding to the service request is not an eye tracking service and the light intensity data indicates that the ambient brightness is greater than a first threshold, the RGB-AO camera starts to collect second data; the second data is transmitted to the first DSP via the first ISP; the first DSP processes the second data to obtain third information, and the third information is transmitted to the first DSP via the I 2C is transmitted to the RGB-AO camera, and the third information represents the exposure parameters used by the RGB-AO camera when collecting the next frame of data; the second data is processed by the first NPU to obtain the operation result of the service corresponding to the service request. In this embodiment, when the service request instruction indicates that the current service is not an eye tracking-related service and is an intelligent perception service, the first DSP can further judge the light intensity in the current environment based on the ambient light data. When the light intensity in the current environment is too small (for example, greater than or equal to the first threshold), the first DSP will determine that the RGB-AO camera is a camera adapted to the current service, and process the AO RAW data (i.e., the second data) collected by the RGB camera in combination with the first ISP, the first DSP and the first NPU to obtain the service result of the intelligent perception service. Since the AO RAW data has smaller pixels than the TOF RAW data, the second data can be transmitted to the first DSP only through the first ISP.

[0025] In this embodiment, the first DSP can select a camera adapted to the domain service based on the service type and / or light intensity of the smart perception service, and process the data collected by the camera with the support of the above-mentioned low-power module, thereby reducing the power consumption of the device and improving the efficiency and quality of service processing.

[0026] In a second aspect, an embodiment of the present invention provides an electronic device comprising multiple processors and one or more memories; wherein the multiple processors include a neural network processor NPU, the one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program code, the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the processor executes the first aspect or any possible implementation of the first aspect, wherein the electronic device is the execution method of the present invention.

[0027] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium comprising computer instructions, which, when executed on an electronic device, enables the electronic device to execute the method of the first aspect or any possible implementation of the first aspect.

[0028] In a fourth aspect, an embodiment of the present invention provides a computer program product, comprising computer instructions, which, when executed on an electronic device, enables the electronic device to execute the method of the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 This is a schematic diagram of a scenario in which an electronic device processes TOF data, provided by an embodiment of the present application;

[0031] Figure 2 This is an architectural diagram of a chip system provided by an embodiment of the present application;

[0032] Figure 3 This is an architecture of a low-power module provided in an embodiment of the present application;

[0033] Figure 4 An embodiment of the present application provides a schematic diagram showing the location of a front-facing TOF camera;

[0034] Figure 5 A flowchart of an image processing method provided in an embodiment of the present application;

[0035] Figure 6 This is an architectural diagram of another low-power module provided in an embodiment of the present application;

[0036] Figure 7 An embodiment of the present application provides a schematic diagram showing the locations of the front TOF and front RGB-AO cameras;

[0037] Figure 8 A flowchart of another image processing method provided in an embodiment of the present application;

[0038] Figure 9 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] To facilitate a clear description of the technical solutions of the embodiments of this application, the words "exemplary" or "for example" are used in the embodiments of this application to indicate examples, illustrations, or explanations. Any embodiment or design described in this application as "exemplary" or "for example" should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0040] In the embodiments of this application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0041] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0042] The terms involved in the embodiments of the present application are explained as follows.

[0043] (1) TOF, TOF imaging, TOF camera, and TOF RAW data.

[0044] Time of flight (TOF) technology is a technology that calculates the distance traveled by measuring the time it takes for light to travel a certain distance in a medium.

[0045] TOF imaging refers to an imaging technology in which a group of infrared lights (laser pulses) that are invisible to the human eye are emitted outward, reflected after encountering an object, and reflected to the camera. The time difference or phase difference from emission to reflection back to the camera is calculated, and the data is collected to form a set of distance and depth data, thereby obtaining a three-dimensional 3D model.

[0046] A sensor capable of TOF imaging is a TOF sensor, also known as a TOF camera. While acquiring depth image data, a TOF camera can also acquire infrared image data of the illuminated object.

[0047] In this application, TOF RAW data (TOF image data) can be the raw data acquired by the TOF camera, or it can be data after preliminary processing or preprocessing of the raw data. After processing, TOF RAW data can be converted into a TOF image, which can include a grayscale image and / or a depth image. Among them, an infrared image can also be called an IR image or an IR grayscale image, and a depth image can also be called a depth image.

[0048] Specifically, the grayscale image converted from the data collected by the TOF camera is an infrared grayscale image. Because the reflectivity of the illuminated material under infrared light is different from that under visible light, this infrared grayscale image exhibits image characteristics different from those of a typical visible light grayscale image. For example, a human face may exhibit bright / dark pupils, reflective nose tips, and dark cheeks, which can be used for anti-counterfeiting in facial recognition scenarios. Furthermore, because infrared images are unaffected by visible light, they provide stable imaging under various lighting conditions and exhibit good light adaptability. Therefore, TOF cameras are highly adaptable in low-light environments.

[0049] It should be noted that, in this application, each time the TOF camera generates a frame of image, the TOF RAW data it outputs may include two parts, one of which is RAW image data, which may also be referred to as TOF image data in this application; the other part is metadata used to describe the attributes of the above-mentioned TOF image data, which may also be referred to as TOF image metadata in this application, which can be used to describe the above-mentioned TOF image data and can be used to indicate storage location, historical data, resource search, file records and other functions. Specifically, in this application, the TOF image metadata in the TOF RAW data can be used to indicate whether the current imaging process of the TOF camera will cause harm to human eye safety, so that the camera system can adjust the shooting parameters of the TOF camera in time when the TOF camera may cause harm to the human eye to ensure the safety of the user. In addition, in this application, the pixel size corresponding to any frame of TOF image can be 483*640, which includes TOF image data with a pixel size of 480*640 and TOF image metadata with a pixel size of 3*640.

[0050] Since infrared light and visible light are in different wavelength bands, TOF images are not affected by visible light. TOF images can be applied to face recognition, face verification, face unlocking, posture recognition, expression recognition, gaze recognition, gesture recognition and other fields to improve recognition accuracy. They can also be used in the field of liveness verification and have anti-counterfeiting capabilities.

[0051] (2) General operating environment and trusted execution environment.

[0052] The rich execution environment (REE), commonly referred to as the non-secure side or non-secure zone, is a common environment for all mobile devices and runs a common operating system (OS), such as Android and iOS.

[0053] A trusted execution environment (TEE), commonly referred to as a secure side or secure zone, is an area that requires authorization to access. It is typically used for digital rights management and privacy protection.

[0054] (3) Client Application (CA), which is an application running in the REE environment, referred to as CA for short.

[0055] (4) Trusted Application (TA), which is an application running in the TEE environment, referred to as TA for short.

[0056] (5) The secure shared cache is a secure storage space created between the application operating on the processor on the security side and the NPU for data exchange between the two parties.

[0057] (6) Intelligent perception business and AO business.

[0058] Smart perception services refer to electronic devices using hardware devices such as cameras, microphones or other sensors to map signals from the physical world to the digital world through cutting-edge technologies such as voice recognition and image recognition, and then further elevate this digital information to a level that can be recognized by electronic devices to provide users with corresponding functional services.

[0059] In this application, intelligent sensing services include but are not limited to:

[0060] 1. Gesture recognition (sensing human gestures, such as flipping the phone, tapping the phone, shaking the phone, picking up the phone, etc. Examples include flipping to mute, rejecting a call, shaking to change the background image, and picking up to answer a call).

[0061] 2. Device state recognition: Detects the device's state, such as whether the phone is face up or face down, on a table, in a pocket, or in the user's possession. Possible implementations include automatically maximizing the ringtone when the phone is in a pocket and switching to silent or vibrate mode when the phone is on a table.

[0062] 3. User status recognition: Senses the user's status, such as running, walking, standing, sitting, etc. Function: When the screen is off, the date and time are displayed when the user is looking at the screen. When the user is not looking at the screen, the screen remains off.

[0063] 4. User transportation behavior: Detects the user's transportation status, such as driving, taking the train, and taking the elevator. Functionality: Automatically connect to Bluetooth when driving; activate noise cancellation when answering a call on the train.

[0064] In some embodiments of the present application, some smart sensing services can be completed by ordinary cameras (such as RGB cameras), but some smart sensing services have high requirements for accuracy and ambient brightness, so these smart sensing services need to be completed based on TOF cameras.

[0065] AO, or always-on mode, is a sensing technology that allows electronic devices to identify changes in the surrounding environment and its content, and to implement autonomous user interfaces based on the information acquired. In the daily use of electronic devices by users, some of the smart perception services involved in electronic devices require electronic devices to autonomously perceive information in the environment in real time. Such services can be called AO services. For example, in the always on display (AOD) service, the electronic device needs to monitor in real time whether the user's gaze points fall on the screen of the device when the screen is off, so as to display information such as time and weather to the user on the screen when the user is looking at the screen. In order to implement these AO services, some electronic devices may include cameras equipped with AO image sensors. These cameras are equipped with a dedicated mode that can run AO functions, which can be used for AO services that require autonomous real-time monitoring. In some embodiments of the present application, the electronic device may include a TOF camera, which serves as an AO image sensor to implement a series of AO services that require real-time monitoring.

[0066] (7) RGB-AO camera.

[0067] An RGB-AO camera has two working modes: AO mode and RGB mode.

[0068] In AO mode, the RGB-AO camera can be used to collect AO data (i.e., AO image, which can be a low-pixel grayscale image). The AO RAW data can be used to implement, for example, gesture recognition and AOD smart perception services. It should be noted that when the AO RAW data obtained by the RGB-AO camera in AO mode is not perceived by the user, and the obtained AO data is only used as input data for the smart perception service to obtain the corresponding business results (such as whether the user is looking at the screen, the meaning of the user's current gesture), and it will not be processed and sent to the display screen for display. In RGB mode, the RGB-AO camera can be used as an ordinary camera, and the RGB RAW data it obtains (i.e., the RAW image corresponding to the RGB image before it is processed into an RGB image, generally a high-pixel RAW image), and the RGB RAW image will be further processed and saved by other hardware in the electronic device.

[0069] In some embodiments of the present application, the electronic device may also be equipped with a TOF camera and an RGB-AO camera at the same time, and both cameras may be equipped with AO image sensors and used to implement a series of AO services that require real-time monitoring.

[0070] (8) Conventional ISP, conventional DSP, low-power ISP, low-power DSP, and low-power NPU.

[0071] In electronic devices, an image signal processor (ISP) is typically used to process data output by image sensors, performing functions such as exposure control, gain control, white balance, color correction, lens shading, gamma correction, bad pixel removal, and automatic black and white levels. After the ISP outputs the digital image signal to the digital signal processing (DSP), the DSP processes it and converts it into standard image signals in formats such as RGB and YUV.

[0072] Generally speaking, electronic devices such as mobile phones and computers typically integrate multiple ISPs and multiple DSPs on a system-on-chip (SOC). Among these ISPs and DSPs, a high-performance ISP and at least one high-performance DSP are responsible for the majority of image processing in the electronic device. These high-performance ISPs and DSPs can be used to process algorithms with high computational loads, such as video encoding and decoding, graphics and image processing, and visual image processing. Furthermore, the SOC may also integrate at least one low-performance ISP and at least one low-performance DSP. These low-performance ISPs and DSPs can perform simple or specialized image processing. For example, the low-performance ISP can be used to transparently transmit image data, while the low-performance DSP can be used to convert TOF-RAW image data into depth and infrared images. It is understood that while the performance of these low-performance ISPs and DSPs is not as good as that of the high-performance ISPs and DSPs, their power consumption during operation is much lower than that of the high-performance ISPs and DSPs. In some services performed by the electronic device, the electronic device may utilize the low-performance ISP and the low-performance DSP to replace the high-performance ISP and the high-performance DSP to complete the image data processing process, thereby reducing system power consumption.

[0073] In this application, the chip system included in the electronic device may include two ISPs and two DSPs.

[0074] Of these two ISPs, the higher-performance ISP can be referred to as a conventional ISP, which can specifically be an ISP integrated into a SoC (e.g., a spectra ISP). The lower-performance ISP can be referred to as a low-power ISP (e.g., an AON ISP). This low-power ISP can be a low-performance ISP included in another sub-chip (non-system-level) or integrated into a SoC, and this application does not limit this. In this application, the performance of a low-power ISP is lower than that of a conventional ISP, but the power consumption of the electronic device caused by the low-power ISP during operation is much lower than that of a conventional ISP.

[0075] Of the two DSPs mentioned above, the one with higher performance can be called a conventional DSP, which can be a DSP integrated on the SOC; the other one with lower performance can be called a low-power DSP, which can specifically be a low-performance DSP contained in other sub-chips (non-system level), such as an external low-power chip integrated with a low-power ISP and a low-power NPU. Optionally, the low-power DSP can also be a low-power DSP directly integrated on the SOC. Similarly, in this application, the performance of the low-power DSP is lower than that of the conventional DSP, but the power consumption caused to the electronic device by the low-power DSP when it is working is much lower than that of the conventional DSP.

[0076] In the present application, the above-mentioned low-power DSP and the above-mentioned low-power ISP may also be included in a low-power module, and the low-power module may also include a low-power neural network processor (neural-network processing unit, NPU) (such as eNPU), and the low-power NPU may store a business application algorithm library (X-Algo Library, where "X" represents the specific name of the smart perception service, for example, "X" can be face unlocking, staring at the screen, etc.), and the NPU may also complete the image calculation process involved in some smart perception services based on the algorithms in the business application algorithm library. In some smart perception services provided by electronic devices (including but not limited to gesture control, AOD and other services), the low-power module can replace the conventional ISP, conventional DSP, conventional NPU and CPU, and independently complete the processing operations of TOF RAW data involved in the smart perception service without waking up the CPU and the entire Android system, and obtain the business processing results of the smart perception service, which can effectively reduce the power consumption of the electronic device.

[0077] Currently, applications such as terminal unlocking, payment, and identity authentication all involve processing sensitive images such as faces and eyes. Due to requirements for accuracy and ambient brightness, electronic devices rely on TOF cameras to perform some of these tasks. However, current electronic devices require extremely complex steps to process TOF RAW data, involving complex software, hardware, and data links.

[0078] Figure 1 The system architecture involved in the current electronic devices using TOF cameras to complete intelligent perception services, as well as the process of processing images based on the system to obtain business results, is shown by way of example.

[0079] like Figure 1 As shown, the operating system of the electronic device can be an Android system, which can be a system running on a SOC. The above system is divided into a general execution environment and a trusted execution environment, and the software and hardware of the general execution environment can adopt a layered architecture. Figure 1 As shown, taking the layered architecture Android system as an example, the hardware and software structure of the general execution environment of the electronic device includes the application (APP) layer, the framework (FWK) layer, the hardware abstraction layer (HAL), the kernel layer, and the hardware layer. The trusted execution environment can include trusted applications, a secure buffer memory area for REE and TEE to exchange data, a TOF image processing and calibration calculation algorithm library (TOF Calc Algo Library) for trusted applications to load corresponding algorithms to calculate business results, and a business application algorithm library (X-Algo Library, where "X" represents the specific name of the smart perception business, for example, "X" can be face unlocking, staring without turning off the screen, etc.). They are explained below.

[0080] The application layer on the REE side may include at least one application, which may be a system application or a third-party application, such as smart perception, TOF camera, etc., or notification, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications ( Figure 1 not shown).

[0081] Among them, the methods or services supported by the smart perception service may include but are not limited to gesture control, face unlocking, face payment, etc.

[0082] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. For example, it may include activity managers, window managers, content providers, view systems, resource managers, notification managers, camera services (CameraService) and intelligent perception services ( Figure 1 Only a portion thereof is shown) etc.

[0083] The system architecture on the REE side can also include system libraries and Android Runtime ( Figure 1 (not shown). The system library can include multiple functional modules, such as the surface manager, media libraries, OpenGL ES, SGL, etc. The Android runtime includes the core library and the virtual machine.

[0084] The HAL layer is an encapsulation of the Linux kernel driver, providing an interface to the upper layer and shielding the implementation details of the low-level hardware.

[0085] The HAL layer can include the TOF camera hardware abstraction module, namely the TOF camera HAL (Camera HAL), and the intelligent perception Daemon module. Figure 2 The HAL layer may further include other hardware abstraction modules, such as Wi-Fi HAL, audio HAL, ambient light sensor HAL, etc.

[0086] Among them, the intelligent perception Daemon module includes intelligent perception CA, intelligent perception HAL and TOF camera control unit.

[0087] The TOF camera control module can be used to implement data interaction between the smart perception HAL and the TOF camera service. For example, the smart perception HAL can send the AE value received from the smart perception TA on the TEE side to the TOF camera service through the TOF camera control module. The TOF camera service can also send the TOE data identifier, such as the file descriptor (FD), to the smart perception HAL through the TOF camera control module. The FD is used to indicate the storage location and reading method of TOF. Data can be transmitted between the TOF camera control module and the TOF camera service through the VNDK interface.

[0088] The smart perception HAL can send an initialization instruction to the smart perception CA. The initialization instruction can include the identifier of the image processing algorithm, so that the smart perception CA and TA can determine from the algorithm library that the image processing algorithm corresponding to the identifier is the image processing algorithm to be used.

[0089] Smart Perception CA can be used to exchange information with Smart Perception TA on the TEE side. Smart Perception CA can send the identifier of the received TOF RAW data to the Smart Perception TA, and can also receive the human eye safety detection results, calculation results, and AE values from the Smart Perception TA.

[0090] The kernel layer may include a TOF camera driver, which is used to provide functional support for the TOF camera.

[0091] The hardware layer can include a TOF camera and an IFE module. The TOF camera is used to collect TOF RAW data and transmit the collected TOF RAW data to a secure buffer memory area through the IFE module and a secure ISP path. The IFE module is present in a conventional ISP, which is included in the SoC of the electronic device.

[0092] The TEE side may include but is not limited to at least one trusted application (TA), such as a smart perception TA, which may include a smart perception module that processes TOF RAW data, calls algorithms, schedules tasks, and transmits smart perception CA data.

[0093] Smart Perception TA can also perform eye safety detection based on TOF RAW data through the eye safety detection algorithm, and determine the AE value of the next frame through the AE prediction algorithm based on TOF RAW data. It can also receive the calculation results sent by Smart Perception TA, and can also send the eye safety detection results, calculation results and AE values to Smart Perception CA.

[0094] The intelligent perception TA is used to read the TOF RAW data from the secure buffer memory area based on the TOF identifier (such as FD).

[0095] Based on the initialization instructions received from the Smart Perception CA, the Smart Perception TA can determine the corresponding algorithm from the TOF image processing and calibration calculation algorithm library and the business application algorithm library. It then inputs the TOF RAW data into the determined algorithm to process the TOF image. For example, the determined algorithm includes an image processing algorithm, an eye safety detection algorithm, and an AE prediction algorithm.

[0096] After the intelligent perception TA obtains the basic TOF RAW data and the calibration data of the TOF camera (the calibration data includes some numbers used when the TOF camera is shooting, such as background noise, depth calculation correction data, etc.), it can perform eye safety detection through the eye safety detection algorithm in the TOF image processing and calibration calculation algorithm library to obtain the eye safety detection result, determine the AE value of the next frame of TOF RAW data based on the AE prediction algorithm, and process the TOF RAW data through the image processing algorithm to obtain a grayscale image and a depth image.

[0097] After obtaining the depth image and grayscale image, the Smart Perception TA loads the corresponding business algorithm from the business application algorithm library to calculate the depth image and grayscale image to obtain the business result. The Smart Perception TA then returns the business result to the Smart Perception TA, which then returns it to the application layer of the REE. The application layer executes the corresponding instructions based on the business operation result.

[0098] For example, the specific working steps of the above system are explained by taking the smart perception service of face unlocking as an example.

[0099] 1. The business application in the application layer issues an image acquisition instruction. The instruction is finally sent to the TOF camera in the hardware layer through the framework layer and hardware abstraction layer. The TOF camera starts to collect TOF RAW data and sends it to the ISP.

[0100] 2. The ISP converts the TOF data into TOF unpacked Raw Data (data under an existing data protocol) and sends it to the secure buffer memory area. At the same time, the ISP sends the FD of the TOF RAW data (i.e., the storage location and reading method of the TOF RAW data in the secure buffer memory area, the same below) to the TOF camera HAL.

[0101] 3. The TOF camera HAL sends the TOF data and the TOF camera calibration data (Cali.data) to the TOF camera service.

[0102] 4. The TOF camera service sends the FD and calibration data of the TOF RAW data to the Face HAL (i.e., X-HAL, where "X" represents the face unlocking service, the same below).

[0103] 5. Face HAL sends the FD and Cali.data of the TOF RAW data to Face CA.

[0104] 6. Face CA sends the FD and Cali.data of the TOF RAW data to Face TA.

[0105] 7. Face TA reads the TOF Raw Data from the secure buffer memory area based on the FD of the TOF Raw Data, and calls the TOF image processing and calibration calculation algorithm library loaded from the REE to the TEE to calculate the TOF data and calibration data. At the same time, it calls the Face ID algorithm model from the business application algorithm library to compare the depth image and infrared image, obtaining the following calculation results:

[0106] 1) Eye safety detection result, used to indicate whether the light emitted by the TOF camera is safe for the human eye. If it is unsafe, the TOF camera is instructed to stop collecting TOF camera raw data. In the embodiment of the present application, the eye safety detection result can be represented by a flag, which can be assigned a value of 1 (True) or 0 (False). 1 (True) indicates that the current shooting method of the TOF camera will cause damage to the human eye, and 0 (False) indicates that the current shooting method of the TOF camera will not cause damage to the human eye.

[0107] 2) AE (automatic exposure) parameters: used to indicate the parameters used to adjust the TOF camera when capturing the next frame of image. The purpose of AE adjustment is to improve the quality of the TOF image obtained by converting TO data.

[0108] 3) Depth image (Depth image) and infrared image (IR image) are used for face unlocking recognition and anti-counterfeiting.

[0109] 4) Call the Face ID algorithm model to compare the Depth image and the IR image to obtain the face comparison result.

[0110] 8. Face TA sends the above face comparison results, eye safety detection results, and the AE of the next frame image to Face CA.

[0111] 9. Face CA sends the face comparison results, eye safety detection results, and AE of the next frame image to Face HAL.

[0112] 10. If the face match is successful, the unlock is successful, and all processes are released. If the face match fails or the eye safety failure occurs, Face HAL will send the eye safety detection results and AE results to the TOF camera service.

[0113] 11. The TOF camera service sends the eye safety detection results and AE results to the TOF camera HAL.

[0114] 12. The TOF camera HAL decides whether to continue the subsequent process (i.e. whether the TOF camera continues to collect TOF data) based on the eye safety detection results, and feeds the AE results back to the TOF adjustment module (i.e. TOF driver, Figure 1 The TOF adjustment module takes effect on subsequent frames.

[0115] 13. Face HAL sends the face comparison result to Face Service, which sends the face comparison result to the lock screen module (or payment module) in the application layer (which exists in the system service / intelligent perception SDK in the application layer). Figure 1 not shown).

[0116] 14. The lock screen module (or payment module) unlocks (or pays) when the comparison result indicates that the comparison is passed, otherwise the lock screen state remains (or payment is not executed).

[0117] Combined with the above description, it can be seen that when using the current TOF RAW data processing method to complete intelligent perception services, starting from the service request issued by the application layer, the electronic device needs to wake up the CPU and the entire Android system to collect, transmit and process TOF RAW data. The system power consumption is too high, especially when processing AO services that require real-time monitoring. The above processing method causes greater power consumption to the electronic device, seriously reducing the battery life of the electronic device and poor user experience.

[0118] In response to the defects in the above-mentioned image processing method, this application combines a low-power module in an electronic device to complete the image processing process included in the smart perception service and obtain the service processing results. The low-power module can independently process the data obtained by the camera on the CPU. When performing image processing, the electronic device does not need to wake up the CPU, memory and the entire Android system running on the SOC in the SOC, which can effectively reduce the power consumption of the electronic device when processing smart perception services.

[0119] First, the low-power module mentioned above is described in conjunction with the chip system in the electronic device provided in this application.

[0120] like Figure 2 As shown, the system-on-chip SOC 20 may be integrated with a CPU 201, a conventional ISP 202, a conventional DSP 203, and a low-power module 204. In some embodiments, the SOC 20 may also be integrated with an ambient light sensor 205.

[0121] The conventional ISP 202 and the conventional DSP 203 can be processing units directly integrated on the system-on-chip SOC 20. The conventional ISP 202 and the conventional DSP 203 can be loaded on the CPU 201 as independent devices, or they can be integrated on the CPU 201. The conventional ISP 202 can be the ISP with the best performance among all the ISPs included in the SOC 20, and the conventional DSP 203 can be the DSP with the best performance among all the DSPs included in the SOC 20. Specifically, the conventional ISP 202 can be used to process algorithms with large computational load, such as video encoding and decoding, graphic image processing, visual image processing, etc. In the present application, the ISP 202 and the conventional DSP 203 can process data collected by various cameras, including but not limited to RGBRAW data collected by an RGB camera, AO RAW data collected by an RGB-AO camera, and TOF RAW data collected by a TOF camera.

[0122] The low power module 204 may include a low power ISP 204a, a low power DSP 204b, a low power NPU 204c, and an integrated circuit bus (IC bus). 2 C) 204d. Of which:

[0123] The performance of the low-power ISP 204a is lower than that of the conventional ISP 202, but it can also perform some relatively simple processing on the image data. For example, in the present application, when the electronic device performs eye tracking and AOD services based on the RGB-AO camera, the low-power ISP 204a can perform exposure control, gain control and other processing on the AO RAW data collected by the RGB-AO camera. However, when the electronic device performs eye tracking and AOD services based on the TOF camera, due to the performance limitations of the low-power ISP 204a, the ISP 204a can only process a part of the TOF RAW data collected by the TOF camera (i.e., the TOF image data, and the other part of the TOF image metadata will be processed by the ISP 204a). 2 C 204d sends it to the low-power DSP 204b for transmission, and the ISP 204a may not perform any processing on the TOF image data and transparently transmit the TOF image data to the low-power DSP 204b.

[0124] The low-power DSP 204b has lower performance than the conventional DSP 203 and can perform some relatively simple processing on the image data. For example, in the present application, when an electronic device performs eye tracking or AOD services, the low-power DSP 204b can obtain the corresponding depth image and infrared image based on the TOF RAW data, determine the exposure parameter AE corresponding to the next frame of the camera, and calculate the human eye safety parameter based on the TOF RAW data (for details, please refer to the subsequent description).

[0125] The low-power NPU 204c can perform operations on the depth image and infrared image sent by the low-power DSP 204b to obtain operation results (ie, business processing results), and return the operation results to the low-power DSP 204b.

[0126] Integrated circuit bus (IC bus) 2 C) 204d can be used for data communication between DSP 204b and TOF camera. For example, among the TOF RAW data output by the TOF camera, a part of the data (TOF image metadata) that cannot be transmitted through the low power ISP 204a can be transmitted through I 2 For another example, after the low-power DSP 204d determines the exposure parameter AE and the eye safety parameter corresponding to the next frame of image for the TOF camera based on the TOF image data, the low-power DSP 204d can transmit the image to the low-power DSP 204b through I 2 C 204d sends the exposure parameter AE and the eye safety setting parameter to the camera, so that the camera can adjust the AE and eye safety related parameters.

[0127] In some embodiments, before the electronic device starts to formally process the smart sensing service, the CPU 201 sends a service request instruction to the low-power DSP 204b. The service request instruction can indicate the service type of the smart sensing (for example, whether it is an AOD service or a face recognition service). In addition, the ambient light sensor 205 can collect ambient light data and transmit the ambient light data to the low-power DSP 204b. 2 C 204d is transmitted to the low-power DSP 204b. The low-power DSP 204b can determine the camera adapted to the current service based on the ambient light data and the above service type, and obtain the image data collected by the camera adapted to the service through corresponding operations (for details, please refer to the subsequent description).

[0128] In some embodiments, the low-power ISP 204a, the low-power DSP 204b, and the low-power NPU 204c are included in an external chip of a SOC, or the low-power ISP 204a, the low-power DSP 204b, and the low-power NPU 204c can be integrated into a low-power module of the SOC at the same time. For example, the low-power ISP 204a, the low-power DSP 204b, and the low-power NPU 204c can be included in the same MCU. In this case, the data transmission between the low-power ISP 204a, the low-power DSP 204b, and the low-power NPU 204c can only occur in the same MCU, which can effectively avoid the problem of image data leakage during the business process. For example, in some embodiments, the ambient light sensor can be mounted on the SensorHub, which can include an MCU 206 ( Figure 2 (not shown in the figure), the low power ISP 204a, the low power DSP 204b and the low power NPU 204c can exist in the MCU206 at the same time.

[0129] It should be noted that, in the embodiments of this application, Figure 2 The chip system architecture shown is only an example and does not constitute a limitation of the embodiments of the present application. In other embodiments of the present application, the SOC may include more or fewer components than shown, or combine or split some components, or arrange the components differently.

[0130] In the chip system provided by the present application, after the low-power module 204 receives the service request instruction sent by the CPU 201, the low-power DSP 204b can independently perform image processing operations without relying on the CPU 201, including sending an instruction to the TOF camera to request data acquisition, and processing the TOF RAW data through the corresponding algorithm after receiving the TOF RAW data to obtain the service processing result. During this whole process, the CPU 201 in the SOC and the entire operating system (such as the Android system) it runs can be in a dormant state (of course, in some cases, the CPU can also be in an awake state due to processing other processes), which greatly reduces the power consumption of the electronic device. When the CPU is in a dormant state, the low-power module is used to perform pre-stage detection and wake up the CPU at the appropriate time (for example, after calculating the service result).

[0131] Next, combine Figure 3 The above description explains the specific working mode of the above low power consumption module.

[0132] like Figure 3As shown, the business system 30 may include a TOF camera 301, an AON ISP 302, a mini DSP 303, an eNPU 304, and an I 2 C 305. AON ISP 302, mini DSP 303, eNPU 304 can be ISP, DSP, NPU in the AON (Always ON) camera processing architecture of the electronic device respectively. 2 The low power consumption module composed of C 305, mini DSP 303 and eNPU 304 can be the aforementioned Figure 2 Low power consumption module 204; specifically, AON ISP 302, miniDSP 303, eNPU 304, I 2 C 305 can be Figure 2 ISP 204a, DSP 204b, NPU 204c and I 2 C204d.

[0133] The TOF camera 301 may be an ITOF camera (i.e., an indirect TOF camera, which measures the time of flight indirectly by measuring the phase difference between the transmitted sine wave / square wave and the received sine wave / square wave). Specifically, the TOF camera 301 may be a front-facing camera or a rear-facing camera of an electronic device.

[0134] Figure 4 Taking the camera 301 as a front camera as an example, the specific style of the TOF camera 301 in the electronic device is shown.

[0135] Reference Figure 4 As shown, the electronic device 300 may include at least: a display screen 310, a middle frame 320, a battery cover 340, and a circuit board 330 located between the display screen 310 and the battery cover 340, wherein the circuit board 330 may be arranged on the middle frame 320, for example, the circuit board 330 may be arranged on a side of the middle frame 320 facing the battery cover 340 (see Figure 4 As shown in (B) in FIG), the circuit board 330 may be disposed on a side of the middle frame 320 facing the display screen 310, and the display screen 310 and the battery cover 340 are located on both sides of the middle frame 320, respectively.

[0136] In order to realize the intelligent sensing function, the TOF camera 301 can be set as a front camera on the side of the middle frame 320 facing the display screen 310, and an opening is provided on the middle frame 320 for exposing the lens end of the TOF camera 301 ( Figure 4(not shown), and a light hole 311 is provided on the display screen 310, and the lens of the TOF camera 301 corresponds to the light hole 311. Of course, the setting position of the front camera can include but is not limited to the above description. In the embodiment of the present application, Figure 4 As shown in (A) in FIG, the TOF camera 301 can be located in an area near the top edge of the display screen 310. It is understood that the location of the TOF camera 301 is not limited to Figure 4 The position shown may also be located at other positions on the display screen 310.

[0137] Reference Figure 4 As shown in (B) in FIG. 3 , the TOF camera 301 can be electrically connected to the circuit board 330. As an embodiment, the TOF camera 301 can be electrically connected to the circuit board 330 via an electrical connector ( Figure 3 (not shown) is electrically connected to the circuit board 330. For example, the TOF camera 301 may be provided with a male socket of an electrical connector, and the circuit board 330 may be provided with a female socket of an electrical connector. The electrical connection between the TOF camera 301 and the circuit board 330 is achieved by plugging the female socket into the male socket.

[0138] Among them, the circuit board 330 may be provided with a central processing unit CPU (such as the CPU 201 in the aforementioned description), and the processor controls the TOF camera 301 to capture images. After receiving a service request instruction issued in response to a user operation in the smart perception service (such as the user picking up the phone in the raise-your-hand-to-light-up service), or a service request instruction generated by a system application (such as a settings application) in the AO service, the CPU may send the service request instruction to the mini DSP 303 in the low-power module, and the mini DSP 303 controls the TOF camera 301 to capture the object according to the service request instruction to obtain TOF RAW data.

[0139] It should be noted in advance that the AON ISP 302 and mini DSP 303 contained in the low-power module of this application can only be used to process AO RAW data or RGB RAW data collected by the RGB-AO camera, and cannot process TOF RAW data output by the TOF camera. This is because the pixel size corresponding to AO RAW data or RGB RAW data is generally 480*640, but the pixel size corresponding to TOF RAW data is 483*640, which exceeds the processing upper limit of the AON ISP 302. The TOF RAW data output by the TOF camera will not be transmitted to the AON ISP 302 and further transmitted to the mini DSP 303, that is, the low-power module will not be used to process related services that need to be completed by the TOF camera in electronic devices.

[0140] However, in this application, each time the TOF camera generates a frame of image, the TOF RAW data it outputs can actually include two parts, one of which is the TOF image data; the other is the TOF image metadata used to describe the properties of the above TOF image data. This part of the data can be used to indicate whether the light currently emitted by the TOF data (such as infrared light) is safe for the human eye. Therefore, in this application, in order to utilize the aforementioned low-power module to process the TOF RAW data output by the TOF camera and reduce the power consumption of the electronic device when processing TOF data, a data link can be established between the TOF camera 301 and the mini DSP 303 through I2C 305. The TOF RAW data with a pixel size of 483*640 output by the TOF camera 301 can be divided into TOF image data with a pixel size of 480*640 and TOF image metadata with a pixel size of 3*640. The TOF image data with a pixel size of 480*640 can be transparently transmitted to the mini DSP 303 via the AON ISP 302, and the TOF image metadata with a pixel size of 3*640 can be transparently transmitted to the mini DSP 303 via I2C 303. In this way, the mini DSP 303 can obtain the complete TOF RAW data output by the TOF camera 301.

[0141] After obtaining the complete TOF RAW data through I2C 305 and AON ISP 302, the mini DSP 303 can load the calibration data of the TOF camera 301 and the human eye safety detection algorithm and process the TOF image metadata to determine whether the infrared light emitted by the TOF camera is safe for the human eye. If it is safe, the TOF image data will continue to be processed; if not, the current processing flow will be stopped.

[0142] Specifically, if the mini DSP 303 determines that the infrared light emitted by the TOF camera is not safe for human eyes, the mini DSP 303 can obtain a first safety parameter for adjusting the light emitted by the TOF camera to a state that is safe for human eyes after processing the TOF image metadata. The first safety parameter can be used to adjust the infrared light emitted by the TOF camera to a state that is safe for human eyes. 2 C 303 is transmitted back to the TOF camera 301 to drive the TOF camera 301 to adjust the corresponding parameters to the first safety parameters and then resume collecting TOF RAW data.

[0143] After the mini DSP 304 determines that the infrared light emitted by the TOF camera is safe for human eyes, the mini DSP 304 will process the TOF image data through the corresponding algorithm to obtain the infrared image and depth image corresponding to the TOF RAW image data, and transmit these two types of images to the eNPU 304. At the same time, it calculates the AE required for the TOF camera 301 to generate the next frame of TOF RAW data, thereby improving the quality of the TOF image converted from the TOF image data.

[0144] After receiving the infrared image and depth image transmitted by the mini DSP 303, the eNPU 304 can run the corresponding algorithm to operate on the depth image and the infrared image to obtain the operation result, which is the business processing result corresponding to the business (for example, the operation result in the face unlocking business is whether the face matches, and the operation result in the AOD business is whether the user's eyes are looking at the screen).

[0145] After the eNPU 304 obtains the above calculation results, the eNPU 304 will return the above calculation results to the mini DSP 303. After obtaining the above calculation results, the mini DSP 303 can send an interrupt signal to the CPU through the corresponding interface, so that the CPU can be awakened from the sleep state, or other processes of the CPU terminal can obtain the above calculation results from the mini DSP 303 and react based on the calculation results (for example, unlock the screen in the case of face matching, turn on the screen when the human eye is looking at it, etc.).

[0146] Combined with the above description and Figure 3 It can be seen that after the TOF camera 301 sends the TOF RAW data to the low-power module (mini DSP 303), the low-power module can process the TOF RAW data without relying on the CPU. During the process of using the low-power module to process the TOF RAW data, the CPU in the SOC and the Android system can be in a dormant state, and the power consumption of the electronic device is reduced. Moreover, by processing the TOF RAW data with the low-power processing module, the electronic device can simplify the software and hardware modules involved in processing the TOF RAW data and shorten the data transmission link. For example, in the embodiment of the present application, the TOF RAW data does not need to be transmitted to the safe buffer area. Therefore, when the low-power module is used to process the TOF RAW data, the DDR in the electronic device basically does not consume power, which can further effectively reduce the total power consumption of the electronic device.

[0147] In this embodiment, the AON ISP 302, mini DSP 303, and eNPU 304 may be referred to as a "first ISP," a "first DSP," and a "first NPU," respectively.

[0148] Combined with the above Figure 3 The specific working mode of the low-power module shown is further described below. The image processing method involved in the embodiment of the present application is further described.

[0149] See Figure 5 The electronic device provided in this application for executing the method may include but is not limited to a TOF camera and a TOF camera driver. A low power consumption module may be Figure 3 The low-power module shown in FIG, which may include a low-power ISP, a low-power DSP, a low-power NPU and I 2 C. Such as Figure 5 As shown, the image processing method may include but is not limited to some or all of the following steps:

[0150] S501: The service application sends a service call request to the low-power DSP.

[0151] The service call request is used to request the call of a service. The service corresponding to the service application may be a smart perception service, including but not limited to face unlocking, face payment, information authentication, gesture control, posture control, etc.

[0152] In the case that the service is a non-AO service, the service application request may be a service request instruction issued by the electronic device in response to a received user operation by controlling the service application program.

[0153] In the face unlock application scenario, the application can be a settings application, and the user operation can be an unlock operation of the electronic device. For example, when the electronic device is in the locked state, the motion sensor detects that the electronic device is picked up, or when a touch operation is detected in the locked state, the electronic device triggers the unlock screen process and executes the above step S501. In this case, the instruction corresponding to the user operation can be an unlock instruction, and the service call request is used to request the call of the face unlock service.

[0154] In the facial recognition payment application scenario, the user operation can be a payment operation through a payment application. For example, the electronic device inputs a user operation indicating payment on the payment interface of a payment application (such as Alipay or WeChat). After detecting the payment operation, the electronic device triggers the secure payment process and executes step S501 above. The instruction corresponding to the user operation can be a payment instruction, and the service call request is used to request the invocation of the facial recognition payment service.

[0155] In the application scenario of information authentication, the user operation can be a user operation that instructs identity authentication. After detecting the user operation, the electronic device triggers the identity authentication process. The instruction corresponding to the user operation can be a verification instruction, and the service call request is used to request the invocation of the information authentication service.

[0156] In the application scenario of gesture control, the user operation can be a user operation for turning on gesture control. After detecting the user operation, the electronic device triggers the gesture control process and executes the above step S501. The instruction corresponding to the user operation can be the instruction corresponding to the detected gesture, and the service call request is used to request the call of the gesture control service. For example, when the electronic device displays the main interface, the instruction corresponding to the gesture of sliding the hand to the right is to display the next page, and the instruction corresponding to the gesture of sliding the hand to the left is to display the previous page.

[0157] In the application scenario of gesture control, the user operation may be a user operation for enabling gesture control. After detecting the user operation, the electronic device triggers the gesture control process and executes the above step S501. The instruction corresponding to the user operation may be an instruction corresponding to the detected gesture, and the service call request is used to request the invocation of the gesture control service.

[0158] In the case where the service is an AO service, the service application request may be a service request instruction automatically issued by a settings application in the electronic device based on the parameters set by the user for the system. The electronic device may continuously (or periodically) execute step S501 according to the specific parameters set by the user for the service in the settings application.

[0159] Not limited to the scenarios listed above, the image processing method provided in the embodiments of the present application can also be used in other scenarios, which are not limited here.

[0160] S502: The low-power DSP sends instruction information to the TOF driver.

[0161] The instruction information is used to instruct the acquisition of TOF data through the TOF camera.

[0162] S503: The TOF camera driver sends a first driving instruction to the TOF camera.

[0163] S504: The TOF camera collects TOF RAW data.

[0164] The TOF camera driver starts the TOF camera based on the first drive instruction (if the TOF camera is not currently started), and the TOF camera starts to collect TOF RAW data. Specifically, the TOF camera driver starts the TOF camera to collect TOF RAW data using the first exposure parameter value.

[0165] During initial acquisition, the first exposure parameter value is a default exposure parameter. In subsequent acquisitions, the first exposure parameter is a received exposure parameter value for indicating the next frame of TOF RAW data.

[0166] S505: The TOF camera sends the TOF image data to the low-power DSP via the low-power ISP.

[0167] The TOF image data is part of the TOF RAW data output by the TOF camera.

[0168] In this application, each time a TOF camera generates a frame of image, the TOF RAW data it outputs includes the TOF image data and TOF image metadata used to describe the attributes of the TOF image data. Figure 3 The relevant instructions will not be repeated here.

[0169] It should be noted that in the present application, the TOF image data can be transmitted to the low-power DSP through the low-power ISP, that is, in the process of the above-mentioned TOF image data being sent to the low-power DSP by the TOF camera via the low-power ISP, the TOF image data may not be processed by any original component or module in the electronic device.

[0170] S506: The TOF camera sends the TOF image metadata to the I 2 C is sent to the low-power DSP.

[0171] Correspondingly, the TOF image metadata is also part of the TOF RAW data output by the TOF camera. Figure 3 The relevant instructions will not be repeated here.

[0172] Similarly, in this application, TOF image metadata can be obtained by I 2 C is transparently transmitted to the low-power DSP, that is, the TOF image data is transmitted to the low-power DSP by the TOF camera through I 2 During the process of sending the TOF image data from C to the low-power DSP, the TOF image data may not be processed by any component or module in the electronic device.

[0173] S507: The low-power DSP performs eye safety detection based on the TOF image metadata.

[0174] Since the TOF camera will actively illuminate when imaging, in order to ensure that the light is safe for the human eye, the low-power DSP can detect whether the current shooting process of the TOF camera is safe for the human eye based on the TOF image metadata. In some embodiments, the low-power DSP can load the calibration data corresponding to the TOF camera and identify whether the TOF camera is working in the human eye safety mode based on the TOF image metadata. If so, the human eye safety detection result is safe, and the TOF camera can continue to illuminate and frame; otherwise, the human eye safety detection result is unsafe, and the low-power DSP uses I 2 The C interface notifies the TOF camera to stop illuminating and outputting frames.

[0175] S508: The low-power DSP calculates the AE of the next frame of image.

[0176] When the eye safety test result is safe, the low-power DSP calculates the AE required for the TOF camera to generate the next frame of TOF RAW data based on the TOF image data. The AE may include exposure parameter values and exposure time.

[0177] Optionally, when the eye safety detection result is unsafe, the low-power DSP may have a business application send a calculation result indicating a calculation failure. Upon receiving the calculation result indicating a calculation failure, the business application may execute a second operation / instruction. For example, in a face unlocking business scenario, the second operation may be outputting a prompt indicating that the unlocking failed.

[0178] Optionally, when the result of the eye safety detection is unsafe, the TOF camera can send an instruction message to the TOF camera driver to instruct to turn off the TOF camera. After receiving the instruction message, the TOF camera driver sends a shutdown command to the TOF camera to turn off the TOF camera.

[0179] S509: The low-power DSP sends the eye safety parameters and the AE of the next frame image to the TOF camera driver via I2C.

[0180] S510: The TOF camera driver sends a second driving instruction to the TOF camera.

[0181] S511: The TOF camera adjusts AE.

[0182] The second driving instruction is used to adjust the exposure parameter of the TOF camera to the AE of the next frame image.

[0183] The TOF camera driver can adjust the exposure parameter of the TOF camera to the above AE value, so that when the TOF camera obtains the next frame of TOF RAW data, it is exposed according to the AE value.

[0184] It should be understood that the above-mentioned eye safety adjustment parameter and AE value can be sent separately.

[0185] S512: Process the TOF RAW data to obtain an infrared image and a depth image.

[0186] When the eye safety parameters are safe, the low-power DSP processes the TOF RAW data to obtain infrared images and depth images.

[0187] The low-power DSP can run image processing algorithms to convert TOF RAW data into grayscale images and depth images.

[0188] S513: The low-power DSP sends the infrared image and the depth image to the low-power NPU.

[0189] S514: The low-power NPU performs calculations on the infrared image and the depth image to obtain calculation results.

[0190] The low-power NPU can run the corresponding target algorithm to perform operations on the depth image and infrared image to obtain the operation result, which is the business processing result corresponding to the business.

[0191] In different scenarios, the target algorithm used may be different, and the meaning of the obtained calculation results may also be different.

[0192] For example, in face unlocking, payment, and verification scenarios, the target algorithms are face recognition and liveness verification. The NPU uses a grayscale image (or grayscale and depth images) to identify whether the face belongs to the device owner. If so, the NPU further detects whether the face is alive based on the grayscale and depth images. If so, the calculation result is determined to be information indicating that the verification has passed (or succeeded). If the face does not belong to the device owner or is not alive, the calculation result is determined to be information indicating that the verification has failed (or failed).

[0193] As another example, in the application scenario of posture / gesture control, the target algorithm is a posture recognition algorithm or a gesture recognition algorithm. At this time, the NPU recognizes the posture or gesture based on the grayscale image (or grayscale image and depth image), and the calculation result is the recognized posture or gesture or indication information used to indicate the recognized posture or gesture.

[0194] S515: The low-power NPU sends the calculation result to the low-power DSP.

[0195] S516: The low-power DSP sends the calculation result to the business application.

[0196] S517: The business application executes the instruction corresponding to the user operation based on the calculation result.

[0197] In different reference scenarios, the instructions (also called operations) executed by the application based on the calculation results are different.

[0198] For example, in a face unlocking scenario, the application unlocks the screen when the calculation result indicates that the verification is passed (or successful). Conversely, when the calculation result indicates that the verification is not passed (or unsuccessful), the smart perception reference outputs an indication of the verification failure and does not unlock the screen.

[0199] For example, in a facial recognition payment scenario, if the calculation result indicates that the verification is passed (or successful), the application performs the payment operation. Conversely, if the calculation result indicates that the verification is not passed (or unsuccessful), the intelligent perception reference outputs an indication of verification failure and does not perform the payment.

[0200] For example, in an information authentication business scenario, when the calculation result indicates that the authentication has passed (or succeeded), the application outputs a prompt indicating that the authentication has succeeded or enters the main interface of the website performing the information authentication. Conversely, when the calculation result indicates that the authentication has failed (or failed), the intelligent perception reference outputs an indication indicating that the authentication has failed.

[0201] For example, in a posture / gesture control business scenario, an application can store the operations corresponding to multiple postures or gestures, and then execute the instructions corresponding to the recognized posture or gesture when the calculation result is a recognized posture or gesture. For example, the instruction corresponding to the gesture of swiping the hand to the right is to display the next page, and the instruction corresponding to the gesture of swiping the hand to the left is to display the previous page.

[0202] In this method, the electronic device can process TOF data based on the low-power module without relying on the main core CPU (a CPU that integrates other devices and can run the Android system); when using the low-power module to process TOF RAW data, the main core CPU and the Android system can be in a dormant state. When completing business involving TOF RAW data, it can effectively reduce system power consumption and extend the battery life of the electronic device.

[0203] In this embodiment, the low-power ISP, the low-power DSP, and the low-power NPU may be referred to as the "first ISP," the "first DSP," and the "first NPU," respectively. The eye-safety adjustment parameter may be referred to as the "first information," and the AE of the next frame of image may be referred to as the "second message."

[0204] In some embodiments, at least one TOF camera and at least one RGB-AO camera may be present in the front camera of the electronic device at the same time. With the support of the above-mentioned low-power module, among the smart perception services involved in the electronic device, some services can obtain better results by using different cameras to output images under different circumstances. In response to this situation, the present application also provides another image processing method, which can select a camera adapted to the domain business in combination with the business type and / or light intensity of the smart perception business, and process the data collected by the camera with the support of the above-mentioned low-power module, thereby reducing the power consumption of the device while improving the efficiency and quality of business processing.

[0205] Next, combine Figure 6 The above description explains the specific working mode of the above low power consumption module.

[0206] like Figure 6 As shown, the business system 60 may include a TOF camera 601, an RGB-AO camera 602, a MIPI switch 603, an AON ISP 604, a mini DSP 605, an eNPU 606, and an I 2 C 607, ambient light sensor 608 and conventional ISP 609. AON ISP 604, mini DSP 605, eNPU 606 can be respectively ISP, DSP, NPU in the AON (Always ON) camera processing architecture of electronic devices. 2 The low power consumption module composed of C 607 can be the aforementioned Figure 2 Low power consumption module 204; specifically, AON ISP 604, mini DSP 605, eNPU 606, I 2 C 607 can be Figure 2 ISP 204a, DSP 204b, NPU 204c and I 2 C 204d.

[0207] The TOF camera 601 and the RGB-AO camera 602 can both be front cameras of the electronic device.

[0208] Figure 7 The specific positions of the TOF camera 301 and the RGB-AO camera 602 in the electronic device are exemplarily shown.

[0209] Reference Figure 7As shown, the electronic device 600 may include at least: a display screen 610, a middle frame 620, a battery cover 640, and a circuit board 630 located between the display screen 610 and the battery cover 640, wherein the circuit board 630 may be arranged on the middle frame 620, for example, the circuit board 630 may be arranged on a side of the middle frame 620 facing the battery cover 640 (see Figure 7 As shown in (B) in FIG), the circuit board 630 may be disposed on a side of the middle frame 620 facing the display screen 610, and the display screen 610 and the battery cover 640 are located on both sides of the middle frame 620, respectively.

[0210] In order to realize the intelligent sensing function, the TOF camera 601 and the RGB-AO camera 602 can both be set as front cameras on the side of the middle frame 620 facing the display screen 610, and the middle frame 620 is provided with an opening for exposing the lens ends of the TOF camera 601 and the RGB-AO camera 602 ( Figure 7 (not shown), the openings may be two or fused into one. A light-transmitting hole 611 is provided on the display screen 610, and the lens of the TOF camera 601 and the lens of the RGB-AO camera 602 correspond to the light-transmitting hole 611. In some embodiments, the light-transmitting hole 611 may also be two or fused into one. The TOF camera 601 and the RGB-AO camera 602 may be arranged arbitrarily in the XY plane, for example, along the X-axis direction, or, more preferably, along the Y-axis direction. In particular, in the present application, the camera module (i.e., lens) contained in the RGB-AO camera includes but is not limited to a color camera module or a black and white camera module. These two camera modules may be integrated into one camera assembly and included in the RGB camera 602.

[0211] In the embodiment of this application, combined with Figure 7 As shown in (A) in FIG, the TOF camera 601 and the RGB camera can be located in an area near the top edge of the display screen 610. It is understood that the positions of the TOF camera 601 and the RGB-AO camera are not limited to Figure 6 The position shown may also be located at other parts of the display screen 610.

[0212] Reference Figure 7 As shown in (B), the TOF camera 601 and the RGB-AO camera can be electrically connected to the circuit board 330. As an embodiment, the TOF camera 301 and the RGB-AO camera can be connected to the circuit board 330 through the electrical connector ( Figure 7(not shown) are electrically connected to the circuit board 630. For example, the TOF camera 601 and the RGB-AO camera may be provided with a male socket of an electrical connector, and the circuit board 630 may be provided with a female socket of an electrical connector. By plugging the female socket into the male socket, the electrical connection between the TOF camera 601 and the RGB-AO camera and the circuit board 630 is achieved.

[0213] Among them, the circuit board 630 may be provided with a central processing unit CPU (such as the CPU 201 in the aforementioned description), and the TOF camera 601 and the RGB camera are controlled by the processor to capture images. After receiving a service request instruction issued in response to a user operation in the smart perception service (such as the operation of the user picking up the phone in the raise hand to light up the screen service), or a service request instruction generated by a system application (such as a setting application) in the AO service, the CPU can send the service request instruction to the mini DSP 605 in the low-power module, and the mini DSP 605 determines whether to control the TOF camera 601 or the RGB-AO camera 602 to capture the subject based on the service request instruction and the ambient light data obtained from the ambient light sensor 608 to obtain image data. Specifically:

[0214] Case 1: When the service request instruction indicates that the current service is an eye-tracking-related service, regardless of the ambient light data indicating the light intensity in the current environment, the mini DSP 605 will determine that the TOF camera 601 is a camera compatible with the current service, and process the TOF RAW data collected by the TOF camera 601 in conjunction with the low-power module to obtain the service results of the eye-tracking-related service. It is understandable that the eye-tracking service requires specific determination of the specific area of the user's gaze. Compared with other intelligent perception services, the eye-tracking service has higher requirements for the accuracy of the service results. The depth data obtained by the TOF camera is highly accurate, while the RGB camera is a 2D camera and does not have the ability to obtain depth data. Therefore, the TOF camera is more suitable for this type of service than the RGB-AO camera. Using the TOF camera 601 to collect the user's eye movement data can ensure the accuracy of the service results of the eye-tracking-related service.

[0215] Case ②: When the service request instruction indicates that the current service is not an eye tracking-related service and is a smart perception service, the mini DSP 605 further determines the light intensity in the current environment based on the ambient light data. When the light intensity in the current environment is too small (for example, less than the first threshold), the mini DSP 605 will determine that the TOF camera 601 is a camera adapted to the current service, and process the TOF RAW data collected by the TOF camera 601 in combination with the low-power module to obtain the service result of the smart perception service. Since the RGB-AO camera has a poor shooting effect in a dark environment, it will affect the accuracy of the service results, and the TOF camera has a better shooting effect in a dark environment. Therefore, when completing the smart perception service in a low-light environment, the mini DSP 605 will determine that the TOF camera 601 is a camera adapted to the current service to ensure the quality of the captured image and thus ensure the accuracy of the service results.

[0216] Case 3: When the service request instruction indicates that the current service is not an eye tracking-related service but a smart perception service, the mini DSP 605 further determines the light intensity in the current environment based on the ambient light data. If the light intensity in the current environment is large enough (for example, greater than or equal to the first threshold), the mini DSP 605 will determine that the RGB-AO camera 601 (AO mode) is the camera adapted for the current service, and process the AO RAW data collected by the TOF camera 601 in combination with the low-power module to obtain the service results of the smart perception service.

[0217] Case ④: When the service request instruction indicates that the current service is not a smart perception service but an ordinary photo-taking service (for example, the user wants to take a selfie with the front camera), the mini DSP 605 will determine that the RGB-AO camera 601 (RGB mode) is the camera adapted to the current service. Since the pixels of the RAW image obtained by this type of service are too large, the low-power module cannot process the RGB RAW data collected by the RGB camera. At this time, the service system can send the RGB RAW data to the regular ISP 609 for processing.

[0218] In the embodiment of the present application, the business type of acquiring image data based on a TOF camera and processing the TOF RAW data collected by the TOF camera in combination with a low-power module as in Case ① and Case ② can be called a "first type of business"; the business type of acquiring image data based on an RGB-AO camera (AO mode) and processing the AO RAW data collected by the RGB-AO camera in combination with a low-power module as shown in Case ③ can be called a "second type of business"; the business type of acquiring image data based on an RGB-AO camera (RGB mode) and processing the RGB RAW data collected by the RGB-AO camera in combination with a conventional ISP as shown in Case ④ can be called a "third type of business". In the business system 60, the modules and data transmission links involved in the first type of business, the second type of business and the third type of business are all different. Specifically:

[0219] The first type of business is completed by the TOF camera 601 and low-power module in the business system.

[0220] In the process of completing the first type of business, after the mini DSP 605 determines that the TOF camera 601 is the camera adapted to the current business, the mini DSP 605 will send a control instruction to the MIPI switch 603. The MIPI switch 603 switches the working camera to the TOF camera 601 according to the instruction (i.e., establishes a data path with the TOF camera 601). The TOF RAW data output by the TOF camera 601 will then be divided into two parts. One part of the TOF image data is transparently transmitted to the mini DSP 605 through the MIPI switch 603 and AONISP 604, and the other part of the TOF image metadata is transmitted through I 2 C 607 is transparently transmitted to miniDSP605. Mini DSP 605 obtains the eye safety detection result of the TOF camera based on the TOF image metadata, and outputs the AE required for the next frame of TOF RAW data based on the TOF image data, and returns the eye safety detection result and AE to the TOF camera 601 through I2C 607. In addition, mini DSP 605 also obtains infrared images and depth images based on the TOF image data, and sends the infrared images and depth images to eNPU 606. eNPU 606 will perform operations on the acquired infrared images and depth images, and return the business results to mini DSP 605 after obtaining them. For details, please refer to the above Figure 3 Related instructions.

[0221] The second type of business is completed by the RGB-AO camera 602 (AO mode) and the low-power module in the business system.

[0222] In the process of completing the second type of business, after the mini DSP 605 determines that the RGB-AO camera 602 is the camera adapted for the current business, the mini DSP 605 will send a control instruction to the MIPI switch 603. According to the instruction, the MIPI switch 603 switches the working camera to the RGB-AO camera 602 (i.e., establishes a data path with the RGB-AO camera 602). Unlike TOF RAW data, the pixel size of the AO RAW output by the RGB-AO camera 602 is generally 320×240 to 640×400. Therefore, the AO RAW data (which is a grayscale image, not an infrared grayscale image) output by the RGB-AO camera 602 can be directly transmitted to the AON ISP 604 through the MIPI switch 603 for processing. The AORAW data can also be further transmitted to the mini DSP 605. The mini DSP 605 will also obtain the AE required for the AO RAW data based on the acquired AO RAW (in some embodiments, the AE can be referred to as the third information), and return the AE to the RGB-AO camera 602 via I2C 607; the eNPU 606 can obtain the business results based on the AO RAW data and return them to the mini DSP 605.

[0223] The third type of business is completed by the RGB-AO camera 602 (RGB mode) and the conventional ISP 609 in the business system.

[0224] During the third service, after the mini DSP 605 determines that the RGB-AO camera 602 is compatible with the current service, it sends a control instruction to the MIPI switch 603. The MIPI switch 603, in response to the instruction, switches the active camera to the RGB-AO camera 602 (i.e., establishes a data path with the RGB-AO camera 602). However, in RGB mode, the pixel size of the RGB RAW image captured by the RGB-AO camera 602 can reach 4000×3000 or even 8000×6000. This cannot be transparently transmitted by the AON ISP 604 in the low-power module, and the mini DSP 605 may not be able to process it. Therefore, even if a data path exists between the MIPI switch 603 and the RGB-AO camera 602, the RGB RAW data output by the RGB-AO camera 602 will not be transmitted to the low-power module through the MIPI switch 603, but will be directly transmitted to the regular ISP 609. The conventional ISP 609 can process the RGB RAW data, such as exposure control, gain control, white balance, and color correction, and then output it to the display screen of the electronic device for display. Optionally, a conventional DSP ( Figure 6 (not shown), after the conventional ISP processes the RGB RAW data, the conventional DSP processes this data, converts the digital image signal into a standard RGB format image signal, and then outputs it to the display screen of the electronic device for display.

[0225] In this embodiment, the AON ISP 604, mini DSP 605, and eNPU 606 may be referred to as a "first ISP," a "first DSP," and a "first NPU," respectively.

[0226] Combined with the above Figure 6 , another image processing method involved in the embodiment of the present application is further introduced below.

[0227] See Figure 8 The electronic device provided in this application for executing the method may include but is not limited to a TOF camera, an RGB-AO camera, a MIPI switch, a low-power module, and a conventional ISP. The low-power module may be Figure 3 The low-power module shown in , which may include low-power ISP, low-power DSP, low-power NPU and I2C. Figure 8 As shown, the image processing method may include but is not limited to some or all of the following steps:

[0228] S801: The service application sends a service call request to the low-power DSP.

[0229] Among them, the service call request is used to request the call of a service. The service corresponding to the service application can be a smart perception service, which includes but is not limited to face unlocking, face payment, information authentication, gesture control, posture control, etc. In the case where the above-mentioned service is a non-AO service, the above-mentioned service application request can be a service request instruction issued by the electronic device in response to the received user operation. For details, please refer to the relevant description of step S501 above, which will not be repeated here.

[0230] S802: The ambient light sensor sends light intensity data to the low-power DSP.

[0231] S803: The low-power DSP determines an adapted camera according to the type of service request and / or light intensity data.

[0232] S804: The low-power DSP sends a control instruction to the MIPI switch.

[0233] Steps S801 to S804 are the decision-making process of the low-power DSP service category in this method.

[0234] The low-power DSP determines whether to control the TOF camera or the RGB-AO camera to shoot the object and obtain image data based on the business request and the ambient light data obtained from the ambient light sensor.

[0235] When the service request instruction indicates that the current service is an eye tracking-related service, or when the service request instruction indicates that the current service is an intelligent perception service and the light intensity data indicates that the light intensity in the current environment is too low (for example, less than the first threshold), the low-power DSP will determine that the TOF camera is a camera adapted to the current service, and process the TOF RAW data collected by the TOF camera 601 in combination with the low-power module to obtain the service result of the intelligent perception service. At this time, the service type is the first type of service. The above control instruction sent by the low-power DSP to the MIPI switch can control the establishment of a data link between the MIPI switch and the TOF camera. The specific components and specific steps involved in the electronic device completing the first type of service can refer to the aforementioned Figure 3 The relevant instructions will not be repeated here.

[0236] When the service request instruction indicates that the current service is not an intelligent perception service related to eye tracking, and the light intensity data indicates that the light intensity in the current environment is large enough (for example, greater than or equal to the first threshold), the low-power DSP will determine that the RGB-AO camera 601 (the RGB camera is working in AO mode at this time) is a camera adapted to the current service, and process the AO RAW data collected by the RGB-AO camera in combination with the low-power module to obtain the service result of the intelligent perception service. At this time, the service type is the second type of service. The above control instruction sent by the low-power DSP to the MIPI switch can control the establishment of a data link between the MIPI switch and the RGB-AO camera. The specific components and specific steps involved in the electronic device completing the second type of service can refer to the subsequent description of steps S805-S816.

[0237] When the service request instruction indicates that the current service is not a smart sensing service but a normal photo service (for example, a user wants to take a selfie with the front camera), the low-power DSP will determine that the RGB-AO camera 601 (the RGB camera is currently operating in RGB mode) is the camera compatible with the current service, and use the conventional ISP block to process the RGB RAW data collected by the RGB-AO camera, and then send the resulting RGB image for display. In this case, the service type is a third-category service. The control instruction sent by the low-power DSP to the MIPI switch can control the establishment of a data link between the MIPI switch and the RGB-AO camera. For the specific components and steps involved in the electronic device completing the second-category service, please refer to the subsequent description of steps S817-S821.

[0238] S805: The low-power DSP sends instruction information to the RGB-AO camera.

[0239] This instruction information is used to instruct the RGB-AO camera to work in AO working mode and obtain AO RAW data.

[0240] S806: RGB-AO camera collects AO RAW data.

[0241] After receiving the above instruction information, the RGB-AO camera starts (if the RGB-AO camera is not currently started), and the RGB-AO camera starts to collect AO RAW data. Specifically, after the RGB-AO camera starts, it collects TOFRAW data using the second exposure parameter value.

[0242] During the initial acquisition, the second exposure parameter value is the default exposure parameter. In subsequent acquisitions, the first exposure parameter is the received exposure parameter value for indicating the next frame of AO RAW data.

[0243] S807: The RGB-AO camera sends the AO RAW data to the low-power DSP through the MIPI switch and the low-power ISP.

[0244] The AO RAW data is the complete data output by the RGB-AO camera. Specifically, the pixels corresponding to the AO RAW data can be 320×240 or 640×400.

[0245] S808: The low-power DSP calculates the AE of the next frame of image.

[0246] The low-power DSP uses AE to calculate the next frame of AOF RAW data generated by the RGB-AO camera. AE can include exposure parameter values and exposure time.

[0247] S809: The low-power DSP sends the eye safety detection result and the AE of the next frame of image to the RGB-AO camera via I2C.

[0248] S810: RGB-AO camera adjusts AE.

[0249] The RGB-AO camera adjusts the exposure parameter to the aforementioned AE value, so that when the RGB-AO camera acquires the next frame of AO RAW data, exposure is performed according to the aforementioned AE value.

[0250] S811: The low-power DSP sends the AO RAW data to the low-power NPU.

[0251] S812: The low-power NPU performs calculations on the AO RAW data to obtain calculation results.

[0252] The low-power NPU can run the corresponding target algorithm to operate on the AO RAW data to obtain the operation result, which is the business processing result corresponding to the business.

[0253] S813: The low-power NPU sends the calculation result to the low-power DSP.

[0254] S814: The low-power DSP sends the calculation result to the business application.

[0255] S815: The application executes the instruction corresponding to the user operation based on the calculation result.

[0256] For the specific information of steps S814 to S815, please refer to the above description of steps S513 to S517, which will not be repeated here.

[0257] S816: The low-power DSP sends instruction information to the RGB-AO camera.

[0258] This instruction is used to instruct the RGB-AO camera to work in RGB mode and obtain RGB RAW data.

[0259] S817: RGB-AO camera collects AO RAW data.

[0260] After receiving the above instruction, the RGB-AO camera starts (if it is not currently started) and begins collecting RGB RAW data in RGB mode. Specifically, after starting, the RGB-AO camera collects TOF RAW data using the third exposure parameter value. During initial collection, the third exposure parameter value is the default exposure parameter.

[0261] S818: RGB-AO camera sends RGB RAW data to a regular ISP.

[0262] S819: Conventional ISP generates RGB image.

[0263] S820: The conventional ISP sends the RGB image to the service application.

[0264] S821: The business application outputs an RGB image.

[0265] The RGB RAW data is the complete data output by the RGB-AO camera. Specifically, the pixels corresponding to the RGB RAW data can be 4000×3000 or 8000×6000.

[0266] The service application program receiving the RGB image may be a camera application.

[0267] A conventional ISP performs processing on the RGB RAW data, such as exposure control, gain control, white balance, and color correction, before outputting it to the electronic device's display screen for display. Optionally, after processing the RGB RAW data, the conventional ISP may also send the processed data to a conventional DSP, which processes the data and converts it into a standard RGB image format. The conventional DSP then outputs the image to the electronic device's display screen for display.

[0268] In this embodiment, the low-power ISP, the low-power DSP, and the low-power NPU may be referred to as a “first ISP,” a “first DSP,” and a “first NPU,” respectively.

[0269] Next, the architecture of the electronic device provided by this application is introduced.

[0270] The electronic device provided in this application may be a device having a TOF camera and / or an RGB-AO camera, including but not limited to smartphones, tablet computers, smart screens, wearable devices, vehicle-mounted devices, augmented reality (AR), virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDAs), artificial intelligence (AI) terminals, and other terminal devices. The embodiments of this application do not impose any restrictions on the specific types of electronic devices.

[0271] For example, Figure 9 A schematic structural diagram of an electronic device 100 provided in an embodiment of the present application is shown.

[0272] The electronic device 100 may include a processor 110, a conventional neural-network processing unit (NPU) 111, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an ambient light sensor 180L, a camera 193, a display screen 194, etc. It will be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0273] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), a conventional image signal processor (ISP), a controller, a memory, a video codec, a conventional digital signal processor (DSP), a baseband processor, etc. The different processing units may be independent devices or integrated into one or more processors.

[0274] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0275] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0276] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an integrated circuit (IC). 2 C) interface, inter-integrated circuit sound (I2S) interface, pulse code modulation (PCM) interface, universal asynchronous receiver / transmitter (UART) interface, mobile industry processor interface (MIPI), general-purpose input / output (GPIO) interface, subscriber identity module (SIM) interface, and / or universal serial bus (USB) interface, etc.

[0277] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0278] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0279] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, etc. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be provided in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be provided in the same device.

[0280] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0281] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0282] The electronic device 100 can implement the shooting function through a conventional ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0283] A conventional ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. A conventional ISP can also perform algorithmic optimization on image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, a conventional ISP can be installed in camera 193.

[0284] The camera 193 is used to capture still images or videos. The camera 193 may include a TOF camera and an RGB-AO camera. The TOF camera can not only collect depth data of the object being photographed, but also obtain an infrared grayscale image of the object being photographed. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to a conventional ISP for conversion into a digital image signal. The conventional ISP outputs the digital image signal to a conventional DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0285] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0286] The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0287] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0288] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light brightness. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with a proximity light sensor to detect whether electronic device 100 is in a pocket to prevent accidental touches.

[0289] The processor 110 may also be integrated with a low-power module, which may include a low-power ISP, a low-power DSP, and a low-power NPU. In some embodiments, the low-power module may receive light intensity data sent by the ambient light sensor.

[0290] In the electronic device 100, the conventional ISP, the conventional DSP, and the low-power module are included in an external chip of the processor 110, and can also be integrated on the processor 110. The conventional ISP can be the ISP with the best performance among all the ISPs included in the electronic device 100, and the conventional DSP can be the DSP with the best performance among all the DSPs included in the electronic device 100. Specifically, the conventional ISP and the conventional DSP can be used to process algorithms with large computational load, such as video encoding and decoding, graphic image processing, visual image processing, etc. In the present application, the conventional ISP and the conventional DSP can process data collected by various cameras, including but not limited to RGB RAW data collected by an RGB camera, AO RAW data collected by an RGB-AO camera, and TOF RAW data collected by a TOF camera.

[0291] The performance of a low-power ISP is lower than that of a conventional ISP, but it can also perform some relatively simple processing on image data. For example, in this application, when an electronic device uses an RGB-AO camera to perform eye tracking or AOD services, the low-power ISP can perform exposure control, gain control, and other processing on the AO RAW data collected by the RGB-AO camera. However, when an electronic device uses a TOF camera to perform eye tracking or AOD services, the low-power ISP can only transparently transmit a portion of the TOFRAW data collected by the TOF camera to the low-power DSP.

[0292] The performance of a low-power DSP is lower than that of a conventional DSP, and it can perform some relatively simple processing on image data. For example, in this application, when an electronic device performs eye tracking and AOD services, the low-power DSP can obtain the corresponding depth image and infrared image based on TOF RAW data, determine the exposure parameter AE corresponding to the next frame of the camera, and calculate the human eye safety setting parameters based on TOF RAW data.

[0293] The performance of the low-power NPU is lower than that of the conventional NPU. It can perform calculations on depth images and infrared images to obtain calculation results.

[0294] In some embodiments, before the electronic device 100 begins formally processing the smart sensing service, the ambient light sensor can collect ambient light data and transmit the ambient light data to the low-power DSP. The low-power DSP can determine the camera that is compatible with the current service based on the ambient light data and the above-mentioned service type, and obtain image data collected by the camera that is compatible with the service through corresponding operations.

[0295] In the present application method, the electronic device 100 can be used to execute the image processing method described above. Specifically, the electronic device can use the low-power module contained therein to process the TOF RAW data without relying on the processor 110, obtain the business results and return them to the processor 110, and the processor 110 performs the operation corresponding to the business result (for example, unlocking the screen when the face comparison result matches). In the process of using the low-power module to process the TOF RAW data, the processor 110 and the Android system running on the processor 110 can be in a sleep state, which can effectively control the power consumption of the electronic device.

[0296] It should be understood that each step in the above method embodiments provided herein can be implemented by hardware integrated logic circuits in a processor or by software instructions. The method steps disclosed in the embodiments of this application can be directly implemented as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0297] The present application also provides an electronic device, which may include a memory and a processor. The memory may be used to store a computer program, and the processor may be used to call the computer program in the memory to enable the electronic device to perform the various functions or steps performed by the electronic device in the above method embodiment.

[0298] The present application also provides a computer program product, which includes: a computer program (also called code, or instruction), which, when executed on the electronic device, enables the electronic device to execute the functions or steps executed by the electronic device in the method embodiment.

[0299] The present application also provides a computer-readable storage medium storing a computer program (also referred to as code or instruction). When the computer program is run on the electronic device, the electronic device executes the functions or steps executed by the electronic device in the above method embodiment.

[0300] The various implementation modes of this application can be combined arbitrarily to achieve different technical effects.

[0301] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0302] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0303] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0304] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0305] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0306] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0307] The above description is merely a specific embodiment of the present application, but the scope of protection of the embodiments of the present application is not limited thereto. Any person skilled in the art can easily conceive of changes or substitutions within the technical scope disclosed in the embodiments of the present application, and such changes or substitutions should be included in the scope of protection of the embodiments of the present application. Therefore, the scope of protection of the embodiments of the present application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: Applicable to an electronic device, the electronic device includes a time-of-flight TOF camera, a first image signal processor ISP, a first digital signal processor DSP, a first neural network processor NPU and an integrated circuit bus I 2 C, the method comprising: The TOF camera collects first data, the first data including image data and image metadata, the image data is transmitted to the first DSP via the first ISP, and the image metadata is transmitted to the first DSP via the I 2 C is delivered to the first DSP; Processing the image metadata by the first DSP to obtain first information, where the first information indicates whether the TOF camera is safe for human eyes; When the first information indicates that the TOF camera is safe for human eyes, the image data is processed by the first DSP to obtain a depth image and a grayscale image.

2. The method according to claim 1, characterized in that The collecting first data by the TOF camera includes: In response to the service request, collecting first data through the TOF camera; After obtaining the depth image and the grayscale image, the method further includes: The first NPU performs operations on the depth image and the grayscale image to obtain an operation result of the service corresponding to the service request.

3. The method according to claim 2, characterized in that The first electronic device further includes a central processing unit (CPU). After responding to the service request, the CPU of the electronic device is in a dormant state.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The image data is processed by the first DSP to obtain second information, and the second information is processed by the I 2 C is transmitted to the TOF camera, and the second information represents the exposure parameters used by the TOF camera when collecting the next frame of data.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: If the first information indicates that the TOF camera is unsafe for human eyes, stop collecting data through the TOF camera; When the first information indicates that the TOF camera is safe for human eyes, control the TOF camera to continue collecting data.

6. The method according to any one of claims 1 to 5, characterized in that The image data is transparently transmitted to the first DSP via the first ISP, and the image metadata is transparently transmitted to the first DSP via the I 2 C is transparently transmitted to the first DSP.

7. The method according to any one of claims 1 to 6, characterized in that The electronic device further includes an ambient light sensor configured to collect light intensity data. In response to the service request, collecting the first data using the time-of-flight (TOF) camera includes: Analyzing and processing the service request based on the DSP, and collecting first data through the TOF camera when it is determined that the service corresponding to the service request is an eye tracking service; Alternatively, the DSP analyzes and processes the service request, and when it is determined that the service corresponding to the service request is not an eye tracking service and the light intensity data indicates that the ambient brightness is less than a first threshold, the TOF camera collects first data.

8. The method according to any one of claims 1 to 7, characterized in that The electronic device further includes an RGB-AO camera, and the method further includes: When it is determined that the service corresponding to the service request is not an eye tracking service and the light intensity data indicates that the ambient brightness is greater than a first threshold, starting to collect second data through the RGB-AO camera; and transmitting the second data to the first DSP via the first ISP; The first DSP processes the second data to obtain third information, and the third information is processed by the I 2 C is transmitted to the RGB-AO camera, where the third information represents the exposure parameters used by the RGB-AO camera when capturing the next frame of data; The second data is processed by the first NPU to obtain a calculation result of the service corresponding to the service request.

9. An electronic device, characterized in that: The electronic device includes: one or more processors, a memory and a display screen; The memory is coupled to the one or more processors, and is configured to store computer program codes, where the computer program codes include computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 8.

10. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method as described in any one of claims 1-8.

11. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 8.

Images