Display method and related device

Abstract

The embodiment of the present application provides a display method and related devices. The method is applied to a terminal device including a TOF camera and an AO camera, and the method includes: the terminal device displays a first interface; at a first time point, a first image is captured by the TOF camera, and the first image includes a face and a first gesture; after the first image is captured, the AO camera is turned on; a second image is captured by the AO camera, and the second image includes a face and a first gesture; according to the first gesture, a second interface is displayed, and the content displayed on the second interface is different from the content displayed on the first interface; at a second time point, a third image is captured by the TOF camera, and the third image includes a face but does not include objects other than the face; after the second time point, the AO camera is turned off. In this way, when the user makes a gesture, the AO camera is turned on to realize gesture detection and realize the function corresponding to the gesture. When no gesture is detected, the AO camera can be turned off to reduce the operating time of the AO camera and reduce power consumption.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a display method and related devices. Background Art

[0002] With the development of technology, mobile phones and other terminal devices have more and more functions. For example, mobile phones and other terminal devices can realize functions such as screenshots and page turning by recognizing user gestures.

[0003] Currently, terminal devices recognize user gestures through always-on cameras (AO cameras).

[0004] However, when the AO camera is running continuously, the terminal device will not be able to sleep and consume a lot of power. Summary of the Invention

[0005] The present invention provides a display method and related apparatus for use in the field of terminal technology. When a TOF camera detects at least two objects, it activates an AO camera for gesture recognition, eye tracking, and other tasks. This allows for tracking user gestures and eyes, while also reducing the operating time of the AO camera, lowering power consumption in the terminal device and improving the user experience.

[0006] In a first aspect, an embodiment of the present application proposes a display method. The method is applied to a terminal device, the terminal device including a time-of-flight (TOF) camera and a normally powered AO camera, and the method includes: the terminal device displays a first interface; at a first time point, the terminal device captures a first image through the TOF camera, the first image including a face and a first gesture; after capturing the first image, the terminal device turns on the AO camera; the terminal device captures a second image through the AO camera, the second image including a face and the first gesture; the terminal device displays a second interface based on the first gesture, the content displayed on the second interface being different from the content displayed on the first interface; at a second time point, the terminal device captures a third image through the TOF camera, the third image including a face but not including any objects other than the face; after the second time point, the terminal device turns off the AO camera.

[0007] Optionally, the terminal device turns on the AO camera based on the fact that the first image includes two objects and the depth information of the two objects is different. In some embodiments, the third image may also not include a face. The terminal device turns off the AO camera based on the fact that the third image includes zero or one object. In this embodiment of the present application, turning on the AO camera can be based on the fact that the image captured by the TOF camera includes two objects with different depth information.

[0008] Optionally, the terminal device turns on the AO camera based on the first image including the human face and the first gesture. In some embodiments, the third image may also not include the human face. The terminal device turns off the AO camera based on the third image including zero or one object.

[0009] In a possible implementation, the terminal device may turn off the AO camera after capturing a third image containing zero or one object within a certain period of time. Alternatively, it can be understood that the terminal device may turn off the AO camera after continuously capturing multiple third images containing zero or one object.

[0010] The first interface may correspond to Figure 5B The first gesture can be palm up, back of hand down, fist, or any other gesture, which is not limited here. The second interface can correspond to Figure 5B The first interface may be an interface of a video application, an interface of a gallery application, or an interface of other applications, which is not limited here.

[0011] In this way, when the user makes a gesture, the AO camera is turned on to detect the gesture and implement the function corresponding to the gesture. When no gesture is detected, the AO camera can be turned off to reduce the operating time of the AO camera and reduce power consumption.

[0012] Optionally, the method also includes: at a third time point, the terminal device captures a fourth image through the TOF camera, the fourth image includes a human face and the first object; after capturing the fourth image, the terminal device turns on the AO camera; the terminal device captures a fifth image through the AO camera, the fifth image includes a human face and the first object; the terminal device continues to display the second interface.

[0013] In this way, when the user does not make a gesture, the AO camera is turned on when two objects are detected. When the terminal device does not detect a gesture, the displayed interface remains unchanged.

[0014] Optionally, the method further includes: at a fourth time point, the terminal device captures a sixth image through the TOF camera, and the sixth image does not include the object; after the fourth time point, the terminal device turns off the AO camera.

[0015] In this way, when no object is detected, the AO camera is turned off, which reduces the running time of the AO camera and reduces power consumption.

[0016] Optionally, after capturing the first image, the terminal device turns on the AO camera, including: starting from the first time point, within the first time period, the images captured by the terminal device through the TOF camera are all the first images; starting from the first time point, after the first time period, the terminal device turns on the AO camera.

[0017] The first duration can be 3 seconds or 5 seconds. The embodiment of the present application does not limit the specific value of the first duration.

[0018] In this way, after the gesture is maintained for a period of time, the terminal device turns on the AO camera. This can reduce false recognition and reduce the frequent activation of the AO camera.

[0019] Optionally, the method also includes: at the fifth time point, the terminal device captures a seventh image through the TOF camera, and the seventh image includes a human face and a first gesture; starting from the fifth time point, after a second time period, the terminal device captures an eighth image through the TOF camera, and the eighth image includes a human face but does not include objects other than the human face; the second time period is less than the first time period; starting from the fifth time point, after the second time period, the terminal device does not turn on the AO camera; the terminal device displays a second interface.

[0020] The second duration can be 1 second or 2 seconds, which is not limited here. The object can be an object such as a book or a living thing such as a cat, which is not limited in this embodiment of the application.

[0021] In this way, when the gesture is maintained for a short time, the terminal device does not turn on the AO camera. This can reduce false recognition and shorten the operating time of the AO camera.

[0022] Optionally, the terminal device displays a second interface according to the first gesture, including: the first gesture corresponds to the page turning function, the second interface includes the next page content of the first interface, or includes the previous page content of the first interface; or, the first gesture corresponds to the screenshot function, and the second interface includes a screenshot of the first interface.

[0023] In this way, different functions can be achieved through various gestures.

[0024] Optionally, the method also includes: the terminal device is in a screen-off state; at a sixth time point, the terminal device captures a ninth image through a TOF camera, and the ninth image includes a face; after the sixth time point, the terminal device displays a lock screen interface, and the lock screen interface includes an input box; in response to the user entering a lock screen password in the input box, the terminal device displays an unlocked interface.

[0025] The lock screen interface may correspond to the interface shown in Figure 8. The input box may correspond to the unlock input box in Figure 8. The unlocked interface may be the main interface of the terminal device or the interface of the application running before the lock screen, which is not limited here.

[0026] In this way, the function of lighting up the screen when someone comes can be realized.

[0027] Optionally, the method also includes: at the seventh time point, the terminal device displays the third interface; from the seventh time point to the eighth time point, the terminal device does not receive any user operation; at the eighth time point, the terminal device captures the tenth image through the TOF camera, and the tenth image includes a human face; after the eighth time point, the terminal device turns on the AO camera; at the ninth time point, when the human face is looking at the display screen, the terminal device captures the eleventh image through the AO camera; starting from the seventh time point, within the third time period, the terminal device displays the third interface.

[0028] In an embodiment of the present application, turning on the AO camera can be based on the image captured by the TOF camera including at least one object after the screen is turned off and the camera-related process (for example, the camera service process) is called.

[0029] The third duration may correspond to the third preset duration described below. The third duration may be 15 seconds or 30 seconds, and the specific value of the third duration is not limited.

[0030] In this way, the terminal device can realize the function of watching the screen without turning it off.

[0031] Optionally, the eighth time point is later than the seventh time point, the duration between the eighth time point and the seventh time point is greater than the fourth duration, and the fourth duration is less than the third duration.

[0032] The fourth duration may correspond to the second preset duration described below. The second duration may be 10 seconds or 20 seconds, and the specific value of the second duration is not limited.

[0033] Optionally, the AO camera may be turned on after the fourth period of screen-off timing, and the image captured by the TOF camera includes at least one object.

[0034] In this way, after a period of time when the screen is turned off, the detection of watching the screen without turning it off is performed, which reduces the frequent opening of the AO camera and reduces the running time of the AO camera.

[0035] Optionally, the method further includes: after the ninth time point, the terminal device turns off the AO camera.

[0036] In this way, the terminal device turns off the AO camera after resetting the screen-off timing, thereby reducing the operating time of the AO camera and lowering power consumption.

[0037] In this way, the terminal device turns off the AO camera after retiming, reducing the operating time of the AO camera and reducing power consumption.

[0038] Optionally, the method also includes: at the tenth time point, the terminal device displays a third interface; at the eleventh time point, the terminal device captures a twelfth image through the TOF camera, and the twelfth image includes a third object; after the eleventh time point, the terminal device turns on the AO camera; starting from the tenth time point, within a third time period, the terminal device captures a thirteenth image through the AO camera, and the thirteenth image includes the third object, but does not include a face looking at the display screen; starting from the tenth time point, after the third time period, the terminal device turns off the screen.

[0039] In this way, when the terminal device does not detect that the user is looking at the display screen and the screen off timer ends, the screen will be turned off, thereby reducing power consumption.

[0040] Optionally, the method further includes: starting from the tenth time point, after a third period of time, the terminal device turns off the AO camera.

[0041] In this way, when the terminal device turns off the screen, the AO camera is turned off, which reduces the operating time of the AO camera and reduces power consumption.

[0042] Optionally, the method also includes: at the twelfth time point, the terminal device displays a third interface; from the twelfth time point, within a third time period, the terminal device captures a fourteenth image through the TOF camera, the fourteenth image does not include the object, and the terminal device does not turn on the AO camera; from the twelfth time point, after the third time period, the terminal device turns off the screen.

[0043] In this way, when the terminal device does not detect an object, the AO camera will not be turned on, thus reducing power consumption. When the timer ends, the screen will be turned off, thus reducing power consumption.

[0044] Optionally, after capturing the first image, the terminal device turns on the AO camera, including: the terminal device obtains first depth information based on the first image, the first depth information includes: secondary depth of field information, the secondary depth of field information indicates at least two objects; the terminal device obtains status information of the terminal device, the status information is used to indicate whether the terminal device has a screen on and whether it is unlocked; based on the first depth information and the status information, it is determined that the terminal device has a screen on and is unlocked, and the first image has a secondary depth of field, and the AO camera is turned on.

[0045] Optionally, the secondary depth of field indicates that the image captured by the TOF camera contains at least two objects at different depths. Depth can be understood as the distance between the object and the terminal device. Different depths can be understood as different distances between the object and the terminal device.

[0046] It should be noted that the object refers to the object within the detection range of the TOF camera, and can be an object within a preset distance directly in front of the terminal device display screen.

[0047] In this way, when the terminal device detects at least two objects, it turns on the AO camera to implement the gesture detection function.

[0048] Optionally, after the second time point, the terminal device turns off the AO camera, including: the terminal device obtains second depth information based on the third image, the second depth information includes: first-level depth of field information, the first-level depth of field information indicates an object; the terminal device turns off the AO camera based on the second depth information and the third image is the first-level depth of field.

[0049] Optionally, the first-level depth of field indicates that the image captured by the TOF camera contains an object.

[0050] In this way, when the terminal device detects an object, it turns off the AO camera to reduce power consumption.

[0051] Optionally, starting from the first time point and after the first time duration, the terminal device turns on the AO camera, including: starting from the first time point and within the first time duration, the terminal device collects N first images through the TOF camera; obtains N first depth information based on the N first images; starting from the first time point and after the first time duration, obtains status information of the terminal device, the status information is used to indicate whether the terminal device has a screen on and is unlocked; the terminal device determines that the terminal device has a screen on and is unlocked, and the first image has a secondary depth of field based on the first depth information and the status information, and turns on the AO camera.

[0052] In this way, when the terminal device detects at least two objects multiple times in a row, it turns on the AO camera to implement the gesture detection function, which can reduce false recognition.

[0053] In a second aspect, embodiments of the present application provide a terminal device, which may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device may be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver functions, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc.

[0054] The terminal device includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the method of the first aspect.

[0055] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of the first aspect.

[0056] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is run, it enables the computer to execute the method of the first aspect.

[0057] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute the method described in the first aspect.

[0058] It should be understood that the second to fifth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0060] Figure 2 A schematic diagram of the software structure of a terminal device according to an embodiment of the present application;

[0061] Figure 3 A schematic diagram of the flow of software module interactions involved in capturing images with a TOF camera according to an embodiment of the present application;

[0062] Figure 4A A schematic diagram of a scene corresponding to a zero-order depth of field in which depth information is provided in an embodiment of the present application;

[0063] Figure 4B A schematic diagram of a scene corresponding to a zero-order depth of field in which depth information is provided in an embodiment of the present application;

[0064] Figure 4C A schematic diagram of a scene corresponding to a first-level depth of field in which depth information is provided in an embodiment of the present application;

[0065] Figure 4D A schematic diagram of a scene corresponding to a first-level depth of field in which depth information is provided in an embodiment of the present application;

[0066] Figure 4E A schematic diagram of a scene corresponding to a first-level depth of field in which depth information is provided in an embodiment of the present application;

[0067] Figure 4F A schematic diagram of a scene corresponding to a first-level depth of field in which depth information is provided in an embodiment of the present application;

[0068] Figure 5A A schematic diagram of a flow chart of a display method provided in an embodiment of the present application;

[0069] Figure 5B A schematic diagram of an application scenario provided in an embodiment of the present application;

[0070] Figure 5C A schematic diagram of a flow chart of a display method provided in an embodiment of the present application;

[0071] Figure 5D A schematic diagram of an application scenario provided in an embodiment of the present application;

[0072] Figure 5E A schematic diagram of a flow chart of a display method provided in an embodiment of the present application;

[0073] Figure 6 A flowchart illustrating the software module interactions involved in the process of activating the AO camera in a gesture detection scenario provided in an embodiment of the present application;

[0074] Figure 7 A flowchart illustrating the software module interactions involved in the process of shutting down the AO camera in a gesture detection scenario provided in an embodiment of the present application;

[0075] Figure 8A A schematic diagram of a flow chart of a display method provided in an embodiment of the present application;

[0076] Figure 8B A schematic diagram of an application scenario provided in an embodiment of the present application;

[0077] Figure 9 An interactive flow chart of a software module corresponding to an automatic screen-lighting method provided in an embodiment of the present application;

[0078] Figure 10A A flow chart of a display method provided in an embodiment of this application;

[0079] Figure 10B A schematic diagram of an application scenario provided in an embodiment of the present application;

[0080] Figure 10C A flow chart of a display method provided in an embodiment of this application;

[0081] Figure 10D A schematic diagram of an application scenario provided in an embodiment of the present application;

[0082] Figure 10E A schematic diagram of an application scenario provided in an embodiment of the present application;

[0083] Figure 10F A flow chart of a display method provided in an embodiment of this application;

[0084] Figure 11 A schematic diagram of the software interaction flow during the AO camera activation process when the user is looking at the screen without turning off the screen, provided in an embodiment of the present application;

[0085] Figure 12 A schematic diagram of the software module interaction process involved in the process of shutting down the AO camera in the "watching without turning off the screen" scenario provided in an embodiment of the present application;

[0086] Figure 13 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0087] 1. Other terms

[0088] In the embodiments of this application, terms such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the terms "first chip" and "second chip" are used solely to distinguish between different chips and do not define their order. Those skilled in the art will understand that terms such as "first" and "second" do not define the quantity or execution order, and do not necessarily define differences.

[0089] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0090] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0091] 2. Terminal equipment

[0092] The terminal device of the embodiment of the present application may also be any form of electronic device. For example, the electronic device may include a handheld device, a vehicle-mounted device, etc. with a gesture recognition function. For example, some electronic devices include: mobile phones, tablet computers, PDAs, laptop computers, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, vehicle-mounted devices, wearable devices, terminal devices in 5G networks or future evolved public land mobile communication networks. The terminal equipment in the network (PLMN), etc., is not limited to this in the embodiments of the present application.

[0093] As an example and not a limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0094] In addition, in the embodiments of the present application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0095] The electronic devices in the embodiments of the present application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0096] In the embodiments of the present application, the electronic device or each network device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also known as main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0097] With the development of technology, mobile phones and other terminal devices have more and more functions. For example, mobile phones and other terminal devices can realize functions such as unlocking and payment by recognizing faces; mobile phones and other terminal devices can also realize functions such as screenshots and page turning by recognizing user gestures.

[0098] Currently, terminal devices use time-of-flight (TOF) cameras to implement functions such as face unlocking and face payment; terminal devices use always-on cameras (AO cameras) to recognize user gestures.

[0099] For example, Figure 1 This is a hardware structure diagram of a terminal device provided in an embodiment of the present application. Figure 1 As shown, the terminal device includes a TOF transmitting device 101, a TOF receiving device 102 and an AO camera 103.

[0100] The TOF transmitter 101 is used to transmit light pulses, and the TOF receiver 102 is used to receive light reflected from the target object. The terminal device obtains the distance to the target object and the depth information of the image by detecting the flight (round-trip) time of the light pulse.

[0101] The AO camera 103 is used to recognize the gesture of the target object.

[0102] The TOF transmitting device 101, the TOF receiving device 102 and the AO camera 103 can be located at the top of the terminal device or at other locations of the terminal device, which is not limited here.

[0103] In some embodiments, the AO camera in the terminal device is integrated into an always-on (AON) module. Specifically, when the terminal device is in the screen-off state, the terminal device can control the AO camera to operate in a low-power mode. When the terminal device detects that the user is in front of the display screen while in the screen-off state, the terminal device can turn on the screen and invoke the always-on display (AOD) function to display information such as the time on the display screen.

[0104] In some embodiments, due to hardware cost constraints, the terminal device does not support the AON module. If the terminal device continues to run the AO camera, it will not be able to sleep, consuming a lot of power. If the terminal device turns off the AO camera, the terminal device will not be able to perform the screen off detection function.

[0105] In light of this, embodiments of the present application provide a display method and related apparatus. A time-of-flight (TOF) camera is integrated into a coprocessor. When the TOF camera detects two objects, the AO camera is activated for gesture recognition, eye tracking, and other functions. This allows for tracking user gestures and eyes, while also reducing the AO camera's operating time, lowering power consumption in the terminal device and improving the user experience.

[0106] For ease of understanding, the software system of the terminal device is described below. The software system of the terminal device can adopt a layered architecture, event-driven architecture, micro-core architecture, microservice architecture, or cloud architecture, etc., which will not be described in detail here. The embodiment of this application takes the Android system with a layered architecture as an example to illustrate the software structure of the terminal device.

[0107] Figure 2 This is a block diagram of the software structure of the terminal device in an embodiment of the present application. The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into five layers: application layer, application framework layer, hardware abstraction layer (HAL), kernel layer, and hardware layer.

[0108] The application layer can include a series of application packages. The application layer runs the application by calling the application programming interface (API) provided by the application framework layer. Figure 2 As shown, the application package may include applications such as camera, calendar, map, phone, system UI, etc.

[0109] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0110] like Figure 2 As shown, the application framework layer may include a window manager (window manager service), a display manager (display manager service), a power manager (power manager service), a camera manager (camera service), etc.

[0111] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, touch the screen, drag the screen, take screenshots, etc.

[0112] The display manager manages the lifecycle of the display. It determines how to control its logical display based on the currently connected physical display device, sends notifications to the system and applications when the state changes, etc. In some embodiments, the display manager can also switch logical screen mappings, screen resolutions, etc.

[0113] The power manager manages the powering on and off of various hardware components in a terminal device. For example, it can power on and off an AO camera or a display. The power manager can also adjust display brightness, enable low-power mode for the terminal device, and keep the CPU awake.

[0114] The camera manager manages the cameras on the terminal device. For example, the camera manager can control the display-side camera (also called the front camera) as the AO camera for gesture recognition, eye tracking, etc.; the camera manager can also control the front camera for taking photos, etc.

[0115] The camera manager includes a camera server and an AO camera management module. The camera server can determine whether to call the AO camera management module or turn on the screen based on the depth information transmitted by the sensor hardware abstraction and the status information of the terminal device.

[0116] The AO camera management module can manage the AO camera and control whether the AO camera is turned on or off.

[0117] The purpose of the HAL layer is to abstract the hardware, providing a unified interface for querying hardware devices for upper-layer applications, or providing data storage services for upper-layer applications. Figure 2 As shown, the HAL layer may include AO camera hardware abstraction (AO camera HAL) and sensor hardware abstraction (sensor hidl).

[0118] The AO camera hardware abstraction includes the following: Smart Service (AO service). The Smart Service can control the AO camera to be turned on or off.

[0119] The sensor hardware abstraction can control sensors and monitor sensor-driven event change notifications.

[0120] Multiple modules in the HAL layer may comply with the Hardware Abstraction Layer Interface Description Language (HIDL) or AIDL.

[0121] The kernel layer is the layer between hardware and software. Figure 2 As shown, the kernel layer may include one or more of the following: AO camera driver, sensor driver, display driver, etc.

[0122] The sensor driver supports reporting the depth information obtained by the TOF camera to the sensor hardware abstraction.

[0123] In the embodiment of the present application, the sensor driver includes a sensor subsystem framework and a sensor subsystem driver, which are located in the coprocessor (SCP).

[0124] It is understood that the terminal device may include an application processor (AP) and a coprocessor (CP). The application processor includes the aforementioned application framework layer, hardware abstraction layer, kernel layer, etc.; the coprocessor includes a sensor hub.

[0125] The sensor control center includes: TOF algorithm and TOF driver.

[0126] It is understandable that the code corresponding to the TOF driver is integrated in the sensor control center. In the embodiment of the present application, the TOF algorithm and the driver code corresponding to the TOF driver must follow the driver format of the sensor control center. The embodiment of the present application does not specifically limit the driver file format of the TOF driver. Figure 2 As shown, the hardware layer includes: AO camera, TOF camera, etc.

[0127] It is understood that in the embodiment of the present application, the power supply of the TOF camera is consistent with the power supply of the sensor control center, and both are always powered. Exemplarily, the power supply corresponding to the TOF camera is set to regulator-always-on.

[0128] It should be understood that in some embodiments, layers that implement the same function may be referred to by other names, or a layer that can implement the functions of multiple layers may be considered as one layer, or a layer that can implement the functions of multiple layers may be divided into multiple layers. This embodiment of the present application does not limit this.

[0129] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be implemented independently or in combination with each other. For the same or similar concepts or processes, some embodiments may not be described in detail.

[0130] For ease of understanding, the following describes the transmission process of depth information obtained from images taken by a TOF camera and the status of the terminal device.

[0131] For example, Figure 3 This is a schematic diagram of the software module interactions involved in capturing images with a TOF camera according to an embodiment of the present application. Figure 3 As shown, the process includes:

[0132] S301: A TOF camera captures an image to obtain a first type of image, and the TOF camera driver transmits the first type of image to a TOF algorithm module. The TOF algorithm module obtains depth information based on the first type of image.

[0133] In the embodiment of the present application, the TOF camera can be a 4×4 direct measurement time-of-flight camera (direct time-of-flight, DTOF camera), or it can be a DTOF camera of other specifications, or it can be an indirect measurement time-of-flight camera (indirect time-of-flight, iTOF camera). The embodiment of the present application does not limit the specific specifications and types of the TOF camera.

[0134] The 4×4 DTOF camera can emit 4×4 shaped laser beams and obtain the depth information of the space covered by the 4×4 laser beams by detecting the return time of the 4×4 laser beams.

[0135] It's understandable that during actual detection, the light received by the TOF receiver in a TOF camera is a combination of ambient light and reflected light. Ambient light interferes with reflected light, so the TOF camera may be unable to detect objects at a distance. Therefore, TOF cameras have a limited detection range.

[0136] In this embodiment of the present application, depth information includes: zero-order depth of field, first-order depth of field, and second-order depth of field. Zero-order depth of field indicates that there is no object within the detection range of the TOF camera; first-order depth of field indicates that there is one object within the detection range of the TOF camera; and second-order depth of field indicates that there are at least two objects within the detection range of the TOF camera. The objects can be living objects such as people and animals, or inanimate objects such as vehicles.

[0137] Taking the TOF camera in the terminal device, and the TOF camera located on the side of the display screen as an example, when there is no object within the detection range of the TOF camera, the depth information transmitted by the TOF algorithm module is the zero-order depth of field.

[0138] For example, Figure 4A and Figure 4B Schematic diagram of scenes corresponding to the zero-order depth of field in two types of depth information provided in the embodiments of the present application. Figure 4A The scene shown includes a terminal device 401. There is no object in the scene, and the depth information is a zero-order depth of field. Figure 4BThe illustrated scene includes a terminal device 401 and a first object 402. In the scene, the first object 402 is located on the back panel side of the terminal device 401, and there is no object within the detection range of the TOF camera of the terminal device 401.

[0139] When there is only one object within the detection range of the TOF camera of the terminal device, the depth information transmitted by the TOF algorithm module is the first-level depth of field.

[0140] For example, Figure 4C and Figure 4D The two types of depth information provided in the embodiment of the present application are schematic diagrams of scenes corresponding to the first-level depth of field.

[0141] Figure 4C The scene shown includes: a terminal device 401 and an object 1 402. In this scene, the object 1 402 is located within the detection range of the TOF camera of the terminal device 401.

[0142] Figure 4D The illustrated scene includes: a terminal device 401, an object 1 402, and an object 2 403. In this scene, the object 1 402 is outside the detection range of the TOF camera of the terminal device 401; the object 2 403 is within the detection range of the TOF camera of the terminal device 401.

[0143] When there is at least one object within the detection range of the TOF camera of the terminal device, the depth information transmitted by the TOF algorithm module is the secondary depth of field.

[0144] For example, Figure 4E and Figure 4F The two types of depth information provided in the embodiment of the present application are schematic diagrams of scenes corresponding to the first-level depth of field.

[0145] Figure 4E The scene shown includes: a terminal device 401, an object 1 402, and an object 2 403. In this scene, both the object 1 402 and the object 2 403 are located within the detection range of the TOF camera of the terminal device 401.

[0146] Figure 4F The illustrated scene includes: terminal device 401, object 1 402, object 2 403, and object 3 404. In this scene, object 1 402 is outside the detection range of the TOF camera of terminal device 401; object 2 403 and object 3 404 are both within the detection range of the TOF camera of terminal device 401.

[0147] It is understandable that depth information is not limited to three categories. In some implementations, it may include a greater number of categories. The embodiments of the present application do not limit the division method and number of categories of depth information.

[0148] S302 : The TOF algorithm module drives and transmits depth information to the sensor.

[0149] It should be noted that the transmission between various software modules in the embodiments of the present application is achieved through function calls.

[0150] In some embodiments, the TOF algorithm module transmits depth information to the sensor driver via Qualcomm native inter-core communication QMI.

[0151] For example, the format of the depth information transmitted between the TOF algorithm module and the sensor driver is as follows:

[0152] Taking the depth information indicating the zero-order depth of field as an example, the depth information includes: the client processor is a null value (for example, SNS_STD_CLIENT_PROCESSOR_NULL=0).

[0153] For example, the format of the depth information transmitted between the TOF algorithm module and the sensor driver is as follows: enumsns_std_client__tof_processor{

[0154] option(nanopb_enumopt).long_names=false;

[0155] SNS_STD_CLIENT_PROCESSOR_NULL=0; / / no object}

[0156] Taking the depth information indicating the first-level depth of field as an example, the depth information includes: the client processor is 1 (for example, SNS_STD_CLIENT_PROCESSOR_ONE=1).

[0157] For example, the format of the depth information transmitted between the TOF algorithm module and the sensor driver is as follows: enumsns_std_client__tof_processor{

[0158] option(nanopb_enumopt).long_names=false;

[0159] SNS_STD_CLIENT_PROCESSOR_ONE=1; / / There is an object}

[0160] Taking the depth information indicating the secondary depth of field as an example, the depth information includes: the client processor is 2 (eg, SNS_STD_CLIENT_PROCESSOR_TWO=2).

[0161] For example, the format of the depth information transmitted between the TOF algorithm module and the sensor driver is as follows:

[0162] enum sns_std_client__tof_processor{

[0163] option(nanopb_enumopt).long_names=false;

[0164] SNS_STD_CLIENT_PROCESSOR_TWO=2; / / There are two layers of depthinformation}

[0165] S303: The sensor driver transmits depth information to the sensor hardware abstraction.

[0166] In some embodiments, the depth information transmitted between the sensor driver and the sensor hardware abstraction is as follows:

[0167] When the depth of field is zero, the depth information includes: SNS_STD_CLIENT_PROCESSOR_NULL=0;

[0168] When the depth of field is level 1, the depth information includes: SNS_STD_CLIENT_PROCESSOR_ONE=1;

[0169] When the depth of field is secondary, the depth information includes: SNS_STD_CLIENT_PROCESSOR_TWO=2.

[0170] S304: After receiving the depth information, the sensor hardware abstraction transmits the depth information to the camera service process in the camera manager.

[0171] In the embodiment of the present application, the TOF camera can capture images at a preset frame rate. It can also be understood that the terminal device executes the above S301-S304 at the preset frame rate. The preset frame rate can be 1 frame / second or 5 frames / second. The preset frame rate can be set based on actual conditions or experience. The specific value of the preset frame rate is not limited in the embodiment of the present application.

[0172] The following describes how to identify the display status of a terminal device, including the screen off and screen on, and the lock screen status, including unlocked and unlocked.

[0173] Specifically, the screen-off state can be determined by the interactive state of the terminal device. For example, the terminal device can determine the interactive state of the terminal device through the boolean isInteractive() function in the power manager. When the return value of this function is true (true), it indicates that the terminal device is in an interactive state, i.e., the screen is on; when the return value of this function is false (false), it indicates that the terminal device is in a non-interactive state, i.e., the screen is off.

[0174] The lock screen status can be determined by obtaining the keyboard lock status from the keyboard management in the system UI. For example, the terminal device can confirm the lock screen status using the boolean isKeyguardLocked() function. When the function returns true, it indicates that the terminal device is in the locked state; when the function returns false, it indicates that the terminal device is in the unlocked state.

[0175] In an embodiment of the present application, the terminal device determines whether the AO camera is turned on or off through the depth information of the image captured by the TOF camera and the status of the terminal device.

[0176] The following describes the process of turning on and off the AO camera in the terminal device in combination with specific usage scenarios. Figures 5A-9 Corresponding to the opening and closing process of the AO camera under the gesture detection function; Figures 10A-12 This corresponds to the opening and closing process of the AO camera under the "Gaze on Screen" function.

[0177] The following combination Figure 5A and Figure 5B This section explains how to display images in scenes involving gestures.

[0178] For example, Figure 5A A flow chart of a display method provided in an embodiment of the present application is shown as follows: Figure 5A As shown, the display method includes:

[0179] S501: The terminal device displays a first interface. During the display of the first interface, the AO camera of the terminal device is turned off.

[0180] The first interface may be a video playback interface, a picture interface, or other types of interfaces, which is not limited in this embodiment of the present application.

[0181] For example, the terminal device Figure 5B For example, the terminal device 501 in FIG. 1 may be configured as follows: Figure 5B As shown in a in FIG. , the AO camera of the terminal device 501 is turned off.

[0182] S502: At a first time point, the terminal device captures an image containing a hand and a face through a TOF camera.

[0183] At the first time point, the face is facing the display screen (screen) of the terminal device, and the user makes a first gesture between the face and the screen of the terminal device. The distance between the face and the display screen is the first distance.

[0184] The face facing the display screen of the terminal device can be understood from the perspective of orientation as being located on the side of the display screen, with the distance between the center of the face and the center of the display screen being the shortest. From the perspective of terminal device recognition, this can be understood as the features recognized by the terminal device from the captured image satisfying pre-set feature conditions corresponding to the face facing the display screen. The present application does not limit the recognition of the face facing the display screen, the feature conditions, etc.

[0185] The first gesture may be a palm facing up, a back of the hand facing down, a fist, or any other gesture. The embodiment of the present application does not specifically limit the first gesture.

[0186] The first distance is less than the preset distance. It is understandable that the range of the TOF camera to capture images is limited. When the distance between the object and the display screen exceeds the preset distance, the TOF camera cannot recognize the object. The embodiments of the present application do not limit the preset distance.

[0187] For example, the terminal device Figure 5B For example, the terminal device 501 in the figure, the object 1 502 is a hand, and the object 2 503 is a face. Figure 5B As shown in b, object 2 503 faces the display screen, and object 1 502 is located between object 2 503 and the display screen. The TOF camera in the terminal device can capture an image containing object 1 502 and object 2 503. The depth information of the image indicates the secondary depth of field.

[0188] Specifically, the terminal device captures an image containing hands and faces through the TOF camera. The terminal device executes the above S301-S304 to transmit the depth information of the image to the camera service process. The depth information indicates the secondary depth of field, and then turns on the AO camera according to the depth information of the image.

[0189] S503: After the first time point, the first gesture is maintained for a first duration.

[0190] The first duration can be 3 seconds or 5 seconds. The embodiment of the present application does not limit the specific value of the first duration.

[0191] It is understood that the TOF camera captures images at a preset frame rate. When the terminal device captures images containing a hand and a face N times in a row using the TOF camera, the terminal device determines that the first gesture is maintained for a first duration. N is the product of the first duration and the preset frame rate.

[0192] In some embodiments, starting at the fifth time point and within the first duration, if the terminal device captures an image containing a hand and a face via the TOF camera, the terminal device executes steps S301-S304 above and transmits depth information of the image to the camera service process, where the depth information indicates a secondary depth of field. Optionally, the AP side of the terminal device activates the AO camera when the depth information received N times consecutively indicates a secondary depth of field.

[0193] In other embodiments, within the first duration starting from the fifth time point, the terminal device captures an image containing a hand and a face through the TOF camera, and the terminal device executes S301 but does not execute S302-S304. During the first duration starting from the fifth time point, the terminal device captures an image containing a hand and a face through the TOF camera, and the terminal device executes the above S301-S304 to transmit depth information of the image to the camera service process, where the depth information indicates a secondary depth of field. Adaptively, the AP side of the terminal device turns on the AO camera when the received depth information indicates a secondary depth of field.

[0194] S504: Starting from the first time point and after the first time period, the terminal device turns on the AO camera.

[0195] Specifically, after the camera service process receives the depth information of the image, the terminal device executes the following S602-S606 to turn on the AO camera.

[0196] S505: After the terminal device turns on the AO camera, the terminal device displays a second interface corresponding to the first gesture.

[0197] Specifically, after the AO camera is turned on, the terminal device executes S607-S609.

[0198] For example, the terminal device Figure 5B The terminal device 501 in Figure 5B For example, when the terminal device 501 recognizes that the first gesture is one of the preset gestures, the terminal device displays the function corresponding to the preset gesture corresponding to the first gesture. Figure 5B The interface shown in c.

[0199] It is understood that the preset gestures may include palm up, back of hand down, fist, palm to the left, palm to the right, palm close to the display screen, etc., and the functions corresponding to the above preset gestures are: next page, previous page, screenshot, swipe left, swipe right, press, etc. The embodiments of the present application do not limit the preset gestures and the functions corresponding to the preset gestures.

[0200] Taking the first gesture of palm up as an example, the second interface includes the next page of content of the first interface. Taking the first gesture of palm down as an example, the second interface includes the previous page of content of the first interface.

[0201] In this way, when the user makes a gesture, the terminal device turns on the AO camera to detect the gesture and implement the function corresponding to the gesture.

[0202] When a user makes a gesture, the AO camera detects the gesture and displays the gesture in accordance with the above steps S501-S505. The terminal device can also turn off the AO camera after detecting the gesture. Turning off the AO camera is described below in conjunction with S506 and S507.

[0203] S506: At a second time point, the terminal device captures an image containing a face through the TOF camera. The second time point is later than the first time point, and the time length between the second time point and the first time point is greater than the first time length.

[0204] At the second time point, the face is facing the screen of the terminal device, there is no other object between the face and the screen, and the distance between the face and the screen is the first distance.

[0205] Specifically, at the second time point, the terminal device captures an image containing a face through the TOF camera, and the terminal device executes the above S301-S304 to transmit the depth information of the image to the camera service process. The depth information indicates the first-level depth of field, and the terminal device turns off the AO camera.

[0206] S507. After the second time point, the terminal device turns off the AO camera.

[0207] Specifically, after the camera service process receives the depth information of the image, the terminal device executes the following S702-S705 to turn off the AO camera.

[0208] In this way, the terminal device can turn off the AO camera when no gesture is detected, thereby reducing the operating time of the AO camera and reducing power consumption.

[0209] The above S501-S507 illustrate the opening of the AO camera of the terminal device in the scene with gesture action, and the subsequent AO closing process. Figure 5C and Figure 5D This section describes the process of turning the AO camera on and off in scenes without gestures.

[0210] For example, Figure 5C This is a flow chart of a display method provided in an embodiment of the present application. Figure 5C As shown, the display method includes:

[0211] S508. At a third time point, the terminal device captures an image containing the first object and the human face through the TOF camera.

[0212] At the third time point, the face is facing the display screen of the terminal device, there is a first object between the face and the screen of the terminal device, the first object is not a hand, and the first object is opaque, and the distance between the face and the screen is the first distance.

[0213] The first object can be an object such as a book, or a living thing such as a cat. The embodiment of the present application does not limit the first object.

[0214] The face facing the display screen of the terminal device, the first distance, etc. can refer to the above corresponding instructions and will not be repeated here.

[0215] For example, the terminal device Figure 5D For example, the terminal device 501 in the example, the object 2 503 is a face, and the object 3 504 is a book. Figure 5D As shown in b, object 2 503 faces the display screen, and object 3 504 is located between object 2 503 and the display screen. The TOF camera in the terminal device can capture an image containing object 2 503 and object 3 504. The depth information of the image indicates the secondary depth of field.

[0216] Specifically, at the third time point, the terminal device captures an image containing the first object and the face through the TOF camera. The terminal device executes the above S301-S304 to transmit the depth information of the image to the camera service process. The depth information indicates the secondary depth of field, and then turns on the AO camera according to the depth information of the image.

[0217] S509: After the third time point, the first object is maintained for the first duration.

[0218] The first duration can refer to the corresponding instructions above and will not be elaborated here.

[0219] S510: The terminal device turns on the AO camera, and the interface displayed on the terminal device remains unchanged.

[0220] Specifically, after the camera service process receives the depth information of the image, the terminal device executes the following S602-S606 to turn on the AO camera. After the AO camera is turned on, if the terminal device does not recognize the gesture, no processing is performed.

[0221] For example, Figure 5DTaking the scenario shown in b as an example, the interface displayed by the terminal device 501 remains unchanged. Figure 5D The terminal device interface shown in b is the same as Figure 5D The interface displayed by the terminal device shown in c is the same.

[0222] In this way, when the terminal device does not detect a gesture, the displayed interface remains unchanged.

[0223] S511. At a fourth time point, there is no non-transparent object within a preset distance in front of the screen of the terminal device, and the terminal device collects images through the TOF camera.

[0224] Specifically, at the fourth time point, the terminal device captures an image containing a face through the TOF camera, and the terminal device executes the above S301-S304 to transmit the depth information of the image to the camera service process, where the depth information indicates a level 1 depth of field. The terminal device turns off the AO camera.

[0225] S512. After the fourth time point, the terminal device turns off the AO camera.

[0226] Specifically, after the camera service process receives the depth information of the image, the terminal device executes the following S702-S705 to turn off the AO camera.

[0227] In this way, when no object is detected, the AO camera is turned off, which reduces the running time of the AO camera and reduces power consumption.

[0228] In some embodiments, if the gesture is maintained for a short time, the terminal device does not turn on the AO camera. This can reduce misrecognition and shorten the operating time of the AO camera.

[0229] For example, Figure 5E A flow chart of a display method provided in an embodiment of the present application, the display method includes:

[0230] S513. At the fifth time point, the terminal device captures an image containing a hand and a face through a TOF camera.

[0231] At the fifth time point, the face is facing the screen of the terminal device, and the user makes a first gesture between the face and the display screen of the terminal device. The distance between the face and the screen is the first distance.

[0232] The face facing the display screen of the terminal device, the first gesture, etc. can all refer to the above corresponding instructions and will not be repeated here.

[0233] S514: Starting from the fifth time point, after a second duration, the terminal device captures an image containing a face of a person through the TOF camera, and the image does not include a hand. The second duration is shorter than the first duration.

[0234] In some embodiments, starting from the fifth time point, within the second time duration, the terminal device captures an image containing a hand and a face through the TOF camera, and the terminal device executes the above S301-S304 to transmit the depth information of the image to the camera service process, and the depth information indicates the secondary depth of field. After the second time duration starts from the fifth time point, the terminal device captures an image containing a face through the TOF camera, and the terminal device executes the above S301-S304 to transmit the depth information of the image to the camera service process, and the depth information indicates the primary depth of field. Adaptively, starting from the fifth time point, if the AP side of the terminal device does not receive depth information indicating the secondary depth of field N times in a row, the AO camera will not be turned on.

[0235] In other embodiments, starting at the fifth time point and within the second duration, the terminal device captures an image containing a hand and a face through the TOF camera, and the terminal device executes S301 but does not execute S302-S304. Starting at the fifth time point and after the second duration, the terminal device captures an image containing a face through the TOF camera, and the terminal device executes the above S301-S304 to transmit the depth information of the image to the camera service process, where the depth information indicates the primary depth of field. Adaptively, when the AP side of the terminal device does not receive depth information indicating the secondary depth of field, the AO camera is not enabled.

[0236] S514. Starting from the fifth time point, after the first time period, the terminal device does not turn on the AO camera.

[0237] It is understandable that if the terminal device displays the second interface at the fifth time point, the terminal device continues to display the second interface after the first time period starting from the fifth time point.

[0238] This can reduce the misrecognition of gestures, shorten the operating time of the AO camera, and further reduce power consumption.

[0239] The following combination Figure 6 and Figure 7 This section describes the software module interaction process involved in starting and closing the AO camera in gesture detection scenarios.

[0240] For example, Figure 6 This is a schematic diagram of the software module interactions involved in the process of turning on the AO camera in the gesture detection scenario provided by the embodiment of the present application. Figure 6 As shown, the process includes:

[0241] S601: The camera service process receives depth information of a first type of image from a TOF algorithm module.

[0242] In the embodiment of the present application, the terminal device can obtain depth information through the above S301-S304 and transmit it to the camera service process.

[0243] For example, Figure 4E and Figure 4F Taking the scene shown as an example, the depth information received by the camera service process is the secondary depth of field.

[0244] S602: After receiving the depth information, the camera service process obtains the status information of the terminal device.

[0245] In an embodiment of the present application, the camera service process can obtain the display screen status from the power manager; the camera service process can obtain the lock screen status from the keyboard manager (keyguard manager) in the system UI (system UI).

[0246] The camera service process first obtains the display status from the power manager, and then obtains the lock screen status from the keyboard management. The camera service process can also obtain the display status from the power manager and the lock screen status from the keyboard management at the same time. The camera service process can also first obtain the lock screen status from the keyboard management, and then obtain the display status from the power manager. This embodiment of the application does not limit the order in which the various status information is obtained.

[0247] In some embodiments, when the camera service process obtains the screen status from the power manager as being on, the camera service process obtains the screen lock status from the keyboard manager. In this way, when the screen is off, the screen lock status may not be obtained, which can reduce some signaling consumption.

[0248] In some embodiments, the camera service process stores the status information of the terminal device. When the status information of the terminal device changes, a message for updating the status information is sent to the camera management process. For example, when the terminal device switches from the screen-off state to the screen-on state, the power manager transmits a message to the camera management process to indicate the display screen status switch; or transmits a message to indicate the screen-on state. When the terminal device switches from the lock screen state to the unlocked state, the keyboard management transmits a message to the camera management process to indicate the lock screen state switch; or transmits a message to indicate the unlocked state. The embodiments of the present application do not limit the acquisition of status information.

[0249] S603: When the depth information is a secondary depth of field and the state information indicates a screen-on state and an unlocked state, the camera service process sends a first message via the AO camera management module. The first message is used to instruct to turn on the AO camera.

[0250] In some embodiments, the terminal device turns on the AO camera when it detects the secondary depth of field multiple times. In this way, the TOF camera detects multiple objects N times in a row, and the status of multiple objects is maintained for a certain period of time, which can reduce misidentification.

[0251] In the first possible implementation method, after receiving the depth information, the camera service process obtains the display screen status of the terminal device; when the camera service process receives the secondary depth of field N times in a row, and the display screen status of the terminal device obtained by the camera service process N times in a row is the screen-on state and the unlocked state, the camera service process sends a first message to the AO camera management module.

[0252] In a second possible implementation, the camera service process obtains the display screen status of the terminal device when receiving the secondary depth of field N times in a row. When the status information indicates the screen-on state and the unlocked state, the camera service process sends a first message to the AO camera management module.

[0253] For example, taking the preset frame rate of the TOF camera as 1 frame and hovering for 3 seconds to trigger the AO camera to turn on as an example, the camera service process receives the secondary depth of field 3 times in a row and obtains the display status of the terminal device.

[0254] In this way, the signaling consumption of the terminal equipment can be reduced and the energy consumption can be reduced.

[0255] In possible implementation method three, when the depth information obtained by the TOF algorithm module is the secondary depth of field for N consecutive times, the depth information is sent to the camera service process. Adaptively, after the depth information obtained by the TOF algorithm module is the secondary depth of field for N consecutive times, the camera service process receives the depth information indicating the secondary depth of field from the TOF algorithm module.

[0256] In this way, the signaling consumption of the terminal equipment can be further reduced, and the energy consumption can be reduced.

[0257] It is understood that if the depth information is level 1 depth of field or level 0 depth of field, the camera service process does not process it. If the depth information is level 2 depth of field and the status information is used to indicate the screen off state or the screen locked state, the camera service process does not process it.

[0258] S604. The AO camera management module transmits a first message to the AO camera hardware abstraction.

[0259] Exemplarily, the AO camera management module transmits the first message to the AO camera hardware abstraction through an opening function (eg, open AOcamrea function).

[0260] S605: The AO camera hardware abstraction transmits a first message to the AO camera driver.

[0261] S606 : After receiving the first message, the AO camera driver turns on the AO camera.

[0262] Specifically, after receiving the first message, the AO camera driver controls the AO camera to power on.

[0263] Adaptively, after the AO camera is turned on, images are collected to obtain the second type of images.

[0264] S607 , the AO camera driver abstractly transmits the second type of image to the AO camera hardware.

[0265] S608 , the AO camera hardware abstraction recognizes portraits and gestures based on the second type of image.

[0266] When the terminal device recognizes that the second type of image includes: a portrait and a gesture action corresponding to a preset gesture, the terminal device executes S609. When the terminal device recognizes that the second type of image does not include a portrait, or does not include a gesture action corresponding to a preset gesture, the terminal device does not execute S609 and does not perform subsequent processing.

[0267] For example, Figure 5D Taking the illustrated scenario as an example, the terminal device recognizes that the second type of image includes: object 2 503 and object 3 504. Since the terminal device does not recognize the gesture action corresponding to the preset gesture, the AO camera hardware abstraction does not perform subsequent processing.

[0268] For example, Figure 5B Taking the scenario shown in b in FIG. 5 as an example, the terminal device recognizes that the second type of image includes: object 1 502 and object 2 503. Since the terminal device recognizes a human portrait and a gesture corresponding to a preset gesture, the terminal device processes the palm-up gesture, thereby changing the content displayed on the display screen of terminal device 501.

[0269] S609: When the terminal device recognizes that the second type of image includes a human portrait and a gesture corresponding to the preset gesture, the AO camera hardware abstraction sends a third message to the foreground application via the window manager based on the gesture, where the third message corresponds to the gesture in the processing result. The foreground application is an application running and displayed in the foreground of the terminal device.

[0270] Exemplarily, when the terminal device recognizes that the second type of image includes a gesture for indicating that the user's palm is facing downward, the third message is a message for indicating turning the page; when the terminal device recognizes that the second type of image includes a gesture for indicating that the user's palm is facing upward, the third message is a message for indicating returning to the previous page; when the terminal device recognizes that the second type of image includes a gesture for indicating that the user is making a fist, the third message is a message for indicating taking a screenshot.

[0271] In summary, when the TOF camera detects multiple objects in front of the display, it triggers the AO camera to turn on and perform gesture detection. This enables gesture detection. Furthermore, when there are no objects or only one object, the terminal device does not turn on the AO camera, preventing it from running continuously. This reduces the operating time of the AO camera and reduces the terminal device's power consumption.

[0272] On the basis of the above embodiment, the terminal device can also control the AO camera to turn off when the depth information is the zero-level depth of field or the first-level depth of field. Figure 7 This is a schematic diagram of the software module interactions involved in the process of shutting down the AO camera in the gesture detection scenario provided by the embodiment of the present application. Figure 7 As shown in the figure, the shutdown process of the AO camera after it is turned on includes:

[0273] S701: The camera service process receives depth information from the TOF algorithm module.

[0274] In the embodiment of the present application, the terminal device can obtain depth information through the above S301-S304 and transmit it to the camera service process.

[0275] For example, Figure 4A and Figure 4B Taking the scene shown as an example, the depth information received by the camera service process is the zero-level depth of field. Figure 4C and Figure 4D Taking the scene shown as an example, the depth information received by the camera service process is the first-level depth of field.

[0276] S702: When the depth information received by the camera service process is a zero-order depth of field or a first-order depth of field, the camera service process sends a second message to the AO camera management module. The second message is used to instruct the AO camera to be turned off.

[0277] In some embodiments, if the depth information received by the camera service process is a first-level depth of field and the camera service process is in a called period, the camera service process does not send a second message to the AO camera management module, and the AO camera remains turned on.

[0278] It is understandable that the terminal device may call the camera service process to detect the user status after entering the screen off timer. If the depth information received by the camera service process during the terminal device calling the camera service process is the first level depth of field, the camera service process will not send the second message to the AO camera management module, and the AO camera will remain on.

[0279] In a possible implementation, the camera service process can determine whether the terminal device is invoking the camera service process by the presence or absence of the first identifier. The first identifier is used to instruct the terminal device to invoke the camera service process.

[0280] When the depth information received by the camera service process is level 1 depth of field and the camera service process is set with the first flag, the camera service process does not send the second message to the AO camera management module, and the AO camera remains on. When the depth information received by the camera service process is level 1 depth of field and the camera service process is not set with the first flag, the camera service process sends the second message to the AO camera management module, and the terminal device executes S703-S705, and the AO camera is turned off.

[0281] The camera service process can also confirm whether the terminal device is invoking the camera service process in any other manner, which is not specifically limited in this embodiment of the present application. S703: The AO camera management module sends a second message to the AO camera hardware abstraction.

[0282] S704: The AO camera hardware abstraction transmits a second message to the AO camera driver.

[0283] S705 : After receiving the second message, the AO camera driver turns off the AO camera.

[0284] Specifically, after receiving the second message, the AO camera driver controls the AO camera to power off.

[0285] In this way, when the TOF camera detects that there are 0 or 1 objects in front of the display, it triggers the AO camera to shut down and stop gesture detection. This can reduce the operating time of the AO camera and reduce the power consumption of the terminal device.

[0286] Based on the above embodiment, the terminal device can also automatically light up the screen based on the depth information transmitted by the TOF camera. Specifically, when the depth information transmitted by the TOF camera is the first or second depth of field and the terminal device is in the screen-off state, the terminal device can control the display screen to power on and display the lock screen interface to automatically light up the screen.

[0287] The following combination Figure 8A and Figure 8B The display method of the screen off scene is explained.

[0288] For example, Figure 8A This is a flow chart of the display method provided in the embodiment of the present application. Figure 8A As shown, the method includes:

[0289] S801: The terminal device is in the screen-off state.

[0290] For example, Figure 8B As shown in a, the terminal device 801 is in a screen-off state. The TOF camera in the terminal device 801 collects images.

[0291] S802: At a sixth time point, the terminal device captures an image containing a second object through a TOF camera.

[0292] At the sixth time point, the face of the person is facing the display screen of the terminal device, and the distance between the face of the person and the display screen is a second distance, which is less than the preset distance.

[0293] For example, the terminal device Figure 8B In the terminal device 801, the object 802 is a face, for example, Figure 8B As shown in b, object 1 802 is facing the display screen. The terminal device can capture an image containing object 1 802 with a TOF camera. The depth information of the image indicates a primary depth of field.

[0294] Specifically, the terminal device captures an image containing hands and faces through the TOF camera. The terminal device executes the above S301-S304 to transmit the depth information of the image to the camera service process. The depth information indicates the first-level depth of field, and then the screen is turned on according to the depth information of the image.

[0295] S802. After the sixth time point, the terminal device displays a lock screen interface.

[0296] In some embodiments, the lock screen interface includes an input box. The input box is used to enter the lock screen password. The input box can be used to enter a fingerprint, a number, or a graphic, without limitation.

[0297] In some embodiments, in response to the user entering the lock screen password in the input box, the terminal device displays the unlocked interface. The unlocked interface can be the main interface or the interface displayed before the lock screen, etc. This is not limited here.

[0298] Specifically, after the camera service process receives the depth information of the image, the terminal device executes the following S902-S908 to display the lock screen interface on the bright screen.

[0299] Exemplarily, after the sixth time point, the terminal device 801 controls the display screen to power on, and the terminal device displays the lock screen interface shown in Figure 8 (c). The lock screen interface includes: time 803 and unlock input box 804. Unlock input box 804 is used to enter the lock screen password. The lock screen password can be the user's fingerprint, a pre-set numeric password, a pre-set graphic password, etc.

[0300] It is understandable that when the terminal device receives a user unlocking operation within the first preset time period after displaying the lock screen interface, the terminal device unlocks and displays the main interface or the interface of the application running before the lock screen.

[0301] When the terminal device does not receive an unlock operation within the first preset time after displaying the lock screen interface, and the camera service process receives a zero-level depth of field, the terminal device turns off the screen.

[0302] When the terminal device does not receive a user unlocking operation within a first preset time period after displaying the lock screen interface, and the camera service process receives a first-level depth of field or a second-level depth of field, the terminal device continues to display the lock screen interface.

[0303] When the terminal device receives a user unlocking operation while displaying the lock screen interface, the terminal device displays the main interface or the interface of the application running before the lock screen.

[0304] It is understandable that the first preset time length can be 10 seconds or 15 seconds, and the embodiment of the present application does not limit the specific value of the first preset time length.

[0305] It should be noted that if at the sixth time point, the image captured by the terminal device through the TOF camera does not include the object, after the sixth time point, the terminal device does not display the lock screen interface and continues to be in the off-screen state.

[0306] For example, Figure 9 This is an interactive flow chart of the software modules corresponding to the automatic screen lighting provided in the embodiment of the present application. Figure 9 As shown, the process includes:

[0307] S901: The camera service process receives depth information from the TOF algorithm module.

[0308] In the embodiment of the present application, the terminal device can obtain depth information through the above S301-S304 and transmit it to the camera service process.

[0309] S902: After receiving the depth information, the camera service process obtains the display screen status of the terminal device.

[0310] For details, please refer to the description of S602 above, which will not be repeated here.

[0311] S903: When the terminal device is in the screen-off state and the depth information is the first or second depth of field, the camera service process sends a screen-on event to the display manager. The screen-on event is used to instruct the terminal device to turn on the screen display.

[0312] S904: After receiving the screen-on event, the display manager sends the screen-on event to the power manager.

[0313] Adaptively, the power manager controls the display power-up.

[0314] S905: After receiving the screen-lighting event, the display manager sends the screen-lighting event to the window manager.

[0315] S906: The window manager calls the process for drawing and rendering in the system UI to draw and render the lock screen interface (keyguard), and performs synthesis processing on the drawn and rendered lock screen interface.

[0316] S907: The window manager sends the synthesized lock screen interface to the display driver.

[0317] In some embodiments, the terminal device further includes a display hardware abstraction, and the window manager sends the synthesized lock screen interface to the display driver via the display hardware abstraction.

[0318] S908. The display driver drives the display screen to display the synthesized lock screen interface.

[0319] It is understandable that when the terminal device receives a user unlocking operation within the first preset time period after displaying the lock screen interface, the terminal device unlocks and displays the main interface or the interface of the application running before the lock screen.

[0320] When the terminal device does not receive an unlock operation within the first preset time after displaying the lock screen interface, and the camera service process receives a zero-level depth of field, the terminal device turns off the screen.

[0321] When the terminal device does not receive a user unlocking operation within a first preset time period after displaying the lock screen interface, and the camera service process receives a first-level depth of field or a second-level depth of field, the terminal device continues to display the lock screen interface.

[0322] When the terminal device receives a user unlocking operation while displaying the lock screen interface, the terminal device displays the main interface or the interface of the application running before the lock screen.

[0323] In this way, when the terminal device detects an object, the screen will light up, realizing the function of lighting up the screen when a person comes, improving the user experience.

[0324] above Figures 5A-9 Corresponding to the opening and closing process of the AO camera under the gesture detection function; the following is combined Figures 10A-12 This corresponds to the opening and closing process of the AO camera under the "Gaze on Screen" function.

[0325] It is understandable that when the terminal device does not receive any user operation for a long time, the display screen can be turned off to reduce the power consumption of the terminal device.

[0326] Specifically, when the terminal device receives a user operation and there is no process holding the lock, the terminal device starts the screen off timer. When the timer ends, the terminal device turns off the screen. When the terminal device receives a user operation and there is a process holding the lock, the terminal device does not start the screen off timer. For example, when the terminal device displays the main interface after being unlocked, the terminal device starts the screen off timer. If the terminal device does not receive a user operation for a long time, the terminal device turns off the screen. When the terminal device is playing a video, the terminal device does not start the screen off timer. If no user operation is received for a long time, the terminal device does not turn off the screen. The embodiments of the present application do not specifically limit the application of locking.

[0327] The following combination Figure 10A and Figure 10B This section describes the display method for scenarios where the screen remains on while being watched.

[0328] For example, Figure 10A This is a flow chart of a display method provided in an embodiment of this application. Figure 10A As shown, the method includes:

[0329] S1001. At the seventh time point, the terminal device displays the third interface.

[0330] The third interface can be a video playback interface, a picture interface, or other types of interfaces, which is not limited in this embodiment of the present application.

[0331] For example, the terminal device Figure 10B For example, the terminal device 1001 in Figure 10B As shown in a in FIG, the third interface may be the interface shown in the terminal device 1001. The AO camera of the terminal device 1001 is turned off.

[0332] S1002. Starting from the seventh time point, within the second preset time period, the terminal device does not receive any user operation, and the terminal device does not turn on the AO camera.

[0333] It is understandable that starting from the seventh time point, within the second preset time period, the terminal device enters the screen off timing, but the camera service process is not called.

[0334] In the embodiment of the present application, the user operation can be an operation of the user clicking the screen, an operation of the user inputting voice, or a user gesture operation. The embodiment of the present application does not limit the type, specific form, etc. of the user operation.

[0335] The second preset time length can be 10 seconds or 20 seconds. The embodiment of the present application does not limit the specific value of the second preset time length.

[0336] In some embodiments, starting at the seventh time point and within a second preset duration, the display screen is at a first brightness; starting at the seventh time point and after the second preset duration, the display screen is at a second brightness. The second brightness is lower than the first brightness. This can reduce power consumption of the terminal device.

[0337] S1003: At an eighth time point, the terminal device captures an image containing a human face via a TOF camera. The eighth time point is later than the seventh time point, and the duration between the eighth time point and the seventh time point is greater than a second preset duration. At the eighth time point, the human face is facing the display screen of the terminal device, and the distance between the human face and the screen is the first distance.

[0338] For example, the terminal device Figure 10B In the terminal device 1001, the object 1002 is a face, for example, Figure 10B As shown in a in FIG, object 1002 is facing the display screen. The terminal device can capture an image containing object 1002 with a TOF camera. The depth information of the image indicates a primary depth of field.

[0339] Specifically, the terminal device captures an image containing a human face via a TOF camera. The terminal device executes steps S301-S304 above to transmit depth information of the image to the camera service process, where the depth information indicates a first-level depth of field. After a second preset duration starting at the seventh time point, the terminal device invokes the camera service process. During the invocation, the camera service process activates the AO camera based on the image depth information.

[0340] S1004. After the eighth time point, the terminal device turns on the AO camera.

[0341] Specifically, after the camera service process receives the depth information of the image, the terminal device executes the following S1102-S1105 to turn on the AO camera.

[0342] S1005: At a ninth time point, the terminal device captures an image containing a person looking at the display screen through the AO camera, and the terminal device displays a third interface.

[0343] In the embodiments of the present application, a portrait can be understood as a portrait of a person, which can be a portrait corresponding to the face of the person, or a portrait corresponding to other parts of the body, without limitation here. A portrait of a person looking at a display screen can also be referred to as a face looking at a display screen or a person's eyes looking at a display screen. A portrait looking at a display screen can be understood as the person's eye's gaze point being located at the display screen of the terminal device. The embodiments of the present application do not limit the specific algorithm analysis of the image. The embodiments of the present application do not repeat the same or similar content.

[0344] For example, Figure 10B As an example of the scenario shown in a, at the ninth time point, the terminal device detects that a person is looking at the display screen, and the terminal device 1001 continues to display the third interface (such as Figure 10B (as shown in b in the figure).

[0345] Specifically, after the AO camera is turned on, the terminal device executes S1106-S1108.

[0346] In some embodiments, after the ninth time point, the display screen of the terminal device is at the first brightness, thereby increasing the brightness and making it easier for the user to browse the content displayed on the display screen.

[0347] S1006: After the ninth time point, the terminal device turns off the AO camera and displays the third interface. The ninth time point is later than the eighth time point, and the duration between the ninth time point and the seventh time point is less than the third preset duration.

[0348] Specifically, when the terminal device detects that the user is looking at the display screen, it stops calling the camera service process.

[0349] In some embodiments, if the terminal device receives a user operation after the eighth time point, the terminal device stops calling the camera service process, and the terminal device controls the AO camera to turn off.

[0350] In this way, when the terminal device detects an object, it turns on the AO camera to achieve the function of watching without turning off the screen. In addition, when the call ends, the AO camera is turned off to reduce the operating time of the AO camera and reduce power consumption.

[0351] S1007: Starting from the seventh time point, within the third preset time period, the terminal device does not receive any user operation, and the terminal device displays the third interface.

[0352] The third preset time length is greater than the second preset time length. The third preset time length can be 15 seconds or 30 seconds. The embodiment of the present application does not limit the specific value of the third preset time length.

[0353] It is understandable that since it is detected that the user is looking at the display screen, the terminal device updates the screen off timer, starting at the seventh time point, and the screen does not turn off within the third preset time period.

[0354] The following combination Figure 10C-10E This section describes the display method for scenarios where the screen is not being looked at.

[0355] S1008. At the tenth time point, the terminal device displays the third interface.

[0356] For example, the terminal device Figure 10D For example, the terminal device 1001 in Figure 10B As shown in a in FIG, the third interface may be the interface shown in the terminal device 1001. The AO camera of the terminal device 1001 is turned off.

[0357] S1009. Starting from the tenth time point, within the second preset time period, the terminal device does not receive any user operation, and the terminal device does not turn on the AO camera.

[0358] S1009 is similar to S1002. For details, please refer to the corresponding instructions of S1002 above, which will not be repeated here.

[0359] Starting from the tenth time point, after the second preset time period, the terminal device calls the camera service process to detect whether the user is looking at the display screen.

[0360] S1010: At an eleventh time point, the terminal device captures an image containing a third object through a TOF camera. The eleventh time point is later than the tenth time point, and the duration between the eleventh time point and the tenth time point is greater than a second preset duration.

[0361] For example, the terminal device Figure 10D In the terminal device 1001, the object 2 1003 is a face, for example, Figure 10D As shown in a in FIG, object 2 1003 faces the display screen sideways. The terminal device can use a TOF camera to capture an image containing object 2 1003. The depth information of the image indicates a primary depth of field.

[0362] For example, the terminal device Figure 10E In the terminal device 1001, the object 3 1004 is a book, for example, Figure 10E As shown in a in FIG, object 2 1004 faces the display screen sideways. The terminal device can use a TOF camera to capture an image containing object 3 1004. The depth information of the image indicates a primary depth of field.

[0363] Specifically, the terminal device captures an image containing a first object via a TOF camera. The terminal device executes steps S301-S304 above to transmit depth information of the image to the camera service process, where the depth information indicates a first-level depth of field. After a second preset duration starting at the eleventh time point, the terminal device invokes the camera service process. During the invocation, the camera service process activates the AO camera based on the depth information of the image.

[0364] S1011. After the eleventh time point, the terminal device turns on the AO camera.

[0365] Specifically, after the camera service process receives the depth information of the image, the terminal device executes the following S1102-S1105 to turn on the AO camera.

[0366] S1012. Starting from the tenth time point, within the third preset time period, the terminal device does not capture an image containing a person looking at the display screen through the AO camera, and the terminal device turns off the screen.

[0367] For example, Figure 10D As an example, from the time when the AO camera is turned on to the third preset time starting at the tenth time point, the terminal device does not detect that a person is looking at the display screen, and the terminal device 1001 turns off the screen (e.g. Figure 10D (as shown in b in the figure).

[0368] For example, Figure 10E As an example, from the time when the AO camera is turned on to the third preset time starting at the tenth time point, the terminal device does not detect that a person is looking at the display screen, and the terminal device 1001 turns off the screen (e.g. Figure 10E(as shown in b in the figure).

[0369] Specifically, after the AO camera is turned on, the terminal device executes S1106-S1107 and S1109.

[0370] In this way, the terminal device turns off the screen when no person is detected looking at the display screen.

[0371] The following combination Figure 10F This section describes the display method for scenarios where the screen is not being looked at.

[0372] S1013. At the twelfth time point, the terminal device displays the third interface.

[0373] S1014. Starting from the twelfth time point, within the second preset time period, the terminal device does not receive any user operation, and the terminal device does not turn on the AO camera.

[0374] S1014 is similar to S1002. For details, please refer to the corresponding instructions of S1002 above, which will not be repeated here.

[0375] S1015: Starting from the twelfth time point, within a third preset time period, the terminal device does not capture an image containing the object through the TOF camera, and the AO camera is not turned on.

[0376] Specifically, the terminal device captures an image containing the first object through the TOF camera, and the terminal device executes the above S301-S304 to transmit the depth information of the image to the camera service process, where the depth information indicates the zero-order depth of field. During the call, the camera service process does not enable the AO camera based on the depth information of the image.

[0377] S1016. Starting from the twelfth time point, after the third preset time period, the terminal device turns off the screen.

[0378] It is understandable that if the terminal device does not detect an object, the terminal device turns off the screen. In this way, when there is no object, the AO camera is not turned on, which reduces the operating time of the AO camera and reduces power consumption.

[0379] The following combination Figure 11 This section describes how to turn on and off the AO camera in the screen-off timing scenario.

[0380] For example, Figure 11 This is a software interaction diagram for the AO camera startup process when the user is looking at the screen without turning it off, provided in an embodiment of the present application. The process includes:

[0381] S1101: After the terminal device enters the screen-off timer and reaches a second preset time, the display manager calls the camera service process to obtain depth information in front of the display screen.

[0382] In the embodiment of the present application, the terminal device can obtain depth information through the above S301-S304 and transmit it to the camera service process.

[0383] In a possible implementation, when the display manager calls the camera service process, the terminal device sets a first identifier in the camera service process. The first identifier is used to indicate that the terminal device has called the camera service process. When the display manager completes calling the camera service process, the terminal device deletes the first identifier set in the camera service process.

[0384] In this way, it is convenient for the camera service process to confirm whether it has been called, and then confirm whether to control the AO camera to be turned on or off.

[0385] The camera service process can also confirm whether it is in the period when the terminal device calls the camera service process through any other method, and this embodiment of the present application does not specifically limit this.

[0386] S1102: The camera service process receives depth information from the TOF algorithm module.

[0387] S1103: When the depth information from the TOF camera is a primary depth of field or a secondary depth of field, the camera service process sends a first message to the AO camera hardware abstraction. The first message is used to instruct to start the AO camera.

[0388] It is understood that when the depth information from the TOF camera reaches zero depth of field, the camera service process sends a message to the display manager indicating that no one is present. If the display manager does not receive a message instructing it to restart the timer at the end of the timer, the display manager sends a message instructing the power manager to turn off the screen, and the power manager controls the display to power off.

[0389] In this way, when there is no object, the terminal device does not turn on the AO camera, thereby reducing the operating time of the AO camera and reducing the power consumption of the terminal device.

[0390] S1104. The AO camera hardware abstraction transmits a first message to the AO camera driver.

[0391] S1105 . After receiving the first message, the AO camera driver turns on the AO camera.

[0392] Specifically, after receiving the first message, the AO camera driver controls the AO camera to power on.

[0393] S1106: The AO camera driver abstractly transmits the second type of image data to the AO camera hardware. The second type of image data is image data collected by the AO camera.

[0394] S1107. The AO camera hardware abstraction recognizes the person looking at the display screen based on the second type of image.

[0395] When it is recognized that the second type of image includes a human portrait and the human portrait is looking at the display screen, the terminal device executes S1108. When it is recognized that the second type of image includes a human portrait but the human portrait is not looking at the display screen, the terminal device executes S1109 and S1110. When it is recognized that the second type of image does not include a human portrait, the terminal device executes S1109 and S1110.

[0396] S1108: When it is recognized that the person in the second type of image is looking at the display screen, the AO camera hardware abstraction sends a message to the timing process in the display manager to instruct the user to restart the timing. Adaptively, the timing process in the display manager restarts the screen-off timing.

[0397] For example, if the screen off timer is 15s, the second preset duration is 12s, and the screen off timer starts at 8:00:00, at 8:00:12, the terminal device screen off timer reaches the second preset duration, and the terminal device executes the above S1101-S1107. When the processing result is used to indicate that the user is looking at the display screen, the terminal device restarts the screen off timer, that is, restarts the screen off timer at 8:00:12. When the terminal device screen off timer reaches the second preset duration again at 8:00:24, the terminal device executes the above S1101-S1107.

[0398] It is understandable that if the terminal device restarts the screen-off timing, the terminal device turns off the AO camera.

[0399] In a possible implementation method 1, the timing process in the display manager stops calling the camera service process after receiving the message for instructing to restart the timing. When the camera service process stops calling, it controls the AO camera to shut down.

[0400] In a second possible implementation, when the processing result indicates that the person is looking at the display screen, the AO camera hardware abstraction transmits a second message to the AO camera driver. After receiving the second message, the AO camera driver drives the AO camera to shut down.

[0401] The terminal device can also control the AO camera to turn off by resetting the screen off timing in any other manner. This embodiment of the present application does not specifically limit this.

[0402] S1109: When it is recognized that the second type of images are all portraits not looking at the display screen, or do not include a portrait, the AO camera hardware abstraction sends a message to the display manager indicating that the portrait is not looking at the display screen.

[0403] In some embodiments, when it is recognized that the person in the second type of image is not looking at the display screen or does not include a person, the AO camera hardware abstraction does not perform any processing.

[0404] For example, if the screen off timer is 15s, the second preset timer is 12s, and the screen off timer starts at 8:00:00, at 8:00:12, the terminal device screen off timer reaches the second preset timer, and the terminal device executes the above S1101-S1107. When the second type of images are all images that the user is not looking at the display screen, the terminal device turns off the screen after the timer ends. Figure 10D As an example, in the scenario shown in a, the terminal device recognizes that the second type of images are all objects 1003 not looking at the display screen, and the terminal device turns off the screen after the timing ends (e.g. Figure 10D (as shown in b in the figure).

[0405] by Figure 10E Take the scenario shown in a as an example, in which the terminal device recognizes that the second type of image does not include a portrait, and the terminal device turns off the screen after the timer ends (such as Figure 10E (as shown in b in the figure).

[0406] S1110: When the timing ends, the display manager does not receive a message for instructing to restart the timing. The display manager sends a message for instructing to turn off the screen to the power manager, so as to control the display screen to turn off.

[0407] Adaptively, the power manager controls the display to power off.

[0408] During the screen-off timer, if the TOF camera detects one or two objects in front of the display, the AO camera is triggered to turn on and perform continuous gaze detection. When the subject is looking at the display, the screen remains on. When the subject is not looking at the display, the screen turns off to reduce power consumption.

[0409] It is understandable that when the screen off timer ends, the display manager stops calling the camera service process. The camera service process controls the AO camera to turn off after stopping the call.

[0410] Specifically, the camera service process sends a second message to the AO camera management module after stopping the call. The terminal device executes S1303-S1305.

[0411] In this way, by controlling the AO camera to shut down at the end of the timing, the operating time of the AO camera can be reduced, and the power consumption of the terminal device can be reduced.

[0412] Based on the above embodiment, the terminal device can also control the AO camera to turn off when the depth information is zero-level depth of field. Figure 12 This is a schematic diagram of the software module interactions involved in the process of shutting down the AO camera in the scenario of watching the screen without turning it off provided by the embodiment of the present application. Figure 12 As shown, the following describes the closing process of the AO camera.

[0413] S1201: The camera service process receives depth information from the TOF algorithm module.

[0414] In the embodiment of the present application, the terminal device can obtain depth information through the above S301-S304 and transmit it to the camera service process.

[0415] S1202: When the depth information received by the camera service process is a zero-order depth of field, the camera service process sends a second message to the AO camera management module. The second message is used to instruct the AO camera to be turned off.

[0416] S1203. The AO camera management module sends a second message to the AO camera hardware abstraction.

[0417] S1204. The AO camera hardware abstraction transmits a second message to the AO camera driver.

[0418] S1205 . After receiving the second message, the AO camera driver turns off the AO camera.

[0419] Specifically, after receiving the second message, the AO camera driver controls the AO camera to power off.

[0420] During the screen-off timer, if the TOF camera detects that there are no objects in front of the display, the AO camera is triggered to shut down. This can reduce the operating time of the AO camera and reduce the power consumption of the terminal device.

[0421] The method of the embodiment of the present application has been described above. The device for performing the above method provided by the embodiment of the present application is described below. Those skilled in the art will understand that the method and device can be combined and referenced with each other, and the relevant device provided by the embodiment of the present application can perform the steps in the above method.

[0422] The display method provided in the embodiment of the present application can be applied to electronic devices with display functions. The electronic devices include terminal devices. The specific device form of the terminal device can refer to the above related descriptions and will not be repeated here.

[0423] An embodiment of the present application provides a terminal device, which includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the above method.

[0424] For example, Figure 13 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. Figure 13As shown, the terminal device includes: the terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0425] It should be understood that the structures illustrated in the embodiments of the present invention do not constitute specific limitations on the terminal device. In other embodiments of the present application, the terminal device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0426] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0427] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0428] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0429] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0430] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the terminal device. In other embodiments of the present application, the terminal device may also adopt a different interface connection method from the above embodiment, or a combination of multiple interface connection methods.

[0431] The terminal device implements display functionality through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0432] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, the terminal device may include one or N display screens 194, where N is a positive integer greater than 1.

[0433] The terminal device can realize the shooting function through the ISP, camera 193, video codec, GPU, display 194 and application processor.

[0434] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0435] Camera 193 is used to capture still images or videos. In some embodiments, the terminal device may include one or N cameras 193, where N is a positive integer greater than 1. The types of the multiple cameras 193 can vary. For example, camera 193 may include a camera for capturing color images or a time-of-flight camera. In the embodiments of the present application, a camera can be understood as a camera. Cameras include: the AO camera, time-of-flight camera, etc. in the embodiments of the present application.

[0436] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0437] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 can include a program storage area and a data storage area.

[0438] The terminal device can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0439] Keys 190 include a power button, a volume button, and other buttons. Keys 190 can be mechanical or touch-sensitive. The terminal device can receive key inputs and generate key signal inputs related to user settings and function control of the terminal device. Indicator 192 can be an indicator light that can be used to indicate charging status, battery level changes, messages, missed calls, notifications, and the like.

[0440] In addition, the device also runs an operating system on top of the above components, such as the iOS operating system, the Android operating system, or the Windows operating system. Applications can be installed and run on the operating system.

[0441] The software system of the terminal device can adopt a layered architecture, event-driven architecture, micro-kernel architecture, microservice architecture, or cloud architecture, etc., which will not be described here.

[0442] An embodiment of the present application provides a terminal device, which includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the method of the first aspect.

[0443] The present embodiment provides a chip. The chip includes a processor configured to invoke a computer program stored in a memory to execute the technical solution of the above embodiment. The implementation principles and technical effects are similar to those of the above-mentioned related embodiments and will not be further described here.

[0444] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the above-mentioned method is implemented. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0445] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage or other magnetic storage devices, or any other medium intended to carry or store the desired program code in the form of instructions or data structures and accessible by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disk and optical disc as used herein include optical disc, laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks generally reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0446] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above method.

[0447] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to produce a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0448] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A display method, characterized in that: Applied to a terminal device, the terminal device includes a time-of-flight (TOF) camera and a normally-powered AO camera, and the method includes: The terminal device displays the first interface; At a first time point, the terminal device captures a first image through the TOF camera, where the first image includes a face and a first gesture; After acquiring the first image, the terminal device turns on the AO camera; The terminal device captures a second image through the AO camera, where the second image includes the human face and the first gesture; The terminal device displays a second interface according to the first gesture, where the content displayed on the second interface is different from the content displayed on the first interface; At a second time point, the terminal device captures a third image through the TOF camera, where the third image includes the human face but does not include any object other than the human face; After the second time point, the terminal device turns off the AO camera.

2. The method according to claim 1, characterized in that The method further comprises: At a third time point, the terminal device captures a fourth image through the TOF camera, where the fourth image includes the human face and the first object; wherein the first object is not a hand; After acquiring the fourth image, the terminal device turns on the AO camera; The terminal device collects a fifth image through the AO camera, where the fifth image includes the human face and the first object; The terminal device continues to display the second interface.

3. The method according to claim 2, characterized in that The method further comprises: At a fourth time point, the terminal device captures a sixth image through the TOF camera, where the sixth image does not include the object; After the fourth time point, the terminal device turns off the AO camera.

4. The method according to any one of claims 1 to 3, characterized in that After acquiring the first image, the terminal device turns on the AO camera, including: Starting from the first time point, within a first time period, all images captured by the terminal device through the TOF camera are the first images; Starting from a first time point, after a first period of time, the terminal device turns on the AO camera.

5. The method according to claim 4, characterized in that The method further comprises: At a fifth time point, the terminal device captures a seventh image through the TOF camera, where the seventh image includes the human face and the first gesture; Starting from the fifth time point, after a second duration, the terminal device captures an eighth image through the TOF camera, where the eighth image includes the human face and does not include objects other than the human face; the second duration is less than the first duration; Starting from the fifth time point, after the second time period, the terminal device does not turn on the AO camera; The terminal device displays the second interface.

6. The method according to any one of claims 1 to 5, characterized in that The terminal device displays a second interface according to the first gesture, including: The first gesture corresponds to a page turning function, and the second interface includes the next page of content of the first interface, or includes the previous page of content of the first interface; Alternatively, the first gesture corresponds to a screenshot function, and the second interface includes a screenshot of the first interface.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: The terminal device is in a screen-off state; At a sixth time point, the terminal device captures a ninth image through the TOF camera, where the ninth image includes the human face; After the sixth time point, the terminal device displays a lock screen interface, where the lock screen interface includes an input box; In response to the user entering the lock screen password in the input box, the terminal device displays the unlocked interface.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: At the seventh time point, the terminal device displays the third interface; From the seventh time point to the eighth time point, the terminal device does not receive any user operation; the eighth time point is later than the seventh time point, the duration between the eighth time point and the seventh time point is greater than the fourth duration, and the fourth duration is less than the third duration; At the eighth time point, the terminal device captures a tenth image through the TOF camera, where the tenth image includes the human face; After the eighth time point, the terminal device turns on the AO camera; At a ninth time point, when the human face is looking at the display screen, the terminal device captures an eleventh image through the AO camera; Starting from the seventh time point, within the third time period, the terminal device displays the third interface; After the ninth time point, the terminal device turns off the AO camera.

9. The method according to any one of claims 1 to 7, characterized in that The method further comprises: At the tenth time point, the terminal device displays the third interface; At an eleventh time point, the terminal device captures a twelfth image through the TOF camera, where the twelfth image includes the third object; After the eleventh time point, the terminal device turns on the AO camera; Starting from the tenth time point, within a third time period, the terminal device captures a thirteenth image through the AO camera, where the thirteenth image includes the third object but does not include a human face looking at the display screen; Starting from the tenth time point, after the third period of time, the terminal device turns off the screen.

10. The method according to any one of claims 1 to 7, characterized in that The method further comprises: At the twelfth time point, the terminal device displays the third interface; From the twelfth time point, within the third time period, the terminal device captures a fourteenth image through the TOF camera, the fourteenth image does not include the object, and the terminal device does not turn on the AO camera; From the twelfth time point, after the third period of time, the screen of the terminal device is turned off.

11. The method according to any one of claims 1 to 10, characterized in that After acquiring the first image, the terminal device turns on the AO camera, including: The terminal device obtains first depth information based on the first image, where the first depth information includes: information of a secondary depth of field, where the information of the secondary depth of field indicates at least two objects; Obtaining status information of the terminal device, where the status information indicates whether the terminal device is in a screen-on state and whether it is unlocked; According to the first depth information and the status information, it is determined that the terminal device has a screen on and is unlocked, and the first image has a secondary depth of field, and the AO camera is turned on.

12. The method according to any one of claims 1 to 11, characterized in that After the second time point, the terminal device turns off the AO camera, including: The terminal device obtains second depth information based on the third image, where the second depth information includes: information of a first-level depth of field, where the information of the first-level depth of field indicates an object; The terminal device turns off the AO camera based on the second depth information and the third image having a first-level depth of field.

13. The method according to any one of claims 1 to 12, characterized in that Starting from a first time point and after a first time period, the terminal device turns on the AO camera, including: The terminal device collects N first images using the TOF camera within a first time period starting from the first time point; The terminal device obtains N first depth information based on the N first images; The terminal device obtains status information of the terminal device starting from a first time point and after a first duration, where the status information is used to indicate whether the terminal device is in a screen-on state and whether it is unlocked; The terminal device determines, based on the first depth information and the status information, that the terminal device has its screen on and unlocked, and that the first image has a secondary depth of field, and turns on the AO camera.

14. A terminal device, characterized in that: include: processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the terminal device performs the method according to any one of claims 1 to 13.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

16. A computer program product, characterized in that The method comprises a computer program which, when being executed, causes a computer to execute the method according to any one of claims 1 to 13.

Images