Display method, electronic device, and readable medium

Abstract

The present application provides a display method, an electronic device, and a readable medium. The electronic device includes a display screen and a first camera. The first camera collects data by emitting and receiving light, indicating that the data collected by the first camera is not affected by the light in the environment. The electronic device is in a first environment at a first moment. The ambient light brightness of the first environment is less than a first threshold value to indicate that the brightness of the first environment is low, indicating that the electronic device is in a dark environment at the first moment. The data collected by the first camera at the first moment can also accurately obtain the direction of a face. The display screen of the electronic device displays a first interface that is the same as the direction of the face, thereby realizing a display interface with a more accurate recognition result of the face direction.

Description

Technical Field

[0001] The present application relates to the field of face recognition technology, and in particular to a display method, electronic device, computer program product, and computer-readable storage medium. Background Art

[0002] When a user uses an electronic device, the electronic device usually uses a low-power camera (always on camera) to capture images and detect the user's face direction based on the image to achieve automatic screen rotation.

[0003] However, low-power cameras have a particularly low recognition rate for facial direction in dark environments, resulting in inaccurate or ineffective screen rotation, affecting the user experience. Summary of the Invention

[0004] The present application provides a display method, electronic device, computer program product and computer-readable storage medium, the purpose of which is to realize a display interface with more accurate recognition results of face direction.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] In a first aspect, the present application provides a display method applied to an electronic device, which includes a display screen and a first camera, and the first camera collects data by emitting and receiving light. The method includes: the electronic device is in a first environment at a first moment, and the ambient light brightness of the first environment is less than a first threshold to indicate that the brightness of the first environment is low; while the electronic device is in the first environment, the display screen displays a first interface, and the direction of the first interface is the same as the direction of a first face, and the direction of the first face is indicated by data collected by the first camera.

[0007] As can be seen from the above, the first camera collects data by emitting and receiving light, indicating that the data collected by the first camera is not affected by ambient light, and the first camera can be a TOF camera. The electronic device is in a first environment at a first moment, and the ambient light brightness of the first environment is less than a first threshold, indicating that the brightness of the first environment is low, indicating that the electronic device is in a dark environment at the first moment. The data collected by the first camera at the first moment can also accurately determine the direction of the face. The display screen of the electronic device displays a first interface that is identical to the direction of the face, thereby realizing a more accurate face direction recognition result display interface.

[0008] In one possible embodiment, the electronic device also includes a second camera, and the resolution of the image captured by the second camera is less than a second threshold. The method also includes: the electronic device is in a second environment at a second moment, and the ambient light brightness of the second environment is greater than the first threshold; while the electronic device is in the second environment, the display screen displays a second interface, and the direction of the second interface is the same as the direction of the second face, and the direction of the second face is indicated by the image captured by the second camera.

[0009] In the above possible implementation, the resolution of the image captured by the second camera is less than the second threshold, indicating that the second camera is a low-power camera. The ambient light brightness of the electronic device at the second moment is greater than the first threshold, indicating that the electronic device is in a non-dark environment. The image captured by the second camera can accurately determine the direction of the face. The electronic device displays the second interface in the direction of the face indicated by the image captured by the second camera, achieving low-power completion interface display.

[0010] In one possible implementation, while the second camera is capturing images to obtain the second face direction, the first camera is not running. This can prevent the first camera from continuously running and increasing power consumption.

[0011] In one possible embodiment, the display screen displays the first interface, including: when the electronic device is in a first usage posture, the display screen displays the first interface, and the method also includes: while the electronic device is in the first environment, receiving an operation of the user controlling the electronic device to adjust from the first usage posture to the second usage posture, the second usage posture being different from the first usage posture; in response to the operation of the user controlling the electronic device to adjust from the first usage posture to the second usage posture, the display screen switches from displaying the first interface to displaying the third interface, and the direction of the first interface is different from the direction of the third interface.

[0012] In the above possible implementation, the first usage posture and the second usage posture can be one of a horizontal screen usage posture and a vertical screen usage posture. When the electronic device is in a dark environment, after the electronic device switches the usage posture, the interface displayed on the display screen follows the adjustment direction, thereby realizing automatic rotation of the screen.

[0013] In one possible embodiment, the display screen displays the first interface, including: when the electronic device is in a first usage posture, the display screen displays a camera preview interface, and the method also includes: while the electronic device is in the first environment, receiving an operation of the user controlling the electronic device to adjust from the first usage posture to the second usage posture, and controlling the electronic device to display a first image, the first image being obtained by the user inputting a shooting operation in the camera preview interface; the second usage posture is different from the first usage posture; in response to the user controlling the electronic device to adjust from the first usage posture to the second usage posture, and controlling the electronic device to display the first image, the display screen switches from displaying the camera preview interface to displaying a fourth interface, the direction of the camera preview interface is the same as that of the fourth interface, and the fourth interface is a browsing interface for the first image.

[0014] In the above possible implementations, the first usage posture and the second usage posture can be one of a landscape usage posture and a portrait usage posture. The user inputs a capture operation on the camera preview interface, and the electronic device captures a first image. When the electronic device adjusts the usage posture and displays the first image, the first image and the camera preview interface are in the same orientation, thereby implementing an auxiliary photography function.

[0015] In one possible embodiment, when the electronic device is in a first environment, the display screen displays a first interface, the direction of the first interface is the same as the direction of the first face, and the direction of the first face is indicated by data collected by the first camera, including: when the electronic device is in the first environment, controlling the first camera to collect data; processing the data collected by the first camera to obtain the direction of the first face; and controlling the display screen to display the first interface based on the first face direction, the direction of the first interface is the same as the direction of the first face.

[0016] In one possible implementation, after processing the data collected by the first camera to obtain the first face direction, the method further includes: controlling the first camera to stop collecting data, thereby preventing the first camera from continuously running and increasing power consumption.

[0017] In one possible embodiment, when the electronic device is in the first environment, after controlling the first camera to collect data, it also includes: when processing the data collected by the first camera and the first face direction is not obtained, determining the timing duration; when the timing duration does not reach the third threshold, obtaining the data collected by the first camera again; processing the data collected by the first camera again to obtain the third face direction; and controlling the display screen to display the first interface based on the third face direction, the direction of the first interface is the same as the third face direction.

[0018] In a possible implementation, the method further includes: controlling the first camera to stop collecting data when the timing duration reaches a third threshold.

[0019] In one possible implementation, before controlling the first camera to collect data, the method further includes: determining whether the gravity direction of the electronic device has changed, thereby avoiding the situation in which the electronic device remains stationary while the first camera continues to operate in a dark environment at night, thereby increasing power consumption.

[0020] In a possible implementation, the method further includes: while the electronic device is in the second environment, the display screen displays a fifth interface, and the direction of the fifth interface is the same as the direction of gravity of the electronic device.

[0021] In a second aspect, the present application provides an electronic device comprising: one or more processors, a memory, a first camera, and a display screen; the first camera collects data by emitting and receiving light; the memory, the first camera, and the display screen are coupled to one or more processors, the memory is used to store a computer program, the computer program includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the display method provided in the first aspect and any one of the possible implementation methods.

[0022] In a third aspect, the present application provides a computer-readable storage medium for storing a computer program. When the computer program is executed, it is specifically used to implement the display method provided in the first aspect and any one of the possible implementation methods.

[0023] In a fourth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the display method provided in the first aspect and any one of the possible implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a diagram showing the application scenario of the automatic rotation function of the screen;

[0025] Figure 2 This is a diagram showing the application scenarios of the auxiliary photo function;

[0026] Figure 3 A software structure diagram of an electronic device provided in an embodiment of the present application;

[0027] Figure 4 A flowchart of a method for recognizing the direction of a face provided in an embodiment of the present application;

[0028] Figure 5 This is a hardware structure diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and claims of this application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless the context clearly indicates otherwise.

[0030] References to "some embodiments" and the like in this specification mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in some embodiments" and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0031] The "multiple" involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first" and "second" are only used for the purpose of distinguishing the description and cannot be understood as indicating or implying relative importance or order.

[0032] To make the description of the following embodiments clear and concise, a brief introduction to the relevant concepts or technologies is first given:

[0033] A Rich Execution Environment (REE), also known as a general or untrusted execution environment, is the system runtime environment for electronic devices, which can run operating systems such as Android, iOS, and Linux. REEs are open and scalable, but lack high security.

[0034] A Trusted Execution Environment (TEE), also known as a secure side or secure zone, is an area requiring authorization for access. The TEE coexists with the REE in an electronic device. Through hardware-based isolation from the REE, the TEE provides security and resistance to software attacks that are common to the REE. The TEE has its own operating space and defines strict protections, offering a higher level of security than the REE. It protects TEE assets, such as data and software, from software attacks and certain types of security threats.

[0035] TA, or trusted application, is an application running in TEE that can provide security services to CA running outside TEE, such as password input, transaction signature generation, face recognition, etc.

[0036] CA, or client application, is typically an application running in the REE. The CA can call the TA through the client application programming interface (API) and instruct the TA to perform corresponding security operations.

[0037] A TOF camera (TOF camera module, TOF sensor) can include a transmitter (TX) and a receiver (RX). The TX is used to transmit infrared light or laser pulses, and the RX is used to receive reflected light and form an image. Because the TX can autonomously emit light signals for imaging, TOF data is not affected by most light in the environment.

[0038] The automatic screen rotation function means that the electronic device automatically adjusts the screen's display orientation based on the user's orientation. For example, if the user holds the device vertically, the screen will display the interface in portrait orientation; if the user holds the device horizontally, the screen will display the interface in landscape orientation.

[0039] In some application scenarios, the screen orientation of electronic devices is inconsistent with the orientation of the user's face. For example, if a user lies on his side and uses the phone vertically, the phone will mistakenly think that the user is using the phone horizontally and will Figure 1 The vertical display interface shown in (a) is adjusted to Figure 1 The horizontal display interface shown in (b) above brings significant changes to the user. Therefore, when a user uses an electronic device, the electronic device typically uses a low-power camera (always-on camera) to capture images and detect the user's face orientation based on the images to facilitate automatic screen rotation. The low-power camera can be configured as the front-end camera of the electronic device, which is turned on by default and periodically captures images.

[0040] The assisted photo function means that when the electronic device is in the screen-on state, if the electronic device detects that the electronic device's camera function is turned on and the electronic device detects the user's face orientation through the low-power camera, the electronic device will automatically rotate the screen according to the user's face orientation, so that the content displayed on the screen always remains in the right direction relative to the user's face. In addition, when the electronic device displays a taken photo, the electronic device automatically rotates the screen according to the user's face orientation, so that the photo displayed on the screen always remains in the right direction relative to the user's face.

[0041] For example, Figure 2As shown, after the user takes a picture with the mobile phone in landscape mode, the user can view the picture. When the auxiliary photo function is not activated, the mobile phone displays the picture interface as follows Figure 2 As shown in (a), it can be seen that the display direction of the image is different from the composition direction when the user takes the image, and is also different from the face direction when the user views the image, which is not conducive to the user viewing the image. After the auxiliary function is activated, the interface of the mobile phone displaying the image is as follows Figure 2 As shown in (b), the display direction of the image is the same as the composition direction when the user takes the image, and is the same as the direction of the user's face.

[0042] However, in the automatic rotation of the screen and assisted photo-taking scenarios, the low-power camera has a particularly low recognition rate for the direction of the face in a dark environment, resulting in inaccurate or ineffective screen rotation, affecting the user experience.

[0043] To address this issue, the inventors discovered a feasible solution: using a Time of Flight (TOF) camera to complement a low-power camera. Compared to low-power cameras, TOF cameras offer higher accuracy and robustness, effectively detecting the direction of a face in low-light conditions. However, TOF cameras consume significantly more power than low-power cameras. Prolonged use can accelerate battery drain and reduce the device's battery life.

[0044] The face direction recognition solution provided in the embodiment of the present application can be applied to the data display link of electronic devices, and can balance the performance and power consumption of low-power cameras and TOF cameras.

[0045] To introduce the face direction recognition solution provided by the embodiment of the present application in detail, the following is an explanation in conjunction with the functional modules in the software structure of the electronic device.

[0046] First, let's discuss the software structure of electronic devices. This can be thought of as the layered architecture of their operating systems. These operating systems run on their hardware components and can include iOS, the open-source Android operating system, and Windows.

[0047] The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device.

[0048] Figure 2 It is a software structure block diagram of the electronic device according to an embodiment of the present application.

[0049] A layered architecture divides software into several layers, each communicating through software interfaces. In some embodiments, the Android system is divided into four layers: application layer, application framework layer, hardware abstraction layer, and kernel layer, from top to bottom. The four-layer architecture also includes a smart sensor hub and a hardware layer.

[0050] The hardware layer includes the hardware of the electronic device, such as Figure 2 The image sensor, accelerometer, ambient light sensor, and time-of-flight sensor are shown. The image sensor can be a low-power camera, and the images it captures have low resolution, typically below a threshold. The functions of the image sensor, accelerometer, ambient light sensor, and time-of-flight sensor are described below.

[0051] Sensorhunb provides a hardware and software solution based on a low-power MCU and a lightweight RTOS operating system. Its main function is to connect and process data from various sensor devices.

[0052] In some embodiments, Sensorhunb may include a first face recognition algorithm. Sensorhunb may process images captured by the image sensor based on the first face recognition algorithm. When the images captured by the image sensor include faces, Sensorhunb may identify the face direction (or face orientation) of the images captured by the image sensor based on the first face recognition algorithm.

[0053] In other embodiments, Sensorhunb may include an accelerometer driver and a gravity rotation algorithm. The accelerometer driver controls the operation of the accelerometer. Detection data from the accelerometer during operation may be reported to Sensorhunb via the accelerometer. Sensorhunb processes the detection data based on the gravity rotation algorithm to determine the direction of gravity.

[0054] In some other embodiments, Sensorhunb may include an ambient light sensor driver for controlling the operation of the ambient light sensor. The ambient light brightness obtained by the operation of the ambient light sensor may be reported to Sensorhunb via the ambient light sensor driver.

[0055] The application layer may include multiple applications. For example, Figure 2 Three applications are displayed: camera, settings, and third-party applications.

[0056] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. For example, Figure 3 The application framework layer displayed may include: a screen automatic rotation (DisplayRotation) module of a window management service (WindowManagerService, WMS), and an intelligent rotation direction service.

[0057] The Window Manager Service (WMS)'s automatic screen rotation (DisplayRotation) module, abbreviated as WMSDisplayRotation, provides applications with an interface and programming framework for automatic screen rotation. WMSDisplayRotation can be used to adjust the window's orientation to achieve automatic screen rotation.

[0058] The intelligent rotation direction service is used to connect WMSDisplayRotation and the lower-level module to achieve communication between WMSDisplayRotation and the lower-level module. The lower-level module can refer to the intelligent fusion module of the hardware abstraction layer.

[0059] The hardware abstraction layer provides a virtual hardware platform for the operating system. Exemplarily, the hardware abstraction layer may include an intelligent fusion module. In some embodiments, the intelligent fusion module may include a face direction acquisition module, a gravity direction acquisition module, an ambient light brightness reading module, an intelligent direction fusion module, a TOF data acquisition module, and a TOF switch decision module. The intelligent fusion module also includes a variety of algorithms, such as Figure 2 The second face recognition algorithm presented is used to identify the face orientation in TOF data.

[0060] In some embodiments, the face direction acquisition module is used to obtain the face direction obtained by Sensorhunb, the gravity direction acquisition module is used to obtain the gravity direction obtained by Sensorhunb, and the ambient light brightness reading module is used to obtain the ambient light brightness obtained by Sensorhunb.

[0061] In some embodiments, the TOF switch decision module can decide whether to turn on the TOF sensor based on the direction of gravity and the brightness of the ambient light. When the TOF sensor is turned on, the TOF data (also known as depth data) obtained by it involves user privacy. Under normal circumstances, the TOF data will be stored in the secure buffer (Secure Buffer) in the TEE. The TOF data transmission module in the face TA can transmit the TOF data stored in the Secure Buffer to the face CA. The face CA is used to transmit the TOF data to the TOF data acquisition module in the intelligent fusion module. The intelligent fusion module can process the TOF data based on the second face recognition algorithm to obtain the face direction.

[0062] It should be noted that Figure 2 This section shows some modules of the operating system that support the operation of TOF sensors and perform subsequent operations based on the TOF data detected by the TOF sensor, but this does not constitute a limitation on the operation of TOF sensors and the use of TOF data to perform subsequent operations. For example, the kernel layer of the operating system may also include a TOF sensor driver, a TOF switch decision, and Figure 2 The TOF sensor is driven to operate under the control of other modules not shown.

[0063] In some embodiments, the intelligent fusion module can determine whether to report data to the intelligent rotation direction service and the content of the reported data based on the face direction obtained by the face direction acquisition module, the gravity direction obtained by the gravity direction acquisition module, and the face direction identified by the second face recognition algorithm. The specific working process of the intelligent fusion module can be found in the following corresponding Figure 3 The content of the embodiment.

[0064] The following describes the process of the face direction recognition method provided in the embodiment of the present application.

[0065] like Figure 3 As shown, the face direction recognition method provided in the embodiment of the present application is applied to an electronic device, and the method includes:

[0066] S301 , detecting the direction of gravity through an acceleration sensor.

[0067] In some embodiments, the acceleration sensor can be set to a normally open state. The normally open state can be understood as the electronic device is turned on and the acceleration sensor starts running to periodically detect the direction of gravity and obtain detection data.

[0068] In other embodiments, the electronic device is equipped with a switch for an automatic screen rotation function. When the user turns on the switch to activate the automatic screen rotation function, the acceleration sensor may be controlled to operate. When the user turns off the switch to activate the automatic screen rotation function, the acceleration sensor may be controlled to stop operating.

[0069] It is understood that the detection data obtained by the acceleration sensor can indicate the direction of gravity of the electronic device. Exemplary gravity directions include: upward, left, right, downward, and unstable directions. Unstable directions indicate that the electronic device is currently in a state of shaking or trembling, and the gravity direction is unstable. Upward, left, right, and downward directions are all stable directions.

[0070] Furthermore, the direction of gravity of an electronic device can indicate the orientation of the electronic device. For example, if the direction of gravity of the electronic device is upward, it indicates that the electronic device is being used in portrait mode with the screen facing upward. Of course, the orientation of the screen is usually defined by the orientation of the camera end of the screen. "Screen-up" means that the camera end of the screen faces upward.

[0071] S302: Determine whether the direction of gravity has changed.

[0072] The electronic device can detect the detection data of the acceleration sensor through the gravity rotation algorithm to obtain the direction of gravity.

[0073] The accelerometer periodically generates multiple detection data points. Based on this data, the electronic device can determine multiple directions of gravity. These directions of gravity can be stored with timestamps. The electronic device can use the directions of gravity at two different moments to determine whether the direction of gravity has changed, that is, whether the electronic device has rotated. The electronic device can also determine the rotation direction of the electronic device, such as when the device has changed from a leftward horizontal orientation to an upward vertical orientation.

[0074] In some embodiments, when the electronic device determines whether the direction of gravity has rotated, if the gravity direction at the previous moment was an unsteady direction, the electronic device may discard this data and use the stable gravity direction at the previous moment to determine whether the gravity direction has rotated. Of course, the electronic device must also confirm that the gravity direction obtained this time is a stable direction. In other words, the electronic device determines whether the gravity direction has changed based on the stable direction and the previous stable direction.

[0075] Combine Figure 2 As shown, the detection data from the acceleration sensor is reported to Sensorhunb. Sensorhunb detects the detection data based on a gravity rotation algorithm to obtain the gravity direction of the electronic device. The gravity direction acquisition module obtains the gravity direction and provides it to the intelligent direction fusion module and the time-of-flight switch decision module. The time-of-flight switch decision module determines whether the gravity direction has changed based on the gravity directions at two previous and subsequent moments. In some embodiments, the time-of-flight switch decision module can also determine whether the gravity direction has changed based on the gravity directions at multiple moments.

[0076] S303: Obtain the ambient light brightness through the ambient light sensor.

[0077] In some embodiments, the ambient light sensor can also be set to a normally-on state. The normally-on state can be understood as the electronic device is turned on and the ambient light sensor starts running to periodically detect the ambient light brightness.

[0078] In other embodiments, the electronic device is equipped with a switch for an automatic screen rotation function. When the user turns on the switch for the automatic screen rotation function, the ambient light sensor can be controlled to operate. When the user turns off the switch for the automatic screen rotation function, the ambient light sensor can be controlled to stop operating.

[0079] Step S301 and step S303 can be understood as being executed in parallel, without any restriction on the execution order.

[0080] S304: Determine whether the ambient light brightness is less than a threshold.

[0081] Combine Figure 2 As shown, the ambient light brightness around the electronic device detected by the ambient light sensor can be reported to the intelligent fusion module through the ambient light sensor driver. The ambient light brightness reading module in the intelligent fusion module obtains the ambient light brightness and sends it to the TOF switch decision module. The TOF switch decision module determines whether the ambient light brightness is less than a threshold value. The threshold value can be an empirical value used to indicate that the light around the electronic device is poor and the electronic device is in a dark light environment. Exemplarily, the threshold value can be 15lux. A dark light environment generally refers to an environment where the ambient light brightness is lower than 15lux. Of course, there may be a certain error in the 15lux. For example, if it fluctuates by a few lux around 15lux, it can also be considered as dark light.

[0082] If the TOF switch decision module determines that the direction of gravity has changed and the ambient light brightness is less than the threshold, steps S309 to S314 may be executed. If the TOF switch decision module determines that the ambient light brightness is greater than or equal to the threshold, steps S305 to S308 may be executed. If the TOF switch decision module determines that the direction of gravity has not changed, the process may be exited.

[0083] S305: Determine whether first valid face direction information is obtained.

[0084] If the ambient light brightness is greater than or equal to the threshold, it indicates that the electronic device is not in a dark environment and that the ambient light around the electronic device is bright enough for the image sensor to capture a relatively clear image. The electronic device can determine the screen rotation direction based on the image captured by the image sensor.

[0085] In some embodiments, an image sensor, such as an accelerometer and an ambient light sensor, can be set to a constantly on state or to be synchronously activated after the screen's auto-rotation function is enabled. Low-power images captured by the image sensor can be processed by Sensorhunb based on a first face direction recognition algorithm to determine face direction.

[0086] Sensorhunb processes low-power images based on the first face direction recognition algorithm and obtains three types of face direction results:

[0087] 1. The low-power image captured by the image sensor contains a face, and Sensorhunb correctly identifies the direction of the face;

[0088] 2. The low-power image captured by the image sensor contains a face, but the sensor does not recognize the direction of the face. This usually happens when the face image in the low-power image is not clear enough.

[0089] 3. The low-power image captured by the image sensor does not contain a face, and Sensorhunb does not recognize the direction of the face.

[0090] The first valid face direction information refers to the first result mentioned above. Valid face direction information is valid information about the face direction. To distinguish different valid face direction information, the valid face direction information here is referred to as the first valid face direction information.

[0091] It is understood that the first effective face direction information obtained by the low-power image captured by the image sensor may include: upward direction, left direction, right direction, and downward direction. In addition, the first effective face direction information may indicate the usage direction of the electronic device.

[0092] If the electronic device determines that the first valid face direction information is obtained, step S306 is executed; otherwise, steps S307 and S308 are executed.

[0093] S306: Control screen rotation using the first valid face direction information.

[0094] When the electronic device obtains the first valid face direction information, it can control the screen rotation based on the first valid face direction information. Of course, controlling the screen rotation here means that the electronic device determines that the face direction and the device direction are different based on the first valid face direction information, and controls the screen rotation; when the face direction and the device direction are the same, the electronic device maintains the display direction of the screen.

[0095] In some embodiments, the electronic device may control screen rotation based on the first valid face orientation information to ensure that the interface displayed on the rotated screen maintains the same orientation as the face orientation indicated by the first valid face orientation information. Alternatively, the electronic device may adjust the screen orientation to the face orientation indicated by the first valid face orientation information.

[0096] For example, when the electronic device uses a landscape orientation to display the interface, the electronic device determines based on the first valid face direction information that the face direction is from the right direction, then the control screen is adjusted from the landscape orientation to the upward portrait orientation; if the face direction is determined to be the left direction, then the control screen is adjusted from the landscape orientation to the downward portrait orientation; similarly, when the electronic device uses a portrait orientation to display the interface, the electronic device determines based on the first valid face direction information that the face direction is from the left direction, then the control screen is adjusted from the portrait orientation to the left landscape orientation; if the face direction is determined to be from the right direction, then the control screen is adjusted from the portrait orientation to the right landscape orientation.

[0097] Combine Figure 2 As shown, Sensorhunb's recognition result of the low-power image based on the first face recognition algorithm, that is, the first valid face direction information, can be sent to the intelligent direction fusion module through the face direction acquisition module. The intelligent direction fusion module determines that the face direction and the device direction are different based on the first valid face direction information, and then reports a message to the intelligent rotation direction service. The message indicates that the screen direction needs to be adjusted and the direction is adjusted. The intelligent rotation direction service receives the message and sends it to WMSDisplayRotation. WMSDisplayRotation controls the screen rotation according to the face direction.

[0098] In this way, in scenarios where the screen automatically rotates and assisted photo taking is used, the interface displayed on the screen of the electronic device remains in the correct direction with the user's face.

[0099] S307: Determine whether valid gravity direction information is obtained.

[0100] If the electronic device does not obtain the first valid face orientation information, it means that the electronic device cannot determine the usage orientation of the electronic device from the low-power image captured by the image sensor, and thus cannot control the screen rotation. In this case, the electronic device can control the screen rotation based on the direction of gravity.

[0101] As described in step S301 above, the gravity direction obtained by the electronic device based on the detection data of the accelerometer can include four stable directions: upward, left, right, and downward, as well as an unstable direction. Valid gravity direction information refers to these four stable directions. If the gravity direction obtained by the electronic device based on the detection data of the accelerometer is any of the stable directions, the electronic device has obtained valid gravity direction information.

[0102] If the electronic device determines that valid gravity direction information is obtained, step S308 is executed; otherwise, the process is exited.

[0103] S308: Control screen rotation using effective gravity direction information.

[0104] If the electronic device obtains effective gravity direction information, it can control the screen rotation based on the effective gravity direction information. Of course, controlling the screen rotation here means that the electronic device determines that the electronic device has rotated based on the effective gravity direction information, and then controls the screen rotation; if it is determined that the electronic device has not rotated, the electronic device maintains the display direction of the screen.

[0105] In some embodiments, the electronic device can determine whether the electronic device has rotated based on the effective gravity direction information and the effective gravity direction information at the previous moment. If the effective gravity direction information is the same as the effective gravity direction information at the previous moment, the electronic device has not rotated; otherwise, it has rotated. Furthermore, the electronic device can determine the rotation direction of the electronic device based on the effective gravity direction information and the effective gravity direction information at the previous moment. The electronic device can control the rotation of the screen according to the rotation direction of the electronic device, so that the screen rotates in accordance with the rotation direction of the electronic device.

[0106] Combine Figure 2 As shown, the detection result obtained by Sensorhunb based on the gravity rotation algorithm, that is, the gravity direction, can be sent to the intelligent direction fusion module through the gravity direction acquisition module. The intelligent direction fusion module determines that the electronic device has rotated based on the two valid gravity direction information before and after, and then reports a message to the intelligent rotation direction service. The message indicates that the electronic device has rotated and the rotation direction. The intelligent rotation direction service receives the message and sends it to WMSDisplayRotation. WMSDisplayRotation controls the screen rotation according to the rotation direction of the electronic device.

[0107] S309: Turn on the TOF sensor.

[0108] Based on steps S302 and S304, the electronic device determines that the electronic device is in a dark environment and the direction of gravity has changed, which means that the screen needs to be adjusted for rotation, and the low-power image captured by the image sensor may not be clear and the direction of the face cannot be determined. The electronic device then turns on the TOF sensor.

[0109] Combine Figure 2 As shown, the TOF switch decision module in the intelligent fusion module can receive the ambient light brightness and gravity direction, and start the TOF sensor when it determines that the ambient light brightness is lower than the threshold and the gravity direction of the device changes.

[0110] In an embodiment of the present application, the electronic device uses a TOF sensor to replace a low-power camera to detect the direction of a face in a dark environment, thereby improving the recognition rate of the face direction, achieving accurate rotation of the screen, and improving the user experience.

[0111] It should be noted that the TOF sensor includes an emitter, which emits infrared light or laser pulses during operation. If a user uses another mobile phone to film an electronic device with an active TOF sensor, the video or image captured by the mobile phone will display a red dot.

[0112] S310 , obtaining TOF data through a TOF sensor.

[0113] The TOF sensor starts running and can collect TOF data.

[0114] S311 . Process the TOF data based on a second face recognition algorithm to obtain a face direction.

[0115] The electronic device can process the TOF data based on the second face recognition algorithm to obtain face direction information. The second face recognition algorithm can be understood as a 3D face recognition algorithm.

[0116] Combine Figure 2 As shown, the TOF data collected by the TOF sensor is stored in the secure buffer of the TEE. The TOF data transmission module in the face TA transmits the TOF data stored in the secure buffer to the face CA. The face CA transmits the TOF data to the TOF data acquisition module in the intelligent fusion module. The intelligent fusion module processes the TOF data based on the second face recognition algorithm to obtain the face direction. Exemplarily, the face direction may include upward, left, right, and downward.

[0117] The electronic device processes the TOF data based on the second face recognition algorithm to obtain the face direction, which can also be divided into three results:

[0118] 1. The TOF data contains a face, and the intelligent fusion module correctly identifies the direction of the face;

[0119] 2. The TOF data contains a face, but the intelligent fusion module does not recognize the direction of the face;

[0120] 3. The TOF data does not contain a face, and the intelligent fusion module does not recognize the direction of the face.

[0121] S312: Determine whether second valid face direction information is obtained.

[0122] Based on step S311, the electronic device processes the TOF data based on the second facial recognition algorithm to obtain a facial direction, which can be divided into three results. The second valid facial direction information obtained by the electronic device refers to the first result. Therefore, in combination with steps S311 and S312, it can be seen that the electronic device processes the TOF data based on the second facial recognition algorithm to obtain a processing result. The electronic device further determines whether the processing result is a facial direction. If the electronic device determines that the processing result is a face reversal, it determines that the second valid facial direction information has been obtained. The second valid facial direction information is valid information about the facial direction.

[0123] If the electronic device determines that the second valid face direction information is obtained, step S313 and step S314 are executed; otherwise, step S315 is executed.

[0124] S313: Turn off the TOF sensor.

[0125] In order to avoid the TOF sensor from continuously running and increasing power consumption, the electronic device turns off the TOF sensor after obtaining the second valid face direction information based on the TOF data obtained by the TOF sensor.

[0126] Combining step 309 and step S313, it can be seen that the electronic device uses the TOF sensor in a limited manner according to changes in the ambient light brightness and the gravity direction of the device, thereby avoiding long-term operation of the TOF sensor, controlling the power consumption of the TOF, and extending the battery life of the device.

[0127] S314: Control screen rotation using the second valid face direction information.

[0128] The electronic device controls screen rotation based on the second valid face orientation information. Controlling screen rotation means that the electronic device determines, based on the second valid face orientation information, that the face orientation and the device orientation are different, and controls screen rotation. If the face orientation and the device orientation are the same, the electronic device maintains the screen display orientation.

[0129] In some embodiments, the electronic device may adjust the screen orientation based on the second valid face orientation information to ensure that the interface displayed on the rotated screen remains in the same orientation as the face orientation indicated by the first valid face orientation information. Alternatively, the electronic device may adjust the screen orientation to the face orientation indicated by the first valid face orientation information.

[0130] Combine Figure 2As shown, after the intelligent fusion module obtains the second valid face direction information, the intelligent direction fusion module determines that the face direction and the device direction are different based on the second valid face direction information, and then reports a message to the intelligent rotation direction service, which indicates that the screen direction needs to be adjusted and the direction needs to be adjusted. The intelligent rotation direction service receives the message and sends it to WMSDisplayRotation, and WMSDisplayRotation controls the screen rotation according to the face direction.

[0131] In this way, in scenarios where the screen automatically rotates and assisted photo taking is used, the interface displayed on the screen of the electronic device remains in the correct direction with the user's face.

[0132] S315: Determine whether the timing reaches a threshold.

[0133] The electronic device starts processing the TOF data based on the second face recognition algorithm and synchronously counts the time. If the electronic device determines that no second valid face direction information is obtained based on step S312, it determines whether the counted time reaches a threshold in step S315.

[0134] If the electronic device determines that the timing duration has not reached the threshold, it returns to step S310 and controls the TOF sensor to continue to collect TOF data. The electronic device executes steps S311 and S312 based on the TOF data collected by the TOF sensor for the second time. If the electronic device recognizes valid face direction information based on the TOF data collected by the TOF sensor for the second time, it turns off the TOF sensor through step S313 and controls the screen rotation through step S315. Otherwise, it executes step S315 and returns to S310 based on the negative judgment result in step S315. This process is repeated until the TOF data collected by the TOF sensor for the new time can recognize the second valid face direction information, or the timing duration of the electronic device reaches the threshold, and the electronic device executes step S313.

[0135] The electronic device determines that the timing duration has reached the threshold, indicating that the TOF sensor has been running for a sufficient time. While continuing to run the TOF sensor increases power consumption, it is also impossible to obtain the second valid face direction information based on the TOF data to control the screen rotation. Therefore, the TOF sensor can be turned off in step S313 to avoid the TOF sensor running and increasing power consumption.

[0136] The method for recognizing the direction of a face provided in the embodiment of the present application combines the low power consumption of a low-power camera and the high precision of a TOF sensor, thereby achieving complementary advantages between the shortcomings of the solution for recognizing the direction of a face through a low-power camera and the solution for recognizing the direction of a face through a TOF sensor, and taking into account both user experience and device power consumption.

[0137] After further research into the technical approach of the face direction recognition solution provided in this application, the inventors discovered that the face direction recognition solution can also be applied to other scenarios where electronic devices rely on low-power cameras to perform their functions. The technical approach of the face direction recognition solution refers to a method that combines a solution for recognizing face direction using a low-power camera with a solution for recognizing face direction using a TOF sensor, thereby achieving a balance between recognition accuracy and device power consumption.

[0138] The scenarios discovered by the inventors include but are not limited to: air gesture recognition, gaze detection, continuous identity detection, etc.

[0139] Taking gaze detection as an example, gaze detection means that when the screen display interface of an electronic device is in the process of the low-power camera collecting low-power images, the electronic device detects that the eyes on the face are looking at the screen based on the low-power image, and then controls the screen to not turn off.

[0140] The solution to apply the technical ideas of the face direction recognition solution to the gaze detection scenario is:

[0141] Electronic equipment execution Figure 3 In step S303 and step S304, the electronic device determines that the ambient light brightness is not less than the threshold value, and then uses the low-power image collected by the low-power camera to determine whether the human eye is looking at the screen. If the human eye is looking at the screen, the screen is controlled not to be turned off.

[0142] If the electronic device determines that the ambient light brightness is less than the threshold, it executes steps S309 to S310. The electronic device determines whether the human eye is looking at the screen based on the TOF data collected by the TOF sensor. If the electronic device determines that the human eye is looking at the screen, it turns off the TOF sensor and controls the screen so that it does not turn off. If the electronic device cannot determine whether the human eye is looking at the screen based on the TOF data collected by the TOF sensor, it executes step S315 to continue the operation of the TOF sensor for a period of time, and continues to determine whether the human eye is looking at the screen based on the TOF data collected by the TOF sensor, until the timing reaches the threshold or the determination result that the human eye is looking at the screen can be obtained based on the TOF data collected by the TOF sensor.

[0143] The electronic devices disclosed in the embodiments of the present application can be mobile phones, tablet computers, personal digital assistants (PDAs), desktop computers, laptop computers, notebook computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, wearable devices and other electronic devices.

[0144] Take mobile phones as an example. Figure 4This is an example of the composition of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 400 may include a processor 410, an internal memory 420, a camera 430, a TOF camera 440, a display screen 450, a sensor module 460, and the like.

[0145] It should be understood that in order to facilitate understanding of the embodiments of the present application, Figure 4 The electronic device 100 shown in the figure only includes some components related to the face direction recognition method provided in the embodiment of the present application. Figure 4 The electronic device 400 shown may have more or fewer components. That is, Figure 4 The electronic device 400 shown does not constitute a specific limitation on the electronic device provided in the embodiment of the present application.

[0146] Processor 410 may include one or more processing units, such as an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP), and a video codec. Processor 410 may also include a memory for storing instructions and data.

[0147] The internal memory 420 can be used to store computer executable program codes, which include instructions. The processor 410 executes the instructions stored in the internal memory 420 to execute various functional applications and data processing of the electronic device 400.

[0148] In some embodiments, the internal memory 420 stores instructions for executing the display method. The processor 410 can recognize the direction of a face and automatically rotate the screen based on the direction of the face by executing the instructions stored in the internal memory 420.

[0149] In other embodiments, the internal memory 420 may also store instructions for performing air gesture recognition, gaze detection, and continuous identity detection. The processor 410 can implement air gesture recognition, gaze detection, and continuous identity detection by executing the instructions stored in the internal memory 420.

[0150] The camera 430 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 400 may include 1 or N cameras 430, where N is a positive integer greater than 1. One camera 430 is a low-power camera and can be configured as a front-end camera.

[0151] The TOF camera 440 is used to obtain TOF data. The specific functions of the TOF camera 440 can be found in the above-mentioned introduction to related probabilities or technologies.

[0152] In the sensor module 460, the acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 400 in various directions (generally three axes). When the electronic device 400 is stationary, it can detect the magnitude and direction of gravity. The ambient light sensor 180L is used to sense the brightness of the ambient light.

[0153] Another embodiment of the present application further provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computer or a processor, the computer or processor executes one or more steps in any of the above methods.

[0154] The computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, a non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0155] Another embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer or processor, causes the computer or processor to execute one or more steps in any of the above methods.

Claims

1. A display method, characterized in that: Applied to an electronic device, the electronic device includes a display screen, a first camera, and a second camera, the first camera collects data by emitting and receiving light, and the resolution of the image collected by the second camera is less than a second threshold, the method comprising: The electronic device is in a first environment at a first moment, and the ambient light brightness of the first environment is less than a first threshold value to indicate that the brightness of the first environment is low; During the process of the electronic device being in the first environment, it is determined that the gravity direction of the electronic device has changed, the first camera is started to operate, the electronic device is in a first usage posture, and when a first facial direction is obtained based on data collected by the first camera, a first interface is displayed on the display screen, and an operation of a user controlling the electronic device to adjust from the first usage posture to a second usage posture is received, and when a fourth facial direction is obtained based on the data collected by the first camera, the display screen switches from displaying the first interface to displaying a third interface; and the first camera is turned off; the direction of the first interface is the same as the direction of the first facial direction, and the first facial direction is indicated by the data collected by the first camera; the direction of the first interface is different from the direction of the third interface, the direction of the third interface is the same as the direction of the fourth facial direction, and the second usage posture is different from the first usage posture; The electronic device is in a second environment at a second moment, and the ambient light brightness of the second environment is greater than the first threshold; When the electronic device is in the second environment, if the second facial direction is obtained based on the image captured by the second camera, a second interface is displayed on the display screen, and the direction of the second interface is the same as the second facial direction, and the second facial direction is indicated by the image captured by the second camera; if the second facial direction cannot be obtained based on the image captured by the second camera, a fifth interface is displayed on the display screen, and the direction of the fifth interface is the same as the gravity direction of the electronic device, and the gravity direction of the electronic device is indicated by the detection data of the acceleration sensor.

2. The display method according to claim 1, wherein: During the process of the second camera capturing images to obtain the second face direction, the first camera does not operate.

3. The display method according to claim 1 or 2, characterized in that: Also includes: When the electronic device is in a first usage posture, a camera preview interface is displayed on the display screen; receiving, while the electronic device is in the first environment, an operation of adjusting the electronic device from the first usage posture to a second usage posture by a user, and controlling the electronic device to display a first image, the first image being obtained by the user inputting a capture operation on the camera preview interface; the second usage posture being different from the first usage posture; In response to the user controlling the electronic device to adjust from the first usage posture to the second usage posture, and controlling the electronic device to display the first image, the display screen switches from displaying the camera preview interface to displaying a fourth interface, the direction of the camera preview interface is the same as the direction of the fourth interface, and the fourth interface is an image browsing interface.

4. The display method according to claim 1, wherein: Also includes: When processing the data collected by the first camera and the direction of the first face is not obtained, determining the timing duration; When the timing duration does not reach a third threshold, obtaining data newly collected by the first camera; Processing the newly acquired data of the first camera to obtain a third face direction; The display screen is controlled to display the first interface based on the direction of the third face, and the direction of the first interface is the same as the direction of the third face.

5. The display method according to claim 4, wherein: Also includes: When the timing duration reaches the third threshold, the first camera is controlled to stop collecting data.

6. An electronic device, characterized in that: include: one or more processors, a memory, a first camera, and a display screen; The first camera collects data by emitting and receiving light; The memory, the first camera and the display screen are coupled to the one or more processors, the memory is used to store a computer program, and the computer program includes computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the display method as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that Used to store a computer program, which, when executed, is specifically used to implement the display method according to any one of claims 1 to 5.

Images