Display method and device in screen-off mode

Abstract

The invention discloses a display method and device in a screen-off mode. The method comprises the steps that the electronic equipment enters a screen-off mode, and a screen-off display object is displayed on a display screen in the screen-off mode; the electronic equipment determines the orientation of a human face or human eyes based on human face or human eye detection, wherein the orientation is associated with the relative position between the human face or human eyes and the electronic equipment; and the electronic equipment dynamically changes the screen-off display object according to the change of the orientation of the human face or the human eyes, so that a dynamic interaction effect between the screen-off display object and a user is realized.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the priority of the Chinese patent application filed with the China Patent Office on August 29, 2023, with application number 202311112182.7 and application name “A method for interaction and electronic device in the screen-off state”, the entire contents of which are incorporated by reference in this application. Technical Field

[0003] The present application relates to human-computer interaction technology, and more specifically, to a display method and device in screen-off mode. Background Art

[0004] Smart devices are increasingly being used because they can provide rich and intelligent services. Smartphones are a type of smart device that has an independent operating system and independent operating space. Users can install software, games, navigation and other applications provided by third-party service providers, and can access wireless networks through mobile communication networks.

[0005] With the popularization of smart devices, users have higher and higher requirements for human-computer interaction. Therefore, how to improve human-computer interaction to improve user experience is a technical problem that needs to be solved at present. Summary of the invention

[0006] The present application provides an intelligent interaction method, which can enable an electronic device and a user to have a good interaction experience even in the screen-off mode.

[0007] In the first aspect, the present application provides a display method in screen-off mode, which is applied to an electronic device with a display screen. The method may include the following steps: the electronic device enters the screen-off mode, and displays a screen-off display object on the display screen in the screen-off mode; determines the orientation of a face or eye based on face or eye detection, and the orientation is associated with the relative position between the face or eye and the electronic device; and dynamically changes the screen-off display object according to the change of the orientation of the face or eye.

[0008] In the above implementation, in the screen-off mode, the screen-off display object can be dynamically changed based on the change in the orientation of the face or eyes, thereby achieving a dynamic interactive effect between the screen-off display object and the user.

[0009] In one possible implementation, the screen-off display object includes a first component, and the first component is used to represent eyes; dynamically changing the screen-off display object according to the change in the orientation of the human face or the human eyes includes: when the orientation of the human face changes, changing the first component according to the change, so that the display position or line of sight direction of the changed first component matches the orientation of the human face or the human eyes.

[0010] The above implementation method can achieve an interactive effect in which, in the screen-off mode, the screen-off display object changes dynamically with the change of the orientation of the face / eyes, so that the line of sight of the screen-off display object follows the user.

[0011] Optionally, the first component includes an eyeball in an eye; and changing the first component according to the change includes: changing a display position of the eyeball in the eye according to the change.

[0012] In one possible implementation, the screen-off display object includes a second component, and the second component is used to express emotions; the method also includes: determining the emotions expressed by the face or the eyes based on face or eye detection, and updating the second component so that the emotions presented by the screen-off display object match the emotions expressed by the face or the eyes.

[0013] In the above implementation, in the screen-off mode, the screen-off display object changes dynamically following the changes in the emotions expressed by the face / eyes, so that the emotions expressed by the screen-off display object follow the interactive effect of the user's emotions.

[0014] Optionally, determining the emotion expressed by the face or the eyes based on face or eye detection includes: utilizing artificial intelligence (AI) capabilities to perform face or eye detection to determine the emotion expressed by the face or the eyes.

[0015] In one possible implementation, the change in the orientation of the human face or the human eye includes: the human face or the human eye changes from a first orientation to a second orientation; the change in the method according to the human face or the human eye dynamically changes the screen display object, including: playing a first animation, the first animation includes N frames, N is an integer greater than 1, and each of the N frames contains the screen display object; among the N frames, the display state of the screen display object in the first frame matches the first orientation, the display state of the screen display object in the last frame matches the second orientation, and the display state of the screen display object in the second frame to the N-1th frame matches an intermediate orientation in the transition process from the first orientation to the second orientation.

[0016] In one possible implementation, the screen-off display object includes a three-dimensional object; and dynamically changing the screen-off display object according to the change in the orientation of the human face or the human eyes includes: when the orientation of the human face or the human eyes changes, changing the orientation of the three-dimensional object accordingly, so that the orientation of the three-dimensional object after the change matches the orientation of the human face or the human eyes.

[0017] In the above implementation, in the screen-off mode, the orientation of the screen-off display object changes dynamically with the orientation of the face / eyes, so that the screen-off display object is always facing the user's interactive effect. In addition, since the screen-off display object is a three-dimensional object, a naked-eye 3D visual effect can be achieved.

[0018] In one possible implementation, the screen-off display object includes a rotatable text object, the text object is used to display at least two texts, and when the text object is rotated, part of the at least two texts is displayed and / or part of the text is hidden; the dynamic change of the screen-off display object according to the change of the orientation of the face or the eyes includes: when the orientation of the face or the eyes changes, the text object rotates to present the hidden text. In some embodiments, the text before the rotation is different from the text presented after the rotation, which can be referred to Fig.13 .

[0019] In the above implementation, in the screen-off mode, the text object can display different text content as the position of the face / eye changes, thereby achieving an interactive effect with the user.

[0020] Optionally, when the orientation of the human face or the human eyes changes, the text object rotates to present the hidden text, including: when the orientation of the human face or the human eyes is a first orientation, the text object rotates in a direction opposite to the first orientation to present the hidden text.

[0021] Optionally, the text object is a three-dimensional object, the first surface of the three-dimensional object displays the first text, and the second surface of the three-dimensional object displays the second text; when the orientation of the face or the eyes changes, the text object rotates to present the hidden text, including: when the orientation of the face or the eyes changes from the first orientation to the second orientation, the text object rotates from the first angle to the second angle; or when the orientation of the face or the eyes changes from the second orientation to the first orientation, the text object rotates from the second angle to the first angle. Wherein, at the first angle, the first surface of the text object is displayed, and at the second angle, the second surface of the text object is displayed.

[0022] In one possible implementation, the text object is displayed as a hollow effect, and the hollow portion of the text object reveals a partial image of the desktop wallpaper of the electronic device; the method also includes: in response to an unlocking operation, switching from the screen-off mode to the screen-on mode through a one-shot visual effect, and displaying the desktop of the electronic device and the desktop wallpaper.

[0023] A possible implementation manner further includes: acquiring information detected by the electronic device using a front camera or a first sensor; and performing face or eye detection based on the detected information.

[0024] In a possible implementation, the change in the orientation of the human face or the human eyes is caused by movement of the user or movement of the electronic device.

[0025] In a second aspect, the present application provides a display method in a screen-off mode, which is applied to an electronic device with a display screen, including: the electronic device enters the screen-off mode; when it is detected that human eyes are looking at the display screen, a screen-off display object is displayed on the display screen in the screen-off mode; based on face or eye detection, the orientation of the face or eye is determined, and the orientation is associated with the relative position between the face or eye and the electronic device; according to the change of the orientation of the face or eye, the screen-off display object is dynamically changed.

[0026] In the above implementation, in the screen-off mode, the screen-off display object is displayed only when it is detected that a person is looking at the display screen, thereby saving power consumption of the electronic device. In addition, since the screen-off display object can be dynamically changed based on the change in the orientation of the face or the eyes, a dynamic interaction effect between the screen-off display object and the user can be achieved.

[0027] Other implementation methods of the second aspect can refer to the aforementioned first aspect and will not be described in detail.

[0028] In a third aspect, a device is provided, comprising a memory, a processor and a display device, wherein the memory is used to store a computer program, and the processor is used to read the computer program so that the device implements a method as described in any one of the first aspect or the second aspect.

[0029] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer, the computer implements the method as described in any one of the first aspect or the second aspect.

[0030] In a fifth aspect, a computer program product is provided, comprising computer-readable instructions, which, when the computer program product is run on a computer, enables the computer to execute a method as described in any one of the first aspect or the second aspect.

[0031] In a sixth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute a method as described in any one of the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of an emoticon in an embodiment of the present application;

[0033] Figure 2 A schematic diagram of a naked-eye 3D object in an embodiment of the present application;

[0034] Figure 3 A schematic diagram of a text object in an embodiment of the present application;

[0035] Figure 4 A schematic diagram of turning on the screen-off display function and setting a screen-off display object in an embodiment of the present application;

[0036] Figure 5 Schematic diagram of several face orientations in the embodiments of the present application;

[0037] Figure 6 A schematic diagram of the hardware structure of an electronic device in an embodiment of the present application;

[0038] Figure 7 A schematic diagram of the software architecture of an electronic device in an embodiment of the present application;

[0039] Figure 8 This is an interactive schematic diagram of the emoticons dynamically changing with the user's position in the screen-off mode in scenario 1 of the embodiment of the present application;

[0040] Fig. 9 This is an interactive schematic diagram of the dynamic changes of emoticons in the screen-off mode according to the user's emotions in scenario 2 of the embodiment of the present application;

[0041] Fig.10 This is an interactive schematic diagram of the emoticons dynamically changing with the user's position and emotions in the screen-off mode in scenario three of the embodiment of the present application;

[0042] Fig.11 This is an interactive schematic diagram of naked-eye 3D objects dynamically changing with user expressions in the screen-off mode in scenario 4 of the embodiment of the present application;

[0043] Fig.12 This is a schematic diagram of the appearance animation of the naked eye 3D object in scene 4 of the embodiment of the present application;

[0044] Fig.13A schematic diagram of a cube dynamically changing with the change of the user's position in the screen-off mode in scenario 5 of the embodiment of the present application, thereby displaying text contents on different surfaces of the cube;

[0045] Fig.14 A schematic diagram of switching from the screen-off mode to the screen-on mode in scenario 5 of the application embodiment;

[0046] Fig.15 This is a schematic diagram of another text object dynamically changing with the change of the user's position in scenario 5 of the embodiment of the present application;

[0047] Fig.16 A schematic diagram of a flow chart of a display method in screen-off mode provided in an embodiment of the present application;

[0048] Fig.17 A schematic flow chart of another display method in screen-off mode provided in an embodiment of the present application;

[0049] Fig.18 A schematic diagram of the structure of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind of", "the", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one, two or more. The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0051] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0052] In order to better understand the present application, some technologies and technical terms involved in the present application are briefly described below.

[0053] (1) Always on display (AOD)

[0054] AOD is a function that allows electronic devices to display useful thumbnail information. Even when the display screen of an electronic device is turned off (that is, it enters the screen-off mode, or the screen-off state), basic information (such as clock, calendar, notification, call, etc.) can be displayed in a minimized manner, so that users can view basic information in the most convenient way anytime and anywhere under low power consumption. That is to say, in the screen-off mode, the display screen of the electronic device can display limited information such as time and notification.

[0055] In the screen off mode, individual pixels on the display screen of the electronic device are lit up to display information such as clocks, notifications, etc., which can save power consumption compared to lighting up the entire display screen to display information such as clocks, notifications, etc.

[0056] AOD technology allows an electronic device to display a certain amount of information when it is in screen-off mode, so AOD is also called screen-off display (or off-screen display).

[0057] In the embodiment of the present application, the content that can be displayed in the screen off mode is called a screen off display object, which may also be called an all-weather display object or an AOD object. The embodiment of the present application does not impose any restrictions on the naming of the object.

[0058] In the screen-off mode, one or more screen-off display objects can be displayed. The screen-off display objects can be pre-set, and the user can view the type and display mode of the pre-set screen-off display objects in the setting interface, and can select the screen-off display objects that they want to display in the screen-off mode. The screen-off display objects can also be generated based on the pictures selected by the user or the text entered, so that a more personalized screen-off display object can be obtained.

[0059] For example, in the embodiment of the present application, the type and setting method of the screen-off display object may include:

[0060] - Basic information. For example, it includes time, date, SMS and call reminders. Users can choose to display basic information or not. Users can also choose how to display basic information. For example, the basic information display mode can include touch display (i.e., basic information is displayed when the screen is touched), timed display, and all-day display.

[0061] -Pictures. The pictures may include static pictures or dynamic pictures. A variety of pictures may be built into the electronic device for the user to choose from. Optionally, the electronic device may also generate personalized pictures (such as personalized static pictures or personalized dynamic pictures) as the screen-off display object based on the pictures selected by the user or the editing operation on the pictures. Optionally, the electronic device may also allow the user to select a picture from the gallery of the electronic device and generate the screen-off display object based on the picture. In addition, the screen-off display object may also include an animation composed of multiple pictures.

[0062] -Emojis. A variety of emoticons may be built into the electronic device for the user to choose from. Optionally, the electronic device may also allow the user to use emoticons downloaded from the network side or emoticons obtained from other channels, which is not limited in the embodiments of the present application.

[0063] An emoticon can be made up of multiple components. For example, an emoticon can include an object for representing eyes, an object for representing a mouth, and so on. Figure 1 An emoticon is shown as an example, and the emoticon includes components such as eyes, glasses, and a mouth. In other embodiments, the screen-off interface can also display multiple different emoticons at the same time. In this application, in order to distinguish from the screen-off display object, the components included in an screen-off display object are referred to as components. For example, for Figure 1 For the emoticon shown in FIG. 1 , the object used to represent eyes in the emoticon is called an eye component or a first component, the object used to represent a mouth in the emoticon is called a mouth component or a second component, and so on.

[0064] In some embodiments of the present application, the electronic device may allow the user to construct (or build) his / her own personalized emoticon. Exemplarily, some components for constructing emoticons may be pre-set in the electronic device, and the user may select these components to construct a personalized emoticon. For example, the electronic device may pre-set multiple styles of eyes, multiple styles of noses, and multiple styles of mouths, from which the user may select a style of eyes, nose, and mouth, and perform some personalized settings or edits to generate a personalized emoticon.

[0065] -Avatars. Similar to emoticons, users can choose avatars built into electronic devices, avatars obtained from other sources, or personalized avatars designed or edited by the user. A avatar can include one or more components.

[0066] It can be understood that the virtual image in the embodiment of the present application may include images of people, etc., and the embodiment of the present application does not limit this.

[0067] -Signature information. The electronic device may have multiple signature information and display modes (such as fonts, colors, etc.) built in for the user to choose. Optionally, the electronic device may also generate a personalized signature of the user as a screen-off display object based on the signature content input by the user and the display mode selected.

[0068] Optionally, an embodiment of the present application further provides a screen-off display object, which has a naked-eye 3D display effect and is referred to as a naked-eye 3D object in this document. Figure 2 An example of a naked-eye 3D object is exemplarily shown, and the naked-eye 3D object is displayed as a 3D “astronaut”.

[0069] Optionally, the embodiment of the present application also provides a screen display object called "text object", which can display multiple texts through the text object, and the displayed text can include signature information, date, weather, time, etc. It can be understood that the embodiment of the present application does not limit the naming of the text object, and the following description takes the text object as an example.

[0070] The text object can present different states, and different texts can be displayed in different states. In one state, part of the text may be displayed and part of the text may be hidden. In a possible implementation, the text object is a three-dimensional polyhedron, and different surfaces of the three-dimensional polyhedron can be used to display different text contents. Figure 3 A schematic diagram of displaying multiple text contents through a three-dimensional cube is exemplified, and the figure only shows the text contents displayed on two surfaces of the three-dimensional cube, one surface is used to display the date (as shown in the figure "SUN SEP 07"), and the other surface is used to display a personalized signature (as shown in the figure "JUST LOVE LIFE").

[0071] In some other embodiments of the present application, other information may be displayed on the surface of the three-dimensional polyhedron, such as weather, etc., which is not limited in the embodiments of the present application.

[0072] In the embodiment of the present application, the electronic device allows the user to turn on or off the screen display function, and also allows the user to set the screen display object. Figure 4 A user interface diagram of a user enabling the screen-off display function and setting the screen-off display object provided by an embodiment of the present application is shown. Figure 4As shown, the user interface includes an off-screen display function option 401, and the user can turn on or off the off-screen display function by clicking on the function option. The user interface also provides a variety of off-screen display objects such as emoticons and naked-eye 3D objects (such as the content displayed in the dotted box 402 area in the figure) for the user to choose. It can be understood that the above only exemplifies several off-screen display objects and the setting methods of the off-screen display objects, and the embodiments of the present application do not limit the types and settings of the off-screen display objects.

[0073] (2) Eyes on display (EOD)

[0074] In one implementation, EOD technology means that when the electronic device detects that the user is looking at the display screen, the AOD display will be lit, otherwise the AOD display will not be lit. That is to say, when the electronic device detects that the user is looking at the display screen, the off-screen display object is displayed on the display screen. When the electronic device does not detect that the user's glasses are looking at the display screen for a set period of time (for example, when the electronic device is put into a pocket or a bag), the off-screen display is turned off to save power consumption of the device.

[0075] The EOD function can also be considered as a type of AOD function.

[0076] (3) Face detection

[0077] Face detection refers to the use of a certain strategy (or algorithm) to search any given image to determine whether it contains a face. If it does, the position, size, and posture of the face are further determined.

[0078] Optionally, the embodiment of the present application may adopt the following face detection method:

[0079] Knowledge rule-based method: A method that uses face rules (such as the distribution rules of eyes, nose, mouth, etc.) to detect faces in input images. This type of method mainly uses some simple rules to describe facial features and the relationship between these features.

[0080] Methods based on invariant features: Face detection is performed on input images by looking for invariant features of the face. This type of method assumes that some features of the face (such as skin color, edges, texture, etc.) will not change under different postures, viewing angles, and expressions.

[0081] Template matching based method: The facial features of the face are described by a standard face template. When performing face detection, the correlation value between the input image subwindow and the standard face template is first calculated, and then the obtained correlation value is compared with the pre-set threshold to determine whether there is a face in the input image.

[0082] Methods based on statistical models: The face region is regarded as a type of pattern, and a large amount of face data is used as a sample for training to learn potential rules and construct a classifier, which is used to detect faces by distinguishing the attributes of all possible regional patterns in the image. This type of method can be implemented based on artificial intelligence (AI), such as using neural networks to implement face detection.

[0083] It can be understood that the above only illustrates several face detection methods by way of example, and the embodiments of the present application do not limit the face detection methods.

[0084] In some embodiments of the present application, the electronic device can collect information through a device or component provided thereon, and then perform face detection based on the collected information and using the above-mentioned face detection algorithm. Optionally, the device or component can be a front camera provided on the electronic device, and the electronic device can perform face detection based on the image collected by the front camera. The device or component may also be other types of sensors, such as an infrared detection sensor, which is not limited in the embodiments of the present application.

[0085] In some embodiments of the present application, based on face detection, the electronic device can determine the orientation of the face, and the orientation of the face is associated with the relative position between the face and the electronic device. The orientation of the face is associated with the relative position between the face and the electronic device, which can be understood as: the relative position between the face and the electronic device can be reflected or represented based on the orientation of the face, or the relative position between the face and the electronic device can be determined based on the orientation of the face.

[0086] Optionally, the position of the face in the image can be obtained by a face detection algorithm, and the position can be specifically the offset of the face relative to the center of the image. The orientation of the face can be determined according to the position of the face in the image, and the orientation can reflect the relative position between the face and the electronic device. Exemplary possible situations include the following:

[0087] Case 1: A reference point on the face (such as the center point of the face, or the position of the tip of the nose on the face, or the center point between the eyes on the face, etc.) is located in the central area of ​​the image, where the central area can be, for example, the center point of the image, or can be a smaller area with the center point of the image as the center and a set length as the radius. In this case, it can be determined that the face is facing the display screen of the electronic device, or that the face is not offset to the left, right, downward or upward relative to the display screen.

[0088] Case 2: The reference point on the face is located in the left area of ​​the central area of ​​the image. In this case, it can be determined that the face is facing the left side of the display screen of the electronic device, or the face is offset to the left relative to the display screen.

[0089] Case 3: The reference point on the face is located in the right area of ​​the central area of ​​the image. In this case, it can be determined that the face is facing the display screen of the electronic device on the right side, or the face is offset to the right relative to the display screen.

[0090] Case 4: The reference point on the face is located in the upper area of ​​the central area of ​​the image. In this case, it can be determined that the upper side of the face is facing the display screen of the electronic device, or that the face is offset upward relative to the display screen.

[0091] Case 5: The reference point on the face is located in the lower area of ​​the central area of ​​the image. In this case, it can be determined that the lower side of the face is facing the display screen of the electronic device, or that the face is offset downward relative to the display screen.

[0092] Furthermore, according to the offset of the face in the image, the offset of the face relative to the display screen of the electronic device can be further determined.

[0093] Figure 5 Schematic diagrams showing the orientation of faces in several situations are shown. Figure 5 In (a), the user's head is located at the center relative to the electronic device display screen, and the black dot on the display screen is used to indicate the orientation of the user's face. Figure 5 In (b), the user's head is located on the right side relative to the electronic device display screen, and the black dot on the display screen is used to indicate the orientation of the user's face. Figure 5 In (c), the user's head is located on the left side relative to the display screen of the electronic device, and the black dot on the display screen is used to indicate the orientation of the user's face.

[0094] It is understood that the above only illustrates several possible relative positions between a face and an electronic device, and the present application embodiment does not limit this. Based on the above principle, the position of the face in the image can be determined according to the face detection information, and the relative position of the face relative to the electronic device (or the display screen of the electronic device) can be determined based on the position.

[0095] In some other embodiments of the present application, the rotation angle of the face can be obtained by a face detection algorithm, and accordingly, the rotation angle of the face relative to the electronic device (or the display screen of the electronic device) can be determined based on the rotation angle of the face. The rotation angle may include one or more of the following: a rotation angle in a horizontal dimension, a rotation angle in a vertical dimension, and a rotation angle in a depth dimension.

[0096] In other embodiments of the present application, the depth of field of the face can be obtained by a face detection algorithm, and accordingly, the distance of the face relative to the electronic device (or the display screen of the electronic device) can be determined based on the depth of field of the face.

[0097] In some other embodiments of the present application, the facial expression can be obtained by a face detection algorithm, and the emotion expressed by the face can be further determined based on the facial expression. Exemplarily, the emotion may include: happiness, sadness, anger, etc. The face detection algorithm can determine the facial expression based on the details of the face, such as the shape of the facial features (such as the opening and closing state of the eyes and mouth), the relative position of the facial features, etc.

[0098] Based on the above description of the face detection algorithm, it can be understood that in the embodiments of the present application, the face detection algorithm can determine one or more of the following: the position offset of the face relative to the display screen of the electronic device, the rotation angle of the face relative to the display screen of the electronic device, the depth of field of the face relative to the display screen of the electronic device, the expression of the face, etc.

[0099] Similar to face detection, eye detection (e.g., information such as the distance between the eyes, the center point between the eyes, the state of the eyes, etc.) can be used to obtain information such as the position offset of the eyes relative to the electronic device display screen, the rotation angle of the eyes relative to the electronic device display screen, the depth of field of the eyes relative to the electronic device display screen, and facial expressions. In the following of this application, face detection or eye detection is referred to as "face / eye detection".

[0100] (4) Eye tracking

[0101] Eye tracking technology mainly studies the acquisition, modeling and simulation of eye movement information. Eye tracking can include tracking based on changes in the features of the eye and its surroundings, or tracking based on changes in the angle of the iris, or actively projecting infrared light beams to the iris to extract features for tracking. Equipment used to implement eye tracking may include infrared equipment, image acquisition equipment, etc. For example, for color or grayscale images captured by an image acquisition device, pattern recognition algorithms such as template matching, projection, and neural networks can be used to implement eye tracking.

[0102] In some embodiments of the present application, eye tracking may be used to determine whether a human eye is looking at a display screen of an electronic device. When it is determined that a human eye is looking at the display screen of an electronic device, an AOD display may be performed.

[0103] In other embodiments of the present application, the line of sight of a person's eyes can be determined by eye tracking, or changes in the line of sight of a person's eyes can be determined.

[0104] In other embodiments of the present application, the state of a person's eyes can be obtained through eye tracking, and the expression of a person's face can be determined based on the state of the person's eyes.

[0105] In the related art, when the AOD display is turned on, the off-screen display object is displayed on the display screen of the electronic device. If the off-screen display object is a static object (such as a static picture or text), the off-screen display object is always displayed as static on the display screen. If the off-screen display object is a dynamic object (such as a dynamic picture), the off-screen display object is displayed as a static state after the dynamic display is completed on the display screen, and the dynamic effect of the off-screen display object is preset. In other words, in the related art, no matter whether the off-screen display object is static or dynamic, it will not interact with the user.

[0106] In order to improve the interactivity in the screen-off mode, the embodiment of the present application provides a display method in the screen-off mode and a related device that can implement the method. In the embodiment of the present application, in the screen-off mode, the electronic device can determine the orientation of the face / eye relative to the display screen of the electronic device based on face / eye detection, so that the screen-off display object can be dynamically displayed according to the orientation of the face / eye relative to the display screen of the electronic device, thereby realizing the orientation change between the face / eye relative to the display screen of the electronic device, so that the user can have a dynamic interactive experience with the screen-off display object.

[0107] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0108] The method provided in the embodiment of the present application can be applied to electronic devices. The electronic device can realize human-computer interaction. For example, the electronic device can specifically be a non-folding screen mobile phone, a folding screen mobile phone, a wearable device (such as a smart watch, a smart bracelet, etc.), a tablet computer, a laptop computer, a smart screen, a vehicle terminal, a computer, a computer, a personal computer (personal computer, PC), an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook, a personal digital assistant (personal digital assistant, PDA), a virtual reality (virtual reality, VR) device / augmented reality (augmented reality, AR) device, an artificial intelligence (artificial intelligence, AI) device and other terminal devices with display functions, and the operating systems installed in the above electronic devices include but are not limited to Or other operating systems. This application does not limit the specific type of the above electronic equipment and the installed operating system.

[0109] See also Figure 6 , which is a possible hardware structure of the electronic device 100 in the embodiment of the present application. The electronic device can execute the method provided in the embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0110] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a display processing unit (DPU), and / or a neural-network processing unit (NPU). Among them, different processing units may be independent devices or integrated into one or more processors. In some embodiments, the electronic device 100 may also include one or more processors 110. Among them, the processor is the nerve center and command center of the electronic device 100. The processor can generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. A memory may also be set in the processor 110 for storing instructions and data.

[0111] The camera 193 may include one camera or a group of cameras. The camera 193 may include a front camera and a rear camera.

[0112] The sensor module 180 may include one or more of the following: a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, a vibration sensor, and an infrared detection sensor.

[0113] The electronic device 100 can realize the display function through the GPU, the display screen 194 and the application processor AP, etc. The display screen 194 is used to display images, videos, etc.

[0114] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0115] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0116] Figure 7 This is a software system architecture block diagram of an electronic device provided in an embodiment of the present application. For example, Figure 7 As shown, the software system architecture of the electronic device can be a layered architecture, for example, the software can be divided into several layers, and the layers communicate with each other through software interfaces. In some embodiments, the operating system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer (framework, FWK), the runtime and system library, the kernel layer, and the hardware layer.

[0117] The application layer can include a series of application packages. Figure 7 As shown, the application layer may include camera, settings, skin module, user interface (UI), third-party applications, etc. Among them, the third-party applications may include wireless local area network (WLAN), music, calls, Bluetooth, video, memo, notes, etc.

[0118] In one possible implementation, applications can be developed using the Java language by calling the application programming interface (API) provided by the application framework layer. Developers can interact with the bottom layer of the operating system (such as the hardware layer, kernel layer, etc.) through the application framework layer to develop their own applications. The application framework layer is mainly a series of services and management systems of the operating system.

[0119] The application framework layer provides application programming interfaces and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. Figure 7 As shown, the application framework layer may include an activity manager, a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0120] The activity manager is used to manage the life cycle of each application and provide common navigation back functions, providing an interactive interface for all program windows.

[0121] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make this data accessible to applications. This data can include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0122] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0123] The phone manager is used to provide communication functions for electronic devices, such as the management of call status (including answering, hanging up, etc.).

[0124] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, etc.

[0125] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0126] The runtime includes the core library and the virtual machine. The runtime is responsible for the scheduling and management of the operating system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the core library of the operating system. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection and other functions.

[0127] The system library may include multiple functional modules. For example: surface manager, media framework, 3D graphics processing library (e.g. OpenGL ES), 2D graphics engine (e.g. SGL), etc. The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The media framework supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media framework can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing. In some embodiments, the 3D graphics processing library can be used to draw 3D motion trajectory images, and the 2D graphics engine can be used to draw 2D motion trajectory images.

[0128] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0129] The hardware layer can include various types of sensors, such as accelerometers, gravity sensors, touch sensors, etc.

[0130] It should be understood that the hardware structure of the electronic device can be as follows Figure 6 As shown, the software system architecture can be Figure 7 As shown, the software programs and / or modules corresponding to the software system architecture in the electronic device can be stored in the internal memory 121, and the processor 110 can run the software programs and applications stored in the internal memory 121 to execute the process of the display method provided in the embodiment of the present application.

[0131] In addition to the above examples, the method proposed in this application can also be applied to other types of software systems, such as Hongmeng operating system, Windows operating system, iOS operating system, etc.

[0132] In a display method in screen-off mode provided in an embodiment of the present application, after the electronic device enters the screen-off mode, the screen-off display object is displayed on the display screen in the screen-off mode. The electronic device performs face / eye detection to determine the orientation of the face / eye, which is related to the relative position of the electronic device and the face / eye. The electronic device can dynamically change the screen-off display object according to the change of the orientation of the face / eye, thereby achieving a dynamic interactive effect.

[0133] It should be understood that the objects displayed on the off screen may be displayed all the time, or may be displayed only when it is detected that the user's eyes are looking at the screen (or a face is detected).

[0134] In order to better understand the display method in the screen-off mode provided in the embodiment of the present application, the following first describes the human-computer interaction behavior and interface changes on the electronic device side in the embodiment of the present application by taking several possible scenarios as examples.

[0135] Scene 1

[0136] Scene 1 Figure 1 Taking the emoticons shown in the figure as an example, an interactive scenario is described in which the emoticons displayed on the screen in the screen off mode change dynamically as the user's position changes.

[0137] See also Figure 8 In (a), the electronic device is currently in the screen-off mode. After the front camera set on the electronic device captures an image, the electronic device performs face / eye detection on the image to determine that the face / eye is located at the center of the image. Since the face / eye is located at the center relative to the display screen at this time, the position of the eyeball 801 and the glasses 802 in the emoticon 800 on the display screen matches the current position of the face / eye. As shown in the figure, the eyeball 801 is located in the middle of the eye, achieving the effect that the emoticon 800 is looking at the user.

[0138] See also Figure 8 In (b), after the user's head moves to the right, the front camera set on the electronic device captures an image, and the electronic device performs face / eye detection on the image to determine that the face / eye is located at a position slightly to the right of the center of the image. Since the face / eye is located slightly to the right of the center relative to the display screen at this time, the position of the eyeball 801 and the glasses 802 in the emoticon 800 on the display screen shifts to the right to match the current position of the face / eye, so that the line of sight of the emoticon 800 follows the user's position and always looks at the user.

[0139] See also Figure 8 In (c), after the user's head continues to move downward, the front camera set on the electronic device captures an image, and the electronic device performs face / eye detection on the image to determine that the face / eye is located at the lower right position of the center of the image. Since the face / eye is located at the lower right position of the center relative to the display screen at this time, the position of the eyeball 801 and the glasses 802 in the emoticon 800 on the display screen is offset downward to match the current position of the face / eye, so that the line of sight of the emoticon 800 follows the user's position and always looks at the user.

[0140] The user's head continues to move, and accordingly, the positions of the eyeball 801 and the glasses 802 in the emoticon 800 change with the change of the user's position. For details, see Figure 8 (d) in.

[0141] Optionally, when the positions of the eyeball 801 and the glasses 802 change with the change of the user's position, the degree of position change of the eyeball 801 and the glasses 802 (such as displacement size and / or angle) may be different. For example, the displacement of the eyeball 801 may be greater than the displacement of the glasses 802, so as to achieve a more natural and realistic dynamic display effect. The embodiments of the present application are not limited to this.

[0142] Optionally, if no face / eye is detected based on the image captured by the front camera, the electronic device may maintain the state of the emoticon displayed last time (or the last time before the current moment), or may also display the default state of the emoticon, such as Figure 8 The state of the emoji in (a) is set to the default state.

[0143] Scene 2

[0144] Scene 2 Figure 1 Taking the emoticons shown in the figure as an example, an interactive scenario is described in which the emoticons displayed on the screen in the screen-off mode change dynamically as the user's emotions change.

[0145] See also Fig. 9 In (a), the electronic device is currently in the screen-off mode. After the front camera set on the electronic device captures an image, the electronic device determines that the emotion expressed by the face is sadness by performing face / eye detection on the image. The electronic device displays the mouth 901 in the emoticon 900 as bending downward to express the emotion of sadness, so that the emotion expressed by the emoticon 900 matches the emotion expressed by the current face.

[0146] See also Fig. 9 In (b), after the user's emotion changes from sadness to happiness, the front camera set on the electronic device captures an image, and the electronic device performs face / detection on the image to determine that the emotion expressed by the face is happiness. The electronic device displays the mouth 901 in emoticon 900 as bending upward to express a happy emotion, so that the emotion expressed by emoticon 900 changes dynamically with the changes in the user's emotion and always matches the user's emotion, thereby establishing an emotional state connection with the user.

[0147] Fig. 9 (c) and (d) in the figure show another emoticon that changes dynamically from sadness to happiness according to the emotion expressed by the user's face.

[0148] In another embodiment, if it is detected that the emotion expressed by the current face is sadness, the mouth 901 in the emoticon 900 can be displayed as bending upward to express a happy emotion. That is to say, the emotion expressed by the emoticon can be different from the emotion expressed by the current face (for example, opposite), thereby achieving the effect of improving the user's mood or pleasing the user.

[0149] Optionally, if no face / eyes are detected based on the image captured by the front camera, the electronic device can maintain the state of the emoticon displayed last time (or the last time before the current moment), or can also display the default state of the emoticon, for example, the state in which the emoticon expresses a happy emotion can be set as the default state.

[0150] Scene 3

[0151] Scene 3 Figure 1 Taking the emoticons shown in the figure as an example, an interactive scenario is described in which the emoticons displayed on the screen in the screen-off mode change dynamically as the user's position and emotions change.

[0152] The implementation method of scenario three can be understood as combining the implementation method of scenario one with the implementation method of scenario two. That is to say, through face / eye detection, both the position of the face / eye and the emotion expressed by the face can be determined. Accordingly, matching emoticons can be displayed based on the position and expression of the face / eye. On the one hand, the line of sight of the emoticon follows the user's position to achieve the effect of always looking at the user. On the other hand, the emotions expressed by the emoticon change dynamically with the changes in the user's emotions and always match the user's emotions.

[0153] Fig.10 The diagram exemplarily shows the dynamic changes of emoticons displayed on the display screen of the electronic device when the relative position between the user and the display screen of the electronic device changes and the user's emotions also change. Fig.10 The figure shows the orientation of the user's face and the facial expression at this time, and correspondingly, the figure shows the display effect of the emoticon on the display screen.

[0154] Scene 4

[0155] Scene 4 Figure 2 Taking the naked eye 3D object shown in the figure as an example, an interactive scene is described in which the naked eye 3D object displayed on the display screen in the screen off mode changes dynamically with the user's position. The naked eye 3D object is a three-dimensional display effect.

[0156] See also Fig.11In (a), the electronic device is currently in the screen-off mode. After the front camera set on the electronic device captures an image, the electronic device performs face / eye detection on the image to determine that the face / eye is located at the center of the image. Since the face / eye is located at the center relative to the display screen at this time, the orientation of the naked-eye 3D object 1100 on the display screen matches the current position of the face / eye. As shown in the figure, the naked-eye 3D object 1100 faces the user, achieving the effect that the naked-eye 3D object 1100 faces the user.

[0157] See also Fig.11 In (b), after the user's head moves to the right, the front camera set on the electronic device captures an image, and the electronic device performs face / eye detection on the image to determine that the face / eye is located at a position slightly to the right of the center of the image. Since the face / eye is located slightly to the right of the center relative to the display screen at this time, the orientation of the naked-eye 3D object 1100 on the display screen deflects to the right, and the changed orientation matches the current position of the face / eye, achieving the effect that the orientation of the naked-eye 3D object 1100 changes with the user's position and always faces the user.

[0158] See also Fig.11 In (c), after the user's head moves to the left, the front camera set on the electronic device captures an image, and the electronic device performs face / eye detection on the image to determine that the face / eye is located to the left of the center of the image. Since the face / eye is located to the left of the center relative to the display screen at this time, the orientation of the naked eye 3D object 1100 on the display screen deflects to the left, and the changed orientation matches the current position of the face / eye, so that the orientation of the naked eye 3D object 1100 changes with the user's position and always faces the user.

[0159] Since the naked-eye 3D object has a three-dimensional display effect and can present images at different angles according to changes in the orientation of the face, the naked-eye 3D visual effect can be achieved.

[0160] Fig.11 The description is based on the example of the change in the horizontal direction of the face orientation. The embodiments of the present application do not limit this. For example, when the face orientation changes in other directions, the naked-eye 3D object can also rotate in that direction accordingly based on the change to achieve a dynamic change effect toward the current user.

[0161] In some embodiments of the present application, when the object displayed on the off-screen is a naked-eye 3D object, the naked-eye 3D object can be directly displayed on the display screen after the display screen is lit. In other embodiments, when the display screen is lit, the appearance animation of the naked-eye 3D object can be displayed first to show the appearance effect of the naked-eye 3D object, thereby increasing the vividness and interest of the display effect.

[0162] by Figure 2 Take the naked eye 3D object shown as an example, Fig.12 The schematic diagram of several frames in the appearance animation of the naked eye 3D object is shown as an example. Fig.12 As shown, through this exit animation, the effect of the naked-eye 3D object "astronaut" gradually exiting the cabin can be presented. Based on this exit animation, when the user's eyes are fixed on the display screen of the electronic device, a hatch appears on the screen, and the "astronaut" comes out of the hatch. After that, while the user is looking at the display screen of the electronic device, he shakes the electronic device or moves his face, and the current picture will change according to the position of the face, presenting the picture associated with the corresponding perspective, thereby achieving the naked-eye 3D effect.

[0163] Scene 5

[0164] Scene 5 takes the display of text through a cube as an example, and describes a scene in which the cube changes dynamically with the change of the user's position, thereby displaying the text content on different surfaces of the cube.

[0165] Text can be displayed on each surface of the cube, and the text may include one or more of date, time, weather, signature, etc. In this example, the display of date and signature information on three surfaces of the cube is described as an example.

[0166] See also Fig.13 After the front camera set on the electronic device captures the image, the electronic device determines the position of the face / eye by detecting the face / eye on the image. When the face / eye is located at the center relative to the display screen, one surface of the cube on the display screen faces the user, and the text displayed on the surface is the date, specifically "SUM SEP 07" as shown in the figure.

[0167] When the user's head moves to the left, the front camera of the electronic device captures an image, and the electronic device performs face / eye detection on the image to determine that the face / eye is located slightly to the left of the center of the image. Since the face / eye is located slightly to the left of the center relative to the display screen at this time, the orientation of the cube on the display screen deflects to the left, and part of the other surface of the rotated cube is displayed. As the user's head continues to move to the left, the orientation of the cube on the display screen continues to deflect to the left, and the left surface of the rotated cube is fully displayed, as shown in FIG. Fig.13 As shown, the left side surface of the cube is fully displayed, and the left side surface faces the user. The text displayed thereon is personalized signature information, specifically "JUST LOVE LIFE" as shown in the figure.

[0168] Similarly, when the user's head moves to the right, the cube gradually rotates to the right as the user's head moves, thereby gradually displaying the right side surface, so that the user can see the signature information displayed on the right side surface.

[0169] Optional, such as Fig.13 As shown, each surface in the cube can be partially hollowed out, for example, the text part can be hollowed out, such as Fig.13 For the text displayed on each surface, the part occupied by the text is hollowed out, through which the desktop wallpaper on the lower layer can be presented, and the other parts are opaque; for another example, on one surface of a cube, the part occupied by the text is opaque, and the rest is hollowed out, through which the desktop wallpaper on the lower layer can be presented.

[0170] Fig.13 The description is based on the example of the change in the horizontal orientation of a human face, but the embodiments of the present application do not limit this. For example, when the orientation of a human face changes in other directions, the text object can also rotate in the opposite direction based on the change to display the text content on the corresponding surface.

[0171] Based on the above scenario 5, the following effects can be achieved: in the screen-off mode, when the user's eyes are looking at the electronic device, a large-character poster with the current time and date information appears on the screen, and the font is hollowed out, revealing the desktop wallpaper currently set by the user. While the user is looking at the electronic device, shaking the electronic device or moving the face, the current picture will change according to the position of the face, showing the hidden information of the corresponding perspective.

[0172] Optionally, when the user performs an unlocking operation and the display screen is unlocked, the electronic device exits the screen-off display mode, and the display screen of the electronic device is fully lit, displaying the desktop and desktop wallpaper. In this process, the electronic device can switch from the screen-off mode to the screen-on mode through a one-shot visual effect, and display the desktop and desktop wallpaper of the electronic device. In the embodiment of the present application, since the text objects in the screen-off mode are displayed as a hollow effect, part of the desktop wallpaper can be seen, so that part of the desktop wallpaper can also be seen in the screen-off mode, and then through the transition animation from the screen-off mode to the screen-on mode, it can be smoothly, coherently, and naturally transitioned to the screen-on mode, thereby achieving a "one-shot" display effect.

[0173] See also Fig.14 , exemplarily illustrates a transition animation provided by an embodiment of the present application for switching from screen-off mode to screen-on mode to achieve a "one-shot" display effect. The effect of the transition animation is: the picture on the current display screen is gradually enlarged with the center point position, and the lower-layer desktop wallpaper is displayed more and more until it is all displayed and switches to screen-on mode. Optionally, in the transition animation, the transparency of the non-hollowed-out part of the text object can be gradually increased to achieve a display effect in which the desktop background gradually becomes clearer from invisible, thereby making the transition process from screen-off mode to screen-on mode smoother and more natural.

[0174] Fig.15 Another dynamic display effect under scene 5 is shown as an example. Fig.13 The animation effect shown is similar to that shown in the figure, except that the style of the text object is the same as Fig.13 The styles of text objects in are different. Fig.15 The text object shown is displayed in a two-dimensional effect. The text object has different states, and the text content displayed in different states is different. For example, in a certain state, the date and time information is displayed (such as the text content displayed by the text object when the user's face is in the middle position in the figure), in another state, the date, location information and weather information are displayed (such as the text content displayed by the text object when the user's face is in the left position in the figure), and in another state, the date and personalized signature information are displayed (such as the text content displayed by the text object when the user's face is in the side position in the figure). Text objects in different states can be displayed according to the different positions of the user's face / eyes. Fig.15 The description is based on the example of the change in the horizontal direction of the face orientation, but the embodiments of the present application do not limit this. For example, when the face orientation changes in other directions, the text object can also display the text content in the corresponding state based on the change.

[0175] Combine the following Fig.16 The display method in the screen off mode provided in an embodiment of the present application is described.

[0176] See also Fig.16 , is a flow chart of a display method in the screen-off mode provided in an embodiment of the present application, and the flow can be Figure 6 or Figure 7 The electronic device shown in the figure may be executed, or the functional module, processor or chip in the electronic device may be executed. The following process is described with the electronic device as the execution subject. As shown in the figure, the process may include the following steps:

[0177] Step 1601: The electronic device enters the screen-off mode, and displays a screen-off display object on the display screen in the screen-off mode.

[0178] In the embodiment of the present application, the screen-off function can be enabled in advance and the screen-off display object displayed in the screen-off mode can be set. In this way, when the electronic device meets the screen-off condition (for example, the electronic device has not been used within a set time), the screen-off mode can be automatically entered, and the screen-off display object can be displayed on the display screen in the screen-off mode. An example of enabling the screen-off mode and setting the screen-off display object displayed in the screen-off mode can be referred to. Figure 4 , which will not be repeated here.

[0179] In the embodiment of the present application, there is no restriction on the objects displayed on the screen, for example, it can be signature information, emoticons, pictures, virtual images, naked-eye 3D objects, text objects, etc.

[0180] Step 1602: The electronic device determines the orientation of the face / eye based on face / eye detection, where the orientation of the face / eye is associated with the relative position between the face / eye and the electronic device.

[0181] In one possible implementation, a front camera or a first sensor provided on an electronic device may collect information and obtain detection information for face / eye detection. The "first sensor" here refers to one or more types of sensors, and the information detected by the sensor can be used for face / eye detection. Exemplarily, the sensor may be an infrared detection sensor, and the infrared detection sensor may collect infrared images for face detection. Taking the front camera as an example, after the front camera collects an image, the image may be passed to a processor, and the processor executes a face / eye detection algorithm to implement face / eye detection for the image. The implementation method of face / eye detection may refer to the above.

[0182] Optionally, the front camera or the first sensor on the electronic device can collect information according to a set period.

[0183] As described above, in the embodiment of the present application, one or more of the following information can be obtained through face / eye detection: the position of the face / eye relative to the electronic device display screen, the rotation angle of the face / eye relative to the electronic device display screen, and the depth of field of the face / eye relative to the electronic device display screen. Optionally, the orientation of the face / eye can be determined by combining the above multiple information to further improve the accuracy.

[0184] Step 1603: The electronic device dynamically changes the screen display object according to the change in the orientation of the face / eyes.

[0185] In the embodiment of the present application, when the electronic device is stationary, the change of the user's position (such as movement) or the rotation or shaking of the user's head may cause the orientation of the face / eye to change, or may cause the relative position between the face / eye and the electronic device display screen to change; when the user is stationary, the position change, rotation or shaking of the electronic device may cause the orientation of the face / eye to change, or may cause the relative position between the face / eye and the electronic device display screen to change. Regardless of the reason for the change, in the embodiment of the present application, the screen display object will change dynamically accordingly.

[0186] Optionally, since the electronic device can periodically capture images and perform face / eye detection on the images, it can obtain changes in the orientation of the face / eye in a timely manner when the orientation of the face / eye changes, and can also obtain changes in emotions expressed by the face / eye, so that the display of the screen-off display object can be updated in a timely manner to achieve the effect of dynamically changing the screen-off display object.

[0187] In one possible implementation, the screen-off display object includes a first component, and the first component is used to represent an eye. In step 1603, when the orientation of the face changes, the electronic device can change the first component in the screen-off display object according to the change (such as changing its position, state or shape, etc.), so that the display position or line of sight direction of the changed first component matches the orientation of the current face or eye. Optionally, the above-mentioned first component includes an eyeball in the eye, and accordingly, the electronic device can change the display position of the eyeball in the eye according to the change in the orientation of the face / eye, so that the line of sight of the screen-off display object changes with the change in the orientation of the user's face / eye, so that the user feels that the screen-off display object is always looking at the user.

[0188] Exemplarily, the effect of the line of sight of the screen-off display object changing with the orientation of the user's face / eyes can be made smoother by playing an animation. Taking the change of a face from a first orientation to a second orientation as an example, during this process, the electronic device can play a first animation, and the first animation includes N frames (N is an integer greater than 1). Each of the N frames contains the screen-off display object. In the N frames, the display state of the screen-off display object in the first frame matches the first orientation, the display state of the screen-off display object in the last frame matches the second orientation, and the display state of the screen-off display object in the second frame to the N-1th frame matches an intermediate orientation in the transition process from the first orientation to the second orientation. By playing the first animation, a smooth dynamic change effect can be obtained.

[0189] A possible example can be as described in the above scenario 1 (for details, see Figure 8 ).

[0190] Optionally, the above-mentioned screen-off display object may also include a second component, and the second component is used to express emotions. In step 1503, the electronic device may also determine the emotion expressed by the face or eyes based on face or eye detection, and update the second component according to the emotion (such as updating the position, state or shape of the second component), so that the emotion expressed by the screen-off display object matches the emotion expressed by the current face or eyes. A possible example may be as described in the above-mentioned scenario 2 (for details, see Fig. 9 ) or scenario three (for details, please refer to Fig.10 ).

[0191] In another possible implementation, the screen-off display object includes a three-dimensional object. An example of the three-dimensional object may be as follows: Figure 2 In step 1603, when the electronic device detects that the orientation of the face or the eye of a person has changed, the orientation of the three-dimensional object may be changed accordingly, so that the orientation of the changed three-dimensional object matches the orientation of the face or the eye of a person, thereby achieving a naked-eye 3D dynamic display effect. A possible example may be as described in the above scenario 4 (for details, see Fig.11 ).

[0192] In another possible implementation, the screen display object includes a text object, which is used to display at least two texts, and when the text object is rotated, part of the at least two texts is displayed and / or part of the text is hidden. An example of the text object can be as follows: Figure 3 In step 1503, when the electronic device detects that the orientation of the face or eyes of the person has changed, the text object rotates to present the hidden text. Exemplarily, when the electronic device detects that the orientation of the face or eyes of the person is a first orientation, the text object rotates in a direction opposite to the first orientation to present the hidden text.

[0193] Taking the text object as a three-dimensional object as an example, the first surface of the three-dimensional object displays the first text, and the second surface of the three-dimensional object displays the second text. When the electronic device detects that the orientation of the face or eyes of a person changes from the first orientation to the second orientation, the text object rotates from the first angle to the second angle; or, when the electronic device detects that the orientation of the face or eyes of a person changes from the second orientation to the first orientation, the text object rotates from the second angle to the first angle. Wherein, at the first angle, the first surface of the text object is displayed and the second surface is hidden, and at the second angle, the second surface of the text object is displayed and the first surface is hidden.

[0194] Optionally, the text object is displayed as a hollow effect, and the hollow portion of the text object reveals a portion of the image of the desktop wallpaper of the electronic device.

[0195] A possible example can be as described in Scenario 5 above (for details, see Fig.13 or Fig.15 ).

[0196] In the embodiment of the present application, when the electronic device satisfies the requirement of switching to the bright screen mode (for example, the user turns on the screen by touching the screen), and the display screen of the electronic device is unlocked, the electronic device switches from the off screen mode to the bright screen mode. In the bright screen mode, the desktop and the desktop background are displayed on the display screen of the electronic device.

[0197] Optionally, in an embodiment of the present application, during the process of switching from the screen-off mode to the screen-on mode, the electronic device may play a transition animation to achieve a one-shot visual effect, switching from the screen-off mode to the screen-on mode to improve viewing smoothness and provide users with a better visual experience.

[0198] Taking the above-mentioned text object displayed as a hollow effect as an example, the electronic device receives an unlocking operation from the user, such as a touch screen operation or other types of operations, which is not limited in the embodiments of the present application. After the electronic device receives the unlocking operation from the user, in response, the electronic device can switch from the screen-off mode to the screen-on mode through a one-shot visual effect, and display the desktop and desktop wallpaper of the electronic device in the screen-on mode. Since the text object has a hollow display effect, its hollow part will show part of the desktop wallpaper. By playing the transition animation, more parts of the desktop wallpaper can be gradually revealed until the entire desktop wallpaper is revealed, so as to improve smoothness. A possible example can be as described in Scenario Five above (for details, please refer to Fig.14 ).

[0199] In some embodiments of the present application, when the electronic device detects a change in the orientation of a face / eye, it can dynamically generate an off-screen display object that matches the current orientation to achieve the purpose of dynamically displaying the off-screen display object.

[0200] In one possible implementation, the electronic device can generate an off-screen display object that matches the current orientation and refresh the display based on the off-screen display object set by the user and the current orientation of the face / eye based on the AI ​​capability. One implementation of generating an off-screen display object based on the AI ​​capability can be: the off-screen display object set by the user and the current orientation information of the face / eye are input as input information to the AI-based image generation model to generate a new off-screen display object.

[0201] In another possible implementation, the screen-off display object may include multiple layers stacked in sequence, and the screen-off display object may be displayed by stacking the multiple layers. When the electronic device detects a change in the orientation of the face / eye, the screen-off display object may be dynamically changed by changing the position of some of the multiple layers, or moving the some of the layers.

[0202] by Figure 1 For example, the emoticon shown in the figure can be formed by superimposing four layers from top to bottom: glasses, eyeballs, whites of eyes and mouth. The electronic device can calculate the moving direction and amount of the layer where the glasses are located, and the moving direction and amount of the layer where the eyeballs are located, according to the change in the orientation of the face / eyes, and then move the layer where the glasses are located and the layer where the eyeballs are located according to the calculation results, so that the new emoticon formed by superimposing the current four layers can match the current orientation of the face / eyes.

[0203] In another possible implementation, the screen display object may include multiple layers superimposed in sequence, and the screen display object may be displayed by superimposing the multiple layers. When the electronic device detects a change in the orientation of the face / eye, the corresponding layer in the multiple layers may be replaced with a new layer to achieve the effect of dynamically changing the screen display object.

[0204] by Figure 1 For example, the emoticon shown in the figure can be formed by superimposing four layers from top to bottom: glasses, eyeballs, whites of eyes and mouth. The electronic device can determine the change from sad emotion to happy emotion based on the emotional change shown by the face / eyes of the person, and the electronic device replaces the layer where the mouth is located with a new layer, that is, replaces the original downward-bending mouth with an upward-bending mouth, so that the new emoticon formed by the superposition of the current four layers can match the emotion shown by the current face / eyes.

[0205] It can be understood that the above only provides several exemplary methods for dynamically updating the screen-off display object, and the embodiments of the present application do not limit this.

[0206] See also Fig.17 , is a flow chart of a display method in the screen-off mode provided in an embodiment of the present application, and the flow can be Figure 6 or Figure 7 The electronic device shown in the figure may be executed, or the functional module, processor or chip in the electronic device may be executed. The following process is described with the electronic device as the execution subject. As shown in the figure, the process may include the following steps:

[0207] Step 1701: The electronic device enters the screen-off mode.

[0208] In an embodiment of the present application, the screen off function can be turned on in advance, and the EOD function can be turned on, so that when the electronic device meets the screen off conditions (such as not using the electronic device within a set time), it can automatically enter the screen off mode.

[0209] Step 1702: When the electronic device detects that a person's eyes are gazing at the display screen, a screen-off display object is displayed on the display screen in the screen-off mode.

[0210] In the embodiment of the present application, after the electronic device enters the screen-off mode, when the electronic device detects that a person is looking at the display screen, the screen-off display object is displayed on the display screen. The setting method of the screen-off display object can be as described above.

[0211] Optionally, the electronic device can capture images through a front camera, and the electronic device can determine whether the human eye is looking at the display screen of the electronic device by executing an eye tracking algorithm on the image.

[0212] Step 1703: The electronic device determines the orientation of the face / eye based on face / eye detection, and the orientation of the face / eye is associated with the relative position between the face / eye and the electronic device.

[0213] The specific implementation method of this step can be found in Fig.16 Related content in .

[0214] Step 1704: The electronic device dynamically changes the screen display object according to the change in the orientation of the face / eyes.

[0215] The specific implementation method of this step can be found in Fig.16 Related content in .

[0216] Fig.17The process shown can achieve the "air triggering" effect of dynamically changing the display of the screen-off display object. Specifically, the user's behavior of looking at the display screen can trigger the display of the screen-off display object. Subsequently, by shaking the electronic device, or by shaking or moving the head, the dynamic change of the screen-off display object can be triggered.

[0217] In some embodiments of the present application, the processing operations of face / eye detection performed in the screen-off mode and the processing operations of dynamically displaying the screen-off display object based on the changes in the orientation of the face / eye can be executed by a low-power AI processor (or low-power AI chip) to reduce the power consumption of the electronic device.

[0218] In the above embodiments of the present application, in the screen-off mode, the screen-off display object can dynamically change with the user's current mood changes, changes in the gaze direction, changes in the electronic device holding angle, etc., thereby meeting the user's demand for personalization, broadening the interesting interactive scenes in the screen-off mode, allowing users to watch the rich and delicate interactive effects in the picture, and bringing users a better visual experience. In addition, it can also effectively reduce power consumption and screen burn-in risks in the screen-off mode.

[0219] Fig.18 The schematic diagram of the structure of a possible device provided in the embodiment of the present application. The device can be used to implement the functions of the electronic device in the above method embodiment, and thus can also achieve the beneficial effects of the above method embodiment. In the embodiment of the present application, the device can be as follows Figure 6 The electronic device shown may also be a module (such as a chip) used in a corresponding device.

[0220] like Fig.18 As shown, the device 1800 includes a processing unit 1810 and a transceiver unit 1820. The device 1800 is used to implement the above Fig.16 or Fig.17 The functions of the terminal device or the sequencing system in the method embodiment shown.

[0221] Optionally, the transceiver unit may include a receiving unit and a sending unit. The receiving unit may receive information, such as signaling or data, and the sending unit may send information, such as signaling or data, under the control of the processing unit.

[0222] When the device 1800 is used to implement Fig.16When the functions of the electronic device in the method embodiment shown are: the processing unit 1810 is used to control the electronic device to enter the screen off mode, and display the screen off display object on the display screen in the screen off mode; the processing unit 1810 determines the orientation of the face or the eye based on face or eye detection, and the orientation is associated with the relative position between the face or the eye and the electronic device; according to the change of the orientation of the face or the eye, the screen off display object is dynamically changed.

[0223] When the device 1800 is used to implement Fig.17 The functions of the electronic device in the method embodiment shown are: the processing unit 1810 is used to control the electronic device to enter the screen off mode; when it is detected that human eyes are looking at the display screen, a screen off display object is displayed on the display screen in the screen off mode; based on face or eye detection, the orientation of the face or eye is determined, and the orientation is associated with the relative position between the face or eye and the electronic device; according to the change of the orientation of the face or eye, the screen off display object is dynamically changed.

[0224] For a more detailed description of the processing unit 1810 and the transceiver unit 1820, please refer to Fig.16 or Fig.17 The relevant description in the method embodiment shown is directly obtained and will not be repeated here.

[0225] The embodiment of the present application also provides a device, which includes a processor and an interface circuit. The processor and the interface circuit are coupled to each other. It is understandable that the interface circuit can be a transceiver or an input-output interface. Optionally, the transceiver can include a receiver and a transmitter. Optionally, the device can also include a memory for storing instructions executed by the processor or storing input data required by the processor to execute instructions or storing data generated after the processor executes instructions. When the device is used to implement Fig.16 or Fig.17 When the method is shown, the processor is used to implement the functions of the above processing unit, and the interface circuit is used to implement the functions of the above transceiver unit.

[0226] When the above device is a chip applied to an electronic device, the chip implements the functions of the electronic device in the above method embodiment. The chip receives information from other modules (such as a radio frequency module or an antenna) in the electronic device, and the information is sent by the network device to the electronic device; or the chip sends information to other modules (such as a radio frequency module or an antenna) in the electronic device, and the information is sent by the electronic device to the network device.

[0227] It is understandable that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0228] In the present application, another example of a device is provided, which includes at least one processor and at least one memory, wherein the at least one processor is coupled to the at least one memory, and the at least one memory is used to store instructions, and when the instructions are executed by the at least one processor, the communication device executes the method in the above embodiment.

[0229] The method steps in the embodiments of the present application can be implemented in hardware or in software instructions that can be executed by a processor. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, register, hard disk, mobile hard disk, CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. The storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device or a terminal. The processor and the storage medium can also be present in a network device or a terminal as discrete components.

[0230] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc; it may also be a semiconductor medium, for example, a solid-state hard disk. The computer-readable storage medium may be a volatile or nonvolatile storage medium, or may include both volatile and nonvolatile types of storage media.

[0231] In the various embodiments of the present application, unless otherwise specified or provided for in any logical conflict, the terms and / or descriptions between the different embodiments are consistent and may be referenced to each other, and the technical features in the different embodiments may be combined to form new embodiments according to their inherent logical relationships.

[0232] In the present application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the text description of the present application, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship; in the formula of the present application, the character " / " indicates that the previous and next associated objects are in a "division" relationship. "Including at least one of A, B and C" can mean: including A; including B; including C; including A and B; including A and C; including B and C; including A, B and C.

[0233] It is understood that the various numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application. The size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic.

Claims

1. A display method in screen-off mode, using an electronic device with a display screen, characterized in that: include: Entering a screen-off mode, and displaying a screen-off display object on the display screen in the screen-off mode; Determining the orientation of a face or eye based on face or eye detection, wherein the orientation is associated with a relative position between the face or eye and the electronic device; The screen-off display object is dynamically changed according to the change of the orientation of the human face or the human eyes.

2. A display method in screen-off mode, using an electronic device with a display screen, characterized in that: include: Enter screen off mode; When it is detected that human eyes are gazing at the display screen, displaying a screen-off display object on the display screen in the screen-off mode; Determining the orientation of a face or eye based on face or eye detection, wherein the orientation is associated with a relative position between the face or eye and the electronic device; The screen-off display object is dynamically changed according to the change of the orientation of the human face or the human eyes.

3. The method according to claim 1 or 2, characterized in that: The screen-off display object includes a first component, and the first component is used to represent eyes; The dynamically changing the screen-off display object according to the change of the orientation of the face or the eyes of the person, comprises: When the orientation of the human face or the human eye changes, the first component is changed according to the change, so that the display position or sight direction of the changed first component matches the orientation of the human face or the human eye.

4. The method according to claim 3, characterized in that The first component includes an eyeball in an eye; The step of changing the first component according to the change includes: The display position of the eyeball in the eye is changed according to the change.

5. The method according to any one of claims 3-4, characterized in that: The screen-off display object includes a second component, and the second component is used to express emotions; The method further comprises: Based on face or eye detection, the emotion expressed by the face or the eyes is determined, and the second component is updated so that the emotion presented by the screen-off display object matches the emotion expressed by the face or the eyes.

6. The method according to claim 5, characterized in that The determining the emotion expressed by the face or eyes based on face or eyes detection includes: Artificial intelligence (AI) capabilities are used to detect faces or eyes to determine the emotions expressed by the faces or eyes.

7. The method according to any one of claims 1 to 6, characterized in that: The change in the orientation of the face or the eyes includes: the face or the eyes change from a first orientation to a second orientation; The dynamically changing the screen display object according to the change of the method of the human face or the human eyes includes: Play a first animation, wherein the first animation includes N frames, where N is an integer greater than 1, and each of the N frames includes the screen-off display object; among the N frames, a display state of the screen-off display object in a first frame matches the first orientation, a display state of the screen-off display object in a last frame matches the second orientation, and a display state of the screen-off display object in a second frame to an N-1th frame matches an intermediate orientation in a transition process from the first orientation to the second orientation.

8. The method according to any one of claims 1 to 7, characterized in that: The screen-off display object includes a three-dimensional object; The dynamically changing the screen-off display object according to the change of the orientation of the face or the eyes of the person, comprises: When the orientation of the human face or the human eyes changes, the orientation of the three-dimensional object is changed accordingly, so that the orientation of the three-dimensional object after the change matches the orientation of the human face or the human eyes.

9. The method according to claim 1 or 2, characterized in that: The screen-off display object comprises a rotatable text object, the text object is used to display at least two texts, and when the text object is rotated, part of the at least two texts is displayed and / or part of the text is hidden; The dynamically changing the screen-off display object according to the change of the orientation of the face or the eyes of the person, comprises: When the orientation of the human face or the human eyes changes, the text object rotates to present the hidden text.

10. The method according to claim 9, characterized in that When the orientation of the human face or the human eyes changes, the text object rotates to present the hidden text, including: When the orientation of the human face or human eyes is a first orientation, the text object rotates in a direction opposite to the first orientation to present the hidden text.

11. The method according to any one of claims 9 to 10, characterized in that: The text object is a three-dimensional object, a first surface of the three-dimensional object displays a first text, and a second surface of the three-dimensional object displays a second text; When the orientation of the human face or the human eyes changes, the text object rotates to present the hidden text, including: When the orientation of the human face or the human eyes changes from a first orientation to a second orientation, the text object rotates from a first angle to a second angle; or When the orientation of the human face or the human eye changes from the second orientation to the first orientation, the text object rotates from the second angle to the first angle; Wherein, at the first angle, the first surface of the text object is displayed, and at the second angle, the second surface of the text object is displayed.

12. The method according to any one of claims 9 to 11, characterized in that: The text object is displayed as a hollowed-out effect, and the hollowed-out portion of the text object reveals a portion of the image of the desktop wallpaper of the electronic device; The method further comprises: In response to the unlocking operation, the screen is switched from the off-screen mode to the on-screen mode through a one-shot visual effect, and the desktop of the electronic device and the desktop wallpaper are displayed.

13. The method according to any one of claims 1 to 12, characterized in that: Also includes: Acquiring information detected by the electronic device using a front camera or a first sensor; Perform face or eye detection based on the detected information.

14. The method according to any one of claims 1 to 13, characterized in that: The change in the orientation of the human face or the human eyes is caused by the movement of the user or the movement of the electronic device.

15. A device, characterized in that: The device comprises a memory, a processor and a display device, wherein the memory is used to store a computer program, and the processor is used to read the computer program so that the device implements the method according to any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a computer, the computer is caused to implement the method according to any one of claims 1 to 14.

17. A computer program product, characterized in that The computer program product comprises computer-readable instructions, and when the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 14.

18. A chip, characterized in that: The chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the method according to any one of claims 1-14.

Images